A Guide to Varnish VCL

I’ve been working with Varnish 2.0 for the last two weeks, going from complete n00b to someone who knows enough to feel I can improve the terrible lack of documentation for Varnish and VCL. There’s not a lot out there and what’s there is hard to find and sometimes erroneous. I’m hoping this post will help others like me who are struggling with Varnish and VCL.

Basics

VCL is essentially a set of stubs which you can override to provide your own behavior. It is very limited in what it can do, primarily for performance reasons. You don’t have access to the filesystem and the language has no variables or loops.

The two stubs you will most often use:

  • vcl_recv – called at the start of a request. This is primarily used to canonicalize the input URL and headers, determine whether to bypass the cache, etc.
  • vcl_fetch – called when the response has been gathered from the backend before placing it in the cache. You can configure a grace period, enable ESI processing, configure different TTLs, remove user-specific cookies, etc before inserting the response into the cache.

Examples

The Varnish VCL examples are rather sparse; here’s a few more which may fill in some gaps. These work with Varnish 2.0.4.

# If the requested URL starts like "/link/?" then immediately pass it to the given
# backend and DO NOT cache the result ("pass" basically means "bypass the cache").
if (req.url ~ "^/link/?") {
  set req.backend = web;
  pass;
}
if (req.url ~ "/$") {
  # Handle URLs with a trailing slash by appending index.html
  # (Useful if you are pulling from S3 which does not have default document logic)
  # Note there's no explicit string append operator.
  set req.url = req.url "index.html";
}
# strip port from the Host header
# (useful when testing against a local Varnish instance on port 6081)
set req.http.Host = regsub(req.http.Host, ":[0-9]+", "");

# /foo/bar.embed -> /foo/bar/embed.js
set req.url = regsub(req.url, "(.*).embed$", "1/embed.js");

# Support feed URLs of the form "/foo/bar.atom" --> "/foo/bar/feed.atom"
if ((req.url ~ ".(rss|atom)$") && !(req.url ~ "feed.(atom|rss)$")) {
    set req.url = regsub(req.url, "(.*).(.*)$", "1/feed.2");
}

The biggest pain in all of this was the very limited logic you can perform on req.url. You don’t have variables in VCL so you need to think in terms of regular expression groups like in the RSS/ATOM regexp above when trying to restructure the URL.

  # use this in vcl_fetch, don't want 404s filling up our cache, so just
  # immediately return a client error and bypass the cache.
  if (obj.status == 404) {
    error 404 "No such file";
  }

Resources

Here’s the best VCL resources I could find:

Good luck!

6 thoughts on “A Guide to Varnish VCL”

  1. First of all, thanks for some useful examples and feedback.

    I agree that the documentation available is a bit diversified but there’s quite alot of it. Maybe that’s part of the problem?

    My primary source for VCL documentation is the vcl manual that comes with Varnish, which should contain all VCL constructs and be up to date. Some times it gets left behind for a few commits, but it’s usually pretty up to date. The Wiki has a few examples and stubs, some of these stubs might be a bit old, if you spot such a page, feel free to fix it or just drop by #varnish at irc.linpro.no or drop us an e-mail at the public mailinglists and we’ll sort it out.

    As for the flexibility of VCL what you have to remember is that VCL isn’t meant to be a fully fledged programming language, and if you compare it to such a language, it will almost always fall short. However, VCL itself is rarely a hindrance. One of our main policies while developing Varnish is to leave policy decisions to VCL, and I belive we’ve been able to do that quite well so far.

    And just in case you really want something that VCL can’t do, you can always resort to inline C code.

    However, mastery of regular expressions makes a significant difference when it comes to writing VCL, and it’s not necessarily easy.

    As for your examples, they are definitely useful. I’ll say they are typical examples of things one can expect to do when working with Varnish.

  2. Hi!

    When we tested a simple VCL filtering IPs, we were stunned to see how much slower Varnish is compared to Gwan, which also uses C for custom behaviors. But we got similar results with an Nginx module, Varnish being so much slower.

    Can you elaborate on why Varnish is so much slower than webservers?

    Our config:

    thread_pool_add_delay = 2
    thread_pools = 4
    thread_pool_min = 200
    thread_pool_max = 5000
    cli_timeout = 15
    session_linger = 100
    malloc = 1G

  3. About a year too late, but what the heck..

    @Frank
    If you’re talking about load testing, the slow speed could have to do with cookies on the site in question. By default, Varnish doesn’t cache anything with cookies, so it would act as a proxy, only adding overhead to the request.

    Your issue could well be something else of course, but this is a common source of confusion when first trying out Varnish.

  4. I’m calling varnish with the url ==> http://cmsfoqua:8180/vgnExtTemplating/stresource?SecurityKey=lVPNsLcf&SiteName=axabanque&ServiceName=DetailLAB&Language=fr&ResourceName=Entete&TTL=3600&CIBLAGE=

    and I would like varnish to rewrite the url like this (http://cmsfoqua/xml/DetailLABEntete.xml) then call the backend.

    Here is my rule:

    set req.url = regsub(req.url, “^/vgnExtTemplating/stresource?SecurityKey=lVPNsLcf&SiteName=axabanque&ServiceName=(.*)&Language=fr&ResourceName=(.*)&TTL=(.*)&CIBLAGE=(.*)$”, “/xml/124.xml”);

    But this is not working, please help.

    Regards,
    Enzo

  5. Here is the answer:

    set req.url = regsub(req.url, “^/vgnExtTemplating/stresource?SecurityKey=lVPNsLcf&SiteName=axabanque&ServiceName=(.*)&Language=fr&ResourceName=(.*)&TTL=(.*)&CIBLAGE=(.*)$”, “/xml/124.xml”);

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>