Install Varnish (with Drupal and Nginx) - developed.be

This articles covers the installation and configuration of Nginx+Varnish+Drupal6(Pressflow) and also about the Purge module.

This article assumes:

  • you are no Drupal newbie
  • you know about the default Drupal caching
  • you have some experience with Nginx (if you have Apache as webserver, this tutorial might help you too.)
  • you know what Varnish is.

Note: I’m still updating this post. It’s a bit messy right now.

Install Varnish (the easy part)

1)

curl http://repo.varnish-cache.org/debian/GPG-key.txt | apt-key add

2)

sudo su

(enter password)

3)

echo "deb http://repo.varnish-cache.org/ubuntu/ lucid varnish-2.1" >> /etc/apt/sources.list 

Note: I installed 2.1, The latest version is 3.0. You can install 3.0 by altering the above line.

(see: https://www.varnish-cache.org/installation/ubuntu )

Config varnish (the hard part)

4)

gedit /etc/default/varnish

Set the port in the config to :80

DAEMON_OPTS=”-a :80 \

5) edit your webserver to listen on port 8080

nginx: open /etc/nginx/sites-enabled/default and change

listen    80 default;

to:

listen    8080 default;

If you have other config-files in sites-enabled, then you have to change those too.

6)

gedit /etc/varnish/default.vcl

make sure the beginning looks like this:

backend default {
  .host = "127.0.0.1";
  .port = "8080";
}

The config-file is different upon the version you use. Version 3 can use this config file: https://wiki.fourkitchens.com/display/PF/Configure+Varnish+for+Pressflow

Version 2.x users will need to have those two changes:

(If you don’t know which version you have, open Synaptic Package Manager and search for “varnish”. It will display the version number)

At: (‘input’ Line 31 Pos 9)

    set req.hash += req.http.Cookie;

instead of: hash_data (req.http.Cookie);

(‘input’ Line 114 Pos 17)

      <title>"} obj.status " " obj.response {"</title>

And futher change “+” concat symbols to two double quotes

<pre>Status: "}  obj.status {"
Response: "}  obj.response  {"
XID: "}  req.xid  {"</pre>

7) Don’t mess up these steps:

    a) STOP your webserver (not reload, stop)

    /etc/init.d/nginx stop

    b) restart Varnish

    /etc/init.d/varnish restart

    It’s very important that varnish doesn’t display any errors         upon restarting, or it won’t work.

    c) START your webserver.

    /etc/init.d/nginx start

If your webserver (nginx) complains that port 80 is taken, it means that a server-config is still listening to port 80. You have to make sure that nothing of nginx listens to port 80, incl all your sites. (see step 3)

[emerg]: bind() to 0.0.0.0:80 failed (98: Address already in use)

[emerg]: still could not bind()

9) You should be able to see your site from http://localhost. If that wouldn’t work, you can still access your site through http://localhost:8080, this way you call your site through your webserver, and not through varnish.

Troubleshooting

If you see something else than expected, then test if varnish is functional. You can do this by adding the following line to varnish default.conf after sub vcl_recv {

sub vcl_recv {
    error 403 "Forbidden by Varnish (test)";
    ...

Restart the setup according to step 7, go to http://localhost and see if you get error 403. If not, Varnish isn’t working properly. Try the previous steps again or look elsewhere on the internet for more info.

Config Drupal

1) Download (or GIT clone) Pressflow

2) Install or overwrite your site with the code from pressflow

3) Check http://localhost and fill in the setup wizard (if new site)

So far you have a normal Drupal site, without session cookies, but there’s no Varnish implementation yet.

4) Download/install varnish module: http://drupal.org/project/varnish (Note: if you want to use Purge, you don’t have to install this module, but as part of the learning curve, you better do so)

5) Download/install Cache Expiration

6) Enable modules: “Coockie cache bypass” and “Path alias cache”. These are already in the Pressflow installation

7) go to /admin/settings/varnish

Enter the values:

* Varnish Control Terminal (this is the value of DAEMON_OPTS – T (in the /etc/default/varnish config file)

* Varnish Control Key: Copy past the content of /etc/varnish/secret

8) Save, the status on the page should indicate: “Varnish running”.

Change your site

Your site can’t absotely have no session or session/tracking cookie. That includes google analytics.

Does it work?

To check wheter it works you can add a random generated string on every page (with php ). By refreshing your page, the string should remain the same.

Troubleshooting:

Use Tamper Data (Firefox plugin), Fiddler or Google Chrome Developer Tools to see if the site writes any session cookies. If it does, find out what modules writes the cookies and disable it.

Clearing the cach

Without Varnish installed

By default Drupal has a caching mechanism that saves html-cache-data in your database.

(To speed that up, you can use memcache. It keeps cache in the RAM instead of in the database. This tutorial isn’t about memcache, though)

By default, every cron cycle will wipe the entire cache. I have a tutorial on how to disable that.

With Varnish installed

(Parts) of the default Drupal caching will still be used with Varnish in front (eg: menu_cache). Varnish is an extra layer of cache, not a replacement of the default Drupal cache.

By installing the Varnish and  Cache Expiration module you can set how the Varnish cache should be cleared.

In settings/varnish

  • Disable to flush cache on cron
  • Use Varnish cache clearing:  This is an important option.
    • For small sites, it might be useful to use “Drupal Default“. This will flush the entire cache when cron runs.
    • Big sites (> 10.000 pages), should not use the Drupal default, but the “selective (expire)”. This is more or less a synonym for saying: use expire. The label “experimatal” can be ignored.
      • Say you take “selective (expire), in settings/performance/expire, I would check every option.
    • The real nerd of course use “None“, and use your own cache flush mecanism.

In settings/performance

  • Set caching mode: external (meaning: Varnish)

Purging

The options in the Varnish module are limited. You better use “Purge”. You can use Purge with Expire.

An HTTP purge is similar to a HTTP GET request, except that the method is PURGE.

(varnish-cache.org)

A purge clears a page from the cache “at lightning speed”. This is were Varnish performs better than Boost.

Drupal needs to be able to call Varnish (which they call: the proxy server) to purge (~ “clear”) a url from the cache.

Nginx: By default Nginx doesn’t know how to purge, so it needs an extra module, which requires a recompilation of nginx.

Before your click away and decide to use Boost: the nginx compilation is rather easy.

(Apache has also a “purge” possibilty. This tutorial does not cover that)

Installation of Purge

Nginx installation

See: http://mcnearney.net/blog/2010/2/28/compiling-nginx-cache-purging-support/ for an excellent tutorial.

Recap:

1) backup the entire /etc/nginx

2) install necessary compile tools

aptitude -y install build-essential libc6 libpcre3 libpcre3-dev libpcrecpp0 libssl0.9.8 libssl-dev zlib1g zlib1g-dev lsb-base

3) download and extract both nginx and cache_purge.

cd /usr/src/
wget http://www.nginx.org/download/nginx-0.7.65.tar.gz
wget http://labs.frickle.com/files/ngx_cache_purge-1.0.tar.gz
tar -xvf nginx-0.7.65.tar.gz
tar -xvf ngx_cache_purge-1.0.tar.gz
cd nginx-0.7.65/

4)

nginx -V (copy /paste that, -V  = Version)

5)

./configure --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --pid-path=/var/run/nginx.pid --lock-path=/var/lock/nginx.lock --http-log-path=/var/log/nginx/access.log --http-client-body-temp-path=/var/lib/nginx/body --http-proxy-temp-path=/var/lib/nginx/proxy --http-fastcgi-temp-path=/var/lib/nginx/fastcgi --with-debug --with-http_stub_status_module --with-http_flv_module --with-http_ssl_module --with-http_dav_module --with-http_gzip_static_module --with-http_realip_module --with-mail --with-mail_ssl_module --with-ipv6 --add-module=/usr/src/ngx_cache_purge-1.0

6)

make && make install

7)

/etc/init.d/nginx stop

8)

cp objs/nginx /usr/sbin/nginx
/etc/init.d/nginx start

9)

check if it works:

nginx -V

Drupal installation

1) Install the Purge module

http://drupal.org/project/purge

(you will see that the requirements covers we are doing in this tutorial)

2) The “Varnish” module is not necessary with Purge. You could disable it.

Nginx config

See: http://labs.frickle.com/nginx_ngx_cache_purge/

1) Add this in your main nginx config file, in the http section:

proxy_cache_path   /tmp/cache   keys_zone=tmpcache:10m;

2) Merge this with your site config file:

location / {
    proxy_pass        http://127.0.0.1:8000;
    proxy_cache        tmpcache;
    proxy_cache_key        $uri$is_args$args;
}
 
location ~ /purge(/.*) {
    allow            127.0.0.1;
    deny            all;
    proxy_cache_purge    tmpcache   $1$is_args$args;
}

3) Restart nginx.

4) You should be able to call yoursite.com/purge/whatever and not end up in Drupal

Varnish config

This piece of code should be in your defaul.vcl file:

See: https://www.varnish-cache.org/docs/2.1/tutorial/purging.html

acl purge {
        "localhost";
        "192.168.55.0/24";
}
 
sub vcl_recv {
        # allow PURGE from localhost and 192.168.55...
 
        if (req.request == "PURGE") {
                if (!client.ip ~ purge) {
                        error 405 "Not allowed.";
                }
                return (lookup);
        }
}
 
sub vcl_hit {
        if (req.request == "PURGE") {
                # Note that setting ttl to 0 is magical.
                # the object is zapped from cache.
                set obj.ttl = 0s;
                error 200 "Purged.";
        }
}
 
sub vcl_miss {
        if (req.request == "PURGE") {
 
                error 404 "Not in cache.";
        }
}

Test purge

To test purge, use the curl command (see README.txt from the purge module):

curl -X PURGE -H "Host: mysite.be" http://mysite.be/taxonomy/term/128

The url is of course a page of my website (can be random)

2) You should see a html page printed in the terminal with “not in cache”

3) Visit the page (as an anonymous user) in your browser

4) Try the purge command again. You should see: “Response: purged”, inbetween some html

5) Visit the page in the browser again. It should create a new version of the page.

Config Drupal

Now, Drupal won’t purge through the curl command as we previously did, but through http.

1) Make sure you have proper rights

/admin/user/permissions

activate “administer purge” for your account.

2) Go to admin/settings/purge and fill in the proxy url

(this is the url to varnish)

I tried: http://local.dewereldmorgen.be

Test Drupal

1) Request a taxonomy page or the homepage as an anonymous user.

2) Make sure you see a Varnished cache page

3) Edit any node on the page.

4) Check if it updates/

Troubleshooting

Check admin/reports/dblog for errors (type: purge)

=> I had to do a small change in the code. @purge_log should be !purge_log

When you’ve visited a site before and you have cookie tracking on (like Google Analytics) the cookies might still be there, avoiding Varnish to cache. Clear your domain cookies and try again. With Chrome it’s the easiest to do so.

Additional custom programming

If Expire by default doesn’t suffice your needs, you can still make your own script togheter or without Expire.

expire_node(&$node) is the key function to purge a node. The function clears “all related pages” to a node that need to be refreshed. Too bad, complex website need more than just the traditional taxonomy terms cleared. (for example: page of the author). I can’t find any hooks in to, so you might write a patch. (more on that soon)


Rss Comments

Comments

No comments yet.

Leave a comment