Configure Varnish logs (varnishnsca), Logrotate and Awstats on Debian Squeeze

Introduction

Assuming you have varnish installed on your server and you started to catch the pages on it.Your webserver is now missing the hits to your pages.

Well, that is great for your webserver load, but not too good for your statistics analysis, because /var/log/apache/access_log will miss the hit.

Configure varnishncsa:

We’ll use varnishncsa to get the logs that awstats will be able to analyse like:

varnishncsa [?a] [?b] [?C] [?c] [?D] [?d] [?f] [?I regex] [?i tag] [?n varnish_name] [?P file] [?r file] [?V] [?w file] [?X regex] [?x tag]

Add this line in the /etc/rc.local file:

varnishncsa -a -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid

-a: To append the logs to an already existing file
-w: To write the logs to the /var/log/varnish/access.log file
-D: To run varnishncsa as a daemon
-P: To write the PID file in the /var/run/ folder

Configure logrotate:

Create the following file /etc/logrotate.d/varnish and put this contents on it:

        /var/log/varnish/*log {
                create 640 http log
                compress
                postrotate
                        /bin/kill -USR1 `cat /var/run/varnishncsa.pid 2>/dev/null` 2> /dev/null || true
                endscript
        }

If you need more info or options run:  man logrotate

We now have varnish writing logs to a file, and logrotate will rotate them everyday, we only need now to analyse them.

Configure awstats with varnish:

Download and install awstats

Get the lastest version from here,  install it basically all you need to do is to copy the contents of the .tar.gz file in /usr/local/awstats/ folder.

Next run the awstats_configure.pl tool. Skip (write none) in the webserver config file question, then follow the questions and you will end up with a config file like: /etc/awstats/awstats.your-domanin.conf and should look more or less like this:

LogFile="/var/log/varnish/access.log"
LogType=W
LogFormat=1
LogSeparator=" "
SiteDomain="www.debian-tutorials.com"
HostAliases="debian-tutorials.com www.debian-tutorials.com 127.0.0.1 localhost"
# Possible values:
# 0 - No DNS Lookup
# 1 - DNS Lookup is fully enabled
# 2 - DNS Lookup is made only from static DNS cache file (if it exists)
# Default: 2
DNSLookup=1
DirData="/var/lib/awstats"
DirIcons="/icon"
AllowToUpdateStatsFromBrowser=0
AllowFullYearView=2

Be sure the read the example located at /usr/share/doc/awstats/examples/awstats.model.conf.gz to fine tune your configuration.

Make awstats update hourly:

Create the file /etc/cron.hourly/awstats and copy these lines inside the file:

#!/bin/sh
/usr/lib/cgi-bin/awstats.pl -update -config=www.your-domain.com

That will update your awstats database hourly, let’s now make if visible, better only for you:

Copy this folder to your www root directory:

cp -R  /usr/share/awstats/ /var/www-root/

Now create etc/apache2/awstats.conf  file and add this lines:

ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
ScriptAlias /awstats/ /usr/lib/cgi-bin/
Options ExecCGI -MultiViews +SymLinksIfOwnerMatch

AllowOverride None Options None AuthType Basic AuthName ‘Private scripts’ AuthUserFile ‘/www-root/.htpasswd’ Require valid-user Order allow,deny Allow from all

Now and add the following line at the end of the  /etc/apache2/apache2.conf file :

Include /etc/apache2/awstats.conf

Create a user that will be able to read the stats:

httpasswd /www-root/.htpasswd username

NOTE: Replace  /www-root/.htpasswd with the file and folder where you want the password file to be in, but be sure the file and upwards folders are owned by the user that runs your Apache server.

Final Step:

Restart Apache, and start varnishnsca.

/etc/init.d/apache2 restart
varnishncsa -a -w /var/log/varnish/access.log -D -P /var/run/varnishncsa.pid