Tag Archives: Lighttpd

Remote Lighttpd Logging and Cloudflare/Reverse Proxy

Posted by on September 2, 2011 No comments

To keep the ‘one post per year’ tradition alive, heres one for 2011!

Logging the right IP with Lighttpd when behind reverse proxies

When you use a reverse proxy like haproxy, varnish, nginx, squid or cloudflare, lighttpd will log the IP of the reverse proxy instead of the user’s IP

While the lighttpd wiki page says

mod_accesslog: In order to see the “real” ip address in access log, you’ll have to load mod_extforward after mod_accesslog

That didn’t work for me even though I get the proper IP within my application.

With google, I found this

accesslog.format = “%{X-Forwarded-For}i %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”"

To get that to work with Cloudflare, just use %{CF-Connecting-IP}i instead of %{X-Forwarded-For}i

accesslog.format = “%{CF-Connecting-IP}i %l %u %t “%r” %>s %b “%{Referer}i” “%{User-Agent}i”"

You could just use that or set your own custom logging format [https://redmine.lighttpd.net/wiki/lighttpd/Docs:ModAccessLog]

[Using ExtForward with Cloudflare]
——

Lighttpd Remote Logging

Now the remote logging with syslog-ng.
rsyslog had been broken on one of my servers for quite some time and was using up 200%+ cpu for some reason. I came across syslog-ng while looking for an alternative. If you have multiple webservers for a site, its nice to have all the logs in one place

Easy to install on ubuntu with:

sudo apt-get install syslog-ng

You’ll need to do that on all the clients (systems we’re logging from) and the server (the system we’re logging to)

On the server edit /etc/syslog-ng/syslog-ng.conf and add ‘udp(ip(0.0.0.0) port(514));’ to the source s_src part so it looks like this

source s_src { unix-dgram(“/dev/log”); internal();
file(“/proc/kmsg” program_override(“kernel”));
udp(ip(0.0.0.0) port(514));
};

and add this to the bottom of the file

destination lighttpd { file(“/var/log/lighttpd.log”); };
log { source(s_src); filter(f_lighttpd); destination(lighttpd); };
filter f_lighttpd { program(lighttpd); };

And on the clients, at the end of  the /etc/syslog-ng/syslog-ng.conf file, add this

destination remote { udp(“[IP-Address-Of-Remote-Host]” port(514)); };
filter f_lighttpd { program(lighttpd); };
log { source(s_src); filter(f_lighttpd); destination(remote); };

restart syslog-ng with

sudo /etc/init.d/syslog-ng restart

on all clients and server, and it should start logging on the server to /var/log/lighttpd.log

Credits: http://www.ruby-forum.com/topic/94465

————————————–
Duplicate logs in /var/log/syslog
If the httpd logs are going into both /var/log/lighttpd and /var/log/syslog
find ‘filter f_syslog3′ in /etc/syslog-ng/syslog-ng.conf
and replace the whole line with

filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug) and not filter(f_lighttpd); };

do the same for the line with ‘filter f_daemon’

filter f_daemon { facility(daemon) and not filter(f_debug) and not filter(f_lighttpd); };

And restart syslog-ng

See you in 2012 :)

Evading far future expires!

Posted by on December 22, 2010 1 comment

I use rsync to sync files between my computer and my webserver. When I make changes to the static content (mostly css or javascript files), most users still see the cached versions in their browsers thanks to my far futures expires header following advice from yslow.

From what I understand, with a far future expires header, the browser simply loads that content from cache (till it expires) without contacting the server. It would probably have gotten a ’304 – Not Modified’ response anyway. So, you save an HTTP request and the little processing required to verify if both parties have the same file (see ETags)

So, back to my problem with browsers using the old file (by respecting the far future expires header) instead of the new one I just uploaded. Theres two ways to get the browser to load the latest file. Either the page has to be refreshed without cache (by pressing the refresh button, or ctrl+f5 in most browsers) or you’ll have to use a different filename in the html and also rename the file on your server. Waiting for all your users to refresh the page is not a good idea :)

What I do is add a version number (usually the date of update) to the filename within the html and get the webserver (lighttpd in my case) to serve the current file no matter what the version. For example, Instead of

<script src=”/js/filename.js”></script>

I use

<script src=”/js/filename.20101222.js”></script>

The webserver will serve the same filename.js file whether you use filename.20101222.js or filename.20050101.js, but the browser will now be using the current version of the file and this will also cache this file until you update again.

I use lighttpd and here is the config ( for filename.{version}.js )

url.rewrite = (“^/js/([a-zA-Z0-9/]*).([0-9]*).js$” => “/js/$1.js”)

If you want it to be /js/{version}/filename.js

url.rewrite = (“^/js/([0-9]*)/([a-zA-Z0-9/]*).js$” => “/js/$2.js”)

I use only numbers (dates as yyyymmdd), hence the [0-9]. Adjust the regex to whatever versioning you plan to use.

I’m not used to apache, but I’m guessing this is what the settings would look like on it

RewriteRule         ^/js/([a-zA-Z0-9/]*).([0-9]*).js$        /js/$1.js

RewriteRule         ^/js/([0-9]*)/([a-zA-Z0-9/]*).js$         /js/$2.js