Logging the right IP with Lighttpd when behind reverse proxies

When you use a reverse proxy like haproxy, varnish, nginx, squid or cloudflare, lighttpd will log the IP of the reverse proxy instead of the user’s IP

While the lighttpd wiki page says

mod_accesslog: In order to see the “real” ip address in access log, you’ll have to load mod_extforward after mod_accesslog

That didn’t work for me even though I get the proper IP within my application.

With google, I found this

accesslog.format = “%{X-Forwarded-For}i %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”"

To get that to work with Cloudflare, just use %{CF-Connecting-IP}i instead of %{X-Forwarded-For}i

accesslog.format = “%{CF-Connecting-IP}i %l %u %t \”%r\” %>s %b \”%{Referer}i\” \”%{User-Agent}i\”"

You could just use that or set your own custom logging format [https://redmine.lighttpd.net/wiki/lighttpd/Docs:ModAccessLog]

[Using ExtForward with Cloudflare]
——

Lighttpd Remote Logging

Now the remote logging with syslog-ng.
rsyslog had been broken on one of my servers for quite some time and was using up 200%+ cpu for some reason. I came across syslog-ng while looking for an alternative. If you have multiple webservers for a site, its nice to have all the logs in one place

Easy to install on ubuntu with:

sudo apt-get install syslog-ng

You’ll need to do that on all the clients (systems we’re logging from) and the server (the system we’re logging to)

On the server edit /etc/syslog-ng/syslog-ng.conf and add ‘udp(ip(0.0.0.0) port(514));’ to the source s_src part so it looks like this

source s_src { unix-dgram(“/dev/log”); internal();
file(“/proc/kmsg” program_override(“kernel”));
udp(ip(0.0.0.0) port(514));
};

and add this to the bottom of the file

destination lighttpd { file(“/var/log/lighttpd.log”); };
log { source(s_src); filter(f_lighttpd); destination(lighttpd); };
filter f_lighttpd { program(lighttpd); };

And on the clients, at the end of  the /etc/syslog-ng/syslog-ng.conf file, add this

destination remote { udp(“[IP-Address-Of-Remote-Host]” port(514)); };
filter f_lighttpd { program(lighttpd); };
log { source(s_src); filter(f_lighttpd); destination(remote); };

restart syslog-ng with

sudo /etc/init.d/syslog-ng restart

on all clients and server, and it should start logging on the server to /var/log/lighttpd.log

Credits: http://www.ruby-forum.com/topic/94465

————————————–
Duplicate logs in /var/log/syslog
If the httpd logs are going into both /var/log/lighttpd and /var/log/syslog
find ‘filter f_syslog3′ in /etc/syslog-ng/syslog-ng.conf
and replace the whole line with

filter f_syslog3 { not facility(auth, authpriv, mail) and not filter(f_debug) and not filter(f_lighttpd); };

do the same for the line with ‘filter f_daemon’

filter f_daemon { facility(daemon) and not filter(f_debug) and not filter(f_lighttpd); };

And restart syslog-ng

See you in 2012

From what I understand, with a far future expires header, the browser simply loads that content from cache (till it expires) without contacting the server. It would probably have gotten a ’304 – Not Modified’ response anyway. So, you save an HTTP request and the little processing required to verify if both parties have the same file (see ETags)

So, back to my problem with browsers using the old file (by respecting the far future expires header) instead of the new one I just uploaded. Theres two ways to get the browser to load the latest file. Either the page has to be refreshed without cache (by pressing the refresh button, or ctrl+f5 in most browsers) or you’ll have to use a different filename in the html and also rename the file on your server. Waiting for all your users to refresh the page is not a good idea

What I do is add a version number (usually the date of update) to the filename within the html and get the webserver (lighttpd in my case) to serve the current file no matter what the version. For example, Instead of

<script src=”/js/filename.js”></script>

I use

<script src=”/js/filename.20101222.js”></script>

The webserver will serve the same filename.js file whether you use filename.20101222.js or filename.20050101.js, but the browser will now be using the current version of the file and this will also cache this file until you update again.

I use lighttpd and here is the config ( for filename.{version}.js )

url.rewrite = (“^/js/([a-zA-Z0-9\/]*)\.([0-9]*)\.js$” => “/js/$1.js”)

If you want it to be /js/{version}/filename.js

url.rewrite = (“^/js/([0-9]*)/([a-zA-Z0-9\/]*)\.js$” => “/js/$2.js”)

I use only numbers (dates as yyyymmdd), hence the [0-9]. Adjust the regex to whatever versioning you plan to use.

I’m not used to apache, but I’m guessing this is what the settings would look like on it

RewriteRule         ^/js/([a-zA-Z0-9\/]*)\.([0-9]*)\.js$        /js/$1.js

RewriteRule         ^/js/([0-9]*)/([a-zA-Z0-9\/]*)\.js$         /js/$2.js

random thought from irc

[d1g1t] I just thought of an awesome blogging platform!
[d1g1t] weird
[d1g1t] but awesome
[d1g1t] plain html pages!
[d1g1t] we just write posts as html in text files in a folder, a program updates pages/links when we add a new file!
[d1g1t] one dynamic file for comments maybe
[d1g1t] rally rally fast
[d1g1t] the program could compress images and js too
[d1g1t] db-less
[d1g1t] might get chaotic for terribly large sites
[d1g1t] if people use comments. otherwise blazing fast!
* d1g1t patents it

Would it work? and what are the downsides?

Image via CrunchBase

Saad Akhtar suggested we use mailing lists to share new Indian webcomics we discover, so that people don’t regularly have to keep checking THIS post. So here it is!

Subscribe to IndianWebComics

Email:

Visit this group

http://groups.google.com/group/indianwebcomics

You can either join by entering your email in the above Box or you can follow it via RSS or Atom feeds. Note that you won’t be able to reply to mails if don’t join the group.

Image via Wikipedia

I was trying to find a way to get live cricket scores on my desktop. I could have used retreived rss feeds every few minutes, but I found a simpler way on a mailing list Here, a python script by Sridhar.

I modified it a bit to fit my horizontal conky, and also the display format.

Click on image to see complete screenshot:

Download cricscore.py
and place it in your home folder.

To get the live score in the terminal, just type

python cricscore.py

If you want it to show in conky, just add this line to your .conkyrc file where you want it to show

${execi 30 python cricscore.py}

that will update the score every 30 seconds

Anyway, heres my .conkyrc script: .conkyrc. (for the rhythmbox script, check here