Include the client IP in apache logs for servers behind a load balancer

When servers are behind a load balancer, Apache’s default configuration produces logs that show the load balancer’s IP address instead of the IP of the remote client that initiated the request. Furthermore, if multiple server’s logs are consolidated into one, it can be difficult to determine which server created a given log entry.

This post covers how to improve on Apache’s default logging situation such that:

  • every web server behind the load balancer includes a unique identifier for itself in its log entries, and that;
  • access log entries include the client’s remote IP address as found in the X-Forwarded-For header set by the load balancer.

This setup is more helpful for troubleshooting, configuring monitoring and alerts, etc. and it helps to maximize the value of 3rd-party log aggregation and analysis services like Papertrail or Loggly).

The example commands in this post are applicable to Ubuntu/Debian however they are easily adapted to other environments.

Enable required modules

Start by ensuring that the required Apache modules: env and remoteip, are enabled:

a2enmod env
a2enmod remoteip 
service apache2 restart

Identify each web server

Add a SetEnv directive in the site/app’s apache conf to instruct the env module to set a new environment variable with a value that uniquely identifies each server behind the load balancer.

The example below is a snippet from an Apache VirtualHost’s conf file. It defines a variable called APP_LB_WORKER with the value ‘unique_identifier’. If you were using a devops automation tool such as Ansible, you could use the template module and use a handy variable such as {{ ansible_host }} in place of the example’s hard-coded ‘unique_identifier’ value.

<VirtualHost *:443>
...
    SetEnv APP_LB_WORKER unique_identifier
...
</VirtualHost>

Configure the Apache RemoteIP module

Create a file named /etc/apache2/conf-available/remoteip.conf and use it to set: the RemoteIPHeader to ‘X-Forwarded-For’, any appropriate RemoteIPInternalProxy or other RemoteIP directives, and to define a new LogFormat that includes both the environment variable that contains the server’s identifier and the client’s remote IP as sourced from the X-Forwarded-For header.

The RemoteIPInternalProxy directive tells the RemoteIP module which IP address(es) or IP address blocks it can trust to provide a valid RemoteIPHeader that contains the client’s IP.

The following example’s RemoteIPInternalProxy value is representative of an environment where the load balancer’s internal network IP address belongs to a public subnet with the CIDR block 10.0.1.0/24. Choose an appropriate value (or values) for your environment.

A full list of configuration directives for RemoteIP can be found in the Apache docs: https://httpd.apache.org/docs/2.4/mod/mod_remoteip.html.

The following example names the new LogFormat as “loadbalance_combined”. You can choose any name you like that isn’t already in use.

/etc/apache2/conf-available/remoteip.conf:

RemoteIPHeader X-Forwarded-For
RemoteIPInternalProxy 10.0.1.0/24

LogFormat "%a %v %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" \"%{APP_LB_WORKER}e\"" loadbalance_combined

Finally, enable the conf:

a2enconf /etc/apache2/conf-available/remoteip.conf

The Apache LogFormat is extensively customizable. Learn more about the placeholders and options from here: https://httpd.apache.org/docs/2.4/logs.html. Remote log aggregation service Loggly also has an excellent overview: https://www.loggly.com/ultimate-guide/apache-logging-basics/.

Tell Apache to use the new LogFormat

Next, tell Apache to use the custom LogFormat that we named loadbalance_combined by editing your site/app’s conf file. The following example builds upon the previous example of a VirtualHost conf:

<VirtualHost *:443>
...
    SetEnv APP_LB_WORKER unique_identifier
...
    CustomLog /var/log/app_name/access.log loadbalance_combined
...
</VirtualHost>

The following example is a more elaborate case that uses the tee command to send the log entry to both an access.log file and to the /usr/bin/logger command to include it in the syslog, only if an environment variable named “dontlog” is not set. An example use-case for an env variable like “dontlog” is to set it (e.g. via the SetEnvIf directive) for any requests that correspond to a “health check” from the load balancer to the web server. This helps keep logs clean and clutter-free.

CustomLog "|$/usr/bin/tee -a /var/log/app_name/access.log | /usr/bin/logger -t apache2 -p local6.notice" loadbalance_combined env=!dontlog

Restart Apache

Confirm the validity of your confs using apachectl configtest and restart apache for the new configuration to take effect:

service apache2 restart