Load Balancing and Scaling with Squid-HTTPD-Tomcat

One of our clients host a content management system built on top of technology I use at work. The technology is a custom server that stacks on top of the J2EE application servers like JBoss or a servlet container like Apache Tomcat. Our standard stack that we prescribe to clients is the MySQL-Apache Tomcat stack.

Over a period of few weeks, the needs of this client grew well beyond the standard stack. Considering the amount of static content being served by the Content Management System, a caching server such as Squid was a logical choice. Soon it was apparent, that this wouldn't scale either. Hence as the next iteration, we prescribed a solution with three Apache Tomcat instances running on 32-bit JVM instances of 1.5GB memory each, being load balanced by Apache HTTPD using mod-jk connnector, and front-ended with a Squid caching server. Since the public side had only read requests, a single MySQL server as the back-end was sufficient.

Being a unique stack in itself, not to mention the abundant complexities involved in between versions and dependencies, I thought it would be great to document the setup for future reference. As of now the stack is working well for the client, with the added ability to bring down one or even two Apache Tomcat instances for maintenance and leaving the site unaffected. The architecture is depicted in Figure 1 below:


The versions for softwares used are:

[root@1233797-app45 ~]# /usr/share/tomcat/bin/version.sh
Using CATALINA_BASE:   /usr/share/tomcat
Using CATALINA_HOME:   /usr/share/tomcat
Using CATALINA_TMPDIR: /usr/share/tomcat/temp
Using JRE_HOME:       /usr/java/jdk1.5.0_17
Server version: Apache Tomcat/5.5.17
Server built:   Apr 14 2006 02:08:29
Server number:  5.5.17.0
OS Name:        Linux
OS Version:     2.6.9-67.0.22.ELsmp
Architecture:   i386
JVM Version:    1.5.0_17-b04
JVM Vendor:     Sun Microsystems Inc.
[root@183787-app9 ~]# rpm -q httpd squid mysql-server
httpd-2.0.52-41.ent.4
squid-2.5.STABLE14-4.el4
mysql-server-5.0.77-1.rs.el4

Following are the steps to setup the servers with reference to Figure 1. Customize it for your local setup.

Edit the /etc/httpd/conf/httpd.conf. Change the IP and port that HTTPD listens on:

Listen 192.168.1.139:8084

Download the mod_jk connector from Apache Tomcat Connectors Download. Since I ran into issues finding pre-packaged RPM for RHEL, I used one of the binaries available from their binaries link on that page. Save the "mod_jk-*.so" file along-with other modules for HTTPD. Append the following lines to the /etc/httpd/conf/httpd.conf

LoadModule jk_module modules/mod_jk.so

JkMount jkstatus
order deny,allow
allow from 127.0.0.1
deny from all



  JkWorkersFile conf/workers.properties
  JkShmFile /var/cache/httpd/mod_jk.shm
  JkLogFile /var/log/httpd/mod_jk.log
  JkLogLevel info
  JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
  JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
  JkRequestLogFormat "%w %V %T"
  SetEnvIf Request_URI "/webmail/*" no-jk
  SetEnvIf Request_URi "/mailman/*" no-jk
  SetEnvIf Request_URI "/awstatsclasses/*" no-jk
  SetEnvIf Request_URI "/awstatscss/*" no-jk
  SetEnvIf Request_URI "/awstatsicons/*" no-jk
  SetEnvIf Request_URI "/awstats/*" no-jk
  JkMount  /* router
  JkMount  /jkmanager/* jkstatus

Create a /etc/httpd/conf/workers.properties file as shown under:

# Define some properties
workers.apache_log=/var/apache/logs
workers.tomcat_home=/usr/share/tomcat
workers.java_home=/usr/java/jrockit-R27.4.0-jdk1.5.0_12
ps=/

# The advanced router LB worker
worker.list=router,tomcat1,tomcat2,tomcat3,jkstatus

# Set properties for tomcat1 (ajp13)
worker.tomcat1.type=ajp13
worker.tomcat1.host=192.168.1.140
worker.tomcat1.port=8010
worker.tomcat1.lbfactor=1

# Set properties for tomcat2 (ajp13)
worker.tomcat2.type=ajp13
worker.tomcat2.host=192.168.1.140
worker.tomcat2.port=8011
worker.tomcat2.lbfactor=1

# Set properties for tomcat3 (ajp13)
worker.tomcat3.type=ajp13
worker.tomcat3.host=192.168.1.140
worker.tomcat3.port=8012
worker.tomcat3.lbfactor=1

# Define the LB worker
worker.router.type=lb
worker.router.balance_workers=tomcat1,tomcat2,tomcat3

worker.jkstatus.type=status

On each of the workers (tomcat1,tomcat2,tomcat3) make the following changes to the /usr/share/tomcatXX/conf/server.xml file where XX indicates the number of tomcat:

• Change the connector configuration so that the port numbers are 8081, 8082, 8083 for tomcat1, tomcat2 and tomcat3 respectively. This helps validate individual tomcat instances.
  
  

• The mod_jk connector talks to the worker instances on the AJP connector port. This port can be specified in the server.xml file as shown below. We picked ports 8010-8012 for tomcat1-tomcat3
  
  
  
  
• Uncomment the lines that set the jvmRoute or append if necessary. Note that the value for the jvmRoute should be the same as that used in the workers.properties file for the respective tomcat instance.
  
  
  
  
• To change the root context ensure that the “Context” directive is updated as under:
  
  

Restart the Apache HTTPD Server and the individual Apache Tomcat Servers and validate the Load-Balancing configuration works as expected. Use the following log files to isolate the problem if any:

tail -f /var/log/httpd/access.log
tail -f /var/log/httpd/mod_jk.log
tail -f /usr/share/tomcatXX/logs/catalina.out
tail -f /var/log/httpd/error_log

Assuming the load balancing setup is working as expected, lets move on to the Squid as a Reverse Proxy. There have been changes between versions 2.5 and 2.6 that affect the configuration file options for the setup. Following are the instructions for version 2.5. Set the IP-Port on which Squid listens. Next, configure the cache-peer of Squid to be Apache HTTPD at port 8084.

http_port 192.168.1.139:80
cache_peer 192.168.1.139 parent 8084 0 no-query

Since, we wanted to cache the pages with parameters as well, we commented the no_cache option. The hierarchy stoplist option was also commented out as shown.

#hierarchy_stoplist cgi-bin ?
#acl QUERY urlpath_regex cgi-bin \?
#no_cache deny QUERY

Depending on your cache size requirements, tweak the following parameters:

cache_mem 512 MB
maximum_object_size 8192 KB
maximum_object_size_in_memory 256 KB
cache_dir ufs /var/spool/squid 1024 16 256

Define access control lists for the destination and trusted networks. This is followed by http_access rules that allow or deny as per security requirements. Note how the rule 'filter_admin_portal' uses a regular expression match to filter access for the admin portal.

acl our_networks src 192.168.1.0/24 10.20.30.0/24
acl dst_mywebapp dst 192.168.1.0/255.255.255.0
acl filter_admin_portal urlpath_regex -i ^/[^/]*admin[^/]*
http_access deny filter_admin_portal
http_access allow dst_mywebapp
http_access allow our_networks
http_access deny all

Squid needs to know the 'Real' HTTP server for which it acts as an accelerator. The parameters httpd_accel_host and httpd_accel_port should point to the Apache HTTPD server setup earlier.

httpd_accel_host 192.168.1.139
httpd_accel_port 8084
httpd_accel_single_host on
httpd_accel_with_proxy on

Once Squid has been restarted, the entire setup should be ready for some testing. Use the log files described above and the following log files to iron out discrepancies, if any.

tail -f /var/log/squid/squid.out
tail -f /var/log/squid/access.log
tail -f /var/log/squid/cache.log

Along the way, we used JMeter for load/stress testing, JConsole for viewing the JVM health in real time, and WireShark/Fiddler to look at HTTP headers and packets. These tools helped find find bottlenecks, tweak parameters and re-architect the solution.

Although, there are multiple points of failure in this architecture, the primary aim was load balancing. Fault tolerance and reliability weren't the primary concerns at this point of time. In a future iteration, the plan is to address these concerns as well.