Intrigue

Load Balancing and Scaling with Squid-HTTPD-Tomcat

2009-04-20T11:26:00.024-05:00

One of our clients host a content management system built on top of technology I use at work. The technology is a custom server that stacks on top of the J2EE application servers like JBoss or a servlet container like Apache Tomcat. Our standard stack that we prescribe to clients is the MySQL-Apache Tomcat stack.

Over a period of few weeks, the needs of this client grew well beyond the standard stack. Considering the amount of static content being served by the Content Management System, a caching server such as Squid was a logical choice. Soon it was apparent, that this wouldn't scale either. Hence as the next iteration, we prescribed a solution with three Apache Tomcat instances running on 32-bit JVM instances of 1.5GB memory each, being load balanced by Apache HTTPD using mod-jk connnector, and front-ended with a Squid caching server. Since the public side had only read requests, a single MySQL server as the back-end was sufficient.

Being a unique stack in itself, not to mention the abundant complexities involved in between versions and dependencies, I thought it would be great to document the setup for future reference. As of now the stack is working well for the client, with the added ability to bring down one or even two Apache Tomcat instances for maintenance and leaving the site unaffected. The architecture is depicted in Figure 1 below:

The versions for softwares used are:


[root@1233797-app45 ~]# /usr/share/tomcat/bin/version.sh 
Using CATALINA_BASE:   /usr/share/tomcat
Using CATALINA_HOME:   /usr/share/tomcat
Using CATALINA_TMPDIR: /usr/share/tomcat/temp
Using JRE_HOME:       /usr/java/jdk1.5.0_17
Server version: Apache Tomcat/5.5.17
Server built:   Apr 14 2006 02:08:29
Server number:  5.5.17.0
OS Name:        Linux
OS Version:     2.6.9-67.0.22.ELsmp
Architecture:   i386
JVM Version:    1.5.0_17-b04
JVM Vendor:     Sun Microsystems Inc.
[root@183787-app9 ~]# rpm -q httpd squid mysql-server
httpd-2.0.52-41.ent.4
squid-2.5.STABLE14-4.el4
mysql-server-5.0.77-1.rs.el4

Following are the steps to setup the servers with reference to Figure 1. Customize it for your local setup.

Edit the /etc/httpd/conf/httpd.conf. Change the IP and port that HTTPD listens on:


Listen 192.168.1.139:8084

Download the mod_jk connector from Apache Tomcat Connectors Download. Since I ran into issues finding pre-packaged RPM for RHEL, I used one of the binaries available from their binaries link on that page. Save the "mod_jk-*.so" file along-with other modules for HTTPD. Append the following lines to the /etc/httpd/conf/httpd.conf


LoadModule jk_module modules/mod_jk.so


  JkMount jkstatus
  order deny,allow
  allow from 127.0.0.1
  deny from all



  JkWorkersFile conf/workers.properties
  JkShmFile /var/cache/httpd/mod_jk.shm
  JkLogFile /var/log/httpd/mod_jk.log
  JkLogLevel info
  JkLogStampFormat "[%a %b %d %H:%M:%S %Y] "
  JkOptions +ForwardKeySize +ForwardURICompat -ForwardDirectories
  JkRequestLogFormat "%w %V %T"

  SetEnvIf Request_URI "/webmail/*" no-jk
  SetEnvIf Request_URi "/mailman/*" no-jk
  SetEnvIf Request_URI "/awstatsclasses/*" no-jk
  SetEnvIf Request_URI "/awstatscss/*" no-jk
  SetEnvIf Request_URI "/awstatsicons/*" no-jk
  SetEnvIf Request_URI "/awstats/*" no-jk

  JkMount  /* router
  JkMount  /jkmanager/* jkstatus

Create a /etc/httpd/conf/workers.properties file as shown under:


# Define some properties
workers.apache_log=/var/apache/logs
workers.tomcat_home=/usr/share/tomcat
workers.java_home=/usr/java/jrockit-R27.4.0-jdk1.5.0_12
ps=/

# The advanced router LB worker
worker.list=router,tomcat1,tomcat2,tomcat3,jkstatus

# Set properties for tomcat1 (ajp13)
worker.tomcat1.type=ajp13
worker.tomcat1.host=192.168.1.140
worker.tomcat1.port=8010
worker.tomcat1.lbfactor=1

# Set properties for tomcat2 (ajp13)
worker.tomcat2.type=ajp13
worker.tomcat2.host=192.168.1.140
worker.tomcat2.port=8011
worker.tomcat2.lbfactor=1

# Set properties for tomcat3 (ajp13)
worker.tomcat3.type=ajp13
worker.tomcat3.host=192.168.1.140
worker.tomcat3.port=8012
worker.tomcat3.lbfactor=1

# Define the LB worker
worker.router.type=lb
worker.router.balance_workers=tomcat1,tomcat2,tomcat3

worker.jkstatus.type=status

On each of the workers (tomcat1,tomcat2,tomcat3) make the following changes to the /usr/share/tomcatXX/conf/server.xml file where XX indicates the number of tomcat:

Change the connector configuration so that the port numbers are 8081, 8082, 8083 for tomcat1, tomcat2 and tomcat3 respectively. This helps validate individual tomcat instances.
```
    
```
The mod_jk connector talks to the worker instances on the AJP connector port. This port can be specified in the server.xml file as shown below. We picked ports 8010-8012 for tomcat1-tomcat3
```
    
    
```
Uncomment the lines that set the jvmRoute or append if necessary. Note that the value for the jvmRoute should be the same as that used in the workers.properties file for the respective tomcat instance.
```
  
  
```
To change the root context ensure that the “Context” directive is updated as under:
```
  
     
      
     
  
```

Restart the Apache HTTPD Server and the individual Apache Tomcat Servers and validate the Load-Balancing configuration works as expected. Use the following log files to isolate the problem if any:


tail -f /var/log/httpd/access.log
tail -f /var/log/httpd/mod_jk.log
tail -f /usr/share/tomcatXX/logs/catalina.out
tail -f /var/log/httpd/error_log

Assuming the load balancing setup is working as expected, lets move on to the Squid as a Reverse Proxy. There have been changes between versions 2.5 and 2.6 that affect the configuration file options for the setup. Following are the instructions for version 2.5. Set the IP-Port on which Squid listens. Next, configure the cache-peer of Squid to be Apache HTTPD at port 8084.


http_port 192.168.1.139:80
cache_peer 192.168.1.139 parent 8084 0 no-query

Since, we wanted to cache the pages with parameters as well, we commented the no_cache option. The hierarchy stoplist option was also commented out as shown.


#hierarchy_stoplist cgi-bin ?
#acl QUERY urlpath_regex cgi-bin \?
#no_cache deny QUERY

Depending on your cache size requirements, tweak the following parameters:


cache_mem 512 MB
maximum_object_size 8192 KB
maximum_object_size_in_memory 256 KB
cache_dir ufs /var/spool/squid 1024 16 256

Define access control lists for the destination and trusted networks. This is followed by http_access rules that allow or deny as per security requirements. Note how the rule 'filter_admin_portal' uses a regular expression match to filter access for the admin portal.


acl our_networks src 192.168.1.0/24 10.20.30.0/24
acl dst_mywebapp dst 192.168.1.0/255.255.255.0
acl filter_admin_portal urlpath_regex -i ^/[^/]*admin[^/]*

http_access deny filter_admin_portal
http_access allow dst_mywebapp
http_access allow our_networks
http_access deny all

Squid needs to know the 'Real' HTTP server for which it acts as an accelerator. The parameters httpd_accel_host and httpd_accel_port should point to the Apache HTTPD server setup earlier.


httpd_accel_host 192.168.1.139
httpd_accel_port 8084

httpd_accel_single_host on
httpd_accel_with_proxy on

Once Squid has been restarted, the entire setup should be ready for some testing. Use the log files described above and the following log files to iron out discrepancies, if any.


tail -f /var/log/squid/squid.out
tail -f /var/log/squid/access.log
tail -f /var/log/squid/cache.log

Along the way, we used JMeter for load/stress testing, JConsole for viewing the JVM health in real time, and WireShark/Fiddler to look at HTTP headers and packets. These tools helped find find bottlenecks, tweak parameters and re-architect the solution.

Although, there are multiple points of failure in this architecture, the primary aim was load balancing. Fault tolerance and reliability weren't the primary concerns at this point of time. In a future iteration, the plan is to address these concerns as well.

Corrupt RPM Database

2009-04-07T15:03:00.004-05:00

The other day I was on a client machine which happened to give me the following exception when looking up an RPM package using yum.


[root@staging ~]# yum search vnc
Loading "rhnplugin" plugin
rpmdb: Lock table is out of available locker entries
rpmdb: Unknown locker ID: 8470
error: db4 error(22) from db->close: Invalid argument
error: cannot open Packages index using db3 - Cannot allocate memory (12)
error: cannot open Packages database in /var/lib/rpm
Traceback (most recent call last):
  File "/usr/bin/yum", line 29, in ?
    yummain.main(sys.argv[1:])
  File "/usr/share/yum-cli/yummain.py", line 85, in main
    base.getOptionsConfig(args)
  File "/usr/share/yum-cli/cli.py", line 163, in getOptionsConfig
    disabled_plugins=self.optparser._splitArg(opts.disableplugins))
  File "/usr/lib/python2.4/site-packages/yum/__init__.py", line 164, in _getConfig
    self._conf = config.readMainConfig(startupconf)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 685, in readMainConfig
    yumvars['releasever'] = _getsysver(startupconf.installroot, startupconf.distroverpkg)
  File "/usr/lib/python2.4/site-packages/yum/config.py", line 755, in _getsysver
    idx = ts.dbMatch('provides', distroverpkg)
TypeError: rpmdb open failed

Apparently, this is the case of a corrupt RPM database and can be fixed by rebuilding the database as follows:


[root@staging ~]# rm /var/lib/rpm/__db.*
rm: remove regular empty file `/var/lib/rpm/__db.000'? y
rm: remove regular file `/var/lib/rpm/__db.001'? y
rm: remove regular file `/var/lib/rpm/__db.002'? y
rm: remove regular file `/var/lib/rpm/__db.003'? y
[root@staging ~]# rpm --rebuilddb

Check Available Disk Space and Notify

2009-02-02T17:41:00.016-06:00

Here's some handy bash script to notfiy support when we are running low on disk space.


#!/bin/bash
#
# Description:Checks the available space for the disk mounted as / (root)
#             If disk space is less than 10GB it will send out a notification
#

# Send emails to these people
EMAILID=support@mydomain.com

# Min Free Space
MINSPACE=10485760

# Get the space available
FREESPACE=`df / | awk '{ if( NR == 2) print $4 }'`

# Send email if required
if [ $FREESPACE -le $MINSPACE ];
then
 logger "Warning: Root Drive Running Out Of Disk Space [${FREESPACE}KB]"
 mail -s 'My Favourite Production Server Disk Space' ${EMAILID} << EOF
My Favourite Production server root drive is running out of disk space. Current free space is ${FREESPACE} KB @ $(date)
EOF
fi

Mantis + LDAP Authentication

2008-05-23T16:49:00.013-05:00

Mantis is one of the many bug tracking softwares available out there (http://www.google.com/Top/Computers/Software/Configuration_Management/Bug_Tracking). For an internal project at work, I needed a bug tracker and Mantis was the logical choice due to prior experience within the development team. The machine at hand was running on CentOS Release 4.4 and required the following rpm packages:



[root@eclipse html]# rpm -q httpd mysql-server php php-mysql php-ldap

httpd-2.0.52-28.ent.centos4

mysql-server-4.1.20-1.RHEL4.1

php-4.3.9-3.22.9

php-mysql-4.3.9-3.22.9

php-ldap-4.3.9-3.22.9

There were issues with using the community-server rpms for MySQL 5.0.51 as the php-mysql and php-ldap packages weren't compatible. Since, this was going to be an internal bug tracker, and we weren't expecting to use any of the 5.0 features, it was a fair compromise for the power of Apache-PHP-LDAP-Mantis.

We have an existing LDAP server which stores the authentication credentials companywide. Out of the box, Mantis allows for new user sign-up but this would mean that every user in the organization would have to keep track of another set of credentials. Hence, LDAP integration was a must. I expected LDAP authentication to work out of the box but it turned out to be a little more involved. I am a novice when it comes to PHP and it took a little bit of searching and reading manuals to wade my way through the Mantis-PHP maze of code.

In order to help people who want to walk down the same path, I aim to simplify their life by providing steps to achieve this feat:

1. Download Mantis and unzip at /var/www/html (or whatever your DocumentRoot is at in /etc/httpd/conf/httpd.conf) and rename the folder to mantis. As of writing this post, the stable version was mantis-1.1.1

2. A bunch of global Mantis settings are listed in the mantis/config_defaults_inc.php. I tried to override the ones that were relevant to this integration. Here's the file:


# --- database variables ---------

# set these values to match your setup
$g_hostname = "10.0.0.4";
$g_db_username = "mantisuser";
$g_db_password = "password";
$g_database_name = "mantis";
$g_db_type = "mysql";

# --- email variables -------------

$g_administrator_email = 'admin@mycompany.com';
$g_webmaster_email = 'webmaster@mycompany.com';

# the "From: " field in emails
$g_from_email = 'noreply@mycompany.com';

# the return address for bounced mail
$g_return_path_email = 'admin@mycompany.com';

# --- file upload settings --------
# This is the master setting to disable *all* file uploading functionality
#
# The default value is ON but you must make sure file uploading is enabled
# in PHP as well. You may need to add "file_uploads = TRUE" to your php.ini.
$g_allow_file_upload = ON;


# LDAP
$g_login_method = LDAP;
$g_ldap_server = 'ldap://10.0.0.4/';
$g_ldap_port = '389';
$g_ldap_root_dn = 'dc=mycompany,dc=com';
$g_ldap_organization = 'objectClass=*'; # e.g. '(organizationname=*Traffic)'
$g_ldap_uid_field = 'uid'; # Use 'sAMAccountName' for Active Directory
$g_ldap_bind_dn = 'uid=root,ou=Users,dc=mycompany,dc=com';
$g_ldap_bind_passwd = 'cleartextsecret';
# Should we send to the LDAP email address or what MySql tells us
$g_use_ldap_email = OFF;

# The LDAP Protocol Version, if 0, then the protocol version is not set.
$g_ldap_protocol_version = 3;

# --- signup ----------------------

$g_allow_signup = OFF;
$g_lost_password_feature = OFF;

3. In the file 'mantis/core/authentication.php', change the lines in the function 'auth_attempt_login' begining "if ( false === $t_user_id ) {" as follows


if ( false === $t_user_id ) {
  if ( BASIC_AUTH == $t_login_method \\ LDAP == $t_login_method ) {
    # attempt to create the user if using BASIC_AUTH
    if ( BASIC_AUTH == $t_login_method) {
      $t_cookie_string = user_create( $p_username, $p_password );
    } elseif ( LDAP == $t_login_method ) {
      $t_cookie_string = user_create( $p_username, '', 
        ldap_email_from_username( $p_username ) );
    }
    if ( false === $t_cookie_string ) {
      # it didn't work
      return false;
    }

4. Restart your Apache HTTPD server and validate.

Truncate a file

2008-04-09T09:37:00.006-05:00

While setting up Apache HTTP Server to rotate logs I came across an interesting problem. How would you truncate a file? The trivial solution that I came up with was


[root@lambda ~]# rm -f /var/log/httpd/foo.log
[root@lambda ~]# touch /var/log/httpd/foo.log

Unfortunately this doesn't work. The HTTP Server ceases logging to the file unless there is a server restart. After prying around the web I came across some interesting tricks but the one that stood out was:


[root@lambda ~]# :> /var/log/httpd/foo.log

TCP Dump

2008-04-09T09:37:00.005-05:00

A very convenient way to find out traffic content on a particular port.


tcpdump -l -i eth0 -w - src or dst port 3306 | strings