How to Share Tomcat Sessions Across a Cluster Using MSM and Memcached

Introduction

When a node in a Tomcat cluster fails, the remaining nodes must take over the failed node's session data. This article presents a solution based on MSM (Memcached Session Manager) and Memcached to share sessions across the cluster.

MSM Overview

MSM‑Memcached Session Manager provides a high‑availability Tomcat session sharing solution. It stores sessions in local memory for fast access (sticky sessions) and backs them up to Memcached for redundancy.

Working Principle

Sticky Session Mode

Tomcat stores the primary session locally; on the first request the session is copied to Memcached. Subsequent requests use the local session, and any changes are synchronized back to Memcached.

When a node crashes, the next request is routed to another Tomcat instance, which retrieves the session from Memcached, updates it locally, and writes back the changes.

Non‑Sticky Session Mode

Tomcat treats the Memcached session as the primary backup. On each request the backup session is loaded into the local container; if loading fails, the primary session is used. After processing, the session changes are written back to Memcached and the local session is cleared.

Implementation Steps

Experimental Topology

Nginx Installation and Configuration

# yum groupinstall "Development Tools" "Server Platform Development" -y
# yum install openssl-devel pcre-devel -y
# groupadd -r nginx
# useradd -r -g nginx nginx
# tar xf nginx-1.6.3.tar.gz
# cd nginx-1.6.3
# ./configure \
    --prefix=/usr/local/nginx \
    --sbin-path=/usr/sbin/nginx \
    --conf-path=/etc/nginx/nginx.conf \
    --error-log-path=/var/log/nginx/error.log \
    --http-log-path=/var/log/nginx/access.log \
    --pid-path=/var/run/nginx/nginx.pid \
    --lock-path=/var/lock/nginx.lock \
    --user=nginx \
    --group=nginx \
    --with-http_ssl_module \
    --with-http_flv_module \
    --with-http_stub_status_module \
    --with-http_gzip_static_module \
    --http-client-body-temp-path=/usr/local/nginx/client/ \
    --http-proxy-temp-path=/usr/local/nginx/proxy/ \
    --http-fastcgi-temp-path=/usr/local/nginx/fcgi/ \
    --http-uwsgi-temp-path=/usr/local/nginx/uwsgi/ \
    --http-scgi-temp-path=/usr/local/nginx/scgi/ \
    --with-pcre
# make && make install
# chmod +x /etc/rc.d/init.d/nginx
# chkconfig --add nginx
# chkconfig nginx on
# vim /etc/nginx/nginx.conf
upstream www.scholar.com {
    server 172.16.10.123:8080;
    server 172.16.10.124:8080;
}
server {
    listen 80;
    server_name www.scholar.com;
    location / {
        proxy_pass http://www.scholar.com;
        index index.jsp index.html index.htm;
    }
}
# service nginx start

Tomcat Installation and Configuration

# rpm -ivh jdk-7u79-linux-x64.rpm
# echo "export JAVA_HOME=/usr/java/latest" > /etc/profile.d/java.sh
# echo "export PATH=\$JAVA_HOME/bin:\$PATH" >> /etc/profile.d/java.sh
# . /etc/profile.d/java.sh
# tar xf apache-tomcat-7.0.62.tar.gz -C /usr/local/
# ln -sv apache-tomcat-7.0.62/ tomcat
# echo "export CATALINA_HOME=/usr/local/tomcat" > /etc/profile.d/tomcat.sh
# echo "export PATH=\$CATALINA_HOME/bin:\$PATH" >> /etc/profile.d/tomcat.sh
# . /etc/profile.d/tomcat.sh
# vim /etc/rc.d/init.d/tomcat
#!/bin/sh
# chkconfig: 2345 96 14
# description: Apache Tomcat servlet/JSP container.
case "$1" in
  start) exec $CATALINA_HOME/bin/catalina.sh start ;;
  stop)  exec $CATALINA_HOME/bin/catalina.sh stop  ;;
  restart) $CATALINA_HOME/bin/catalina.sh stop; sleep 2; exec $CATALINA_HOME/bin/catalina.sh start ;;
  *) echo "Usage: $(basename $0) {start|stop|restart}"; exit 1;;
esac
exit $?
# chmod +x /etc/rc.d/init.d/tomcat
# chkconfig --add tomcat
# chkconfig tomcat on

Memcached Installation

# yum groupinstall "Development Tools" "Server Platform Development" -y
# tar xf libevent-2.0.22-stable.tar.gz
# cd libevent-2.0.22-stable
# ./configure --prefix=/usr/local/libevent
# make && make install
# echo "/usr/local/libevent/lib" > /etc/ld.so.conf.d/libevent.conf
# ldconfig
# tar xf memcached-1.4.24.tar.gz
# cd memcached-1.4.24
# ./configure --prefix=/usr/local/memcached --with-libevent=/usr/local/libevent
# make && make install
# vim /etc/init.d/memcached
#!/bin/bash
# chkconfig: - 86 14
# description: Distributed memory caching daemon
PORT="11211"
USER="nobody"
MAXCONN="1024"
CACHESIZE="64"
prog="/usr/local/memcached/bin/memcached"
lockfile="/var/lock/subsys/memcached"
start() {
  echo -n "Starting Distributed memory caching: "
  daemon $prog -d -p $PORT -u $USER -c $MAXCONN -m $CACHESIZE
  RETVAL=$?
  [ $RETVAL -eq 0 ] && touch $lockfile || failure
  echo
  return $RETVAL
}
stop() {
  echo -n "Shutting down Distributed memory caching: "
  killproc $prog
  RETVAL=$?
  [ $RETVAL -eq 0 ] && rm -f $lockfile || failure
  echo
  return $RETVAL
}
case "$1" in
  start) start ;;
  stop)  stop ;;
  restart) stop; start ;;
  status) status $prog ;;
  *) echo "Usage: $0 {start|stop|restart|status}"; exit 1;;
esac
exit $RETVAL

# chmod +x /etc/init.d/memcached # chkconfig --add memcached # chkconfig memcached on # service memcached start

MSM JAR Deployment # cp *.jar /usr/local/tomcat/lib/ Tomcat MSM Configuration

# vim /usr/local/tomcat/conf/server.xml
<?xml version='1.0' encoding='utf-8'?>
<Server port="8005" shutdown="SHUTDOWN">
  ...
  <Service name="Catalina">
    <Connector port="8080" protocol="HTTP/1.1" connectionTimeout="20000" redirectPort="8443" />
    <Engine name="Catalina" defaultHost="localhost">
      <Host name="www.scholar.com" appBase="webapps" unpackWARs="true" autoDeploy="true">
        <Context path="/test" docBase="/usr/local/tomcat/webapps/test/" reloadable="true">
          <Manager className="de.javakaffee.web.msm.MemcachedBackupSessionManager"
            memcachedNodes="n1:172.16.10.126:11211,n2:172.16.10.212:11211"
            failoverNodes="n1"
            requestUriIgnorePattern=".*\.(ico|png|gif|jpg|css|js)$"
            transcoderFactoryClass="de.javakaffee.web.msm.serializer.javolution.JavolutionTranscoderFactory" />
        </Context>
        <Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="scholar_access_log." suffix=".txt" pattern="%h %l %u %t \"%r\" %s %b" />
      </Host>
    </Engine>
  </Service>
</Server>

# scp /usr/local/tomcat/conf/server.xml node2:/usr/local/tomcat/conf/ # service tomcat restart

Testing

Create a simple JSP page under /test/WEB-INF/ that displays the session ID and creation time. Deploy it on both Tomcat nodes, start the services, and access the page through Nginx. The session ID remains consistent across nodes, demonstrating successful sharing.

Stop one Tomcat node; the other node continues to serve the same session without changing the session ID.

Stop one Memcached node; the session is automatically retrieved from the remaining node, and the session ID stays unchanged.

The End

This experiment demonstrates that Tomcat session sharing using MSM and Memcached provides high availability and seamless failover for web applications.