How do I Troubleshoot Performance Problems like "server is slow" and "server hangs"?

Application dynamics, deployment topologies and incoming load profile are all factors that show up with "server is slow", "server hang" symptoms. First step in debugging such scenarios is to gather the Server health indicators when "server is slow". Pramati Server has a status all command that dumps the health indicators of all the constituent Pramati Services, viz. EJB Container, Web Container, JMS Server. A thread dump at the time when "server is slow" will also be helpful.

However, it is hard for clients to manually monitor health of the Server and send us the collected data. Following ANT build script - healthCheck.xml (See Attachment) intends to automate such a task. The build script derives that "server is slow" when Pramati Web Server does not respond to a ping in given amount of time. At that instant the build script takes a Thread dump, and connects to ailing Pramati Server to gather server health indicators by doing a status all.

The ANT script uses the waitfor ANT task to ping the Pramati Web Server. When the ping does not respond with in a stipulated time period this task exits and set a property. badHealth.

The threadDump feature of the script will not work on Windows. Gathering status all output though will work on any OS.

How do I use the ANT Script?

1. Save the XML file - healthCheck.xml (See attchment) to computer which has the Pramati Server node that needs to be monitored.
2. Download Install ANT if its not already present.
3. Open the saved healthCheck.xml and edit the property appServer.home to point to Pramati Server installation on your computer.
4. Find out the process id of the JVM in which Pramati Server is running. By doing something like: ps -eaf | grep Server.
5. From the shell windown launch command ant -Dpid=<procId of Server> -f healthChecks.xml -verbose.
6. You should see an output like the following:

   $ ant -Dpid=10978 -f healthCheck.xml -verbose
Apache Ant version 1.6.2 compiled on June 28 2005
Buildfile: healthCheck.xml
Detected Java version: 1.4 in: /home/prashant/programs/java/j2sdk1.4.2_08/jre
Detected OS: Linux
parsing buildfile /home/prashant/working_dir/PS30/util/healthCheck.xml with URI =
file:///home/prashant/working_dir/PS30/util/healthCheck.xml
Project base dir set to: /home/prashant/working_dir/PS30/util
[available] Found: /home/prashant/builds/SP41122/server
Build sequence for target `dump' is [healthCheck, dump]
Complete build sequence is [healthCheck, dump, statusAll, win32ThreadDump, threadDump, ]

healthCheck:
Checking for http://localhost:8181
Checking for http://localhost:8181
Checking for http://localhost:8181
Checking for http://localhost:8181
Checking for http://localhost:8181
Checking for http://localhost:8181

dump:
[echo] Could not access localhost:8181


Apart from the thread dump, the ANT script also connects to the ailing Pramati Server and dump the status of all constituent Pramati Services to a file named statusAll. This file will be in the directory from where you have launched the ANT command.

Suggested Improvements

• Better name for the output file of the status all command. Appending time stamp, or simple counter to the file name, is a good idea, otherwise consequtive invocations will override data collected previously.
• Account for more ways to determine badHealth. Perhaps wait on a SQL Querry instead of the HTTP ping request.
• Thread dump on Windows.
• Connecting to RemoteShell should be possible by giving nodeName instead of ip, namingPort, and credentials, specially since it may not be ideal to store password in plain text.

The healthCheck.xml ANT Build file

<?xml version="1.0" encoding="UTF-8"?>
<project name="PramatiServerHealthCheck" default="dump">
<property name="appServer.home" value="/home/prashant/builds/SP41122/server"/>
<property name="server.ip" value="localhost"/>
<property name="server.naming.port" value="9191"/>
<property name="server.http.port" value="8181"/>
<property name="server.security.realm" value="system"/>
<property name="server.security.user" value="root"/>
<property name="server.security.password" value="pramati"/>
<property name="statusAllOutputFile" value="statusAll"/>
<property name="pramati_ant_handler.jar" value="${appServer.home}/lib/pramati_client/pramati_ant.jar"/>

<condition property="setup.complete">
<and>
<available file="${appServer.home}"/>
<available classname="com.pramati.AntTaskHandler" classpath="${pramati_ant_handler.jar}"
property="pramati.classpath.set"/>
</and>
</condition>

<target name="dump" if="badHealth" depends="healthCheck">
<echo message="Could not access ${server.ip}:${server.http.port} "/>
<antcall target="threadDump"/>
<antcall target="statusAll"/>
</target>

<property name="ping.interval.millis" value="10000"/>
<property name="expiry.time.minutes" value="1"/>

<target name="healthCheck">
<waitfor maxwait="${expiry.time.minutes}" maxwaitunit="minute" checkevery="${ping.interval.millis}"
timeoutproperty="badHealth">
<http url="http://${server.ip}:${server.http.port}"/>
</waitfor>
</target>

<condition property="os.is.linux">
<os family="unix" />
</condition>

<target name="threadDump" if="os.is.linux" depends="win32ThreadDump" >
<echo message="Taking Threaddump of process with id : ${pid}"/>
<exec command="kill -3 ${pid}"/>
</target>

<target name="win32ThreadDump" unless="os.is.linux">
<echo message="Taking thread dump on Windows OS is not implemented.."/>
</target>

<taskdef name="serverTask" classname="com.pramati.AntTaskHandler"
classpath="${appServer.home}/lib/pramati_client/pramati_ant.jar"/>

<target name="statusAll">
<echo message="About to execute status all on Pramati Server at ${server.ip}"/>
<serverTask verbose="true" ip="${server.ip}"
port="${server.naming.port}" realm="${server.security.realm}"
user="${server.security.user}" password="${server.security.password}"
debug="true" wait="true"
classpath="${appServer.home}/lib/pramati/classpath.jar"
commands="status all > ${statusAllOutputFile},exit "/>
<target>
</project>


© Pramati Technologies 2007	Runs on Pramati Server | Feedback | Legal