Insight into Performance
Testing J2EE Applications
Sep 2008
Presented by
Agenda
The Declaration
J2EE Introduction
Performance testing strategy for J2EE applications
J2EE Application Architecture
Need for effective Performance Monitoring strategy
Performance metrics for J2EE system
J2EE Application Server level Performance monitoring
Advanced Instrumentation based Monitoring tools
Approach to Performance Analysis
Case studies
The Declaration
We are NOT Performance Testers!
We are Performance Doctors and Applications are our patients! We, as doctors:
» Test to validate
» Monitor health
» Analyze & Diagnose abnormalities
» Recommend cures
Testing – To validate/acknowledge the problem
» E.g., The average transaction response time (of 9.5 seconds) doesn’t meet the SLA (of 5 seconds)
Provide value – Help customers spot the performance bottleneck and give them the remedy
» E.g., The high response time is due to high GC activities, which can be minimized by increasing the JVM heap size to from current 256MB to 512 MB.
Java 2 Enterprise Edition (J2EE)
Introduction
J2EE is a widely used platform for server programming in Java language
It offers libraries which provide functionality to deploy fault-tolerant, distributed, multi-tier Java software
Based on modular components deployed on a container (application server)
Different types of J2EE systems
Web based system using JSP/Servlets presentation tier Webservices based system
J2EE Application Architecture
Understand Application Architecture during the Requirements analysis phase
Technology used to implement the system
» E.g., J2EE web based, Web services based, Message based
The components that make up the system
» E.g., Session Bean, Message Driven Beans, WebServices, JMS
Interaction between the components
» E.g., Invoking Session Bean methods, WebService calls, JMS Messages
Database Connectivity
» JDBC Connection pool (or) Hibernate
JVM 2
JVM 3
HTTP Servlet
EJB Client
EJB Method
Web Service client
Web Service JDBC Connectivity JVM 1 DB Concurrent Requests
Need for Effective Performance Monitoring strategy
Bottlenecks
“My application responds poorly – But WHY?”
Bottlenecks in your application – Performance limiters Bottlenecks can be in any tier of J2EE architecture Identify bottlenecks, resolve them – tier by tier
Bottlenecks in a J2EE system are caused by 2 factors
Code related performance issues
Improper Configurations in Web/Application/Database servers
• E.g., JDBC Connection pool in Application server
Hardware/Software limitations
Fast Web server SLOW Application Server
Network
LRFast DB Server
Slow Response time
Performance Metrics for J2EE system
What to Monitor? – Depends on Application architecture, Project scope and access restrictions Client side statistics – Response Time, Throughput, Hits/Second.
Server side statistics (for illustration only – Not complete list)
High Level Performance Statistics
(Application server specific)
System Performance Statistics
(OS specific)
JVM Performance Statistics
(JVM specific)
• CPU Utilization – user, system, total • Memory utilization – used, free, total
• Paging info – page in, page out, page faults • Processor Run Queue, blocked queue
• Network Statistics – Bytes Sent/Received, etc • Disk I/O – Disk %used time, %red, %write • DB Connection Pool – free, active, total • Thread Pool – free, used, total
• Session info – active, invalidated • Web service call response time • EJB Method execution time (IBM JVM)
• Heap used/Free/Total Limit • GC Duration
• GC Frequency
J2EE Application Server level Performance monitoring
Identify counters to monitor
VM Performance – Monitor & Identify VM Performance issues
» Heap memory (used, Free, Total) » GC time & frequency
JDBC Connection Pool Utilization – Monitor bottlenecks due to Connection pool settings
» Connections In Use » Connection Use time » Wait time
HTTP Thread Pool Utilization - Monitor » Percent Maxed
» Threads in Use Other Counters
» EJB Method execution time » Object Pool Utilization » Webservice response times
JDBC Connection Pool HTTP Thread Pool Object Pool
JAVA Virtual Memory
Heap Memory Web Server Method1 EJB Method2 Method1 Method2 Web Services
Advanced Instrumentation based Monitoring tools
Advanced Monitoring
Advanced monitoring tools are available in market
Based on JVM byte code Instrumentation
Can break down response time across tiers (Web/Business/DB/External)
Can pin-point method level delays and CPU consumptions Example: J2EE Diagnostics from HP
Has less overhead on the system
Integrates with Loadrunner
Has built in ‘light-weight’ profiler – Start Profiling on the fly
Approach to Performance Analysis
Start from Client side metrics
Valid Test?
Analyze Transaction failures Analyze Throughput, TPS, Hits/Sec
Analyze server logs for errors Analyze the backend transactions
Analyze Average Transaction Response time and 90% response time
Within SLA?
Resolve the errors Repeat the performance tests NO
YES
Create Results document Sign off performance testing YES
Stop NO
A
NOTE - For illustration only Not a complete list
Approach to Performance Analysis (Contd.)
Analyze GC logs Tune JVM Profile Application
A
Analyze System Statistics (CPU)
Is CPU of App Server within applicable limits? (65%) NO YES Is CPU of DB within acceptable limits? (65%) NO
Analyze DB performance using DB performance tools like STATSPACK
Analyze JDBC Connection Pool, thread pool, EJB pool
Analyze for thread contention Analyze for network issues
YES
Stop
NOTE - For illustration only Not a complete list
Case Study 1 – Poor JAVA coding practices
Problem Definition
“After 20 minutes of load test, my application stops responding” Analysis
CPU Utilization is good (within 30%)
Analysis on data collected from Tivoli Performance viewer:
All the JDBC connections are under use
Number of active connections to DB increases, but never comes down even after the test is over
Issue Identification
Developer error / Poor coding practice
Connections are not closed / not returned back to the pool Sample Code with error
import java.sql.*; import javax.sql.*;
public ProductPK ejbCreate() { try {
// initialize JNDI lookup parameters Context ctx = new InitialContext(parms);
ConnectionPoolDataSource cpds = (ConnectionPoolDataSource)ctx.lookup(cpsource); ...
// Following parms could all come from a JNDI look-up PooledConnection pc = cpds.getPooledConnection(); Connection conn = pc.getConnection();
...
// Execute queries, do business logic …
conn.close();
} catch(Exception ex){ }
Connection will not be closed if exceptions are thrown before this statement.
So, connection close should be written within a finally{ } block
Case Study 2 – JVM Heap memory issues in
Application server
Problem Definition
“My application responds poorly with very high response time..”
Analysis
CPU Utilization of the application server is very high throughout the test
“Out Of Memory” Exceptions were thrown in the server logs
Analyzing the GC Log statistics
40% heap memory was free in spite of “OutOfMemory” Exceptions
The GC Frequency was very high (one in every 100 milli Seconds)
The GC time was very high (~6 seconds for each GC)
Issue Identification
Analyze the “compaction” time
Heap memory Fragmentation issue
Free memory is scattered around the JVM Heap
Contiguous space could not be found to allocate the objects
Analyze the reason for fragmentation
“Large Objects” created by the application
“Pinned” or immovable objects created by the application
High GC Time
Possible pitfalls during
Performance Analysis
Consider both Average & 90% response time
Sometimes Average response time may be misleading
Analyze the Server logs/errors during Performance tests
Errors/exceptions might have failed the server transactions, but error messages not shown in the web pages
Loadrunner transactions may PASS though
Consider the asynchronous communication between the components (like JMS messages) and devise a test strategy
Understand the middleware components in the architecture and monitor them if possible
Consider the external calls made from the system and watch the call response times
