Realizing Continuous Performance
Management
Steven Haines
Quest Software
Java Domain Expert
February 14, 2008
Your
logo here
Agenda
State of the Market
Cost of Mistakes
Test-Driven Development
Continuous Integration
Continuous Performance Management
CPM in Practice
Business Value
Industry Experts Say…
Forrester Research reports nearly 85 percent of companies with revenue of more
than $1 billion reported incidents of significant application performance problems.
Survey respondents identified the architecture and deployment as the primary
causes of these problems.
-Forrester Research
Medium-sized businesses* are losing an average of 1% of their annual
revenue, or $867K, to downtime. Application outages and degradations are
the biggest sources of downtime, costing these companies $213K annually.
-Infonetics Research
Impact of Poorly Performing Applications
Business-to-consumer
−
Direct loss of customer revenue
−
Loss of confidence
Business-to-business
−
Loss of reputation
−
Potential loss of business relationships
Intranet
−
Loss of productivity
Cost of Mistakes
Performance problems found later in
production cost almost 10x more to fix…
Why is the Current State of Testing Failing?
Test suites are written to out-of-date artifacts
Test suites are not comprehensive enough
Test suites are not automated
−
Not performed regularly
−
No regression testing
Test-Driven Development
Test cases must be written prior to
writing code
Test cases written by developers
Each code component must
Test Suite
Test
Code
Test
Code
Test
Code
Test-Driven Development
Continuous Integration
Source code
Repository
Continuous Performance Management
Picture to show profiling
of a test: JProbe or PAS
type of screen
Profile
Analyze
React
Test-Driven Development Lifecycle
Add a Test
Prove
Failure
Add a Test
Test Suite
Test-Driven Development Lifecycle
Prove
Failure
Add a Test
Implement New
Functionality
Test Suite
Test-Driven Development Lifecycle
Prove
Failure
Add a Test
Prove
Success
Implement New
Functionality
Test Suite
Prove
Failure
Re-factor
the Code
Add a Test
Prove
Success
Implement New
Functionality
Test Suite
Test-Driven Development Lifecycle
Shorter Development
Cycles
Limited Debugging
Faster Triaging
Clearly Solved Business
Problems
Confidence in
Deployment
Continuous Integration
The Challenge
−
With complicated applications, integration
can take longer than development
The Solution
−
Integrate after each code check-in
The Tool
Test Suite
Test
Code
Test
Code
Test
Code
Test-Driven Development
Continuous Integration
Source code
Repository
CI
Server
Build
Test
Publish
Continuous Performance Management
Picture to show profiling
of a test: JProbe or PAS
type of screen
Profile
Analyze
React
Continuous Integration Lifecycle
Monitors
SourceCode
Repository
Continuous
Integration
Server
Continuous Integration Lifecycle
Detects New
Check-Ins
Monitors
SourceCode
Repository
Continuous
Integration
Server
Continuous Integration Lifecycle
Detects New
Check-Ins
Monitors
SourceCode
Repository
Compiles Entire
Application
Continuous
Integration
Server
Continuous Integration Lifecycle
Detects New
Check-Ins
Monitors
SourceCode
Repository
Runs
Exhaustive
Test Suite
•
Compiles Entire
Application
Continuous
Integration
Server
Continuous Integration Lifecycle
Detects New
Check-Ins
Publishes
Build Results
Monitors
SourceCode
Repository
Runs
Exhaustive
Test Suite
• Commit Build
Compiles Entire
Application
Continuous
Integration
Server
Shorter integration
times
Integration issues
resolved as they
occur
Higher quality end
product
What is Continuous Performance Management?
Challenge in Performance Testing
−
Time consuming and error prone
because manual
−
Problems discovered at the end of
development effort often affect
assumptions made at the beginning of a
development effort
Requires re-factoring the architecture
Solution
−
Performance Management in a
Continuous Integration Environment
Proactive approach
Execute performance tests on every
Continuous Performance Management (CPM)
• Create additional
Secondary Builds that
implement performance
tests
1.
Performance Unit
Tests
2.
Performance
Integration Tests
• Business case
• Use case scenario
3.
Performance
Stress/Load Tests
Secondary
Build 2
Secondary
Build 3
Secondary
Build 1
CPM
Test Suite
Test
Code
Test
Code
Test
Code
Test-Driven Development
Continuous Integration
Source code
Repository
CI
Server
Build
Test
Publish
Continuous Performance Management
Profile
Analyze
Repair
Performance Unit Tests
• Memory
– Lingering Object References
(memory leaks)
– Object cycling
• Performance
– Identify slow running code
• Coverage
– Quantify the percentage of
classes, methods, lines of
code and conditions
Performance Integration Tests
Performance analysis of a single user
−
Identify request response times that exceed service-level
agreements
−
Triage to offending methods and SQL statements
−
Follow requests across tiers (and across JVMs)
Identify application problems prior to load
−
Find problems in isolation
−
Easier to pinpoint root cause
Performance Stress/Load Tests
Performance analysis under load (multiple users)
−
Identify request response times that exceed service-level agreements
−
Triage to offending methods and SQL statements
−
Follow requests across tiers (and across JVMs)
−
Validate environmental configuration
−
Identify resource contentions
Identify scalability problems prior to deployment
−
Identify environmental configuration issues
−
Resolve scalability and reliability problems
Continuous Performance Management Toolset
Continuous Integration Server –
CruiseControl
Build Environment – Apache Ant
Test Bed – JUnit
Load Generator – Apache JMeter
Performance Testing:
−
Unit tests:
Quest JProbe
CPM – Too Arduous a Task?
It is a common misconception
that CPM is too risky and time
consuming of an undertaking
We’ve gone through painstaking
efforts to make CPM as easy as
possible to implement
Only requires a handful of build
CPM In Practice – Performance Unit Tests
Additions to Ant build script to profile all your JUnit tests
Easy to do – can paste on a single slide
<!-- Set location of the JProbe Enterprise Tools -->
<property name="jprobe.enterprisetools.home“ location="c:\jprobe-ent" /> <!-- Define the Pre-processor CLASSPATH -->
<path id="classpath.preprocessor">
<fileset dir="${jprobe.enterprisetools.home}" includes="lib/*.jar"/> </path>
<!-- Execute the preprocessor -->
<java classname="com.javasrc.anttools.JProbePreprocessor"> <sysproperty key="src.dirs" value="${src};${src.test}" />
<sysproperty key="jprobe.home" value="C:\Program Files\JProbe 7.0" />
<sysproperty key="jprobe.build.dest" value="${jprobe.enterprisetools.home}\jprobe.xml" /> <classpath refid="classpath.preprocessor"/>
Performance Unit Testing – Quest JProbe
Three analysis enablers:
−
Memory
−
Performance
−
Coverage
CPM In Practice – Integration / Load Tests
Additions to Ant build script to launch JMeter
Easy to do – can paste on a single slide
<!-- Import the JMeter task -->
<taskdef name="jmeter“ classname="org.programmerplanet.ant.taskdefs.jmeter.JMeterTask"/>
<!-- Execute the test script -->
<jmeter jmeterhome="C:\lib\jakarta-jmeter-2.2" testplan="${basedir}/mytests/MyTest.jmx"
resultlog="${basedir}/mytests/MyTestResults.jtl">
<property name="jmeter.save.saveservice.output_format" value="xml" /> </jmeter>
<!-- Generate an HTML Report -->
<xslt in="${basedir}/mytests/MyTestResults.jtl" out="${basedir}/mytests/MyTestResults.html"
CPM In Practice – Integration / Load Tests
Additions to Ant build script to record a performance session
<property name="pas.home" location="j:\PerformaSure5.0" />
<target name="pas" description= "Start a PerformaSure recording" > <!-- Execute nexusctl --> <exec executable="cmd"> <arg value="/c" /> <arg value="${pas.home}/scripts/nexusctl.cmd" /> <arg value="start-recording" /> <arg value="-user" /> <arg value="user" /> <arg value="-l" /> <arg value="30m" /> <arg value="-t" /> <arg value="10s" /> <arg value="-f" /> </exec>
Performance Integration/Load Testing – Quest PerformaSure
Quest Management Server
Business Value
Benefits
−
Reduce R&D and QA time in
performance testing
−
Better performing applications
Results
−
Faster time-to-market = more time to
add features that give you a
competitive advantage over your
competition
−
Higher quality applications = higher
Next Steps…
Download one of my whitepapers from
www.quest.com
under Application
Management, solutions for Java:
−
Extending the Continuous Integration Environment with Performance
Testing (Part 1)
−
Achieving Continuous Performance Management (Part II)
−
Best Practices in Java Environment Performance Testing
−
A Formal Performance Tuning Methodology: Wait-Based Tuning
Sign up for my upcoming “Java Performance Winner’s Circle” webcast
series on performance testing, tuning, and management at
www.quest.com
under Application Management, solutions for Java (starts in March 2008)
Watch my Continuous Integration OnSoftware Video Podcast on
www.informit.com
(or iTunes)
Read my forthcoming digital shortcut book, Agile Java Development with
Next Steps cont’d…
Quest Software Product Information:
JProbe –
www.quest.com/jprobe
PerformaSure –
www.quest.com/performasure
Other Resources:
JProbe Live Product Demo –
www.quest.com/events
PerformaSure Product Demand –
www.quest.com/performasure
