APM EVALUATION GUIDE
Application Performance Management Evaluation Guide
Page 1
FROM THE TRENCHES:
THE RESEARCH IN ACTION
APPLICATION PERFORMANCE
EVALUATION GUIDE
EXECUTIVE SUMMARY
Knowing which Application Performance Management platform is right for you can be a
challenging task. The term Application Performance Management is thrown around pretty loosely
these days and it can be difficult to isolate facts from the marketing noise. However, there are
some key considerations that will help you shortlist, evaluate, and select the best Application
Performance Management solution for your unique needs.
The purpose of this document is to gather and organize evaluation criteria, define why they
matter, and provide you with an example of the typical evaluation process and timeline for an
Application Performance Management buyer.
APM EVALUATION GUIDE
Application Performance Management Evaluation Guide
Page 2
INTRODUCTION
Over the last 10 to 15 years, application complexity has increased significantly
1
. Today, most
enterprises face the challenge of managing not only legacy client-server or even mainframe
systems, but also a huge variety of Web-apps and private, hybrid and public Cloud services.
Furthermore, users now demand proactive IT Service Management (ITSM)
2
, focused on the
end-user experience, ITIL
3
principles and concepts like a CMDB
4
and BSM
5
. All these factors combined
lead to the rise of Application Performance Management (APM) solutions.
Market Definition
Research In Action defines APM as the monitoring and managing of performance and availability
of software applications
6
.
Today, the APM market is roughly $5 billion in software licenses, maintenance and Software as a
Service (SaaS)
7
revenues. The annual growth is between 10 – 15%, which makes it one of the most
dynamic markets in the IT industry
8
. Therefore, it should be no surprise that a market as attractive
as APM, has entered into a period of intense competition of technology and strategy with a
multitude of vendors and viewpoints
9
. As a result, IT decision makers and project managers are
looking for support and lessons learnt when entering an APM initiative. The first step is usually the
vendor longlist
10
, typically aided by tools like the Magic Quadrant or the Vendor Selection
Matrix
11
. Further into the project, the project team needs to evaluate the vendor capabilities in
more detail. This is where this evaluation guide comes into play. The content is compiled from
numerous client interviews, best practices and real-life project experiences. It is a true “from the
trenches” lessons learnt checklist.
APM EVALUATION GUIDE
Application Performance Management Evaluation Guide
Page 3
EVALUATION AREAS
Modern APM software should do much more than help you see the health and availability of your
systems and production code: It should help you find the source of a problem in minutes and
pinpoint the line of code that created the problem; it should help you shorten your release
process by giving you a view of your performance metrics and potential issues before you go to
production; it should show you how external or internal users perceive your mobile or web
applications as well as anticipate how your applications will perform ahead of changes or traffic
spikes. For the purpose of this evaluation guide, we grouped capabilities and evaluation criteria in
6 functional categories with set of additional general considerations.
To master the challenges of modern APM, ensure your solution is able to:
1.
Identify problems and pinpoint root-cause.
2.
Understand your application or website’s readiness before go-live.
3.
Shorten application release cycles (i.e. DevOps
12
).
4.
Understand and optimize your end-users’ journey.
5.
Understand outside forces affecting your application and/or website.
6.
Simplify deployment and management of your APM platform.
EVALUATION PROCESS
As it is with any software consideration, the evaluation process requires a combination of research
and a hands-on proof of concept to make sure you picked the right solution for your needs.
Here is a typical timeline for the evaluation process:
1.
Scope the project.
1.1.
Establish and document business requirements driving the project (e.g. lack of visibility,
poor metrics, customer satisfaction issues, reputation, etc.) – 1 day.
APM EVALUATION GUIDE
Application Performance Management Evaluation Guide
Page 4
1.2.
Identify which application(s), use case(s) or processes(s) are tied to the business objectives
– 2 days.
1.3.
Identify all key architectural components that are affected. This includes all databases,
browser types, mobile devices, app servers, packaged applications, custom code, etc. – 1
to 3 days.
2.
Select vendors to evaluate.
2.1.
Finalize evaluation areas and selection criteria – 1 day.
2.2.
Select vendors and fill out criteria – 5 days.
2.3.
Shortlist top 2 vendors – 1 day.
3.
Run Proof Of Concept (POC).
3.1.
Install the software - 0.5 day for each product.
3.2.
Evaluate software – 1-2 days for each product (depending on depth of POC).
4.
Select a vendor/product.
APM EVALUATION GUIDE
Application Performance Management Evaluation Guide
Page 5
EVALUATION CRITERIA
1.
Go beyond monitoring dashboards and evaluate your code and its performance
Capability Benefit
Vendor1 Vendor2 Vendor3
Availabilityofhealthmonitoring Providesbasicmonitoringofsystemandapplicationhealth
Baseliningandalerting Createsathresholdforacceptableperformancelevelsthatareautomaticallybaselinedon historicaltrafficpatternsandcantriggeralerts
Adaptivealerting Removesmost“falsepositives”anderroneousalertsbysettingnotificationsonabsolute andrelativedeviationfrompercentiles
Sampledorpartialdatacapture Enablesyoutocollectruntimemetricsfromanapplication:eitherthroughcode-level instrumentationorthroughanagent.
Gaplessdatacapture Enablesyoutoidentifyrootcauseinanyscenario,bycapturingeventsthathappened betweenfailuresoreventsleadingtoafailure,24x7,withmethod-leveldetails.Also allowsperformanceinprovementforallusers,notjusttheoneexperiencingproblemsand
comparesgoodwithbadperformingtransactions
SmartAnalytics Providesinstantproblemidentificationandrankingsoyouknowinamatterofseconds wheretofocuseffortsforthefastestservicerestoration.Italsoallowsforcontinuous
performanceimprovementstoyourqualityofservice.
Client-sidecrashanalysis Capturinganddisplayingthecrashreports,withalltheeventsthatleadtothecrash,is criticaltounderstandingtheissueandgettingacodefixwithouthavingtoreproducethe
issue
Completeuser-visitcapture Tofullyunderstandyourenvironmentandwhycertaineventsoccur,youmustbeableto lookupanindividualuser,seealltheiractions,andtiethoseactionstobackend
performancebeforeanalertistriggered
AgentforWeb2.0applications Havingabrowseragentprovidesaviewofallrequestsmakingupapageload,aswellas allsubsequentrequests.Thatviewiscriticaltounderstandingthenetworkbehaviorofthe browserindetail(e.g.howmanynetworkconnections,whatwascached,etc),getinsights intotheexecutionofJavascriptsub-functions,andunderstandhowthebrowserrenders
thewebpage Abilitytocaptureallcontentontheweb
servertier Staticcontentdeliveredbythewebservermightimpactyourapplicationperformance VisitlevelSQLstatements EnablesyoutocaptureindividualSQLstatementsforeachusertransactionwithbind
variablesandeasilyidentifythereasonforslowqueries
Supportforasynchronouseventmodels Scenariosexistinwhicheventsareexchangedasynchronouslybetweencomponentsofa system(ex:AJAX,RabbitMQ,MQSeries).Withoutsupportforasynchronousevent models,yourAPMsolutioncannotautomaticallyrecreateacompletetransaction Supportforsynchronouseventsmodels Pooruseofsynchronousmethodcallscanleadto performanceissuesthataredifficultto
spot.Lackofsupportforsynchronouseventswilllimityou.Thissupportenablesyouto quicklypin-pointaproblem’srootcause.
Supportforanalysisofsuspensiontimes Enablesyoutocapturegarbagecollectioninformationtoidentifyitsimpactonuser experienceandoverallperformance
Abilitytotakememorysnapshotsforleak analysis
Enablesyoutoquicklyunderstandwhatcausesmemoryissues.Givesyoudependencies withintheJava/CLRheapobjects.
AbilitytotakethreaddumpsforJavaand
.Netprocesses Allowsdiagnosisofthreadcontentionproblems(e.g.threadsblockingeachother) Visibiltyintonativeapplications LargerapplicationsoftenrelyonlegacycomponentsthatarenotwritteninJava/.Net.
Lookfortheabilitytofindinsightsintonativecomponents,andanintegrationintothe transactionflowfordiagnosis
Transactional,method-levelinsightfor Mainframecomponents
Criticaltoanswersingquestionslike:whichregionsorprogramsontheMainframe consumemostCPU/takethelongest?HowoftenistheMainframecalledforaend-user
action?
Securedatacapture Keepalltrackeddatawithinyourenvironment;thisiscriticalifyourmobileorwebapp handlesanyformofsensitiveinformation
Stableoverheaddatacapture Oneofthekeyenemiesofproductionoperationsischange.Yourdatacaptureapproach shouldbepredictableinallconditions.Approachesbasedonsamplingtechniquescan haveadynamicoverheadandwilladjusthigherbaseduponproductionevents.Dynamic
overheadaddsoverheadpreciselyatthetimewhenthesystemisencountering performanceissuesandcanleastaffordit.Somesolutionshaveastableoverheadwhere
theimpactofdatacaptureisfullypredictable.
APM EVALUATION GUIDE
Application Performance Management Evaluation Guide
Page 6
2.
Plan ahead and make sure your application is ready to handle traffic spikes.
3.
Find performance problems early in the development cycle and foster an environment that
promotes collaboration and innovation across Development and Operations
Capability
Benefit
Vendor1 Vendor2 Vendor3
Lastmiletesting Enablesyoutounderstandifyoursitecanbeaccessedandwhichlevelofperformance frommanygeographicallocationsacrosstheworld
Backbonetesting Mobilebackbonetestingallowsyoutotestyourmobileservicesinacleanroomsoyou candetermineiftheproblemisserviceorcarrierrelated.
3Gand4Gtesting Youneedtobeabletounderstandhowyourappperformsunderdifferentrealworld bandwidthconditions. Sinceperceptionisrealityyouhavetoensurethat,evenwith2 barscoupledwitha3Gconnection,yourapplicationisgoingtogiveagoodexperience. APIPerformancetesting Sincemobilesolutionsrelyonseveraldifferentservicestoperformtasks,youneedtobe
abletotestallAPIstoensuretheperformanceofanapplication. Justhavingdevice testingisnotgoingtogettothebottomofslowperformanceifithastodowitha
backendservice
Understand
your
site
or
application
readiness
before
go-live
Capability
Benefit
Vendor1 Vendor2 Vendor3
Offendingtransactionflagging Enablesyoutospotapoorlyperformingtransactioninproduction Exportandcapturetransactionsincluding
method-levelinformationforanalysis
Developerscananalyzeaproblemwithoutaccessingtheproductionsystem,without reproductingtheissue,bygettingthecompletecontextofanoffendingtransactionfor
code-levelanalysis AutomaticallycaptureandcompareKPIs
fromindividualunittestexecutions Providefeedbacktodevelopersontheperformanceimpactoftheirchangesveryearlyon inthereleasecycle.Leveragesunittestingforperformancetesting,byautomatically comparingindicatorsaffectingperformancebetweenanewbuildandanypreviousbuilds. Compareloadtestresultsacrossreleases Understandhowperformancehaschangedfromoneloadtesttothenext LoadRunnerandsynthetictestintegration EnablesyoutodisplayspecificresultsanddashboardsforLoadRunnertransactionsor
Syntheticusertestingplatforms.ImportantforenablingacompleteQAprocess AutomationHooks Enablesyoutointegratewithanautomatedcontinuousdeliverypipelineandtopinpoint
problemsbeforethebuildisapprovedfortesting.Enablesyoutointegratewithleading automationplatformssuchasMaven,Hudson,Bamboo,Jenkinsforautomaticbuild
successandfailures
Defineandrestrictaccesstosensitivedata Userspecificdatapointsmightbenecessaryforbusinesslevelmonitoring,butshouldnot besharedwithdevelopersduringtroubleshooting.Thiscapabilitiyenablesteamsto excludeconfidentialorsensitivedatawhensharingtransactionalproductiondatawith
internalpartiesorexternalpartners
APM EVALUATION GUIDE
Application Performance Management Evaluation Guide
Page 7
4.
End-users drive your business and your brand; their experience is the only one that
matters.
5.
Save time, money, and resources by determining what is within your control and ensure
Service-Level Agreements (SLAs) with 3
rd
parties.
Capability
Benefit
Vendor1 Vendor2 Vendor3
Userinteractionanalysis
Understandhowauserisinteractingwithyourapplication. Notonlydoestheviewneed tobeseenatanaggregateviewbutalsogroupedtogethertounderstandeachuser’svisit.
Abilitytomaptransactionstobusiness metrics
HowdoesyourMobilebusinesscomparetoyourWebbusiness? Areconversionrates whereyouwantthemtobe?Areyousuccessfulinyourtargetmarkets?Allthose questionscanbeansweredbyleveragingtheinstrumentationofend-usertransactions
andconnectingthosetransactionswithbusiness-relevantevents. GaplessUserVisitsandUserInteractions Capturinguservisitswithoutgapsenablesadetailedunderstandingofhowaspecific
customerisusingtheapplicationregardlessofwhetherornottheyareexperiencing performanceissues.Gaplessuservisitsenablesyoutoextractaccuratebusinessinsights,
understand usersnavigationpath,understandtheirlandingandexitpages,howlonga usertypicallystaysonyoursite,andhowtocorrectlycalculatebounceorconversion
rates.
Searchanddisplayaspecificuser Inmostcases,complaintresolution requiresthefullhistoryoftransactionsforauser. Thisenablesyoutounderstandhowauserwasinteractingwithyourapplicationunder normalconditionsaswellasafteraproblemwasdetected.Supportteamscansearchfor
aspecificuservisitandgetastep-by-stepviewofallassociateduseractionsaswellas eachspecificsystemsupportingthatvisit.
Appusagepattern Justknowingaboutproblemsdoesnothelpwhenitcomestomakingthedecisionabout what'snext. Youhavetobeabletoviewandanticipatewhatyourcustomerslikeand don’tlikeaboutyourapplicationandbeabletoseewhattheyaredoingfromtouchto swipe. Youhavetoseewheretheyaregoingandhowtheyareusingtheapplication. This
datahastobegroupedtogetherintoalaunchoraclickonabrowser. End-to-endoperationaldata Enablesyoutotiealltypesofoperationaldatatogether.Forexample,youneedtobeable
toviewallinstancesofmobile batterylife,clienttype,memory,Ram,andsignalstrength andthentiethatinformationtodatafromthebackendAPIsthatsupportthemobile
applications.
OperationalBigData Storeperformancedataforavirtuallyunlimitedamountoftimetoestablishbenchmarks andtrendperformanceovertime.Thisfunctioniscriticaltomaketherightbusiness decisionswithregardtoresourceplanningaswellasanticipatingtheusageofapps
Understand
and
optimize
your
end-users’
journey
Capability
Benefit
Vendor1 Vendor2 Vendor3
APIdependenciesanalysis MobileappsandmodernwebsitesrelyonarangeofthirdpartyservicesandAPIsto providefunctionstotheend-user. Thiscapabilityidentifiesthosedependenciesand enablesyoutoassesstheriskofperformanceissuescomingfromelementsoutsideof
yourdirectcontrol. 3rdpartyAPIandwebcontentoutage
monitoring
Enablesyoutoknowwhich3rdpartyservicesimpactyourend-userperformance. Monitorstheinternetforoutagesorslowresponsetimesfromthirdpartycomponents
thatareusedbyyourapporwebpages.
CDNperformancetrackingandanalysis Trackend-userexperienceandanalyzewhetherornotyourCDNproviderisdeliveringon theirSLAsinallgeographies
Location-basedperformancecomparison Havinganunderstandingofuserexperiencebasedonlocationiscrucial.Thiscapability enablesyoutounderstandwhyspecificlocationsareslowerthanothersandprovides
datatooptimizetheuser-perceivedperformanceforspecificphysicallocations Carrier-basedperformanceanalysis Helpsyouanswercarrier-relatedquestionssuchas:Docertainmobilenetworksaffectthe
performanceofyourapp?Isitamisconfigurationonyourpartorisitbadmobileservice?
APM EVALUATION GUIDE
Application Performance Management Evaluation Guide
Page 8
6.
Reduce the complexity of instrumenting, configuring, and managing your systems and
applications.
Do not hesitate to demand proof of these capabilities from the vendors you evaluate. Any
organization that can conquer this checklist will be equipped to master today’s exploding
complexity as well as be able to attack and adapt to tomorrow’s changes. As it is with any software
consideration, the evaluation process requires a combination of research and a hands-on proof of
concept to make sure you picked the right solution for your needs.
Capability
Benefit
Vendor1 Vendor2 Vendor3
Easeofuse,outoftheboxdashboards Enablesyoutoquicklyvisualizeandtrackkeymetricsusingasetofeasy-to-understand visualindicators
Speedofset-up Enablesyoutoinstrumentapplicationsandstartcapturingdatainlessthan60minutes AbilitytodefineBusinessTransactions EnablesyoutoconfigureBusinessTransactionsusingasimpleUIorwizard CustomizableDashboards&DataAnalysis Differentstakeholdersoftenwanttoseedatainawaythatmakessensetothem; this
capabilityenablesyoutocreatedashboardsthatmaximizetheuseofyourAPMsolution acrossabroadrangeofusers.Forinstance,acustomdashboardcouldshowthelistofthe top10slowestSQLstatementsonaOpsdashboardand thetop10mostoftenrequested
URIstoaneCommercedirector
AutomaticMobileinstrumentation Enablesyoutoinstrumentanyappinminutesbyusinganauto-instrumentationengine withoutrequiringaccesstotheappsourcecode
AutomaticBrowserinstrumentation Manualinstrumentationofpagesisimpossibletomaintainandtest. Anend-usersolution shouldbeabletoautomaticallyinjectandcollectbrowsermetricswithouttheneedfor
manualcoding. ItshouldalsoworkwithCDNand3rdpartyservices. Automatictransactiondiscovery Today'scomplexapplicationsmakeitimpossibletokeepupwithallthechangesin
transactionbehavior. Sinceyoumusthaveboththeserverandtheclientsidedetails,the solutionmustbeabletodetectanddisplayallthedifferenttypesoftransactionsinyour
environment. Drilldownintoindidivualtransactionsfrom
high-levelbusinessdashboard
Whenyourbusinessdashboardshowsaproblemorquestionablepattern,youcandrill downintototheindividualtransactiondetailsfromthesamedashboard
AutomaticTopologyDiscovery Enablesyoutoviewallthedependanciesandvisualizethem,foreveryexecutionofa businesstransaction,andforthewholeapplication
Abilitytoadd/removenewdataormethod capturepoints atruntimewithoutrestarts
Providetheabilitytogetdetailsduringdeepdiveanalysisbyaddingnewsensorswithout restarting(andpotentiallynotseeingtheissueagain).Also,tocaptureanewdatapoint forbusinessdashboardswithoutrequiring downtime
Automaticdeploymentupdates& rollbacks
Automaticallyupdatingallorselectedagentswithouthavingtomanuallycopyfilesto eachandeveryserver.YourAPMsolutionshouldjusttakecareofthat
APM EVALUATION GUIDE
Application Performance Management Evaluation Guide
Page 9
Author: Dr. Thomas Mendel Ph.D., Managing Director Research In Action GmbH,
Research In Action GmbH is a leading independent information and communications technology
research and consulting company. The company provides both forward-looking as well as practical
advice to enterprise as well as vendor clients.
© 2014, Research In Action GmbH reproduction prohibited.
1
The Research In Action IT Trends Surveys 2013and 2014 reveal additional interesting results on budget and spending
trends in Cloud, virtualization, performance management as well as ITIL and general IT operational benchmarks.
2
IT Service Management (ITSM) is a discipline for managing IT systems, philosophically centered on the customer's
perspective of IT's contribution to the business. ITSM stands in deliberate contrast to technology-centered approaches
to IT management and business interaction. See http://en.wikipedia.org/wiki/IT_service_management.
3
IT Infrastructure Library, see http://de.wikipedia.org/wiki/IT_Infrastructure_Library.
4
A Configuration Management Database (CMDB) is a repository of information related to all the components of an
information system. It contains the details of the configuration items in the IT infrastructure. Although repositories
similar to CMDBs have been used by IT departments for many years, the term CMDB stems from ITIL. See
http://en.wikipedia.org/wiki/Configuration_management_database.
5
Business Service Management (BSM) is a strategy, approach and methodology for aligning IT elements to the goals of
the business. BSM is about communicating the right message at the right time to the right level within IT and the
business in business terms or IT terms as appropriate for each audience. See
http://dougmcclure.net/blog/business-service-management/business-service-management-bsm-defined.
6
See http://en.wikipedia.org/wiki/Application_performance_management.
7
SaaS (Software as a Service) sometimes referred to as "on-demand software", is a software delivery model in which
software and associated data are centrally hosted on the Cloud. SaaS is typically accessed by users using a thin client
via a web browser. See http://en.wikipedia.org/wiki/Software_as_a_service.
8
The overall global IT budget spending grows typically at a two percent to three percent rate annually.
9
APM Convergence: Monitoring vs. Management - See
http://apmdigest.com/apm-convergence-monitoring-vs-management.
10
This can be a formidable challenge, since the global IT operations and service management market has more than
1,000 players. It is one of the markets with a very long tail of competitors. This is primarily due to the fact that the
customer requirements are diverse, large in number, sometimes local and specific and forever changing, as the
application landscape progresses. See http://www.researchinaction.de/ria%20mo%20itosm%20en.html.
11
An example from the ATP space can be found at
http://www.researchinaction.de/Vendor%20Selection%20Matrix%20en.html.
12