First Class Mobile Application Performance Management

(1)

First Class Mobile Application

Performance Management

August 2012

Jim Rapoza

~ Underwritten, in Part, by ~

(2)

August 2012

First Class Mobile Application Performance

Management

The rise of mobile applications and the devices that they run on has been long expected. However, the rate at which mobile web and native mobile applications are quickly becoming a primary way that critical business applications, content and digital properties are consumed has clearly taken many organizations by surprise. Based on initial findings collected in July 2012 from the Aberdeen group survey The Challenge of Application

Performance in a Mobile Application World, the number one pressure that is driving the application performance strategies of organizations is the increased importance of mobile application access (at 69% of responses).

However, only 30% of organizations have currently implemented any kind of mobile application performance infrastructure or strategy but 54% plan to do so. In this Aberdeen Group report, we’ll look at how tools are evolving to make mobile applications first class APM citizens and we’ll also analyze how businesses are working to manage and improve performance for their mobile application infrastructure and look at what the most successful organizations are doing to achieve their results.

Best-in-Class Performance

Aberdeen used the following four key performance criteria to distinguish Best-in-Class organizations:

• End user satisfaction: Measured as the year-over-year percent change in the number of end user complaints about mobile application performance and user experience

• Mobile Application adoption: Measured as the percent of users actually using the organization’s mobile applications, on a scale of 0%

to 100%.

• Employee Satisfaction with application performance: Measured on a scale from extremely satisfied to extremely dissatisfied.

• Customer Satisfaction with application performance: Measured on a scale from extremely satisfied to extremely dissatisfied.

The Challenge of Understanding Performance for

Mobile Apps

One of the biggest challenges when it comes to managing performance and improving end-user experience of mobile applications is the unique

deployment and device delivery requirements of mobile apps, which typically require deployment to multiple device types and across different app stores

Research Brief

Aberdeen’s Research Briefs provide a detailed exploration of a key finding from a primary research study, including key performance indicators, Best- in-Class insight, and vendor insight.

Defining the Best-in-Class Aberdeen’s research for, First Class Mobile Application

Performance Managements, used four key performance criteria to distinguish the Best-in-Class (top 20% of aggregate

performers) from the Industry Average (middle 50%) and the Laggard (bottom 30% ) organizations:

√ Number of end user complaints about mobile application performance

√ Mobile application adoption rate

√ Employee satisfaction with mobile application performance

√ Customer satisfaction with mobile application performance

(3)

First Class Mobile Application Performance Management Page 2

www.aberdeen.com Fax: 617 723 7897

or app deployment systems. Also, the difficulty of getting visibility into mobile app performance once the connections leave company networks and travel over external and carrier networks means that traditional monitoring and end-user experience management tools often have gaps in what they can detect.

As shown in Figure 1, the approaches that businesses are taking to track performance and user experience for mobile applications differ greatly from how they tend to track traditional applications.

Figure 1: Key Indicators Used to Evaluate Application Performance

Source: Aberdeen Group, July 2012

The top two methods for tracking traditional application performance (application response times and SLA achievements) are among the lowest for tracking mobile apps while the main methods used to track mobile app performance (social trend analysis and user ratings on app stores) barely register as ways to track traditional applications.

87%

86%

85%

81%

63%

58%

21%

19%

57%

53%

75%

67%

69%

73%

86%

88%

0% 20% 40% 60% 80% 100%

Application response times Level of SLA achievements Calls to help desk Application availability Carrier network performance Transaction conversion rate Social trend analysis User ratings on app stores

Percentage of respondents, n=63 Mobile Applications Traditional Applications

Fast Fact

√ 25% of users abandon a mobile application after three seconds of delay

(4)

This data reveals one of the biggest problems facing mobile applications. The primary ways that the performance and user experience of mobile apps are tracked are based on detecting user problems AFTER the app has been deployed. Ideally, a business should be able to find the problems in their apps before users start sending negative tweets or posting bad reviews on app stores.

Proactively Improving Mobile App Performance

One key way to find application issues before an application is deployed to end-users is to carry out effective testing during the development stage. As shown in Figure 2, 42% of organizations plan to do performance testing of mobile applications in the development stage and 35% plan to do mobile app functionality testing during development.

Figure 2: Top Actions Taken to Address Mobile Performance Issues

Along with the usage of testing in the development stage, businesses are also planning to take actions to redesign and optimize mobile apps and improve mobile apps based on end-user experience. All of these actions have the benefit of providing valuable information about mobile application performance and user experience before the apps are deployed to the end- user population.

Building Out to Mobile App Domination

Not too long ago, mobile application access (both mobile web and native) was seen as a nice option to have but not typically vital to business interests.

But nowadays more organizations are finding that mobile devices are indeed being used for vital business interests, such as securing sales while in the

33%

35%

39%

42%

0% 10% 20% 30% 40% 50%

Improve mobile apps based on end- user experience

Mobile Functionality Testing During Development

Convert existing applications to mobile apps

Redesign or optimize mobile apps Mobile Performance Testing During

Development

Percentage of respondents, n=69

"Mobile is a disconnected use case with occasional

connections."

~ Executive at UK-based social software developer

(5)

field or detecting new opportunities for business growth. And, in many ways, these mobile users require high performance and reliability as much as, if not more then, workers on the company network.

While many businesses are still behind in their adoption of tools and technologies to manage, monitor and understand mobile application performance, looking at their planned adoption shows that, within a year, many organizations expect to be deploying dedicated systems to manage mobile application performance.

Figure 3: Mobile Application Performance Management Capabilities

Source: Aberdeen Group, July 2012 17%

19%

20%

27%

21%

20%

18%

7%

40%

46%

51%

55%

61%

62%

79%

0% 20% 40% 60% 80% 100%

Software or service for mobile traffic and load simulation

Simulation and testing of loads and real-world traffic scenarios for mobile applications Software or agent for monitoring mobile end-

user experience

Test mobile performance of real-world users Use end-user experience to inform mobile

application design or changes Capture of mobile end-user experience data

Measure mobile application response times Ability to perform root-cause analysis on

mobile application issues Define baselines for acceptable mobile app

performance

Percentage of respondents, n=66

Currently Implemented Plan to Implement Fast Fact

Top Challenges of Mobile Application Performance

√ 51% - Diversity of mobile devices and operating systems

√ 48% - Lack of tools to measure, monitor and test mobile application

performance

√ 43% - Complexity of mobile development platform

(6)

The top four capabilities that organizations plan to deploy in the next year include some of what would be considered core elements of traditional performance management systems. These include the ability to define performance baselines (79% plan to adopt), perform root-cause analysis of mobile application performance (62% plan to adopt) and measure mobile application response times (61% plan to adopt). These basic but necessary capabilities make it possible for organizations to gain a better understanding of their overall mobile application performance and user-experience and lay the groundwork for the use of more advanced testing, analytics and

optimization technologies.

User Satisfaction is Key to Mobile App Performance

While the focus of traditional performance management tools has often been centered around network performance and bandwidth utilization, recently, more and more products in the performance management space have realized that the focus needs to be on end-users. No user really cares how the network is running, they only care if the applications that they rely on are performing well and providing good usability.

Given this, the most important metric for any application is user satisfaction.

In our survey, organizations achieving the best results in mobile application performance were clearly providing a good end-user experience for their mobile applications.

Figure 4: User Satisfaction with Mobile App Performance and Usability

As shown in Figure 4, organizations identified as being Best-in-Class had 86%

of users identified as Satisfied or better with mobile app performance and they had 59% very or extremely satisfied. This is compared to Industry Average companies that had 43% of users less than satisfied and 10% of

10%

33%

22%

2%

0%

14%

27%

45%

14%

0% 10% 20% 30% 40% 50%

Very Dissatisfied Somewhat Satisfied Satisfied Very Satisfied Extremely Satisfied

Percentage of respondents, n=71 Best-in-Class Industry Average

"Businesses need to capture feedback from users on a real time basis and innovate to make applications offer critical value to customers"

~ IT Manager, African-based payment processing firm

(7)

users identified as very dissatisfied. In the end, no number of green lights on a performance management dashboard will matter if end-users are

dissatisfied with a mobile apps performance and usability. Happy users don't light up social networks with complaints or post negative reviews on app stores.

Key Insights

More and more users are now turning to mobile devices such as tablets and smartphones as their primary computing platform. And businesses are responding to this trend by delivering more and more of their core web and native applications to these mobile systems. But if an organization is relying on outdated methods and tools to handle application performance

management and user experience monitoring, they will find that these older tools can't address mobile application performance in the same way that they handle traditional applications. Organizations should look to these insights in order to be better prepared for the rise of mobile applications:

• Understand the difference between mobile and traditional applications. It can be easy to assume that there is little difference between managing the performance of a mobile web or native app and managing the performance of traditional web and desktop applications. But these are completely different beasts, as shown in Figure 1 in the radically different ways that organizations are

currently tracking mobile application performance. Once businesses understand these differences, they can take the necessary actions to build a mobile ready performance management infrastructure.

• Don't wait for users to complain about apps. Relying on user responses on social networks and app store rating to find out how mobile apps are performing means that it is already too late for a business to fix problems. 42% of organizations plan to utilize testing during the development stage to head off potential performance and usability problems before the apps end up on user mobile devices.

• Take action now. Every bit of information and insight that an organization can get on mobile application performance and usability can pay off in avoiding problems. Among the top capabilities that businesses plan to deploy in the next year are core performance management features such as root cause analysis, performance baselines and mobile application response times. Using these capabilities will help these organizations lay the groundwork to utilize all of the capabilities of an end-to-end mobile performance infrastructure.

• Focus on the user. While every bit of performance and uptime data is useful to understanding mobile application performance, in the end the only metric that matters is end-user satisfaction.

Organizations that focus on the user are able to build and optimize mobile applications that provide high levels of user satisfaction.

(8)

While the rise of mobile applications is coming more quickly than some may have expected, organizations that are well prepared are already seeing success when it comes to providing high-quality mobile application

experiences. As more of the performance management products turn their focus towards the management, optimization and monitoring of mobile apps, the second class status of mobile applications will end and organizations will expect to have all the performance management and optimization capabilities for mobile apps that they currently enjoy for traditional applications.

For more information on this or other research topics, please visit www.aberdeen.com.

Related Research End-user Experience Management:

Questions to Ask; July 2012

Convergence of Social, Mobile and Cloud Requires User Focused End-to-end App Approach; February 2012

Boosting Enterprise Application

Performance in Distributed Environments;

April 2012

Enterprise Mobility Requires New Strategies for Optimizing Application Performance; January 2012

Author: Jim Rapoza, Senior Research Analyst, Network and Application Performance ([email protected])

For more than two decades, Aberdeen's research has been helping corporations worldwide become Best-in-Class.

Having benchmarked the performance of more than 644,000 companies, Aberdeen is uniquely positioned to provide organizations with the facts that matter — the facts that enable companies to get ahead and drive results. That's why our research is relied on by more than 2.5 million readers in over 40 countries, 90% of the Fortune 1,000, and 93% of the Technology 500.

As a Harte-Hanks Company, Aberdeen’s research provides insight and analysis to the Harte-Hanks community of local, regional, national and international marketing executives. Combined, we help our customers leverage the power of insight to deliver innovative multichannel marketing programs that drive business-changing results. For additional information, visit Aberdeen http://www.aberdeen.com or call (617) 854-5200, or to learn more about Harte-Hanks, call (800) 456-9748 or go to http://www.harte-hanks.com.

This document is the result of primary research performed by Aberdeen Group. Aberdeen Group's methodologies provide for objective fact-based research and represent the best analysis available at the time of publication. Unless otherwise noted, the entire contents of this publication are copyrighted by Aberdeen Group, Inc. and may not be reproduced, distributed, archived, or transmitted in any form or by any means without prior written consent by Aberdeen Group, Inc. (2012a)

(9)

Featured Underwriters

This research report was made possible, in part, with the financial support of our underwriters. These individuals and organizations share Aberdeen’s vision of bringing fact based research to corporations worldwide at little or no cost. Underwriters have no editorial or research rights, and the facts and analysis of this report remain an exclusive production and product of Aberdeen Group. Solution providers recognized as underwriters were solicited after the fact and had no substantive influence on the direction of this report. Their sponsorship has made it possible for Aberdeen Group to make these findings available to readers at no charge.

SOASTA’s web and mobile app test automation solution enables developers, QA professionals, and IT Operations teams to test with unprecedented speed, scale and precision. Our innovative solution

streamlines test creation, automates provisioning and execution and distils analytics to deliver actionable intelligence faster. With SOASTA, you can have confidence that your app will perform as designed, even in peak traffic.

For additional information on SOASTA:

SOASTA

444 Castro Street, Fourth Floor Mountain View, CA 94041 Telephone: 866.344.8766 www.soasta.com

[email protected]

(10)

New Relic is the all-in-one web application performance tool that lets you see performance from the end user experience, through servers, and down to the line of application code. New Relic provides visibility to all major web application languages – Java, .net, Python, Ruby and PHP. With customers that include Comcast, Intuit, Groupon and AT&T Interactive, New Relic is the largest SaaS-based performance monitoring provider, measuring over 60 Billion metrics for its more than 25,000 customers each and everyday. The product is delivered as a service and available for free at newrelic.com . For additional information on New Relic:

New Relic

101 Second Street, 15th Floor San Francisco, CA 94105 Telephone: 888.643.8776 newrelic.com