Developing a Mobile Application Performance Management Strategy

(1)

Whitepaper

Evidant Inc.,

Developing a Mobile Application

Performance Management

Strategy

(2)

Preface

Mobile has rapidly become the new battleground for acquisition and retention of new customers in the Business-to-Consumer (B2C) space. In 2011, 38% of mobile subscribers in the United States had a

smartphone. In 2012 that number grew to over 50% and the usage of applications has jumped by 28%. The case is clear that mobile is not only here to stay but must be a key part of any B2C business strategy. Today 80% of the U.S. smartphone market is accounted for by Android and iOS operating systems.

In developing a strategy around mobile websites and applications, there are three key elements to focus on:

1. Determine what your customers want to do.

Typically, designers will focus on potential use cases that tie to a series of mobile transactions which enable the customer to perform a task (e.g. view available checking account balance in a mobile banking application)

2. Implement the user interface to enable customers to accomplish these key tasks efficiently.

Limited screen space requires a focus on the most important tasks that customers want to accomplish. A typical rule-of-thumb is no more than 2-3 screens to accomplish a task.

3. Monitor the Performance and Availability of your mobile website or application to optimize the customer’s experience.

This paper will focus on this third element of building a mobile presence strategy.

(3)

COMMON PROBLEMS

Mobile Customer Expectations Not Considered During Development

Understanding the customer’s experience is an often unsatisfied need when it comes to deploying either mobile websites or applications. Typically, a tremendous amount of effort is spent on understanding the key tasks that users want to perform on a mobile device along with the “look and feel” of the user interface.

These elements are absolutely critical in the development of a successful mobile website or application.

Efforts around assuring that the mobile customer expectations are satisfied from an availability and performance perspective are typically not considered or are relegated to traditional IT monitoring.

Mobile “Bolted on” to Core Systems

As mobile websites and applications are deployed, they are frequently “bolted on” to existing platforms and infrastructure. Although typically not an issue, often these “core systems” were not architected with mobile as a consideration. The challenge is assuring that these “core systems” and the mobile “bolt-ons” work together well and provide a seamless environment for the customer. The brunt of poor performance or availability is most often felt in your customer call centers or on social networking sites, resulting in increased support costs, the potential for losing existing customers, and negatively influencing prospective customers.

Not Capturing the Customer Experience

This element is typically only thought about after customers are complaining, and by then the perception of your organization is being affected negatively. IT organizations will typically first rely on traditional server-based or network-based monitoring approaches to determine whether the application is performing well. These approaches, while useful, only provide a marginal part of the picture of performance. To truly quantify and understand the customer experience, a strategy around monitoring the actual end-user experience itself must be developed. This means measuring mobile website and application performance

“from the outside in”.

Failing to Use Tools Strategically

There are numerous “tool” vendors that provide capabilities for measuring mobile performance and availability. Tools are not the issue. The challenge is having a strategy that drives the action regarding

mobile user experiences that can negatively impact usage and perception.

Typical IT Infrastructure Monitoring Puts Too much Emphasis on Monitoring the Wrong Thing and Fails to Identify the Impact on End-Users

(5)

A TESTED SOLUTION

There are three key elements to a valuable mobile monitoring strategy:

1. Capture the customer experience in a way that most closely represents it.

2. Enable visualization of the customer experience through effective reporting and dashboards.

3. Establish a Mobile Customer Experience team to monitor the customer experience and proactively establish programs to prevent negative impacts on the mobile customer experience.

Capturing the Customer Experience

The key to all successful Application Performance Management strategies is capturing the “Customer Experience” as close to what it looks and feels like from the customer’s point of view. In the world of mobile website and application monitoring, this effort centers around two basic methods: Active and Passive monitoring.

Active monitoring uses a “robot” to simulate what a customer would do either on a mobile website or application utilizing. This “robot” goes through a predefined set of actions that simulate what an actual customer would perform. The benefits to active monitoring are;

 it enables a baseline of mobile performance, because it uses the same set of transactions over and over again, and

 it establishes when the mobile services are unavailable from a customer perspective.

A weakness of passive monitoring is that when users are not doing anything there is nothing to monitor.

Passive monitoring captures the experience of actual users. There are a range of technologies available to accomplish this which typically requires an insertion of code into either mobile web pages or the mobile application itself. Although development groups are often reticent to add more code to their mobile platform, these solutions can provide a wealth of detailed information such as: Wi-Fi connectivity or cellular network strength, in-call or roaming status, low power mode, CPU load, available storage or OS version. The benefits of Passive monitoring are;

 it helps understand the performance mobile customers are actually experiencing, and

 the relative volume of these transactions to help identify what they do the most, where they are abandoning out of a task and what the trend looks like for the mobile customer community, and

 the ability to monitor processing transactions (e.g., “Pay a Bill”, “Submit a Claim”, etc.)

This is cannot be done with Active monitoring unless your system is set up to allow dummy account activity (a rare situation in our experience).

The main advantages that passive monitoring has over active is insight into the performance actual customers are experiencing, and the availability of volume-per-transaction metrics that can provide additional insight into user behavior within the application or website.

There are a range of tools vendors on the market today that can deliver one or both of the solutions above.

Selecting these solutions often becomes the key effort that a Mobile Customer Experience Team will focus on. During this process, it is critical to remember that tools do not deliver value; how you use the tools delivers value. Many business executives often incorrectly believe that they have a mobile customer

(6)

experience strategy once a “tools vendor” has been selected. When in fact they should be asking, who will deploy and maintain the tool, how will the organization be able to visualize what customers are experiencing and, most importantly, how will the organization take action on this information.

This paper does not attempt to compare and contrast the various active and passive tools vendors in the mobile monitoring space. This market is moving rapidly and the selection process, although important, is the least critical element of executing a well-developed strategy.

Enabling Reporting and Visualization of the Customer Experience

The ability for an organization to visualize the customer experience is the next important piece of a mobile monitoring strategy. It clearly doesn’t make sense to capture the customer experience if it cannot be easily viewed and understood by decision makers in the business and IT teams. We’ve identified some of the typical mistakes that organizations will make when trying to provide customer visualizations:

 Too technically focused. Tools

vendors provide rich visualizations, which can be extremely valuable when doing technical root cause analysis; however, they are typically far too complex to be valuable for obtaining a rapid

understanding of the customer experience.

 Excessive use of “cool” dashboard graphics. There has been a tendency to over utilize a whole range of flashy elements to produce “cool looking” but difficult to quickly interpret dashboards.

This dashboard is an example of one with multiple different presentation elements.

This requires the user to understand and interpret the various elements and then make a decision. Experience has taught us that simplicity, as with most things in life, removes obfuscation (i.e., keep it simple).

 Reporting that is oriented toward “baselines” versus expectations. A common mistake is to use current mobile website or application performance as the benchmark for what needs to be delivered to customers. This approach, although valuable from a technical or even six-sigma perspective, misses the point. It is critical to understand customer expectations and then measure against them. Performance that fails to meet customer expectations means a frustrated and

potentially lost customer. Exceeded customer expectations, while at first sounds terrific, potentially means you overspent on your mobile platforms, and yet it also shows the room for growth you have before requiring additional expenditure.

The “Too Cool” to Understand Dashboard

(7)

The key to presenting a view of the mobile customer experience is having a clear understanding of their expectations and then measuring against those expectations. The following are some of the critical parameters to identify for mobile customers:

 The key tasks that mobile customers perform (e.g. view available checking account balance in a mobile banking application) and the specific screens that the customer needs to walk through.

 The customer expectations of performance for each screen, ideally characterized by seconds, into:

Acceptable:

“Good enough” to meet customer expectations.

Critical:

Slow enough that some customer will begin to abandon the webpage or application.

Unavailable:

So slow that most customers will begin to abandon the webpage or application.

Although IT organizations don’t view unavailability in this way, customers, mobile or otherwise, definitely perceive application or website availability this way.

Getting it perfect is not nearly as important as getting a mobile customer experience initiative going. Never-the-less, you want to avoid the following traps.

 Trying to monitor it all versus monitor what is most important. It is far better to focus on monitoring the 20% of tasks that customers perform 80% of the time than attempting to monitor 100% of what customers might do. The 80:20 Rule should drive this effort and will not only speed up the process and but takes less time and money to execute.

 Trying to get is right the first time. In the mobile market, time to knowledge is critical and organizations will often waste time in an attempt to perfect reporting, versus taking an educated guess and then tweaking it later. This is especially true when it comes to determining mobile customer expectations around mobile website or application performance. You don’t need a focus group of mobile customers to tell you that at 15 seconds for a screen update, some mobile

customers will get aggravated and abandon the task they were attempting to accomplish.

 Avoid the use of “averages”. This is a typical IT view that focuses on average response time. The adage that a man with his feet in a freezer and his head in an oven “on average” is just fine,

highlights why averages should be avoided. Instead, we recommend you focus on numbers of instances that meet, or fail to meet, mobile customer expectations.

(8)

Assembling this information into an easy to understand presentation that shows the overall mobile customer experience, such as this dashboard, is the next step. In this example, it is easy to see that the

“Android App” has not been meeting customer expectations over the past 2 hours and the amount of

“orange” indicates that a sizeable number of mobile customers are probably abandoning the task they were trying to accomplish. The next step is taking action.

Establish a Mobile Customer Experience team

This team is responsible for developing and delivering a mobile monitoring strategy that is functional and echoes the desired experience of the customer. Companies will often attempt to develop and even deliver this strategy by cobbling together people versus assigning a single team to focus on this effort. Customer experience, especially in the mobile space, is too important not to give it a strong focus. The Mobile Customer Experience team is the closest “voice of the customer” a company will typically have.

Capturing and visualizing the customer experience enables the organization to understand and make decisions about the mobile customer experience. It does not mean that every dip in performance or unavailability needs to be acted on. What it does mean is making a conscious decision to take action based on what mobile customers are experiencing. The following are the keys to success in this area:

 Establish a Mobile Customer Experience Team that is focused on metrics – this can be a single person. The leader of this team must:

o “Own” the mobile customer experience metrics. This means having a deep understanding of tools and underlying data to provide information on how mobile customers are impacted.

o Be able to communicate with both mobile business executives and key IT stakeholders about how monitoring information is indicative of the impact on mobile customers. This

Mobile Customer Experience Dashboard that BOTH IT and Business canUnderstand

(9)

individual needs to use mobile customer experience metrics to provide an “outside-in” view and does NOT need to be from IT.

o Have senior executive support from both IT and the mobile business.

 Developing “actionable” alerts for the organization. There is an important distinction between alerts and ones that are actionable. Organizations will often turn on alerting systems and then don’t maintain or tune them. This often results in either the organization not getting alerts, or worse, getting so many alerts that it isn’t possible to decide when to take action. Here are some keys to developing “actionable” alerts:

o The Mobile Customer Experience Team needs to “own” the alerting, especially early in their establishment. A best practice is to initially send out too many alerts (i.e., risk the potential of a high number of “false positives”) to a small group or individual who can then tweak or tune the alerting to something with a small number of “false positives”.

o The Mobile Customer Experience team needs to drive alerts to the team or individual that it relates to. The idea of sending out all alerts to everyone is flawed and misses the concept of “actionable” alerts. These alerts, in order to be “actionable”, must go the team or individual who are most likely to be able to address the issue.

o The alerts must have a sufficient amount of information about the problem so recipients understand the issue and how it is impacting mobile customers, enabling them to take action.

 Review trends on a regular basis. The Mobile Customer Experience Team should engage mobile business executives and IT stakeholders in regular review of the mobile customer experience against the customer expectations that were established. The trend analysis should review the mobile customer experience over weeks and even months to capture slow degradations and periodic events that could negatively impact mobile customers. The mobile business teams should bring reports of customer experiences from customers call centers or on social networking sites to help identify gaps.

The leader of the Mobile Customer Experience Team needs to drive this effort working across the organization.

 Use the Mobile Customer Experience information to make decisions about changes, improvements, investments and event to request IT to perform root cause analysis. This

information should provide evidence for return-on-investment (ROI) decisions and then utilized to validate that the ROI was achieved. In addition the organization should use this information to validate that infrastructure changes do not negatively impact mobile customers by integrating the Mobile Customer Experience team into the organization’s change control process.

There is often a concern that the establishment of “another team” means more meetings and potentially less time for “real work” to get done. This is a reasonable concern and makes the selection of the Mobile Experience Team leader, whether internal or external to the organization, critically important to the success of this initiative.

CONCLUSION

There is little doubt that mobile is a key element to retaining and gaining new customers in the B2C market space. The ROI justifications for mobile website and applications are challenging; however, it has moved from a “nice to have” to a “must have” for the customer. The mobile community is also less forgiving when it comes to responsiveness of a website or application. Unlike the desktop web environment when the user has the flexibility of doing something else if a website is slow, on a mobile device, this isn’t an

(10)

option. When you add that many mobile platforms are integrated with core systems that may not have been designed with a mobile experience in mind, it becomes critical to have a strategy around understanding the mobile customer experience that as closely as possible represents the customer’s experience and taking action on this information.

The establishment of a small Mobile Customer Experience Team provides the kind of focus needed to develop and execute an effective mobile customer experience strategy. This team should be small with the ability to work effectively between across the organization to bridge the communication gap between Business and IT. The Mobile Customer Experience Team leader should have “customer experience” as their key and ideally as their only focus. Great effect can be realized by bringing an individual or

organization that is independent and has experience in the customer experience space.

(11)

ABOUT EVIDANT

Since being established in 2002 Evidant has been a pioneer in aligning Application Performance Management with End-User Productivity requirements.

Evidant's core belief is that the value to the enterprise of Application Performance Management (APM) does not lie in the capabilities of the tool alone, but in the way it is configured to match End-User Productivity requirements. This requires working with both the IT and Business units of an enterprise to form a common metric - and that is what we excel at. We partner with your IT and business units to define, implement, analyze and report on your APM needs.

Developing a Mobile Application Performance Management Strategy

Whitepaper