Load Testing
W H I T E P A P E R
How to Make Sure Your Site is Available When Your Users Want it Most
I
ntroduction
The stories hit the news every year shortly after Black Friday, Cyber Monday and Super Bowl Sunday – websites and apps that failed miserably by slowing to a crawl or crashing when users wanted them most.
Sometimes companies know the horde of traffic is coming – if you’re a retailer, the holiday shopping season will slam your apps; if you’re advertising during a highly watched televised event, people will flock to your site; if you’re launching the next installment of a much loved game, users are counting down the seconds until they can access it. For other companies, heavy load is unexpected. Maybe a celebrity name drops your boutique, or your game becomes an overnight sensation or you break an important news story. In either situation, if your app fails under load, users will come away disappointed, likely voice their displeasure on a range of social media and might never return to your app again.
With people increasingly turning to websites and mobile apps for research, news, purchases, entertainment and other tasks, your app faces the very real possibility of experiencing heavier load than you ever anticipated. That’s why regular load testing is so important.
Load testing is performed for many reasons, each to address a different issue. Here are a few of the more common load testing terms and scenarios:
• Software Stress Testing: Stress testing is intended to determine where the appli-cation begins to break down when under load.
• Soak Testing: Taking place over days or weeks, soak testing tests for perfor-mance degradation over time caused by things like slow memory leaks or even something as simple as hardware fatigue or subpar hosting conditions.
• Spike Testing: This is your typical Super Bowl situation. Spike testing helps com-panies determine an app’s scalability under a sudden increase in load (like when millions of viewers see your commercial at the same time). The ability of an ap-plication to shift gears from handling a moderate number of users to handling a large number of users is extremely important, especially if your company is preparing for a major campaign or busy season. Even if you don’t have an antici-pated peak season, sudden word of mouth or media buzz can unexpectedly slam your app with users.
• Platform Testing: This allows companies to test their app’s front end under a wide variety of hardware and software configurations. This helps teams deter-mine software requirements, which they can publish as their suggested mini-mum configuration.
Whether you have a website, a web app, an online game supported by an army of backend servers or a mobile app relying on web APIs, load testing is vital to making sure your application holds up under a trickle or a wave of traffic, for a few minutes or a few days.
L
oad Testing Approaches
In the past, load testing was expensive, time consuming and difficult to coordinate. Not anymore. Today, there are several approaches to load testing to help you identify breaking points, pinpoint bottle necks and address functionality under load.
Synthetic Load
Synthetic load testing simulates heavy traffic with remotely-run load scripts to test your app under a variety of scenarios. The benefits of this approach
are that you can simulate extremely high traffic situations and run test cycles in relatively little time.
Synthetic load testing is good for testing situations such as:
• High-Traffic Apps: If you’re expecting a lot of concurrent users, synthetic load testing can provide you with hundreds of thousands of simulated users – helping you identify any load-related weaknesses and ensure performance during peak usage.
• Fast-Approaching Deadlines: If you already have load testing scripts, a synthetic test cycle can be completed in relatively little time while providing you with valu-able insight.
• Testing Changes: This relatively simple test can help assure you that any recent changes in your infrastructure “chain” (i.e. features, functionality or even con-tent) will not cause an issue for your users.
• Limited Resources: The advent of open source load scripts and cloud-based load testing services have eliminated much of the heavy lifting once associated with load testing, making adequate testing much easier and simpler for teams of any size.
While synthetic load can help you determine when performance starts degrading and highlight bottlenecks, it’s not a one-stop-shop for ensuring a good user experience. To make sure your app looks and functions well in the eyes of actual humans, a degree of live testing is crucial.
Live Load
Live load testing is essentially manual load testing without the aid of simulators or load tools. Instead, a team of real live testers simultaneously converge on your app or the core features you want tested.
Load testing with live users gives you the added benefit of a human perspective. Where a synthetic load test can tell you exactly how long it takes a page to load, a human tester can tell you how that wait feels – did the app seem to load quickly or did it feel like they were waiting forever. Seconds are an
important benchmark, but ultimately your app isn’t being used by machines and the app needs to meet real user expectations. While everyone will have a different opinion of how long is too long, load testing with live testers will give you a sense of how long people are willing to wait and if your app might lose users because of wait time.
In addition to load time insight, live testers can give you true feedback on how your application performs. While your app might technically be able to perform an action – such as loading a video or clicking a link – while under load, the synthetic test won’t tell you if the video is choppy or if the link takes so long to react that people are compelled to keep clicking. Remember, when dealing with real people perception is just as important as actual functionality. If users think your app isn’t work, they’ll leave.
Common live load test use cases include:
• Multimedia: If your app includes media such as images or videos, an automated test may indicate that all is well, even when it’s not. Live testers can determine if sound quality is off, if videos or images are pixilated, if files take too long to buffer or other problems that simulated load testing rarely catches.
• Limited Traffic Apps: If you anticipate relatively stable app usage and peak traf-fic that doesn’t spike too high, you may be able to test entirely with live users. This will give you the most accurate representation of what your users are likely to experience.
• AppTurns: The number of components (images, scripts, CSS, etc.) needed for the page to render.
• End-to-End Load Testing: For companies that want in-the-wild testing that mir-rors the true user experience – including users located around the world and using different device configurations – live load testing is an ideal solution.
However, getting enough concurrent users to perform a live load test can be expensive and hard to coordinate, especially if you expect a lot of traffic or want to prepare for an especially unexpected spike. Luckily, a hybrid load
testing approach has emerged to offer you the best of both worlds.
Hybrid Load
Hybrid load testing is exactly what it sounds like – a mix of synthetic load and live load testing. Your application is put under synthetic load to simulate heavy traffic, and then a smaller number of live testers put your app through its paces to see how it performs while under pressure.
In this scenario, the live testers perform more like functional testers, exploring your app or executing test scripts to evaluate real and perceived load time, validate that core functionality is still running smoothly and to see which features might suffer. This two-pronged approach gives you the real data of synthetic load coupled with live user feedback about perceived wait times and actual feature functionality.
Hybrid testing also allows you to test your application under the heaviest load (you can’t really get 100,000 live users for a test, but you can simulate the traffic) without losing the benefits of a live load test. Other benefits include:
• Multimedia: For companies that need complete load testing, but also want to verify that multimedia components are working properly with no degradation under peak load, hybrid testing makes the most sense.
• Load-Related Defects: Some bugs – notably GUI bugs – only show themselves when an application is under heavy stress. Hybrid testing enables companies to discover these defects through first hand bug reports from real testers.
T
ips
Before you decide on a load testing approach, there are a few other things to take into consideration. Reviewing these tips before you embark on performance testing may save you time and effort going forward.
Test What Matters
Performance testing is more than just measuring how long it takes for your app to launch and fully load. To get the most out of your load testing efforts, make sure your scripts address anticipated or popular use cases and features that will have the greatest negative impact if they fail.
For instance, a load test for an eCommerce app should cover launching the app, searching for a product, selecting that product page, adding the item to the cart and following through with the purchasing process. An online game that requires players to log in should test page load, the login process, any profile setup or selection, entering and playing the game, saving profiles and high scores and logging out.
Loading the initial page of your app is only the threshold of app use; load test to be sure the rest of the application runs smoothly and quickly no matter the traffic.
Test In-The-Wild
Synthetic load testing is good for an initial load test to address early problems, but getting live eyes on your app while it’s under pressure could be the difference between success and utter failure. When live or hybrid load testing, don’t rely solely on a team sitting in a central office. Your users will be accessing your application from around the country – or the world – and it’s
important to make sure that everything works as planned from their point of view.
Your users will be accessing your application from different networks and connectivity speeds, using different software versions and hardware configurations. All these factors may result in some unanticipated issues if your application is already struggling under heavy load. To accurately mirror real life usage, enlist live testers from a wide geographic area to assist with your load testing.
Know What You’re Looking For
For an effective load test, it’s important to you know what you need to get out of the effort. It may seem obvious, but it’s a routine mistake. To reach any actionable conclusions (which is the entire point of load testing), companies must be able to identify where specific failures occurred in the delivery chain. Did the major failure first occur with your database or code? Did the issues originate from your web server, load balancer, content delivery network or ISP? Companies that fail to pinpoint these issues will learn a valuable lesson in the costs of redundant testing.
Don’t Rest on Your Laurels
Your app may have held up to a tidal wave of traffic last time, but that doesn’t mean it will this time. Each time you make a major update (to the front end or backend), a little technical tweak, add a new feature, start highlighting a new type of media, see a steady uptick in traffic or are about to enter a busy season or launch a big campaign, a new round of load testing is in order. Again, know what you’re looking for so you can quickly address an issue
should one arise. “Better safe than sorry” may be a cliché, but when your brand’s reputation, public relations and bottom line depend on an app that works, you need to take steps to ensure it does work.
Testing in Production
If you don’t have a built-up staging environment, don’t worry, you can absolutely load test in production. Be mindful to run tests during slow times – while it may be tempting to use the load your app already has, it’s not a good idea to potentially crash your application while your real users are engaged during peak traffic times. If you plan to load test in production, start small. Most defects are discovered by low-volume tests and by live testers, so why destroy your live app with unnecessarily heavy load?
Once any issues that appear during low-volume tests are identified and addressed, you can move on to more comprehensive load testing. At that point, it will be helpful to have a staging environment. If a staging environment just isn’t in the cards, test during your lowest traffic times and consider warning your users about the upcoming test by posting a message in your app.
C
onclusion
For users, the cost of switching apps is low – a few taps, swipes or clicks later and they’ll never have to deal with your slow or downed app again. In today’s economy, that could mean the difference between a company thriving or disappearing forever.
Load testing is a simple step you can take to make sure that you app holds up to your everyday traffic and to the pleasant surge of visitors you’d always
hoped for. With the extensive options of load testing approaches, tools and services available today, company’s have no reason to ignore this vital step of app stability and success.
About Applause
Applause is leading the app quality revolution by enabling companies to deliver digital experiences that win - from web to mobile to wearables and beyond. By combining in-the-wild testing services, software tools and analytics, Applause helps companies achieve the 360° app quality™ they need to thrive in the modern apps economy. Thousands of companies – including Google, Fox, Amazon, Box, Concur and Runkeeper – choose Applause to launch apps that delight their users. Learn more at www.applause.com.
