Testing & Assuring Mobile End User Experience Before Production
Neotys
Agenda
Introduction
The challenges
Best practices
NeoLoad mobile capabilities
Mobile devices are used more and more…
At Home
In 2014, more people will connect to the Internet from mobile devices than from PCs
In 2015, 98 billion mobile applications will be downloaded
1.2 billion new smartphones enter the market in the next 5 years
In the Office
30% of companies have or develop applications for their
employees. 88% of employees use their
personal devices in their work.
Different type of mobile testing
Functional
• Is this application does what it is supposed to do?
Equipment
• How is the
application works on the device?
• Rendering
• CPU, Battery
Performance
• Is 1000 users can use this application simultaneously?
• What is the user experience when the server is loaded?
• What is the behavior of the application with different network conditions?
Functional testing Device testing Performance Testing
Agenda
Introduction
The challenges
Best practices
NeoLoad mobile capabilities
Mobile Performance Testing is Challenging for Enterprise Organizations
6
For 64%
Mobile Performance Testing is #1 Testing Priorityhttp://www.capgemini.com/thought-leadership/world-quality-report-2012-13
Don't have the right tools
65%
Don't have the devices they need
readily available
52%
Don't have the right processes or
methods
34%
But
Mobile Project are shorter with less budget
Back End services
• Common services for all devices
• Don’t need the mobile application to start testing
• Testing driven development
Mobile application
• Use the service calls to retrieve the data
• Last mile of the development life cycle
Validate the Performance of a Mobile Application is complex!
Back-End
EXECUTION PLATFORM
RIA/FRAMEWORKS
Database
Network
3G, 4G / LTE, Wifi
Video
HTTP Adaptive Streaming
Throughput Latency
Packet loss Server
Infrastructure
OS Client APP
Device
App
iPhone 4, 5, 6 Samsung Blackerry
…
(7,000 different device types are used to access Facebook)
Screen size CPU Memory
Native Mobile Mobile Browser Hybrid
Impact of the Network
Latency and bandwidth
The Most important network constraints are :
• Latency
• Bandwidth
From a certain level of bandwidth, increasing the
bandwidth will have a minor effect.
Packet size on a 3G Network
What is the average packet size on mobile networks ?
Traffic recorded on a 3G Network by an European Operator during 1 week in October 2010
0 50000 100000 150000 200000 250000 300000 350000
100b 1kb 10 kb 100 kb 1 Mb 10 Mb More
Number of packet
Packet LosT influEnce
Packet loss consequence
• Packet Retransmission
• End user experience degraded because Packet loss will increase jitter
• Errors could occur if Packet loss are in pure data transmission
Packet loss is inevitable in Mobile Network
• Mobility/Fast switch to next cell/temporary loose the signal for congestion of the current cell
Mobile protocols try to reduce
the packet loss rate
Wrap-up…
Different kind of application
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Mobile Context – Lots of Devices
Device Challenges
• Operating Systems - iOS
- Android - Windows
• Technologies used in the app - HTML 5
- Web Socket
- Streaming Video/audio - Google SPDY
• Screen Sizes
- Different content for different devices
- Design of what content is above/below the fold - When that content gets delivered
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Mobile Context – High, Distributed Load
Lots of users
• Mobile applications often have large user bases
• High growth in popularity can create the need for highly scalable load generation infrastructure
Geographical distributed load
• Users are accessing apps from locations around the globe
• Distance and location can greatly affect performance
Mix of Web & Mobile Users
• Many mobile apps have a corresponding web app with a shared backend
• High load on the web app can affect performance on mobile
www.neotys.com
© 2014 Neotys. All Rights Reserved.
CDN principles
www.neotys.com
© 2014 Neotys. All Rights Reserved.
CDN advantages example
www.neotys.com
© 2014 Neotys. All Rights Reserved.
How to test CDN-based architecture
2 kinds of CDN
• Caching CDN
• Streaming CDN
Tests must be realistic
• Spread the load among different geos and network
• Cache usage
• Streaming emulation
• CDN Backbone are hosted by Cloud providers: must choose appropriate Cloud provider(s)
• DNS resolution is mandatory in order to avoid stress a cdn node
Measure response times from different locations
• Cloud or multiple load generators are needed to stress several node of the CDN
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Agenda
Introduction
The challenges
Best practices
NeoLoad mobile capabilities
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Constructing Scenarios
Creating Real World Conditions
• Application under realistic load
• Real devices for capturing actual
• application performance on the device
• Real mobile CDN paths for testing the whole delivery chain
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Include Network Constraint during your test
Select most represented location and carries
• Measure the latency /packet Loss from the major locations
• Store your measurements into a network Matrix
Involve the Management to get statistics
• Ratio of user connected on Wifi/4G/3G/Edge
• Ratio or number of user per locations
Include the Network constraint in your load policy
• Organize your load policy depending on the network constraints
• Combine the usage of Cloud Based Load generator and Wan emulation
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Take profit of the Cloud Based Load Generator
Test the entire chain
•Firewall
•DNS
•ISP
•Network
Traffic generated from different geos
•Different networks
•Multiple Cloud providers
•Response time comparison
Save Time
•Deployment time is reduced
•Administration time is reduced
•No maintenance
•No Upgrades
Save Cost
•Pay only for what you use
•No machines acquisition
•No Maintenance
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Analytics to Enable Decision-making
Actionable Insight
• Monitoring on all infrastructure layers (App, Web, DB)
• Stats on backend and frontend to pin point root cause of performance problems
Shift Left
• Test integration with CI systems to get feedback to development teams earlier
• Fix problems earlier when they’re less expensive
Mobile Performance Testing Product
What are you really measuring?
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Real Device
Testing Tool Real Device Users
00:00:00 00:00:30 00:01:00 00:01:30
Load
NeoLoad
Application Server
App Error Rate
Virtual Users Load
Rendering Time
Complete Mobile EUE
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Report proper metrics
Pie I have not yet eaten
Pie I have eaten
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Agenda
Introduction
The challenges
Best practices
NeoLoad mobile capabilities
www.neotys.com
© 2014 Neotys. All Rights Reserved.
NeoLoad’s Mobile Capabilities
Record from any device
• Browser-based application
• Native application
Simulate any device or browser
• Ensures the right content is delivered for that device or browser
• Simulates parallel connections for each specific device and browser
Emulate realistic network conditions
• Bandwidth
• Latency
• Packet Loss
Generate load from the Cloud
• Geographic realism
• Rapid scalability for large tests
Load Command
H+ 3G 4G WiFi
Monitor APP
Network Emulation
& Device Simulation
Cloud Load Generators On-premise Load Generators
Database Server
App Server Web Server
www.neotys.com
© 2014 Neotys. All Rights Reserved.
NeoLoad + MobileCloud
Real Device Metrics Under Load
• Automatically trigger MobileCloud test with the start of a NeoLoad test
• See real device rendering times and vitals while the
backend of the application is under load
• Results are
correlated in real time for accurate and fast analysis
Load Command
H+ 3G 4G WiFi
Monitor APP
Cloud Load Generators On-premise Load Generators
Database Server
App Server Web Server
NeoLoad Controller
Real Devices &
Networks MobileCloud
Simulated Devices w/ Network
Emulation
Command Real Device Metrics
4G H+ 3G
www.neotys.com
© 2014 Neotys. All Rights Reserved.
Next Steps
NeoLoad PoC Assistance: Email sales@neotys.com
Visit Neotys Community at answers.neotys.com
Download NeoLoad Free Edition from www.neotys.com
Evaluation support at www.neotys.com/support/contact.html
Questions?
Visit Neotys booth