Mobile App Development: The CD Recipe
— Jenkins + Functional and
Non-functional Testing + Real Devices
Introducing Perfecto Mobile
Getting Noticed Good Friends At a glanceEnabling manual and automated mobile application testing as well
as application monitoring for increasingly agile delivery processes
“Perfecto Mobile marks shift in mobile
apps QA with Continuous Quality Lab”
• Founded in late 2006, Business HQ in Boston and R&D in Israel • Proven management team with over 200 employees globally • Funded by leading VCs, Total - $57.8M
Overview
• Mobile changes test planning
• Describe CI challenges with
mobile
• Describe a working pattern
• Share some examples
Objectives
• Share best practices for
Time to Market
Innovation
Time to Market
Innovation
RaboBank Digital Portfolio
Current online presence
- Website for desktop and for mobile
- Native app for iOS, Android and Windows Phone
Development
- 1 new hybrid environment that will replace all native apps and the website for desktop created by 10 scrum teams
- Total of 18 scrum teams, 200 FTE
- Core of a team: Development, PO, QA, Operations - Extended team: BA, Appl Eng, project mngt
Platform #users # login / month
Desktop 3,6M 18,6M
App iOS 1,2M 30M
App Android 1,2M 30M App Windows Phone 0,02M 0,3M
Automation Strategy
Technology facing
Business facing
C
ri
ti
cal
pr
ojects
Suppo
rt pr
ogr
am
mi
ng
Functional tests(system (integration) & chain, 200 portability testing – 10% x 10 devices)
Exploratory testing User acceptance testing
Accessibility Visuals (partially)
Unit testing
Unit integration testing
Non-functional testing
(performance, security, production acceptance testing)
Regression testing is cross-category
Automated
Automated
Manual
Easily Applied for Browsers …
Mobile Create Challenges …
UI
Integration
Unit Testing
90 Minute Commute
• Check Facebook
• Skip that OS update
• Watch YouTube videos
• Clear easy emails
• Start expense reports
• Answer the phone
• Finish expense reports
• Order a gift on online
• Send several text
messages
UI
Integration
Unit Testing
Performance Security
Mobile Requires Shifting the Plan …
Target
Devices
Target
Env.
OS
OS versions Form factors CPU & Memory Manufacturers Wearables IOT Networks Orientation Location Notifications Events Device events
Our Working Cookbook
Shared Lab
Align QA tools to Dev
Continuous Integration
CI Blockers:
Adding Real Devices
Wha
t
• Struggling to add real devices to the CI process
Wh
y
• Uncertain device availability • Test readiness and configuration control of real devicesHow
• Managed lab • Devices SLAA Lab that provides…
1. SLA assured devices availability
2. Parallel execution capability
Best Practice: Select Devices By Attribute
Select targets
by attribute
CI Blockers:
Inefficient Test Development
Wha
t
• Developing UI automation takes too long • Automation requires constant maintenanceWh
y
• Scripting per platform • Skillset - shifting from scripting to codingHo
w
• Easy object identification • Develop framework • Reuse predefined code functions • Write crossBest Practice:
Collaborate for Test Friendly Code
Catalog objects
Object Properties
Inherit Properties
Best Practice: Cross Platform Test Code
Cross platform
execution
CI Blockers:
Excessive Execution Failures
Wha
t
• > 25% execution
failures and errors
Wh
y
• Unexpected notifications • Environment readiness • Device errorsHow
• Separate test code & execution mgt • Perform pre-testreadiness checks • Add Error /Event
Best Practice:
Graceful Test Recovery
System level control Application control Device control
Manage Mobile
Events to
Reduce False
Negatives
CI Blockers:
Poor Execution Performance
Wha
t
• UI Automated test execution exceeds available test windowWh
y
• All the tests wait for the device(resources issue ) • Lack of execution management • Hardcoded “Waits”
Ho
w
• Apply automation best practices • Parallel execution • Use sync mechanismBest Practice:
Parallel Execution
NativeDriver.manage().timeouts().implicitlyWait(15, TimeUnit.SECONDS);
NativeDriver.manage().timeouts().pageLoadTimeout(15, TimeUnit.SECONDS);
Implement
dynamic waits &
Parallel Execution
Manage the Time Budget …
Build Frequency Test Duration Test Scope Device CoverageUnit System Smoke Regression Performance
Hourly 20-40 Min ±4
3-4 times a
day 30-60 Min ±6
Nightly 2-7 Hours ±10
Recommendation's
• Automation requires an available lab
• Maintain test code guidelines
• Error & event handling
• Framework maintenance
• Readiness checks
• Externalize Resource mgt
• Optimize execution
Reporting Wind Tunnel
MobileCloud Platform Testing and Monitoring Srvcs
The Continuous Quality Lab
Continuous Feedback and Reporting
Functional Testing Integration Testing Performance Testing
Parallel Test Execution
Automated Deployment on Real Devices
Continuous Quality Lab for a full App Delivery Lifecycle
Dev Team
Source Control
CI Server
Launch Perfecto Job (Release Package) Bug Tracking
Slide 27 Test Automation Engineering Group ©Paychex 2010 All Rights Reserved
Nightly CI builds
Perfecto Connectivity – Daily / 4:45 AM
Web Service Connectivity – Daily / hourly & Post Deploys BATs (IOS & Android) – Daily / 5:30 AM / across 4 VMs/cradles
Majority of Business Logic is within the Web Services
Web Service Regression – Daily / 4:30 AM & On-Demand
Jenkins App Builds (5 AM) copied to Perfecto Cloud / Post Builds
Test Continuously
Continuous Quality
(to-be) Build Unit Test Check-out & Build application Security/Perf. code analysis Quality code analysis Check-inCheck-in code & review Deploy to TST Deploy to TST Smoke test on TST Regression Test System Test Deploy to ACC Deploy to ACC Smoke test on ACC Acceptance Chain Test Performance Test PAT UAT Portability Test Pre-PAT Security Production Schedule P-deployment Deploy to PROD PROD verification Automated step Manual step Code review Exploratory test Done