Cost effective methods of test environment management
Prabhu Meruga
Director - Solution Engineering 16th July
SCQAA – Irvine, CA
© 2013 CSS Corp
2
Agenda
Basic complexity
Dynamic needs for test environments
Traditional test environment challenges
Cost impact of not having dedicated test environments
Solution – Leverage Cloud based test environments
Who is using it ?
Who provides such infrastructure?
Types of cloud infrastructure
Is the cloud infrastructure secure
Cost comparison of On premise Vs Cloud based test environments
Steps to move into Cloud
What kind of testing can be done on Cloud?
Case Study
Illustrative shift of test environment landscape
Before and after cloud – what is the change in testing process?
Benefits Summary
3
Basic complexity !!
• Enterprise Information systems are increasingly becoming dynamic, complex, distributed and component-based.
• In-house test environments are proving to be
increasingly expensive and ineffective when handling these
complexities
• With consumer base accessing the end user applications in different browsers, devices and operating systems, certifying and testing the applications in similar production like
environments needs huge investments
How do I relate to this? Let’s review …
Conventional Application development scenario in enterprises
• Multiple applications in enterprise with parallel releases, various development life cycles Need for end to end testing of all the applications
• Interim releases, technology upgrades
2013
Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
Application 1 (Agile)
Application 2 (waterfall)
Release 1
Dev Test AT PT
Dev Test AT PT
Dev Test AT PT
Dev Test AT PT
Dev Test AT PT
Dev Test AT PT
Release 2
Dev Test AT PT
Dev Test AT PT
Dev Test AT PT
Dev Test AT PT
Dev Test AT PT
Dev Test AT PT Regre
ssion Regre
ssion
Release 1
Dev Test
AT PT
Regression
• Effort required to setup so many test environments
• Utilization levels of each test environment
• Maintenance of test environments
Automation Testing Performance Testing
5
Traditional test environment - Challenges
Test environments end up being different than production environments in terms of Operating Systems, Patch Level synchronization, Software Versions, Application Configuration, Multiple devices (in case of Mobile testing).
Challenges
• Lack of Test Environments management process
• Loosely coupled Asset control mechanisms
• Lack of change control mechanisms
• Increase demands of effort, process and cost
• Test environments are seen as expensive and providing little real business value.
Compromising steps
• Testing executed in shared environments
• Partially functioning test environments
Impact
• Most test servers run at less than 10 percent utilization.
• Approximately 30 % - 58%percent of defects are caused by wrongly
configured test environments.
• Testing backlog is often very long and the single largest factor in the delay of new application deployments.
• Production Issues
• Increased rework costs
Cost Impact due to lack of dedicated testing infrastructure
Statistics Source : National Institute of Standards and Technology; Infrastructure here refers to servers, storage and testing tools
Industry Cost of Inadequate testing infrastructure
Potential Cost reduction from infrastructure improvements Aerospace 237.4 Bn USD 54.5 Bn USD
Automotive 1.7 Bn USD 377 Bn USD Financial
Services
3.3 Bn USD 1.5 Bn USD
• On one side the cost of procuring hardware for enterprises is increasing while the other side it provides an opportunity for optimization.
• Of the overall hardware cost 40% is attributed to setup testing
infrastructure
7
Solution – Leverage cloud based test environments
Who is using it in the market?
Few real time applications working on cloud.
….and many more
Cloud-based testing environment – This refers to use a selected cloud infrastructure (or platform) as a base to form a test bed equipped with diverse and scalable computing resources, system infrastructures, and licensed tools, which are allocated using auto- provision based on static/dynamic requests. Both virtual and physical computing resources can be included and deployed inside.
How is the shift happening in the industry for cloud adoption?
Solution – Leverage cloud based test environment (IaaS)
9
Who provides the cloud infrastructure?
Public Private Hybrid
Definition When the services are rendered over a network that is open for public use
When the services and infrastructure are
maintained on a private network
Includes a variety of public and private options with multiple providers
Benefits Basic Average High
Risk High Medium Low
Cost Pay as you go Pay as you go Pay as you go
Security High (needs to be managed)
High High
Scalability Best Fair Good
Maintenance Easy Medium complexity High complexity
Effort required to setup
In few hours 2- 4 business days 5-7 business days
What can be setup
Typical Usage E-Commerce, Payment gateways, Intranet and Secured applications
Types of cloud infrastructure
11
Is infrastructure on cloud secure?
Secure Access
Built-in firewalls
Unique users VPN option
Hardware Security
Compliance
• Access Control and Identity management
• Ongoing compliance concerns
• Co-mingling of other customers data
• Security Standards & Certification
Security Challenges Security mitigation
Cost comparison – hypothetical analysis
On Premise infrastructure Cloud based infrastructure
13
Making the business case
Initial procurement cost IT Support staff cost Infrastructure maintenance cost
CO$T
Physical test Infrastructure Cloud based test infrastructure
Steps to move into cloud based test infra
Step 1
• Kick off pilot cloud involving non critical pieces – staging/IT test environments
• Start with shifting of storage servers
• Move low critical applications from IT
Step 2
• Focus on moving the entire test infrastructure
• Pilot with few mission critical applications
• Setup a Virtual private cloud and assess the security
Step 3
• Evolve into running the infrastructure as a service
• Focus on service and cost part of it
15
Where can cloud based test infrastructure help?
Functional testing Regular Manual Testing Regression Testing Web Services Testing Product Testing
Cross browser testing
Non Functional testing Load Testing Stress Testing Endurance Testing Vulnerability Testing Penetration Testing
Functional Automation testing Custom framework setup Industry standard tools Automated regression testing
Mobile Applications testing
Case study : Cloud based performance testing
Cloud based Framework for High Load Performance Testing, Functional Testing, Test Management,
Vulnerability Scanning & App monitoring
17
Eg: Leverage cloud based infrastructure for load generation in performance testing
Load Generation over the cloud
Illustrative test environment landscape shift
Coding Testing UAT
Development Environment
Unit Testing Environment
System Testing Environment
Automation Environment
Performance Environment
UAT Environment
Prod
Prod Environment
Coding Testing UAT
Development Environment
Unit Testing Environment
System Testing Environment
Automation Environment
Performance Environment
UAT Environment
Prod
Prod Environment
Cloud Management Console – leverage cloud infrastructure for test environments
Shift to Cloud based test Infrastructure
Tightly coupled environment instances
19
Before and after cloud environment. Let’s talk
Environment Setup – 1 time activity. Typical duration (few days to a week based on the complexity)
• Select the appropriate partner for infrastructure setup. (Amazon, Google or Microsoft)
• Wrap up and apply the security policies as applicable for your Organization
• Cloud Management console logins for restricted user access (development, QA and Configuration teams)
Deployment process – Iterative !! As it was earlier
• Plan the deployments when needed
• No restriction on the number of deployments Usability – the way environments are used
• Remains same – u will find no change in the way environment is used
• You can use all the commands that you use in the physical environment
• You can deploy all the licenses for your custom package software Utilization and Cost – Pay as you go cost model.
• Cost effective especially in the case of performance testing where the execution is done when needed.
• Multiple load generators can be setup run time as needed
• No worries about re setup as the same is available with a minimal cost
• Environment can be rebuilt very swiftly since the components are known
• Share or shift the environments very easily with other priority projects within the enterprise
Scalable and on-demand cloud
environments
Accelerated testing cycles
• Create multiple test environment instances on-demand
• Create isolated test environments for a broad range of OS, database, browser, and application combinations
• Run them concurrently to create parallel work streams
• Capture the entire state (memory, network settings, and disk) of a multi-machine configuration and save it as a template
• Perform destructive tests, install intrusive tools, and get back to a "clean slate" easily
Enterprise IT collaboration
Enable collaboration for multiple product and release teams
• Create product and release specific projects and assign groups of users and resources
• Enable easy collaboration including secure, limited access for contract or off-shore test teams
• Share bug snapshots with remote development and testing teams
Advanced networks
Recreate customer use cases with virtual data centers
• Create custom networks, routing and security policies
• Replicate customer use cases including clustering and fail-over configurations, N-tier secure configurations, and shared resource configurations in the cloud
Total Cost of ownership
• Pay-for-what-you-use’ model
• Eliminates vendor commitment overheads
• Accelerates 'go to market‘ by minimizing overall lead time in infrastructure provisioning
• Shifts spending from Capex to Opex
• Ensures security and standards compliance
• Provisions hardware only for the required duration
Benefits summary
©2013 CSS Corp
The information contained herein is subject to change without notice. All other trademarks mentioned herein are the property of their respective owners.
Thank You!
Appendix
© 2013 CSS Corp 22
23
Evolution of cloud based infrastructure maturity
Define the objective – End to End testing, functional automation, Performance testing, Mobile testing Identify the Infrastructure requirements – Storage Capacity, frequency of use, Service providers
Security Policies : Identify the security, regulation challenges with the internal IT risk and security team
Dedicated resources : Assign dedicated environment personnel
Pilot implementation : Kick off test execution and collect the metrics
Assessment : Review the ease of deployments,
maintenance and security of applications on the cloud infrastructure
Standardize: roll out standard environment process for synchronization, new setup, sharing and deployment of applications
Continuous Improvement: Review the maturity and expand the infrastructure for development and production needs as well
DefineTransformManage
24
25
26
References :
• IBM
• Amazon
• Microsoft
• Bluemountainlabs
• National Institute of standards and technology (USA)
© 2013 CSS Corp 27
