Software Testing as a Service STaaS

1 | P a g e

Software Testing as a Service – STaaS

December 06 - 07, 2012, Bangalore

Authors:

Smriti Verma

[email protected]

Noida, UP-201301, India

Rahul Kumar

[email protected]

Element Infomatics (India) Private Limited #439/30, 3rd Cross,

Wilson Garden Bangalore – 560027, India

2 | P a g e

Abstract:

The World is a global village, and money speaks the universal language. In the womb of today’s converging technological era, lies in the conceptualization of Software testing as a Service.

Testing as-a-service has become a challenging activity for enterprises due to its ever increasing techno – functional complexities, software sourcing challenges, rising cost and security issues. For all the pain areas and impertinent issues faced before delivering an exquisite product release, STaaS shall be the way to choose in future.

With this paper we intend to enlighten the latest trends in the field of software testing which would help us to Innovatively Adopt the technological Convergence which precisely is the theme for this year.

We take this opportunity to showcase the impact of STaaS on the changes in Software Business Model also how its adoption factors like Availability & Reliability, Fast Deployment, Reduced dependencies on internal IT infrastructure, Flexible consumption & Simplified Pricing model catalyzes the production for stakeholders.

What is STaaS?

Software testing as a Service (STaaS) is an outsourcing model, in which testing activities are outsourced to a third party which specializes in simulating real world testing environments as per client requirements. In other words, STaaS is a process in which companies ask service providers to provide software testing services as and when required.

STaaS involves outsourcing testing activities to service providers, by providing all relevant data and test documents, so that service providers can simulate real world testing environment and deliver detailed test results on time.

Services that are well-suited for the STaaS model include automated regression testing, performance testing, security testing, testing of major ERP software, and monitoring/testing of cloud-based applications.

STaaS solutions are particularly useful, when companies need to automate test activities using expensive tool sets. Instead of buying these tools, STaaS allows companies to pay as they use, hence saving costs and achieving optimum results.

3 | P a g e

STaaS “As a Service” solutions

Most organizations today like automating testing activities to get the best results in the least possible time. But the cost of automation is really expensive, which is beyond the budget of many organizations. Hence, due to the inability to access automation, testing activities are hampered and everything needs to be manual. This leads to lot of time consumption and results are prone to risks.

But STaaS has a solution for everything!!

STaaS allows organizations to buy what they need and when they need. This means, organizations can use automation tools and skilled people on demand, as and when needed. Full testing services will be provided by STaaS providers to the demanding organizations. Hence, no matter how small or large an organization is, it can afford STaaS due to its simple and flexible approach.

STaaS is becoming popular now-a-days, as organizations get the best of service without huge investments.

4 | P a g e

When to use STaaS?

It is critical for organizations to decide when they really want to use STaaS, to get the best possible results. STaaS is useful for:-

1. Performing specialized tasks which require a high degree of automation and quick results 2. Performing tasks which require people with automation skills

3. Performing testing efforts that don’t require in-depth knowledge of the design or the system

To be in the market, organizations have to compete, produce the best of products and deliver out of the box services. Market pressure is so high that no organization can compromise on the quality of products/services. To deliver out of the class service in limited budget and time constraints, STaaS is the answer.

Types of services that are well-suited for the STaaS model include functional, performance and security testing.

STaaS model follows all phases of Software Testing Life Cycle, confirming to international standards.

5 | P a g e

Types of STaaS

Figure 3: STaaS Types

STaaS not only ensures that the application under test is working properly; it also ensures that the application runs on various platforms and devices. STaaS is so popular today because of its availability in different versions across the industry.

Software Testing-as-a-Service includes:-

1. Functional Testing-as-a-Service

STaaS providers enable functional testing services for clients which include, but not limited to, UI/GUI testing, integration, regression and automated user acceptance testing (UAT). It brings down the cost of regression testing, increases test coverage and gives assured results. 2. Performance Testing-as-a-Service

Performance testing is very crucial for an organization where multiple users are accessing the application at the same time. STaaS simulates real world users’ environment by creating virtual users, thereby testing to check if the system can withstand the load and stress. It is a “pay-as-you-go” method for such tests.

3. Security Testing-as-a-Service

Security testing is very crucial for any web based application. With an increasing number of websites and applications obtaining critical information such as credit card or account number, the security for such information is of utmost importance. STaaS scans the websites and the applications for any vulnerability.

6 | P a g e

STaaS Providers speak out

1. Bogdan Chernyachuk

Project Manager & Resource Manager at EPAM Systems, Ukraine

My team (40+ QA Engineers) executes functional, integration and automation testing services for a large customer. We are not developing products, only test them. We test various applications built by various development teams (which are teams on customer side or other outsourcing vendors) and testers need to adjust their work with the methodologies and processes implemented in each of that teams (so it varies from true scrum approach to some kind of waterfall). We have projects with Test Cases such complex that passing one integration test case through all front-to-back systems may take 1 day. We have projects with large number of UI related regression test cases where tester can pass 100+ in one day.

Having all this difference, it’s a documented way to define "Software Testing as a Service" operation model and to develop metrics that can measure team success and improvement in a long term run.

2. Ethel Evans

SDET at SEOmoz, Greater Seattle Area

As a service, testing product is information about the development product. What you might be able to measure, then, is timeliness and quality of information about the product. With so many different project types, you might not be able to compare between projects or testers easily, but maybe you can get some trends over time. Here are some thoughts about specifics, drawn largely from metrics I've seen in the past when working with a "service" mentality:

•How fast can you get testing results back to the developers after a build? This could include multiple times, such as the time to initial results (pass / fail), first-pass analysis (test issue, code issue, needs analysis, etc.), and final analysis (which could include more thorough bug reduction). You might also want to consider if you are getting the most interesting (the riskiest) results out sooner.

•How thoroughly are failures investigated? Are developers just given a stack trace, logs,

and test case number? Are testers able to reproduce the bug and reduce it to the simplest possible set of steps that generates a consistent reproduction of the issue? Note that there are trade-offs here, and some customers might prefer to pay you less and investigate their own issues as soon as they are detected - but long-term, bug-reducing and debugging skills are great for testers to build.

7 | P a g e

•How many bugs are detected by your testing on the first pass, and how many are found later through other means? How many are found by the testers, and how many by developers or - worse - users? Automatically detected bugs are great signs. Bugs found in exploratory testing are good signs if they should have been detected automatically but were still caught in-house on the first build where the bug existed; and great signs if they are "weird" bugs. Bugs found outside the team or several builds later should be reduced as much as possible.

•How many of your automated tests are running at any given time, and how many are broken, blocked, or otherwise unusable? When tests are broken for reasons other than being blocked by bugs, how long does it take to fix them? This can help identify brittle tests or infrastructure issues that are technical debt, so you can fix them and quit 'paying interest' on them.

•How accurate are your time estimates? How frequently does testing run long and why

do this happen? How can you recognize risk factors for extended testing cycles and point them out to the client sooner?

•How easy is it for the client to get information to and from testers? Can they gather information 'on demand' from a web portal, or do they need to wait for a report to be emailed to them? How do your clients feel about the transparency of the testing process?

STaaS Benefits

Software Testing-as-a-Service or STaaS is used in all spheres of the industry. With the advent of testing an application on not just computers, it’s increasingly important to test on laptops, notebook PCs, smart phones, tablets, dual monitors etc. with different versions of the OS and different types and versions of the browsers. All this requires a huge expenditure and round the clock support.

STaaS provides all the required testing support on 'n' number of devices and platforms. You name it and STaaS has it.

8 | P a g e

Core benefits of STaaS include:- 1. In depth coverage

STaaS allows in depth coverage of the application under test, ensuring a good amount of code and statement coverage.

2. Quick and timely results

Standardized testing processes and the availability of automation tools and skilled resources enhances quality and reduces the time to market.

3. Flexibility

STaaS allow testers to conduct performance tests at a higher load by creating a virtual

environment. STaaS also verifies the compliance of applications on various platforms and devices.

4. Low cost and trusted result

STaaS provides great cost reduction to the customers, as customers do not have to build the environment themselves. STaaS provides trusted results by following all phases of STLC.

Software Testing as a Service over Cloud

Cloud Testing is a form of software testing in which web applications use cloud computing environment to simulate real-world user traffic.

Cloud Testing offers effective unlimited storage, quick availability of the infrastructure with scalability, flexibility and availability of distributed testing environment reduce the execution time of testing of large applications and lead to cost-effective solutions.

Figure 5: STaaS over Cloud (Courtesy: Wikipedia)

Companies simulate real world Web users by using cloud testing services that are provided by cloud service vendors such as Advaltis, SOASTA, HP, Load Impact, Compuware and Keynote systems. Once user scenarios are developed and the test is designed, these service providers leverage cloud servers (provided by cloud platform vendors such as Amazon.com, Google, Rackspace, etc.) to generate web traffic that originates from around the world. Once the test is complete, the cloud service providers deliver results and analysis back to corporate IT professionals through real-time dashboards for a complete analysis of how their applications and the internet will perform during peak volumes.

9 | P a g e

Case Study 1:-

A dedicated offshore Software testing service center provided end – to –end testing service to client realizing a cost reduction over 35% along with faster product launch and reduction in testing cycle time.

•Client Situation – One of the world’s largest financial groups delivering service to over 60 countries across Europe, America, Africa and Asia.

•The Challenge – For expansive growth in India and other emerging markets was facing excessive overhead and operation costs in its product rollout for overall business plan. Lack of location specific domain knowledge and poor quality of product would delay the launch of retail operations on multiple occasions.

•The Solution – With incorporation of STaaS there was established a dedicated testing center which offered a wide spectrum of key services encompassing –

•Test Consulting •System Testing •Integration Testing •Performance Testing •Functional Testing •Regression Testing •UAT •Test Automation •Security Testing

•Test Environment management •The Benefits -

•Faster resource ramp- ups through investment in training and knowledge management of niche skills.

•33% improvement in time-to-market and faster product launch / roll outs through leveraging reuse & business process testing solutions , leveraging product knowledge & competencies and generic regression test packs.

•35% reduction in total testing cost through leveraging offshore team, extensive test automation and extensive reuse through libraries.

10 | P a g e

Case Study 2:- An Agile approach

The implementation of managed software testing as service resulted in a 60% decrease in cost, spanning over a short period of two years. The planned approach to workflow was effective enough to have a positive effect on throughput of the projects.

•Client Situation - In the autumn of 2007, a global telecom provider based out of UK hot listed a number of challenges being faced with the management and supply of software testing services. The major objectives were outlined as:

•Overall cost reduction of the services

•Utilization of a cheaper offshore resource where appropriate •Reducing the cost of managing the service

•Increasing the stability of resource base

Major concern of client in performing this end-to-end testing was retention of the control of the testing function and key knowledge within the domain, as loss of these would have eroded the competitive edge that the client held over its other competitors based on products and their performance.

•The Challenge – A crest – trough pattern of testing demand resulted out of client’s dynamic and fast –paced environment. Ad hoc testing that was utilized as a reaction to the crest and troughs of demand resulted in the client dealing with a lack of accountability from vendors as testing was not being planned holistically.

•The Solution – STaaS carved out a solution to the scenario and the proposed resolution included :

•Depending on the amount of planned project over a rolling period – workflow requirements , tester infrastructure and costs were calculated

•A year - on - year cost reduction competitive pricing model was devised.

•Wide range of KPIs (Key Performance Indicators) ensured that both the client and solutions provider had agreed measures for success.

•The Benefits

•Cost reduction in services management as account/resource management was an integral part of service which was delivered to the client at no cost.

•Long term, localized staff commitment ensured the testing was continuous.

•Efficiency of projects increased with appropriate mix of onshore, offshore and automation within the testing services undertaken.

11 | P a g e

Figure 6: A graphical representation of evolution of the service (X – Time, Y - Staff)

Summary

Software testing as a Service is a model which allows organizations to pay for what they need, when they need it. By using a flexible approach, STaaS allows organizations to get the benefits of automation testing without the risk of huge investments.

STaaS is used all across the industry in almost all spheres of testing activities, however to get the best use of STaaS, organizations need to make intelligent decisions like when to use STaaS to avail all possible benefits. Organizations need to monitor STaaS activities at regular intervals to ensure expected results.

Organizations need to work on the bugs encountered by STaaS activities and get them fixed on time, so that STaaS work flow is not hampered. Bug logging and bug fixing is an ongoing interaction between STaaS providers and organizations, to develop a highly bug-free product in minimum possible time. The sooner a product reaches the market, the more it will succeed.

With all the known benefits of STaaS, it’s a great way to lessen the burden on organizations and help them concentrate on developing world class applications. But at the same time, organizations should be smart enough to make the best use of STaaS.

Hence, STaaS solutions may be the answer to give your organization the state-of-the-art testing solution without the high price tag.

12 | P a g e

References

1. Testing-as-a-Service: Outsourcing your specialized software testing Yvette Francino

2. Stack Exchange: Software Quality Assurance and Testing 3. IEEE Explore Digital Library: Testing as a Service over Cloud

4. Silicon India QA City: Testing-as-a-Service: Its Types and Core Benefits 5. Euro star Software Testing Conference by Edwin van Loon, Valid, Belgium 6. Case study reference - www.tcl-global.com & L n T InfoTech , India

Biography

Smriti Verma is Software Quality Engineer at Hexaview Technologies Pvt. Ltd. She has an experience of more than two and a half years in the Software Industry working on various domains and testing methodologies. Currently, Smriti is working for a client which is one of the top software companies of the world, and developing one of their next gen products scheduled to come out next year.

Prior to this, she has worked with Soft Tech Group, Gurgaon as a Database Officer.

Smriti holds an Engineering Degree in Computer Science from Apeejay College of Engineering. In free time, she likes listening to music, playing badminton and traveling.

Rahul Kumar

Rahul Kumar is Quality Analyst at Element Informatics Pvt. Ltd. He is a passionate software test professional with a determination to carve a niche in the industry by being innovative and flexible in approach. He has an experience of close to two years in the Software testing domain, involved in the unearthing of quality related issues.

Prior to this, he has worked as a Quality Assurance Engineer with Soft Tech Group, Gurgaon. Rahul holds an Engineering Degree in Computer Science from Apeejay College of Engineering. He was consistently participating and winning in various oratory events held at inter-college level.