Cloud testing and support to the software lifecycle







Full text



di Milano

Cloud testing and support to the software lifecycle


The software lifecycle and the cloud


PaaS/IaaS approaches seen so far mostly dedicated

to offer run-time features


Google App Engine and MS Azure also offer

programming features




Programming models (e.g., web role/worker role in

Azure, parallelism managed through task queues in



Coverage of other aspects of the software lifecycle


Cloud testing and testing Cloud applications




Software testing


Software testing is the process of assessing, through

execution, the functional and non-functional

characteristics of a program


For large systems:


High number of test cases (thousands)


Need to execute the same tests several times during

the day


Need to handle testing for multiple configurations (e.g.,

multiple browsers, various operating systems, various

versions/types of databases,



Software testing and the Cloud

Cloud characteristics

Testing problems

Possibility to have a large

number of parallel executions


Large test suites

Possibility to exploit resources

(and pay for them) only when



Execute tests along the day

Possibility to build virtual

machines each one with a

different configuration


Handle different



Cloud testing


Cloud testing means using the Cloud to support the

testing phase


Risks of Cloud testing


Testing artifacts need to be migrated to the cloud

even though still kept in synch with other

development activities


Artifacts to be migrated


Test cases/test suites




Software to be tested


Testing tools


Specific Cloud issues, e.g., security, should be taken

into account


Test code, libraries, dependencies may not be


Characteristics of applications that can

be tested in the Cloud


Test case independence


If test cases are temporally dependent from each other,

they cannot be run in parallel


Standard operating environment


Otherwise incompatibilities may arise


Availability of programmatic interfaces


GUI-based testing could be less convenient on the


Types of testing suitable for the Cloud


Unit testing


Easy to automatize


A killer application of Cloud testing


High Volume Automated Test


Looks for failures that happen only after a prolonged test

(e.g., deadlock, race conditions,



Resource intensive


It can rely on the large number of resources offered by the



Performance testing


Used to determine application throughput, resolving

concurrency issues, tuning server response time, assessing



In a Cloud environment it is possible to emulate load on



What do we install on the Cloud?


The entire system to be tested, together with all it is

needed for performing testing


This is what we have

assumed so far


We use the Cloud for hosting/emulating part of the


Examples of testing tool: D-Cloud


Support execution of tests with fault injection

machine using the fault injection facility implemented in the virtual machine layer.

• The computing resources can be managed flexibly. If

resources are available, then test cases can be executed quickly by simultaneously using the resources.

• D-Cloud automates testing using descriptions of the

sys-tem configuration and the test scenario to execute tests on cloud computing systems.

A. Fault injection in a virtual machine

D-Cloud uses a virtual machine to execute system tests. The virtualized hardware device can simulate failures on the guest OS, and fault injection using virtual machines allows system tests to be executed without changing the program. Using the virtual machine, D-Cloud can test software running in not only the userland layer but also in the kernel layer. When software bugs on the kernel layer are detected during a system test, the OS may hang-up automatically due to a kernel panic. When the system runs on real machine, it is difficult to obtain helpful information for the bug fix in this case, because the user cannot manipulate the OS under the kernel panic. However, when using a virtual machine, a bug in the OS running on the virtual machine does not affect the host OS running on a physical machine. Therefore, the tester can continue system tests, and the tester can collect information for debugging even if the guest OS crashes. Furthermore, the snapshot of the previous state in the guest OS permits the operation to return until the desired state repeatedly.

B. Management of computing resources

For developing reliable systems, it is important that system tests must be executed for as many cases as possible in order to find and fix as many bugs as possible. In addition, in order to execute many tests, a large amount of resources must be managed efficiently and flexibly. In D-Cloud, resources are managed by a cloud computing system. For example, a number of systems that require high reliability and dependability consist of multiple nodes linked by a network. In this case, D-Cloud can test such distributed systems using several guest OSes.

C. Automating system configuration and testing

D-Cloud automates the system setup process of the tested system and the test process, including the fault injection, based on a scenario written by a tester. When the tester writes a number of configurations of system test environments in a scenario description file, D-Cloud sets up appropriate test environments and executes appropriate tests automatically. Therefore, D-Cloud enables the behavior of dependability functions on the system to be tested exhaustively and enables system tests to be executed quickly.


D-Cloud uses QEMU as virtualization software and Euca-lyptus, which is an open-source implementation having the same API as AmazonEC2[3]. Figure 1 shows an overview of D-Cloud. D-Cloud consists of the following components:

Fig. 1. Configuration of D-Cloud

1) QEMU nodes, which are available with the fault injec-tion facility.

2) The controller node, which manages the QEMU nodes using Eucalyptus.

3) D-Cloud frontend, which issues test and fault injection commands and transfers input/output data with QEMU nodes.

A. Fault injection using QEMU

D-Cloud uses QEMU as a virtualization software. The advantages of using QEMU are described following.

• QEMU can emulate a number of hardware devices. Thus,

QEMU may control several hardware faults in the guest OS.

• The QEMU source codes are available as open-source

software. This allows modification to the hardware emu-lation codes in order to add the fault injection function.

• QEMU can isolate the host OS from the guest OS. In

addition, QEMU may insulate the computing host from anomalous behavior of the guest OS during various tests. 1) Type of fault injection: Table I lists the types of fault injection . Here, we implement the fault injection mechanism for the hard disk controller, the network controller, and the memory as target devices.

2) Reproducibility in fault injection: Reproducibility is important in fault tolerant testing using fault injection. If a testing environment has reproducibility, then the tester can find errors that occur under specific conditions and can fix such bugs. In particular, in many cases of hardware device errors, we cannot predict the effects of bugs, and these effects are usually not repeatable. However, since hardware errors can be emulated anytime and anywhere using a virtual machine, the costs associated with using a virtual machine are lower than those associated with using physical machines.




device contents value Hard disk Error of specified sector badblock

Specified sector is read-only readonly Error detection by ECC ecc Received data contains error corrupt Response of disk becomes slow slow Network 1bit error of packet 1bit 2bit error of packet 2bit Error detection by CRC crc

Packet loss loss NIC is not responding nic Memory Bit error bit Byte at specified address contains error byte

B. Managing resources using Eucalyptus

D-Cloud manages virtual machine resources using Euca-lyptus. Eucalyptus is a cloud computing infrastructure that manages machine resources flexibly using a virtual machine. D-Cloud consists of multiple QEMU nodes, which execute guest OSes, and a controller node, which controls all of the guest OSes.

The procedures to manage machine resources are shown as follow.

1) A tester uploads OS images to the controller node and registers the machine images to the D-Cloud frontend. 2) The controller node transfers OS images to QEMU


3) The OS images are booted as a guest OS on QEMU nodes.

After the controller node receives the request to boot a guest OS, the controller node transfers the OS images to QEMU nodes, which are available to run the OS images. Thus, the tester does not need to be aware of computing resources on D-Cloud.

C. D-Cloud frontend

D-Cloud frontend manages guest OSes, configures system test environments, and transfers various data from the tester to a guest OS executed for the purpose of system testing.

D-Cloud frontend performs its function as follows:

1) D-Cloud frontend receives a test scenario, a test pro-gram, input data, and an execution script from a tester. 2) D-Cloud frontend then issues a request to boot a guest

OS to the controller node.

3) D-Cloud frontend then transfers the test program, the input data, and the execution script to the guest OS. 4) D-Cloud frontend then issues the fault injection

com-mand to the target guest OS.

5) Finally, D-Cloud frontend collects the output data, logs, and snapshots.

Since D-Cloud frontend collects data obtained in the test, the tester can download these data anytime. If the tester checks the output and traces the operation using saved snapshots, the tester can discover some bugs and investigate these bugs in detail.

Fig. 2. Example XML scenario


D-Cloud executes system tests according to a scenario written in XML. By describing the scenario, the tester si-multaneously executes various system tests. Figure 2 shows a complete example of a scenario statement. The scenario statement consists of four parts defining the test as follows:

• Description for hardware environment • Description for software environment • Definition of faults for injection

• Procedure of the entire test

A. Configuration for the hardware environment

The definition of the hardware environment is given in the “machineDefinition” element. Table II lists the contents of the “machineDefinition” element. Next, the definition of the hardware environment is described in the following. The “machineDefinition” element can hold multiple “machine” el-ements. The “machine” element includes five elements (name, cpu, mem, nic, and id), and these elements are needed for each definition of a hardware environment. The “name” element


Using the Cloud for emulating part of the



Emulation of large traffic in stress and performance



Application-Behavior Virtualization to replace


SOASTA CloudTest


Use the Cloud to inject large traffic hitting the

application under test



can happen

also in


Application-Behavior Virtualization


Focuses on enabling testing when the software under

test includes


Missing/unstable components


Non fully accessible third-party/partner systems and



Systems that are too complex for test labs


Enables testers to rapidly isolate and virtualize just

the behavior of the specific dependent components

that they need to exercise in order to complete their

end-to-end transactions.


Application-Behavior Virtualization




Identify the components to virtualize


Capture the behavior of the associated transactions,

messages, services in

virtual assets


Configure the virtual assets parameterizing conditional

behavior, performance criteria, test data


Use the virtual assets to emulate the actual behavior of


Test environments in the Cloud (1)

24 © 2011 Computer Sciences Corporation

Test Environment in Cloud

Attributes of Cloud


Characteristics Benefits

Advanced virtualization Test resources (infra, tools and

people) are pooled and virtualized

Providing efficient

implementation of independent infrastructure

Rapid Provisioning Test resources are provisioned

on demand

Reducing test setup and

execution time and eliminating errors

Service Catalog ordering Test environment are readily


Enabling visibility, control and automation

Elastic scaling Test environment can be

scaled up or down by large factor as the need emerges

Optimizes, infra and software license usage

Flexible pricing Test resources are priced on

supported topology and project phases

Offering pricing options tailored to user resource need

Metering and billing Test resources used in

reserved are charged back to LOBs


Test environments in the Cloud (2)

25 © 2011 Computer Sciences Corporation

Test Environment in Cloud (Contd..)

Attributes of cloud solutions

Characteristics Benefits

Re-platform ability (Compatibility Testing)

Platform based configurations (for testing) can be created and managed

Comprehensive assurance on product / service

Maintenance of multiple test beds based for multiple release testing

Release based configurations (for testing) can be created and managed

Complete assurance on maintenance of product / service

Test factories and TCoE Setups Emerging alternative models for supporting client testing needs

Collaborative platform enables collective productivity

Service virtualization In a multi component

architecture, availability of a dependant component(s) managed for testing

Effective completion of component level testing, despite the dependencies on Critical components

ALM (Application Lifecycle Management) / TLM (Test Lifecycle Management) support

In a multi component

architecture, availability of a dependant component(s) managed

Effective completion of component level testing, despite the dependencies on Critical components


Test environments in the Cloud (3)

26 © 2011 Computer Sciences Corporation


Present and future (Test Environment)

Capability Past Present and future

Server / Storage utilization 10-20% 70-90%

Cost Reduction Nil 20-30%

Self Service None Unlimited

Test Provisioning Weeks Minutes

Change Management Months Days / Hours

Release Management Weeks Minutes

Metering / Billing Fixed cost model Granular

Re-platform ability (Compatibility Testing) Prohibitively expensive

Engineering possibility with affordable cost

Maintenance of multiple test best beds for multiple release testing

Prohibitively expensive

Engineering possibility with affordable cost

Multiple Tools testing (Application Security –for false positives –false negatives triangulations)

Prohibitively expensive

Affordable because of utility pricing and improved coverage New Alternate for BCP (Business Continuity

Plan) and DR (Disaster Recovery)

Not Applicable A Distinct possibility


Testing as a Service (TaaS)


Potential benefits


Tool license and infrastructure costs on a pay-per-use



Flexibility and wide range


No setup and procurement time wastage


No need of hiring expert people


The landscape of current offering [CSC]

30 © 2011 Computer Sciences Corporation

Cloud Based Testing Service Offerings under TaaS Ecosystem


Windows (Multilingual, Multi

CPUs) Linux (Red hat, Linux, Debian)

MS-SQL. Oracle, DB2,

Informix, MySQL, Teradata Sun Solaris

Elastic servers (Tomcat, My

SQL combination s)

Borland ES, Cold fusion, Tomcat, Web logic, Web Sphere, Web Application server

(SAP), ATG Dynamo, JBOSS

Tools Sandbox


& Services Client / Server

Enterprise Applications – SAP, Oracle, Cadence, PTC, Sugar CRM CSC TRUSTED CLOUD Powered by Web Application Performance Testing Web Applications Industrialized Solution – CAT / SAPT FUTURE OFFERINGS CSC OPEN SOURCE CLOUD


TaaS experience at CSC

33 © 2011 Computer Sciences Corporation

CSC Client

Test Requirements & Application Components

CSC On-Demand Testing in Cloud


Virtual Test lab using SkyTap

Database servers Web Server Application Servers

Application Under Test Automated Software Quality Tools

Infrastructure Stabilization

Test Strategy & Planning

Test Execution Test Summary Reports

Certify Application Customer requests Testing from CSC Online portal / Telesales Provisioning / Scheduling of Tests CSC conducts Testing CSC publishes Test Results &

Reports Collaboration Customer Sign-off Cloud Orchestration CSC Client CSC & Client SOASTA Cloud

Typical Engagement Model





What things are the same about

testing cloud solutions?

From Paul Selway and Scott Rostal, Testing in the Cloud, in the Cloud.ppt


Requirements coverage based testing


Risk-based structured test approach


Test Plan / Test Phases


Test Cases / Test Data / Test Automation


Defect Management / Functional Test


Use of off-shore resources / Virtualization


Politics {Quality / Schedule / Resources}


What is different about testing

cloud solutions?

From Paul Selway and Scott Rostal, Testing in the Cloud, in the Cloud.ppt


Shared multi-tenant test environments


Security (Test User IDs / SSO)


Integration of on/off premise systems


Performance / volume test


Defect isolation


Documentation of

Out Of the Box



Release Management


What is harder than expected?


Isolating the architectural components for a performance



Understanding that a cloud solution vendor delivers new

releases which you have to take!


Accepting that you have a

black box

in your environment


Determining who owns defects


What is easier than expected?


Overlapping Test phases


End User Adoption


Use of off-shored test resources






Enhancements to testing methodology

for cloud testing


Test planning


Test design


Cloud setup


Requires familiarity of the target Cloud environment and of

its key functions


Understanding of Cloud monitoring tools is essential


Test execution


Requires usage of additional resources (computational units,

storage, communication channels), possibly TaaS






All used resources need to be released


Storage resources could be used for archiving test cases,

results, used VMs could be saved as images and for later



Testing focuses when migrating an

application in the Cloud


Test application changes needed to adapt the

application to the Cloud


Check on-premise interfaces


Check data migration and backup mechanisms


Focus on new security aspects that may have added


Encrypted data in cloud storage


New secure channels


Performance and scalability


Compare with on-premise setting


Make sure that you are making good use of the

features offered by the Cloud


CloudBees main characteristics (1)


Differently from GAE (and Azure), relies on well-known, state of

the art components




Jboss 6 Application Server




Can execute any standard Java web application


An application built for CloudBees can be moved to any other

environment, provided that you install the state of the art

components it uses


No vendor lock-in


Offers a load balancer and manages an application cluster


The developer, however, is in charge of maintaining HTTP

session on some repository


CloudBees main characteristics (2)


Supports a continuous integration process through an

on-demand Jenkins (Hudson) service


Each time a new version of a software component is

released, the system is automatically built and verified

against an array of tests


The build process can take into account the definition


CloudBees partners eco-system




New Relic: performance monitoring service


Papertrail: aggregation and analysis of log files


Sauce Labs (Selenium): automated in-browsed testing

of web applications


MongoHQ (MongoDB): noSql database


SendGrid: SMTP service




Cloud testing


Parveen, T.; Tilley, S.; , "When to Migrate Software Testing to the

Cloud?," Software Testing, Verification, and Validation Workshops

(ICSTW), pp.424-427, 6-10 April 2010


Banzai, Takayuki et al., "D-Cloud: Design of a Software Testing

Environment for Reliable Distributed Systems Using Cloud

Computing Technology," Cluster, Cloud and Grid Computing

(CCGrid), 2010 10th IEEE/ACM International Conference on , pp.

631-636, 17-20 May 2010


SOASTA website


Lef Grant Briefing, H.KOTHANDARAMAN, Testing applications n




WAYNE ARIOLA, The Next Generation of Test Environment

Management: Application-Behavior Virtualization,




Testing cloud-based applications


Paul Selway and Scott Rostal, Testing in the Cloud, in the Cloud.ppt


RAJAGOPAL SATTALURI, Testing Considerations for Application Migration

to Cloud Computing,







Related subjects :