Test automation is a use of special software to run and configure the test and test environment, and collect test results. The software which runs the tests should differ from the software which is tested. Automation is very beneficial for performance testing, as it allows to run preparation of the test in similar flow, when actions for preparation are performed in the same sequence, with the same delay for all scenarios. Besides, the tests can be run faster.
5.2.1 Test automation tools
Jenkins is widely used for test automation. Jenkins, originally called Hudson, is an open source Continuous Integration tool written in Java.
Continuous Integration, in its simplest form, involves a tool that mon-itors a version control system for changes. Whenever a change is de-tected, this tool automatically compiles and tests the application[18].
Jenkins can download the changes of the tests from the repository, there is a possibility to specify many repositories, and then to run the scripts, where further actions on tests automation are specified. In Jenkins there is a possibility to provide some settings, for tests run, including, the names of the host where to run the scripts, and the version of JDK to use, and others.
The client-server test environment requires flexible management of testing process on multiple hosts. This can be done by usage of Smart-Frog.
SmartFrog is a powerful and flexible Java-based software framework for configuring, deploying and managing distributed software systems[6].
SmartFrog has it’s own language to describe the sequence of actions, which should be implemented to run SmartFrog components. Smart-Frog components are Java classes, which implement predefined inter-faces, and are considered by SmartFrog a single unit of work. In the main configuration SmartFrog script, one can specify the interaction
5. Testing environment and test automation of SmartFrog components, for example to run them in parallel or in sequence, specify the host where to run them, and what should happen with the whole system if one component fails.
6 Task formulation and test automation implementation
The main goal of this diploma thesis is to compare performance of JBoss Fuse and SwitchYard.
In order to achieve more fair comparison of performance results, performance of JBoss Fuse and SwitchYard was tested with the same testing tool. PerfCake was chosen as a testing tool, because it matches all requirements that were set by design of performance tests. The per-formance tests execution was automated on Jenkins, using SmartFrog.
I will focus on description of automation for JBoss Fuse, because au-tomation for SwitchYard is implemented in a similar manner and uses the same design model.
6.1 Implementation of test automation
The code with test scenarios, the messages which are sent for load generations, and applications, which are deployed to JBoss Fuse to test some functionality (let’s call them tested service (TS)), are stored in the repository, managed by version control system.
All performance test scenarios for JBoss Fuse are accessible through the Jenkins job.
Figure 6.1: Tests automation in Jenkins.
In order to rebuild the tests only one click on the link in web browser is required. This will execute sequential run of all tests automatically.
6. Task formulation and test automation implementation Let’s consider the actions which are done during the automated execu-tion of one performance test scenario.
1)The Linux script kills all possible residual processes from previous test runs, including the process of JBoss Fuse, exactly Karaf on the server machine, and PerfCake process from the client machine. Besides, new code changes are upload to the SmartFrog components, scenarios, messages and tested services from the repository. The folders with all code, are located in the folders shared between all machines in the network, so that both host and server could access them. This script is run by Jenkins.
2)After the Linux script is finished Jenkins runs SmartFrog on the client machine, and provides it with its configuration script. The further steps are described in the SmartFrog configuration script.
3)SmartFrog script performs the following sequence of actions, as drown on the picture below.
Figure 6.2: Components sequence execution in SmartFrog.
SmartFrog runs PrepareServer component at the server host, and PrepareTestsAtClient at the client host. The execution of components at host and server start at the same time and in parallel. When Pre-pareServer component is finished, SmartFrog starts PrepareTestsAt-Server component at server host. So, SmartFrog executes PreparePrepareTestsAt-Server and PrepareTestsAtServer in one block, which corresponds to execution of PrepareTestsAtClient component. When PrepareTestsAtServer and PrepareTestsAtCleint are finished, SmartFrog proceeds to another step of execution. It run StartServer component at Server, and RunTest-sAtClient component at Client. These components run in parallel and start at the same time. If any of the components fails, the whole system stops execution and the test is marked as failed.
6. Task formulation and test automation implementation Most of the components, except StartServer use Groovy scripts in which is written the sequence of actions. The component StartServer extends Fuse component, which can start and stop JBoss Fuse, with some settings like which version of Java to use. I created it during the work on this thesis. Groovy is an object-oriented programming language designed for the Java platform as a addition to the Java language with the possibilities of Python, Ruby and Smalltalk. Now, let’s take a look at each component:
PrepareServer: PrepareServer script deletes old JBoss Fuse instance from server and installs a new one. Then it adds the user which can access JBoss Fuse.
PrepareTestsAtServer: PrepareTestsAtServer builds the specified tested service with maven, using mvn-bundle-plugin, as each tested service is an OSGi-bundle. OSGi-bundles are described in the Subsection 3.1.4 of the Chapter 3. After the bundle was built, the component either deploys a feature, if it is present in the re-source directory of the bundle, or deploys the jar archive with the bundle itself. The deployment happens through ”hot de-ploy” mechanism, which means the artifact (a bundle or a fea-ture) is copied into the deploy directory of JBoss Fuse. After the server will be started from the StartServer component, JBoss Fuse will try to start all bundles which are in deploy directory.
PrepareTestsAtClient: PrepareTestsAtClient script deletes old Per-fCake instance from server and installs a new one. Then it downloads the activemq.jar to the lib directory of PerfCake to be able to run tests for ActiveMQ client through JMS.
StartServer: This is Java-based component which starts JBoss Fuse, and stops it when the component is terminated from RunTest-sAtClient component.
RunTestsAtClient: RunTestsAtClient script specifies which version of Java to use to run PerfCake. After that it runs PerfCake for the specified test scenario.
4) When SmartFrog finishes the execution, runs another Linux script in Jenkins, which collects and saves the results. The results are stored in comma-separated values (CSV) format.
6. Task formulation and test automation implementation