Volume 2, No. 12, June 2014 ISSN – 2278-1080
The International Journal of Computer Science &
Applications (TIJCSA)
RESEARCH PAPER
Available Online at
http://www.journalofcomputerscience.com/
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved1
A Survey on Functional and Non-functional
Testing Tools released during 2000-2013
Lalitha Shastry
Dept. of MCA
Siddaganga Institute of Technology
Tumkur, India
Asha Gowda Karegowda
Dept. of MCA
Siddaganga Institute of Technology
Tumkur, India
Latha Raju
Dept. of MCA
Surana College PG Dept
Bangalore, India
Abstract
The paper attempts to provide a wide-ranging view of the field of software testing tools in particular functional
and non-functional testing tools. The objective of the paper is more biased towards the relevant functional and
non-functional testing tools released during 2000-2013 and depicted in a unified context. The system is tested by
functional testing tools by providing appropriate input, verifying the output and comparing the actual results
with the expected results. Non-functional requirements tend to be those that reflect the quality of the product,
particularly in the context of the suitability perspective of its users. This paper does not mean here to give an
inclusive survey of testing tools rather it gives brief description of a total of 44 functional testing tools and 15
non-functional tools with their year of release and platform/framework.
Key Terms
-
Software testing, functional testing tools, non functional testing
1.
Introduction
Software testing is a process of verifying and validating that a software application or program .It meets the
business and technical requirements that guided its design and development, and works as expected
[1].
Software testing refers to process of assessing the software with intention to find out error in it. It is technique,
aimed at evaluating an attribute or capability of a program or product and determining that it meets its quality.
Quality factors may include reliability, usability, integrity, security, capability, efficiency, portability,
maintainability, compatibility etc.
[2]
. It focuses on finding defects in the final product rather in the initial stage.
Software testing has three main purposes: verification, validation, and defect finding. The verification process
confirms that the software meets its technical specifications. A “specification” is a description of a function in
terms of a measurable output value given a specific input value under specific preconditions. The validation
Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved2
process confirms that the software meets the business requirements. A defect is a variance between the expected
and actual result. Software testing can be carried out using both manual testing and using automated software
testing tools. Since manual testing is time-consuming and error-prone, there is increasing need and demand for
testing tools. The software testing tools not only reduce manpower, time, effort and money spent on testing, but
also improve the quality and reliability of the software. Software testing can be categorized using different
methods. Software testing is classified into unit testing, integration testing, system testing and Acceptance
testing based on levels of testing. The software testing can further be grouped based on different approaches as
top-down vs. bottom-up approach, functional testing versus structural testing, mutation testing and regression
testing. This paper attempts to provide the purpose of the functional and non-functional testing tools released
during the year 2000-2013. Section 2 describes the functional and non-functional testing tools. Section 3
describes the functional and non-functional testing tools in terms of purpose, year of release and platform
followed by conclusions in Section 4.
2.
Functional Testing and Non functional Testing [3]
There are different ways of categorizing the testing tools. One of the broad ways of categorizing the testing tools
is functional vs. non-functional testing tools. Functional Testing usually describes what the system does. It
verifies a program by checking it against design document(s) or specification(s). It verifies that each function of
the software application operates in conformance with the requirement specification. This testing mainly
involves black box testing and it is not concerned about the source code of the application. The system is tested
by providing appropriate input, verifying the output and comparing the actual results with the expected results.
It involves checking of User Interface, APIs, Database, security, client/ server applications and functionality of
the Application under Test. Functional testing can be done either manually or using automation. The major
types of functional testing are unit testing, integration testing, acceptance testing, system testing, smoke testing,
sanity, black box testing and regression testing [3, 49].
The non Functional Testing is the type of testing done against the non functional requirements. Most of the
criteria are not considered in functional testing so it is used to check the readiness of a system. Non-functional
requirements tend to be those that reflect the quality of the product, particularly in the context of the suitability
perspective of its users. It checks the Performance, reliability, scalability and other non-functional aspects of the
software system. It can be started after the completion of Functional Testing. The non functional tests can be
effective by using testing tools. The non-functional testing describes how good the product works in contrast to
Functional testing which describes what the product does. The major types of non-functional testing are
Performance Testing, Load Testing, Volume Testing, Stress Testing, Security Testing, Installation Testing,
Penetration Testing, Compatibility Testing, and Migration Testing [3, 49].
3. Functional and non-functional testing tools released during 2000-2013
Many functional testing tools are released compared to non-functional tools. Authors have attempted to collect
the details of both functional and non-functional tools released during 2000-2013. Since the numbers of
functional tools are many in number, the details of these tools are shown in different tables year wise. Table 1 ,
Table 2 , Table 3 , Table 4 and Table 5 shows the details of various functional testing tools with year of release,
purpose and platform during the year 2000-2002, 2003-2006, 2007-2009, 2010-2011and 2012-2013
respectively. In addition, the graph in figure 1 depicts the total number of functional testing tools released in
individual years from 2010 to 2013. Moreover, Figure 2 shows the number of functional testing tools released
for different platforms/framework. Furthermore, Table 6 and Table 7 gives the details of the different
non-functional tools with year of release, purpose and platform, released during 2000-2010 and 2011-2013
respectively. Figure 3 depicts the number of non-functional testing tools released during the year 2000-2013
year wise. Figure 3 provides a clear indication of highest number of non-functional testing tools is released
during the year 2013.
4. Conclusion
The use of software testing tools not only simplify testing, but helps in achieving a better quality and reliable
software product making them more productive and effective from the customers perspective. The
non-Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved3
functional testing describes how good the product works in contrast to Functional testing tools which describes
what the product does. This paper briefs about the various functional and non functional testing tools released
during the year 2000 -2013. It is observed that, more number of functional tools were released when compared
to non-functional tools during the 2000-2013.
SL. No
Name of the Functional Testing Tool &
Year of release Purpose
Platform &Reference
1 DejaGnu 2000
DejaGnu is a framework for testing applications such as Tcl, C, C++, Java and network applications and cross testing of embedded systems. Its purpose is to provide a single front end for all tests.
MacOS, Windows, POSIX [44] 2 Doit:Simple Web Application Testing 2000
Doit is a scripting tool and language for testing web applications. It uses a console-based web client tool (like Curl or Wget) to send and receive HTTP requests and responses respectively.
Perl 5.6 [4]
3 Harness 2000
Harness is an open source JAVA API for creating java test software.
It provides stubs and drivers, which are small programs that interact with the software under test. This tool provides support for object-oriented software. This type of tool enables the developer to test, identify and localize any defects. The stubs and drivers supply any information needed by the software being tested (e.g. an input given by the user) and also receive any information sent by the software (e.g. a value to be displayed on a screen).
Java [5]
4 LogiTest 2001
LogiTest is the core application in the LogiTest suite. The LogiTest application provides a simple graphical user interface for creating and playing back tests for testing Internet-based applications.
JDK 1.2 or higher [6]
5 Scalable Platform Test
2001
STP is a test framework for the Linux kernel. Patches can be created and built through the Patch Lifecycle Manager (PLM), and then run through our choice of performance and scalability tests. Kernel patches are loaded and built against stock kernels, and tests are executed on our own selected hardware platform.
Linux [7]
6 XML Test Suite 2002
XmlTestSuite provides a powerful way to test web applications. It is adopted by testers, business analysts, and web developers. The main aims of XmlTestSuite are Tests are written in XML and it also allows use of syntax checking. XMLeditors uses excellent free editor called jEdit.
Windows 95/98/2000, Windows NT/2000, Linux, SunOS/Solaris [8] 7
Abbot Java GUI Test Framework 2002
Abbot is a simple framework for unit and functional testing of Java GUIs.It facilitates the automated event generation and validation of Java GUI components. The framework may be invoked directly from Java code or accessed without programming through the use of scripts. In addition, it supports record and playback on any Java application.
Java [9]
8 Anteater 2002
Anteater is a testing framework designed around Ant, from the Apache Jakarta Project. It provides an easy way to write tests for checking the functionality of a Web application or for an XML Web service.
OS Independent [10]
9 Pounder 2002
Pounder is a utility for testing Java GUIs. It allows developers to dynamically load components, record scripts, and then use those scripts in JUnit. It supports custom
components, drag and drop. Junit [11]
10 QMTest 2002
QMTest is an open-source, general-purpose, cross-platform software testing tool written entirely in Python. QMTest can be used to test software applications such as compilers, databases, graphical user interfaces, or embedded systems. QMTest provides a convenient graphical user interface for creating, managing, and executing tests, provides support for parallel test execution.
Unix, Linux,
GNU and
Windows [12] Table 1. Functional testing tools released during the year 2000-2002
Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved4
SL. No Name of the Functional Testing Tool& and Year of Release Purpose Platform &Reference 1 Jameleon 2003
Jameleon is an automated testing tool. One of the main concepts behind Jameleon is to create a group of keywords or tags that represent different screens of an application. All of the logic required to automate each particular screen can be defined in Java and mapped to these keywords. The keywords can then be organized with different data sets to form test scripts. The test scripts are then used to automate testing and to generate manual test case documentation.
OS Independent, JDK 1.4 or higher [13]
2 Canoo WebTest 2003 Canoo WebTest is a free open source tool for automated testing of web applications. It calls web pages and verifies the results, giving absolute reports on success and failure. It
helps us to reduce the defect rate of your web application.
JDK 1.2 and ANT v1.3 [14]
3 WebInject 2004 WebInject is a free tool for automated testing of web applications and services. It can be used to test any individual system component with an HTTP interface. Windows, OS Independent,
Linux [15]
4 Arbiter 2004
Arbiter is a requirements gathering and verification tool for web projects. Its aim is to sharpen the communication between stakeholders and developers. Arbiter itself runs as a simple document repository on a web server. When requirements documents are added or updated, they are frayed of test cases and these tests are run. The tests are just examples of how someone would use a web browser to get information from the site.
OS Independent [44] 5 Citrus Integration Testing framework 2004
Citrus is an integration testing framework written in Java that tests your software application to fit into your customer’s environment. The tool simulates surrounding systems across various protocols (e.g. Http, JMS, TCP/IP, SOAP,) in order to perform automatic end-to-end use case testing. Citrus provides strong validation mechanisms for XML message contents and allows to build complex testing logic.
Java VM [16]
6 jWebUnit 2004
JWebUnit is a Java framework that facilitates creation of acceptance tests for web applications. JWebUnit provides a high-level API for navigating a web application combined with a set of assertions to verify the correctness of the application
JUnit and Http Unit [17]
7 Tclwebtest 2004
Tclwebtest is a tool for writing automated tests on web applications in tcl. It implements some basic html parsing functionality to provide comfortable commands for operations
on the html elements of the result pages. HTML [18]
8 Avignon 2005
Avignon is an acceptance testing tool that allows you to write executable tests in a language that we define. It uses XML to define the syntax of the language but, if you choose to extend the language, leaves the semantics of the tests up to the developers choice. Avignon includes modules for testing HTML applications (through either IE or Firefox), Swing and .NET WinForm applications.
Java (MS
Windows only
for .NET
testing) [44]
9 WET Tester) (Web
2005, 2006
WET is opensource web automation testing tool which uses Watir as the library to drive web pages. WET allows you to perform various checks as a part of the testing process by using Checkpoints. Windows 98/ME/2000 SP3/XP SP2/Server 2003 [19] 10 Win32::IEAuto mation 2005, 2006
A Perl module which automates functional testing of web applications. It can be used to
automate any complex web application including dynamic frames and popup windows. Windows (only Internet
Explorer is
supported) [20]
11 Netbeans 2006
Netbeans is an integrated development environment (IDE) for developing primarily with JAVA, but also with other languages, in particular PHP, C/C++ and HTML.5.It is also an application platform framework for Java desktop applications and others. The Netbeans platform allows developing applications.
Java 1.2 or above [21] Table 2. Functional testing tools released during the year 2003-2006
Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved5
SL. No Name of the Functional Testing Tool/Year of Release Purpose Platform& Reference 1 Autonet 2007
Autonet is a tool for checking binary compatibility of a shared C/C++ library. It is a
GUI network test platform; internally it is based on CLI to communicate with devices. It helps in arrangement of test cases, setup commands to devices, run commands and record test results simultaneously.
windows, Linux and any other platform which support tcl [44]
2 DBFeeder 2007
DBFeeder can automatically generate testdata for Oracle Databases which fits primary and foreign keys of tables. It is very easy to use since in a few clicks lot of tables get filled immediately and start testing our processes. This is especially used when we are first time db-process developments and we don’t have data on which they can work on.
OS Independent[22]
3 DbFit 2007
It is a testing tool for database code. It is an extension to FIT/FitNesse for test-driven database development and enables developers to manipulate database objects in a relational/ tabular form, making database testing and management much easier with xUnit-style tools.
Java/.NET [23]
4 MaxQ 2007
MaxQ is a free web functional testing tool. It includes an HTTP proxy that records the test script, and a command line utility that can be used to playback tests. The proxy recorder automatically stores variables posted to forms, so we don’t have to write that stuff by hand.
Java 1.2 or later [24]
5 2007,2009,2010 csvdiff/
Csvdiff is a perl script for comparing two files of comma-separated values. In contrast to standard difference, it will show the number of the record where the difference occurs, the column number, and (if provided) the fieldname which is different. The separator can be set to any value you want, not just a comma. It also provides support for multiple column keys, the ability to ignore case and trim leading/tailing spaces, and the ability to ignore selected columns such as timestamps.
Everywhere with perl [25]
6 Marathon 2008
Marathon is simple, flexible testing tool for exercising Java GUI applications developed using Swing. It consists of a recorder, player and an editor. Marathon scripts are Jython scripts.Marathon is a rich suite of components to help you interact
with your application at the User Interface Level (GUI). Marathon is used mainly for
automation of functional tests (also called as acceptance tests, end user tests or customer tests). Marathon itself adds only a few well defined constructs to Jython library. It is conceivable that Marathon can be used to develop tests even before an application is available.
Java 1.3 or later [26]
7 WebTst
2009
WebTst is a Web development test infrastructure. It comes with support for digital certificates, and a number of simple tests, such as cookie setting, pattern matching, response status, and many others. It features an extensible plug-in system.
POSIX, Linux [27]
8 ABI Compliance Checker
2009
A tool for checking backward binary compatibility of a shared C/C++ library. The tool checks header files and shared libraries of old and new versions and analyses changes in API and ABI. The tool is intended for developers of software libraries and maintainers of operating systems, who are interested in ensuring backward compatibility, i.e. allow old applications to run or to be recompiled with newer library versions.
Linux, Unix, BSD, Haiku [28]
9 CubicTest 2009
Cubic Test
is
anopen sourcetool for writing functional tests for web applications.It is a graphical Eclipse plug-in for writing Selenium and Watir tests. It makes web tests faster and easier to write, and provides abstractions to make tests more robust and reusable.
Eclipse plug-in [29] Table 3. Functional testing tools released during the year 2007-2009
Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved6
SL. No Name of the Functional Testing Tool &Year of Release Purpose Platform & Reference 1 BlueDuck SDA 2010
BlueDuck SDA (Selenium Driver AutoIt) is an open-source automated testing framework that can be easily used by technical and non-technical users alike. It allows you to write tests that are easy to read and maintain. BlueDuck SDA is a powerful, easy-to-use framework designed to help you build dynamic and robust test scripts for Web and Win32 apps.
Windows XP - Vista – Seven
[30]
2 ITP
2010
It is a stand-alone Java application that can test your website from a user's perspective. It is amazingly simple and lightweight. There is no programming involved, Test scripts are written in XML. No programming required and no changes required to web application. Supports sessions/cookies, POST form data and also useful for regression and smoke testing
Java Runtime Environment 1.6+ [31]
3 Ivalidator 2010 Regression testing framework written in java but by no means restricted to java testing. Test suites are declared in XML. Especially designed for complex testing
scenarios and integration testing.
JDK 1.3 [32]
4 Latka
2010
Latka is a functional (end-to-end) testing tool. It is implemented in Java, and uses XML syntax to define a series of HTTP (or HTTPS) requests and a set of validations used to verify that the request was processed correctly.
JDK 1.3 or better [33]
5 Solex 2010
Solex is a free open source Web application testing tool built as a plug-in for the Eclipse IDE. It provides functions to record a client session, adjust it according to various parameters and replay it later typically in order to ensure non regression of the application's behaviour (with stress testing capabilities being added at a later stage).By recording, we mean that Solex acts as an HTTP proxy and records all HTTP requests and responses going through the wire between a Web client (eg. a Web browser) and a Web server. The task of replaying a scenario consists in sending the previously recorded and eventually customized HTTP requests to the server and asserting each response.
Eclipse 2.1 or above [34]
6 WebDriver 2011
WebDriver is a tool for automating testing web applications. It aims to provide a friendly API that's easy to explore and understand, which will help make our tests easier to read and maintain. It's not tied to any particular test framework, so it can be used equally well with JUnit, TestNG.
Any java-compatible platform [35]
7 Jemmy 2011
A tool allowing you to create automated tests for Java GUI applications. It contains methods to reproduce all user actions which can be performed on Swing/AWT components (i.e. button pushing, text typing, tree node expanding,).It can be used with any test harness (such as XTest, for example). The programmer simply has to adopt his tests to the harness requirements.Tests are written on Java, using Jemmy as a regular Java library. No recording (yet), no GUI, no XML, and no bells and whistles – all the work being done in Jemmy is dedicated to test stability, so it could be used for big, complicated and “dynamic” Java GUI applications.
Java.net [36] Table 4. Functional testing tools released during the year 2010-2011
Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved7
SL. No Name of the Functional Testing Tool &Year of Release Purpose Platform& Reference 1 DB Sanity 2012
DB Sanity performs checks on a database for verifying data consistency and correctness of application deployments. It is invoked by command line and creates HTML reports with collective analytic summaries and listings of faulty database entries. Windows, Mac OS X, Linux, Unix, Solaris, HPUX, Oracle, MS SQL Server, DB2, MySQL, HSQL, H2, Derby, Firebird [37] 2 JFunc 2012
JFunc is an extension to the JUnit testing framework to make it easier for use with functional tests It offers many features like easing the pains of manual suite construction like use one test object for a series of tests, rather than the one test instance per test, Enhanced Test Runner like Pass arguments to tests on the CLI
Junit [38]
3 Concordion 2013
Concordion is an open source tool for Java that lets you turn a plain English description of a requirement into an automated test. This makes the specification much more natural to read and write. Its specifications are written in HTM with special attributes. Concordion enriches agile processes and has been successfully applied in industrial context.
Java 1.5 or above [39]
4 Enterprise Web Test 2013
Enterprise Web Test allows Java programmers to write re-usable tests for web
applications unlikeHttpUnit "drive" the actual web browser on the actual platform
they intend to support.
Microsoft, OS Independent,
Linux[40]
5 Linux Test Project 2013
The Linux Test Project is a collection of tools for testing the Linux kernel and related features. Our goal is to improve the Linux kernel by bringing test automation to the kernel testing effort.
Linux[41]
6
QAT (Quality Assurance Tests)
2013
QAT was developed to ease the issues encountered by having to perform Quality Assurance tests across a variety of hardware and software combinations. The QAT tool can be divided into two main sections, the Agent, responsible for actually running each test or group of tests, and the Harness, which is responsible for test selection, management, and result and agent co-ordination.
Java 2[42]
7 Selenium 2013
It is a testing tool for browser-based testing of web applications. Selenium provides a record/playback tool for authoring tests without learning a test scripting
language (Selenium IDE).It also provides a test domain-specific
language (Selenese)[1] to write tests in a number of popular programming
languages, including Java, C#, Groovy, Perl, PHP, Python and Ruby.
Windows, Linux or Mac[43] Table 5. Functional testing tools released during the year 2012-2013
Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved8
0
1
2
3
4
5
6
7
200
0
200
1
200
2
200
3
200
4
200
5
200
6
200
7
200
8
200
9
201
0
201
1
201
2
201
3
Year of Release
N
um
be
r
of
F
un
ct
io
na
l T
es
ti
ng
To
ol
s/
ye
ar
3
13
11
3
1
6
16
1
1
2
0
2
4
6
8
10
12
14
16
18
Ma c O S Wi n d o w s 95/ 98/ N T /2000/ M E /S P 3/ S P 2/ S er ve r 2 0 0 3 /X P LI N U X UNI X SU N O S/ So la ri s OS I n d e p e n d e n t JA V A ,JV M ,JD K ,JU N IT ,E cl ip se pl ugi n GN U BSD .N e tPlatform/Framework
#
N
u
m
b
er
o
f
to
o
ls
Fig1: Number of functional testing tools released per year during 2000-2013
Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved9
Sl No.
Name of the Non functional Testing tools &
year of release
Purpose
Platform & Reference
1. Httperf 2000
Httperf is a high performance testing tool for measuring and analysing the performance of any web services and web applications. This is mainly used to the test the HTTP servers and its performance. The main objective of this testing tool would be to count the number of responses generated from this particular server. This generates the HTTP GET requests from the server which helps in
summarizing the overall performance of the server. Through this tool, we will be able to conclude the rate at which the response is sent from each server and thereby the efficiency can be calculated.
Windows and Linux [49]
2. NeoLoad(1.0) 2005
NeoLoad is a load and stress testing tool to measure the performance of web and mobile applications. NeoLoad provides realistic solutions to developers to help them optimize performance before the application goes into
production.NeoLoad's exclusive technology allows you to perform web testing more quickly, efficiently, and frequently, which means we can immediately identify the root cause of any bottlenecks or issues.
Microsoft Windows, Linux,
Solaris [47]
3. OpenSTA(1.4.4) 2007
Open STA stands for Open System Testing Architecture. This is a GUI based performance tool used by application developers for load testing and analysing. This is believed to be a complex tool among the all other performance testing tools. It has proven capability in the past and the current toolset is capable of performing the heavy load test and analyses for the scripted HTTP and HTTPS. Here, the testing is carried out by using the recordings and simple scripts. To carry out the test successfully, results and other statistics are taken through various test runs. These data and results can be later exported to software for creating reports.
Windows[49]
4. Loadstorm 2008
Loadstorm is the cheapest available performance and load testing tool. Here, we have the option of creating our own test plans, testing criteria and testing scenario. Through this tool, you can bring an end to all the expensive performance testing tools. The cloud infrastructure is used in this tool, which enables you to send huge amount of requests per second. There are thousands of servers available around the world for this software. They are proudly known as the lowest cloud load testing tool.
Windows OS[49]
5. LoadImpact 2009
Load Impact is a leading online load testing service that lets us load test our website over the Internet. Load scripts are used to program the behaviour of simulated users in a load test. Apart from native functionality of the Lua language, load script programmers can also use Load Impact’s load script APIs to write their advanced load scripts. Now we can script the user scenarios in the simple but powerful language Lua, using our programmer friendly IDE and new APIs such as: JSON and XML parsing, HTML form handling, Bit-fiddling, and more. This also helps in website optimization and improvising the working of any web application. This tool generates traffic to the website by simulating users so as to find the stress and maximum load it can work. This LoadImpact comprises of two main parts; the load testing tool and the page analyser. The load testing can be divided into three types such as Fixed, Ramp up and Timeout. The page analyser works similar to a browser and it gives information regarding the working and statistics of the website.
Windows OS&Linux [49] 6. QEngine (ManageEngine) 7.1 2009
QEngine (ManageEngine) is a most common and easy-to-use automated testing tool helping in performance testing and load testing of your web applications. Many developers find it to be the most simple and easy tool to use for finding out any leakage in their web services or websites. The key important feature of this testing tool is its ability to perform remote testing of web services from any geographical location. Other than that, QEngine (ManageEngine) also offers various other testing options such as functionality testing, compatibility testing, stress testing, load testing and regression testing. This automated testing tool has the capacity to generate and simulate lot if users so that the performance can be well analysed during the maximum load.
Windows and Linux&[49]
7. Anywhere6.0.1 Testing
2010
Test Anywhere is an automated testing tool which can be employed for testing the performance of any web sites, web applications or any other objects. Many developers and testers make use if this tool to find out any bottlenecks in their web application and rectify them accordingly. It is a powerful tool which can test any application automatically. This testing tool comes along with a built in editor which allows the users to edit the testing criteria according to their needs.
Windows [49]
8. LoadUI(1.0-beta2 2010
LoadUI is an open source API load testing solution that is free and cross-platform. With a visual, drag-and-drop interface, it allows us to create, configure and redistribute load tests interactively and in real-time. In a single test environment, LoadUI provides complete test coverage and supports all the standard protocols and technologies. And it’s so powerful; it generates scalable, high-volume and real-world load from any number of local and remote computers. Windows XP/Vista/7; Windows Server 2003/2008 (Standard and Enterprise editions) [48] Table 6. Non Functional testing tools released during the year 2000-2010
Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved10
Sl No.
Name of the Non functional Testing tools &
year of release
Purpose
Platform & Reference
1. Loadster 2011
Loadster is a desktop based advanced HTTP load testing tool. The web browser can be used to record the script which is easy to use and record. Using the GUI you can modify the basic script with dynamic variables to validate response. With control over network bandwidth you can simulate large virtual user base for your application stress tests. After test is executed HTML report is generated for analysis. This tool is best to identify performance bottlenecks in our application.
Windows 7/Vista/XP [49]
2.
LoadRunner 11.50
2012
HP LoadRunner is the industry standard for application performance testing. It is extremely flexible for organizations and projects of all sizes, this testing tool enables you to test a range of applications including mobile, Ajax, Flex, HTML 5, .NET, Java, GWT, Silverlight, SOAP, Citrix, ERP and legacy. The load testing tool helps us to prevent issues by detecting bottlenecks and to obtain an accurate picture of end-to-end system performance before going live.
Windows XP Professional SP3/ Windows Server 2003 R2 Standard Edition/Enterprise Edition SP2 / Windows Server 2008 Standard Edition/Enterprise Edition SP2 / Windows 7 SP1/Windows Server 2008 R2 SP1 [45] 3. WAPT(3.0) 2012
WAPT is a load and stress testing tool that provides an easy-to-use and cost-effective way to test any web site, including business applications, mobile sites, web portals, etc. With this tool you have the advantage of testing the web application performances in various different environment and different load conditions. WAPT provides detailed information about the virtual users and its output to its users during the load testing. The WAPT tools can test the web application on its compatibility with browser and operating system. It is also used for testing the compatibility with the windows application in certain cases.
Microsoft Windows XP/2003/Vista/2008/Win7/ Win8 [50] 4. Apache JMeter(2.10) 2013
Apache JMeter is a fully open source pure java tool designed for the purpose of load testing and measuring performance of web applications. It can be used to test both static and dynamic web resources such as files, Servlets, web services, Perl scripts, Java Objects, Data Bases and Queries, FTP Servers etc. JMeter can effectively use to simulate heavy loads on the above resource types and analyse the overall performance under different load conditions such as concurrent loads. It also provides various types of visualization techniques to graphically analyse the results obtained from the load tests.
JAVA 6+[46]
5. WebLoad(10.1) 2013
WebLoad is one such tool used for load testing and stress testing. This tool can be used for Load testing any internet applications such as Ajax, Adobe Flex, Oracle Forms and much more. Through this tool, you have the ability to measure the working performance and also its response to the users. This tool is widely used in the environment where there is a high demand for maximum Load testing. This tool gives you clear information on the functionality and the actual capacity of the web applications. Windows 8/12 server[49] 6. Rational Performance Tester 8.5.1 2013
The Rational performance tester is an automated performance testing tool which can be used for a web application or a server based application where there is a process of input and output is involved. This tool creates a demo of the original transaction process between the user and the web service. By the end of it all the statistical information are gathered and they are analysed to increase the efficiency. Any leakage in the website or the server can be identified and rectified
immediately with the help of this tool. This tool can be the best option in building an effective and error free cloud computing service.
Windows and Linux [49]
7.
CloudTest
(SOASTA49 )
2013
CloudTest is a performance testing tool for the cloud computers. The users or the developers can use the cloud platform as their virtual testing lab. The developers can carry out their performance or load testing in the cloud platform in the cost effective way through this CloudTest tool. This CloudTest has the capacity to enable number of users to use the website at the same time. It also increases the traffic of the website to know the actual performance under stress and heavy load. They provide many services for testing the websites and other web applications and now they also help in testing the mobile applications.
Windows, Linux and Mac OS.[49] Table 7. Non Functional testing tools released during the year 2011-2013
Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved11
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
20
00
20
01
20
02
20
03
20
04
20
05
20
06
20
07
20
08
20
09
20
10
20
11
20
12
20
13
Year of ReleaseN
um
be
r
of
N
on
-f
un
ct
io
na
l Te
st
in
g
To
ol
s/
ye
ar
References
[1]
http://softwaretestingfundamentals.com/differences-between-black-box-testing-and-white- box-testing/
[2]
http://www2.sas.com/proceedings/sugi30/141-30.pdf
[3]
http://www.softwaretestingclass.com/functional-testing-vs-non-functional-testing/
[4]
http://doit.sourceforge.net/
[5]
http://sourceforge.net/projects/harness/
[6]
http://logitest.sourceforge.net/logitest/index.html
[7]
http://www.osdl.org/lab_activities/kernel_testing/stp/
[8]
http://sourceforge.net/projects/xmltestsuite/?source=navbar
[9]
http://www.automatedtestinginstitute.com/
[10]
http://aft.sourceforge.net/
[11]
http://pounder.sourceforge.net/
[12]
http://www.codesourcery.com/qmtest
[13]
http://www.coderanch.com/t/94815/Testing/Automation-Testing-Tool
[14]
http://webtest.canoo.com/webtest/
[15]
http://webinject.org/download.html
[16]
http://www.citrusframework.org/license.html
[17]
http://jwebunit.sourceforge.net/
[18]
http://tclwebtest.sourceforge.net/ http://toster.sourceforge.net/
[19]
http://wet.qantom.org/news.html
[20]
http://search.cpan.org/~prashant/Win32-IEAutomation-0.5/
[21]
http://jemmy.netbeans.org/
[22]
http://dbfeeder.sourceforge.net/
[23]
http://sourceforge.net/projects/dbfit/
[24]
http://maxq.tigris.org/
[25]
http://csvdiff.sourceforge.net/
[26]
http://marathonman.sourceforge.net/
[27]
http://www.vietnamesetestingboard.org/zbxe/
[28]
http://ispras.linuxbase.org/index.php/ABI_compliance_checker_Downloads
Lalitha Shastry, Asha Gowda Karegowda, Latha Raju, The International Journal of Computer
Science & Applications (TIJCSA) ISSN – 2278-1080, Vol. 2 No. 12 June 2014
© 2014,
http://www.journalofcomputerscience.com
- TIJCSA All Rights Reserved12
[29]
www.cubictest.org
[30]
http://sourceforge.ne t/projects/blueducksda/
[31]
http://www.incanica.com/itp.html
[32]
http://www.ivalidator.org/
[33]
http://jakarta.apache.org/commons/latka/index.html
[34]
http://solex.sourceforge.net/
[35]
http://www.infoq.com/news/2011/07/Selenium-2
[36]
https://jemmy.java.net/
[37]
http://wet.qantom.org/news.html
[38]
http://jwebunit.sourceforge.net/
[39]
http://www.concordion.org/Download.html
[40]
http://sourceforge.net/projects/webunitproj/
[41]
http://ltp.sourceforge.net/
[42]
http://qat.sourceforge.net/
[43]
http://selenium.thoughtworks.com/index.html
[44]
http://www.opensourcetesting.org/functional.php
[45]
http://www.myloadtest.com/loadrunner-11-5/
[46]
https://jmeter.apache.org/download_jmeter.cgi
[47]
http://en.wikipedia.org/wiki/NeoLoad
[48]
http://www.loadui.org/
[49]
http://www.softwaretestinghelp.com/performance-testing-tools-load-testing-tools/
[50]
