Comparative Analysis of Various Automated Test Tools for Flex Application
Nisha Kaushal, Rupinder Kaur
National Institute of Technical Teachers’ Training & Research, Punjab University, Chandigarh, Panjab
Abstract
This paper is aimed to carry a comparison study of automated testing frameworks for Rich Internet Applications (RIA) made using Flex framework.. Test cases will be written in different languages like Ruby, ActionScript etc. using identified frameworks on a
reference application. Frameworks (FunFX and
FlexMonkey) are evaluated on the basis of Execution time, CPU Utilization, Memory used, Number of lines of code, ease of automation, extensibility options, maintainability of automated test scripts, options to run test suites, options to generates test reports etc. Results of this study will be helpful to software development professionals to select better test automation framework for their application.
Keywords:
Flex Application, Automated testing FunFX, FlexMonkey,1. Introduction
Software testing is an integral part of Quality Assurance in Software development. Software testing is aimed at determine how well a software system meets the required results. Test Automation is an emerging field to meet these challenges and increase efficiency, report and catch bugs and software defects as early as possible in SDLC. Test automation overcomes few of the manual testing limitations of being slow and costly for the repetitive functional and regression testing. Manual testing is also not consistent or deterministically repeatable. Automating test cases leaves manual testers to focus more on writing test cases and analysis of defects and quality issues in the project. There was lack of automated test tools so Researches and professionals in test automation industry have produced many tools to automate testing of web [1]. These tools vary in terms of their capabilities and commercially fall in two either paid or open source tools. With so many tools in the market, in the past comparison studies have been done to evaluate features and usability of different tools. Watir and Selenium has come up top in the list of Open source test automation frameworks available and Rational robot, Quality Test Professional are most preferred in the list of commercial test tools/framework. Flex framework developed by Adobe in around 2004 is an answer to problems like browser compatibility, an adaptive framework to develop both web and desktop applications. Flex provides additional capabilities like adding charts
and graphs to applications. Using Flex it is claimed to be easy to put a fresh new face to legacy applications. Flex framework has picked up in the industry and thus there has been a demand to have test tools to automate testing of flex based applications. As the Flex based applications are becoming popular, recently enhancements are being made in exiting tools to support automated testing of flex applications. Few new test tools and frameworks are being written to provide automation of flex-based applications.
2. Flex Applications
Adobe Flex is a collection of technologies for the development and set up of cross platform Rich Internet Application. Flex enables to rapidly build applications that are installed to Flash player which means that Flash Player runs all Flex Applications. Features of flex are- • Flex is capable of providing integrated solutions
that utilizes data services, interactive UI design, media elements such as audio and video and even real time communications.
• It utilizes metaphors from desktop computing such as drag and drop and double click.
• It has small download time.
• It is a cross-platform and cross-browser compatible. It runs on Window, OS X and Linux and all the major browsers including Firefox, Internet Explorer, Safari and Opera.
3. Flex Applications And Rich Internet
Applications (RIA)
Flex is a framework developed by Adobe Systems Incorporated for developing RIA for the Flash platform. It was a result of the increased demand for more desktop-like applications for the Web.The initial release in March 2004 by Macromedia included a software development kit, an IDE, and a J2EE integration application known as Flex Data Services [2].Rich internet application (RIA) was introduced by Macromedia (in Macromedia flash MX-A next generation rich client) in 2002. RIA are web applications that have the features and functions of traditional desktop applications [3].
4. Automated Test Tools
Because of the more advantages of the automation testing over manual testing, various companies are
engaged in developing various automated test tools for various applications. There are two types of test tools.
• Open source test tools • Commercial test tools
Open Source Test tools- These test tools are free for the users to use. It can be downloaded from the internet or can be obtained by the vendor without any charges e.g. Watir, Selenium, FunFx.Commercial Test Tools- These test tools are not free and user has to pay for it e.g. TestComplete, Quality Test Professional, Rational robot.FunFX [4] is developed by Peter Motzfeldt as a result of his master thesis at the Norwegian University of Science and Technology in June, 2007 .FunFX is created as a framework that will enable to drive a Flex application through a web browser and it was the only solution of testing flex application.FunFX is a framework for functional testing of Adobe Flex applications. The framework encourages test-driven development, and provides a Ruby API for interacting with Flex application or it is a Ruby tool for functional testing of Adobe Flex applications. It is the only free alternative to achieve functional testing of Adobe Flex applications. FunFX is open-source and uses Ruby to write tests.
FunFX framework:
• The framework supports any Flex application compiled with the FunFX library file.
• The framework must be able to drive a Flex application programmatically.
• The framework must be able to support test-driven development.
• The framework must support writing tests in Ruby. • The framework must be able to run multiple test
suites, and provide useful results from the tests. FlexMonkey is the only Flex record/playback tool that is entirely Flex-based and was created by adapting Adobe's sample application, AutoQuick.. It requires no browser plug-ins (other than the Flash player of course) or special scripting languages. If you know Flex, you already know 99% of what you need to know to use FlexMonkey, and you can run FlexMonkey anywhere you can run Flex (ie, virtually any platform) [5].FlexMonkey is an open source Flex application and library that records and plays back user interface interactions and generates replayable test cases that can be run from a continuous integration framework such as Cruise Control. It is a unit testing framework for Flex apps that provides for automating the testing of Flex UI functionality. FlexMonkey can record and play back Flex UI interactions, and generates editable ActionScript-based testing scripts that can easily be included within a continuous integration process. It uses the Flex Automation API and was created by extending Adobe's sample automation adapter, AutoQuick. FlexMonkey is a new tool to perform automated testing on Flex applications. With it, a set of interactions can be recorded, save those commands into a TestCase class, add
the TestCase to the TestSuite, and include it as part of a FlexUnit testing workflow. Features:
• Records and plays back Flex UI interactions. • UI Interactions can be edited and replayed.
• Generates FlexUnit TestCases, and can also be used with non-FlexUnit-based testing frameworks. • Tests can be run from build systems such as Ant. • Handles all Flex UI events.
•
Uses Flex Automation API to provide native control over your flex app requires no javascript or browser plug-ins to use.•
Unit tests are written entirely in ActionScript. No other programming or special purpose scripting languages are needed to develop comprehensive UI test suites.•
Non-invasive. Requires no modifications to your application source.5. Methodology
Figure5 shows Reference flex application will be taken from internet for automated testing. Various automated test tools will be searched and test cases will be written using identified frameworks on a reference application. Test cases will be written in JavaScript, Ruby, ActionScript etc, depends upon the automated tools through which flex application being tested.
Frameworks are evaluated on the basis of Execution time, CPU utilization, Memory used, Number of lines of code, Ease of automation, Extensibility options, Maintainability of automated test scripts, options to run test suites, options to generate test reports etc.
• Execution Time – This parameter measures the time
taken by the tools for testing the applications in seconds.
• CPU Utilization – This parameter identifies the percentage of CPU usage during testing of applications by test tools.
• Memory usage – This parameter identifies the
percentage of RAM or memory usage during testing of an application.
• Number of lines of code – This parameter identifies
the number of line of code required to write the test cases for each tool. This will be calculated through a Line Counter tool.
Figure 5. Method of Automated Testing
6. Implementation
FunFX Automated test tool takes 59.688 seconds during testing of library application and FlexMonkey takes 45.122 seconds. Various test cases like Test_verify_addition_contents,
Test_add_success,Test_add_failure,Test_delete_success, Test_delete_failure are written in FunFX and FlexMonkey tool to do the automated functional testing of a library application. As these test cases provides constraints like the Test_add_success testcase provide constraint that the book details can only be added if the addbook button be clicked. All other testcases provide different constraints to do validation. FunFX will open Internet Explorer browser having url of home page. Home page consists of two buttons(add book and delete book) and one datagrid. Whereas FlexMonkey opens application in Adobe Flash player for automated functional testing.
In figure 6.1 FunFX through
Test_Verify_addition_contents testcase do functional testing of library application. As it checks that after filling various textboxes both the buttons work properly or not.During execution of Test_verify_addition_contents test case ,the FunFX will test the application automatically by opening the application in internet explorer and then automatically fills the textboxes(book title, author etc ) and then click on adddbook button ,after clicking the button book details will be added into a datagrid. Hence application works properly.in this way funfx can do automated functional testing and it takes 13.594 seconds during testing.
Test_Verify_addition_contents
Figure 6.1 Home page with FunFX
Figure 6.2 Functional Testing with
Test_verify_addition_contents testcase in FunFX
Figure 6.3 Execution Time of
Test_verify_addition_contents testcase in FunFX Test_add_success
Flex application
Download Automated test tools along with their compatible files
Write test cases in ruby, ActionScript for
Automated test tools
Link test cases of an application to Automated test tools
Figure 6.4 Execution Time of Test_add_success test case in FunFX
est_delete_success
Figure 6.5 Execution Time of Test_delete_success test case in FunFX
Figure 6.6 Functional testing of library application with FlexMonkey
7. Qualitative Comparison between FunfX
and FlexMonkey
7.1 Ease of Usage – FunFX
Installation of framework is tricky.Framework is distributed as a ruby gem file.
• FunFX framework has to be compiled with specific same version of flex sdk as your application before you can use framework.
• To use Framework your application needs to be compiled using FunFXAdapter.swc, automation.swc and automation_agent.swc library files.
• Your application need to include AutomationGenericEnv.xml file to use FunFX. • Test cases are written in ruby language.Testers
have to be proficient in ruby to write test cases.
Ease of Usage – Flex Monkey
• Installation is easy. Tool is distributed as a swf file which comes with a record and play interface.
• Tool does not require any special changes to your application.
• You need to copy swf file in same location where you application swf file is located and then launch the application through tool.
• Framework uses action script based language to write test cases.
• Testers have to write assertions using this language in the test cases.
7.2 Extensibility – FunFX
• Support for new type of components can be added through making entries in AutomationGenericEnv. Xml file.
• There are no instructions or user manual on how to add such a support. In absence of this it is not very easy to extended capabilities of tool.
Extensibility – FlexMonkey
• Framework supports advance flex components. • Framework comes with a Flex spy tool which is
very helpful in identify Flex components used in application.
7.3 Maintainability of automated test cases–
FunFX
• Test cases written in ruby. Different versions of test cases can be maintained in source control. • Existing test scripts has to be changed every time
during structure of application changes.
Maintainability of automated test cases– FlexMonkey
• A swf file of your test cases can be made which can be maintained in source control.
• Existing test scripts has to be changed every time during structure of application changes.
• Both the frameworks provide ability to run individual test cases in a sequence to run as test suites.
8. Qualitative Comparison between FunfX
and FlexMonkey
8.1 Execution time
Table 8.1 Test_verify_additon_contents execution time Test_Verify_addition_
contents Execution Time (seconds)
FunFX 13.594
FlexMonkey 12.122
Table 8.2 Test_add_success Execution Time Test_add_success Execution Time
(seconds)
FunFX 13.937
FlexMonkey 11.202
Table 8.3 Test_add_failure execution time Table 8.3 Test_add_failure execution time
Table 8.3 Test_add_failure_Execution _Time
Table 8.4 Test_delete_successs Execution time
Table 8.5 Test_delete_failure Execution time Test_delete_failure
Execution Time (seconds)
FunFX 10.407
FlexMonkey 9.122
Table 8.6 Total Execution Time
Total Execution Time (seconds)
FunFX 59.688
FlexMonkey 45.122
8.2 CPU Utilization
Table 8.7 CPU UtilizationCPU Utilization (%)
FunFX 50
FlexMonkey 20
Test_add_failure Execution Time (seconds)
FunFX 12.547 FlexMonkey 10.352 Test_add_failure Execution Time (seconds) FunFX 12.547 FlexMonkey 10.352 Test_delete_success Execution Time (seconds)
FunFX 9.203
FlexMonkey 9.122
Test_Verify_addition_contents Execution Time (seconds)
11 11.5 12 12.5 13 13.5 14 FunFX FlexMonkey Test_Verify_addition_cont ents Execution Time (seconds)
Test_add_success Execution Time (seconds)
0 2 4 6 8 10 12 14 16 FunFX FlexMonkey Test_add_success Execution Time (seconds)
Test_delete_success Execution Time (seconds)
9.08 9.1 9.12 9.14 9.16 9.18 9.2 9.22 FunFX FlexMonkey Test_delete_success Execution Time (seconds)
CPU Utilization (%) 0 10 20 30 40 50 60 FunFX FlexMonkey CPU Utilization (%)
8.3 Memory space
Table 8.8 Memory spaceMemory Space (K) FunFX 25,192 FlexMonkey 34,880 Memory Space (K) 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 FunFX FlexMonkey Memory Space (K)
8.4 Lines of code
Table 8.9 Lines of codeLines of Code FunFX 115 FlexMonkey 95 Lines of Code 0 20 40 60 80 100 120 140 FunFX FlexMonkey Lines of Code
9. Conclusion
Automate test cases is one of the most important aspect of Flex Application and Web Application testing. There is limited research in effectively testing of Flex Application using automated test tools. This paper focuses on Flex Application testing through Automated Test Tools. In this FunFX and FlexMonkey open source automation tools are compared on the basis of various Quantitative and Qualitative parameters. In this paper library Flex application is being implemented in FlexBuilder tool and its validation is performed with FunFX and FlexMonkey automated test tool. FunFX and FlexMonkey are compared on the basis of various Quantitative (Execution time, CPU Utilization, Memory used, lines of code) and Qualitative parameters(Ease of Usage, Extensibility, Maintainability of automated test cases, Options to run test suites).This paper concluded that FlexMonkey automated test tool is a better functional testing tool as compared to FunFX as it takes less Execution time, require few lines of code for writing test cases and require less CPU Utilization during functional testing of Library Flex Application. Although FlexMonkey requires more memory space during execution but still it is better than FunFX. This paper also concluded that from Qualitative parameters point of view FlexMonkey is better than FunFX because it is easy to install, does not require any modifications to flex application, components of flex application can easily be identified etc. Results of this study will be helpful to software development professionals to select better test automation framework for their application. In future a new tool can be implemented for testing Flex application and then it can be compared with the existing automated test tools so that the companies can select the best tool for their applications.
References
[1] Jason I. Hong, Jeffrey Heer , Sarah Waterson , and James A. Landay “WebQuilt: A Proxy-based Approach to Remote Web Usability Testing” ACM Transactions on
Information Systems, vol. 19, no. 3, 2001.
[2]“AdobeFlex”http://en.wikipedia.org/wiki/Adobe_Flex ,
Retreived January 22nd 2009
[3]“RichInternetApplication”,http://en.wikipedia.org/wiki
/Rich_internet_application, Retrieved March 9, 2009. [4] Peter Nicolai Motzfeldt , “FunFX A Framework for Functional Testing of Flex Applications” , Norwegian
University of Science and Technology Department of Computer and Information Science , 2007.
[5] Jon Rose, “Flex Testing with FlexMonkey”, InfoQ
Tracking change and innovation in the enterprise software development community, 2008.
[6] AutomatedQA,“TestComplete 5 FAQ”,
