new_admin
Snippet 21. Test Case Execution in ATP4Romulus > roma module project test run
At the end of this step, test cases are reported. The following table shows a summary of the defected revealed by ATP4Romulus:
Table 41. ATP4Romulus Results
148 http://graphml.graphdrawing.org/
Application Tests Failures Errors Success Rate Time
EUProjectManager 130 6 0 95.38% 60.66 sec
Cornelius 493 84 0 82.96% 112.465 sec
Scrooge 89 4 0 95.50% 51.365 sec
ATP4Romulus categorizes the found failures in four levels: i) INFO: information, actually not error, typically for performance data; ii) WARN: warning, not a real bug but a behaviour which can be improved; iii) ERROR: real error or bugs; iv) FATAL: when a test cases ends because an error or exception. For example, the following table summarizes the found six failures in EUProjectMannager:
Table 42. EUProjectManager Test Results
Test Type Description
TestCRUDJDBCEligibleCosts ERROR Attempt to insert null into a non‐nullable column: column: MAXIMUN_FUNDING table: ELIGIBLECOSTS
TestDomainPrimitiveEligibleCosts WARN Getter getMaximunFunding cannot return a primitive type
TestI18N WARN Default locale (admin_messages.properties)
NOT found for properties admin_messages_en.properties
TestDBPerf INFO [Performance] Num samples: 70 samples
Total time: 0.512 sec Average response time: 0.73142856 sec Max response time: 42.0 sec Min response time: 0.0 sec Throughput (num requests/ total time): 136.71875 samples/s Error: 0.0 (0.0%)
DynamicTestWebPerf INFO [Performance] Num samples: 240 samples Total time: 42.562 sec Average response time: 3.7786112 sec Max response time: 956.0 sec Min response time: 3.0 sec Throughput (num requests/ total time): 5.6388326 samples/s Error: 0.0 (0.0%) TestStartupTime INFO [Performance] Roma context starting up in 10
seconds with 0 errors and 0 failures
ATP4Romulus also generates charts for the performance in test cases TestDBPerf and DynamicTestWebPerf. First, some charts are created to gather performance information about the web navigation. In Scrooge, the generated charts are the illustrated as follows:
Figure 72. Scrooge Web Performance Charts
Finally, some charts are generated to measure the performance of the different table of the database application. The following charts are generated to as a picture of the database performance in Cornelius:
Figure 73. Cornelius Database Performance Charts
8.4. Conclusions
Once the case studies has been finished, I can draw certain conclusions and answer the research questions proposed at the beginning. Regarding RQ1 (Does the SUT accomplish its
functional requirements?) I can conclude that both Factur@ as the Romulus demonstrators has
been well implemented since not any navigation problem has been detected. The elements in charge of transitions were always found by ATP/ATP4Romulus and the state validation never detects any problem. Since the navigation of the web application under test is an implementation of the use cases and activity diagrams, I can determine that the SUT are functionally correct.
RQ2 (Does the SUT have an acceptable behaviour in terms of non‐functional requirements?) applies mainly to the case study Factur@, since the ATP4Romulus only checks for performance attributes and this assessment is only informative, i.e., there is no performance oracles in ATP4Romulus. Therefore and focusing in Factur@, concerning performance there is a warning about the expected average traffic and an error due to the concurrently repetition of request to the server. Concerning security, there has been only detected low errors, so the potential security risk is low. Regarding compatibility, usability, and accessibility, a lot of problems have been detected in the source code of the application. These problems can be solved by following some best practices guidelines when coding the HTML and CSS. All in all, I can
conclude that Factur@ has an acceptable behavior in terms of performance and security, but it can be improved in terms of compatibility, usability and accessibility.
The answer to RQ3 (Is ATP able to reveal defects in web application of different types?) is straightforward: it does. ATP has proved that it is an effective tool finding warning, errors, and potential problems. In addition, it is fully automated and data‐driven, so it can become a powerful framework to web developers and testers. I get the same conclusion for ATP4Romulus, since it is a tool fully integrated in the Roma Framework, ease of use, and it discovers several failures in different kinds of Roma applications automatically. In addition, the family of tools ATP/ATP4Romulus has proven able of assessing different web applications in the client side: based on Struts/JSP (Factur@), XHTML/CSS (EUProjectManager) AJAX/Echo2 (Cornelius), and JSP templates (Scrooge).
The last question, i.e. RQ4 (What are the advantages and disadvantages of different types of
input (UML, XML, and R&P) to ATP?) is about the way that ATP works. ATP needs any of the
accepted types of input to automate the quality control of web applications in terms of functionality, performance, security, compatibility, usability and accessibility. In this case study the three types of inputs has been used, so I can draw the pros and cons of these inputs. This information is summarized in the following table:
Table 43. Pros and Cons of XMI, XML and HTML
Input Pros Cons
XMI (UML models in NDT) 1. Analysis/design models are reused for assessment 2. Every possible path is depicted in the models 1. It is not possible to attach test data nor oracle in the models 2. Post‐automation step is mandatory XML (based on XSD Schema) 1. Every possible path can be depicted using XML files 2. Data and oracles can be attached to XML files 1. The XML files must be coded and maintained by hand HTML (Selenium R&P scripts) 1. The creation of the scripts is done using Selenium IDE against the real application. 2. Data and oracles can be attached to HTML scripts 1. Each recording is linear, therefore there is always a single path by HTML script 2. Error paths should be defined in different scripts. 3. ATP uses a subset of Selenium commands, so the recorded script should be done using only that subset
Therefore, each type of input (XMI, XML, and HTML) has advantages and disadvantages. Depending on the nature of the web application under test and the development/assessment process in which it is created, I would recommend one choice or another. The UML (XMI) models are suitable in projects where analysis and design models are created before the coding phase. The HTML scripts (R&P) can be used in general for agile development in terms of saving time‐to‐market, since the recording of scripts is a quick and easy way to establish the correct navigational path of an application including test data and oracles. Finally, XML files are a middle way between UML and R&P, since it can incorporate each possible path in the
navigation (just like in UML) and at the same time they can include different values for test data and oracles in the navigation (just like R&P).
Regarding RQ5 (Does ATP provide any advantages in testing and analysis (defects revealed,
reduction of efforts, and so on)?), I can conclude that ATP/ATP4Romulus do help to perform
quality control (testing and analysis) and find defects in a short amount of time, reducing efforts. The main advantage of the presented approach is that it is based in the navigation of web applications in which each state is a point of observation point reached via the above states and transitions. In the traditional testing/analysis approach, a tester would need to do this browsing manually. This fact involves performing the same actions in a mechanical way again and again.
With the approach proposed in this piece of research, the correct navigation is established once (pre‐automation phase) and the browsing is done automatically. Furthermore, test data for the generated test cases will be stored in Excel spread‐sheets. In order to create new tests, it is as simple as adding new rows of data in the spread‐sheets (post‐automation phase). In addition, these sheets can also contain test oracles. All in all, test data and oracles, which are crucial in efficiency of quality control, are centralized in form of Excel files, and therefore are these data is easily extensible.
Furthermore, in the Factur@ case study I am able to compare the figures of the effort in testing (depicted in Table 32) with the results of assessing this application with ATP (summarized in Table 40). The effort is much lower by using ATP (0,033 vs. 1 PM). In addition, the number of defects found by ATP is significantly higher, and the kind of assessment in much more complete (functionality, performance, security, compatibility, usability and accessibility vs. only functionality).
Besides the above, for the automatic navigation performed another series of assessment that would be very difficult to do otherwise. I refer to the assessment in terms of performance, compatibility, accessibility and usability in each of the web states.
Finally, the number of different found defects is quite significant. These defects are summarized in an automatically created report. This report is a useful artefact for testers and developers since it describes the found defects in the SUT. Therefore it could help to find the errors causing faults and failures within the web application under test, helping to avoid incidences, which is as a last resort the important part in software quality control.