Crawljax and Artemis comparison

Running Crawljax targetting the Contacts Agenda application produced the state machine depicted in Figure 7.15. Each state in the Figure 7.15 is com- posed of a name and a screenshot of the page in that current state. Although the last state discovered is called state29 the state machine is composed by 13 states. We can click in each screenshot and the information respective to that state is displayed. This includes the number o interaction elements tested, their XPATH and the state we reach when we trigger those elements.

Figure 7.14: Contacts Agenda abstract state machine

In order to obtain this state machine, we had to add the correct authenti- cation values to the Crawljax configuration. However, a immediate difference from our tool is that if we add specific input values for fields, no other values are tested on those fields. That means Crawljax misses the two error states that can be reached from the Login frame. Also the Login frame information showed that 16 elements were tested in the initial page, and only one triggered a change which was the Ok button. This showed that Crawljax tried to trigger all the buttons including those that were invisible since they belong to other frames. A Crawljax run without the authentication information specified re- turn two states, the Login frame and the Error frame. The error frame that is found depends the configuration option to use random values. If we choose to use random values, Crawljax never tests the empty cases so it only discovers the "login failed" error. If we choose not to use random values, Crawljax discovers only the "no text entered" error.

The MainForm frame analysis also presents several differences in compari- son to our framework. The radio-buttons are not tested by Crawljax. Similar to our framework, the elements on the list are selected, therefore Crawljax is able to trigger both the Remove and the Edit buttons. Nevertheless, the edit button transition does not lead to the add frame and the remove button is a transition from a list with 2 elements to a list with 4 elements. Therefore, something

7.1. Contacts Agenda 119

Figure 7.15: Crawljax contacts agenda state machine

was wrong with the analysis and this state was incorrectly mistaken with some other state. This problem was further analyzed in the Class Manager case study in Section7.2.3.

The Find frame analysis discovered events in all the buttons. The search button led to the "no contact found" Error frame. Since the empty field was not tested and a correct value was not entered the other two states were not discovered. A strange thing happens with the analysis of the show button. This is a button that was referred before as not containing any behavior whatsoever associated with. Crawljax however detects a new state when triggering this button.

A more thorough analysis of the state machine identified why this problem happens. At some point in the crawl, Crawljax is able to add more contacts to the contact list. When it tries to return to the Find frame (state2 in Crawljax state machine) it in fact reaches a different state (with more contacts in the list without noticing it. It is only when it tests the Show button that it analyzes the state and considers that a new state was found (state 19). State 19 corresponds to a state exactly the same as state2 the only difference being that now the contact list (which is inf fact invisible) now has more elements. When Crawljax returned to the Find it should have discovered that it now was not state 2

but a different state since it was considering invisible elements. The same situation was what we identified previously in the Mainform frame and happens throughout the entire state machine.

Since we used the random values option which makes Crawljax fill all the input fields, the Edit frame analysis found that triggering the Ok button would generate a new state, which corresponds to the Mainform state with one more contact. Therefore, it did not find the error state in the Edit frame, since the fields were always all filled.

The Artemis tool has three different run modes. The default uses a feedback directed approach and generates execution traces of the application. The man- ual mode is used for manual testing and therefore will not be used for analysis. The concolic mode performs a concolic analysis of form validated code and generates a tree with the concolic results.

An analysis of the path traces Artemis generates shows a problem which is that the traces generated are triggering elements that are not visible on the page. Since there is no option to use predefined values in fields, the application is never able to pass the login frame. Nevertheless, most iterations are using these elements which are invisible on the page. This is very problematic since for example a run with 100 iterations only contained three iterations where the Ok button present in the login frame was triggered. Moreover, in none of those three iterations the Ok button from the error frame that appears afterwards was triggered. This means that if we ran all the 100 iterations we would only have found 2 states and only one of both transitions between them.

An excerpt of the generated Artemis tree from its concolic mode is depicted in Figure 7.16. In terms of the concolic analysis of this application Artemis failed to discern both branches in the only condition it found on the source code. That condition corresponds to a function, in the Mainform frame, that checks if a contact was selected on the list and enables the edit and remove buttons accordingly. One of the reasons the concolic solver failed might be that the tool is not able to reach a state where the contact list is displayed since it cannot solve the authentication. It is peculiar that all the other branches present in the source code were not discovered and analyzed.