Incremental Web Application Testing using Page Object

(1)

Incremental Web Application Testing using Page Object

Bing Yu Waseda University [email protected] Lei Ma University of Tokyo [email protected] Cheng Zhang University of Waterloo [email protected]

Abstract—Software testing is indispensable to assure soft-ware quality. Traditional softsoft-ware testing techniques are in-sufﬁcient for modern web applications due to their unique characteristics, such as highly dynamic generation of web pages and interactions with DOM elements. Meanwhile, existing web testing techniques are cumbersome to use in long-term maintenance and evolution because the tests are often brittle in the presence of fast changes to web pages.

In this paper, we propose an automatic test generation technique for dynamic web applications. Our technique ef-fectively decouples test code from web pages by automatically generating page objects. Based on page objects, our technique performs iterative feedback directed random test generation. Our evaluation on seven real-world web applications demon-strates that the proposed technique achieves reasonable code coverage (from 23.3% to 90.8%) and can effectively explore dynamically generated web pages (around 14.7 times more than the original static web pages), revealing hundreds of HTML issues and ﬁve real errors in the applications.

Keywords-Automated test generation; page object; web ap-plications; static analysis

I. INTRODUCTION

Web applications are gaining popularity with the develop-ment of richer user interactions on client side and numerous web services at backend. While web applications, such as Gmail and Microsoft Ofﬁce Online, are taking the place of traditional programs, their defects become more likely to cause severe consequences. Therefore, quality assurance is of profound importance to web applications.

In the era of Web 2.0, AJAX (Asynchronous JavaScript and XML) greatly improves user experience by commu-nicating asynchronously with server side. When part of a page needs to be updated, an AJAX-based application only refreshes that speciﬁc part, without reloading the whole page [2]. Such dynamic features bring challenges for testing, as web pages generated at run-time have to be tested as well. To test a web application, the registered events in the elements of its web pages need to be tested by triggering them (e. g., clicking links, typing texts) to run the code of corresponding functions. Thus, a test case usually consists of an event sequence, which is one of the many possible combinations of events. Manually writing sufﬁcient test cases to cover non-trivial event sequences tends to be costly and time-consuming. Automated web testing techniques have been proposed [7], [1], [9] to reduce human effort. However, the existing techniques do not separate the testing

logic code and the web page interface code: web elements are directly retrieved and accessed by some test code which is intertwined with the code to perform real testing. As web applications often evolve to update web page contents, ﬁx errors and add new features, test cases generated by existing techniques can be broken easily and are difﬁcult to maintain. The page object development model is proposed to reduce the cost of test generation and maintenance for web pages. Page objects introduce a middle layer between web pages and test code so that the web page elements are abstracted by the page objects and the test code only contains testing logic code. In this way, a tester can write test cases based on page objects, without concerning about the actual represen-tation of the web application. Previous work [3], [13] has demonstrated that test cases written based on page objects require less maintenance cost.

In this paper, we propose an automated incremental test case generation technique for modern web applications. To generate test cases for the application under test (AUT), we automatically create page objects to represent web pages of AUT. We perform feedback-directed random test gen-eration for these page objects to test the AUT and also explore new dynamic web page states by executing the generated test cases. Upon detecting new web page states, we create corresponding page objects for the new states and iteratively perform test case generation until the testing resource is exhausted. We implement our technique as a tool called InwertGen, and evaluate it on seven real-world web applications to demonstrate its effectiveness in terms of obtaining high code coverage, discovering new web page states, detecting violations of W3C HTML standards, and revealing error pages. The contributions of our work include: 1) To the best of our knowledge, our technique is the ﬁrst to use page objects to automatically generate test cases for dynamic web applications. The use of page objects also enables potential applications of traditional testing techniques to web applications.

2) A comprehensive evaluation on the implementation of the proposed technique. The evaluation results show that our technique achieves reasonable code coverage (from 23.3% to 90.8%), effectively explores dynam-ically generated web pages (around 14.7 times more than the original static web pages), detects hundreds of HTML issues, and reveals ﬁve real errors.

(2)

1 public classAddressBookLoginTest{

2 public voidtest_Login_success(){

3 WebDriver driver = newFirefoxDriver();

4 driver.get("http://wwww.addressbook.com/");

5 driver.findElement(By.name("user")).sendKeys("admin");

6 driver.findElement(By.name("pass")).sendKeys("pwd");

7 driver.findElement(By.name("submit")).click();

8 String pageText = driver.findElement(By.xpath

9 ("/HTML[1]/BODY[1]/DIV[1]/A[1]")).getText();

10 Assert.assertTrue(pageText.contains("Welcome addressbook"));

11 }

12

13 public voidtest_Login_fail(){

14 WebDriver driver = newFirefoxDriver();

15 driver.get("http://wwww.addressbook.com/");

16 driver.findElement(By.name("user")).sendKeys("admin");

17 driver.findElement(By.name("pass")).sendKeys("wrongPwd");

18 driver.findElement(By.name("submit")).click();

21 Assert.assertTrue(pageText.contains("User not found!"));

22 }

23 }

Figure 1: Test case without using page object pattern.

II. MOTIVATINGEXAMPLE

Figure 1 shows two test cases for the login func-tion of a web applicafunc-tion, called Address Book. A large portion of the test code is written to retrieve elements from web pages, using hard coded element names (e.g.,

findElement(By.name("user"))). Besides being redundant and cumbersome to write, such test code tends to be broken by changes to web page elements. For example, if the name of the element"pass"is changed or the element is removed, both test cases need to be ﬁxed. Figures 2 and 3 show the page object created for the login web page and the test cases using the page object, respectively. In case of changes to the page element named "pass", we only need to update few lines of code of the page object, instead of revising code scattered in many test cases. Since a web application usually has a large number of test cases and changes to web pages are common in the long-term evolution of web applications, writing test cases using page objects can reduce considerable cost of test maintenance [3]. Nevertheless, signiﬁcant effort is still required to create page objects and to write test cases to exercise possible method combinations in page objects. It can be very useful to generate page objects and test cases automatically.

III. TESTCASEGENERATION BASED ONPAGEOBJECT Figure 4 shows the overall architecture of our technique. To test a web application, a set of initial page objects are ﬁrst created and passed to the test generator. The test gener-ator (also refer to function TESTGENERATOR in Figure 5) extracts public methods from the page objects to generate test cases. A method pool and an object pool are used to store and provide necessary elements of test cases. During each iteration of test generation, our approach captures new DOM states, introduced by modiﬁcations to existing web pages or by following links to new web pages. Our approach creates new page objects for the captured DOM states (described as function PAGEOBJECTGENERATOR in

1 public class AddressBookLogin{

2 private WebDriver driver;

3 @FindBy(name="user") 4 WebElement username; 5 @FindBy(name="pass") 6 WebElement password; 7 @FindBy(name="submit") 8 WebElement login;

9 String url="http://wwww.addressbook.com/";

10

11 publicAddressBookLogin(WebDriver driver){

12 this.driver = driver;

13 PageFactory.initElements(driver,this);

14 driver.get(url); }

15 public voidsetUserName(String strUserName){

16 username.sendKeys(strUserName); }

17 public voidsetPassword(String strPassword){

18 password.sendKeys(strPassword); }

19 public voidclickLogin(){ login.click(); }

20

21 public voidloginAddressBook(String strUserName,

22 String strPassword){ 23 this.setUserName(strUserName); 24 this.setPassword(strPassword); 25 this.clickLogin(); 26 } 27 }

Figure 2: Page object of AddressBook login page.

1 public class AddressBookLoginTest{

2 public voidtest_Login_success(){

3 WebDriver driver =newFirefoxDriver();

4 newAddressBookLogin(driver).

5 loginAddressBook("admin","pwd");

8 Assert.assertTrue(pageText.contains("Welcome addressbook"));

9 }

10

11 public voidtest_Login_fail(){

12 WebDriver driver =newFirefoxDriver();

13 newAddressBookLogin(driver).

14 loginAddressBook("admin","wrongPwd");

17 Assert.assertTrue(pageText.contains("User not found!"));

18 }

19 }

Figure 3: Test case based on page object of AddressBook login page.

Figure 5) and starts a new iteration of testing on these page objects. This section describes our technique in detail.

A. Page Object Creation

Page objects form a layer of abstraction to separate test code from actual elements of web pages under test. In our technique, page objects are represented as plain Java code, where a Java class is created for each web page, and the member ﬁelds and methods correspond to the page elements and the actions on the elements, respectively. Page objects can be either written by developers or generated by

! "# $ %&!

(3)

1: functionTESTGENERATOR

Input: A initial set of page objectsO={o1, o2, . . . , on}, and visited DOM setSvisited

Output: Test suiteT and new DOM statesS={s1, s2, . . . , sm}

2: T ← ∅,S←S_visited

3: ObjectPool op← {PrimitiveValues,BrowserInitializer} 4: MethodPool mp←getPublicMethods(O)

5: while!StoppingCriteriado

6: Method m←randomSelectMethod(mp)

7: MethodSeq seq←genSeq(m,selectMethInput(m,op)) 8: Object o←executeSeq(seq) Execute method sequence

9: ifo=nullthen The execution successes

10: T←T∪seq Store generated test sequence

11: op←op∪o Store generated objects in object pool 12: end if

13: DOM dom=getCurrentDOM()

14: ifdom∈/S then Check if current DOM is visited 15: S←S∪ dom,seq

16: end if

17: end while

18: Output(T,S) Output the generated tests and DOM states 19: end function

20: functionPAGEOBJECTGENERATOR(DOMSetS) 21: PageObjectClassSet poSet← ∅

22: for alldom, seq ∈Sdo

23: WebElemSet elemSet←extractWebElems(dom) 24: Class c←createPageObjectClass(elemSet,seq) 25: poSet←c Store page object class 26: end for

27: end function

Figure 5: Algorithm of test generation and page object generation

tools automatically. Developer-written page objects normally have member methods that combine multiple elementary actions on page elements to represent a meaningful action. In comparison, automatically generated page objects usually map page elements to member ﬁelds and create member methods for all possible actions on the elements. As a result, the member methods are less organized than those written by developers. However, automatic techniques reduce manual effort to create and maintain page objects. Moreover, for some test generation techniques, ﬁner-grained per element based member methods enable more extensive testing.

Before the ﬁrst iteration of test generation, our technique automatically creates an initial set of page objects for the homepage (e. g., index.html, index.php) of the web appli-cation under test. The page object generator analyzes web pages to extract web elements and uses the elements to generate ﬁelds and public methods of page objects.

B. Feedback Directed Random Testing on Page Objects

Our technique adapts and extends the framework of feed-back directed random testing [11] to generate test cases for page objects. The function TESTGENERATOR in Figure 5 describes the high-level algorithm of test generation.

Taking the initial set of page objects as input, the test generator ﬁrst initializes themethod pooland theobject pool

(lines 3 and 4). The method pool is initialized with public methods from the input page objects. The object pool is initially populated with random primitive values, such as

constant numeric values and string literals, and predeﬁned statements to initialize browser instances. After initializa-tion, the test generator randomly selects a method under test (MUT) from the method pool (line 6). To generate a test case, the test generator searches for the input parameters of the MUT in the object pool. Then it concatenates the MUT with the found objects provided as input parameters for test generation (line 7). Each test case is executed upon generation (line 8). If passing, it is added to the set of output test casesT and its resulting object is put to the object pool as input objects for further iterations (lines 9 to 12).

The feedback directed random testing framework is in-herently iterative. The information obtained from former iterations is used to improve the test effectiveness of future iterations. Our technique generates test cases for one MUT in each iteration, and the process continues until the stopping criteria is met (e. g., a time budget is hit).

Although our technique generates test cases based on page objects, the invocation of methods of page objects eventually operates on web pages. This requires a browser instance as the execution environment. By using the Selenium Web-Driver [12], we create and initialize a browser instance to run each test case, and dispose the browser instance after test execution. This strategy avoids inadvertent dependencies between different test cases, as executing a test case usually puts the browser into a speciﬁc state and can inﬂuence the subsequent test execution. Creating a browser instance for every single test case tends to be expensive, and we are planning to optimize the current strategy in our future work. A promising solution is to cluster non-interfering test cases into groups and create a browse instance for each group.

C. Testing New Page Objects

In traditional feedback directed random testing, the set of classes under test does not change during test generation. However, it is common that new web pages come out when testing web applications, due to the their dynamic characteristics. To discover new web pages, we monitor the DOM states when executing each test method (lines 13 to 16 in Figure 5). More speciﬁcally, we build a DOM tree and perform structural comparison with existing DOM states. Our algorithm of tree comparison ignores irrelevant differences, such as tree nodes representing code comments. This treatment is critical to the efﬁciency of our technique. In addition to identifying new DOM states and storing their web page code, our technique records the method sequence that leads to each new DOM state. In test cases for the new web pages, the recorded method sequences are necessary to initialize the DOM and the browser instance. After the cur-rent test generation completes (i.e., the stopping criteria are met), the test generator outputs both the generated test cases and the newly discovered web pages with information (e. g. URLs, method sequences) to create the DOM states.

(4)

To generate test cases for new web pages, our page object generator ﬁrst performs static analysis on the recorded source code to extract essential web elements (e.g., those can be interacted by users). This information is then combined with the method sequences that create the new DOM states to generate new page objects. The newly generated page objects are used as the input to perform the next round of test generation. The algorithm of page object generation is de-scribed by function PAGEOBJECTGENERATORin Figure 5.

IV. EVALUATION

We have implemented our technique as a tool called In-wertGen, which has three major components:test generator,

DOM state comparator, and page object generator. We implement the test generator based on the random testing tool Randoop [11]. We use Selenium [12] to create browser instances for the initialization and execution of page objects. We implement the DOM state comparator as a pluggable component and different DOM comparison algorithms can be applied. The current implementation performs DOM tree equality comparison (excluding comments) to detect a new DOM state if the DOM tree changes after the test execution. The page object generator performs static analysis on new DOM states to extract web elements that users can interact with (i. e. buttons, input text box) and their locators using XPath or CSS. The extracted web elements of a DOM state are used as the fields of the corresponding page objects. The object fields are initialized by Selenium page factory using web element locators. To enable testing the new DOM states through the web elements, page object generator automatically synthesizes methods for the corresponding fields to create interfaces to be invoked by testing code.

Table I: Metrics of Subject Web Applications

Subject Server-side Client-side JavaScript LOC #Static Page # Line # Bran. # Fun. Addressbook 59.1k 24 1,134 32,240 6,611 Chess 0.4k 1 195 320 15 Form 17k 1 4,145 5,786 840 Fractal 6.5k 1 5,234 9,630 2,756 Photo Gallery 7.7k 6 1,015 5,324 686 Tinymce 57.9k 79 920 7,350 1,375 TuDu 12.5k 20 5,615 66,416 11,614 Total 161.1k 134 18,258 127,066 23,897

A. Subject Programs and Experimental Setup

To evaluateInwertGen, we use it to generate test cases for seven web applications shown in Table I. Columns 2 and 3 are the server side code size and the number of static pages. Columns 4 to 6 give the size in lines, branches and functions in the client side JavaScript code. Note that some client side JavaScript code is automatically generated and encoded as a single line of code (without formatting with white-spaces or line breaks), and the code line can contain multiple branches and functions. Consequently, for some subjects, the number

Table II: Experimental Results

Subject Iter. Code coverage (%) #New #HTML #Error Line Bran. Fun. State Issue Page Address 1 0.2 0.3 0.1 4 12 1 Book 2 0.4 0.5 0.3 16 15 3 57.4 16.6 26.7 8 46 4 74.9 18.8 34.0 29 3 5 74.9 18.8 34.0 17 5 Chess 1 86.2 59.4 80.0 77 3 0 2 86.2 59.7 86.7 92 0 3 86.2 60.9 86.7 91 0 4 86.2 60.9 86.7 59 0 5 90.8 61.6 86.7 128 0 Form 1 27.7 13.3 21.8 3 9 3 2 27.7 13.3 21.8 1 4 3 27.7 13.3 21.8 0 0 4 27.7 13.3 21.8 0 0 5 27.7 13.3 21.8 0 0 Fractal 1 26.5 8.4 16.7 508 1 0 2 33.6 13.5 18.3 284 28 3 33.6 13.5 18.3 83 3 4 33.6 13.5 18.3 54 0 5 33.6 13.5 18.3 62 0 Photo 1 10.7 0.4 1.3 3 11 0 Gallery 2 21.5 0.7 2.0 22 21 3 22.8 16.5 25.5 36 8 4 22.8 17.9 26.7 39 126 5 23.3 18.0 26.7 30 1 Tinymce 1 30.1 30.9 50.1 47 9 0 2 37.2 32.3 51.5 18 0 3 37.2 32.4 51.5 21 0 4 37.2 32.4 51.5 14 0 5 37.2 32.9 52.4 8 0 TuDu 1 5.6 0.1 0.3 40 3 1 2 9.8 1.0 1.9 14 16 3 9.8 1.0 1.9 36 0 4 13.8 4.8 8.7 70 186 5 25.5 11.0 21.6 58 59

of branches and functions of client side JavaScript code can be larger than the number of lines (see Table I).

We evaluate the effectiveness of InwertGen from four aspects. First we measure the code coverage of client side JavaScript. It is a common standard in software testing research to use code coverage to assess the quality of tests. Most dynamic features of web applications (such as user interactions and AJAX features) are implemented by JavaScript code. Higher code coverage usually indicates that the tests are more likely to detect errors in code.

As discussed in Section III-C, it is essential for a testing technique to test dynamically created web pages. Therefore, beyond code coverage, we measure the number of new states that are detected in each iteration of testing.

Malformed HTML code is a main cause of browser portability problems, and even raises vulnerability issues if not handled properly. However, most existing HTML validators only apply to static HTML code, they cannot handle dynamic HTML code generated at run-time. Inwert-Gen facilitates the validation of dynamic HTML code by triggering various event sequences during test generation.

(5)

Upon executing each method sequence, we use a W3C standard HTML validator to check issues of malformed HTML.1 We measure the detected issues of invalid DOM states that fail the HTML validation.

In addition to malformed HTML issues, the occurrences of error pages reveal errors in client side (e. g.,400 cannot resolve request error, 404 page not found error) or server side (e. g.,HTTP 500 internal server error). They are often unexpected and can lead to negative user experience. We use the non-existence of error messages as the oracle of the generated test cases to check the DOM state after executing each test sequence. Whenever an error message occurs in a DOM state, a test failure is reported, and we perform post-mortem manual analysis on the report to conﬁrm its validity. For each subject, we give 10 minutes for each iteration of test generation and perform 5 iterations to measure the code coverage, new states discovered, and the detected invalid DOM states and DOM with error messages.2 _All experiments were run on an Intel Core i7 Mac 2.5 GHz with 16 GB of RAM, running MAC OSX 10.10.2.

B. Experimental Results

Table II summarizes the experimental results. Every sub-ject has ﬁve rows of code coverage data, since we use

InwertGen to perform 5 iterations of test generation for each of them. Columns 3 to 5 show the accumulated code coverage of each iteration (merging the code coverage from previous iterations). Columns 6 and 7 show the number of new DOM states and the number of error pages detected by

InwertGenin each iteration, respectively.

From columns 3 to 5, we can see that test cases generated by InwertGen can achieve high code coverage of client-side JavaScript for two subjects, namely Chess and Address Book, while the coverage data is lower for the other subjects (Note that the function coverage of Tinymce is still higher than 50%). Overall, the code coverage can be considered quite reasonable for a fully automatic random test generator. By analyzing the uncovered code manually, we ﬁnd two major causes of low code coverage for some subjects:

• Non-registered functions. These functions are deﬁned in client-side JavaScript code, but they are neither registered in HTML code of web pages or invoked by other functions. In other words, they cannot be covered by testing on interfaces exposed by web pages. • Complex coverage condition. Covering some code

re-quires to satisfy complex conditions, including special input values and sequences of events in special orders. These conditions are inherently difﬁcult for random techniques that search for solutions in a large com-binational space. The problem can be mitigated by 1_{We use an ofﬂine W3C HTML validator http://validator.github.io.} 2_{Code coverage is measured by JSCover http://tntim96.github.io/} JSCover/.

using program analysis techniques to extract application speciﬁc knowledge to improve testing effectiveness [6]. In the context of testing web applications, testing new web pages is arguably more important than covering more JavaScript code in a limited set of web pages. From column 6, we can see that InwertGen discovers a large number of new web pages for all subjects, except Form, with the average 56.3 and the median 30 states in each testing iteration over all subjects. Considering the number of the original static web pages in each subject (i. e., data in column 3 in Table I), InwertGen is highly effective in testing the dynamic behavior of web applications. Moreover, we can observe that the code coverage of each subject in the ﬁrst iteration is often limited, but it continues to improve when more new web pages are included in subsequent iterations. By generating page object classes for newly detected states,

InwertGen can further generate test cases to improve code coverage and detect even more new DOM states. The code coverage usually stabilizes after a number of iterations.

On the seven subject web applications, InwertGen has identiﬁed 569 malformed HTML issues (see column 7 in Table II) from the dynamic web pages generated at run-time. The issues do not necessarily cause errors, but they often need to be ﬁxed to improve code quality and reduce security risks. Similar to code coverage, the number of detected HTML issues increases along with newly discovered DOM states. We observe that issues in existing web pages often remain in variants of the web pages, and new types of issues are likely to occur in completely new web pages.

Through manual analysis on web pages containing error messages, we have conﬁrmed ﬁve error pages on the studied subjects, including four 404 page not found errorsand one

CSS loading error. Such error pages are generally more severe than HTML issues, and thus require immediate ﬁxes to remove code that generate the erroneous states on client code or to provide the missing pages on server side.

C. Discussion

Application Scope. We propose to automate test gen-eration for web applications by introducing an abstraction layer. This makes the challenges in testing web application amenable to traditional automated software testing tech-niques, such as state-of-the-art testing tools Randoop [11] and GRT [6] for Java. However, a ﬁxed abstraction of web interface limits the scope of functionality that testing tools can cover. We thus propose the incremental strategy to combine test generation and page object generation to address this issue. Overall we demonstrate the effectiveness of our technique to automatically cover client-side code, explore new DOM states, and detect web application errors.

Abstraction Scope of Page Object. We use one page object to abstract the user interface of each individual web page and to automate the page object generation. In practice, a web page consists of multiple independent functional

(6)

groups, each of which contains multiple web elements for user interaction. Ideally, test cases should test combinations of events or actions on web elements in the same functional group, as cross-group combinations are generally irrelevant or invalid. Representing a web page by a single page object may induce unnecessary testing effort for web elements from different groups. A possible solution is to perform program analysis on web pages to identify functional groups, generate ﬁner-grained page objects for individual functional groups, and control the method sequence generation.

Testing Oracle for Error Detection. In InwertGen, we use general test oracles that are applicable to different web applications, including malformed HTML code (that does not pass W3C validation) and web pages with error messages. To further improve the error detection ability, application-speciﬁc knowledge, such as security rules and performance requirements, can be incorporated into the test cases generated byInwertGen.

V. RELATEDWORK

Traditional testing techniques have been studied exten-sively [8]. Representative techniques, such as feedback directed random testing and guided random testing, are proposed and implemented in Java tools Randoop [11], and GRT [6], [5], [4], respectively. Such automated tools are useful in automatically achieving high code coverage and detecting software defects. However, they do not support test generation for web applications which use web browser as execution environment and are often implemented in languages not supported by mainstream testing tools.

Automated testing of AJAX-based web applications is attracting more attention recently. Mesbah et al. [10] propose to use the state flow graph to model the user-interface states and their transitions in AJAX web applications. Their follow-up work [9] uses the inferred model to generate test cases with predefined invariants as test oracles. Artzi et al. [1] implement a JavaScript testing framework Artemis. They perform random testing to generate test inputs and use the information extracted during the execution to generate new inputs. Marchetto et al. [7] propose a state based testing approach that performs dynamic analysis to extract semantic interaction event sequences for test case generation. Different from previous work, we are the first to propose test generation for web applications using page objects, which makes it particularly useful to test evolving web applications. The use of page objects bridges the gap between traditional testing techniques and web testing.

Our work is inspired by the recent work of Leotta et al [3], [13]. They demonstrate the advantages of generating test cases based on page object pattern in reducing development effort and maintenance cost. We propose a general frame-work that leverages page objects as an abstraction layer and incrementally generates test cases, detects new DOM states and their potential problems.

VI. CONCLUSION ANDFUTUREWORK

In this paper, we present a fully automatic test generation technique of web applications based on page object. We adopt feedback directed random testing to generate test cases and incrementally explore more states. We show that page object based approach enables traditional testing technique to tackle challenges in testing AJAX-based web applications. We demonstrate the effectiveness of our approach on seven real-world web applications. To the best of our knowledge, we are the ﬁrst to automatically generate test cases for web applications based on page objects. Our future work includes: (1) improving test generation performance by removing irrelevant method sequence combinations, and (2) studying the inﬂuence of page object abstraction scope on testing effectiveness.

REFERENCES

[1] S. Artzi, J. Dolby, S. H. Jensen, A. Møller, and F. Tip, “A framework for automated testing of javascript web ap-plications,” in Proc. of the 33rd Int. Conf. on Softw. Engg. (ICSE’11), 2011, pp. 571–580.

[2] A. T. Holdener, III, Ajax: The Deﬁnitive Guide. O’Reilly, 2008.

[3] M. Leotta, D. Clerissi, F. Ricca, and C. Spadaro, “Improving test suites maintainability with the page object pattern: An industrial case study,” in6th Int. Conf. on Softw. Test., Verif. and Valid. Workshops (ICSTW’13), 2013, pp. 108–113. [4] L. Ma, C. Artho, C. Zhang, H. Sato, J. Gmeiner, and

R. Rudolf, “GRT: An automated test generator using orches-trated program analysis,” in IEEE/ACM Int. Conf. on Auto. Softw. Engg. (ASE’15), 2015.

[5] ——, “GRT: Program-analysis-guided random testing,” in

IEEE/ACM Int. Conf. on Auto. Softw. Engg. (ASE’15), 2015. [6] L. Ma, C. Artho, C. Zhang, H. Sato, M. Hagiya, Y. Tanabe, and M. Yamamoto, “GRT at the SBST 2015 tool competi-tion,” inThe 8th Int. Workshop on Search-Based Softw. Test. (SBST’15), 2015, pp. 48–51.

[7] A. Marchetto, P. Tonella, and F. Ricca, “State-based testing of ajax web applications,” inProc. of the 2008 Int. Conf. on Softw. Test., Verif., and Valid. (ICST’08), 2008, pp. 121–130. [8] P. McMinn, “Search-based software test data generation: A survey: Research articles,”Softw. Test. Verif. Reliab., vol. 14, no. 2, 2004.

[9] A. Mesbah, A. van Deursen, and D. Roest, “Invariant-based automatic testing of modern web applications,”IEEE Trans. on Softw. Engg., vol. 38, no. 1, pp. 35–53, 2012.

[10] A. Mesbah, E. Bozdag, and A. van Deursen, “Crawling ajax by inferring user interface state changes,” inProc of the Int. Conf. on Web Engg. (ICWE’08), 2008, pp. 122–134. [11] C. Pacheco, S. K. Lahiri, M. D. Ernst, and T. Ball,

“Feedback-directed random test generation,” inProc of the 29th Int. Conf. on Softw. Engg. (ICSE’07), 2007, pp. 75–84.

[12] Selenium, “https://selenium.openqa.org.”

[13] A. Stocco, M. Leotta, and P. Ricca, F.and Tonella, “Why creating web page objects manually if it can be done au-tomatically?” in10th Int. Workshop on Auto. of Softw. Test (AST’15), 2015, pp. 70–74.