Dissertations, Theses, and Masters Projects Theses, Dissertations, & Master Projects
2015
Enhancing Bug Reports for Mobile Apps
Enhancing Bug Reports for Mobile Apps
Kevin Patrick MoranCollege of William & Mary - Arts & Sciences
Follow this and additional works at: https://scholarworks.wm.edu/etd
Part of the Computer Sciences Commons
Recommended Citation Recommended Citation
Moran, Kevin Patrick, "Enhancing Bug Reports for Mobile Apps" (2015). Dissertations, Theses, and Masters Projects. Paper 1539626966.
https://dx.doi.org/doi:10.21220/s2-sps5-3f59
This Thesis is brought to you for free and open access by the Theses, Dissertations, & Master Projects at W&M ScholarWorks. It has been accepted for inclusion in Dissertations, Theses, and Masters Projects by an authorized administrator of W&M ScholarWorks. For more information, please contact [email protected].
E nhancing Bug R eports for Mobile Apps
Kevin P atrick M oran
M aitland, FL
Bachelor of Science, College of the Holy Cross, 2013
A Thesis presented to th e G rad uate Faculty
of th e College of W illiam and M ary in C andidacy for th e Degree of M aster of Science
D epartm ent of C om puter Science
The College of W illiam and M ary A ugust 2015
This D issertation is su b m itted in p artial fulfillment of th e requirem ents for the degree of
M aster of Science
Kevin Patrick Moran
A pproved by th e C om m ittee, A ugust 2015
Committee CJiair
Associate Professor Denys Poshyvanyk, Computer Science The College of William a n d ^ a r y
Associate Professor Gang Zhou, Computer Science The College of William and Mary
Professor Xu( Liu, Computer Science The College of William and Mary
ABSTRACT
Software bug reports are artifacts leveraged by users, testers or developers in order to docum ent defects in software applications or projects. Bug reports are typically in sta n tiated when a problem (e.g. unexpected or unw anted behavior) is observed in a software application and are created through a m echanism known as an issue-tracker. T he prim ary purpose of issue-tracking system s are to allow reporters to construct a detailed bug rep o rt th a t accurately describes a software problem so th a t it can be faithfully reproduced and subsequently fixed. In general,
issue-trackers employ n a tu ra l language descriptions of problems, and sometimes sup p o rt the addition of augm ented inform ation such as the uploading of
screenshots. These system s have been used, with few exceptions, m ostly
unchanged to docum ent and form ulate fixes for bugs across m any different types of software projects. However, some applications th a t rely on complex user
interactions are not well suited to these types of trad itio n al systems, as they are not able to cap tu re intricacies of com plicated application event-flows. The most prom inent exam ple of a classification of applications th a t exhibits this type of behavior, and for which issue-tracking system s could be improved, is the rapidly growing category of mobile apps which rely on touch-based gestures for navigation on sm artphones and tablet devices.
T he m odern software development landscape has seen a shift in focus tow ard mobile applications as “sm art” devices near ubiquitous adoption. Due to this trend, the com plexity of mobile applications has been increasing, making
developm ent and m aintenance particularly challenging. However, it is clear th a t current bug tracking system s are not able effectively support construction of reports w ith actionable inform ation th a t will directly lead to a b u g ’s resolution. To address th e need for an improved reporting system, we introduce a novel
solution, called FUSION, th a t helps users auto-com plete reproduction steps in bug reports for mobile apps. FUSION links inform ation, th a t users provide, to
program artifacts ex tracted through static and dynam ic analysis perform ed before testing or release. The approach th a t FUSION employs is generalizable to other current mobile software platform s, and constitutes a new m ethod by which off-device bug reporting can be conducted for mobile software projects. We evaluate FUSION by conducting a study th a t quantitatively and qualitatively m easures the user experience of th e system for b o th reporting and reproducing bugs, as well as the quality of the bug rep o rts it produces. In a study involving 28 p articip an ts we apply FUSION to support the m aintenance tasks of reporting and reproducing defects on 15 real-world bugs found in 14 open source A ndroid apps. O ur results dem onstrate th a t FUSION allows for more reliable reproduction of bugs from rep o rts by aiding users in reporting more detailed application-specific inform ation com pared to trad itio nal bug tracking systems.
Acknowledgments iii
D edication iv
List of Tables v
List of Figures vii
1 Introduction 2
1.1 M o tiv a tio n ... 2
1.2 C ontributions ... 4
2 R elated Works 6 2.1 Existing Bug R eporting S y s te m s ... 6
2.2 Bug R eporting Studies ... 7
2.3 In-Field Failure R eproduction ... 9
2.4 Bug and E rror R eporting R e s e a rc h ... 10
3 The FUSION A pproach 12 3.1 Analysis P h a s e ... 12
3.1.1 S tatic Analysis ( P r im e r ) ... 14
3.1.2 Dynam ic Analysis ( E n g i n e ) ... 14
3.2 R eport G eneration P h a s e ... 16
3.2.1 R eport G enerator User I n t e r f a c e ...16
3.2.2 A uto-com pleting Bug R eproduction S t e p s ... 18
3.2.3 R eport G enerator A uto-C om pletion E n g i n e ...20
3.2.4 Handling FU SIO N ’S A pplication Model G a p s ... 21
3.2.5 R eport S t r u c t u r e ... 22
4 Design of Em pirical Studies 25 4.1 Study C ontext: Bug R eports Used in the S tu d i e s ... 26
4.2 Study 1: R eporting Bugs w ith F U S I O N ... 28
4.3 Study 2: R eproducibility of Bug R e p o r t s ...31
5 Em pirical Study R esults 34 5.1 Study 1 (Bug R eport C reation) R e s u l t s ... 34
5.1.1 Study 1 Bug C reation Tim e R e s u l t s ...34
5.1.2 Study 1 UX & U P R esults ...37
5.2 Study 2 (Bug R eport R eproducibility) R e s u l t s ... 39
5.2.1 Study 2 UX and U P R esults ... 39
5.2.2 Study 2 Bug R eproduction R e s u l t s ... 41
6 L im itations and T h reats to V alidity 45 6.1 Lim itations of th e FUSION a p p r o a c h ...45
6.2 T h reats to the Validity of th e Em pirical S t u d y ...46
7 Conclusion 48 A Instructions for User Study P articip an ts 49 A .l Study 1 In s tru c tio n s ... 49
A .2 Study 2 In s tru c tio n s ... 51
T his thesis and th e work perform ed tow ards its w riting would not have been possible w ith o u t th e su p p o rt and guidance of several individuals. F irst and forem ost, I would like to th a n k m y advisor. Dr. Denys Poshyvanyk for tak in g th e tim e to m entor and su p p o rt a physics m ajo r w ith an initially lim ited background in com puter science. He has tirelessly su p p o rte d this work and set a great exam ple illu stratin g how to conduct m eaningful research in th e field of softw are engineering w ith integrity, vision, passion and thoroughness. It goes w ith o u t saying th a t w ith o u t him th is work would not have been possible, and I look forw ard to continuing to grow as a researcher under his m entorship as I pursue my PhD .
N ext, I would like to th a n k all of th e m em bers of th e SEM ER U group for th eir su p p o rt in th is p ro ject and o th ers as I began my career as a g ra d u a te stu d e n t. It is not often th a t you find individuals who are willing to use th eir tim e to explain p ro p er research m ethods and code conventions to a new research s tu d e n t and I am extrem ely grateful for th eir guidance. In p a rticu la r. I would th a n k to M ario Linares-V asquez, Carlos
B ernal-C ardenas. and C h risto p h er Vendome. M ario, your research expertise and co nstructive feedback c o n trib u te d im m ensely to this p ro ject and to my education as a softw are engineer and research p ractitio n er. Carlos, your coding expertise and patience in teaching me new techniques has been invaluable to m e and this project. Chris. I am grateful for your su p p o rt and friendship since I s ta rte d at W illiam & Mary. I cannot possibly give you all enough credit.
I th a n k my p a re n ts B rian and E liz a b e th for th eir continued su p p o rt of my academ ic pu rsu its, especially during difficult tim es. I look to you b o th for inspiration and
m otivation in th e way you have raised my b ro th e rs and me and in th e way you live your lives. I would also like to th a n k m y g ran d p aren ts, Tom and Jeanne M oran, and M ary and P au l Lavin. for always encouraging me in m y studies and offering crucial advice w hen needed.
M ost Im p o rtan tly . I would like to th a n k m y fiancee Em ily for her unw avering su p p o rt, especially during th e late nights and deadline work m arath o n s. Your c h aracter and accom plishm ents inspire all th a t I do, and my full ap p reciatio n for having you in my life can not be ad eq u ately p u t into words.
In loving m em ory of Thom as P. M oran, M ary E. Lavin, and Paul B. Lavin; your lives inspired m any people - myself included - to accomplish great things.
4.1 S u m m a r y o f th e b u g r e p o r ts u se d for th e e m p ir ic a l stu d ies: G D E = Gui Display Error, C = Crash, DIC = D ata In p u t/C a lc u latio n Error, NE = N avigation Error; (Links to the original bug rep o rts can be found at our online a p p e n d ix ) ...27 4.2 S tu d y 1 P a r tic ip a n t D e sig n M a trix : This table shows the indicies
of bug rep o rts assigned to p articipan ts during Study 1. The bugs corresponding to the index num bers can be found in Table 4.1 . . . . 28 4.3 S tu d y 1 U se r P r e fe r e n c e Q u e stio n s: Q uestions used during Study
1 to evaluate User Preferences regarding FU SIO N ...30 4.4 S tu d y 1 U se r E x p e r ie n c e Q u estio n s: Questions used during Study
1 to evaluate the User Experience of FU SIO N ... 30 4.5 P a r tic ip a n t P r o g r a m m in g E x p e r ie n c e Q u e stio n s Q uestions used
to evaluate the relative program m ing experience of p articip an ts in b o th em pirical studies...30 4.6 S tu d y 2 U se r P r e fe r e n c e Q u estio n s: Q uestions used during Study
1 to evaluate User Preferences regarding FU SIO N ...31 4.7 S tu d y 2 U se r E x p e r ie n c e Q u estio n s: Questions used during Study
1 to evaluate the User Experience of FU SIO N ... 31 4.8 S tu d y 2 P a r tic ip a n t D e s ig n M atrix: This table shows th e indicies
of bug reports assigned to particip ants during Study 2. T he bugs corresponding to the index num bers can be found in Table 4.1 . . . . 33
5.1 C r e a tio n T im e S ta tis tic s for F U S IO N B u g s: All of the times rep o rted in this tab le are in the form at (m:ss); an (*) next to the tim e indicates th a t FUSION was able to cap ture all of th e steps for reproduction w ith th e autocom pletion engine and a replayable can be gen erated... 35 5.2 C r e a tio n T im e S ta tis tic s for G C IT B u g s: All of th e times
reported in this table are in th e form at (m:ss) ...35 5.3 A v e r a g e B u g R e p o r t R e p r o d u c tio n T im e: Average reproduction
tim e results for each type of bug report evaluated... 40 5.4 N o n R e p r o d u c ib le B u g R e p o r ts: N um ber of bugs th a t could not
be reproduced per Bug R eport T yp e...43 5.5 B u g R e p o r t Q u a lity S ta tistic s: G C IT = Google Code Issue Tracker,
NR = # Instances not reproducible, Time = Average tim e to repro duce, E = bug rep o rt created by experienced particip an t, I = bug rep o rt created by inexperienced p a r t i c i p a n t ... 44
2.1 T h e G o o g le C o d e Issu e Tracker: An example of a popular Issue
Tracking System w ith sem i-structured fields for reporters to construct
issue tickets. Image taken from [? ] 8
2.2 B u g z illa Issu e Tracker: An exam ple of a popular Issue Tracking
System w ith separate fields th a t prom pts users to enter reproduction steps and ex p ec te d /ac tu a l results of the steps. Image taken from [5] . 11 2.3 M a n tis Issu e Tracker: An example of a popular Issue Tracking
System containing a large am ount of contextual inform ation. Image taken from [1 2 ]... 11
3.1 O v e rv iew o f F U S IO N W orkflow : First static and dynam ic app
analysis is perform ed on th e targ et app, then the A uto-C om pletion Engine uses th e inform ation gleaned by the analyses in order to help th e user auto-com plete reproduction steps of a bug for the targ et app. 13 3.2 F U S IO N R e p o r te r In terfa ce: This figure shows th e FUSION re
po rter web interface, w ith an area for contextual inform ation and a n a tu ra l language description of the bug at the top of the page, an area for reporting reproduction steps, and an area showing th e history of the steps entered w ith an option to view or delete p ast step s...17 3.3 A u to -C o m p le tio n D r o p d o w n M en u s: This figure shows examples
of the auto-com pletion dropdow n menus th a t present reporters w ith possible choices during course of creating a re p o rt... 19
3.4 D e c is io n T ree U tiliz e d b y A u to -C o m p le tio n E n gin e: This fig ure outlines the decision tree utilized by FU SIO N ’S autocom pletion engine which helps to predict the possible com ponents th a t a user can interact w ith at a specific place in the event flow of an application. . . 22 3.5 R e la tiv e L o c a tio n E n u m e r a tio n a n d E x a m p le A u g m e n te d S creen -
sh o t: This figure shows FU SIO N ’S enum eration for th e relative loca
tion of G U I-com ponents on the screen and an exam ple of an aug m ented full screenshot... 23 3.6 E x a m p le F U S IO N B u g R e p o r t: This rep o rt shows th e three m a
jor categories of inform ation contained w ithin FUSION bug repo rts 1) C ontextual inform ation and n a tu ra l language description of th e bug. 2) A detailed set of reproduction steps including G U I-com ponent spe cific screenshots. 3) A list of fullscreen screenshots w ith th e GUI- com ponent acted upon in each step highlighted...24
5.1 S tu d y 1 U se r E x p e r ie n c e Q u e stio n R e su lts: Answers to the U X -related questions for Study 1 (Bug R eport C reation) ... 36 5.2 S tu d y 2 U se r E x p e r ie n c e Q u e stio n R e su lts: Answers to the
U X -related questions for S tudy 2 (Bug R eport R e p r o d u c tio n )...39 5.3 S tu d y 2 B u g R e p o r t R e p r o d u c tio n R e su lts: R esults for the
num ber of bug reports reproduced and th e average tim e taken to re produce each b u g ...42
2
C h a p te r 1
I n tr o d u c tio n
1.1
M o tiv a tio n
S m artp h o n es and m obile com puting have skyrocketed in p o p u larity in recent years, and ad o p tio n has reached n ear-ubiquitous levels w ith over 2.7 billion active sm artp h o n e users in 2014 [42]. An increased d em and for high-quality, robust m obile applications is being driven by a growing user base th a t perform s an increasing num ber of com puting tasks on “s m a rt” devices. D ue to th is dem and, th e com plexity of m obile applications has been increasing, m aking developm ent and m ain ten an ce challenging. T he intense com petition present in m obile application m arketplaces like Google P lay and th e A pple A pp Store, m eans th a t if an app is not perform ing as expected, due to bugs or lack of desired features. 48% of users are less likely to use th e app again and will ab andon it for a n o th e r one w ith sim ilar fu n ctionality [13].
Softw are m aintenance activities are know n to be generally expensive and challenging [89]. One of th e m ost im p o rta n t m ain tenance tasks is bug rep o rt resolution. However, cu rren t bug tracking system s such as B ugzilla [5]. M antis [12]. th e Google Code Issue Tracker [9]. th e G itH ub Issue Tracker [8]. and com m ercial solutions such as JIR A [11] rely m ostly on u n stru c tu re d n a tu ra l language bug descriptions. T hese descriptions can be augm ented w ith hies uploaded by th e rep o rters (e.g.. screenshots). As an im p o rta n t com
po n en t of bug rep o rts, rep ro d u ctio n steps are expected to be rep o rted in a stru c tu re d and descriptive way, b u t th e q u ality of description m ostly depends on th e re p o rte r’s experience and a ttitu d e tow ards providing enough inform ation. Therefore, th e rep o rtin g process can be cum bersom e, and th e ad d itio n al effort m eans th a t m any users are unlikely to enhance th e ir rep o rts w ith e x tra inform ation [26. 37. 25. 18].
A past survey of open source developers conducted by K oru et al. has shown th a t only « 50% of developers believe bug rep o rts are always com plete [61]. P revious studies have also shown th a t th e inform ation m ost useful to developers is often th e m ost difficult for rep o rters to provide and th a t th e lack of th is inform ation is a m ajo r reason behind non-reproducible bug rep o rts [41. 24]. Difficulty providing such inform ation, especially rep ro d u ctio n steps, is com pounded in th e context of m obile applications due to th eir com plex event-driven and G U I-based n a tu re . F urtherm ore, m any bug rep o rts are created from te x tu a l descriptions of problem s in user reviews. A ccording to a recent s tu d y by C hen et al. [31], only a reduced set of user reviews can be considered useful a n d /o r inform ative. Also, unlike issue rep o rts and developm ent em ails, reviews do not refer to details of th e app im plem entation.
T he above issues point to a m ore prom inent problem for bug tracking system s in general: th e lexical gap th a t norm ally exists betw een bug rep o rters (e.g.. testers, b e ta users) and developers. R ep o rters typically only have functional knowledge of an app. even if th ey have developm ent experience them selves, w hereas th e developers working on an app ten d to have in tim a te code level knowledge. In fact, a recent stu d y conducted by Huo et al. co rro b o rates th e existence of this knowledge gap as th ey found there is a difference betw een th e way ex perts and non-experts w rite bug rep o rts as m easured by te x tu a l sim ilarity m etrics [49]. W hen a developer reads and a tte m p ts to com prehend (or reproduce) a bug rep o rt, she has to bridge this gap. reasoning ab o u t th e code level problem s from th e high-level functional description in th e bug rep o rt. If th e lexical gap is too wide th e developer m ay not be able to reproduce a n d /o r subsequently resolve th e bug rep o rt.
C H A P T E R 1. IN T R O D U C T IO N 4
1.2 C o n tr ib u tio n s
To address th is fu n dam ental problem of m aking bug rep o rts m ore useful (and repro ducible) for developers, we in tro d u ce a novel approach, which we call FU SIO N , th a t relies on a novel Analyze —> Generate p aradigm to enable th e auto-com pletion of A ndroid bug re p o rts in order to provide m ore actionable inform ation to developers. In th e context of th is work, we define auto-com pletion as suggesting relevant actions, screen-shots, and im ages of specific G U I-com ponents to th e user in order to facilitate rep o rtin g th e steps for reproducing a bug. F U SIO N first uses fully a u to m a te d s ta tic and dynam ic analysis techniques to g a th e r screen-shots and o th er relevant inform ation a b o u t an app before it is released for testing. R ep o rters th e n in teract w ith th e w eb-based rep o rt generator using th e auto-com pletion features in order to provide th e bug rep ro d u c tio n steps. By linking th e inform ation provided by th e user w ith features e x tra cte d th ro u g h sta tic and dynam ic analyses, FU SIO N presents an au gm ented bug report to th e developer th a t contains im m ediately actionable inform ation w ith well-defined steps to reproduce a bug. T he work p resented in th is thesis represents an extension of a published conference p a p e r at F S E ’15
[69].
We evaluate FU SIO N in a s tu d y com paring bug rep o rts s u b m itte d using our system to th e bug rep o rts produced using Google Code Issue Tracker, involving 28 p articip an ts, rep o rtin g bugs for 15 real-w orld failures stem m ing from 14 open source A ndroid apps.
O ur p ap er m akes th e following n otew orthy contributions: We evaluate FU SIO N in a s tu d y com paring bug rep o rts s u b m itte d using our system to bug rep o rts produced using Google Code Issue Tracker, involving 28 p a rticip a n ts rep o rtin g bugs for 15 real-w orld failures stem m ing from 14 open source A ndroid apps.
O ur pap er m akes th e following notew orthy contributions:
1. We design and im plem ent a novel approach for auto-com pleting and augm enting A ndroid bug rep o rts, called FU SIO N , which leverages sta tic and dynam ic analyses, and provides actionable inform ation to developers. T he tool facilitates th e reporting,
reproduction and subsequent resolution of Android bugs. The program analysis techniques of the apps can be run on both physical devices and emulators;
2. We design and carry out a comprehensive user study to evaluate the user experience
of our approach and the quality of bug reports generated using FUSION compared to the Google Code Issue Tracker. The results of this study demonstrate that FUSION enables developers to submit bug reports that are more likely to be reproducible compared to reports written entirely in natural language;
3. We make FUSION and all the data from the experiments available for researchers [57] in hope th at this work spurs new research related to improving the quality of bug reports and bug reporting systems.
6
C h a p te r 2
R e la te d W orks
B ug and error rep o rtin g has been an active area of research in th e software engineering com m unity. However, little work has been conducted to im prove th e lack of s tru c tu re in th e rep o rtin g m echanism for entering rep ro d u ctio n steps, and adding corresponding s u p p o rt in bug tracking system s. Therefore, in this section, we briefly survey th e features of cu rren t bug rep o rtin g system s and th e studies th a t m o tivated this work. We outline th e current types of inform ation th a t bug rep o rtin g system s a tte m p t to elicit from users a nd explain how FU SIO N im proves upon th is m ethod for collecting inform ation. T h e n we differentiate our work from approaches for reproducing in-field failures and explain how our work com plim ents existing research on bug reporting.
2.1
E x is tin g B u g R e p o r t in g S y s te m s
T h e purpose of a bug rep o rtin g system , som etim es referred to as an issue tracker, is twofold: F irst, such system s m ust provide a coherent and easy to use m echanism for re p o rte rs to accurately and com pletely describe a software defect or feature a d d itio n /e n h a n ce m en t. Second, th e y m ust organize this inform ation and present it to th e developer in a m eaning ful fashion. M ost cu rren t bug rep o rtin g system s rely upon u n stru c tu re d n a tu ra l language descriptions in th eir reports. However, some system s do offer m ore functionality. For in stance. th e Google C ode Issue Tracker (G C IT ) [9] (See F igure 2.1) offers a sem i-stru ctu red
area w here rep o rters can en ter rep ro d u ctio n steps and expected in p u t/o u tp u t in n a tu ra l language form (i.e.. th e online form asks: ’’W h at steps will reproduce th e problem ?” ). N early all current issue trackers offer s tru c tu re d fields to en ter inform ation such as tags, severity level, assignee, fix tim e, and p ro d u c t/p ro g ra m specifications. Some w eb-based bug rep o rtin g system s (e.g. Bugzilla [5] (See Figure 2.2). J ira [11]. M antis [12] (See Figure 2.3). U serSnap [15], BugD igger [87]) facilitate rep orters including screenshots. O ne com m onality th a t m ost of these rep o rtin g system s share is th a t rep o rters and developers share th e sam e view of th e bug rep o rt re p o rt. T h a t is, there is no differentiation betw een th e inform ation and view of the issue re p o rt th a t rep o rter sees and th a t which th e developer sees. T his is yet a n o th e r exam ple of how current issue tracking system s do n ot effectively h andle th e lexical gap th a t exists betw een developers and te s te rs /re p o rte rs . Ideally, th e user-facing rep o rtin g m echanism should be fam iliar and easy to use, and th e developer rep o rt should be detailed and su itab le for th e m aintenance ta sk at h an d (e.g. feature a d d itio n /e n h a n ce m en t, bug fixing). In oth er words, th e issue tracker system , n o t th e
d e v e lo p e r should bridge th e lexical knowledge gap th a t typically exists betw een rep o rters
and developers. T his is precisely w h at FU SIO N accom plishes. By leveraging inform ation gleaned from program analysis. FU SIO N is able to present to th e rep o rter th e fam iliar interface of a m obile application GUI. and suggest the steps to reproduce a bug.
2 .2
B u g R e p o r t in g S tu d ie s
T h e problem facing m any current bug rep o rtin g system s is th a t typical n a tu ra l language rep o rts c a p tu re a coarse grained level of detail th a t m akes developer reasoning about defects difficult. T his highlights th e underlying task th a t bug rep o rtin g system m ust accom plish: bridging the lexical knowledge gap between typical reporters of a bug and the
developers that m ust resolve the bugs. In order for an issue track in g system to effectively
accom plish this task, it m ust facilitate th e e n try of certain types of crucial inform ation th a t developers find useful. Previous studies on bug report quality and developer inform ation
C H A P T E R 2. R E L A T E D W O R K S 8
P ro je c t H o m e Wiki ! Issues S o u rc e
N ew is s u e S e a rc h Open issues for S e a rc h A d v a n c e d s e a r c h S e a rc h tip s S u b sc rip tio n s
Template: User defect report
Tip: P le ase search for an existing issu e before reporting a problem a s a new issue.
Summary: Enter one-line sum m ary
Description: w p a t s t e p s will re p ro d u c e th e p ro b lem ? S te p 1.
S te p 2. S te p 3.
Rem ember: This report will
be publicly visible. So, d o n t Include p assw ords or other confidential information. W hat is th e e x p e c te d o u tp u t? W hat d o you s e e in s te a d ?
W hat b ro w se r (or g it/h g /s v n client) a re you u s in g ? On w h a t o p e ra tin g s y s te m ?
P le a s e p ro v id e a n y ad d itio n al inform ation below .
^ A ttach a file m Notify m e of is s u e c h a n g e s , if e n a b le d in s e ttin g s
Figure 2.1: The G oogle C ode Issue Tracker: An example of a popular Issue Tracking System
with semi-structured fields for reporters to construct issue tickets. Image taken from [? ] needs highlight several factors th a t can im pact th e quality of bug rep o rts [28, 41, 24]:
• O th e r th a n “In tc rb u g dependencies” (i.e., a situ a tio n w here a bug was fixed in a previous p a tc h ), insufficient in fo rm a tio n in bug rep o rts is one of th e leading causes of non-reproducible bug rep o rts [41];
• Developers consider (i) steps to reproduce, (ii) stack traces, and (in) test cases/scenarios as th e m ost helpful sources of inform ation in a bug re p o rt [24];
• Inform ation needs arc g rea test early in a b u g ’s life cycle, therefore, a way to easily add th e above features is im p o rta n t du rin g bug re p o rt creation [28].
Using these issues as m otivation, we developed FU SIO N w ith two m ajo r goals in m ind: (i) provide bug reports to developers w ith im m ediately actionable knowledge (reliable
reproduction steps) and (ii) facilitate reporting by providing this inform ation through an
auto-com pletion m echanism .
It is w orth n o ting th a t one previous s tu d y conducted by B h a tta c h a ry a et. al. [27] concluded th a t m ost A ndroid bug re p o rts for open source apps arc of high-quality, however in th eir s tu d y only « 46% of bug rep o rt contained steps to reproduce, and and even lesser am ount ( » 20%) contained ad d itio n al inform ation (e.g. bug-triggering in p u t or even an
app version). T herefore, th ere is clearly room for im provem ent in term s of th e ty p e of inform ation th a t is contained w ithin open source A ndroid bug reports. By helping auto- com plete th e rep ro d u ctio n steps using guided suggestions for re p o rte r GUI actions and corresponding com ponents, we facilitate th e rep o rter providing th is inform ation in th e bug rep o rt which is b o th useful from a developer’s perspective, and typically difficult to provide from a re p o rte r’s perspective.
2 .3
I n -F ie ld F a ilu re R e p r o d u c tio n
A bo d y of work known as in-held failure rep roduction [23. 52. 97. 33. 51. 19. 58. 30] shares sim ilar goals w ith our approach. T hese techniques collect ru n -tim e inform ation (e.g.. execution traces) from in stru m e n te d program s th a t provide developers w ith a b e tte r u n d e rsta n d in g of th e causes of an in-held failure, which will subsequently help expedite th e fixing of those failures. However, th ere are several key differences th a t set our work a p a rt and illu strate how FU SIO N im proves upon th e s ta te of research.
First, techniques regarding in-held failure rep roduction rely on po ten tially expensive
program in stru m e n ta tio n , which requires developers to m odify code, and introduce over head. FU SIO N is com pletely au to m atic, our sta tic and dynam ic analysis techniques only need to be applied once for th e version of th e program th a t is released for testing. F u rth e r m ore. th e analysis process can be done w ith o u t th e need for in stru m e n ta tio n of program s in th e held. Second, cu rren t in-held failure rep roduction techniques require an oracle to signify when a failure has occurred (e.g.. a crash). FU SIO N is not an approach for crash or failure detection, it is designed to su p p o rt testers during th e bug rep o rtin g process.
Third, these techniques have not been applied to mobile apps and would m ost likely need
to be optim ized fu rth e r to be applicable for th e corresponding resource-constrained env- io rn m e n t.
C H A P T E R 2. R E L A T E D W O R K S 10
2 .4
B u g a n d E rror R e p o r t in g R e se a r c h
A subset of prior work has been focused on bug and crash triage [86. 71. 50. 59, 94. 60. 85. 18. 62. 68, 45, 48. 54. 56]. T he techniques associated w ith th is topic typically em ploy different program analysis and m achine learning or n a tu ra l language processing techniques to m atch bug rep o rts w ith ap p ro p riate developers. O ur proposed research com plim ents developer recom m endation fram ew orks, as FU SIO N can provide these fram ew orks w ith m ore d etailed “knowledge” th a n cu rren t s ta te of practice bug rep o rtin g system s.
A significant am ount of research has been conducted concerning th e sum m arization [66. 26, 82. 61. 93. 35], fault localization [97. 91. 81. 22. 90. 95. 67. 20, 34. 36]. classification and d etection of d u p licate bug rep o rts [41. 72. 92. 47, 96, 46. 75]. R esearch on these topics is prim arily concerned w ith d u plicate bug rep o rt detection, localizing bug rep o rts to specific areas of source code, and effectively sum m arizing rep o rts for developers w ith th e m ost p e rtin en t inform ation.
Again, th e work presented in th is p a p e r com plim ents these categories of research as bug re p o rts created w ith FU SIO N can provide m ore detailed inform ation, easliy linking the bug back to source code, allowing for b e tte r localization, sum m arization and. potentially, d u p lic a te detection. It is w orth n o ting th a t work by B ett.enburg et. al. on ex tractin g s tru c tu ra l inform ation from bug re p o rts is also related, however, we aim a t helping auto- com plete th e s tru c tu re d rep ro d u ctio n steps a t th e tim e of rep o rt creation, ra th e r th an e x tra ctin g it after th e fact [26].
Bugzilla@Mozilla
Home New Browse Search j (help) Reports My Dashboard Product Dashboard
Enter A Bug
• Please fill out th is form clearly, precisely and in as much detail as you can manage. • Please report only a single problem a t a time.
• These guidelines explain how to write effective bug reports.
Summary: ^
Product: Firefox (Change) Version: Q I* W hat did you do? (step s to reproduce) #
What h appened? (actual results)
What should have h appened? (expected results)
Attach a file: Choose File no file selected
Security:
Additional Details: This is a problem with Firefox on my phone or tablet.
Many users could be harm ed by this security problem: it should be kept hidden from th e public until it is resolved.
Figure 2.2: B u gzilla Issue Tracker: An example of a popular Issue Tracking System with
separate fields th at prompts users to enter reproduction steps and expected/actual results of the steps. Image taken from [5]
View Issu e Details [ Jump to N otes J ( Wiki ] [ < < ) [ > > ] [ Issu e History ] ( Print ] ID Project Category View S tatu s Date S ubm itted Last Update
0019828 m a n tisb t bugtracker pubiic 2015-06-11 1 7 :33 2 015-07-14 11:47 R eporter raro
Assigned To
Priority norm al Severity m ajor Reproducibility h a ve not tried S tatu s feedback Resolution open
Product Version 1 .3 .0 -b eta .2
T arget Version 1.3.x Fixed in Version Sum m ary 0019828: Escaping
D escription Since wc up d a ted to 1.3.0 Beta 2 we noticed th e re a re som e escaping issues All q uotation m arks are displayed escaped ( a s in V instead of ’) We noticed th is a t le ast o n th e following locations: - Bug description
- Notes - Email
Quick fix w as strip slah e s on th e following flies: - bug_vfcw _m c.php line 668
bugnote_view_wK .php line 275 - cora/em ail_api.php line 973
Ofcourse this is n o t th e preferred solution, escaping should be handled by th e d a ta b se wrapper o r so If th e re is a setting for this please let m e know'
T ag* No ta g s a ttac hed. A ttached Files
Figure 2.3: M antis Issue Tracker: An example of a popular Issue Tracking System containing
12
C h a p te r 3
T h e F U S IO N A p p ro a ch
F U S IO N ’S Analyze —> Generate workflow corresponds to two m ajo r phases. In th e A n a l
ysis Phase FU SIO N collects inform ation related to th e GU I com ponents and event flow
of an app th ro u g h a com bination of sta tic and dynam ic analysis. T h en in th e Report
Generation Phase FU SIO N takes advantage of the G U I centric n a tu re of m obile apps to
b o th auto-com plete th e steps to reproduce th e bug and augm ent each step w ith contex tu a l application inform ation. T h e overall design of FU SIO N can be seen in Figure 3.1. We encourage readers to view videos of our tool in use. com plete w ith com m entary th a t are available at our online ap p endix [57] T he key idea behind th e FU SIO N workflow is:
program analysis, performed preemptively before an app is released fo r testing, can be used as a means to aid reporters to easily provide information that developers need during the bug reporting process to reproduce and fix app issues.
3.1
A n a ly s is P h a s e
T he Analysis Phase collects all th e inform ation required for th e Report Generation Phase o p eratio n . T he first phase has two m ajo r com ponents: 1) sta tic analysis (Primer), and 2) dynam ic program analysis (Engine) of a targ e t app. T he inform ation generated by
(P rim er) and (Engine) is required by th e R ep o rt G eneration P hase. T he Analysis phase
Physical Device or Emulator
Analysis Phase__________________ ( l ) - Static App Analyzer (Primer)
apktool dex2jar Static Extraction of C omponents and Associated Attributes jd-cmd Decompiler
(2) - Dynamic Program Analyzer (Engine)
/ \ / " ' \
Hierarchy Step-by-Step
Viewer & Execution
uiautomator Engine t + . . » ... s ' GUI- b Screenshot Component Capture Information \ / Extraction /I FUSION Database Testers
JL
l
A
l
(4 ) -Auto- Completion EngineReport Generation Phase
(5 )- Report Entry (FUSION Ul)
(6)- Generated Reports (FUSION Ul)
Application Developers
Figure 3.1: O verview o f F U SIO N Workflow: First static and dynamic app analysis is per
formed on the target app, then the Auto-Completion Engine uses the information gleaned by the analyses in order to help the user auto-complete reproduction steps of a bug for the target app.
C H A P T E R 3. TH E FU SIO N A P P R O A C H 14
published to end users. B oth com ponents of th e Analysis Phase store their e x tra c te d d a ta in th e FU SIO N d a ta b a se (Fig. 3.1 - (3)).
3 .1 .1 S ta tic A n a ly s is (P rim er )
T he goal of th e P r im e r (Fig. 3.1 -
CD )
is to e x tra c t all of th e GU I com ponents and associated inform ation from th e app source code. For each GU I com ponent, th e P rim e r e x tracts: (i) possible actions on th a t com ponent, (ii) ty p e of th e com ponent (e.g.. B u tto n . Spinner), (iii) activities th e com ponent is contained w ithin, and (iv) class files w here th e com ponent is in sta n tia te d . T hus, this phase gives us a universe of possible com ponents w ithin th e dom ain of th e application, and establishes traceab ility links connecting GU I com ponents th a t rep o rters o p e ra te upon to code specific inform ation such as th e class or activ ity th e y are located w ithin.T h e P r im e r is com prised of several steps to e x tra c t th e inform ation o utlined above. F irst it uses th e dex2jar[6] and jd-cmd [10] tools for decom pilation, then, it converts th e source hies to an X M L-based rep resen tatio n using srcML [14]. We also use apktool [2] to e x tra c t th e resource hies from th e a p p ’s A P K . T h e id s. and types of GU I com ponents were e x tra c te d from th e xm l hies located in th e a p p ’s resource folders {i.e.. /res/layout
and /res/menu of th e decom piled application or src). Using th e srcML rep resen tatio n of th e source code we are able to parse and link th e G U I-com ponent inform ation to e x tra cte d app source hies.
3 .1 .2 D y n a m ic A n a ly s is (E n g in e)
T h e Engine (Fig. 3.1 - (2)) is used to glean dynam ic con tex tu al inform ation, such as th e location of th e GU I com ponent on th e screen, and enhance th e d a ta b a se w ith b o th ru n tim e GUI and ap p lication event how inform ation. T he goal of th e Engine is to explore an app in a sy stem atic m an n er ripping and e x tra ctin g ru n -tim e inform ation related to th e G U I com ponents du rin g execution including: (i) th e te x t associated w ith different GU I com ponents (e.g.. th e “Send” te x t on a b u tto n to send an em ail m essage), (ii) w hether
th e G U I component, triggers a tra n sitio n to a different activity, (iii) th e action perform ed on th e GU I com ponent during system atic execution, (iv) full screen-shots before and after each action is perform ed, (v) th e location of th e GU I com ponent object on th e test device’s screen, (vi) th e cu rren t A ctiv ity and window of each step, (vii) screen-shots of th e specific GU I com ponent, and (viii) th e object index of th e G U I com ponent (to allow for differentiation betw een different in sta n tia tio n s of th e sam e GU I com ponent on one screen).
T he Engine perform s this sy stem atic exploration of th e app using th e UIAutomator [1] fram ew ork included in th e A ndroid SDK. T his system atic execution of th e app is sim ilar to existing approaches in G U I ripping [16. 8 8. 83. 17. 21. 65. 32. 74]. Using th e UIAutomator
fram ew ork allows us to c a p tu re cases th a t are not c a p tu red in previous tools such as pop-up m enus th a t exist w ithin m enus, in tern al windows, and th e onscreen keyboard. To effectively explore th e application we im plem ented our own version of a system atic depth- first search (D FS) algorithm for application trav ersal th a t perform s click events on all the clickable com ponents in th e GU I th a t can be reached using th e D F S-based traversal.
D uring th e ripping, before each step is executed on th e GUI. th e Engine m akes a call to UIAutomator su b ro u tin es to e x tra c t th e con tex tu al inform ation outlined above regarding each GU I com ponent displayed on th e device screen. We th e n execute th e action associated w ith each G U I com ponent in a depth-first m anner on th e cu rren t screen. O ur cu rren t im plem entation of D FS only handles the c lic k /ta p action, however, as this is the m ost com m on action, we are still able to explore a significant am ount of an ap p lic atio n ’s functionality.
In th e DFS algorithm , if a link is clicked th a t would norm ally tra n sitio n to a screen in an ex ternal activ ity (e.g.. clicking a web link th a t would launch th e C hrom e web browser app) we execute a back com m and in order to stay w ithin th e current app. If th e DFS exploration exits th e app to th e hom e screen of th e d ev ice/em u lato r for any reason, we sim ply re-launch th e app and continue th e G U I traversal. D uring th e DFS exploration, th e Engine c a p tu res each activ ity tra n sitio n th a t occurs after each action is perform ed (e.g.. w hether or not a new a c tiv ity is s ta rte d /re s u m e d after an action to launch a m enu).
C H A P T E R 3. TH E FU SIO N A P P R O A C H 16
T his allows FU SIO N to build a m odel of th e app execution th a t we will la te r use to help tra c k a re p o rte r’s relative position in th e ap p when th ey are using th e system to record th e steps to reproduce th e bugs.
3 .2
R e p o r t G e n e r a tio n P h a s e
We h ad two m ajo r goals when designing th e Report Generation Phase com ponent of FU SIO N :
1. Allow for tra d itio n a l n a tu ra l language in p u t in order to give a high-level overview of a bug.
2. A uto-com plete th e rep ro d u ctio n steps of a bug th ro u g h suggestions derived by tra c k ing th e position of th e re p o rte r’s step e n try in th e app event flow.
D uring th e Report Generation Phase FU SIO N aids th e rep o rter in c o n stru ctin g th e steps needed to recreate a bug by m aking suggestions based upon th e “p o te n tia l” GU I s ta te reached by th e declared steps. T his m eans for each step s. FU SIO N infers — online — th e G U I s ta te G U I S in which th e ta rg e t app should be. by tak in g into account th e h isto ry of steps. For each step, FU SIO N verifies th a t th e suggestion m ade to th e rep o rter is correct by presenting th e rep o rter w ith contextually relevant screen-shots, w here th e rep o rter selects th e screen-shot corresponding to th e cu rren t action th e rep o rte r w ants to describe.
3 .2 .1 R e p o r t G e n e ra to r U se r In te r fa ce
A fter first selecting th e app to rep o rt an issue for, a rep o rter in teracts w ith FU SIO N by filling in some brief contextual inform ation (i.e., nam e, device, title) and a brief te x tu a l description of th e bug in question in th e to p half of th e UI. N ext, th e re p o rte r in p u ts th e steps to reproduce th e bug using th e auto-com pletion boxes in a step-w ise m anner,
FUSION - Bug report
(Android Ape Bug Raocrt)
R e p o rt d e ta ils fo r d o c u m e n t view e r.ap k v ersio n 2.2 Issu e I d : 1434857235016
H I R ep o rte d by: Reporter Q Device: Nexus 7 Q O rientation: Portrait }
R Title for th e b u g report:
The G old Page feature d o es not work property Q Brief desc rip tio n o f th e bug you en countered:
- What should happen: ~
Tapping the GoTo button Brings you to the corresponding page. • What happens instead: —
You stay on the sam e page
D e ta ils for s t e p 2
Q I Clicked } Button "Search" located at C enter ■»
0 Select the screenshot showing the com ponent retorted to th e step
ca®.
S te p s h isto ry (S um m ary)
1. CUCKirVon Sutton "OK" oca-nc at Confer
p Additional inform ation:
Type any ad d ’tionai ;ntormation lor this step
Yes jnext step) + I No. I am done *6
Figure 3.2: F U SIO N R eporter Interface: This figure shows the FUSION reporter web inter
face, with an area for contextual information and a natural language description of the bug at the top of the page, an area for reporting reproduction steps, and an area showing the history of the steps entered with an option to view or delete past steps.
C H A P T E R 3. TH E FU SIO N A P P R O A C H 18
s ta rtin g from th e initial screen of a cold app lau n ch 1, and proceeds until th e list of steps to reproduce th e bug is exhausted. Let us consider a ru n n in g exam ple w here th e user is filling out a re p o rt for th e D ocum ent Viewer bug in Table 4.1. According to th e various fields in F igure 3.2 th e rep o rter would first fill in th eir (i) nam e (Field 1), (ii) device (Field 2). (iii)
screen orientation (Field 3). (iv) a bug report title (Field 4). and (v) a brief description of the bug (Field 5).
3 .2 .2 A u to - c o m p le tin g B u g R e p r o d u c tio n S te p s
To facilitate th e re p o rte r in entering rep ro d u ctio n steps, we m odel each step in th e re p ro d u ctio n process as an {action, component} tu p le corresponding to th e action th e re p o rte r w ants to describe at each step, (e.g., ta p , long-tap, swipe, type) and th e com ponent in th e app G U I w ith which th ey in te rac ted (e .g ..“N am e” textview . “O K ” b u tto n . “D ays” spinner). Since rep o rters are generally aware of th e actions and G U I elem ents th ey in te rac t w ith, it follows th a t this is an intuitive m anner for th em to co n stru ct repro d u ction steps. FU SIO N allocates auto-com pletion suggestions to drop down lists based on a decision tre e tak in g into account a re p o rte r’s position in th e app execution beginning from a co ld -start of th e app.
T h e first drop down list (see Figure 3.3-A) corresponds to th e possible actions a user can perform a t a given point in app execution. In our exam ple w ith th e D ocum ent Viewer bug. le t’s say th e rep o rter selects click as th e first action in th e sequence of steps as shown in F igure 3.3-A. T he possible actions considered in FU SIO N are chck(tap), long-chck(long-
touch), type, and swipe. T he type action corresponds to a user entering inform ation from
th e device keyboard. W hen th e rep o rter selects th e type option, we also present them w ith a te x t box to collect th e inform ation th e y ty p ed in th e A ndroid app.
T h e second dropdow n list (see F igure 3.3-B) corresponds to the com ponent associated to th e action in th e step. FU SIO N presents th e following inform ation, which can also
1 C o ld -start m eans th e first step is executed on th e h rst window and screen displayed directly after the ap p is launched
A ) Action List
B) Component List
---■— --- — --- --- --- — Select GUI component—
-- Select action/event —
Button "OK" located at CenterClicked
Long-Clicked
Button “Restore?” located at TopRightSwiped
Button “Search" located at CenterTyped
Button “Remove?- located at CenterRiaht
Figure 3.3: A u to -C o m p letio n D ropdow n M enus: This figure shows examples of the auto-
completion dropdown menus th at present reporters with possible choices during course of creating a report.
be seen in F igure 3.3: (i) Com ponent Type: this is th e ty p e of com ponent th a t is being o p e ra ted upon, e.g., b u tto n , spinner, checkbox, (ii) Component Text: th e te x t associated w ith or located on th e com ponent, (iii) Relative Location: th e relative location of th e com ponent on th e screen according to th e p a ra m ete rs in F igure 3.5, and (iv) Component
Image: an in-situ (i.e., em bedded in th e dropdow n list) im age of th e in stan ce of th e
com ponent. T h e relative location is displayed here to m ake it easier for rep o rters to reason a b o u t th e on-screen location, ra th e r th a n reasoning a b o u t pixel values. In our ru n n in g exam ple, FU SIO N will p o p u late th e com ponent dropdow n list w ith all of th e clickable com ponents in th e M ain A ctivity since th is is th e first ste p and th e selected action was click. T he user would th en select th e com ponent th e y acted upon, in this case, th e first option in th e list: th e “O K ” b u tto n located a t th e center of th e screen (sec Figure 3.3-B).
O ne p o ten tia l issue w ith com ponent selection from th e auto-com plete drop-dow n list is th a t th ere m ay be du p licate com ponents on th e sam e screen in an app. FU SIO N solves this problem in two ways. First, it differentiates each d u p licate com ponent in th e list th ro u g h specifying te x t “O ption Second FU SIO N a tte m p ts to confirm th e com ponent entered
C H A P T E R 3. TH E FU SIO N A P P R O A C H 20
by th e re p o rte r a t each step by fetching screen-shots from th e FU SIO N d a ta b a se repre senting th e entire device screen. Each of these screen-shots highlights th e representative G U I com ponent as shown in Fig. 3.5. To com plete th e step e n try th e rep o rte r sim ply selects th e screen-shot corresponding to b o th th e app s ta te and th e GU I com ponent acted upon. In our running exam ple th e rep o rter would select th e full augm ented screenshot corresponding to th e com ponent th ey selected from th e dropdow n list. In our case an illu strativ e p o rtio n of th e screenshot for th e “O K ” b u tto n is shown in F igure 3.5.
A fter th e re p o rte r m akes selections from th e drop-dow n lists, th ey have an o p p o rtu n ity to en ter add itio n al inform ation for each ste p (e.g., a b u tto n h a d an unex p ected behavior) in a n a tu ra l language text, e n try field. For instance in our running exam ple, th e rep o rter m ight in dicate th a t after pressing th e “O K ” b u tto n th e p op-up window to o k longer th a n expected to disappear.
3 .2 .3 R e p o r t G e n e ra to r A u to -C o m p le tio n E n g in e
T h e A uto-C om pletion Engine of th e w eb-based rep o rt gen erato r (Figure 3.1-(4)) uses the inform ation collected up-front du rin g th e Analyze Phase. W hen FU SIO N suggests com pletions for th e drop-dow n m enus it queries th e d a ta b a se for th e corresponding s ta te of th e app event flow, aird suggests inform ation based on th e p ast steps th a t th e rep o rter has entered. Since we always assum e a “cold” application s ta rt, th e A uto-C om pletion Engine s ta rts th e rep ro d u ctio n steps e n try process from th e a p p ’s m ain A ctivity. We th en track th e re p o rte r’s progress th ro u g h th e app using predictive m easures based on p a st steps.
T h e Auto-C om pletion Engine o p erates on application steps using several different pieces of inform ation as in p u t. It m odels th e re p o rte r’s rep ro d u ctio n steps as an or dered stream of steps S w here each individual step s L m ay be either em p ty or full. Each step can be m odeled as a five-tuple consisting of {step-.mim, action, compjnam,e, activity,
history). T he action is th e gesture provided by th e rep o rter in th e first drop-dow n menu.
T h e componenUname is th e individual com ponent nam e as rep o rted by th e U la u to m a to r interface during th e Engine phase. T h e activity is th e A ndroid screen th e com ponent
is found on. T h e history is th e h isto ry of steps preceding th e cu rren t step. T he a u to com pletion engine pred icts th e suggestion inform ation using decision tree logic which can be seen in Figure 3.4.
FU SIO N presents com ponents to th e rep o rter a t th e g ran u la rity of activities or ap plication screens. To sum m arize th e suggestion process, FU SIO N looks back th ro u g h th e h isto ry of th e p ast few steps and looks for possible tra n sitio n s from th e previous steps to fu tu re steps depending on th e com ponents in te rac ted w ith. If FU SIO N was unable to c a p tu re th e last few step s from th e re p o rte r due to th e incom plete application execution m odel m entioned earlier, th en FU SIO N presents th e possibilities from all known screens of th e application. In our ru n n in g exam ple, le t’s consider th e re p o rte r m oving on to re p o rt th e second rep ro d u c tio n step. In this case, FU SIO N would query th e h istory to find th e previous activ ity th e “O K ” b u tto n was located w ithin, and th en present com ponent suggestions from th a t activity, in th e case th a t th e user stayed in th e sam e activity; and th e com ponents from possible tra n sitio n activities, in th e case th e user tra n sitio n e d to a different activity.
3 .2 .4 H a n d lin g F U S I O N ’S A p p lic a tio n M o d e l G ap s
B ecause D F S-based exploration is not exhaustive [73], th ere m ay be gaps in F U S IO N ’S d a ta b a se of possible app screens (e.g.. a dynam ically g enerated com ponent th a t triggers an activ ity tra n sitio n was not acted upon). Due to this, a re p o rte r m ay not find th e a p p ro p riate suggestion in th e drop-dow n list. To handle these cases gracefully, we allow th e re p o rte r to select a special option w hen th ey cannot find th e com ponent th ey in teracted w ith in th e auto-com plete drop-dow n list. In our running exam ple, le t’s say th e rep o rter wishes to indicate th a t he clicked th e b u tto n labeled “O pen D ocum ent” . b u t th e option is not available in th e auto-com plete com ponent drop-dow n list. In this case th e user would select th e “Not in this list...” option and m anually fill in (i) T he ty p e of th e com ponent (to lim it confusion, we present th is option as a drop-dow n box auto-com pleted w ith only th e G U I-com ponent types th a t exist in th e application, as e x tra c te d by th e Primer, in our
C H A P T E R 3. THE FUSION A P P R O A C H 22 No Yes Yes No Yes No Yes No Yes No Display all p ossib le app com ponents. Display com p onents for the ap p ’s Main Activity steps_history- l verified by FUSION? Is steps_history = 0? steps_history-2 verified by FUSION?
Display com ponents from previous activity
and possible transition activities. Is steps_history >=2?
Display com ponents
from Main Activity
and two sta g e s of transition activities. Display
com p onents from previous activity and p ossib le transition activities. Is steps_history = 1 and is steps_history-1 confirmed? Display com p onents from
the activity in
steps_history-2
and two sta g e s of transition activities.
Figure 3.4: D ecision Tree U tilized by A u to-C om p letion Engine: This figure outlines the
decision tree utilized by FUSION’S autocomplction engine which helps to predict the possible components th at a user can interact with at a specific place in the event flow of an application.
case th e user would choose ’’B u tto n ” ), (ii) any te x t associated w ith th e G U I-com ponent (in this case “O pen D ocum ent” , and (iii) th e relative location of th e G U I-com ponent as denoted in F igure 3.5 (in this case “Top C e n te r” ).
3 .2 .5 R e p o r t S tr u c tu r e
T h e Auto Completion Engine saves each step to th e d a ta b a se as rep o rters com plete bug rep o rts. Once a rep o rte r finishes filling o ut th e steps and com pletes th e d a ta e n try process, a screen containing the final rep o rt, w ith an a u to m atically assigned unique ID, is presented to th e rep o rter, and saved to th e d a ta b a se for a developer to view later (sec F igure 3.6 for an
Relative Location Enumeration Example Augmented Screenshot 25% 50% 25% T O P TO P T O P L E F T C E N T E R R IG H T C E N T E R LE FT C E N T E R C E N T E R R IG H T 50% B O T T O M B O T T O M B O T T O M L E F T C E N T E R R IG H T 25%
Figure 3.5: R elative L ocation E num eration and E xam ple A u gm ented Screenshot: This
figure shows FUSION’S enumeration for the relative location of GUI-componcnts on the screen and an example of an augmented full screenshot.
exam ple re p o rt from D ocum ent Viewer). T h e R eport presents inform ation to developers in th re e m ajo r sections. F irst, prelim inary inform ation including th e rep o rt title , device, and sh o rt description (shown in F igure 3.6 in blue). Second, a list of th e Steps w ith th e following inform ation regarding each ste p is dispalycd (highlighted in blue in F igure 3.6): (i) T h e A ction for each step, (ii) th e ty p e of a com ponent, (iii) th e relative location of th e com ponent, (iv) th e A ctiv ity class w here th e com ponent is in sta n tia te d in th e source code, and (v) th e com ponent specific screenshot. T h ird , a list of full screen-shots corresponding to each ste p is presented a t th e b o tto m of th e page so th e developer can trace th e steps th ro u g h each application screen (this section is highlighted in green in F igure 3.6).
C H A P T E R 3. THE FU SIO N A P P R O A C H 24
Bug C S E S 2 S 2 > < °rG
S o m etim es Go to d o e s n ’t w ork (by P articipant 4) j J j j J B
- W hat sh o u ld h ap p e n : — It sh o u ld go to th e p a g e - W hat h a p p e n s in ste ad : -- It d o e s n 't. T he s te p s for re producing th e b u g a re th e following: 1. CLICK in/on ReiativeLayout located at Top left. The com ponent is In the activity "rg.ebookdroid.ui.library.RecentActivity'.
Component image: Screen Image: i H . i (Also see thumbnail in corresponding step below)
2. CLICK m/on TextView located at Top. The component is in the activity "rg.ebookdroidui.viewer.ViewerActivityV
_
3. CLICK in/on LinearLayout located at Center. The component is in the activity "rg.ebookdrold.ui. viewer.Viewer Activity*. C cm porent mage Screen image 0 3 (A 50 see thumbnail n corresponding step below)
4. TYPE *2" in/on EditText "1“, located at Top. The component is in the activity “rg.ebookdroid.ui.viewer.ViewerActivity' Component image: m m K Screen Image: (Also see thumbnail in corresponding step below)
5. CLICK In/on TextView located at Top. The component is m the activity ‘rg.ebookdrold.ui. viewer. ViewerActivity*. Additional info: Prior to clicking again, swipe back up to page one
m m
Component image: Screen image m s (Also see thumbnail in corresponding step below)
6. CLICK m/on LinearLayout located at Center. The component is in the activity “rg.ebookdroid.ui.viewer.ViewerActivity".
E 3 (Also see thumbnail In corresponding step below)
7. TYPE "2" in/on EditText “1”, located at Top. The component is in the activity "rg.ebookdroid.ui.viewer.ViewerActivity. Additional info: After clicking this, it doesn't go to page 2
anymore
C omponent image: t Screen Image: KSE1 (Also se e thumbnail in corresponding step below)
If th e re a re an y s t e p s with 0 , it m e a n s S y s te m A w a s n ot ab le to find inform ation for th o s e s te p s
1 1 /3 1
CESfB'f
(■ESSES
Figure 3.6: E xam ple F U SIO N B u g R eport: This report shows the three major categories of
information contained within FUSION bug reports 1) Contextual information and natural language description of the bug. 2) A detailed set of reproduction steps including GUI-component specific screenshots. 3) A list of fullscreen screenshots with the GUI-component acted upon in each step highlighted.
D e s ig n o f E m p irica l S tu d ie s
T h e two m ajo r design goals behind F U SIO N are: 1) facilitate and encourage reporters
to submit useful bug reports f o r Android applications. 2) provide developers with more actionable inform ation regarding the bugs contained within these reports. In order to m ea
sure F U S IO N ’S effectiveness at achieving these goals, we have designed two com prehensive em pirical studies which evaluated two m ajo r aspects of our approach: 1) the user expe
rience of reporters using FUSION, and 2) the quality o f the bug reports produced by the system . To this end. we investigated th e following research questions (RQs):
• R Q p What types o f information fields do developers/testers consider important
when reporting and reproducing bugs in Android?
• R Q 2'- Is F U S IO N easier to use fo r reporting and reproducing bugs than traditional
bug tracking system s?
• R Q3: Do bug reports generated with F U S IO N allow f o r fa ste r bug reproduction
compared to reports submitted using traditional bug tracking systems?
• R Q4 : Do developers/testers using F U S IO N reproduce more bugs compared to tradi
tional bug tracking systems?
T he em pirical studies used to evaluate these research questions m odel two m aintenance activities involving rep o rtin g and reproducing real bugs in open source apps. In the