Introduction to Defect Classification

GUI defects can be roughly classified into three groups: directly detectable defects, undetectable defects and comparably detectable defects.

Directly detectable defects are those which can be detected without involving an AUT’s specifications and base version information. These defects usually include crashes, and recognizable error messages. Crashes cause the AUT to stop responding to inputs.

Graphic User Interface Modelling and Testing Automation

Recognizable error messages are usually the system error dialogs or specifically customized message boxes which pop up while the AUT is running. This type of defect can be detected by random testing tools. When these types of defect occur, they can be easily recognized, and no specifications or regression information are needed. Random input testing is also referred to as “stochastic” testing or “monkey” testing [93]. The latter designation is used to impart the idea of someone without a brain, or without knowing what he's doing, seated in front of a computer and interacting randomly with the keyboard or mouse. Microsoft has reported that 10-20% of the bugs in their software projects are found by monkey test tools [93].

Undetectable defects are those defects that GUI testing automation is not able to detect. Because GUI defects are manifested as failures observed through the GUI, and only the effects on the GUI can be extracted, the defects whose results are not reflected in the GUI are not detectable. Examples of this type of defect include wrong files being saved with a successful process, emails being sent successfully but failing to reach their destinations, wrong data being saved into background databases with successful responses, and background thread execution without outputs to the foreground GUI. These types of defects require testing via direct methods and are primarily conducted manually.

Comparably detectable defects are the type which GUI testing automation principally deals with. This kind of defect can be detected by comparing the actual state affected by the defects with the expected state. The expected state can be a state from a base

Graphic User Interface Modelling and Testing Automation

version in regressive testing, or derived from the specifications of an AUT. When a test case is executed, the actual states will be compared with the expected results and the differences will be reported.

Some researchers have worked on defect classification with particular focuses. L. Briand and Y. Labiche, and I. Krsul [111, 112] have addressed defect classes at different phases of the development life cycle, e.g., defects in the requirements specification document. K. Weidhenhaupt et al. [113] have focused on defects in e- commerce applications. M. L. Lough and G. J. Meyers [114, 115] have focused on taxonomies for security issues. In general, typical defect classes of an AUT are listed as follows [105]:

 Completeness defects subsume all defects related to an incomplete implementation of the specified functionality. This usually includes missing functionality defects and undesired functionality.

 Input/Output (I/O) defects subsume all defects related to wrong input respectively to wrong output data of the AUT. Boundary defects, defects concerning wrong size, shape or format of the data or combination defects are typical I/O defects.

 Calculation defects subsume all defects resulting from wrong formula or algorithms in the AUT.

 Data handling defects subsume all defects related to the lifecycle and the order of operations performed on data. This usually includes duplicated data or dataflow

Graphic User Interface Modelling and Testing Automation

defects (defects related to the sequence of accessing a data object (e.g., a data update before the data has been created).

 Control flow and sequencing defects subsume all defects related to the control flow or the order of actions. Typical control flow defects concern wrong sequencing of the actions performed or iteration and loop defects, which subsume all defects related to the control flow of iterations and loops.

 Concurrency defects subsume all defects related to the concurrent execution of parts or of multiple instances of the AUT.

 Display and navigation defects subsume all defects related to the user interface, which are not usability defects. Typical defects of this class are display defects and navigation defects.

 NFR (non-functional requirement) defects subsume all defects related to the quality of the AUT. This usually includes defects concerning reliability, usability, efficiency, maintainability and portability.

 Inter-Software defectssubsume all defects concerning the interface of the AUT with other software systems. Typical defects of this class are input/output defects, concurrency defects or completeness defects such as missing third-party software.

 Other defects subsume all other defects including hardware defects, test design defects, OS and compiler defects etc.

The defect classes are not orthogonal, i.e., a defect can be categorized into more than one class. Additionally, a defect can also be associated to a combination of defect

Graphic User Interface Modelling and Testing Automation

classes. Different types of defects accounts for different percentages of the total defects in software systems. By analyzing and re-organizing the results of Brooks, Robinson and Memon [53, 108], the distribution of different types of defects is shown in Table 4.1. Not all classes of defects mentioned above are detectable during GUI testing automation.

Table 4.1 Distribution of defect types

Fault Type Defects (%)

Completeness 13.53

I/O 6.5 Calculation 11.69

Data handling 17.34

Control flow and sequencing 17.86

Concurrency 5.18

Display and Navigation 8.52

NFR 4.26 Inter-software 4.85

Other 10.27 Among these classifications, Completeness defects, Calculation defects, Input/Output

(I/O) defects, Display and navigation defects are detectable using GUI testing automation. Inter-Software defects are not, unless they manifest as crashes or typical error messages. Data handling defects, control flow and sequencing defects, concurrency defects and, NFR defects are usually detected by traditional software testing methods which involve large amounts of manual analysis and observation. My taxonomy will be restricted to defects which can be detected during GUI testing automation, about 40% of all the software defects listed in Table 4.1. Other types of

Graphic User Interface Modelling and Testing Automation

example, traditional variable boundary value testing shall be tested with conventional white-box testing techniques rather than with GUI testing techniques.