The Operation

After designing and planning the experiment, it must be carried out in order to collect the data that should be analyzed. This is the operation phase, which is basically consisted of three steps:

preparation, where subjects are chosen and experiment material is prepared; execution, where the subjects perform their tasks according to different treatments and data is collected, and data validation, where the collected data is validated (Wohlin et al.,2000). Each will be following detailed.

Preparation

The subjects were 32 undergraduate students from the MATB15 course, from Computer Science department at Federal University of Bahia, Brazil - UFBA -, representing all students regularly enrolled in such a course. All students had previously enrolled in courses on object-oriented programming, Java programming language, and most recently on software verification and validation, thus including testing concerns. Hence, we believe this set is representative to be involved in this experiment.

The subjects have to agree to the research objectives. They comply with the experiment by signing up the consent form, as previously mentioned in Section5.2.1. The results of the personal subjects performance in the experiment will be kept confidential. Moreover, they were informed that we would like to investigate the outcome of the process application. However, they were not aware of which aspects we intended to study, thus including the hypotheses stated.

5.2. FIRST EXPERIMENT

Figure5.2RiSEChairFeatureModelusedintheexperiment.

5.2. FIRST EXPERIMENT

Execution

The experiment was conducted from November to December in 2009, according to the definition and planning documented, and data was collected. The activities involved in the experiment were followed by the subjects. Table 5.2.1sketches the agenda of experiment training and execution sessions.

The characterization, when subjects filled in the background form, enabled us to divide subjects into two groups, one to perform the experiment in an ad-hoc fashion (group 1), and other to apply the RiPLE-TE process (group 2). Moreover, the balancing strategy was then applied in order to have two groups as balanced as possible in terms of expertise. Subjects’

expertise was considered for balancing purposes.

Table5.2presents the initial division of subjects into their respective groups. The division is not equal since as the rounds would be held in different days, it caused the transfer of some subjects from group 1 to group 2 and vice-versa, so that everyone could participate. Indeed, this fact does not negatively influence on the results.

Table5.3and Table5.4present a condensed view of the subjects’ profile, respectively group 1 - which performed the experiment in an ad-hoc fashion -, and group 2 - which performed the experiment using the RiPLE-TE Process. In these tables, their experience with software development, Java testing, SPLand JUnit is presented, in terms of months dealing with such aspects. Experience with Eclemma was not presented in this table, since no subjects had experience with such a tool. We believe this lack of prior expertise does not represent a threat to the validity of this experiment since the commands required for this experiment were addressed in the training sessions, and were considered elementary for a group of subjects in the last period of a Computer Science undergraduate course. Moreover, subjects had to their disposal a tutorial with a step-by-step guide on how to correctly use the tool in the experimental study context.

Moreover, their participation in industrial development and test projects is presented as well. This information is a clear indicator on how experient the subjects are in the topics required to execute this study. Yet, the experience of the subject regarding english reading, in a trichotomous categorisation: advanced, intermediate and basic. This information may impact on the results since all the artifacts that comprise the set of instruments are english-writen. Another information that can be seen in this table refers to their knownledge on other framework for unit tests, in addition to JUnit - [A] NUnit¹², [B] RSpec¹³, [C] Test::Unit¹⁴, [D] Selenium, and

12NUnit is a unit-testing framework for all .Net languages - http://www.nunit.org/

13RSpec provides a behaviour driven development framework for Ruby - http://rspec.info/ http://rspec.info/

14Test::Unit is Ruby Unit Testing Framework - http://test-unit.rubyforge.org/

5.2. FIRST EXPERIMENT

[E] other. The form included a field in which they could write down tools that they had some expertise. Much probably subjects with have already worked with any unit testing framework can easily understand how JUnit works.

Table 5.2 Subjects divided into groups

Group Subjects ID # of subj.

1 1 2 3 5 6 8 10 11 14 18 21 22 23 24 25 26 27 17

2 4 7 9 12 13 15 16 17 19 20 28 29 30 31 32 15

Table 5.3: Subjects’ Profile from Group 1 - Ad-hoc fashion

Subject English Particip. in Industry Experience in Testing

ID Reading Dev. Project Test. Project Programming* Java* Testing* SPL* JUnit* Tools

1 advanced No Yes 48 0 2 0 2 A

2 intermediate No Yes 24 0 5 0 1 A

3 advanced Yes Yes 18 0 12 0 0 C

5 advanced No Yes 50 30 4 0 1 C

6 intermediate No No 12 6 6 0 2 B,C

8 advanced No Yes 53 30 11 0 2 C,E

10 intermediate No No 60 36 6 0 6 E

11 advanced No No 48 30 6 0 6

-14 intermediate No Yes 42 24 12 0 6 C

18 basic No No 24 24 0 0 0

-21 advanced No No 54 6 3 0 1 C

22 advanced No Yes 48 6 4 0 4 A

23 intermediate No Yes 48 6 4 0 0 C

24 advanced No Yes 48 30 4 0 4

-25 intermediate No Yes 30 30 4 0 4 A

26 intermediate No Yes 30 12 3 0 3

-27 intermediate No Yes 36 18 4 0 0 C

(*) the experience is expressed in months.

5.2. FIRST EXPERIMENT

Table 5.4: Subjects’ Profile from Group 2 - RiPLE-TE

Subject English Particip. in Industry Experience in Testing

ID Reading Dev. Project Test. Project Programming* Java* Testing* SPL* JUnit* Tools

4 advanced No Yes 30 0 2 0 0 E

7 intermediate Yes Yes 50 36 6 0 3 C

9 intermediate Yes Yes 40 28 10 0 2 D

12 basic Yes Yes 32 24 6 0 3

-13 intermediate No Yes 24 24 6 0 2 B,C

15 advanced Yes Yes 30 24 18 0 18 D,E

16 advanced No Yes 54 30 12 0 8 C,E

17 intermediate No Yes 24 3 2 0 0

-19 advanced No Yes 60 6 3 0 0 E

20 intermediate No No 36 4 4 0 0

-28 intermediate No Yes 48 24 3 0 2 A,E

29 advanced No No 32 24 4 0 2 E

30 basic No No 36 24 4 0 0 C

31 advanced No No 40 24 3 0 3 B

32 intermediate No Yes 36 12 4 0 0 E

(*) the experience is expressed in months.

The first training consisted of an explanation on the topic SPL, in which subjects from both groups were presented to this new software development approach. Moreover, explanation on the experiment’s objective, their roles in this study were addressed.

Then, the tools to be used in the experiment were subject of training. It included JUnit framework¹⁵and Eclemma code coverage tool¹⁶. The former was used in order to create test cases and automate unit test suites. The second, in order to gather the code coverage they covered with test cases, as they were requested to.

The subjects were said for not to implement new features, but rather analyze the components they were given and implement the test cases to evaluate those. Thus, their tasks involved analyze the code and specifications, create test assets, execute test cases and suites, and report

15http://www.junit.org/

16http://www.eclemma.org/

5.2. FIRST EXPERIMENT

the findings in the proper form (Appendix B.3). They had to perform unit tests only in the EventManagement component, although they should access the code of the other components and their specification, in order to known the flow of information as well as classes, methods and attributes. In addition, subjects are encouraged to reuse test cases they create in terms of saving effort, regarding to the variability inserted in the portion of code they have at their disposal. The source code they produced containing the unit test cases must be recorded in a repository for further analysis.

At the end of the experiment, each of the 32 participants completed a feedback questionnaire, either type A (AppendixB.4) or B (AppendixB.5). Moreover, the type C (AppendixB.6) was answered by 6 subjects from group 1, randomly chosen.

Data Validation

Data was collected from 32 subjects. However, data from 2 subjects (IDs 22 and and 29) were removed, since they either did not participate in all activities of the experiment, or they did not complete the forms as requested at the beginning of the experiment. Table5.5shows the final grouping of subjects. Although we were counting on all subjects, we believe that the absence of two of them does not invalidate our work, in terms of statistical analysis and interpretation of the results.

Table 5.5 Final grouping of subjects

Group Subjects ID # of subj.

1 1 2 3 5 6 8 10 11 14 18 21 23 24 25 26 27 16

2 4 7 9 12 13 15 16 17 19 20 28 30 31 32 14