Experiment Design

In order to answer the previous research question, we need to measure the success of our Systematic Evaluation Methodology criteria. For this purpose, we began by establishing two groups of DSL producers, one for those who evaluated their DSLs based on our methodology and another for those who used an ad-hoc evaluation approach. In both cases, DSL producers only assessed their languages after completing

7. SYSTEMATICEVALUATIONMETHODOLOGYVALIDATION 7.2. Experiment Design

84

the development process. In the end of their language evaluation we asked both groups to answer a questionnaire over the internet.

During their evaluation process, we never gave any kind of advice, but we did clarify any doubts about our methodology. We think that this process helped us avoiding the contamination of the results according to our desires or expectations.

7.2.1. Subjects

Eight subjects were engaged to our experiment. Four of them used our methodology. They developed: a DSL for ubiquitous devices, a DSL for languages composition, between ―I*‖ and ―KAOS‖, a DSL tool for transformations, and the last subject did not actually develop a DSL, but used two DSLs to perform transformation rules between ―I*‖ and ―KAOS‖. The other four subjects based on an ad-hoc evaluation approach developed: a DSL to specify ―I*‖ language rules, a DSL for queries optimization, a DSL to specify ―KAOS‖ language rules, and a DSL to specify applications of augmented reality. In both cases subjects were MSc Students from DI/FCT.

The four subjects, who used our methodology, have followed a single iteration of our iterative evaluation model presented in Figure 6.3 of section 6.4.2. This has happened since they could only assess their languages after the development process has been fully completed.

In order to explain our evaluation methodology, presented in section 6.4, we carried out a debriefing session. Here, we gave the details about what should be developed and established in each stage of the evaluation process (Figure 6.6), the actors that should be involved and their tasks (section 6.3), and the number of domain experts that we recommend for each group. The material of our methodology, scenario’s questionnaire (Table 6.5), final questionnaire (Table 6.4) and template to store domain experts’ results (Appendix 1) was also made available to them.

7.2.2. Questionnaire

A questionnaire has been answered by both groups (Table 7.1). Every question relates to one of the two main areas. The first one, questions E1 to E13, concerns the procedures that DSL producers have made to set up the experiment. Here, we tried to

7. SYSTEMATICEVALUATIONMETHODOLOGYVALIDATION 7.2. Experiment Design

85

identify their experimental foundations and advantages from using them. For those who have followed our methodology we tried to understand the extent to which they used of our evaluation methodology. The second area of interest, questions Q14 to Q17, concerns DSL producers’ general impressions. This time, for each group of DSL producers we sought to understand their satisfaction level with the evaluation process followed.

Subjects’ responses have been collected in two ways: through a Likert scale and open answers. We used both, in order to get more knowledge about certain options they have made during language’s assessment.

ID Question

Experiment

Q1 Have you used the checklist based validation? Yes/No If you answered No, explain why.

Q2 How many changes have you done to the DSLs Concrete Syntax based on the checklist based validation?

Q3

Have you followed the entire Evaluation Process? Yes/No

If you answered No, which of the following steps did you not use? Subject Recruitment Task Preparation Pilot Session Training Session Exam Scenario’s Questionnaire Final Questionnaire Result Analysis

Why did you not use them? (Provide a reason for each unused step) Q4 How many groups of domain experts did you create?

(E.g. Experienced Group, Student Group, etc.)

Q5 How many domain experts did you assign to each group? Q6 How many scenarios have you defined?

Q7

Did you add new questions or update the existing ones from the provided questionnaire? Yes/No

Identify the new questions and the updated ones, and give a reason Why you needed it.

Q8

Did you remove any question? Yes/No

Identify the ID of the removed questions, and give a reason Why you removed it.

Q9

How did you set up experiment’s environment? In-house Environment

Asynchronous Communication (offline communication, e.g. email) Which one?

7. SYSTEMATICEVALUATIONMETHODOLOGYVALIDATION 7.3. Results of the Experiment

86

Synchronous Communication (online communication, e.g. VOIP) Which one?

Q10

Did you establish any Data Collection Tools (e.g. Recording, Log Analysis)? Yes/No

Which tools? (If you answered Yes)

Q11 How much effort (Man-Hours) did it take to set up the experiment until domain experts’ Exam?

Q12 How many changes have you performed to the DSL after the experiment? General Impressions

Q14 How useful did you find the checklist based validation? (Very Good, Good, Satisfactory, Bad, Very Bad)

Q15 How demanding did you find establishing the experiment? (Undemanding, Simple, Regular, Tough, Severe)

Q16

Which step(s) of the Evaluation Process did you find more challenging to follow? Subject Recruitment Task Preparation Pilot Session Training Session Exam Scenario’s Questionnaire Final Questionnaire Result Analysis

Why? (Please express your feelings for each selected one)

Q17 Did you feel lost in What and How to do, to establish the experiment? (Very often, Often, Sometimes, Seldom, Never)

Table 7.1 – Questionnaire