Towards a Semantic Knowledge Base on Threats to Validity and Control Actions in Controlled Experiments

(1)

Towards a Semantic Knowledge Base on Threats to

Validity and Control Actions in Controlled Experiments

Stefan Biffl1 _{Marcos Kalinowski}2 _{Fajar Ekaputra}1

Amadeu Anderlin Neto3 _{Tayana Conte}3 _{Dietmar Winkler}1

1_{Vienna University of Technology, Austria} 2_{Fluminense Federal University, Brazil}

3_{Amazonas Federal University, Brazil}

(2)

Introduction

 Validity is a property of inferences and every experiment faces

Threats to Validity (TTVs).

 We assume that experiment planners can benefit from a structured

Risk Identification Risk Assessment Risk Mitigation Risk

Monitoring Risk Control Risk Management Steps

Experimt Definition Experimt Planning Experimt Operation Analysis & Interpretation Presentation & Packaging Experiment Process Steps

Intensity of Risk Management Tasks in Experiment Process

Examples of typical experiment risk elements

Subject selection, Subject characteristics, Subject groups Process, Pretest, Training, Treatment, Replication Artifact, Material, Environment, Instrumentation Cause/effect, Metrics, Observation, Data collection Hypothesis, Statistical test Risk Assessment Risk Monitoring

(3)

Introduction

3



Context and challenges:

 (1) No semantic access based on domain concepts.

– Risk of missing important TTVs related to their experiment plan.

 (2) SLRs publish static reports. There is no way for systematically integrating evidence and sharing it with other researchers to reuse and extend.

(4)

Introduction



A semantic knowledge base (KB) on TTVs and control

actions could provide access using domain concepts,

helping to identify:

– Relevant TTVs related to their research context; and – Information on how TTVs have been addressed.



A good TTV KB should be flexible enough to allow

adding new evidence and the analysis of data

(5)

Research Issues

 RI-1: TTV KB Stakeholder Needs. The starting point for

developing a TTV KB is: What are the most relevant needs and requirements for a TTV KB prototype from the viewpoint of EMSE process stakeholders?

 RI-2: TTV KB Design. Which data elements are necessary to address the most relevant queries from EMSE stakeholders on TTVs and control actions?

 RI-3: TTV KB Content Analysis. How can the TTV KB content bring new insights into the use of TTVs in SE experiments compared to the available textbooks?

(6)

RI-1: TTV KB Stakeholder Needs



We identified stakeholder queries in an informal survey

with 10 EMSE experts from six different research

groups.



Three most relevant Stakeholder Queries (SQ)

– SQ1: Which research areas have been addressed by the experimental studies reporting TTVs?

– SQ2: Which TTVs have been reported in a given research area (filtered by TTV type and experiment domain concept)?

– SQ3: Which control actions have been reported for a specific TTV?



Other requirements

– Query interface (via a web user interface);

(7)

RI-2: TTV KB Design

7



The TTV KB prototype was implemented with semantic

technologies using the Protégé framework.



Interfaces:

– A web prototype for querying.

– An interface to import data from spreadsheets. – Glossary to add synonyms for domain concepts.

(8)

RI-2: TTV KB Design



To populate the TTV KB, we used a TTV SLR

spreadsheet containing data on 206 experiments, their

related publications, TTVs, and control actions.



Data extensions to address the requirements:

– SWEBOK Chapters and BOK Topics related to each of the 206 experiments;

– Experiment domain concepts related to each TTV and control action;

– Generic TTVs reported by (Wohlin et al., 2012) (most-cited TTV

identification source, cited as source by 47 experiments); and – Mappings between the specific and generic TTVs.

(9)

RI-2: TTV KB Design

9

(10)

RI-3: TTV KB Content Analysis



The TTV KB content analysis showed a high variation in

the number of specific TTVs reported in an experiment.



There are only a few SE research areas with a

significant number of experiments reporting TTVs.

(11)

RI-3: TTV KB Content Analysis



Even in research areas with a medium-sized set of

experiment reports, this set discusses a wide range of

specific TTVs, which can be useful to consolidate.

– Topic “UML” associated to just 15 experiments still covers 64% of the specific TTVs and 75% of the generic TTVs found in all 206 experiments.



A small set of TTVs has been reported in many

experiments and the relevance of TTVs may vary

notably for different research areas.



More than half of the 206 experiment reports discussed

specific TTVs that could not conservatively be mapped

to generic TTVs in the most-cited textbook.



Five of the generic textbook TTVs were not directly

reported in any of the 206 experiment reports.

(12)

Discussion and Conclusion



Experiment planners may benefit from a structured

overview on TTVs and control actions collected from

actual experimental SE research

– Complementing other valuable sources for identifying TTVs, such as guidelines and generic checklists.



We defined a strategy of providing such overview by

introducing a semantic TTV KB populated with data

from a TTV SLR.



Concerning the TTV KB content quality:

– the SLR’s primary studies were selected based on strict quality criteria; and

– data extensions (e.g., mapping specific to generic TTVs) were added using a peer-reviewed process.

(13)

Discussion and Conclusion



The resulting TTV KB is extensible and available online.

– Enables experts in the EMSE community to incrementally improve its content for their research areas.



Future work:

– Integrate other experiment-related data (e.g., experiment design, hypotheses, and response variables) into the KB, significantly extending the querying capabilities.

– Provide collective intelligence functions to foster community involvement for frequent updates and quality assurance.

13

“Software engineering requires a community supported living experience base”

V.R. Basili, “A personal perspective on the evolution of empirical software engineering” In: J. Munch and K. Schmid (editors), Perspectives on the Future of Software Engineering, pp. 255–273. Springer, 2013.

(14)