Kaplan and Duchon (1988) classify the field of Information Systems under the field of social systems. They say that such social systems involve many “uncontrolled and uni- dentified” variables and hence methods used in closed systems are not very appropriate for such systems. They use a combination of quantitative and qualitative techniques in their case study. The qualitative methods used in their paper include open-ended inter- viewing, observation, participant observation, and analysis of responses to open-ended items on a survey questionnaire, while quantitative methods used include analysing the data collected from questionnaires (Kaplan and Duchon 1988).
Saracevic (1995) describes six levels of prototype evaluation, namely: (i) Engineering (ii) Input (iii) Processing (iv) Output (v) Use or user, and (vi) Social level. According to Saracevic (1995) most of prototype evaluations focus on only one or two of these levels. Vokurka et al. (1996) discuss a list of quantitative and qualitative methods to validate prototypes. Zelkowitz and Wallace (1998) describe twelve validation methods that fall into three categories, namely observational, historical and controlled. An observational method collects relevant data as the project proceeds and there is little control over the project development. The historical method collects data from projects that are com- pleted and for which the data is available. A controlled method provides data from mul- tiple instances of observations and can thus check statistical validity of the data (Zelkowitz and Wallace 1998).
Hevner et al. (2004) list five different types of design evaluation methods, namely, ob- servational, analytical, experimental, testing and descriptive. They classify the different evaluation methods as follows:
1) Observational:
a. Case Study: Study the artefact in a business environment b. Field Study: Monitor use of artefact in multiple projects
2) Analytical:
a. Static Analysis: Examine the structure for static qualities (e.g. complex- ity)
b. Architecture Analysis: Study the fit of the artefact into the IS architecture c. Optimization: Demonstrate inherent optimal properties of the artefact or
provide optimal bounds on artefact behaviour
d. Dynamic Analysis: Study artefact in use for dynamic qualities (e.g. per- formance)
3) Experimental:
a. Controlled Experiment: Study artefact in a controlled experiment for dif- ferent qualities like usability
b. Simulation: execute artefact with artificial data 4) Testing:
a. Functional (Black Box) Testing: Execute artefact to discover failures and identify defects
b. Structural (White Box) Testing: Perform coverage testing of some metric in the artefact implementation like execution paths.
5) Descriptive:
a. Informed Argument: Use information from the knowledge base (e.g. rele- vant research) to build a convincing argument for the artefact’s utility b. Scenarios: Construct detailed scenarios around the artefact to demonstrate
its utility
As the TESNA tool as well as the method is the first of their kind, we use a soft observa- tional evaluation method like case study research for evaluation. The tool TESNA was tested in each iteration of its development using functional (black box) as well as struc- tural (white box) testing methods. Here we describe the two ways in which the TESNA method and tool have been evaluated.
2.3.1 Evaluation through Case Studies
The case studies were conducted using the Case Study research methodology as pre- scribed by Yin (2003). The tool was used to gather as well as display data in the differ- ent case studies. The data from interviews of the different employees was used to aug- ment and verify data collected from the different artefacts. The interview data analysed using the qualitative methods including the data coding scheme by Miles and Huberman (1984). On the basis of the different network and graph displays, coordination problems related to the development process were identified. During the second case study (at
eMaxx, Chapter 6) a presentation was held in which seven of eMaxx employees (who were also involved in the projects studied) participated. After the presentation a feedback questionnaire on the TESNA method and tool was distributed among the attendees. The feedback from the questionnaire served as evaluation of the TESNA method and tool and can be seen in Chapter 6 (section 6.7.3).
No formal comparison with related artefacts was conducted as there are no other tools that address the specific research problem. Although, there are tools that address sub- problems like for the identification of specific coordination problems (called the Con- way’s Law STSC as will be explained in Chapter 3) (de Souza, Froehlich et al. 2005). We have conducted case studies in two comparative Software Development companies, Mendix and eMaxx apart from multiple Open Source case studies. Both the companies (eMaxx and Mendix) adopted an iterative software development process and had devel- opment teams of similar size (between 15 to 20 developers each). The products the two companies develop are also comparable. While Mendix develops a web-based Service Oriented Application (SOA), eMaxx develops Mid-office solutions for city administra- tions in The Netherlands. Both companies employ three tier architecture for their prod- ucts, namely a client (thin client), server and database.
In the eMaxx case study the following s methodological steps were followed:
(i) In a meeting with the company’s CTO; the technical architecture, the task and team allocation of the different employees were discussed (ii) The different code repositories used by the developers were analysed.
The important core modules relating to different projects were then determined through interviewing the Managers and select employees. The call graph structure, the clustering and the coupling metrics of these modules were then analysed in more detail.
(iii) The different communication and coordination mechanisms used by the employees were first analysed. Then, the most representative mode of communication and coordination was determined through various interviews of the managers and employees. The communica- tion repositories (e-mail, chat and bug tracker) corresponding to these modes of communication and coordination were then analysed in more detail.
(iv) After analysing the data from the code repositories, the data in the form of graphs was taken back to the developers for their feedback (v) The same procedure (as (iv)) was repeated with the data analysed
from the communication repositories. This time, the data (especially the accuracy of the mined social network) was discussed with each of the employees involved through face-to-face interviews.
(vi) After determining if the data was valid and an accurate representation of the social and technical structures, the data was analysed to identify coordination problems.
(vii) After identifying coordination problems, the Managers responsible for the particular projects in which the coordination problems were iden- tified, were again interviewed. This was done to obtain their feedback on the findings.
(viii) A research presentation and feedback session was arranged with the employees of the company in order to get their feedback on many of the coordination problems.
(ix) Follow-up interviews of the project members were then held in order to ascertain the reason for the occurrence of some of the coordination problems.
In the Mendix case study only steps (i) to (vii) was followed, as it was a first case study and we thought that the data was not too extensive to warrant a presentation to the com- pany’s employees.
On the other hand, in the Open Source case studies, only data from the repositories was considered. This was done, as it was very difficult to obtain a large number of responses to our online questionnaires from the Open Source developers. Hence, we used the data from the repositories and verified the presence of coordination problems using additional data (in the case of the Modularity Pattern) or additional case studies (in the case of the Core-Periphery Shift Pattern).
2.3.2 Evaluation using Feedback on TESNA
After the presentation (on the analysed data) given to the eMaxx employees, we also dis- tributed questionnaires for feedback on the method and tool. We later discussed the feed- back with all the participants of the talk, in order to get more qualitative data on their feedback. In total eight developers, support personnel and project leaders attended the presentation and also filled the questionnaires. The summary of the responses can be seen in the Chapter on the eMaxx case study (Chapter 6).
In the next Chapter (Chapter 3) we describe the theoretical underpinnings of this re- search.