Data analysis enables a researcher to scrutinize the data to make sense out of it, rule out any other possible interpretations, and to put together appropriate conclusions (Miles & Huberman, 1984).
There are several approaches for data analysis such as building case study data favourable for statistical analysis, creating a template of categories, recording the
frequency of different incidents, doing statistical calculations to determine relationships, and using methods to classify information (Miles & Huberman, 1984; Pelz, 1981). Yin (1994) suggests using the theoretical propositions that are used to investigate the research problem.
This study employed the theoretical propositions approach using the adaptation framework to guide data analysis involving applying the adaptation framework to identify agile adaptation factors. The data analysis involved describing the data in a meaningful way by using adaptation components identified in the framework including the background of the case study to answer “how and why” research questions, which were the sub-questions to the main research question about agile adaptation.
This study used the coding scheme approach (Creswell, 1994). Chapters 4 and 5 present the two case studies using meaningful descriptive data, referred to as “thick
description”, for the reader to understand and to draw their own interpretations (Denzin, 1978; Patton, 1990).
This research used the Nvivo software tool (qualitative analysis tool) to organise and analyse the large amount of data. Separate projects folders were created for the two case studies. This was done since each case study was treated as a single unit for making generalizations. The transcribed interviews and other information saved as Word documents were imported into Nvivo.
Nvivo allowed the organising of data into various categories based on the adaptation framework (theoretical framework) and using new adaptation categories as they emerged by enabling data to be merged from different documents. Nvivo also allowed the flexibility to rearrange data as analysis proceeded and as new categories emerged.
For each case study, data was organised and analysed in categories according to Fitzgerald’s (1998) adaptation framework (Figure 2) using methodology-in-action, development environment (organisational) factors, overt factors, and covert factors and their sub-categories. An additional category, company background, was included and further sub-categories emerged as analysis proceeded; these are identified in Chapters four and five. Similarly for methodology-in-action further sub-categories emerged as analysis proceeded, which are also identified in Chapters four and five. Figure 4 identifies the four main categories adopted from Fitzgerald’s adaptation framework (methodology-in-action, development environment factors, overt factors, and covert factors) which were used to organise and analyse data. Figure 4 also includes the company background category and the sub-categories associated with methodology-in- action, which emerged during analysis.
The template used to collect data, to provide case study description and to do analysis is based on the adaptation framework. Hence, a single model strategy is used.
The use of multiple models or theories requires a more demanding research design for different types of data collection (Allison, 1971). While this approach is often regarded as providing more accuracy and explanation of interesting events, often it leads to awkwardness and unsatisfying results (Langley, 1999) and development of a complex theory (Eisenhardt, 1989).
Besides, the results are complementary when multiple models are used (Baskerville & Pries-Heje, 2001). Hence, the single model was considered to be most appropriate to identify the agile method adaptation factors as accurately as possible.
The use of a single model does not overwhelm and confuse the software development community with different models, and with non-practical and varying constructs. Fitzgerald’s adaptation framework provides constructs and an analysis model, both relevant to theory and practice; it is an empirically grounded framework. This enables research convergence, hence achieving both theoretical and practical contributions (Markus, 1997), while maintaining simplicity (Eisenhardt, 1989).
Description and interpretation are two different tasks; interpretation activities followed after describing all the data in a meaningful way (Patton, 1990). Data interpretation (in-
case and cross-case analysis) answered the main research question “how does adaptation work in an agile approach”.
Patton points out that interpretation involves explaining the findings and attaching significance to particular results, and putting patterns into an analytic framework. The pattern matching technique was employed to explain the adaptation factors identified through empirical data using the key adaptation literature. This was used to confirm the internal validity of the research (Yin, 1994; Patton, 1990; Eisenhardt, 1989).
Creswell (1998) suggests that data analysis is a spiral activity (refer to Figure 5). This approach emphasizes that data analysis should take the following steps: (a) data is organised (b) data is scrutinized to make sense of it, (c) data is classified according to identified categories, and (d) synthesis- integrate; a summary that may include tables, figures etc to interpret the data. Using the approach described by Creswell, data analysis was done in a systematic manner as shown in Figure 5.
Figure 4: The main categories used to organise and analyse case study data.
Profile of Development Environment
Covert/Political Roles of Methodology Methodology-In-Action
- Rationale for adaptation
- Adaptation process
- Adaptation responsibility
Shape s
Influences
Overt/Intellectual Roles of Methodology Company Background
- Product development
- Culture for improvement
Figure 5: Data analysis approach based on Creswell (1998)
The identified steps for data analysis were applied first to write up individual case studies and in-case analysis for both case study organisations. The in-case analysis identified and explained the adaptation factors. Cross-case analysis provided a
perspective on agile method fragments and practices that are employed by the two case study organisations.
Patton suggests two tests for components used to categorize data: (a) internal
homogeneity- the extent to which the data that belongs in a certain category fits together in a meaningful way and (b) external heterogeneity- the extent to which the disparity among categories is bold and clear.
If a large amount of adaptation data was not able to be classified into any of the categories of the theoretical framework, it would have indicated a fault in the category system (Guba, 1978). According to Guba it would then have required working back and forth between the data and classification system to verify the meaningfulness and accuracy of the components and placement of data in components.
5. Report writing
4. Synthesize
Data interpretation, explaining the findings on emerging patterns
3. Classify
Categories data according to identified components of
Fitzgerald’s framework
2. Scrutinize
Make sense of data
1. Organize
Create database
Create meaningful description of data
To flesh out the patterns, Guba suggests the following: (a) build on information already known, (b) make connections with different information, and (c) recommend and confirm new information that will fit in. Patton (1990) suggests that this process finishes only when it is deemed that: sources of information have been exhausted, new sources lead to redundancy, and the investigator starts to go over the boundaries of the guidelines used for analysis. These tests, listed in Table 10 were applied to check for completeness.
Table 10 Test for completeness
Tests Description
1. Integration
2. Inclusive
3. Replication
4. Credible
Internal and external plausibility: (a) internal-individual components that identify different types of adaptation factors should be consistent and (b) external- the set of components should provide the complete picture of the adaptation of the agile method.
The components should be realistically inclusive of adaptation data that has been collected- the components of the framework should be able to match adaptation data from all cases that will be investigated. The other test is to match the components with the agile adaptation research questions and if there appears to be discrepancies between these two then the adaptation information about agile methods will be deemed to be incomplete.
Similar analysis results should be achieved by another independent capable investigator. The judgment that components of the
framework, that identifies adaptation factors, make sense and data on adaptation has been appropriately classified into various components is important.
The complete set should be convincing and realistic to the participants who provide the adaptation information.
Adopted from Patton (1990, pg 404)