The study protocol involved creating procedures for data collection and organisation to facilitate analysis and report writing.
The primary source for data collection was the interview technique. Observation was also used to determine the individual(s) to interview on a particular practice or issue. Discussions were held with the managers of both participating agile teams to determine the possible candidates. However, individual participations were voluntary. Table 6 and Table 7 provide a separate list of participants from the two case study organisations who took part in the study.
Table 6 List of participants from case study one (Akldevelopment)
Role in the agile team Company experience as at 2006) Key responsibilities Engineering manager
3 years experience with company but a total of 15 years of development experience including substantial agile
development experience
Responsible for all project deliverables of the agile team. Works closely with product strategy group and senior executives of the company to determine new feature developments. Co-located and takes part in all the development activities of the agile team. Other activities include field trips (overseas client sites), working with sales group to host potential clients and taking part in sales negotiations.
Senior product analysts (was product manager) 15 years of development and product management experiences with the company
Part of product strategy group but work full-time in agile development team. Plan and manage product backlogs, write user stories (development-task), write acceptance test, test implemented stories, provide domain support for engineers and plan sprint cycles.
Product analysts 5-6 years of
development experiences with the company
Part of product strategy group but works full-time in agile development team. Plan and manage product backlogs, write user stories, write acceptance test, test implemented stories, provide domain support and plan sprint cycles with engineers.
Principal engineer 1 year with the company but 20 years of development experiences including 6 years of agile development experience
Provide team and technical leadership, and team coaching. Provide input into vision plans. Take part in product backlog and sprint planning sessions and scrum meetings. Implement user stories in sub-teams.
Senior engineers 6 or more years of
development experience with the company
Provide team and technical leadership, and team coaching. Provide input into vision plans. Take part in product backlog and sprint planning sessions and scrum meetings. Implement user stories in sub-teams.
Engineers 4 or more years of
development experience with the company
Take part in sprint and scrum planning meetings. Implement user stories in sub-teams.
Table 7 List of participants from case study two (Meldevelopment)
Role in the agile team
Company experience as at 2006)
Director of software engineering
10 years of development experience with the company
Responsible for all project deliverables of the engineering department. Works closely with product planning group and senior executives of the company to determine new product developments. Co-located and takes part in all the development activities of the team.
Product manager 6 years of development and
product management experiences with the company
Part of sales and marketing group, co-located and works closely with project mangers and development teams; provides high-level requirements. Plan and manages vision and roadmap plans. Takes part in design phase with engineers.
Project managers (became product development managers)
10 years of development and project management
experiences with the company
In-charge of projects and are official team leaders. Work closely with product mangers to implement vision plans. Plan and manage product backlogs, provide domain support, plan iteration cycles and take part in daily stand-up meetings with engineers. As product development managers they also work in field (product management function).
Principal engineer 7 years of development experiences with the company
Provide team and technical leadership and team coaching. Provide input into vision plans. Plan product backlogs and iterations, and attend daily stand-up meetings. Responsible for product architecture and enhancement of products. Implement tasks in sub-teams.
Senior engineers 2 to 20 years of development
experience with and outside the company
Provide team and technical leadership, and team coaching. Provide input into vision plans. Plan product backlogs and iterations, and attend daily stand-up meetings. Implement tasks in sub-teams.
Engineers 2 or more years of
development experience with the company
Plan product backlogs and iterations, and attend daily stand-up meetings. Implement tasks in sub- teams.
Quality assurance manager
7 years of company experience
Overall responsibility for quality of product releases. Works closely with project managers, product manager and documentation team.
Quality assurance engineers
2-4 years of company experience
Test iteration builds, take part in design phase (backlog planning), support software engineers to implement unit tests and coach them with quality assurance activities.
Communication engineer
Over 10 years of company experience
Provide documentation for releases. Coach and provide information to the engineers on usability issues. Take part in design phase.
Marketing manager
2 years of company experience (substantial marketing experience outside the company)
Provide product information to the engineering team. Attend stand-up and design meetings.
The participants who took part in the study were co-located with their agile teams. Case study one had made the quality assurance team redundant on their agile adoption and the documentation team worked separately from their development teams. The study participants from both case studies were agile method users and were empowered to
adapt their methods. Senior executives of both case study organisations were not requested for the interviews because of their busy schedules and field commitments. Likewise, the usability engineer with case study one could not be interviewed.
The following were the interview instruments or key themes based on which individual interview questions were formulated; product development prior to agile adoption; agile method adoption, adaptation and agile culture; planning practices; product backlog practices; test-driven practices; development setup, teams, roles and skill set; short development cycles and project sizes; integration practices; code development practices; domain knowledge support; development tool support; empowerment and productivity practices, training and learning practices; and communication and interaction practices.
Each interview session was planned for an hour and an additional session was scheduled if an interview was not completed in a session. Interview sessions were held in the meeting rooms at the production lab of the case study participants. In total forty sessions were held per case study organisations. Table 8 and Table 9 provide a separate list for the two case study organisations indicating number of interviews held with each participant.
The requests for interviews were made through the managers of the agile teams of two participating organisations. A timetable for interviews was agreed with both
organisations. Initially, it was planned that all interviews were to be done within a six month period. However, this was extended to over a year because of the changes in product development commitments of both teams.
The following principles for interview technique were applied as suggested by Marshall & Rossman (1999), Yin (1994) and Patton (1990). A semi-structured interview method was used allowing the interviewees to explain their method practices. Observation was used to see methods in practice (achieved data triangulation). The best possible
informants were used and interviews focused on method practice rather than personal belief. Questions were formulated as single questions. Interviewees were informed of their right not to answer any particular question they were not comfortable with. Participant anonymity was also ensured. The impact on their work schedule was minimised by having interviews early in the mornings, late afternoons or during lunch time (as convenient to the individual), and limiting the interview sessions to one hour at
Secondary sources for data collection were also employed. Interviews done by company executives for business publications and made available on the web were also used. These were validated with the team manager. With both case study organisations, it was also agreed that the case study report rather than individual interviews would be
validated to minimise impact on their development commitments. Agreement was made on recording of interviews and individual consent was taken before a session began. These recorded interviews were later transcribed. Recording enabled accuracy for data collection and to be more focused on the interviewee (Patton, 1990). However,
important notes or emerging questions were noted down during the session.
The interview questions were knowledge questions about their agile product
development environment, development practices and related adaptation. Interview questions also included background questions relating to their organisation and previous development environments and development problems (Patton, 1990).
Prior to an interview session all questions were read and revised. Questions were also read out to colleagues to ensure their clarity. Any probing or follow-up questions resulting from the previous session were included.
Table 8 Number of interviews per participants at case study 1 (Akldevelopment)
Table 9 Number of interviews per participants at case study 2 (Meldevelopment)
Role in the agile team Number
interviewed
Interviews per Individuals
Total interviews
Director of software engineering 1 7 8
Product manager 1 1 1
Project managers
(became product development managers)
2 3 & 4 7
Principal engineer 2 2 4
Senior engineers 4 2 8
Engineers 2 2 4
Quality assurance manager 1 2 2
Role in the agile team Number
interviewed
Interviews per Individuals
Total interviews
Engineering manager 1 9 9
Senior product analysts (was product manager) 1 2 2 Product analysts 2 2 4 Principal engineer 1 5 5 Senior engineers 6 2 12 Engineers 4 2 8
Quality assurance engineers 2 2 4
Communication engineer 1 1 1
Marketing manager 1 1 1
3.8.2 Data organisation- case study database
A process was adopted to create a repository for all the data collected for this research. This repository facilitated systematic organisation of data from various sources into categories to scrutinize data and make sense of it (Creswell, 1998). Systematic data organisation established a chain of evidence linking research question, empirical
evidence and analysis approach used (Maxwell, 2005; Lincoln & Guba, 1985). This was done to ensure research validity.
Each interview was tape recorded, labelled (given a number and job title) and
transcribed. The transcribed interviews were saved as Microsoft Word documents using tape number and interviewee title. This labelling allowed for quick access to the
appropriate tape to validate transcription with audio. Separate folders were created for two case study organisations to save the transcribed interviews as Word documents. Any other relevant information (electronic) was kept under respective folders.