Research Question 1: What is the perception of experienced programmers on the peer behaviours that help or hinder them in their own tasks?
Objectives:
1. To collect opinions from experienced programmers working in a variety of application domains about the helps and hindrances that affect them in the breadth of their everyday work.
2. To identify common themes in their responses.
This section outlines the design considerations for a study to address the objectives of Research Question 1, repeated here for convenience. The resulting design is a study in which experienced professional programmers are interviewed about real-life experiences in their work and asked to reflect on how the work is affected by the behaviours of their peers. This study, referred to throughout as the Exploratory Study, is described in Chapters 4 and 5.
3.3.1 Participant demographics
Research Question 1 seeks to identify patterns in the opinions of experienced programmers. New graduates do not yet have all the skills needed to work effectively in a commercial software development team (Radermacher, Walia, & Knudson, 2014; Johnson & Senges, 2010), so to eliminate any issues that are specific to novices the participants need to have enough experience to have developed a degree of “developer fluency”. This is a term used by Zhou and Mockus (2010), who suggested that
fluency continues to grow for at least the first three years if taking into account the increasing difficulty of the tasks that can be assigned to novice developers as they gain experience. Job advertisements for “experienced” programmers do not always specify how much experience; of those that do, some demand at least five years while others ask for as little as two. All the managers questioned by Zhou and Mockus (2010) would be willing to commit customer-critical and mentoring tasks to developers with five years’ experience, so this is chosen as the criterion for the study. By this stage of their careers developers are likely to have encountered different colleagues’ styles and be familiar with tackling significant tasks involving existing code. To gain an overall picture of participants’ career experience, a questionnaire will be used.
3.3.2 Elicitation
Observation or introspection
Identifying patterns in the opinions of experienced programmers calls for the research to include a breadth of programmer perspectives across a variety of types of software application. It precludes detailed case studies or observational approaches such as ethnography or participant observation (Symon & Cassell, 2004, chapters 26, 25 & 13), which are particularly suited to gathering rich, in-depth data in the context of a particular environment but not appropriate in this research. Here it is not the setting but the individual’s perceptions that are important, synthesised from any and all of the environments they have encountered during their whole career. Seeking this “long view” over years of experience also has practical implications. Events happening during a workplace observation, for instance, would afford an excellent opportunity to record in great detail the nature of an issue and its consequences, but the data would be limited to those occasional incidents. Such an approach would be more feasible as part of an extended participant observation, but even then would only yield data for one location and a limited number of projects.
If the issues cannot be observed in person, an alternative is to ask programmers themselves to reflect on circumstances in which they perceive that they have been helped or hindered in their work by others’ behaviour. A questionnaire would be a fast way to canvas the opinions of a large number of respondents, but structured questions cannot capture the richness of data required (Coolican, 2014, p.170). This research seeks not only to catalogue the behaviours which are perceived as having most impact (whether positive or negative), but also to understand why this is the
case. A questionnaire could investigate the former with a simple Likert scale, but is not well suited to collecting the latter. Even if respondents took the time to a enter a sufficiently detailed account in a text box asking them to explain the impact of important behaviours, there would be no opportunity to seek clarification or pursue interesting aspects of their response in more detail.
Rich and detailed introspection
To fully explore the reasons why as well as cataloguing which peer behaviours have most impact requires a more personal approach than a questionnaire — a one-to-one interview in which a rich and frank account can be elicited (Gray,
2013, pp.382–388). The interviews need to remain grounded in the everyday
details of participants’ workplace experience; without those details, the account risks becoming generalised and over-simplified rather than explicitly explaining the symptoms and their consequences, much as good practice maxims like “seek to write clear, self-documenting code” (Goodliffe, 2006) are insufficient without the accompanying explanation of the principles behind them. It is helpful in this respect if participants can talk to a fellow specialist for whom they do not need to interpret (and therefore simplify) their very technical world, but can give detailed accounts using the vocabulary of the profession. Coolican (2014, p.179) explains the benefits of an interviewer understanding the normal language mode of the community; it prevents misunderstanding, and interviewees are “at their most comfortable and fluent” when using it.
In this research, understanding the language of the profession is also a prerequisite for the ongoing data analysis that occurs during the interview process:
“[T]he interviewer is not just listening and recording. They are or should be attempting to establish the full meaning of the respondent’s
account from their point of view. For this reason the qualitative
interviewer is not a passive recipient of information but needs to be active in checking what the interviewee is saying, watching for inconsistencies, encouraging fuller detail where a story is incomplete and generally keeping the research aims in mind throughout the process.” (Coolican, 2014, p.177)
In interviews which seek detailed accounts of a technical specialism, a lack of subject knowledge would make such checking and follow up questions difficult. It would also affect the story that participants tell. Interviewees respond differently depending on who the interviewer is; this includes social categories such as age and gender (Miller
& Glassner, 2004) but also their role (Coolican, 2014, p.171). The researcher has the relevant experience to interview participants about their work, although this comes with the risk of her own experience influencing their responses by the form in which questions are asked (see §3.5). Interviews by the researcher herself are therefore chosen as the appropriate means to elicit rich and reflective accounts of participants’ workplace experiences, with awareness of this risk reflected in their design.
Facilitating recall
The interview will use what Tashakkori and Teddlie (1998) described as a ’funnel
interview’: starting with open-ended questions before moving on to a more
structured format. The interview will thus be designed to first capture the thoughts that spring freely to participants’ minds and then move on to structured techniques to facilitate further recall.
Just as observational methods would be limited to capturing only the situations that happen to arise, interview techniques can only collect what participants actually recount. Cognitive psychology has shown that recent events are more accessible than older ones (Baddeley, Eysenck, & Anderson, 2015). Autobiographical recollection is also better for emotional events, whether positive or negative, and unusual incidents that stand out (Bower & Forgas, 2000). Cued recall — asking people to recall a memory associated with a prompt, such as a simple word or a particular period of their life — has long been understood by cognitive psychology as an effective means of probing for memories (Baddeley et al., 2015).
There is a problem, too, with implicit knowledge (Berry, 1987): things which are so “obvious” that they are not usually consciously articulated. Aids are therefore useful to help prompt participants to recall a wide range of their experiences at work. Card sorting offers a means of tapping into implicit knowledge; it has been used, for example, to explore mental constructions of programming concepts (e.g., Sanders et al., 2005). It is also a tool within the computing industry itself, where website builders use it to understand how users categorise information so that the website can be structured in a way that makes sense to them (Usability.gov, 2013). It is an engaging and informal process which also has the advantage of helping to “solidify” the experience (Chope, 2015); physical artefacts such as cards do not just carry content but also facilitate the communication process itself (Sharp, Robinson, & Petre, 2009). A card sorting task is therefore included as a part of the interview procedure to cue participants to reflect on the breadth of their experience and
articulate their thoughts about it. This has similarities with a Q sort in eliciting first- person perspectives by asking participants to rank a set of items in order of personal significance (Watts & Stenner, 2012) but differs in the use that these are put to. Q methodology applies a factor analysis to reduce the list of items to a smaller number of factors and thus recognise individuals who share a similar perspective. In this research, the goal of the card topics is to prompt recall.
The process of ranking of the cards, although it will also produce quantitative data on what matters, is intended as an aid to help participants reflect deeply on their career experiences and elaborate on why the topics presented have an impact. Interpretative statistics would be inappropriate for a method designed as a means to provoke thought and discussion rather than as a quantitative measurement. Descriptive statistics, however, will be helpful to illustrate the trends in the sorting task and the topics which dominated the subsequent discussion.
3.3.3 Analysis of common themes
The methods selected to elicit programmers’ perceptions about peer behaviour are chosen to produce rich data which addresses the full breadth of tasks involved in their work. The approach taken not only seeks to find out what matters to them, but to delve more deeply to understand why.
Some of their responses might be anticipated. Others might be unforeseen or express different priorities to those emphasised by the field of programming good practice. Indeed it is important that the opinions expressed should not be constrained by the researcher’s own experiences or training as a programmer. This research calls for an approach to analysis which can accommodate both anticipated and new themes. Template analysis (King, 2012) is chosen for thematic analysis of the interview data because it flexibly accommodates both an open discussion, in which participants can raise any topic, and themes anticipated prior to the study. “Bottom up” approaches such as interpretative phenomenological analysis (IPA) and grounded theory avoid any prior theorising, preferring themes to emerge only from the data. Matrix analysis and other “top down” analyses, in contrast, define themes from a theoretical basis and apply these to the data (King, 2012). Template analysis uses such a priori themes more provisionally: as coding progresses they are refined and new codes are added, as needed. Interviews can be coded until the data does not give any new information or possible codes.
Template analysis
Template analysis begins with preliminary coding of a subset of interview transcripts. Material relevant to the research question is coded either to one of the a priori codes (refined as necessary) or to a new code. The end of this stage is reached when the coding no longer produces distinctly different new themes. The modified a priori codes and the new codes are then organised under a smaller number of higher order codes describing broader themes, creating the initial template. The initial template is then developed by applying it to remaining transcripts, modifying the template when text relevant to the research question does not fit within an existing theme. Significant changes to a template require re-coding of earlier work. This development of the template continues until no further new and relevant information is emerging (i.e. saturation is reached and reading of the remaining transcripts yields no new material that could not be coded to the existing template). The template at the end of the process is the final template, a hierarchical model of the themes.