Evidence and Data

Yin identifies six main forms of evidence that can be used in a case study. Namely:

documentation, archival records, interviews, direct observations, participant-observation and physical artefacts (2009 p.101) . Researchers are encouraged to analyse the strengths and weaknesses of the different forms of evidence, and combine them in ways that can mitigate their weaknesses. Yin presents a table that evaluates the strengths and weaknesses of each source.

Yin also encourages researchers not be be restricted by this list and to look as broadly as possible to any forms of evidence. He even suggests the value of seeking out conflicting evidence. As a practice based research its clear that the possible sources of data are indeed wide and diverse, and with many different characteristics. This should be taken as an overall strength of the study.

Table 1 lists the possible sources of evidence in my project and their strengths and weaknesses. I adapt Yin's 'source of evidence' categories to be specific for my project. I use some of his terms of analysis of the strengths and weaknesses of the sources (Yin 2013), and add some of my own to be more specific. The two major sources of evidence are additional to Yin's published table but are the most significant data sources for my research. The first of these sources are the 'reflective interviews' that took place in March and April 2014. These interviews were arranged specifically to reflect upon the project processes. It was decided to do these interviews in a group situation to be consistent with the collective working that characterised the project. 3 interviews took place which were audio-recorded, reviewed several times and partially transcribed. In addition a group interview took place in November 2015, at the conclusion of the research element of this project. This was also reviewed and transcribed.

The second main source of evidence is the quantitative data in which digital data which is collected to facilitate production was re-used for research purposes. This style of research is inspired by work in 'digital humanities' (Spiro 2012; Kirschenbaum 2012) in which data pre- existing on digital platforms is re-purposed for research. For this project, the commit logs from git5 _{were analysed and visualised to gain insight into the timespans of changes to the code base} from the Drupal community on the one hand and to custom code on the other.

Data was also analysed and visualised that looks at the types of issues raised with the

developers from the team at Real. The source of this data was a mixture of data automatically generated from the issue tickets system, plus data added by reviewing emails. This was necessary because not all issues were logged on the ticketing system. The raw data is available as supporting material to this thesis, and the process for generating and analysing the data is included with the data in section 6 of the project description.

The theme of mutual learning and tailorability had emerged through practice rather than through research data so 'grounded theory' methods were not appropriate or necessary. The interview data was reviewed for relevant references to mutual learning. Quotes from participants that illustrated the foundational principles of PD were also identified for inclusion in the project description document, especially those that illustrated important contradictions or paradoxes. The main source of validity for this study comes from participant checking. Key participants6 were sent copies of the project description for review and invited to respond to the following questions:

5 See section 6 of the Hublink project description for more explanation of Git.

6 Mike Smith, Karen Linnane, Cathie Duncan, Edward Pickering, Ailidh MacCloed, Kate Lomax, Paula Graham.

- Do you think this is an accurate account of our work together? - Have any important points been missed or mis-represented? - What do you think of the summary/conclusions?

- Do you have any additional comments?

They were also asked if they could give their permission for quotes to be attributed to them personally. The case study paper about Hublink presented at PDC 2012 (Haskel & Graham 2014) was fully checked by all participants who gave their agreement to the account. They also gave permission for their names and and photographs to be used in the conference presentation.

Source of Evidence Strengths Weaknesses

General administrative documentation eg. Emails, agendas, follow up to meetings - stable - unobtrusive - exact - broad - easily reviewable - biased by incomplete selection

- access can be withheld - reflects unknown biases of author

Participatory Design & training workshop

documentation eg. Agendas, meeting notes, audio recordings

- real - contextual - insightful

- easily reviewable

- difficult to analyse for research purposes - targeted to production

Written eg. Feature requests/discussions, bug reports - real - contextual - exact - reflect participation - reviewable, but not easily

- pragmatic rather than insightful

- targeted to production

- incomplete as complemented by phone calls/meetings - difficult to analyse Design artefacts eg. Workshop

results, paper prototypes with notes, working prototypes

- real - contextual

- insightful into technical considerations - record participation - reviewable - targeted to production - incomplete without accompanying discussions

Diaries and other self- documentation - insightful - reflective - targeted to research - contemporaneous with events

- reviewable, but not easily

- subjective - incomplete

Reflective interviews (participants)

- targeted to research - insightful into context - easily reviewable

- bias due to poor questioning - participant bias

Questionnaires - targeted to research

- easy analysis for research -easily reviewable

- bias due to poor design - narrow in scope Technically derived evidence

eg. version control logs

- exact - reviewable

- insightful into technical operations

- not insightful into usage context

- possible bias in categorisation

- possible bias in data interpretation