Chapter 3: Methodology and Methods
3.3 Methods
3.3.2 Streams of evidence to be considered
In case study research a number of different ‘data streams’ may be utilised in order to gain an in-depth understanding of that case. These can include quantitative surveys, interview data, participant observer data, user logs, archive material, external
publications (e.g. media articles, web content) and audio-visual material. For each online citizen science project I have selected three different data sources to address the
research questions. While some information is already available on the project websites and from external publications there is not enough that addresses my areas of interest and so data has been generated through questionnaires and interviews.
66 The following sources of data have been considered for all three cases and incorporated into the research analysis.
1. Information and experiences gathered through being a participant-observer in each of these projects;
2. Online surveys of citizen scientists taking part in each project;
3. Qualitative, semi-structured interviews of a sub-set of citizen scientists who completed the survey, and of the professional scientists and developers involved in managing the projects.
The timeline for data collection / generation for each of the projects is illustrated in Table 3.4.
Table 3.4 Timing of data collection and generation
Project Date I joined project as a participant Online question naire launch Follow-up interviews of citizen scientists Follow-up interviews of project scientists
Foldit April 2012 June
2012
Jan – Feb 2013 June 2012 - Feb 2014
Folding@home Feb 2012 March
2013
Jan – Feb 2014 Feb 2014
Planet Hunters Jan 2011 23 May
2013
Nov 2013 – Jan 2014
Jan-Feb 2014
Data was collected over a number of months, and the approach I have adopted can be described as ‘exploratory sequential’ meaning that there is a sequence of data collection, with the results obtained through one method informing the approach or content of another (Guest, 2013). For example, online survey questions have been informed by
23
I initially took part in Planet Hunters shortly after its launch when it was highlighted during the 2011 BBC Stargazing Live programme. This is a programme designed to promote astronomy and often features amateur astronomy and citizen science activities. While this involvement only lasted a few weeks, I became a more active participant in February 2012.
67 participant observation, while the interview schedule has been influenced by the
questionnaire data and my experience as a participant observer. The process of conducting surveys and interviews has influenced my experience as a participant observer. Figure 3.3 illustrates how the data collection processes have been interconnected.
Figure 3.3: Data collection process (‘exploratory sequential’)
Participant-observer role
I have been a participant observer in all three case studies. In one of the projects, Planet Hunters, I began participating prior to this research (See Table 3.4) and had become familiar with some of the other Zooniverse projects as part of an MSc dissertation on astronomy outreach (Curtis, 2013). Being a participant observer is a direct way of
68 exploratory technique, as a supplementary or additional source of information (as in this research) or it can form the basis of a research project resulting in the production of a detailed ethnography (Hammersley and Atkinson, 2007).
Participant observation has a number of advantages, namely, that it gives the researcher access to the ‘background culture’ and can enable a detailed description of behaviours, situations, intentions and events that they may not have access to otherwise (Kawulich, 2005). However, there can be problems with this research tool. For example, the researcher may be reliant on a small number of informants, and the experiences and views of these individuals may not be typical of the community in question (Kawulich, 2005). Perhaps one of the most important considerations is researcher bias. The researcher must be aware that their gender, ethnicity, class and theoretical approach may affect their observation and interpretation (Iacono et al., 2009).
The degree to which a researcher becomes integrated into the community can vary greatly, and some researchers talk of a ‘continuum of participation’ (DeWalt and DeWalt, 2002) or ‘boundaries of insiderness’ (Labaree, 2002). Determining the degree of
participation and membership of the group is sometimes done by the researcher, and sometimes it is decided by the community (DeWalt and DeWalt, 2002). The fact that the communities that I was observing were online also influenced the degree to which I could become involved. For example, much of the communication with other participants was asynchronous and conducted through online forums or discussion boards. This alters the temporal nature of exchanges, and can create distance between participants (James and Busher, 2009a). Interactions that took place during real-time synchronous internet relay chat felt more spontaneous and immediate. The project tasks also influenced the degree to which I could enter the project community. For example, one project task (Foldit) was
69 quite complex and necessitated much more interaction with other participants (in order to learn how to complete the task) than in the other two projects.
The degree to which I achieved membership varied. Table 3.5 summarises the ‘continuum of participation’ as proposed by DeWalt and DeWalt (2002) and I have marked where my involvement with each of the projects is situated in this continuum.
Table 3.5 The continuum of participation (from DeWalt & DeWalt, 2002)
Non-participation No membership role.
Observation only. Researcher remains external to the community
Passive participation No membership role.
Researcher is on the spot, but acts as an observer (as a spectator or bystander)
Moderate participation
(Folding@home)
Peripheral membership.
Researcher is present at the scene of action but does not participate, or only occasionally interacts with people.
Active participation
(Planet Hunters)
Active membership.
Researcher engages in almost everything that other people are doing.
Complete participation
(Foldit)
Full membership.
Researcher is, or becomes a member of that group.
This approach describes the various roles that the participant-observer may take. The roles range in degree from non-participation (activities are observed from the outside), to passive participation (activities are observed in the research setting but without
participation in any of the activities), to moderate participation (activities are observed in the research setting with occasional participation on activities), to active participation (activities observed in the setting with almost complete participation in activities), to complete participation (there is complete participation in the ‘culture’). While the degree of participation varied between projects, this did not affect my ability to collect data or to make observations of other participants.
70 For each project a similar participant-observer protocol was adhered to (for Planet
Hunters this protocol was followed as soon as I received permission from the project managers to carry out my research in February 2013). This is summarised in Table 3.6.
Table 3.6 Participant Observer Protocol
1. Register with project.
2. Download software, complete tutorials on project task.
3. Register (when required) with online forum, and become familiar with project etiquette (e.g. online codes of conduct, presence of moderators).
4. Observe content and frequency of interactions between participants on online forum, internet relay chat (if available), and on project blogs. Identify key participants. 5. Attempt to estimate active participants in the project.
6. Take part in online forum discussions and internet relay chat when relevant or of interest (state research interests and affiliations on all first forum postings).
7. Explore other online project content produced by either participants or professional scientists (e.g. wikis, FAQs).
8. Participate regularly (4-5 times a week on average).
9. Write summaries of observations, reflections and of any interactions in project diary during active participation.
10. Compile a library of ‘screen shots’ to illustrate the online setting. These will supplement notes, and provide graphical illustrations of the areas where participants interact. 11. Share research findings with participants.
Acting as a participant-observer was the first stage in data collection. It enabled an exploration of the individual projects and provided an insight into how the projects were organised (see Chapter Four). This experience informed both the content of the surveys and the interview schedule, as well as providing a context for these findings during the subsequent analysis. Participant observation continued throughout the period of the research, although it was at a reduced level during the final stages of analysis and writing. By continuing to participate, I retained my familiarity with the projects, kept abreast of new developments, and maintained contact with some individual participants. The
71 participant-observer protocol was formulated at the beginning of the research period with the exception of steps five and nine which were later considerations. Step five became important once it became clear that the number of active participants was significantly smaller than the number of registered participants. Step ten was introduced as the importance of screen shots in illustrating participant interaction began to emerge.
Online surveys of project participants
Online surveys can help a researcher reach a potentially large number of participants at relatively little expense. They can be set up quickly, and there are a number of online tools available to facilitate their development and administration (Kozinets, 2010). These tools also enable results to be effectively collated, and some enable basic statistical analyses to be undertaken (Sapsford, 2007). Many packages allow a significant degree of flexibility in layout and type of question that help the researcher tailor the questionnaire design to their specific research questions.
In addition to addressing issues relating to motivation and interaction, the online surveys were also used as an opportunity to collect demographic information about the
participants which may have a bearing on the topics under consideration (particularly motivation to participate or how they perceived their contribution to the project). The results of the online surveys were later used to inform the direction and content of the interviews with both scientists and citizen scientists.
The design of a questionnaire must be carefully considered (Meadows, 2003, Tourangeau et al., 2000). A survey should not be too long, and it should be clearly set out and easy for the respondent to understand (Adams and Cox, 2008). I was also aware of the options relating to style of questions and did consider the use of Likert scales instead of more open-ended questions. Likert scales have been used in some previous work that explores
72 motivation to participate in online citizen science, and can enable a more quantitative analysis of results (Nov et al., 2011, Raddick et al., 2013). However, there was some concern that they may be too prescriptive. There is the possibility that if the researcher is not inclusive enough in the options / statements presented to respondents that
something that was not previously considered may not be brought to light (Carifo and Perla, 2007, Jamieson, 2004, Pell, 2005). Open-ended questions give respondents greater freedom to express themselves, although the responses may well be more time-
consuming to analyse.
A questionnaire was designed that contained a mixture of closed questions and open- ended, opinion-based questions. The closed questions related to demographic
characteristics such as age, sex, level of education, and indicators of their involvement, such as how long they had been with the project, and how many hours per week they spent on the project. The open-ended questions related to their opinions and attitudes. For example, why they took part in the project, what they liked best about the project, things they would change, and how they would describe their interactions with other players. The design of the questions was informed by my participant-observation of each project, as well as previous research, and my own research questions.
Some questions were specific to certain projects. For example, the Foldit questionnaire asked participants whether they played any other computer games, as participants are regularly portrayed as ‘gamers’ in external media articles (Bourzac, 2008, McGonigal, 2012, Gross, 2012). In all instances, I offered the project scientists the opportunity to suggest one or more questions for inclusion in the survey. As they were permitting me access to their project, it seemed appropriate to offer them the chance to explore
73 to suggest a question, and they asked participants if they had had any technical
difficulties with the latest version of the project software. Each questionnaire had an introductory page that outlined who I was, my institutional affiliation and the objectives of the research (Appendix A). The secure storage of data and the fact that data would not be passed on to third parties was clearly stated, as was the stipulation that only those aged 16 and over should complete the survey. This introductory section stated that I would share my research findings with project participants, as well as with the project scientists and the wider academic community24. A copy of each questionnaire can be seen in Appendices B-D.
A number of tools are available for constructing and carrying out online surveys. I used the Bristol Online Surveys (BOS)25 tool as this is widely used by the Open University to carry out its own student satisfaction surveys. All Open University postgraduate researchers are covered by a site license and there are personnel available to provide guidance and support. Once a final version of the survey was constructed, a test version was sent to each project team, who tested it with their own responses, and then
approved it. A hyperlink to each survey was produced, and made visible to project participants via a number of channels. Table 3.7 illustrates where the survey links were posted for each project, and the final number of respondents.
24
Findings of the online surveys have been shared with key contacts in each of the project teams. A summary of the survey findings has been prepared for participants in all 3 projects. This has been shared by the project manager at Folding@home, and with those who took part in the Foldit survey. The Planet Hunters managers have stated that they intend to post a link to my findings on a blog post sometime in the near future.
25
74
Table 3.7 How participant surveys were advertised and final numbers of respondents
Project Where survey link was placed Final number of respondents
Foldit Online forum on website.
The ‘what’s new’ section that is visible when game is launched.
37 Folding@home Online forum on website.
Project blog.
Folding@home Twitter feed and Facebook page.
407 Planet Hunters Online discussion boards.
Email to participants.
Planet Hunters Twitter feed and Facebook page.
118
One of the main problems with online surveys is that the response rate can sometimes be poor (O'Brien and Toms, 2010). The number of respondents for the three surveys is relatively small in comparison to the overall number of registered participants, so these groups of respondents may not be representative of the entire population (see Table 3.3). However, the small sample sizes may also be related to the fact only a relatively small proportion of the total registered users are contributing on a regular basis, and these sample sizes may therefore constitute a greater proportion of active participants. The fact that survey respondents were recruited through project forums and discussion boards also supports the likelihood that these individuals were actively making
contributions to the project at the time of the surveys. Ascertaining the proportion of
active participants is an important component of understanding who participates in these
projects (RQ1), and an estimate of this figure will help to characterise the types of contribution (RQ5). A greater of understanding of registered vs. active participants will also help to place these samples sizes into the appropriate context and estimates of the active population for each project are presented in Chapter Four.
Those who participated in the survey are a self-selected sample as some individuals are more likely to respond to a questionnaire than others, or will have a greater interest in its
75 subject(s) (Sterba and Foster, 2008, Tourangeau et al., 2000). This survey could also be described as an example of ‘convenience sampling’, as it is composed of individuals who were available and chose to make themselves accessible to the researcher (Battaglia, 2008, Castillo, 2009). This could also result in the groups of respondents not being representative of the population of active participants.
On average, the surveys remained open for two months (this ranged from 1-3 months). Survey administration was relatively straightforward with the exception of the Foldit survey. Approximately two weeks after I launched this survey, another researcher (an undergraduate from a UK university) posted a request for Foldit players to take part in some interview-based research exploring motivations for playing. Several Foldit players complained to one of the project scientists about the presence of too many external researchers and I was told that the link to my survey would no longer appear on the Foldit start-up page (which would have been viewed each time a player launched the game). However, I was still free to post a link to my survey on the general forum. This illustrates some of the problems with online research and the fact that some project participants are not always comfortable being the subject of someone’s study. This can directly affect access to subjects, and it can also impact on the subsequent relationship between
subjects and researchers. It also illustrates the importance of ‘gatekeepers’, and that their support may be required in order for the researcher to gain access to a potential pool of respondents.
76 Interviews of project participants
The next part of the data generation process involved semi-structured interviews with a sub-set of survey respondents, and with some of the scientists and developers26 involved in setting up and managing the projects. An interview gives the researcher the
opportunity to explore issues in greater depth than a survey or questionnaire, and data is generated through the interaction between the interviewer and interviewee (Holstein and Gubrium, 2003). Indeed, Talmy (2011) describes an interview as a “collaborative achievement”. Interviews may also provide a way of verifying some of the findings of the surveys, and thus serve as tool for both methodological and data triangulation (Gillham, 2000). Interview questions should be ‘open’ giving the interviewee freedom to express their views, while the semi-structured nature of an interview can give the researcher flexibility and freedom to follow-up other topics of interest that come up during the discussion (Gillham, 2005). This was the approach taken in my research, and I aimed to design research questions that were general enough to enable respondents to talk around a point of interest, yet retained a focus on the overarching themes and objectives of this research. However, the researcher needs to direct the interview and can use prompts to steer the conversation back to the areas of research interest (Bryman, 2012). The interview questions are listed in Appendices E and F.
Interviews have ‘traditionally’ been carried out face-to-face, or perhaps over the
telephone (Shuy, 2003). However, when the focus of research is an online citizen project and participants are in numerous geographical locations and time zones, other
approaches had to be considered. There are now a number of online and internet-based
26
By developers I am referring to those involved in designing, maintaining and updating the project