Evaluating and combining methods

I have suggested that each of the discussed methods, DR, AR and case study research, in different ways, have some characteristics that fit my project and design approach. I therefore suggest that to be capable of addressing my research question and capturing the important characteristics of my production activity, the methods need to be combined.

AR clearly provides a method for research that is complementary to PD. Like PD it is able to cope with emergent questions and its cycles of reflection in collaboration with participants set an ongoing agenda for research as well as practice. AR with PD potentially brings a unifying to an action/intervention and research, the AR element providing an overarching framework for

drawing knowledge out of a (collaborative) action and providing principles for validity. Most valuably: PD and AR are valuable together in being adaptable to change. They need to be able to accommodate emergent questions and should follow opportunities.

AR has many resonances with PD and it is tempting to see AR as a natural partner for a PD project. AR and PD value participation and inclusivity, and acknowledge both the unique situation of research and the active role of the researcher. However, they have different aims in terms of their outputs. As Gillian R. Hayes points out, while the cyclic process of AR may look like an iterative design process, AR enables knowledge production and is much broader than PD which aims to produce an artefact (Hayes 2011). Foth and Axup highlight the different

approach to goals in PD and AR. They see PD as having set goals while AR is more open-ended and exploratory (Foth & Axup 2006). Other writers draw on experience to problematise

participation in research activities when they are twinned with production. Cal Swann acknowledges that the 'systematic reflection' aspects of AR are more difficult to achieve in a design project than participation in the creation of the designed artefact (Swann 2002). Hearn et. al also suggest that the principles of AR - that includes “all participants have a right to exert mutual influence over the research process” - represents an “ideal situation”, with the reality existing along a “continuum of participation”. They also point out that there are many barriers to equally high levels of participation in all aspects of a project (Hearn et al. 2009). In her case study on ACTION for Health, Ellen Balka describes discomfort when participants may feel they are “walking a thin line between end-user participation in design and consultancy in academic or research orientated projects” (Balka 2012 p.276). These writers point to difficulties in defining participation for research while strongly advocating for participation in design practice.

It is certainly my experience that participation in PD project is extremely demanding for all concerned, stretching people's time commitments and sometimes straining relationships when problems arise. Echoing Swann, asking people to additionally take part in reflective activities is difficult to ask among many other demands, and simply difficult for participants to fit in to their schedule. Furthermore, while participants will be genuinely invested in the practical outcome and its impact on their organisation, they may not be interested in making a contribution to the academic research in design and software development methods.

Nevertheless, giving careful consideration to the methods and values of AR have been essential in shaping this project and positioning the research. On conceptual level, AR has enabled this research to accommodate emergent themes and provided a model of research in which a broad theme may be present at the outset of research but within which the most salient topic might not at first be clear. In the Hublink case specifically, it could not have been predicted that the project would have been so long running, or that the design in use and adoption activities would have provided such fertile ground for research. Action Research, therefore, has provided a framework in which, through cycles of reflection on practice either individually or in groups, the research themes could be developed along lines that are responsive to the development of the project. In addition, the strong and inclusive principles of participant verification, and the consideration of the accessibility of academic outputs to participants, that are so important in AR have guided my interactions with participants around this research and had a positive effect on my relationships with participants.

Therefore, despite the difficulties I describe in combining AR and PD, I have reached an understanding of AR as having the capability to extend PD by drawing the participants into an additional iterative, reflective process that can generate knowledge of wider relevance while also providing an opportunity for practice to be enhanced by those shared reflections. The group reflective interviews which are a primary source of data in this study are based on this understanding.

Elements of the reflective methods developed by Douglas Schön(1985, 1990) and later Steven Scrivener (2000) encourage practitioner reflection. DR provides a way to link practice and research, explaining how design activity can itself generate knowledge through transparent reflection. The approach of Design through Research could be complementary for some PD projects, with its emphasis on the development of theory and the consequent development of new methods for practical purposes (Zimmerman et al. 2010). The values of reflection are strongly held by both PD and DR, however, DR on its own privileges the individual whereas PD seeks to value and reflect collaborative work and co-ownership of ideas and results. For my project, I have combined some individual reflection in keeping with DR, with reflections from participants, mostly stimulated by collective interview situations, that are more in keeping with AR.

A case study approach is appropriate to my research for several more reasons. It allows a focus on a single piece of practice in the real world, and does not restrict, though any predefined theory or method, the possible findings from that study. Moreover, in the PD context, the case study, loosely defined, is a common format. Therefore, though equally unique and situated, taken together case studies in PD build a body of knowledge that is applicable to further

practice and open questions for further work. The case study also provides a potential format for my research by allowing a deep description of a single case that is complex and multilayerd, and combines multiple sources of data and evidence. The case study approach provides a

framework for relevance, despite being unique and situated, by being part of a set of similar studies that together contribute to knowledge. It also provides a route to validity, by being able to combine different sources of evidence.

The research question identifies a link – between social and technical processes during design and those processes after design. From a PD perspective, it is the mutual learning characteristic that has the potential to make that link, and exploring that link must involve the participants. The researcher can observe phenomenon such as the developing interest and ability of participants in ongoing customisation. Those phenomena can be shown to exist via empirical evidence such as actual customisations that have taken place. But only the participants can tell us if and how the mutual learning outcomes of PD link to this ongoing 'design in use'. Design Research, therefore, with its emphasis on practitioner reflection is not enough to answer the question. Action Research provides a better and broader framework for participant inclusion but in practice, the context has not allowed for the capture of ongoing reflection by the participants in formats such as diaries. Nevertheless reflections have been captured through group reflective and evaluative interviews that have taken place through the process, and although participants have not guided research, key participants have reviewed and commented upon the project description. However, other artefacts such as emails and tickets are available whose texts may reveal further insights, but which are forms of evidence that do not fit easily

within Action Research. Therefore a number of qualitative research methods, as are expected in a case study and in Action Research but which are not expected to be seen in Design Research, can be used for this study.

In summary, therefore, the methodology for this research is the Case Study, with practitioner reflection (as suggested by Design Research), and collaborators' reflections (as suggested by Action Research). It is through these reflective methods that the research question and hypothesis have been narrowed, in the spirit of Action Research. However, in addition to the reflective element, my research seeks to collect and analyse other sources of evidence,for instance quantitative data gathered from the code repositories, issue queues and emails, which can be included in the case study.