Step 4a – Select methods

At first glance there are two methods which are likely to significantly contribute this research. Action Research allows for the researcher to be positioned within the context of the study of an intervention, whilst Design Science encourages cycles of exploration and validation as a process towards emergent artefacts. All these aspects are

applicable to this work. To explore the likely contribution of each method towards this research both were thoroughly explored.

Action Research

A common approach for research into education is action research. There are many variations on action research which are diverse (Hult and Lennung 1980; Tripp 2003, p.2). Kemmis and McTaggart (2005, p.273) identified many different ‘families’ of action research including one which emphasises the practical element, and is the most appropriate to this study: ‘classroom action’.

When to use Action Research Corey Corey (1952) believes that action research is conducted where a hypothesis says that the research approach results in better decisions and actions than if decisions are based on intuitive and subjective opinion. Action research problems never emanate from theory alone but emerge from a desire to solve a practical problem (Hult and Lennung 1980). It requires a rational and systematic examination of a phenomenon with a view to problem solving, competence

enhancement and theory expansion (Hult and Lennung 1980; Zuber-Skerritt 1996)

Action research can be viewed as an iterative structure of ‘self-reflective cycles’

including several stages such as planning, action and observation, and reflection (Lewin 1946, Tripp 2003, p.14, Kemmis and McTaggart 2005, p.278). The research is

completed within the system under scrutiny creating an immediate impact, without the need to wait for findings to be published prior to action (Hult and Lennung 1980). Of course, making the findings public is what makes researchers accountable

(Zuber-Skerritt 1996) and is therefore, still an essential part of action research.

Criticisms of Action Research It is the variety of Action Research which also causes the most problems for researchers. It is frequently misunderstood (Tripp 2003, pp.1-2) and regarded as confusing (Hult and Lennung 1980;Tripp 2003, p.2). Kemmis and McTaggart (2005, p.273) warn of the philosophical and practical perils of proposals

based in abstraction and idealism, rather than in practice.

Most researchers agree that the phenomenon is studied within the context that gives it meaning (Hult and Lennung 1980, p.5). However, controversially some state that participation and collaboration is not essential to action research (Hult and Lennung 1980, p.7, Tripp 2003, p.6) and the degree of the presence of these elements should be determined by method and strategy (Hult and Lennung 1980, p.7), where other

advocates insist that its participatory nature is a key feature of action research (Zuber-Skerritt 1996).

Kemmis and McTaggart (2005, p.273), define ‘classroom action’, specifically as the collection of data by teachers, but explicitly limits this to qualitative modes of inquiry with a view to teachers making judgements regarding their own practice. Hult and Lennung (1980, p.2) say action research simultaneously assists practical problem solving and expands scientific knowledge, however they immediately point out that this is not a universally upheld idea.

Hult and Lennung (1980) suggest the method of gathering and analysing data must be pragmatically chosen to suit the problem by any valid and reliable method. Therefore, they suggest action research is not a method in its own right, but a way of employing methods. Tripp (2003, p.3) holds the opposing view, identifying action research explicitly as a method, and says it is ’a mistake’ to define it as a ‘category of processes’ and even suggests that the confusion could threaten research funding by undermining the

stakeholder’s confidence in researchers.

Relevance to this study This research is to be conducted by a researcher within the context of a teaching role, but intends to go beyond the micro reflection by teachers implied here. The goal is to expand the relevance of findings to the global community of interested teaching practitioners.

Design Science

Design science is concerned with devising artefacts which serve a human purpose (Formosa in Dresch et al. 2014, p.v). It concerns itself with innovative development defining ideas, practices, technical capabilities, and products through which IT systems can be developed, managed and used (Hevner et al. 2004). Cole et al. (2005) outlines the pragmatic nature of design science and the inextricable link between truth and utility

(Dresch et al. 2014, p.70). Hevner et al. (2004, p.80 and 89) differentiates between the objectives of design science and behavioural science research by saying that

behavioural science research seeks the truth, whilst design science research seeks what is effective.

Criteria for the use of Design Science Research From an inductive perspective, design science builds on a foundation of experience to produce knowledge (Dresch et al. 2014, p.18). Dresch et al. (2014) interpreted the guidelines for design science by Hevner et al. (2004) as a criteria for its use, and a similar exercise has been carried out here to determine its suitability. Each of the guidelines by Hevner et al. (2004) are described here, and the relevance of each one to this study is explained.

Guideline 1: Design as an artefact

• Guideline

Design-science research must produce a viable artefact in the form of a construct, a model, a method, or an instantiation

• Relevance to this study

The aim of this study is to produce artefacts of guidance for teaching practitioners in the production and distribution of video feedback. A taxonomy has the potential to form the foundation of many such artefacts of guidance.

Guideline 2: Problem relevance

• Guideline

The objective of design-science research is to develop technology-based solutions to important and relevant business problems.

• Relevance to this study

Modern cohort sizes and expectations of modern students mean technology offers the best chance of making video feedback feasible to produce and valuable to students.

Guideline 3: Design evaluation

• Guideline

The utility, quality, and efficacy of a design artefact must be rigorously demonstrated via well-executed evaluation methods.

• Relevance to this study

This study will test the validity and usefulness of all artefacts produced in practice.

Guideline 4: Research contributions

• Guideline

Effective design-science research must provide clear and verifiable contributions in the areas of the design artefact, design foundations, and/or design methodologies.

• Relevance to this study

This research aims to produce clear and verifiable contributions in the form of design artefacts.

Guideline 5: Research rigour

• Guideline

Design-science research relies upon the application of rigorous methods in both the construction and evaluation of the design artefact.

• Relevance to this study

Methods selected must befrom those which are already well established, well documented and well used. However, the unique context may require a

combination of methods designed to fit the circumstances of the work. Methods will be carefully applied during both the construction and evaluation phases.

Guideline 6: Design as a search process

• Guideline

The search for an effective artefact requires utilising available means to reach desired ends while satisfying laws in the problem environment.

• Relevance to this study

The selected pragmatic paradigm allows the researcher to use any available resources, whilst the case study approach encourages the use of multiple sources of data, to create artefacts of theoretical and practical contribution.

Guideline7: Communication of research

• Guideline

Design-science research must be presented effectively both to technology-oriented as well as management-oriented audiences.

• Relevance to this study

The language and style of presentation must be tested by trial presentation to an audience who are expert in practice.

In their framework for conducting design science Hevner et al. (2004, p.78) acknowledge how emerging technologies play a significant role in determining the strategies of an organisation and their capability to engage with new ways to do business, or in our case, education. The underlying design theories are articulated by Walls et al. (1992) as both product and process, noun and verb. These inextricably linked aspects must both be considered in the development of the artefact. The

researcher shifts perspective many times between the product and the process evolving both aspects towards a product (Hevner et al. 2004, p.3). Many iterations of build and evaluation cycles may occur before a finished artefact is created.

Identification of the problem Many of the proposed methods of design science research suggest the first step is to identify the problem, (like Cole et al. (2005) and Peffers et al. (2007)), while others begin at the point of understanding the problem, (like Eekels and Roozenburg (1991) and van Aken and Romme (2009)). Vaishnavi and Kuechler ((2009) in Dresch et al. 2014) merge the two steps into one step known as ‘awareness of the problem’.

Contributions The contributions themselves could be the artefact itself; other models, constructs, or ontologies developed during the process; innovative systems; or

developments in the method or methodology used (Hevner et al. 2004).

Development of artefacts According to Hevner et al. (2004, p.77) the products of design science may be :-

• Constructs of meaning consisting of words and symbols which determine the language used to share understanding and ideas.

• Models of abstraction or representations of the real world.

• Methods explaining algorithms or practices provide guidance towards the solution.

• Implementations of complete or prototype systems which validate the work as concrete evidence of feasibility.

Evaluation of artefacts Measures of design science contributions found in product or production fall into one of three categories: novelty, generality and significance. The designer may also bring an element of style, which while difficult to quantify and measure, should also be assessed during the evaluation phase (Hevner et al. 2004, p.86). Of the evaluation methodologies and corresponding methods for design science suggested by (Hevner et al. 2004, p.86) three are applicable to this work.

• Methodology: Descriptive

An ’informed argument’ can be developed using information from the knowledge base (e.g., relevant research) to build a compelling case for the artefact’s utility.

Relevance to this research: In this case the informed argument is to be provided

by the relevant research reviewed.

• Methodology: Analytical

A ’static analysis’ method can be used to examine structure of artefact for static qualities e.g., complexity. Alternatively an ’architecture analysis’ method can be used to study the fit of artefact into the technical architecture.

Relevance to this research: This can be employed as a strategy to test

implementation of any artefacts developed. This type of analysis can be conducted with each iteration of development.

• Methodology: Observational

Employing a ’case study’ method to study the artefact in depth in a business environment.

Relevance to this research: This would be useful to ensure the effectiveness of

artefacts in a real-world context.

Criticisms of Design Science There are currently insufficient constructs models and other tools available for modelling the real world and rigorously adhered to methods can result in high abstraction levels, which reduces relevance. This inadequate knowledge base results in the researcher becoming reliant on intuition and experience. At that point the researcher is experimenting, and iterations of prototyping and evaluation are of high importance. Rigorous evaluation methods are notoriously difficult to apply in design science research. Finally, as in any field of technology, the results may have been superseded before reaching a state of useful implementation (Hevner et al. 2004, p.99).

The case for a combination of methods

Cole et al. (2005) suggest a method which combines design science research with action research. The ‘in context’ position of the researcher in this study (see section 2.2.4) fits with this pragmatic suggestion. Cole et al. (2005, p.332) examines similarities between design research and action research. For instance, regarding ontology, both approaches depend on the phenomenon being studied evolving during study. The epistemology assumes that knowing involves intervention which is required to effect change. In action research the intervention occurs in practice, and design science specifies an artefact that enables change. The shared axiology is evident as both methodologies value the research problem, and theoretical knowledge as well as change in practical application. Cole et al. (2005) concludes that the mapping between the two approaches is not perfect but that they are paradigmatically compatible.

Structural similarities also make Action Research and Design Science compatible. Action research models are often cyclic in nature (Ferrance 2000; Coughlan and Coghlan 2002; Cohen et al. 2013), as are the 16 tenets of participatory action research by McTaggart (2018). Although Cole et al. (2005, p.329) does not specify a cyclic model in the synthesised research approach it is discussed as a criterion. Therefore, there is a strong case for a combined approach of design science and action research in this case.

A combined method, as suggested by Cole et al. (2005), is the best fit for this research. In this case it would be an action research study within each iteration of a design

science structure. The qualities of action research position the researcher within the real world context. An overarching design science approach pulls together the separate studies as evolutionary stages enabling change in practice, and progression towards an improved artefact, for validation.

Selected methods

- A combined method (Cole et al. 2005) - An overarching design science structure.

- Action research studies within each iteration, positioning the researcher in the real world context.