Step 4b – Data collection instrument selection

Taking the pragmatic perspective on the instrument selection means collecting from any sources available, in whatever way they can contribute to the knowledge. All

instruments require ethics consideration and require approval before use (Stake 1995; Cohen et al. 2013).

Literature is a useful starting point, but it also has the disadvantage of being beyond the researchers control. Therefore, primary data must be developed to work out to what extent the conclusions are applicable in other practice contexts. Having discounted a survey as a single approach, as a source of information among many it still has

significant value as a tool for data gathering. In addition, there is the data held as part of the video storage system regarding individual interaction with the media. Finally,

students can be interviewed on an individual basis about their experience with video feedback. Therefore, the potential types of data sources are: -

• Literature

• Questionnaires

• Interviews

• Documentation

Literature

When implementing a new element of practice, and starting from scratch, the literature offers the opportunity to avoid repetition of the pitfalls already found. The variety of studies available is anticipated to provide insight into the advantages and disadvantages of implementation details and the impact of context.

Questionnaire

The questionnaire is a widely used and useful instrument for collecting data. The contributions from a large number of participants can be collected in far less time than individual interviews would take. Like any other instrument it has limitations, in that the preparation time must be considered. The questions must be constructed with care and should be analysed regarding ambiguity (Cohen et al. 2013; Collis and Hussey 2013), and their ability to obtain information pertinent to the research question.

There are limits regarding the types of data that can be collected, particularly regarding the lack of flexibility available for the participant in their response. However, this feature is also considered an advantage in that it makes analysis easier in some respects. In addition there must be the consideration of ‘non-response bias’ (Collis and Hussey 2013), in that there may be a reason why a particular set of participants do not respond, and therefore the participant set is not necessarily representative of the general

population. In the particular case of students on the first year of an undergraduate degree, may suffer from what Collis and Hussey (2013) calls questionnaire fatigue. It is known that many organisations, commercial, governmental and academic, ask new students to answer many online questionnaires at this point of their academic career.

Interview

In an interview, the interaction between two humans emphasises the social nature of the knowledge since it is constructed through conversation (Cohen et al. 2013). Although the researcher can control some elements, such as ensuring topics covered, there is the opportunity to be flexible in responses. The structured or unstructured style questions can result in answers of ‘yes’ or ’no’, or in depth answers (Cohen et al. 2013; Collis and Hussey 2013), taking directions the interviewer was not expecting. It allows for the flexibility of adding in questions mid way through the interview, and changing the questions for future participants, in a way not always possible with e.g., a questionnaire (Collis and Hussey 2013).

The multi-sensory communication informs by the additional non-verbal communication in ways a questionnaire cannot (Cohen et al. 2013). However, questions still need careful preparation to get to the detail that is useful to knowledge construction without becoming invasive in ways intolerable to the participant (Cohen et al. 2013). Its results can be compromised by the attitude or tone of the interviewer, causing the interviewee to respond in superficial ways, or even shut down the interview. It is a resource hungry mode of elicitation where anonymity is usually not possible (Cohen et al. 2013). c

Documentation

There are two main sources of other documentation that would provide useful data. There is data from students enrolment on the course, and once produced, the data from the videos made and returned to students as feedback.

Feedback videos The feedback videos themselves contain evidence to determine if feedforward of learning is taking place as anticipated. The system on which the videos are stored contains evidence of student interaction with the videos. Whilst each of these sets of data has the potential to make a valuable contribution to the research, each of these systems is also limited to their design and original purpose. After closer

examination the data collected may be flawed or not fit for the purpose of this research.

Student information system Summary information regarding the cohort is available through the student information system. The results of student surveys and interviews are often examined in the context of the total number of students on the course at the time, for example, response rates. The number of students will vary at different points in the year. Therefore, depending on when a particular analysis is done, or a survey is released, the total number of students in the cohort may vary. In general, the number of students is at its highest approximately two weeks after the start of term when all enrolments are complete. Beyond that point the main cause of change is students changing course within the university or withdrawing to pursue an external option.

Students occasionally wish to transfer to a course from another subject after the first two weeks of teaching have passed. However, unless the student has relevant prior

knowledge of the subject these applicants are not normally accepted on to computing due the volume of work that would need to be caught up on to succeed.

Consequently,increases in numbers after the first fortnight are rare and made only in exceptional circumstances. As a result, changes in student numbers normally follow a shallow decline from that point onward through the year. This is a normal general trend within the yearly cycle of first year undergraduate students.

The variation across the year in the total numbers of students on the course may appear inaccurate at first glance when e.g.: two different values are given for the cohort size when referring to the same year group. However, the numbers reported are accurate at the time and to report any other figure would be give an inaccurate reflection. To give the reader a point of comparison values should be additionally shown as percentages of the cohort wherever possible.

Summary of data collection instrument selection

The data gathering techniques planned for use in the studies for this work have been selected based on resource availability and purpose. The pragmatic perspective is

gained by evaluating all the potential resources of information available. The depth and complexity of each case in its context can be drawn on in this way.

The questionnaire, and documentation are sources of quantitative data, whilst the interviews, and parts of the questionnaire, supply qualitative data. Mitigation against bias and the limitations of a single data source may be reduced by examination and comparison of different data sources, although investigator triangulation is not possible due to the single researcher involved (Collis and Hussey 2013). Therefore, the

combination of instruments is employed to form a well-informed and rich picture of the student perspective of video feedback used in practice.

Selected data collection instruments

Questionnaire, documentation and interviews.