The method used to search for relevant literature begins by identifying topic areas, which in turn, narrows the selection of relevant databases to be searched with a suitable search string. The process of excluding irrelevant material is repeated for several
criteria. Finally the remaining publications are catalogued and sorted.
3.2.1 Step 1 - Identify topics
The first stage was to identify the topic areas of literature to search. Not only were publications on the use of video feedback likely to be listed under education, but because the implementation is likely to be electronic, they may also be listed under computing or information systems. Therefore databases covering these topics were to be included.
3.2.2 Step 2 - Identify databases and sites to search
A list of 8 research databases covering education, and 12 on computing or information systems was compiled. One of the computing/information systems databases was excluded as it applied exclusively to business and industry, leaving 19 sources to search (details of the research databases considered can be found in appendix B.1). In addition other sources of research considered were the websites of professional bodies. The websites examined were BCS (formerly British Computer Society), the Higher Education Academy (HEA), and the Staff and Educational Development Association (SEDA).
3.2.3 Step 3 - Develop a suitable search string
The search string was selected by starting with a wide scope and gradually reducing it so that the numbers of articles returned became manageable. However,it was also important not to make the search so narrow that relevant publications are omitted, so a balance was sort. Candidate strings were tested on EBSCO Industries, Inc. (2019) (see Appendix B.5 for details of search strings tested and the results). The string with the widest scope included any one of a set of keywords related to feedback, and any one of a set of keywords related to media, which returned an unmanageable 85,717 articles. Gradually 5 different search strings were developed as potential candidates. In addition filtering options were applied to only include peer reviewed articles, where the full text is both accessible and written in English. Twenty different combinations of search string and options were recorded. The selected string sort a term related to feedback and a term related to media in the title of the article, and reference to HE, university, or
undergraduate. In combination with all three options for peer review, English and full text selected, the number returned was 671 but that still contained duplicates.
3.2.4 Step 4 - Removal of duplicates
EBSCO Industries, Inc. (2019) has the facility to automatically remove as many duplicates as it can find, but it is not perfect. Once the duplicates were removed automatically by EBSCO Industries, Inc. (2019) 439 articles were returned. Finally, duplicates missed by the automatic system were removed manually leaving 394 unique publications returned by EBSCO Industries, Inc. (2019).
3.2.5 Step 5 – Exclusion by abstract review
The abstract of each paper was reviewed to identify, and remove from the data set, papers which are not relevant. The publications to be included needed to be relevant to
the use of video feedback used in practice for the purpose of marker feedback on academic subjects in higher education.
• Exclusion by subject: Effect in a system
The term feedback is commonly used in electronic systems to describe when the route taken by the signal becomes a loop, and the effect that might create.
Feedback from a system might also refer to a response to a user to confirm correct use e.g., haptic feedback on a hand held device. Both of these types of articles are excluded from the study.
• Exclusion by subject: Feedback on products
Articles were excluded where feedback was referring to feedback from consumers on products and not as a response to student work. These scenarios could be in a completely different domain or they could be as closely related as e.g., an
instructor getting feedback from students on a video they used in a lecture. However, the feedback is not being completed by the instructor in response to student work, and is therefore excluded.
• Exclusion by subject: Not in teaching or instructional domain
Articles were excluded where the domain was not instruction of any kind e.g., to see how computer gamers perform.
• Exclusion by subject: Assessment but not feedback
Some articles were concerned with the development of assessments, but not with feedback, and were therefore, excluded from the data set.
• Exclusion by subject: Performance and behaviour
Video feedback is a term used frequently in contexts regarding skills, performance or behaviour. In these cases videos are used to reflect on action for the purposes of evaluating a performance. It comprises of evaluation post performance and often out of context, to allow the performer to be present at the review, or where the presence of the reviewer would interfere with the performance. The reflection activity is potentially completed by the performer or performers; with or without, the instructor and/or peers. These articles are excluded from the study. This study is looking at video as an asynchronous response to student work by teaching staff.
• Exclusion by reviewer type: peer, self or automated
Articles referring to work on assessment feedback by student peers, by the student themselves, or by an automated system, are excluded here. This study is
Some of the articles fell into one or more of these categories and all of these were excluded from the data set. In some cases, the eligibility of papers was indiscernible from the abstract, or the abstract was not available, and those papers go through to the next step for further examination.
Steps 3 to 5 repeated per research database
Steps 3 to 5 were repeated for each database and website. However, some of the databases are included in the results by EBSCO Industries, Inc. (2019) searches and therefore did not to be repeated separately. Of the remaining 11 databases covering computing or information systems all but 1 was included in an EBSCO search. Of the 8 on the topic of education 3 were not included in an EBSCO search. That meant 5 separate searches needed to be completed in total. Table 3.1 summarises the numbers of publications resulting from searching each resource, and the number of publications remaining following the abstract review process.
Research database or website No. search
results
No. publications post exclusions
by abstract
EBSCOHost 439 85
Higher Education Empirical Research (HEER) 157 19
Gartner 65 3
Research into Higher Education Abstracts (RHEA) 18 13
Google 146 52
Higher Education Academy (HEA) 62 22
BCS (formerly British Computer Society) 296 8
Staff and Educational Development
Association (SEDA) 100 0
Total 1283 202
Table 3.1: Summary of results from research databases and websites
3.2.6 Step 6 - Catalogue publications for review
The remaining publications were downloaded to a single location. Now duplicates that had come from different sources could be identified and removed. Each publication was labelled and the references were recorded in reference management software.
3.2.7 Step 7 - Sorting publications
Of the studies remaining, 34 are about the use of audio only as feedback, and one is about automated feedback. These are to be filed away in case they happen to make interesting points of comparison, but are excluded from the main data set for review. That left 30 which are about the use of video as feedback and 19 that required a more detailed review to ascertain the media formats of the feedback in the study.
Publications were sorted into folders by the type of media they used in the study. These were text, audio or video or a mixture. Only the publications categorised into the ’video’ folder, and those in the ’mixed’ folder which include video in the study, were selected for inclusion in the core data set for this review. This folder structure was duplicated in the three ways: -
1. The file system containing the articles as files.
2. The node structure in the software to be used during the thematic analysis process which would contain the imported annotated files (Nvivo (QSR International 1999)).
3. The folder structure in the reference management software (EndNote (Clarivate Analytics 2001)).
The data set was sorted into the video feedback folder in all three structures. Articles were also kept on related topics e.g., audio feedback, since they may still provide useful insights. Publications excluded were recorded along with the reason for their exclusion.