• No results found

The accuracy and reproducibility of video assessment in the pitch-side management of concussion in elite rugby

N/A
N/A
Protected

Academic year: 2019

Share "The accuracy and reproducibility of video assessment in the pitch-side management of concussion in elite rugby"

Copied!
21
0
0

Loading.... (view fulltext now)

Full text

(1)

This is a repository copy of The accuracy and reproducibility of video assessment in the

pitch-side management of concussion in elite rugby.

White Rose Research Online URL for this paper:

http://eprints.whiterose.ac.uk/116973/

Version: Accepted Version

Article:

Fuller, G.W., Kemp, S.P.T. and Raftery, M. (2017) The accuracy and reproducibility of video

assessment in the pitch-side management of concussion in elite rugby. Journal of Science

and Medicine in Sport, 20 (3). pp. 246-249. ISSN 1440-2440

https://doi.org/10.1016/j.jsams.2016.07.008

Article available under the terms of the CC-BY-NC-ND licence

(https://creativecommons.org/licenses/by-nc-nd/4.0/)

eprints@whiterose.ac.uk https://eprints.whiterose.ac.uk/ Reuse

Unless indicated otherwise, fulltext items are protected by copyright with all rights reserved. The copyright exception in section 29 of the Copyright, Designs and Patents Act 1988 allows the making of a single copy solely for the purpose of non-commercial research or private study within the limits of fair dealing. The publisher or other rights-holder may allow further reproduction and re-use of this version - refer to the White Rose Research Online record for this item. Where records identify the publisher as the copyright holder, users can verify any specific terms of use on the publisher’s website.

Takedown

If you consider content in White Rose Research Online to be in breach of UK law, please notify us by

(2)

1 The accuracy and reproducibility of video assessment in the pitch-side management of concussion in 1

elite rugby 2

3

Abstract 4

Objectives: To investigate the accuracy and reliability of side-line video review of head impact events to 5

aid identification of concussion in elite sport 6

Design: Diagnostic accuracy and inter-rater agreement study 7

Methods: Immediate care, match day and team doctors involved in the 2015 Rugby Union World Cup 8

viewed ‘

9

matches. Subjects subsequently recorded whether any criteria warranting permanent removal from play 10

or medical room head injury assessment were present. The accuracy of these ratings were compared to 11

consensus expert opinion by calculating mean sensitivity and specificity across raters. The 12

additionally assessed using raw agreement and Gwets AC1 13

chance corrected agreement coefficient. 14

Results: Forty rugby medicine doctors were included in the study. Compared to the expert reference 15

standard overall sensitivity and specificity of doctors decisions were 77.5% (95%CI 73.1-81.5%) and 16

53.3% (95%CI 48.2-58.2%) respectively. Overall there was raw agreement of 67.8% (95% CI 57.9-77.7%) 17

between doctors across all video clips. Chance corrected Gwets AC1 agreement coefficient was 0.39 18

(95% CI 0.17-0.62), indicating fair agreement. 19

Conclusions: ‘ W C rated moderate accuracy and fair reproducibility in 20

(3)

2 time video may improve the identification, decision making and management of concussion in elite 1

sports. 2

Key Words: Brain concussion, Rugby, Video analysis, Reproducibility of results, Sensitivity and specificity 3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

(4)

3 Introduction

1

Concussion is a common injury in elite sport , with concussion rates of 10.5 concussions per 1000

player-2

match-hours reported in professional Rugby Union.1 Continued participation following concussion may 3

risk further injuries, exacerbate concussive symptoms, or predispose to second impact syndrome.2 4

The in-game assessment and management of head impact events has been a contentious issue.3 Media 5

attention has focused on high profile incidents where players clearly demonstrating symptoms or signs 6

of concussion have controversially been allowed to continue playing 4 7

Conversely, inaccurate evaluation and incorrect withdrawal of non-injured players could have 8

detrimental effects on team performance. 9

Elite adult Rugby Union, in common with other collision sports, has introduced systems to improve the 10

in-game management of head impact events with the potential to cause concussion. The World Rugby 11

Head Injury Assessment (HIA) process (formally the pitch-side concussion assessment, or PSCA, process) 12

was developed by a working group of international concussion specialists and was informed by expert 13

opinion, recent consensus statements, and review of scientific literature. The HIA process, recently 14

updated for the 2015 Rugby World Cup, has been described previously and is summarised in the web 15

appendix.5 16

Side-line video review of head impact events has been introduced in several elite sports, including 17

American Football, Rugby Union and Ice Hockey, to identify possible concussions and inform removal 18

from play decisions.6 Video has three main roles in the World Rugby HIA process identification of 19

suspicious head impact events that warrant off-field assessment, identification of signs that confirm a 20

concussion leading to a permanent removal, and a final video review prior to a decision that clears a 21

(5)

4 Despite the introduction of video review into side-line concussion assessment by many sports little is 1

known about the accuracy or reliability of these assessments. This study aimed to characterise the 2

performance of rugby medicine doctors when evaluating video clips of head impact events. Specific 3

objectives were to determine the accuracy of match day and team doctors opinions against an expert 4

reference standard, and to describe the reproducibility of video clip assessments in the World Rugby HIA 5

process. 6

7

8

9

10

11

12

13

14

15

16

17

18

19

(6)

5 Methods

1

This was a cross-sectional diagnostic accuracy and inter-rater agreement study investigating the 2

assessment of head impact event video clips in elite Rugby. The study population consisted of all 3

immediate care, match day and team doctors involved in game day concussion management during the 4

2015 Rugby World Cup. All participants were rugby medicine doctors, experienced in providing pitch-5

side medical care in elite Rugby, who had completed W ‘ on-line course in concussion 6

management, and accreditation in immediate care in Rugby.7-9 7

The study was performed immediately prior to the October 2015 Rugby World Cup. In preparation for 8

this tournament a face-to-face training session was conducted to re-inforce HIA protocols, including the 9

identification of suspicious head impact events, criteria for immediate and permanent removal, and 10

indications for a medical room head injury assessment. This study was performed immediately after the 11

training session. 12

Video recordings of all Elite Rugby matches are reviewed for head impact events by World Rugby as part 13

of the HIA process. A selection of 20 video clips were identified a priori W ‘ C M

14

Officer. The clips included a range of head injury severities, in addition to trivial head impact events with 15

no evidence or suspicion of concussion. Twenty clips were chosen to provide a sufficient range of clinical 16

presentations within realistic time constraints. 17

Videos showed the incident in real time with the relevant player identified by jersey colour and number. 18

S

19

footage. Sound and commentary was removed. Clips lasted a mean of 61.5 seconds (range 21-129). The 20

resulting clips therefore mimicked the availability of video within the HIA process as much as possible. 21

The clips were then evaluated by a team of 7 international concussion experts in a web-based consensus 22

(7)

6 positions in Sports Medicine. Expert opinion on each video clip was determined by consensus. In cases 1

where a majority consensus of greater than 70% was not reached the World Rugby Chief Medical Officer 2

held a casting vote. 3

Following the training session, clips were then shown to Rugby World Cup doctors with each participant 4

rating the clips individually without conferring. A single replay was available on request. The first 10 clips 5

simulated the side-line identification of suspicious head injury events. Participants were asked whether 6

the clip showed any HIA process criteria for removal from play (either permanent removal or medical 7

room head injury assessment), or if the head impact event was not significant. The remaining 10 videos 8

simulated post-removal medical room video review and asked participants to indicate whether 9

permanent removal or further head injury assessment was indicated. For each clip participants were 10

also asked to identify which HIA process criterion for permanent removal or head injury assessment was 11

the most important in influencing their decision. HIA process criteria for permanent removal and head 12

injury assessment are presented in the web appendix. 13

The statistical analysis proceeded in 2 stages. Firstly, the accuracy of match and team doctors decisions 14

were compared to the consensus expert opinion. Participant responses were pooled across video clips 15

and classified according to the reference standard. Mean sensitivity and specificity was then calculated 16

across raters. Secondly, the reproducibility ‘ 17

initially determined, but to account for the fact that some agreement would be expected even if the 18

participants were guessing, a chance corrected agreement coefficient was also calculated. Gwets AC1 19

coefficient was calculated as the preferred measure of agreement due to theoretical considerations of 20

increased stability, robustness to marginal probabilities, and lack of dependence on rating prevalence. 10-21

12 L K benchmark values were used to interpret the magnitude of agreement coefficients 22

with: 0 0.20 indicating slight, 0.21 0.40 fair, 0.41 0.60 moderate, 0.61 0.80 substantial, and 0.81 1 23

(8)

7 overall, and separately for each stage of the HIA video process. Statistical analyses were carried out in 1

Stata version 13 (StataCorp, College Station, USA) and AgreeStat 2011.3 (advanced Analytics, 2

Gaithersburg, MD, USA). 3

A study protocol defining an a priori analysis plan received ethical approval from the University of 4

Sheffield. Prior consent was available for the use of each video clip and individual consent was obtained 5

from all participants prior to the study. Research funding was provided by the World Rugby. Reporting is 6

in accordance with EQUATOR guidelines for reliability studies.14 7

8

9

10

11

12

13

14

15

16

17

18

19

(9)

8 Results

1

All forty immediate care, match day and team doctors involved in the 2015 Rugby World Cup 2

participated in the study, rating all videos with no missing data. 3

Expert consensus was reached on 16 of the 20 videos. A casting vote from the World Rugby Chief 4

Medical Officer was therefore necessary for 4 videos where agreement amongst experts ranged from 5

43-57%. 6

Compared to the expert reference standard overall sensitivity and specificity of doctors decisions were 7

77.5% (95%CI 73.1-81.5%) and 53.3% (95%CI 48.2-58.2%) respectively. For the side-line identification of 8

suspicious head impact events sensitivity was 87.0% (95%CI 81.5-91.3%) and specificity: 39.0% (95%CI 9

32-46%). Sensitivity was relatively lower for medical room video review, determining whether 10

permanent removal or head injury assessment was appropriate, at 68.0% (95%CI 61.1-74.4%), but with 11

higher specificity of 67.5% (95%CI 60.5-73.9%). 2x2 contingency tables, pooling participants responses 12

across relevant video clips and classifying them according to the reference standard, are presented 13

separately for the accuracy of side-line and medical room video review in Tables 1 and 2. 14

Overall there was raw agreement of 67.8% (95% CI 57.9-77.7) between doctors across all video clips. 15

Chance corrected Gwets AC1 agreement coefficient was 0.39 (95% CI 0.17-0.62), indicating fair 16

agreement. Figure 1 presents the level of agreement across each video clip. 17

Agreement was relatively higher for the side-line identification of suspicious head impact events with 18

raw agreement of 71.2% (95%CI 54.5-88.0%), and a Gwets AC1 coefficient of 0.53 (95%CI 0.14-0.93), 19

denoting moderate agreement. Less agreement was observed for medical room video review 20

determining whether permanent removal or head injury assessment was indicated. Raw agreement was 21

64.3% (95%CI 50.1-78.0), with a Gwets AC1 coefficient of 0.29 (95%CI 0.015-0.56) demonstrating fair 22

(10)

9 There was similarly lower agreement on which of the HIA process criterion was most important in each 1

scenario, with a mean raw agreement between participants of 30.6% (95%CI 25.9-35.3%) across the 2

clips, and a Gwets AC1 coefficient of 0.26 (95%CI 0.2-0.31). No discernible association was evident 3

between the specific criterion and the degree of agreement. Notably, only one of the five videos with 4

the lowest inter-rater agreement corresponded to a clip where a firm expert consensus was not 5

reached. 6

7

8

9

10

11

12

13

14

15

16

17

18

19

(11)

10 Discussion

1

Rugby World Cup moderate accuracy and fair reproducibility when assessing 2

video clips of head impact events. There was high sensitivity (87%) and moderate agreement (Gwets 3

AC1 coefficient 0.53) when detecting suspicious head injury events. Sensitivity for identifying permanent 4

removal criteria in the medical room was lower (68%) with fair reproducibility evident (Gwets AC1 5

coefficient 0.29). 6

There are a number of limitations that could adversely affect the validity of the results. Firstly, it is 7

questionable whether a true reference standard can be defined for each video clip, challenging 8

calculation of diagnostic accuracy metrics. Video clips of head impact events are inherently subjective as 9

illustrated by the minority of video clips where considerable disagreement was present amongst the 10

expert panel. However, derivation of a reference standard through a formal consensus process is an 11

established method to obtain a credible gold standard in the absence of an objective reality.15Secondly, 12

although comparing favourably to other published reproducibility studies, the sample size is relatively

13

low, resulting in imprecise results consistent with either poor or moderate agreement. These findings

14

should therefore be considered as preliminary, requiring confirmation in larger studies. Thirdly, although 15

every effort was taken to imitate the HIA video review process, controlled study procedures cannot fully 16

simulate the real-life application of video technology. On-field observations, competitive pressures or 17

crowd distractions could all conceivably influence actual performance of video review. Finally, we

pre-18

specified the relatively liberal Landis and Koch scale for benchmarking agreement coefficients. Although

19

well established and widely used, this may overstate agreement compared to other benchmarks e.g.

20

F M H 16, 17

21

The accuracy and reproducibility of video review in sports concussion assessment has not been

22

extensively studied. Makdissi and Davis (2016) examined video clips of 82 concussion cases sustained

(12)

11

during professional Australian Football matches, demonstrating a similar level of fair to poor inter-rater

1

agreement for a range of clinical signs indicating head injury (kappa statistics 0.13 to 0.37).18 Gardner 2

and colleagues (2016) evaluated 162 video clips where players were temporarily substituted for

3

assessment of suspected concussion during the 2014 Australian National Rugby League season.19 4

Reported agreement between two raters was relatively higher, both overall (kappa 0.6, 95% CI

0.56-5

0.64) and for specific observable signs (kappa statistics 0.36 0.7). However, the comparability of these

6

results, with videos scrutinised by two researchers and multiple replays allowed, is uncertain.

7

In the World Rugby HIA process, independent match-day doctors can supplement side-line observation 8

with video reviews of incidents to identify any suspicious head impact events. The finding of favourable 9

sensitivity and moderate agreement for this stage of side-line recognition is reassuring, suggesting that a 10

real-time videoreviewprocess integrated into the HIA process has the potential to identify head injuries 11

or incidents requiring removal from play or further assessment. The high media profile of the Rugby 12

World Cup and the potential consequences to practitioners for not removing players who are 13

subsequently deemed to be concussed may be associated with a reduced threshold for further off-pitch 14

evaluation. This could explain the lower specificity for identifying non-significant head impact events. 15

Once a suspicious head impact event has been identified, the HIA process includes a second video 16

review undertaken off-field in the medical room with the match-day and team doctors. This decides 17

whether the player should be immediately and permanently removed (without a further head injury 18

assessment) or the player should have a HIA1 screening head injury assessment. The less favourable 19

accuracy and reproducibility apparent for medical room video review may be of lesser concern, as the 20

player has already been removed from the field of play. The minority of players who were not identified 21

for permanent removal (video false negative) will all undergo a head injury assessment and a final 22

further review of the video as additional safeguard prior to any return to play. Previous research has 23

(13)

12 unlikely that such cases would return to play.5 Furthermore, permanent removal of play for a minority of 1

incidents where a HIA screening assessment was indicated (video false positive) may be acceptable in 2

the context 3

The lack of reproducibility on the single most prominent HIA process criterion present in each scenario is 4

unsurprising as agreement tends to decrease as the number of rating categories increases. Furthermore, 5

more than one HIA process criteria was evident in several clips. Ultimately, the final decision may 6

matter more than the exact underlying reason. Conversely, the lack of conformity in decision making 7

highlights the importance of robust training of doctor in the video review process, specifically focusing 8

on definitions of each assessment criterion. 9

The sample of World Cup match-day and team doctors should ensure good external validity. However, it 10

is possible that accuracy and agreement could be lower in other tournaments utilising less experienced 11

medical staff, if specific concussion education is not provided to video-reviewing medical personnel, or if 12

the effects of concussion management training wane. Furthermore, Hawk-Eye/Smart 13

Replay video system was recently integrated into the HIA process during certain competitions 14

(including the RWC).20 This technology allows the user to fully control replays from 20-30 camera feeds, 15

which might be expected to increase the sensitivity of head impact evaluation compared to the standard 16

video review with limited broadcaster feeds investigated here. Generalizability to other sports 17

implementing differing systems for reviewing video, using non-medical practitioners and exhibiting 18

other patterns of play is less certain. 19

Conclusion 20

Elite contact and collision sports are played at a fast pace in a disorganised environment, where the view 21

of medical staff is often obscured, challenging the evaluation of head impact events. This study suggests

22

(14)

13 reproducibility in head injury event decision making when reviewing video footage. Despite these 1

encouraging results it is likely that further improvement in identification and decision making could be 2

achieved with development of education resources and greater operational clarification of permanent 3

removal from play criteria. 4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

(15)

14 Practical Implications

1

 Side-line video review is an additional modality to help in the identification and assessment of 2

potential concussive episodes 3

 Sports physicians show moderate accuracy and fair reliability when reviewing video clips of head 4

impact events 5

 Introduction of side-line video review could improve the management of concussion in elite 6

sport 7

8

Acknowledgments 9

The authors would like to match day doctors, team doctors and intermediate care doctors participating 10

in the 2015 Rugby World Cup for their assistance with the study. Funding was received by World Rugby. 11

12

13

14

15

16

17

18

19

20

(16)

15 References

1

1. Moore IS, Ranson C, Mathema P. Injury Risk in International Rugby Union: Three-Year Injury 2

Surveillance of the Welsh National Team. Orthopaedic journal of sports medicine. 2015; 3

3(7):2325967115596194. 4

2. Brolinson PG. Management of sport-related concussion: a review. Clinical journal of sport

5

medicine : official journal of the Canadian Academy of Sport Medicine. 2014; 24(1):89-90. 6

3. Helmy A, Agarwal M, Hutchinson PJ. Concussion and sport. BMJ (Clinical research ed.). 2013; 7

347:f5748. 8

4. Cusimano MD, Sharma B, Lawrence DW, Ilie G, Silverberg S, Jones R. Trends in North American

9

newspaper reporting of brain injury in ice hockey. PLoS One. 2013; 8(4):e61865.

10

5. Fuller GW, Kemp SP, Decq P. The International Rugby Board (IRB) Pitch Side Concussion 11

Assessment trial: a pilot test accuracy study. British journal of sports medicine. 2015; 49(8):529-12

535. 13

6. NFL Head NaSC. NFL Head, Neck and Spine Committee's Protocols Regarding Diagnosis and 14

Management of Concussion. USA: National Football League; 2015. 15

7. Rugby W. Concussion Management for Doctors and Healthcare Professionals. 16

http://playerwelfare.worldrugby.org/?documentid=module&module=22. Accessed 2016 8th

17

February 2016. 18

8. Rugby W. Medical Protocols for Match Day Medical Staff. 19

http://playerwelfare.worldrugby.org/?documentid=module&module=23. Accessed 2016 8th

20

February 2016. 21

9. Rugby W. Concussion Management for Elite Level Match Day Medical Staff. 22

http://playerwelfare.worldrugby.org/?documentid=module&module=24. Accessed 2016 8th

23

(17)

16 10. Wongpakaran N, Wongpakaran T, Wedding D, Gwet KL. A comparison of Cohen's Kappa and 1

Gwet's AC1 when calculating inter-rater reliability coefficients: a study conducted with 2

personality disorder samples. BMC medical research methodology. 2013; 13:61. 3

11. Gwet K. Handbook of Inter-Rater Reliability: How to Estimate the Level of Agreement Between

4

Two or Multiple Raters., Gaithersburg, MD, STATAXIS Publishing Company; 2001. 5

12. Gwet K. Inter-Rater Reliability: Dependency on Trait Prevalence and Marginal Homogeneity. . 6

Statistical Methods for Inter-Rater Reliability Assessment. 2002:1-9. 7

13. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics.

8

1977; 33(1):159-174. 9

14. Kottner J, Audige L, Brorson S, et al. Guidelines for Reporting Reliability and Agreement Studies 10

(GRRAS) were proposed. Journal of clinical epidemiology. 2011; 64(1):96-106. 11

15. Rutjes AW, Reitsma JB, Coomarasamy A, Khan KS, Bossuyt PM. Evaluation of diagnostic tests 12

when there is no gold standard. A review of methods. Health technology assessment

13

(Winchester, England). 2007; 11(50):iii, ix-51. 14

16. McHugh ML. Interrater reliability: the kappa statistic. Biochemia medica. 2012; 22(3):276-282. 15

17. Fleiss JL, Levin BA, Paik MC. Statistical methods for rates and proportions, 3rd ed. / Joseph L. 16

Fleiss, Bruce Levin, Myunghee Cho Paik. ed, Hoboken, N.J. ; [Chichester], Wiley-Interscience; 17

2003. 18

18. Makdissi M, Davis G. The reliability and validity of video analysis for the assessment of the 19

clinical signs of concussion in Australian football. Journal of science and medicine in sport /

20

Sports Medicine Australia. 2016. 21

19. Gardner AJ, Iverson GL, Stanwell P, Moore T, Ellis J, Levi CR. A Video Analysis of Use of the New 22

C I ‘ N ‘ L Int J Sports Med. 2016; 37(04):267-23

(18)

17 20. Hawk-Eye. Hawk-Eye in Rugby Union. http://www.hawkeyeinnovations.co.uk/sports/rugby-1

union. 2

3 4

5

6

7

8

9

10

11

12

13

(19)
[image:19.612.107.506.125.479.2]

18 Table 1. 2x2 contingency table summarising the R W C

head impact events

Reference standard Total

D Suspicious head impact event* Not suspicious

Suspicious head impact event 174 [TP] 122 [FP] 296

Not suspicious 26 [FN] 78 [TN] 104

Total 200 200 400

*A suspicious head impact event exhibited a HIA process criterion for permanent removal or off-pitch head injury

assessment. Non-suspicious head impact events exhibited no HIA process criteria. TP: True positive, FP: False

(20)
[image:20.612.115.495.163.482.2]

19 Table 2. 2x2 contingency table summarising the R W C

removal criteria

Reference standard Total

D Permanent removal* H

Permanent removal 136 [TP] 65 [FP] 201

Head injury assessment 64 [TN] 135 [TN] 199

Total 200 200 400

*One or more HIA permanent removal criteria present (see web appendix) N HIA

(21)

20 Figure 1. Overall agreement across each of the 20 video clips.

Light grey shading represent the percentage majority opinion. Dark grey shading represents the percentage

minority opinion. The dashed black line indicates the 50% agreement that would be expected if participants were

Figure

Table 1. 2x2 contingency table summarising the �������� �� R���� W���� C�� �������� ��� ����������
Table 2. 2x2 contingency table summarising the �������� �� R���� W���� C�� �������� ��� ���������

References

Related documents

As a final conclusion, university teachers must favour self-regulation of learning, offer the students the opportunities to discover the usefulness of digital technologies and

The multimode Radio cellular mobile system has improved performance on the basis of BER and constellation size for different modulation algorithms.. Thus, the mode may improve

19% serve a county. Fourteen per cent of the centers provide service for adjoining states in addition to the states in which they are located; usually these adjoining states have

Given that coronary artery disease is the major cause of death in the Western world, we speculated that this protective Ser249Ser TLR-6 genotype would be overrep- resented in an

Sexual compulsivity is an intriguing, though unpopular, topic in China as it is seldom discussed among the general public, in what is characteristically a sexually conservative

MAP: Milk Allergy in Primary Care; CMA: cow’s milk allergy; UK: United Kingdom; US: United States; EAACI: European Academy of Allergy, Asthma and Clinical Immunology; NICE:

It was decided that with the presence of such significant red flag signs that she should undergo advanced imaging, in this case an MRI, that revealed an underlying malignancy, which

Also, both diabetic groups there were a positive immunoreactivity of the photoreceptor inner segment, and this was also seen among control ani- mals treated with a