Determining the Reliability of the ALiEM AIR Score for Rating Educational Online Learning Materials

16 

Loading....

Loading....

Loading....

Loading....

Loading....

Full text

(1)

Rating Educational Online Learning Materials

Michael Paddock, DO, MS

@mikepaddock

(2)

Je n’ai aucune affiliation (financière ou autre)

avec une entreprise pharmaceutique, un fabricant

d’appareils médicaux ou un cabinet de communication.

communications organization.

Author: Michael Paddock, DO, MS

Date: October 23

rd

, 2015

(3)
(4)

Growth of EM/CC OLM

(5)
(6)

ALiEM AIR Series Grading Tool

BEEM*

Content Accuracy

Educational Utility

1-7

1-7

1-7

Adapted from http://www.aliem.com/wp-content/uploads/Air-Series-Grading-Tool.pdf

*Acad Emerg Med. 2013 Oct;20(10):1004-12. doi: 10.1111/acem.12235

EBM

Referencing

(7)

Methods

Curated modules of curricular topics

»

CORD EM Modular Curriculum

1

ID/Heme, HEENT, Endocrinology, Toxicology, etc.

»

Highest score using the Social Media-index

2

»

Top 6-10 scored posts used

Validate first 3 modules

»

July-Dec 2014

»

Nine Volunteer EM Educators on AIR Exec Board

Scored 83 OLM

1

CORD EM Modular Curriculum, CORD EM Tests; http://www.cordtests.org

2

Thoma, Brent, et al. "The Social Media Index: measuring the impact of emergency medicine and critical care websites."

Western

(8)

Methods

Reliability Studies

»

Generalizability Study (G-Study)

*

IRR and rating (inter-item) variance

Rater, blog, posts, and topic

Phi coefficient, Epsilon (G-)coefficient

»

Decision Study (D-Study)

*

Predicts number of raters required for reliability

(9)

Methods

Validation Study

»

Random selection of 40 OLM

»

Separate group of expert medical educators

Seven-point Likert scale for quality of materials

Pearson r statistics compared to AIR score

(10)

Results – Reliability Study

Generalizability Studies

»

Absolute IRR (9 AIR raters) phi = 0.81*

(11)

Results – Decision Study

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 0 2 4 6 8 10 12 14 16

C

oeff

ici

en

t v

al

ue

Number of Raters

Decision Study on Inter-rater Reliability

Absolute Inter-Rater Reliability (Phi Coefficient) Relative Inter-Rater Reliability (Epilson Coefficient)

(12)
(13)
(14)

Conclusion & Limitations

ALiEM AIR Score

»

Moderate to highly reliable for rating OLM

»

Correlates well with educator gestalt

»

First validated tool for use by medical educators

Sources of variance

»

Website interface/design

(15)

Teresa Chan, HBSc, BEd, MD, FRCPC

»

McMaster University

Andrew Grock, MD

»

University of California-Olive View

Kulamaken Kulasegaram, PhD

»

University of Toronto

Lalena Yarris, MD, MCR

»

Oregon Health & Science University

Michelle Lin, MD

»

University of California San Francisco

Medical Educator Contributors

(16)

Download the ICRE App,

Visit the evaluation area

in the West Ballroom Foyer,

near Registration, or

Go to: http://www.royalcollege.ca/

icreevaluations

to complete the

session evaluation.

Téléchargez l’application de la CIFR

Visitez la zone d’évaluation, foyer

de la salle de bal Ouest, près du

comptoir d’inscription, ou

Visitez le http://www.collegeroyal

.ca/evaluationscifr

afin de remplir

une évaluation de la séance.

You could be entered

to win 1 of 3

$100 gift cards

.

Vous courrez la chance de gagner

l’un des trois

chèques-cadeaux

d’une valeur de 100 $.

Figure

Updating...

References

Updating...

Related subjects :