Session Number:
T153
Session Title:
Crea6ng a
Longitudinal Educa6on
Database: Conceptual and
Methodological Issues
Perri Morgan, PA-‐C, PhD
Chris6ne EvereF, PA-‐C, MPH, PhD
Duke University
Crea%ng a Longitudinal Educa%onal
Database
for Research
:
Objec6ves
•
Discuss poten6al uses of a longitudinal educa6onal
database.
•
Describe examples of educa6on studies using
longitudinal databases.
•
Summarize a conceptual approach to crea6ng
educa6on databases.
•
Iden6fy exis6ng sources of informa6on for inclusion
into a database.
•
Describe processes associated with development and
maintenance of a longitudinal database.
Why go longitudinal?
•
To move toward the dream
studies
•
Dream ques6on
How do we select and educate
students to produce PAs with the
best and largest impact on the
health of the na6on?
Why go longitudinal?
•
Longitudinal analysis allows analysis of
change
s
at both the
group
and
student
level.
•
As educators, we are interested in changes in our
students/graduates over 6me.
•
This is the example we give our students when
An example:
cross-‐sec%onal vs.
longitudinal
data
Anne and Sue both respond to a survey about their
aWtudes toward working in surgery.
How likely are you to choose a career as a surgical PA?
(1-‐10 scale with 1= very unlikely and 10=very likely
Student
1
styear response
Response at gradua6on
Anne
1
10
An example:
cross sec%onal
data
How likely are you to choose a career as a surgical PA?
(1-‐10 scale with 1= very unlikely and 10=very likely
Student
1
styear response
Response at
gradua6on
1
10
10
1
Mean student
response
5.5
5.5
Conclusion: Student aWtudes toward working in surgery do
NOT change over the course of their PA educa6on
An example
: longitudinal
data
How likely are you to choose a career as a surgical PA?
(1-‐10 scale with 1= very unlikely and 10=very likely
Change in student
response
Student
1
styear response
Response at
gradua6on
Y
1
10
+9
X
10
1
-‐9
Conclusion: Student aWtudes about working in surgery change during their PA
educa6on.
For the longitudinal analysis, we have to be able to link each student’s first response
to their later response.
Research vs. evalua6on
Research
•
Produces generalizable
knowledge
•
Uses scien6fic methods
•
Requires human subjects
review (IRB)
Evalua%on
•
Intent is to improve a
specific
program
•
Findings are expected to
directly impact a program
and to iden6fy poten6al
improvements
•
Geared toward program
decision-‐making
•
Some6mes does not require
Why might you want a longitudinal
database for
evalua6on
purposes?
•
To help organize your data
•
To use for program improvement
•
To analyze issues specific to your students or your
program
–
Ex: Does a specific admissions factor predict a specific
problem in your program?
–
Ex: Does a specific educa6onal interven6on work beFer
for a par6cular type of student?
•
You do not want to deal with human subjects review
and informed consent (but we think this is a weak
excuse!)
Why would you want a longitudinal
database for
research
?
•
To share your findings with other programs
and the educa6on community
•
To facilitate use of previously-‐collected data
into research on new ques6ons
–
This might lead to shorter surveys and
–
might
reduce survey fa6gue among your students
•
You might be able to combine your program
data with that of other ins6tu6ons in the
future
When does
evalua6on
NOT require
human subjects review?
•
When the ac6vity does not involve non-‐
standard interven6ons
•
The intent is to only provide informa6on for
and about the seWng in which it is conducted
•
The ac6vity is part of standard opera6ng
Examples of educa6on research using
longitudinal databases
•
Jefferson Medical School started a longitudinal
database in 1970. Over 150 ar6cles have been
published based on it
•
Papadakis MA, Teherani A, Banach MA, et al.
Disciplinary ac6on by medical boards and prior
behavior in medical school. N Engl J Med.
2005;353:2673–2682.
•
Tamblyn R, Abrahamowicz M, Dauphinee D,et al.
Physician scores on a na6onal clinical skills
examina6on as predictors of complaints to medical
regulatory authori6es. JAMA. 2007;298:993–1001.
Examples of longitudinal research in
PA educa6on
•
Hoops S, Barley G, Chung A.
The design and implementa%on of a
longitudinal clinical competency assessment of PA students.
JPAE,
2004; 15(1).
•
Higgins R et al.
Admissions variables as predictors of PANCE scores
in PA programs: A comparison study across universi%es.
JPAE,
2010; 21(1):10-‐17.
•
Warner ML, Maio C, Hudmon KS.
Career paMerns of PAs: A
retrospec%ve longitudinal study.
JAAPA.
2013; 26(6):44-‐8.
•
Beck B, Scheel MH, DeOliveira K, Hopp J.
Cultural competency in a
PA curriculum in the US: A longitudinal study with two cohorts.
J
Our conceptual approach:
The big picture
Data Sources
•
Admissions data
Data Sources
•
Midpoint student survey
–
Repeats select items from
new student survey
•
Academic data during PA
training
•
PANCE pass/fail
•
Gradua%on student
survey
Data Sources
•
Alumni survey
–
To be developed
•
Prac%ce-‐related
data
–
Claims data
–
State medical board
sanc%ons data
Examples of Research Ques6ons
•
What student characteris6cs predict admission
into the Duke PA program? (Pre-‐PA school PA
School)
•
Which PA program experiences are associated
with post-‐graduate leadership posi6ons? (PA
school Post-‐PA School)
•
What PA program experiences are associated
with the delivery of high quality care? (PA school
Post-‐PA School)
Data that is NOT included
•
Data not included because anonymity is
necessary
–
Student evalua6ons of courses
–
Other student evalua6ons of the program (exit
survey, etc.)
•
Data not included because we consider it
mandatory for every student
–
Data required for repor6ng to HRSA for grant
applica6ons and progress reports (data for
Prac6cal issues
•
Student par6cipa6on
•
Human subjects review
•
Privacy protec6on
•
Choosing soqware
•
Maintenance of database
•
Linking data
Student par6cipa6on
•
Program leadership emphasizes the
contribu6on that students can make to
knowledge about the profession by
par6cipa6ng
•
2013 entering class 85/90 consented
–
Midpoint survey 89/90 completed the survey—
snacks might have helped.
•
2014 entering class 89/90 consented
Human subjects review: our
experience
•
We have a separate protocol approved for
crea6on of the database.
–
Each new survey that is added to the database
requires IRB approval. These are expedited, with
2-‐3 day turnaround
•
Any research using the database will require
Informed consent
•
We give a 10 minute presenta6on to new
students about the database and distribute the
consent forms electronically.
•
The next day, in the classroom, staff distributes
paper consent forms and collects them. Faculty
are not present.
•
In order to obtain applica6on data for all
applicants (including those not admiFed), we
added a one paragraph consent statement to our
supplemental applica6on.
Privacy protec6on
•
Faculty does not know which students consented to
par6cipate
•
Staff assign a database iden6fier to each student and
keep the code with student names under lock and key.
•
Faculty who wish to use the database will be issued
limited datasets by staff that include only the variables
required for their project.
•
Even without student names, faculty could iden6fy
many students using other variables. However, this
would be a breach of research ethics and possibly
illegal.
Secure storage
•
Data on a protected server
•
Access to iden6fiable data limited
•
De-‐iden6fied datasets created for individual
Choosing Soqware
•
Ins6tu6onal resources
–
Any exis6ng programs available through ins6tu6on? (e.g.
REDCap)
–
Support readily available?
•
Interface preferences – overall usability, security issues
–
Desktop-‐based (e.g. Microsoq Access, FileMaker Pro)
–
Server-‐based (e.g. MySQL)
–
Web-‐based (e.g. REDCap, Medrio)
•
Import/export file type op6ons (e.g. SAS, Stata, SPSS, Excel,
others)
Database Soqware Op6ons
SoTware Website
Where is database
located?
Data
export
op%ons
Cost
Microsoq
Access
hFp://
office.microsoq.co
m/en-‐us/access/
On user’s computer
Excel, txt,
Word, XML
Office 365—individual
license $70
REDCap
hFp://www.project-‐
redcap.org/
On Internet; need user rights to
access
Excel, PDF,
SPSS, SAS,
Stata, R
Ins6tu6onal
partnership required;
no cost
Medrio
hFp://medrio.com/ On Internet; need user rights to
access
Excel, SAS,
SPSS, STATA
Free for inves6gator-‐
ini6ated trials; $1200/
year once you hit
100k data points
StudyTrax
hFp://
www.sciencetrax.co
m/studytrax/
Hosted on own server or
ScienceTrax secure servers
Excel, CSV,
SAS, SPSS,
Word
$99 Student License
OpenClinica hFps://
www.openclinica.co
m/
On user’s computer (aqer free
download)
HTML, tab-‐
delimited,
Excel, SPSS
Open source; no cost
QuesGen
hFp://
www.quesgen.com/
On Internet; need user rights to
access
Stats
packages
and Excel
Pay as you use, with
per-‐user, per-‐month
charge as set-‐up fee
Linking data
•
Format maFers
•
IRB issues
•
Data use agreements
The future
•
Many PA programs combining data for
Discussion
•
Sugges6ons
References
• Ander-‐Peciva, S. (2005). Construc6on of longitudinal databases -‐ for flexibility, transparency and longevity.
Interna6onal Commission for Historical Demography. Sydney, Australia.
• Chen, H. (2013). "Designing Educa6on Lab: Evalua6on vs. Research -‐-‐ What's the Difference?" Retrieved
September 20, 2014, from
hFp://web.stanford.edu/group/design_educa6on/wikiupload/2/27/Helen_Evalua6on.pdf.
• Cook, D. A., D. A. Andriole, S. J. Durning, N. K. Roberts and M. M. Triola (2010). "Longitudinal research databases in
medical educa6on: facilita6ng the study of educa6onal outcomes over 6me and across ins6tu6ons." Acad Med
85(8): 1340-‐1346.
• Ellaway, R. H., M. V. Pusic, R. M. Galbraith and T. Cameron (2014). "Developing the role of big data and analy6cs in
health professional educa6on." Med Teach 36(3): 216-‐222.
• Gonella, J., M. Hojat and J. Veloski (2005). "Abstracts: Jefferson Longitudinal Study of Medical Educa6on, 3rd
edi6on [full volume]." Jefferson Longitudinal Study of Medical Educa6on Paper 1.
• Papadakis, M. A., A. Teherani, M. A. Banach, T. R. KneFler, S. L. RaFner, D. T. Stern, J. J. Veloski and C. S. Hodgson (2005). "Disciplinary ac6on by medical boards and prior behavior in medical school." N Engl J Med 353(25):
2673-‐2682.
• Tamblyn, R., M. Abrahamowicz, D. Dauphinee, E. Wenghofer, A. Jacques, D. Klass, S. Smee, D. Blackmore, N.
Winslade, N. Girard, R. Du Berger, I. Bartman, D. L. Buckeridge and J. A. Hanley (2007). "Physician scores on a
na6onal clinical skills examina6on as predictors of complaints to medical regulatory authori6es." JAMA 298(9):
993-‐1001.
• Triola, M. M. and M. V. Pusic (2012). "The educa6on data warehouse: a transforma6ve tool for health educa6on
