Conceptualizing
College and
Career Readiness
through Concept
Mapping
Wenjie Tang, M.A. Thomas Fetsco, Ph.D. Arizona GEAR UP
NAU/Arizona GEAR UP 1
Arizona GEAR UP (2012-19)
An initiative of Northern AZ University
Partnerships with schools in rural
communities statewide
Proven model for increasing
graduation and college-going rates
Seven-year project begins with 7th
grade cohorts, continues through
first year in postsecondary
education
Range of services and interventions
with emphasis on rigor, relevance,
relationships, resources and
reflection.
NAU/Arizona GEAR UP 2
GEAR UP Mission
“The mission of GEAR UP is to significantly
increase the number of low income
students who are prepared to enter and
succeed in postsecondary education.”
Although there is a shared mission, we
wondered if there was a shared
understanding of that mission.
NAU/Arizona GEAR UP
“The Devil is in the Details.”
Implementation of a single program
across different contexts
Different Challenges and Opportunities
Different Understandings of the Meaning
What knowledge, abilities, skills, and behaviors or other
characteristics are the most important for a student
to be college- and career-ready?
NAU/Arizona GEAR UP
Research Questions
1.
What knowledge, abilities, skills, and behaviors or
other characteristics are the most important for a
student to be college- and career-ready?
2.
How does the importance of one characteristic
differ from that of others?
3.
To what extent are the administration team and site
coordinators consistent on the perceived
importance these characteristics?
4.
To what extent are the important characteristics
feasible to implement according to GEAR UP site
coordinators? What are the facilitators and
inhibitors?
NAU/Arizona GEAR UP 6
Methods
Needs
Structured
conceptualization
Mixed method
Methods
Concept mapping
(Trochim, 1989)
NAU/Arizona GEAR UPUses of Concept Mapping
Strategic planning – What should be done by an
effective arts council?
Needs assessment – What types of support do
professionals need?
Assessment of fidelity - Whether program models are
8In this study, CM is used for
Conceptualizing program outcomes
Assessing fidelity
Facilitating organizational learning
NAU/Arizona GEAR UP
Concept Mapping Procedure
NAU/Arizona GEAR UP 10
“Complete the following sentence:
If our district’s GEAR UP program
successfully prepares students to be
college/career ready, our high
school graduates should…”
NAU/Arizona GEAR UP
Brainstorming
NAU/Arizona GEAR UP NAU/Arizona GEAR UP 14
Editing for
•
Clarity
•
Relevance
•
Uniqueness
•
Manageable # of
statements
ideas from
literature
statements
for sorting
synthesized
“In this activity, we want you to sort
ideas about college and career
readiness into categories based on
their conceptual similarity. By
conceptual similarity, we mean that
the ideas seem to make sense
together.”
Then name the piles.
NAU/Arizona GEAR UP
Sorting
Logistical Challenges
Distance and scheduling
Face to face brainstorming
Online sorting and rating
Drag-n-Drop Sorting
NAU/Arizona GEAR UP
Re-name Piles
NAU/Arizona GEAR UP 18
One Person’s Sorting
NAU/Arizona GEAR UP
Aggregate Sorting Data
Display ideas on a 2-dimension
map
Multidimensional Scaling (MDS)
Point Map
NAU/Arizona GEAR UP
7
Cluster analysis
Group statements into clusters
Hierarchical Cluster Analysis
Decide on the number of clusters
NAU/Arizona GEAR UP 22
AZ GEAR UP CCR Concept Map
Points: statement # 1 to 66
Statements are grouped into 7 Clusters
Concrete Abstract Academic Non-academic
Ratings
Importance as a college-readiness
outcomes(1-6)
Importance as a career-readiness
outcomes(1-6)
Feasibility of reaching the CCR outcomes(1-5)
Ratings
NAU/Arizona GEAR UP
Cluster Rating Map
NAU/Arizona GEAR UP 26 1 8 21 25 27 39 45 48 60 1116 33 34 37 52 64 5 6 7 10 12 20 26 32 44 46 49 51 58 62 66 14 17 43 65 2 3 13 15 24 28 31 40 4142 47 50 57 4 9 18 35 53 55 61 19 22 23 29 30 36 38 54 56 59 63 1. Academic Foundations 2. College Preparation Courses/Exams 3. Postsecondary Knowledge & Skills 4. Financial Literacy 5. Social/Emotional Intelligence 6. Metacognitive Skills 7. Motivational Beliefs/Skills Cluster Legend Layer Value 1 4.71 to 4.83 2 4.83 to 4.95 3 4.95 to 5.06 4 5.06 to 5.18 5 5.18 to 5.29
Pattern matching
“Ladder graphs”
NAU/Arizona GEAR UP
A
B
A
B
A
B
Compare importance ratings of
College Readiness and Career
Readiness
Importance for College Readiness
(n = 14) Importance for Career Readiness(n = 14)
Motivational Beliefs/Skills Motivational Beliefs/Skills
Academic Foundations Academic Foundations
Metacognitive Skills Metacognitive Skills Financial Literacy
Social/Emotional Intelligence
6.00 6.00 28
Group difference: Importance ratings
on College Readiness
Implementers (n=11) Administrators (n=3) Motivational Beliefs/Skills Motivational Beliefs/Skills Academic Foundations Academic Foundations Metacognitive Skills Metacognitive Skills Financial Literacy Financial Literacy College Prep Courses/ExamsCollege Prep Courses/Exams
Postsecondary Knowledge & Skills
Postsecondary Knowledge & Skills Social/Emotional Intelligence Social/Emotional Intelligence 4.00 4.00 6.00 6.00 r = 0.83 Implementers and administrators are highly similar to each other in terms of ratings of important clusters for college-readiness. The 3 most valued clusters
remain the same. NAU/Arizona GEAR UP
Which clusters are more important to
college readiness but less feasible
Importance for College Readiness
(Program) (n=14) Feasibility (Implementers) (n=11)
Motivational Beliefs/Skills Motivational Beliefs/Skills Academic Foundations Academic Foundations Metacognitive Skills Metacognitive Skills Financial Literacy Financial Literacy College Prep Courses/Exams
College Prep Courses/Exams
Postsecondary Knowledge & Skills
Postsecondary Knowledge & Skills
Social/Emotional Intelligence
Social/Emotional Intelligence
4.71 3.68 5.29 4.07
r = -0.36
The 3 most important clusters are more challenging than
other clusters, such as PSE Knowledge and Financial Literacy.
Results
The three most important clusters are:
Academic foundations
Motivational beliefs/skills
Metacognitive skills
However, they are not necessarily as
feasible as other goals.
Implementers prioritized CCR goals similar
to the designers.
NAU/Arizona GEAR UP
Dissemination of the Results
Interpreting results with program staff
Check validity of results
Approaches:
Meeting
Webinar & Online posts
Dissemination of the Results
(Cont.)
Questions for the program administration
team:
NAU/Arizona GEAR UP What has AZ GEAR UP administrators been doing/ is planning to do to make it more feasible for implementers to achieve these outcomes? What may havecaused these outcomes to be rated as less feasible to achievement than others? To what extent are outcomes in the cluster important to the program?
Dissemination of the Results
(Cont.)
Discuss with site coordinators:
• How would you relate the CCR goals in the
map with your work?
• Which clusters do you think are relatively
more feasible to accomplish at your site?
• What are the resources and strategies you
and the school district/school have to help
accomplish the goals?
NAU/Arizona GEAR UP 34
Coordinators’ Feedback
NAU/Arizona GEAR UP The CCR goals keep causing me to reflect on my work plan
and what it should consist of for the next four years. There are
certain clusters like Social/Emotional Intelligence, Meta Cognitive Skills, Motivational Beliefs, etc., that I can help students with as a coordinator, but there are some clusters
that need to be a part of the school culture/expectations and classroom environment. When we plan with the high
school, I would really like to have these clusters available so we can thoughtfully plan parent nights, student workshops,
guest speakers, and professional development, etc. We
need administrators and teachers to "buy-in" to the importance of Academic Foundations and College Preparation. I'm sure they will all agree that fostering and
building these skills are valuable, yet we need them to understand the role GEAR UP and the Coordinators have in
implementing and supporting these CCR goals.
Coordinators’ Feedback
(Cont.)
36
I really like how the statements were organized into the seven-cluster concept map. When we were all originally coming up
with different ideas they seemed to cover a lot of different areas, but when grouped together it was clear that they overlapped in so many ways. I think the CCR goals directly
relate to our everyday tasks and goals for our students. The
Our Use of Results
Research in Practice papers
Assessment of Metacognition
NAU/Arizona GEAR UP
Recap
NAU/Arizona GEAR UP 38
Limitations and Future Studies
Process depends to a large extent on successful
brainstorming.
Brainstormed ideas may evolve
Follow-up study on the same topic during high school
Need a deeper discussion with site coordinators about
facilitators and inhibitors of reaching CCR goals.
Perspectives needed from other groups of program
stakeholders
Compare program perspectives with input from colleges
and employers
NAU/Arizona GEAR UP
Reference
Kane, M. & Trochim, W.M.K. (2007). Concept
Mapping for Planning and Evaluation. Thousand
Oaks, CA: Sage Publication.
Trochim, W. (1989). An introduction to concept
mapping for planning and evaluation. Evaluation
and Program Planning. 12, 1, 1-16.
Trochim, W M.K. (1999) Technology for Mapping
Where We’re Going and Where We’ve Been.
Thank you!
Tom Fetsco, Ph.D., Director of Research
[email protected]
