DATA COLLECTION AND ANALYSIS
Quality Education for Minorities (QEM) Network
HBCU-UP Fundamentals of Education Research Workshop Gerunda B. Hughes, Ph.D.
August 23, 2013
Objectives of the Discussion
2
Discuss important principles of research when engaging in the data collection and analysis phases of the project?
Operationalize the variables in the research question(s)
Choose appropriate data collection methods
Choose appropriate data collection tools
Identify and collect from the proper data sources
Be acutely aware of timing
Avoid sampling error and bias
Ensure privacy and confidentiality
Store data properly
Define the different types of validity and reliability and the relationship between these two important characteristics of data and the results of data analysis.
Describe the types of measuring instruments used to collect data in qualitative and quantitative studies.
Variables
3
A variable is a construct that can take on two or more values. A constant takes on only one value.
For Data Collection, variables must be “operationalized.”
That is, the researcher must define a rule for how a variable is to be measured.
“Interest in Science” may be operationalize as (1) the score on a science interest inventory or questionnaire, or (2) the number of science courses that an individual took during grades 9-12.
Quantitative and Qualitative Variables
Quantitative variables are ordinal, interval and ratio variables. Variates differ in magnitude.
scores , heights, speed, age, weight
Quantitative variables exist on a continuum that ranges from low to high or less to more.
Qualitative Variables are nominal or categorical
variables. Variates differ in kind.
political party affiliation; eye color; gender; race/ethnicity
Qualitative variables are qualities about how people or objects differ with no relation to natural order.
4
Measurement Scales
5
Measurement is the process of assigning numbers to characteristics of an object or person.
The four measurement scales:
Nominal
Ordinal
Interval
Ratio
Data collected on different measurement scales require different methods of statistical analysis.
Nominal and Ordinal Scales
Nominal scales define variables that are categorical.
Examples: gender, employment status, marital status, type of school.
These scales classify persons or objects into two or more
categories. It is the lowest level of measurement.
Ordinal scales classify persons or objects and they also rank them in terms of the degree to which they possess a particular characteristic.
Examples: class rank, order of finishing a race.
These scales permit comparisons of higher/lower, for example, but do not indicate “how much higher or lower.”
6
Interval and Ratio Scales
Interval scales have all the properties of nominal and ordinal scales, and also have equal intervals.
Examples: Achievement, attitudes, motivation, etc. (educational
measures).
Interval scales do not have a true zero point. A score of zero may indicate the lowest level of performance possible, but does not indicate total absence of the characteristic.
Ratio scales have all the properties of the other scales and represents the highest, most precise level of measurement.
Examples: Height, weight, time, distance, speed (physical
measures).
Ratio scales have a true zero point. It is meaningful to talk about “no
distance.” Ratio scales also permit comparisons by ratios (Aisha weighs twice as Linda).
7
Types of Scores from Instruments
8
Raw Scores
The number or point value of items a person answered correctly on an assessment
Norm-referenced Scoring
A scoring approach in which an individual’s performance on an assessment is compared to the performance of others
Criterion-referenced Scoring
A scoring approach in which an individual’s performance on an assessment is compared to a predetermined external standard.
Self-referenced Scoring
A scoring approach in which an individual’s repeated performances on a single assessment are compared over time.
Types of Scores from Instruments
9
Raw Scores
Cedric earned a raw score of 92 on his biology test.
Norm-referenced Scoring
Jenelle’s has a percentile rank of 92 on her algebra test.
Criterion-referenced Scoring
Richard earned 92% on his chemistry test.
Self-referenced Scoring
Sheri scored 92% higher on this week’s geometry quiz than she did on last week’s geometry quiz.
Independent and Dependent Variables
10
Experimental Research
The independent variable (causal or manipulated variable) is the intended cause of the dependent variable (outcome, effect, or criterion variable).
Non-Experimental Research
The independent variable (status variable – not manipulated) is the variable that “logically” has some effect on a dependent variable. Examples of IVs include: gender, race-ethnicity,
marital status, eye color, employment status, etc.
Research Questions: Identifying Independent and Dependent Variables
11
Do ninth-grade girls will have different attitudes toward science than ninth-grade boys?
Is there a relationship between middle-school students’
grades and their self-confidence in science and math?
Is personalized instruction from a teacher more
effective for increasing students’ critical thinking skills than computer-based instruction?
Data Collection Methods
Tests
Surveys
Questionnaires
Rubrics
Checklists
Observations
Interviews
Document Reviews
Focus Groups
Photographs/Drawings
Recordings
Social Media/E-mail
Phone Calls/Recordings
12
Quantitative Methods Qualitative Methods
Formats for Data Collection Tools
13
Selection & Supply Methods: Used predominately by quantitative researchers; paper and pencil or electronic.
Selection Methods
Multiple-choice, true-false, and matching items
Supply Methods
Administer a short answer/essay question tests; fill in the blank items; and performance assessments (assessment of a product or a process)
Rubrics
Interviews, Focus Groups, Observations: Used predominately by qualitative researchers. Data are collected by observation, conversation, or extended written communication.
Validity of Assessment Results
14
Validity
is the most important characteristic of the assessment results;
is concerned with the appropriateness of the interpretations made from assessment results;
is best thought of in terms of degree;
is specific to the interpretation being made and to the group being assessed.
Types of Validity
15
Content Validity- the degree to which the assessment results are a reflection of the intended content area.
Criterion-Related Validity- determined by relating performance on one measure to performance on a second measure.
Concurrent Validity (SAT scores and ACT scores)
Predictive Validity (SAT scores and freshman g.p.a.; GRE scores and success in first year of graduate school)
Types of Validity
16
Construct Validity- is the most important form of validity because it asks the fundamental validity question: What is this assessment tool really
measuring? Examples…
Mathematics tests and reading levels
Reading and language tests
Interest in STEM
Consequential Validity - the extent to which the use of assessment results has intended or unintended effects for the user.
Test scores and graduation, teacher certification, teacher effectiveness
A narrowing of the curriculum and classroom teaching to focus only on what is tested
Reliability of Measuring Instruments
17
Reliability is the degree to which a test consistently measures whatever it is measuring.
Reliability is expressed as a reliability coefficient which is obtained by using correlation.
Error is present in all measurement.
High reliability means small errors of measurement.
Types of Reliability
18
Test-retest Reliability
Equivalent-forms Reliability
Internal Consistency Reliability
Split-Half Reliability
Cronbach’s Alpha Reliability
Scorer/Rater Reliability
Validity and Reliability of Instruments
A valid test is
always reliable, but a reliable test is not always valid.
19
Validity and Reliability of Observational Data
20
Factors that influence the validity and reliability of observational data:
The research question
Errors of measurement
Training of the observer(s) results in familiarity with
the setting
the culture
the focus of the study
the observation protocol
how to record data (and not summaries or personal opinions)
Data Collection Procedures and Environments
21
Every effort should be made to ensure appropriate data collection procedures and ideal environments (e.g., test administration conditions such as proper lighting,
minimum noise level, comfortable seating, etc.)
Failing to administer procedures precisely or altering the administration procedures, especially on standardized tests, lowers the validity of the test. High noise levels may be a distraction to study participants during data collection.
DATA ANALYSIS
Types of Statistics
Descriptive statistics are used to organize,
describe, and summarize a set of data.
Inferential statistics are used to draw inferences about the conditions that exist in a population
from study of a sample drawn from that
population.
Descriptive Statistics Inferential Statistics
Types of Statistics Analyses in Quantitative Research
Measures of central tendency
Mean, median, mode
Measures of variability
Range, variance, standard deviation, semi-
interquartile range
Effect Size
Parametric Tests
t-tests, Analysis of variance (ANOVA), Regression analysis
Non-parametric Tests
Chi-Square test; the sign test
Descriptive Statistics Inferential Statistics
Inferential Statistics
Types of hypotheses
Research hypothesis
Null hypothesis
Tests of the null hypothesis among
Relationships
Means
Proportions
Correlational Techniques
Correlation: A measure of the degree of association between two or more variables.
Pearson’s correlation, r; (both variables are continuous and quantitative)
Mathematics achievement and mathematics anxiety
Phi coefficient is the Pearson correlation for two
variables that are both qualitative and dichotomous
Gender and Science major or not
Spearman’s rho (both variables are expressed as ranks)
Class rank and ranking in a science fair competition
Tests of Significance
Simple Analysis of Variance: one independent variable- gender, and one dependent variable - college gpa.
Multi-Factor Analysis of Variance: (two or more independent variables - gender, SES, participation in summer bridge
program; and one dependent variable, college freshman gpa).
Multiple Regression: tells us how much of the variance in the dependent variable (e.g., “on-time graduation”) is explained by the set of independent variables (e.g., high school gpa, SAT/ACT mathematics and verbal scores, freshman college gpa).
Correlational Techniques
Correlation: A measure of the degree of association between two or more variables.
Bi-serial correlation (one variable is continuous and
quantitative and the other would be, expect it has been reduced to two categories)
Multiple correlation, R, is the Pearson correlation between the variable to be predicted and the best- weighted combination of several predictors. To
calculate R, we must know Pearson’s r between each pair of variables.
Data Analysis in Qualitative Research
Engage in a great deal of analysis before data collection is complete.
Reflect on two questions—
Is the research questions still answerable?
Are the data collection techniques catching the kind of data that is wanted and filtering out the data that is not wanted?
Avoid premature actions based on early analysis and interpretation of data.
Data Analysis in Qualitative Research
Qualitative data analysis is a cyclical, iterative process of reviewing data for common topics or themes. One approach is to follow three iterative steps:
Reading and memo-ing
Describing what is going on in the setting
Classifying research data
Data Analysis in Qualitative Research
Constant Comparative Analysis
Phenomenological Approaches
Ethnographic Methods
Narrative Analysis & Discourse Analysis
Data Analysis Strategies
Identifying Themes -- emerges for ideas found in the review of the literature and the data collection.
Coding -- the process of marking units of text with codes or labels as a way to indicate patterns and meaning in data.
Asking questions – “Who is centrally involved?”; “What major activities or issues are relevant to the problem?” Then seeking answers in the data.
Concept Mapping – a visual display of the major influences that have affected the study.
Questions?
33
