Conceptualizing and
Measuring Variables, part 1
Quote of the day:
“I have been struck again and again by how important measurement is to improving the human condition.”
-- Bill Gates
Midterm exam will be open book and take home. Will be distributed by email soon and due next Tuesday at 8:00 PM.
The questions will avoid things you can easily look up.
Instead, the questions will be integrative and will ask you to apply and extend what you have learned. There will be a word limit for each question.
First section assignment, due to Canvas by tonight at 8:00 PM.
some important concepts in political science:
• state power
• the economy
• bureaucratic autonomy
• judicial autonomy
• political efficacy
• size of government
• electoral competition
• corruption
concept: an abstract mental construct used to capture some aspect of the empirical world
Theories operate at an abstract level and make use of concepts. When we arrow-diagram a theory, each
variable is a concept.
X Y
Problem: we don’t have direct access to concepts.
Instead, we construct indicators (or measures) of them.
What we’re interested in: X Y (concepts) What we can study empirically: x y (indicators)
Suppose we find an association between our two
empirical indicators. To learn whether it is causal, we have to clear the tests covered in our last class. For today, however, let’s focus on whether an association between the indicators reveals an association between the concepts.
X Y (concepts)
x y (indicators)
Researchers must provide reasoning and evidence for why each indicator accurately taps its designated
concept. The alternative is “measurement by fiat.”
The dashed lines indicate the reasoning and evidence.
Let’s work through an example where we try to develop the necessary reasoning and evidence connecting
concepts and indicators.
social class support for globalization
Testing the relationship between social class and suppot for globalization requires that we understand the meaning of both concepts, then develop indicators of them. Let’s start with social class—what does it mean at a conceptual level?
How might we measure a person’s social class? Maybe we could just ask them to give their subjective
perceptions.
General Social Survey. “If you were asked to use one of four names for your social class, which would you say you belong in: the lower class, the working class, the middle class, or the upper class?” (2018)
lower class 9.0%
working class 43.4%
middle class 43.4%
upper class 3.5%
don’t know 0.4%
no answer 0.3%
In addition, this indicator relies on a person’s perception of what social class means, along the with labels for
different social classes. That approach could arguably be a strength or a weakness.
All self-reports, this one included, have some potential problems. They are widely used in political science and other social sciences largely because of the ease of
measurement.
An alternative approach for social class uses objective indicators rather than a person’s perception. What are some possible indicators?
● Education. Measure the person’s highest level of education completed.
● Income or wealth. Measure personal income, household income, and/or net wealth.
● Occupation. Measure the person’s occupation, then pair that information with a list assessing the status of different occupations.
● Should tastes/habits/values be included? If so, which ones?
Once we gather this data, we could either (a) construct a single index of social class, or (b) keep the components separate as different dimensions of social class.
The higher the correlation between the indicators, the stronger the case for constructing an index. Lower
correlations suggest that the concept is multi-dimensional and you need separate indicator(s) of each dimension.
After we build our indicator(s) of social class, we can turn to conceptualizing and measuring support for
globalization.
Once we have measures of the key variables in a theory or hypothesis, we can assess the relationship between
them—a topic for future classes. Today we’re focusing on the logically prior question of conceptualization and
measurement.
https://www.youtube.com/watch?v=O7VaXlMvAvk
News articles and media commentators typically take a simpler view of class than do scholars. For example, in reports on voting in 2016, 2018, and 2020, class was
usually measured as education (everyone without a college degree classified as working class).
Saturday Night Live’s Black Jeopardy with Tom Hanks:
Levels of measurement
nominal (categorical): variation in kind or quality but not in amount. Can classify the data into categories.
Includes binary/dichotomous, but can be more than two.
examples: race/ethnicity, religious identification, political party affiliation
ordinal: the order of the categories matters (relative
rankings), but you can’t place a quantitative value on the differences between the categories
examples: rankings of all kinds, Likert scales (strongly agree, agree, neutral, disagree, strongly disagree)
interval: you can measure the quantitative difference
between each value, but you cannot necessarily multiply or divide. The zero point has no inherent meaning.
examples: temperature, many constructed indexes
ratio: the zero point is meaningful. You can not only add or subtract (as with interval measurement) but also
multiply or divide.
examples: population, weight, unemployment rate
What can go wrong in a measure? Lots, which we can summarize as random and non-random error.
reliability: the extent to which a measure is free of
random error. If you measure the concept again, how
close will the second score be to the first? The closer the scores, the higher the reliability.
Imagine a bathroom scale. You step on it and get a reading of 150 pounds. You step on it again and it says 153. You step on it a third time and it says 147.
We expect bathroom scales to have 100% reliability.
In the social sciences, successive scores of 150, 153, and 147 are about as good as it gets.
● Test-retest method, as in the bathroom scale example.
Can be hard to implement. For aggregate data, there normally won’t be repeat data available for the same period and unit. For individual data, would the same
people be willing to take a survey two days in a row, and would the results be meaningful if they did?
How can we assess a measure’s reliability?
● Internal consistency. If you have different measures of the same concept, calculate the correlation among them.
The higher the correlation, the higher the reliability of an index containing them.
● Intercoder reliability. When content analyzing a text, researchers work from written coding rules. You can calculate intercoder reliability from two different people implementing the same coding rules. The higher their agreement, the higher the reliability.
By the way, laypeople often use the term “reliable” when they really mean “valid.”
validity: the extent to which a measure is free of non- random error. Essentially, are you measuring the
concept you think you’re measuring, or are you systematically missing the mark?
A measure can be reliable but not valid. However, if it’s not reliable, it’s not valid either.
● Face validity. Does the measure appear valid on its
face? Would it pass muster among experts in that area of study?
How can we assess a measure’s validity?
● Content validity. Does the measure contain the full range of content that falls within the scope of the
underlying concept?
● Convergent validity. Does the measure correlate highly with other, previously established measures of the same concept? Does the measure have a clear relationship with other independent and dependent variables in ways that match our theoretical expectations?
● Discriminant validity. Does the measure lack a
correlation with variables we would not expect it to be related to?
