is present in the environment during an observation.There may be very good reasons why you would choose an observational approach over an experimental design: for example, while there is a large body of work in social psychology in which the influence of many intermediate variables is controlled in the labora- tory, the ultimate ecological validity of the work is clearly most accurately demonstrated in the real world.Conversely, the variables measured during obser- vation may have had their construct validity determined in controlled laboratory experiments.So, over the long term, a research plan might set out a combination of experimental and observational work to be performed.
Observational Studies
Not all observational studies (also known as quasi-experimental studies) are the same in terms of their effectiveness in answering research questions.The least biased observational studies are “forward-looking” (prospective) and focus on a randomly selected group (acohort).Thus, a prospective longitudinal study, also known as acohort study, observes people forward in time from their entry into the study.Cohort studies often start from birth or another common point in time, such as the start of school.Systematic error can be reduced by ensuring that all participants have as much in common as possible, for example, by selecting on birthdate, social class, etc.In contrast, aretrospective longitudinal studyreviews participants backward in time from their entry into the study.The goal of this type ofcase control studymight be to determine, for example, what social activi- ties in the clinical group have led to catching a specific disease.However, if there is a long gap in time between the study and the events in question, thenrecall bias may influence the results.
Case control designs are also the only ethically supportable designs where a treat- ment may be harmful, as is the case in disease; for example, in order to link the AIDS syndrome with HIV infection, it would be clearly unethical to administer the virus to an experimental treatment group and compare their immune responses and clinical outcomes with a control group.For some case control studies, even if an experimental manipulation was possible, there simply may not
Observational Studies | 87
Research
Design
be enough potential participants—particularly for very rare disease cases—so an observational design in preferable.Experimental designs are usually only appro- priate where the treatments are either known to be all not harmful or all beneficial.For example, a researcher investigating the effectiveness of a new form of pain relief might use baseline responses for other pain relief medications that are known to be safe in the quantities administered.
Two other types of observational study are worth mentioning.Across-sectional
designinvolves a single observation (e.g., a questionnaire or an interview), which
may be useful if an immediate response to a specific question is required (e.g., what donut flavors are popular in New York City today), but have clear limits on their generalizability (since donut flavor preferences may change seasonally). Another technique is so-calledsecondary analysis, where data from many different sources is combined to investigate a particular problem.In this case, the investi- gator does not exert any influence over the data collection, and the analysis is generally retrospective, since data is usually discovered post-hoc from a range of sources.While some researchers have questioned the validity of relying on other people’s data, secondary analysis can be very useful for developing new leads when investigating difficult or complex questions.For example, cohort studies undertaken in different countries may have systematic bias related to geograph- ical or social factors, and secondary analysis can be used to trace and/or eliminate this bias by examining whether the relationship between variables is consistent across these different countries.Note that the only source of control in a secondary analysis is the selection of variables, although it may be possible for a secondary analysis to specify new variables to be measured in a future prospective study.
While observational studies are generally considered weaker in terms of statistical inference, they have one important characteristic: response variables (like human behavior) can often be observed within the natural environment, enhancing their
ecological validity, or the sense in which what is being observed has not been arti-
ficially constrained by engaging in a narrowly defined experimental paradigm. Going one step further, some observational studies use participant observation methods, where a researcher becomes involved in the activity under study.If this participation is hidden from the actual participants, then ethical issues may arise around the use of deception.However, in other cases, there may be ethical objec- tions to withholding an intervention in which investigator participation is required.For example, in clinical settings in speech language pathology, the clin- ical investigator would typically play a very active role in eliciting responses to various treatments, since these would not normally be forthcoming from the participants.Observational studies may be the only solution to investigate a research problem where ethical considerations prevent the use of randomized experimental trials.
Observational studies potentially suffer from a number of biases, including biases in selection, which are either known or suspected.One way of removing these biases is to make adjustments for those that are known (perhaps using a covari- ance correction), and at least making clear those that are suspected to exist. Often, even if the magnitude of a bias is not known, the direction of bias can be easily determined.For example, members of a conservative political party may be reasonably assumed to hold conservative social values, even if the strength of
these beliefs or their extent is unknown (as an illustration, all conservatives may favor jailing of drug dealers, but a proportion may favor the death penalty over imprisonment).
Case control studies match patients with nonpatients based on covariates to elimi- nate bias, without having to use an experimental design.A covariate in the context of observational studies is a variable that is unaffected by the administra- tion of a treatment (e.g., collected prior to a study), such as age, sex, or IQ, whereas a variable that is predicted to be affected by a treatment is known as an outcome. This type of matching is also common in experimental designs (i.e., a
matched pair design).
For some observational studies, such as clinical populations, the treatment group may be relatively small, but in order to match as closely as possible on all covari- ates, a large potential pool of controls is generally useful.Indeed, it may also be possible to further eliminate biases by matching a single member of the treatment group to several controls who are all matched on key covariates.You can imagine, though, that as the number of covariates increases, and/or as the number of controls per treatment unit increases, the likelihood of finding “perfect matches” decreases.If you are concerned that bias may be creeping in, you can calculate a
propensity score, which is the probability of being assigned to a treatment or
control group based on the covariate values.Think of it as a form of validation for the assignments that you have made during case control.*
So far, I’ve focused on known biases, but what about biases that are unknown? In experiments, randomization takes care of both identified and unknown sources of bias—if the selection is performed randomly, both types of bias will be controlled for.But in an observational study, there may simply be a confounding, hidden source of bias that is unknown, such as a hidden covariate.In this situation, it may be possible to use a sensitivity analysis to determine whether there is a source of hidden bias, especially one of great magnitude.