A case-ordered descriptive meta-matrix contains first-level descriptive data from all cases, but
A case-ordered descriptive meta-matrix contains first-level descriptive data from all cases, but the cases are ordered (e.g., high, medium, low) according to the main variable being examined.
Thus, it coherently arrays the basic data for a major variable across all cases (see Displays 8.9 and 8.10).
Display 8.9
Ordered Meta-Matrix: Format for Student Impact Data
Source: Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage Publications.
Applications
Case-ordered displays array data case by case, but the cases are ordered according to some variable of interest, so that you can easily see the differences among high, medium, and low cases.
It’s a powerful way of understanding differences across cases to help answer questions such as the following: What are the real patterns of more and less X in the cases? Why do we see more X at one site or case and less of it in another? This makes the method particularly suitable for evaluation and action research studies.
Example
In our (Miles and Huberman) school improvement study, we were interested in a variable we called “student impact.” We had interviewed not only students themselves but also teachers, parents, administrators, and counselors—and had looked at formal evaluation data as well. We wanted to assemble this information in one place and understand the differences among cases that showed high, medium, or low student impact.
Our interviewing and data retrieval focused not only on “direct” student outcomes (e.g., improved reading test scores) but also on what we called “meta-level” or more general outcomes (e.g., increased interest in school activities) and “side effects” (e.g., “zest in experimental program leads to alienation from regular classes”). We also wanted to track both positive and negative effects of the innovation on students.
How can the cases be ordered? You can look at the relevant sections of each case report and note the general level of student impact the writer of the case report had claimed. You place the cases in rough order, and then you reskim the text in each case report to see whether the first impression is justified or whether the ordering should be changed. Some programs aimed high, shooting for everything from student achievement to self-concept, while others were more focused, hoping only, say, for the improvement of reading skills. The reading also showed us that the data were of variable robustness—ranging from detailed and thorough achievement data, interviews, and questionnaire responses to nearly unsupported opinion.
That range suggests the usefulness of a format like Display 8.9. The idea here is to show the objectives and to enter data in the form of phrases or sentences in the appropriate cells. This way, we keep more of the raw data, and we can see how outcomes are related to objectives. Our own matrix legend/shorthand to show the data source (i.e., Attribute Codes) was as follows: U for user, A for administrator, E for evaluator, S for student, P for parent, and so on. If the impact was seen as strong, the letter would be underlined; if uncertain, a question mark would be added; if the evidence was conflicting, an X would be added (i.e., Magnitude Codes).
From relevant sections of each case report, we look for coded material on direct and meta-level
From relevant sections of each case report, we look for coded material on direct and meta-level outcomes and summarize the material in a set of brief sentences or phrases, one for each distinct outcome. We can arrange the data in the Program Objectives and the Direct Outcome columns in parallel fashion to make comparisons easier. Skills and achievement goals might be placed first, followed by affective/attitudinal and other outcomes.
After the data are all assembled into the matrix, review again: Are the cases correctly ordered from high to low student impact? In this case, some criteria for a “high” rating are as follows:
1. The program is achieving most of its aims.
2. The program is also achieving other positive meta-level and side effects.
3. These judgments are corroborated, either through repeated responses from one role through cross-role agreement, or through evaluation data.
Display 8.10 shows how this kind of data display worked out. The rows show two cases (Perry-Parkdale, Masepa) where student impact was high, two where it was moderate (Carson, Calston), one where it was moderate to low (Dun Hollow), and one where it was low (Burton). (For simplicity, we show an excerpted table, with only 6 of our 12 cases.)
Analysis
At first glance, Display 8.10 looks daunting and overloading. But first try a “squint analysis” by asking where in the table the data look dense or sparse.
Looking across rows, we can see that positive effects are more frequent than negative ones (tactic:
Looking across rows, we can see that positive effects are more frequent than negative ones (tactic:
making contrasts, comparisons). Another squint shows us that cases where many negative effects are noted (e.g., Perry-Parkdale, Masepa) are cases where much is being attempted (in terms of the Program Objectives column). Apparently, large efforts are more likely to spin off negative effects along with the positive ones (tactic: noting relations between variables).
Looking at the two right-hand columns shows significant meta-level and side effects in nearly all cases (tactic: noting patterns, themes); in only the lowest impact case (Burton) do we see none at all. Perhaps this pattern means that the claims for meta-level changes in moderate-impact cases are suspect. Maybe so—they are less likely to have repeated mention (underlines) and have fewer instances of multiple-role confirmation (tactic: using extreme cases).
To clarify the last two columns, we defined meta-level outcomes as congruent with the program’s purposes but affecting the more general aspects of students’ functioning. Note the reading program at Calston: It led to the direct outcome of improved reading skills; but by providing interesting materials and opportunities for independent student work, it may have induced the meta-level outcome of increased student self-direction. Side effects are more unintended: Note the alienation of Perry-Parkdale students, who liked their program’s concreteness and relevance but thereby came to dislike their regular high school’s courses and activities.
During analysis of a case-ordered descriptive meta-matrix, first do a general squint. What does the matrix look like—where is it dense, where empty? Draw conclusions, and write them out. Then, look down particular columns, comparing/contrasting what things look like for high, medium, and low cases. It may be useful to look at more than one column at a time to note relations between variables.
A first-cut case-ordered descriptive meta-matrix can be big. We sometimes have used up to 20 columns in trying to sort out a wide range of variables that might bear on the main “ordering”
variable. Excel database software comes in very handy for these types and sizes of matrices. Expect to make several cuts at the format of the matrix before you settle on the one best for your purposes.
Recheck the ordering of cases at several points as you proceed; ask a colleague to confirm or argue with your ordering. For less complex databases, it often helps to enter actual direct quotes from case reports, rather than constructed sentences or phrases. They give an even more direct grounding in the data for the analyst and the reader.
Notes
Miles and Huberman’s classic Display 8.10 employed a legend of single letters and symbols to differentiate between sources and corroboration of data. Today, software such as Word and Excel can color code or use rich text features and varying font sizes to separate different types of data from one another in a matrix and make them much more visually discernable at a glance.
A case-ordered descriptive meta-matrix is usually a fundamental next step in understanding what’s going on across cases. Assuming that you have a good basis for ordering the cases, it’s far more powerful than a partially ordered matrix: Patterns can be seen for high, medium, and low cases; and the beginnings of explanations can emerge. “Aha” experiences are likely. Generally speaking, time spent on this is well worth the investment because so much later analysis rests on it.
Also remember that rows and columns can be transposed for your analytic goals. It’s usually easier to show effects by rows rather than columns, simply because there is more space to write headings there. You can create a case-ordered effects matrix that sorts cases by degrees of the major cause being studied and shows the diverse effects for each case. The focus is on outcomes across multiple cases when multiple variables also need to be examined simultaneously (see Display 8.11).
Display 8.11
Case-Ordered Effects Matrix Template
Source: Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage Publications.
Case-ordered effects matrices are especially useful when you anticipate a wide diversity of effects from a general cause—itself varying across cases. They fall just about at the fuzzy boundary between cross-case description and explanation, providing leads for theory building and testing. The matrix also helps you avoid aggregation, keeping case data distinct, and lets you look at deviant cases. In sum, a case-ordered effects matrix enables you to bring together and order several cases, variables, and time simultaneously to see their multiple outcome configurations.
Closure and Transition
Most ordering displays are complex because we are attempting to multidimensionally and simultaneously document cases, ranks, variables, actions, time periods, outcomes, and/or assessments. Basic software literacy and access for most of us are limited to two-dimensional renderings, when we wish we could draw in three dimensions or animate some of our graphics.
But two-dimensional matrices and figures bring us one step closer toward understanding complex participant processes.
Exploring, describing, and ordering data are necessary prerequisites for the next chapter’s methods: explaining. Rich and insightful answers are generated from posing “how” and “why”
questions.