Data Analysis

In an IA, data analysis is concurrent with data collection. Each new piece of information is considered in light of previous information starting with the original quantitative data that drove the focus of the inquiry. Thus, the analysis and development of findings is an iterative process as described below.

25The process and its related conceptual framework influenced the design of the federal Child and Family Services Review, the evaluative methodology utilized by the Administration on Children, Youth and Families to review all 50 state child welfare programs, as well as child welfare systems of the U.S. Virgin Islands and Puerto Rico.

1. Debriefings

The debriefings are the bridge between data collection and data analysis. Debriefings in an IA consist of members of the team reporting on their interviews, observations and case file reviews each day, often including representatives of the system under study. The reporting process begins with this instruction: Describe who you interviewed or what you observed; what step of the process or text were you discussing or observing; name any insights you received about how the work is organized that might be relevant to our inquiry. Debriefings are a facilitated process that provide the opportunity for the team to begin identifying important themes to pursue through additional information collection and analysis as well as feedback and clarification from the agency.

For example a team member may have understood from interviews that the current policy on placement does not allow a child to be placed under the care of someone who would not be a suitable adoptive parent. If this is not actually the policy, a local representative is likely to identify and clarify the misunderstanding and produce the actual current policy. The team member must then determine if this was a practitioner’s misunderstanding of policy or a widely held view of current policy. It is not uncommon for an IA to find “policy mythology,” with practitioners believing a certain practice to be required by policy or law when it is not.

Individual daily debriefings are also conducted. Individuals reviewing case files debrief their reading with one of the IA leaders while another team member takes detailed notes. When debriefing a file, the readers note any places where practices may have had a significant impact on the case outcome and discussing any problematic conceptual or administrative practices that were noted. Readers also look for what was documented and how.

In Fresno County, the on-site data collection team debriefed as a whole at the end of each of the four days of on-site data collection and analysis. This meeting included representatives of Fresno County Quality Assurance Staff and leadership. In addition team members debriefed individual file reviews with Dr. Ellen Pence, Gayle Samuels and Kristen Weber.

2. Developing Findings

In an IA, a final debriefing is convened at the end of the intensive data collection period to formulate preliminary findings based on emerging themes in the observations. ²⁶ This preliminary list is used as the starting point for the findings that will be enumerated in the IA final report. This process generally requires a half day. It is facilitated by a senior team member and recorded.

Prior to the final debriefing, each subgroup of the team meets to compile its observations. During the final debriefing, each subgroup shares its observations. The facilitator asks for any and all insights or statements that link poor case outcomes to an organized, coordinated practice in the system. At the same time they are asked to comment on practices that seemed to produce equity rather than disparity.

The facilitators have three frames for helping to guide the discussion:²⁷

• the map of the process developed after the big picture interviews, asking members to comment on each step of case processing;

• the eight trails of inquiry asking members to locate a source of a problematic practice (is it linked to the mission statement or job description, a rule or regulation, a procedure or administrative tool); and

• the problematic features of legal and human service institutions previously enumerated.

While these three frames are not systematically discussed during debriefings they do inform the facilitator’s questions and lines of inquiry with reporting teams to generate the most complete list of possible findings from the week.

Each practice that is observed or uncovered as a likely contributor to a poor case outcome is first noted as a practice, then screened out if it seems to be an example of poor case work, individual practitioner error or a one-time event linked to either unusual circumstances or the idiosyncratic practice of an individual worker.²⁸

Keeping the focus on the way the worker is organized rather than on his or her personal skills or work habits is not always simple or straightforward to do especially when doing an investigation with people who may be involved in training workers on best practices. Investigators are often quick to conclude: this worker should have, if well trained, done x or y or z. While such an observation may be true, the question for the team is how this practice comes about and is allowed to stand in for acceptable processing. Take, for example, case file documentation of a worker’s initial interview with an 8-year old child who had just moments before arrived at a foster home:

The worker asked the child about what had happened that night and the child started to cry.

The worker rephrased the question two more times and both times the girl cried and gave no information. The worker then notes that she concluded the interview and told the child she would return in the morning.

27 This facilitation requires a person or persons who are experts in the IA process and the three frameworks as applied to legal and human service agencies. It is a directed debriefing process. By using these frameworks to ask questions of the IA team, the facilitators keep the inquiry focused on managerial systems rather than skills or failures of individual practitioners.

28 Such an example might lead to an exploration of why an individual practitioner’s mistake goes unchecked in the system, thus uplifting the question of how people in the system are held accountable to a standard of practice.

This is one of many examples of workers not engaging with children in supportive and nurturing ways when the children were obviously suffering. The worker did not document any attempt to comfort the child, only her attempt to conduct the initial post separation interview. The IA process seeks to find out how this practice becomes acceptable, not to point out how a well-trained worker would have comforted this child. The eight methods provide a starting point for that inquiry: what task or function is the worker asked to accomplish in this encounter? Is support and comforting part of that? If so, how is the worker held accountable to that work? Does this behavior go unchecked by supervision?

It is the repetitive nature of an observation that allows the IA team to identify it as a finding. If a practice is seen only rarely in the process it is not considered a finding of the analysis. However, if many examples are found in the files and observations of children and parents, then it rises to the level of a supported finding.

On-site, the IA facilitators and leaders take a few hours to synthesize the findings into an organized presentation for agency and community leaders. Then a large debriefing is held to discuss the preliminary findings with the agency leadership. It is up to the host agency to determine who to invite to this meeting.

The IA leaders make a presentation calling on the leaders of the subgroups to supply the details and case examples for key points or findings. Agency leaders are asked to comment on each finding in order to determine if there are explanations for practices that were not considered or if investigators misunderstood how a certain practice is organized in the agency. Detailed notes are taken. This process is very useful in indicating what follow-up information is needed before formulating final findings It also ensures that management foresees the findings of the report rather than feeling blindsided.

On the final day of the on-site work the team followed this debriefing process and met with Fresno County child welfare leadership to provide an overview of the preliminary findings and receive feedback.