Post-School Outcome Data
Collection and Use: Questions
State Data Managers Should Ask
Prepared by Marsha Brauen
Vice President, Westat
University of Oregon
July 2006
Questions for Data Managers To Ask on Their Role in Supporting Post-School Outcome Data Collection and Use
PURPOSE:
The purpose of this document is to give new and experienced data managers a guide to ascertain their roles and responsibilities in regard to the collection, analysis, and reporting of post-school outcome data for Indicator 14 of the Part B State Performance Plan (SPP). Indicator 14 requires states to report annually on the “percent of youth who had IEPs, are no longer in secondary school and who have been competitively employed, enrolled in some type of postsecondary school, or both, within one year of leaving high school.”
To assist data managers, NPSO has developed a series of questions that will guide data managers through the key activities states will need to complete to collect, analyze, and report data for Indicator 14. As you review this document, remember that the activities your state is undertaking related to Indicator 14 are likely intertwined with the activities your state is undertaking for Indicators 1, 2, and 13.
HELPFUL HINTS:
1. Many of the questions listed below are best asked of your state transition coordinator, your state director of special education, or your personnel supervisor. 2. There are numerous technical assistance documents on the NPSO web site that will
help your state to answer some of the questions listed. A list of NPSO tools is provided at the end of this document.
3. Before asking the questions below, read your state’s SPP response for Indicator 14; your state’s SPP should be available on your web site.
4. If your state is contracting for some of the activities noted below, you may need to know the answers to some of these questions as you may have some roles or responsibilities related to the contracted activities, e.g., providing a database of student names and identifiers.
QUESTIONS TO ASK: Background
Timeline
3. What is the timeline for data collection, data analysis, and reporting of data? What are the critical dates I need to be aware of?
Data Collection Approach
4. What is the approach our office will be taking in the collection of these data? Will data be collected directly from students? Will extant data be used?
5. Will we be doing an exit survey as well as a follow-up survey? If so, how will this survey be completed? By whom?
6. How will we find age-outs, dropouts, and early leavers?
7. Do we have incentives in place to increase the response rate (especially for age-outs, dropouts, and early leavers)?
8. How will we ensure that students have not re-enrolled? Can we use our unique student identifiers or combinations of student characteristics to remove such students?
9. Will a special web application be developed to collect these data? If so, who will develop this? Is there an expectation that I will have a role in this process? What will it be?
10. What is our state’s rule, if any, on obtaining consent from students or parents for the collection of these data? Should this be discussed with our state’s legal counsel? If consent is needed, how will we get consent?
11. Will we use sampling? Of districts? Of students within districts? Will we use the NPSO sampling calculator? Will we use an internal or external sampling statistician to draw the sample(s)?
Information To Be Collected
12. What will the data collection instrument (or instruments if there will also be an exit survey) look like? Has it been piloted? If so, what were the results of the pilot?
13. What information will be collected in the exit survey, if applicable, and followup survey?
14. Has the data collection instrument been approved by our forms clearance committee (if applicable)? Does our state have such a committee? How does it operate?
15. How will we monitor the collection of these data by LEAs (if applicable)? How will we assure the accuracy, validity, and reliability of the data collected? Can this be part of our regular monitoring of LEAs? Can we put in place a set of checks of data for reliability and validity when received from LEAs? What would these look like?
Data Collectors
16. Who will be collecting these data? The special education office? A university? School districts? A contractor? Another entity?
17. Who will the data collectors be? Teachers? Transition coordinators? Data collectors paid by a contracted entity?
18. Who will train school district data collectors (if applicable)? When? How will we train new data collectors as needed? Can this be done annually as part of the regular training done for school district data collectors?
Information To Be Provided to Data Collectors
19. What will we need to provide to the data collectors (university, school district, contractor, another entity) from the special education database?
a. A list of students?
b. A list of students with basic demographic characteristics, contact information, etc.? c. A list of students with basic demographic characteristics, contact information, and
other information (e.g., information from IEPs re: transition goals)?
d. Nothing, as school districts will get this information from their own databases? 20. When will it be necessary to provide this information to the data collectors (if
applicable)?
21. If student lists are to be provided by districts, who will be responsible for creating a database of the names and other data provided?
Data Managers’ Role(s)
22. What is the expectation of my role in this process? a. To provide a list of exiting students?
b. To draw a representative sample of districts for this collection? c. To weight the sampled data?
d. To verify that responses are representative of the state? e. To analyze the data collected?
f. To develop reports on the data? Data Collected
23. Will individual student record data or aggregate data be received from LEAs (if applicable)?
Data Analysis and Reporting
25. What is our state’s rule regarding the masking/suppressing of data based on small cell sizes for purposes of confidentiality? Should we consult our legal counsel? If we have such a rule, is the small cell size number the same as or different from our AYP sub-group size for students with disabilities under NCLB? What are the implications of this rule for reporting our post-school outcomes data to the public for each school district? 26. What is the plan for the analysis of the data collected? Will analyses beyond those
needed for the APR be completed? What will they be? Will drill-down analyses be needed?
27. What types of reports will be generated? Who is the audience, e.g., OSEP, the public, school districts, schools? What data will we report to OSEP and the public? How will the reports be structured? What graphics will be needed?
28. Are there plans to continue to collect data on students into the future? If so, how will this occur?
Data Use
29. How will the data be used to drive change/improvement and align with SPP measurable and rigorous targets?
NPSO Tools
*8/8/08 - The documents below have been revised and updated. Please go to our website to view all of our current tools and products.
http://www.psocenter.org
Sampling Checklist: To Sample or Not To Sample. That IS the question. This checklist provides guiding questions for a state to consider. (draft 3/2006) Data Collection Procedures Checklist. This checklist provides guiding questions about who, what, how, by whom, and when data are to be collected. (draft 3/2006)
Reporting Checklist. This checklist includes public reporting requirements for APR submissions in 2007 and 2008 and other reporting considerations to facilitate usability and accessibility of the data. (draft 3/2006)
An Indicator #14 Checklist (MS Word doc) (August 2005)
A Case Study (MS Word doc) for Indicator #14 provides states with a sample of a completed Part B-SPP for 2006-2011. (August 2005)
Timeline for Part B SPP (MS Word doc) for Indicator #14 includes timelines, state activities, and resources. This document links directly to the tools needed for a state to consider Sampling and to examples of Student Exit Surveys and Data Collection Surveys. (October 2005)
