sample eligibility criteria and selection

Round 1 sample.

Sample eligibility criteria and selection. Participants were practicing teachers in North Carolina that provided a variety of perspectives from traditional, charter, and private schools. The target number of teachers for the Round 1 questionnaire was 150. Figure 7 provides an overview of the sample selection process. I selected the panel from ten counties in North Carolina that were designated based on the proportion of: private/public schools, school size, school locale, and school level compared to national proportions.

I derived the list of ten counties from a ranking system that I devised to single out the ten counties most representative of each of the categories within each variable. I reverse-ranked each county (the school with the most of any category was given the lowest rank, which is the highest number) by the number of schools with each characteristic. For example, with the school

location variable, each county was reverse-ranked separately on how many schools are

characterized as city, sub-urban, town, and rural. A county with many schools located in cities ranked high (had a high number) in the city category but ranked low in the rural category if it did not have many schools in rural settings. Then, I added all of the county reverse-ranks across categories and variables to create an overall reverse-rank composite. The counties with the highest numbers indicated those counties that had the most schools that represented all

categories. Due to the incompatibility of school variables across traditional, charter, and private school datasets, I only used traditional school information to calculate ranks. The presence on the list of counties of some of the most urban and most rural counties in North Carolina suggests that the process was successful in narrowing the sampling frame to ten counties that contain the most schools with a wide range of important school characteristics from which to sample.

117

Figure 7. Round 1 sample selection process

All counties in NC

•School-level variables: level, size, location, & setting

•Reverse-rank: # in EACH category of each school-level variable

•Reverse-rank Composite Score

•Top ten reverse-rank composite scores

Ten counties in NC

•NC Administrative Data

SCHOOLS in ten counties:

Traditional, Charter, &

Private

•Match national proportions of school-level variables for each school setting

•Target sample size: 150

•SCHOOLselection: random number generator

Stratified random

SCHOOL

Sample

•TEACHERselection

•School websites

•Subject, grade, last name (alphabetical)

Quasi-random

From these counties, I drew my sampling frame. Using North Carolina administrative data and the Common Core of Data, I created three separate lists of schools from the ten counties for traditional, charter, and private settings. The list of traditional public schools was

accompanied by teacher names and emails. However, I found that a significant portion of these names did not appear on the matching school website, indicating that the teacher may no longer work at that school.

With the three lists of schools ready, I calculated target proportions of school-level variables based on national averages for each category of each variable, using the 2015-16 Common Core of Data and the Institute of Education Sciences’ 2016 Digest of Education Statistics. For example, I calculated the percent of private and public schools nationally (26% and 74%, respectively in 2016) and applied this proportion of public and private schools to my sampling frame. These proportions indicate that in order to approximate national proportions, 39 private schools and 111 public schools should be in the sample. I also separated public schools into traditional and charter schools. Within the private and public school division, I calculated the same proportions for each category within each of the following school-level variables: school setting, location, school level, and school size.

For each list, I used a random number generator supplied by random.org to perform stratified random sampling. For each school-level variable, I counted the number of schools within each category and randomly drew from within that number the previously calculated target proportion of schools within that category. However, I drew five times the target number for each category in an effort to create a large enough sampling frame to arrive at the desired sample size of 150. For example, on the ten county list of private schools, there are a total of 205 private elementary schools. In order to create a proportionate sample, 10 private elementary

schools were needed, so I randomly drew 50 private elementary schools from this list. I continued this process for each school setting (traditional, charter, and private) applying it to school level (elementary, middle, and high), school size (these categories varied by school setting), and school location (city, sub-urban, town, rural). I applied the same proportions to traditional and charter schools. This step completed the stratified random selection of schools from the top-ten ranked counties by school variables. The list included 88 traditional schools, 29 charter schools, and 92 private schools.

I found that a larger list of private schools was necessary because a substantial portion of private schools originally selected do not publish teacher email addresses on their school

website. The online feature of this study makes email addresses of participants imperative. The inclusion or exclusion of teacher email addresses among the three school settings may introduce bias into the sample, if schools that post teacher email addresses are systematically different in their working conditions compared to those that do not.

With a stratified list of schools prepared, next, I began the process of selecting teachers. Three factors led me to quasi-randomly sample teachers and record their email address from individual school websites. The quasi- characteristic of the sampling results from the lack of a truly systematic random sampling method. First, I had no data on individual teachers from charter or private schools that could be matched with school-level data, so procuring names and email addresses from individual school websites was my only option to select teachers for these two settings. Second, I found that a significant number of the teacher names provided on the traditional school list were not found on the school website, most likely indicating that these teachers were no longer teaching at this school. Third, there was substantial variation among school websites in the information they provided, including labels for teacher positions and types

of information available for teachers (including whether or not an email address was provided). These conditions necessitated that I design a method for extracting names from school websites.

I devised a method to approximate stratified random sampling with the array of schools randomly sampled from the ten counties. First, I visited the website of each school to determine if the school included a list of faculty and faculty email addresses. If they did not, I could not include teachers from that school in the study. If they did include teacher names and email addresses, I randomly chose 4-5 teachers from the school website, depending on the size of the faculty. To make these selections, I cycled through the alphabet by teacher last name and simultaneously cycled through grade levels. I moved through subjects as they were listed on websites (tested and non-tested); the organization of subjects varied greatly across websites, which likely added more randomness to the process. I picked up on the next school website where I left off at the last. For example, if the last teacher I selected on school X’s website taught fourth grade and their last name began with J, I began selecting teachers on the next school website in fifth grade and with the letter K. Since I sampled each school level as a group (e.g. all elementary schools at one time), there was no systematic matching of lower grades to letters at the beginning of the alphabet. When more than one teacher had a last name beginning with the same designated letter, I systematically chose in sequence. The first time, I chose the first name, the second time, I chose the second name, and so forth. I selected five teachers from each public school and four teachers from each private and charter school, since these schools most often had smaller faculties. A few private schools have faculties with only 2-5 teachers. From these

schools, I only selected one teacher. Thus, from the randomly selected list of schools and using the described process, I quasi-randomly selected teachers from 88 traditional school websites, 21 charter school websites, and 30 private school websites. In all, I selected 572 teachers from all

school settings, 348 from traditional public schools, 89 from charter schools, and 134 from private schools.

Round 1 study initiation. Each person in the sampling frame received an emailed letter of