each stakeholder, to determine the reliability and sub-construct validity of the
questionnaire. To obtain a sample size for each stakeholder large enough to determine reliability, it was necessary to recruit participants for the pilot study from more than one school. Although the instrument was intended to measure a school’s STEM culture, the reliability of the items could be tested using participants from different schools in that the reliability is not measuring the construct itself, but the ability of the items to measure the intended domains. For each category of stakeholders, 50 participants were recruited to complete the survey.
Participant recruitment. Administrators were contacted through the district offices of two local school districts and asked to complete the surveys online. The surveys were completed anonymously, with no record of the administrator’s personal information or IP address. Counselors were contacted through a local email list of counselors in local districts after approval by the districts, and surveys were completed anonymously, with no record of the counselor’s personal information or IP address. Teachers were recruited through school-wide emails to the schools in one school district. The surveys were completed anonymously, with no record of the teacher’s personal information or IP address.
High school teachers of core subjects were asked to choose their main teaching assignment from a list of options. Students from one school in a local school district were asked to complete the survey on school computers during a homeroom period. The surveys were completed anonymously, with no record of the students’ personal
information or IP address. Parents were contacted through a school-wide email list, and a local parent function conducted by a guidance department. The surveys were completed anonymously, with no record of the parents’ personal information or IP address. No individual demographic information was asked of any of the stakeholder groups within the survey. The survey was conducted via Qualtrics, an online survey tool, and data was compiled using Microsoft Excel.
Reliability analysis. Five reliability tests were completed using items from each stakeholder group within the STEM culture for a school including the administrators, counselors, teachers, parents, and students. A sample for each stakeholder was chosen to complete a pilot study of the questionnaire. It was not imperative that the sample be a part of the same school community to determine the reliability of the items. Once the pilot studies were completed, reliability of each item was calculated using Cronbach’s Alpha (Chronbach, 1951). As suggested by Cronbach in his notes on alpha (Cronbach & Shavelson, 2004), standard error of each item was calculated as well. Cronbach
suggested that despite the rampant use of alpha in research, stating that his 1951 paper had been cited 5,590 times, standard error is a better determinant of reliability than alpha because it shows variability of each item. Items were removed as necessary based on Cronbach’s Alpha and standard error to maintain a high level of reliability. Acceptability for alpha was set as follows for this study: below 0.7, unacceptable; between 0.7 and 0.75; minimally acceptable; between 0.75 and 0.8, acceptable; between 0.8 and 0.9, very good; above 0.9, consider shortening the scale (DeVellis, 2003).
The number of participants for each stakeholder was determined to be 50 participants. It is a commonly debated subject determining the necessary number of participants for a reliability study using Cronbach’s Alpha. Common practice states that
the larger a sample size using Cronbach’s Alpha, the more accurate the results will be. Varying conclusions have been made in the literature stating sometimes 50 participants is acceptable, sometimes the minimum number of participants can be as large as 400
(Yurdugul, 2008). Because the reliability study was done with members of five different types of stakeholders, 50 participants from each area were chosen to include a total of 250 participants.
Sub-construct validity within domains. Once the Cronbach’s alpha for many of the sub-constructs was found to be unacceptable, an exploratory factor analysis (EFA) was completed using the SPSS computer software package to determine if the items would cluster according to categories other than the categories outlines in the literature review. A Kaiser-Meyer-Olkin Measure of sampling adequacy was used to indicate that items in the survey were strongly correlated enough to conduct a factor analysis
(Tabachnick & Fidell, 2001). Factors were extracted using maximum likelihood methods with oblique rotation (promax) because the factors were assumed to be related. The data presented was complete with no missing data; therefore, no reduction of data was
necessary. A factor solution was obtained using Kaiser’s criteria, the scree plot, the interpretability of the results, and the model fit indices indicated by SPSS, which indicated a three-factor model (Preacher & MacCallum, 2003). The EFA was used to determine the retention or removal of items from the survey being analyzed (Worthington & Whittaker, 2006). Items with a factor loading of .40 or higher were retained based on study criteria (Tabachnick & Fidell, 2001). The exploratory factor analysis was used to determine a new organizational schema for beliefs within the stakeholder groups and identified a new structure including two major sub-constructs under beliefs: beliefs about student activity in the classroom, and beliefs about curriculum and lesson design.
Final instrument development. After the five iterations of the pilot study were completed, and items were determined to be reliable, a sub-construct validity was developed, and the final version of each iteration of the questionnaire was completed. The survey was constructed using Qualtrics (Qualtrics, Provo, UT), with one survey that asked participants to identify which type of stakeholder they were, and then provided the appropriate items for each stakeholder.