Reporting

The scoring rubric developed by Abd-El-Khalick (2013) assigns a score to a unit of analysis ranging from +3 to -3 depending on the extent to which that unit represents a targeted NOS aspect. The rubric draws a distinction between an explicit versus an implicit representation of the target NOS aspect. Research by Abd-El-Khalick et al. (1998) on implicit versus explicit instructional approaches revealed that implicit strategies, such as engaging in scientific activities, do not translate into an

41 understanding of NOS, whereas an explicit approach is more effective in ensuring comprehension of NOS. An explicit approach would entail educators differentiating between observation and inference during activities as opposed to the learners having to infer this crucial distinction from their activities.

When one uses the scoring rubric, all units of analysis, including paragraphs, diagrams, black boxes, tables and captions in the textbook targeting the same NOS aspects, are grouped together and examined holistically. The corresponding score for the target NOS aspect is based on an examination of all relevant materials. Below is the scoring rubric used in the analysis, providing the distinction between explicit and implicit representation.

3.5.3.1 Scoring rubric

Points are allocated by the raters from a scale of positive three points to negative three points. This allocation depends on the degree of explicitness or implicitness of the NOS represented in the units of analysis.

 Three points = Explicit, informed, and consistent representation of the target

NOS aspect:

i. Explicit statements that convey an informed representation,

ii. Consistency across the selected chapters or sections in addressing the target NOS aspect, and

iii. Consistency in addressing other directly related NOS aspects.

 Two points = Explicit, partially informed representation of the target NOS

aspect:

i. Explicit statements that convey an informed, but incomplete representation, and ii. Consistency across the selected chapters or sections in representing the target

42 remaining silent in terms of addressing other related NOS aspects that ensure a complete informed representation.

 One point = Implicit, informed, and consistent representation of the target

NOS aspect:

i. An informed representation of the target NOS aspect could be inferred from the textbook materials (e.g., relevant explanations, activities, examples, or historical episodes lacking structured, reflective prompts or explicit statements), and

ii. Absence of other explicit or implicit messages, which are inconsistent with the inferred implicit representation.

 Zero points = The target NOS aspect is not addressed: i. No explicit or implicit treatment of the target NOS aspect, or

ii. Not enough materials (statements, examples, historical vignettes, etc.) to make an informed judgment or to convey to the textbook reader a sense about the target aspect of NOS one way or the other.

 Negative one point = Implicit misrepresentation of the target NOS aspect: A naïve representation could be inferred from the textbook materials.

 Negative two points = The textbook materials convey mixed explicit and/or

implicit messages about the target NOS aspect:

i. Implicit, informed representations that could be inferred from some parts of the textbook materials are countered by explicit, naïve statements in other parts, or ii. Explicit statements that convey conflicting messages about the same NOS aspect.

 Negative three points = Explicit, naïve representation of the target NOS

aspect: Explicit statement or statements that clearly communicate a naïve

representation of the target NOS aspect.

(Source: Abd-El-Khalick: NOS textbook analysis methods/ UIUC: April 20, 2013/ Scoring rubric)

43 Consistent with Elo and Kyngas`s (2008) definition of deductive content analysis approach, only the aspects from the units of analysis that connect with the tenets of the NOS are chosen and recorded in the scoring sheet. The outstanding data which could not fit into the predetermined category was categorised as “other”. Elo and Kyngas (2008) propose that the outlying data form their own categories based on principles of inductive content analysis. The generation of new categories is beyond the scope of this research and may be recommended for further studies.

3.6 DATA ANALYSIS

Scores agreed upon by the three coders through discussions, and with reference to the scoring guidelines, were assigned to each of the eleven tenets of the NOS, and the number of occurrences per tenet counted. Each textbook has four strands, therefore, the extent to which each NOS tenet is represented in each strand was documented and a comparison made amongst the strands. Next, a strand by strand comparison was made across the three textbooks. Finally, a cumulative score for each textbook was calculated. Individual scores for NOS tenets range from -3 to +3 for 11 aspects giving a cumulative score of -33 to +33. The higher the cumulative score, the more explicit, informed and consistent is the representation of the NOS.

3.6.1 Reliability

Reliability is achieved in a study when the tool used for assessment produces consistent and stable results. According to Krippendorff (1980) three categories of reliability can be established: stability, reproducibility and accuracy. A stable analytical tool will reproduce the same set of results when applied to the same data set at different times; reproducibility is shown if similar findings emerge from the same data set when different methods of analysis, based on the same constructs, are employed; and accuracy is how close the results of an analysis are to a known standard.

44 Three raters were used for this study and so the degree of consistency between the findings of raters could be measured in what is known as inter-rater reliability. Inter- rater reliability is the degree of agreement amongst raters, or how much homogeneity exists between the raters’ scores (Multon, 2010). It is crucial to calculate inter rater reliability as the content being assessed may be interpreted in different ways by different observers.

In calculating inter-rater reliability, Cronbach`s alpha, an intraclass correlation coefficient (ICC) (Norusis, 2003) is computed to provide a reliability measurement. Cronbach’s alpha coefficient measures internal consistency reliability. This provides an understanding of the extent to which the raters` scoring shows consistency (Stemler, 2004) within a class. A class in this case is a tenet of the NOS. ICC produces two sets of correlation coefficients: a single measurement showing the reliability for one single rater and an average measurement for the reliability of different raters averaged together.

Cronbach`s alpha is selected over alternate coefficients such as Cohen`s Kappa or Pearson`s correlation coefficient as the latter are used for pairs or for two judges. Cronbach`s alpha is useful for more than two raters and pairing is not required. A random model measuring consistency was employed to calculate Cronbach`s alpha at a 95% confidence interval which provides a 95% chance of obtaining true reliability amongst the raters.

3.6.2 Validity

A valid test is one in which the findings are supported by empirical evidence. According to Krippendorff (1980) there are several forms of validity based on the nature of data used in the validation process. Validity ranges from data oriented validity, semantical validity, sampling validity, process oriented validity, construct validity, pragmatical validity, correlational validity to predictive validity. Construct validity relevant to this study is when the inferences made from a content analysis

45 are acceptable as evidence owing to the process of deductive category application in which the categories are based on accepted theories or models.

3.7 CONCLUSION

The research methodology chapter has explained the use of the qualitative paradigm as the most appropriate approach to collect the data. The design followed a deductive content analysis in which units were selected, coded, and scored in connection with predetermined categories of the eleven tenets of the NOS. In order to test this analytical framework, a pilot study was conducted, which is reported in chapter 4 below.

46