Item generation

The rationale for the construction of a school leadership capability scale was that it would enable the description of the core capabilities required by school leaders at different levels of leadership, and facilitate self-reflection and capability building. The initial step, item generation, required the researcher to provide theoretical support for the initial item pool based on the literature review and existing scales of similar nature (Pacico, Zanon, Bastianello, & Hutz, 2013). Systematic steps taken in the generation of items will enhance the content validity of the measurement tool. In this study, a deductive approach was therefore taken in the item generation process. This involved developing an adapted version of the theoretical model that guided this study, the Leadership Pipeline Model (Charan et al., 2001). Three other capability frameworks/professional standards documents were also used as references to generate capability items specifically relevant to school leadership.

The foundation and core reference of this research study is the Leadership Pipeline Model described by Charan et al. (2001), which endorses different skills and capabilities sets for different stages of leadership development. To increase relevance to the Australian school leadership requirements, other documents were used as references to construct the survey content. They included:

• the NSW Public Sector Capability Framework (2010) • the Australian Professional Standards for Teachers (2014) • the Australian Professional Standards for Principals (2011).

87 Table 4.1 gives an overview of each of the documents used as a reference in item generation for the SLCS, which was used as a measurement scale to capture data in the quantitative part of the study. This is followed by Table 4.2, an item matrix indicating items generated using these documents as references.

88 Table 4.1 References used in item generation

89

Key:

1= Leadership Pipeline Model 2=NSW Public Capability Framework 3= Australian Professional Standards for Teachers

4= Australian Professional standards for Principals

90 In the item generation process, DeVellis’s (2003) recommendations on scale development were followed, with attention focused on the writing of short items, the use of relevant language and the avoidance of double-barrelled items and the use of double negatives. To answer Research Question 2: “How do school leaders in NSW public schools perceive their strengths and development needs as organisational leaders?” the five capability sets created for the SLCS were designed to measure participants’ perceptions of the importance of each capability item and how they associated their strengths with each item. The capability sets illustrated progressive leadership capability, from leading self to leading others, to leading other leaders, leading the organisation and leading the community. Each of these sets comprised ten capability items.

Against each capability item, two columns were created to capture the ‘importance rating’ and ‘strength rating’. This was to facilitate easy reading and participant reflection on leadership strengths. Participants were asked to rank each capability using a five-point Likert scale (1 Very important; 2 Important; 3 Moderately important; 4 Important to a small extent; 5 Not important) for the ‘importance rating’ column, and similarly, using a five-point Likert scale (1 Very strong; 2 Strong; 3 Moderate; 4 Weak; 5 Very weak), to rate their strengths in the ‘strength rating’ column for the strength rating.

The choice of a five-point Likert scale was based on suggestions from the scale development literature, which pointed out that a five-point Likert scale is less mentally demanding than a seven-point or nine-point scale, and therefore more suitable for a larger scale study (N>100), while the use of a seven-point scale would be preferable for smaller-scale studies (Finstad, 2010; Leung, 2011).

The survey was administered online using software known as Qualtrics. Qualtrics is a web- based survey tool licenced for use by Western Sydney University. It is a user-friendly software with a range of dynamic data-capturing and reporting tools for researchers.

A copy of the survey questions, incorporating the BFI and the SSEIT, is attached in Appendix 2.1.

91

4.3.2.1 Assessing face and content validity of the survey

Step two comprised two parts: the assessment of the face validity and content validity of the survey. In this study, the Delphi technique, a widely used and accepted method in a survey validation process (Taylor & Judd 1989; Young & Jamieson, 2001), was used to establish the face and content validity of the survey. This technique seeks to aggregate opinions from an expert panel comprising very experienced personnel in leadership development. The backgrounds and experience of the expert panel members are briefly outlined in Table 4.3.

Experts Background and experience

Expert 1 Senior consultant and executive leadership coach; adjunct professor in an Australian university on leadership coaching. Very experienced with competency/capability frameworks.

Expert 2 Former (retired) CEO of a trade association in NSW; author of a leadership book.

Expert 3 Senior learning & development consultant. Very experienced with competency/capability frameworks.

Expert 4 CEO of an engineering company. Experienced in leadership development.

Expert 5 Retired primary school principal and leadership scholar.

The input of an expert panel helped to prove the validity of the instrument: i.e. that it is measuring what it is meant to measure. This process confirmed the face validity, measuring whether, at face value, the questions appear to be measuring the construct. The panel provided a thorough review of the appearance of the survey, the clarity of question wording, terms of feasibility, consistency of style and formatting, helping to indicate the survey’s

92 appropriateness for its purpose and content area, a necessary step in establishing face validity (DeVon, Block, Moyle-Wright, Ernst, Hayden, Lazzara, Savoy, & Kostas-Polston, 2007).

Content validity is the extent to which the elements of a measurement procedure are relevant to appraising the different facets of a construct. Establishing content validity is a necessary preliminary task in the construction of a new measurement instrument (Bell, 2007). It measures whether all important aspects of the construct are covered and clearly defined.

4.3.2.2 Expert panel input

Prior to the construction of the survey, three consultants experienced in leadership competency/capability frameworks development in the public sector and business sector were consulted in a group discussion. They suggested that a maximum of ten capabilities per stage was optimum, as this would sufficiently cover the capabilities deemed necessary for each capability set, and the number of items listed would not overwhelm participants. Subsequently, the expert panel of five was requested to rate each item on the survey scale on a three-point rating scale (1= unacceptable, 2= modifications required, 3= acceptable) to identify deficient areas, and to provide recommendations or suggestions regarding the evaluation criteria to measure the face and content validity of the survey. Members of the panel were also asked to indicate any perceived inconsistencies or potential difficulties regarding the clarity and succinctness of individual items. In addition, similar feedback was sought from the researcher’s academic supervisors to improve the first draft of the survey. Recommendations for improvement were identified in round one of this evaluation. They included:

1. Instructions unclear: some parts of the instructions were confusing, especially on the interchange of levels of leadership.

2. Design of the survey layout: suggestions were made to list all 5 sets of capabilities in one questionnaire to enhance clarity and continuity; a side-by-side layout with subheadings “importance rating” and “strength rating” was suggested to allow participants to complete the questionnaire in one go, reducing fatigue and making it less time-consuming.

3. Some examples of behaviours needed to be more concise to minimise confusion. 4. Typos, grammatical errors and spelling errors were pointed out and corrected.

93 5. Wordings and semantics: some members of the panel considered the word

“competency”, appearing on the original survey, to be “old fashioned”, “last century” and “threatening” to participants. A replacement word, “capability”, was accepted.

6. Jargon was spotted and replaced.

7. Word choice: the panel cautioned against using vague terms such as effectively and efficiently, instead suggesting listing behaviours to illustrate meaning.

8. Rephrasing: a few examples of behaviours needed to be rephrased to increase clarity and create a more neutral tone.

A new layout for the survey was designed and mapped as an online survey in Qualtrics, the online survey software used by Western Sydney University. The revised instrument was presented to the expert panel for a second round of rating, and was accepted by all panel members. The scoring from the expert panel’s two rounds of evaluation is presented in Appendix 2.2.