Concourse development

Using the literature review as a starting point, I collected succinct statements about evaluative reasoning, including statements from value theory, evaluative reasoning theory, and articulations of evaluator practices in relation to evaluative reasoning. I then reviewed the approximately 300 statements against Figure 3 (refer chapter 5)

6 Expressed in statistical terms, the factors derived from a Q study are orthogonal, that is, independent and at 90 degrees to each other (Stainton Rogers, 2005).

(developed from the review of literature) to ensure the key theoretical positions about evaluative reasoning were included. This review proved very useful as it highlighted omissions in both the statements and the conceptual framework. It also revealed duplicative ideas in the statements.

6.3.2

Item selection

A workshop was held with five evaluation colleagues (who are experienced practitioners and familiar with my doctoral study) to select statements for the Q set. This collaborative approach helped to ensure the Q set was representative and comprehensible. The importance of context in textual comprehension was demonstrated in this collaborative approach to statement selection. While I understood the statements from having read them in situ, some had become less understandable as a result of being extracted from their context. The selection group began by sorting the statements into identified themes as shown in Table 6.2.

Table 6.2 Themes for statement selection

Themes Sub themes

1. How evaluation is defined and its purposes

2. How values get privileged in evaluation Contextual factors Methods

Worldviews 3. Who does/is involved in valuing Evaluator only

Stakeholders only Both

4. Evaluative criteria: explicit/implicit/non existent

Explicit - prescriptive/external

Explicit – descriptive/emergent/consensual Implicit

No criteria 5. Evaluative judgments: how evaluative

judgments are arrived at, and by whom

Emphasis placed on evaluative judgments Who is involved in judgment-making Logic/warranted argument

Intuitive approaches 6. The stances/roles/behaviours adopted

by evaluators in relation to thinking about values and practising evaluative reasoning

The selection group then reviewed the statements by theme/subtheme. The aim was to select 35 or 43 statements. I chose to have 35 statements to enable participants to complete a Q sort and participate in a short discussion about the way they had sorted the statements within their sixty minute lunch break. Any ambiguous or duplicated items were rejected. The items that best addressed the theme/subtheme were then selected. Any aspects about the theme/subtheme (e.g. a specific theoretical position) that was missing from the statements were identified, and one or more other statements were subsequently included. Following discussion, themes one and six were excluded from the Q set as it was assumed they would be implicit in, or emerge through the factors (orientations). The selected statements (hereinafter referred to as items or the Q set) were subsequently reviewed by my two supervisors and New Zealand-based Q

methodology expert Dr Amanda Wolf, for their clarity and readability. I was advised to re-write some of the items, replacing academic terms with more accessible language to make it easier for the participants to self-reference (so as to avoid feeling they were undertaking a cognitive task) (A. Wolf, personal communication, 23 May 2012). The Q set was then trialled with an experienced researcher (an ex-academic and evaluation practitioner) who suggested some minor wording changes to some items and gave feedback on my administration of the Q sort.

6.3.3

Q sort preparation

I drew the distribution shown in Figure 5 onto large (122.5 x 91.5 centimetre) heavy-duty cardboard. A velcro tab was glued in the centre of each of the boxes in the matrix. Each of the statements was written onto a card, the cards were laminated and a velcro tab attached to the back of each card. This enabled participants to easily attach (and detach) the cards on the board as they wished. The shape of the distribution, was guided by the advice of Watts and Stenner (2012). They advise that a more spread-out distribution is appropriate for Q participants who are familiar with the topic of interest. Whereas a steeper shape with more boxes in the middle columns is preferable for participants who are less informed about the topic, providing sorters with more opportunity to express uncertainty.

–4 –3 –2 –1 0 +1 +2 +3 +4

Figure 5 Q distribution matrix

6.3.4

Participant selection

My aim was to recruit a differentiated group of 30 participants, including evaluators and evaluation commissioners who are government employees and consultants, and New Zealand European, Māori and Pasifika. (Note: In Q, the term sample refers to the Q items, not the participants). This number of participants was recommended as an appropriate size for a study of this nature (Wolf, 2012). I prepared an email about the Q study which was sent to 300 or so people who subscribe to the WEG email list. My rationale for this approach was that since subscribers are Wellington-based, they are likely to be undertaking public sector evaluation, either as a public servant or contractor. (The majority of evaluations undertaken or commissioned by government agencies are done so from head offices which are based in Wellington where the government is located). Although 21 people expressed an interest in participating, only 15 subsequently completed a Q sort (the others did not respond when asked to identify a suitable time for the Q sort). I then undertook some targeted recruitment (using professional networks) to ensure a differentiated sample of 30 participants, in particular Māori and Pasifika evaluators (of whom there are few). This involved a visit to evaluators based outside the Wellington region who undertake work for government agencies. (While the term evaluator is used in this chapter, it should be noted that one of the

participants in the Q study is a policy analyst who commissions evaluations and a second participant undertakes evaluations but does not refer to themselves as an evaluator).

Information about the study (Appendix A) and a consent form (Appendix B) were emailed to participants before the agreed time for the Q sort. The participants (P set) consisted of: public sector employees (18) working in 10 government agencies; consultants (11) (either working independently or as an employee in a private research organisation or university); and one person who works for a non-government organisation. The ethnicity of the P set is as follows: New Zealand European (17), Māori (5), Pasifika (3), other (2), not specified (3).

6.3.5

Q sort administration

The Q sorts were conducted in a room at the participant’s workplace, usually during their lunch hour. Participants were given another copy of the information sheet and then invited to sign a consent form. Participants were asked to provide a unique identifier to enable them to identify their results when the results were subsequently emailed to them, and so individual sorts could be referenced anonymously. Participants were then given written instructions about how to do the Q sort (Appendix C), which included the following condition of instruction:

“Please sort the statements to reflect your point of view as a professional evaluator, +4 being the two statements that are most similar to your views, and -4 being the two statements that are most different to your views.” The instructions also suggested

an approach to sort the items, as follows: “Sort the items initially into three piles (i)

items aligned with your point of view, (ii) items not aligned with your point of view, (iii) items you need to think about, do not understand, or about which you do not have a view. Then proceed to the detailed sorting of the items.” It was also suggested that

participants begin by placing items onto the board at the far left and far right ends, and to work inwards towards the centre.

The sort board was placed upright on a table. It was large enough for me to sit behind so I was not visible to the participant. The board appeared to provide a space for participants to think, and I was surprised to hear a few people talk to themselves as they sorted the items. On completion of the sort, I asked the participant an open ended question about their reasons for selecting the items placed at +4, +3, -3, -4 and recorded their responses on paper (Wolf, 2012).

I was also interested in participants’ choice of items in the zero column. For some participants, the items in this column were the taken-for-granted aspects of evaluation practice, whereas for other participants these were the items about which they did not have a view, or did not understand. I recorded participants’ comments about their sort on paper and subsequently referred to them to aid interpretation of the three factors that emerged (discussed in sections 6.3.6 – 6.4.5).

Q methodology was a new experience for all but one of the 30 participants. The majority of participants made unprompted comments about enjoying the task of sorting the items and arranging (and rearranging) them on the board. Some also expressed interest in learning more about Q and how they might use it in their work. This prompted me to write a two-page overview about Q methodology which I emailed to interested participants after the sort (Appendix D). I also

asked participants who expressed interest in my research whether they would be interested in being part of a discussion with other participants about the Q results. While most signalled their willingness to do so, only three responded to an email invitation (some months later) to a discussion (discussed below).

6.3.6

Factor results

The data from the 30 Q sorts were inputted into PQMethod software (version 2.33, December 2012). For each Q sort, the data comprises the number allocated to each statement representing where each card was positioned in the distribution matrix, and the unique identifier for the participant. The software undertakes the following statistical procedures (McKeown & Thomas, 2013): (i) correlation of each Q sort with each other Q sort, (ii) the intercorrelational matrix is factor analysed (to identify clusters of common meaning), (iii) the factor scores are rotated (using Varimax rotation), and (iv) factor arrays produced (identifying people who are statistically associated with a factor).7

The unrotated factor matrix showed that factors one, two and three have

eigenvalues of greater than 1 (factor one explains 47% of the variance, factor two

7 Factor analysis in Q is based on Centroid Method as this was the preferred statistical approach of Stephenson for theoretical reasons (McKeown & Thomas, 2013). Stainton Rogers (2005) describes Varimax rotation as the usual approach used by Q methodologists, but there is disagreement among some Q practitioners over this.

6% and factor three 4%).8 _{Table 6.3 shows the results of the rotated (Varimax)} factor matrix, with the three factors accounting for 51 percent of the variance. I then calculated the significant factor loading for my study (0.43) to identify the sorts that are statistically significantly associated with one or more of the three factors (referred to as a defining sort) as shown in Table 6.3.9 _{There are 28 defining} sorts. Twenty-two of the 28 sorts are associated with one factor, and six sorts are associated with two factors. Factors one and three are strongly correlated (0.8485), whereas the correlation between factors one and two is 0.5895, and between factors two and three is 0.5705.

Table 6.3 Summary of factor results

Percentage of study variance accounted for

No. of participants significantly statistically associated with one or more factors

Factor 1 19% 14, of which 3 are also associated

with factor 3

Factor 2 12% 6, of which 3 are also associated

with factor 3

Factor 3 20% 14, of which 3 are also associated

with factor 1, and 3 with factor 3

Total 51% 28 10