Chapter 3 Method
3.3. Survey
Surveys have a long, rich history within educational research and have been applied to a wide variety of contexts with different purposes (Cohen et al., 2000, p. 169ff, Andres, 2012, p. 4ff). Andres in particular emphasises the ‘multifaceted and versatile’ nature of survey
research (Andres, 2012, p. 17), including the types of question which can be asked, alongside
the sampling and response methods, not omitting the survey’s place within the overall project design. Recognising the limitations of survey usage in terms of reliability, validity and bias, the approach taken in this project is more limited. Rather than looking for a highly reliable set of data, with good statistical significance, the goal was to use the survey to expose key ideas, that is indicators of change in social practice, especially meanings (see Figure 2.1), which could then be explored through other methods, such as interviews, in the main ethnographic phase of the study in the departmental context. It was also envisaged that some small indications may be exposed owing to differences between different steps of the implementation staircase (Figure 2.2) or owing to gender. This is also consonant with the model described in Figure 2.3, which has a focus on interventions and their evaluation. The design of the survey was informed by documentary analysis of successful departmental submissions from within FenU and by material from the ECU (ECU, 2015a). FenU was chosen as the site of the study partly for pragmatic reasons of proximity but also since it gave potential to exploit the benefits of insider research which Trowler highlights in terms of improved access to implicit meanings and to different types of data (Trowler, 2012b, Chapter 1, paragraph 5). Whilst there are inherent risks (see Section 3.8), this approach has attracted considerable support amongst ethnographers (see, for example, Taylor, 2011).
60
Focusing on FenU ensured that all common elements of Athena SWAN programmes and initiatives within the institution were included in this initial analysis, which would not have been as possible with another institution. That is, by engaging with multiple sources and acknowledging the limitations in the quality of information obtained, the potential for bias owing to selectivity was minimised. Using notes made from these documents, six main themes emerged which then formed the basis for generating a set of questions for piloting. The themes were also cross-referenced against the social practice model (Figure 2.1) to ensure a degree of theoretical integrity and on a dimension of internal to the department or external (a change tactic, scaffolder or inhibitor from the central university or beyond). This is illustrated in Table 3.1.
Theme Internal External Theory link
Communications – reporting, availability of documents, action plans, applications, survey results
Competency, meaningKnowledge – rationale, meaning of initiative
MeaningProgrammes – tactics for change, staff review and development, academic promotions, workload models
Competency, material, meaningData – collection, awareness of issues of student numbers and attainment, recruitment, retention
Competency, materialLeadership – evidence, structures, family-friendly practices
Competency, meaning Resources – surveys, runningactivities and courses
Material, meaningTable 3.1 Analysis of themes
61
Two issues need to be addressed here. Firstly, the division into these themes is arbitrary: indeed, a specific tactic to encourage improvement in practice could address more than one of these areas. For example, leadership and resources could be seen as overlapping: without committed leadership it is unlikely that resources will be made available. Without adequate resourcing, the effectiveness of leadership may be compromised. Secondly, by identifying themes with particular elements of the theoretical model, this is not indicating any form of exclusivity within the model to particular elements or interactions between elements. This exercise simply acted as a check that the survey was capable of identifying potential sites of change in elements of practice.
From this set of themes, questions were generated, which were appropriate to the institution. Given the limited goals of this survey stage, further clear parameters were established at the design stage. These were that data should be anonymous; the survey should be equally relevant to all STEMM departments within FenU; the questions should be capable of implementation through a computer-based system; the survey should not take more than 15 minutes to complete (to avoid fatigue); and the data should be capable of relatively straightforward analysis to inform the next stages of the project, given the limited resources of time and effort. The principles for question designed were drawn from existing good practice. For example, consideration of accessibility, fairness, audience, difficulty and background knowledge (Welch, 2006, p. 311) in the use of language and format. Andres provides advice on the suitability of different question types (Andres, 2012, p. 69ff) including the use of rating scales. Question design was also informed by previous work (Peet, 1993).
The questions were all loaded onto FenU’s Qualtrics® platform, using a variety of question
62
The target audience for this survey was all university employees in STEMM departments, so the questions had to be as accessible to staff in lower-graded support roles as to senior professors. At this point, it should be noted who the survey, and indeed the project,
excluded. Owing to the division of their time, responsibilities and loyalties, academic visitors and employees of other academic institutions who might be research collaborators were omitted owing to potential conflicts of interest as well as the degree of distance from the practices within the department. Further, owing to their particular circumstances and limited experience in time, students, both undergraduate and postgraduate, were also omitted. The mixed student experience between departments (owing to the structure of the FenU course) and with other agencies would mitigate against valid investigation, in
particular with issues relating to separating different experiences and meanings from different contexts. Athena SWAN does have clear links to students, but given the particular constraints within FenU, it was felt this precluded these groups from the current study, although they could easily form the subject of separate work. Equally, even if access had been possible, it is doubtful that data from another HEI would have been sufficiently comparable, without the same degree of insider involvement.
The survey was therefore piloted twice given the reputational risk to the whole project of not getting it right first time and the very wide audience. The potential risk leading to a limit on future work was seen as significant. (It should be noted at this stage that no department within FenU had agreed to participate in the second ethnographic phase of the project). The survey was first tested by staff in a specialist section of FenU with experience of evaluation and then with a group of fellow mature postgraduate students researching higher education to provide a research-orientated focus. Revisions included the division of questions to
improve accessibility, the addition of ‘not appropriate’ answer options in some questions
63
alongside the raw data. It should be recognised that the on-line survey format does not transfer easily to print.
The survey was launched through departmental Athena SWAN contacts and human resources business managers in the four STEMM units. One of the four units opted out of
the process owing to an overlap in timing with their own unit’s staff survey. It could be
argued that a second survey on similar or overlapping material might have led to a
displacement in responses to this study’s Athena SWAN survey anyway. The target was to
achieve 200 responses: the initial approaches yielded over 300 complete (i.e. finished) responses (a response rate of about 8%) and a further 200 partial answers, where the survey was not finished. Consonant with the questionnaire rubric, and in line with the ethical framework, unfinished responses were discarded. The indicators as exposed by the survey are discussed in the following sections, but were sufficient to inform the next stage of the project, an in-depth quasi-ethnographic study of the research questions in the context of a particular department. As well as standard reports of the responses, cross-tabulation of data, analysing responses by staff within particular units, was the most frequent analytical tool. Given the focus on an individual department, it could be argued that it would have been more appropriate to survey just that department. The potential to deliver more specific information through this tactic is mitigated by the much smaller numbers involved (in terms of potential participants) and associated issues of validity and also of ethics, since individuals might become identifiable through their responses. Therefore, the design decision was to prefer the results from a larger survey which would give indicators more relevant to the practice-as-entity, which can then be explored in the instantiated practice.