Question testing phase

The question testing phase involved an academic review in which an academic who was also a RC with a master’s degree in Research was invited to review the survey. The main objectives of the academic review were to review the survey in light of the research question and identify potential concerns in the survey. The academic reviewer provided recommendations to me during debriefing meetings (Belson cited in de Leeuw et al., 2008).

75 During these meetings the academic reviewer worked through the survey with me. These meetings were informal and free flowing. Based on the academic review I made certain adjustments to the survey where necessary (Personal Communication, Julie MacFarlane, 4th May 2016). Table 4-3 provides an example of the survey design process as described in this above section.

76 Table 4-3 An Example of the Survey Design Process

Original Survey (Young, 2015).

Initial Adaptation Development Stage – Supervisor’s so it was simplified to just ask: In which felt that there was no problem asking this

77 Original Survey

(Young, 2015).

Initial Adaptation Development Stage – Supervisor’s the overall aim of the research to also look did not feel that these items were laborious despite the fact that a period of recall was

78 Original Survey

(Young, 2015).

Initial Adaptation Development Stage – Supervisor’s

79 Original Survey

(Young, 2015).

Initial Adaptation Development Stage – Supervisor’s aware of this scope of practice?

The HPCSA

prescribes a scope of practice for RC in Form 258. Are you aware of this scope of practice? would like to add that you feel would enhance the visibility

Stellenbosch University https://scholar.sun.ac.za

80 Original Survey

(Young, 2015).

Initial Adaptation Development Stage – Supervisor’s

Review

Item testing phase - Academic Review

Dress Rehearsal - Peer review

Final Survey Design

incident that particularly stands out as having been significant to you in your personal

practice or work as a counselling

psychologist.

that particularly stands out as having been significant to you as a RC.

specifically trying to understand.

the aim of the item was to allow participants to

comment on anything they may have felt was left out by the survey. The item was therefore refined.

unclear/difficult wording that could make the item ambiguous.

of the RC in South Africa?

Stellenbosch University https://scholar.sun.ac.za

81 4.5.3.3 The dress rehearsal stage.

Before the dress rehearsal stage the survey was converted into an interactive computerised, self-administered survey using Survey Monkey (Survey Monkey, 1999). Survey Monkey is an online survey development solution that allows for surveys to be formatted in a way that is clear, uncluttered and ensures that a great level of attention is given to question wording, order, format, structure and visual layout of the survey (Survey Monkey, 1999). During the dress rehearsal two RCs were invited to participate in a peer review (de Leeuw et al., 2008) of the survey (Personal Communication, Kirsten Penderis, 8th July 2016; Personal Communication, Henk Mostert, 8th July 2016). The peer reviewers were sent the link to the survey. Using the condensed expert survey appraisal system proposed by Snkjkers (cited in de Leeuw et al., 2008). Respondents were asked to fill in the survey while at the same time giving attention to the condensed expert survey appraisal system (Snkjkers cited in de Leeuw et al., 2008). Table 4-4 provides an overview of this.

82 Table 4-4 Condensed Expert Survey Appraisal System

Condensed expert survey appraisal system (Snkjkers cited in de Leeuw et al., 2008).

Survey comprehension: Attention should be given to the following:

• Whether items made sense;

• If items were clear;

• Unclear/difficult wording.

Information processing: Attention should be given to the following:

• Whether items were laborious to answer;

• Whether a long period of recall was required; and

• Whether the participant needed to access information to answer a question.

Reporting: Attention should be given to the following:

• Overlapping categories;

• Questions being too repetitive.

Overview of the survey: Attention should be given to the following:

• Whether they felt the survey fulfilled its objectives;

• Structure and ease of moment through the survey;

and

• The length of the survey.

During the debriefing interviews the RC spoke about their experience of filling in the survey with specific reference to question comprehension, information processing, reporting and a survey appraisal as described above (Snkjkers, cited in de Leeuw et al., 2008). Table 4-5 provides a summary of the survey appraisal.

83 Table 4-5 Summary of the Survey Appraisal

Summary of the survey appraisal (Snkjkers cited in de Leeuw et al., 2008).

Did items make sense? The peer reviewers felt generally items made sense, were clear and did not contain unclear/difficult wording that could make the item ambiguous.

Were items clear? Where the peer reviewers felt that the item could be simplified this adjustment was made for example, the original item: The HPCSA promulgated a new scope of practice for RCs in September 2011.

Item Are you aware of this revised scope of practice? was adjusted to: The HPCSA prescribes a scope of practice for RC in Form 258.

Are you aware of this scope of practice?

Was there unclear/difficult wording?

Most items were clear and did not contain unclear/difficult wording.

However, if an item was considered problematic in that people’s interpretation of a term could differ quite substantially. The item was discussed with the peer reviewers. For example, the item:

Please indicate you level of satisfaction with the HPCSA was considered potentially problematic in that people’s interpretation of

"satisfaction" could differ quite substantially. In discussion with the peer reviewers, however, the peer reviewers agreed that the item was important and that they all interpreted the word satisfaction in a similar way. It was understood that they needed to comment on whether they were confident in the role of the HPCSA and whether they were fulfilled by what the HPCSA provided for them as RCs.

Were items laborious to answer?

Peer reviewers commended the grouping of certain items to obviate against respondent fatigue in answering items that were considered more laborious than other items. For example: The peer reviewers felt that the items regarding training of RCs were tedious in that there was a period of recall was required in answering the question because participants needed to think back to their studies. The peer reviewers highlighted however that having these items grouped closely together obviated against respondent fatigue in answering the items. The peer reviewers, however, felt that there was no problem asking this item and that generally people would find it easy to remember when they were first registered.

Was there a long period of recall required?

See above Did participants need to

access information to answer a question?

Concern was expressed by the academic reviewer regarding the cognitive effort that may be required in remembering when the participant was registered and that participants may have to stop the survey in order to check when they were registered and they may then not complete the survey. This did not appear to be the case.

84 Summary of the survey appraisal (Snkjkers cited in de Leeuw et al., 2008).

Overlapping categories?

Generally, not. Although the item regarding overall level of

satisfaction with postgraduate training to become a RC was cited as being too repetitive and overlapping with the item: Do you feel satisfied that your studies prepared you for the work you do as a RC? The item was removed from the final survey.

Were questions too repetitive?

Did the survey fulfil its objectives?

Yes Structure and ease of

movement through the survey.

A recommendation was made by the peer reviewers that the survey be split into the 5 sections of the survey to make it easier to use.

Recommendation was also made regarding the use of drop down menu options and Likert scale options where applicable, for example: In the item what is you highest qualification? The peer reviewers recommended that rather than leaving this open ended a drop down menu should be provided. The peer reviewers also provided insight into items where an open-ended response could be added. For example, in the item: If you were to start over again knowing what you know now about being a RC, would you choose a different career? The reviewers felt that the provision of a "please explain" section would help to frame the findings if necessary.

Length of the survey The survey took on average about 15 minutes to complete which the peer reviewers felt was appropriate. The reviewers felt that the survey was enjoyable to complete and the research would play a principal role in ultimately enhancing the visibility of the RC category.

Adjustments were made where necessary and the survey was finalised (Personal Communication, Kirsten Penderis, 8th July 2016; Personal Communication, Henk Mostert, 8th July 2016).