LIMITATIONS OF STUDY ONE

Cohen and Manion (1980) report that the postal questionnaire survey is one o f the most commonly used descriptive research methods. Typically, surveys gather data at a particular point in time with the purpose of "a. describing the nature o f existing conditions, or, b. identifying standards against which existing conditions can be compared, or c. determining the relationships that exist between specific events" (p.71). In this study the IPS Questionnaire was designed so that data could be collected describing the views o f course tutors on aspects o f interviewing and problem solving (purpose "a" in Cohen and Manion's terms).

In critiquing study one, the following question unifies this section, "In drawing conclusions and making generalisations about the training courses surveyed, what weight can be placed upon the results?". In discussing the usefulness o f the data via the postal questionnaire a range o f issues will be highlighted which caste real doubt on the reliability, validity and ultimately the generalisation o f the main results obtained.

3.5.1. Questionnaire design and construction

"I would include a big section in your methodology on the problems o f using structured questionnaires to do research on complex social phenomenal" (Respondent 8)

Missing data or non responding affects all forms o f questionnaire research. A survey of this type, with relatively small numbers, is much more influenced by these effects. The

design of the questionnaire and the construction of its questions becomes an important factor in the efficacy of such small scale surveys.

The first phase in the design o f this questionnaire was a decision on its purpose and focus. In study one the general purpose was "to investigate course tutors' views on aspects of interviewing and problem solving". This was seen as being important in terms o f providing contextual information for subsequent studies detailed in this thesis. The next phase involved deciding upon the specific primary objectives o f the survey. This involved identifying and itemising a list o f subsidiary topics that related back to this general purpose, based upon past research, logic and researcher questions. Once topics were identified then specific questions were constructed, piloted (on a small sample of only four tutors who were not involved in the final survey), and then the final questionnaire was sent out. On hindsight and for future reference there were several important topic omissions, and several examples o f inadequate question construction which may have increased the non responding phenomenon.

3.5.1.1. Topic omissions and inadequate question construction

"I've tried my best but struggled with some o f the terms presented and/or their definitions" (Respondent 6, referring to the personality descriptors)

"Some o f these facets are just impossible to respond to" (Respondent 14, referring to the personality descriptors)

The questionnaire did not ask any specific questions about EPIT placements, as they related to supporting and developing interviewing and problem-solving skills. There was only a brief reference made to placements in questions 3 (k) and 14 (k). It would have been helpful in building up a more complete picture if information on this area had been collected. This is relevant given that a third of the training year is involved in placement activities. Carroll’s (1974) study reported that placement supervisors would be asked to rate trainee interviewing competence. The present study does not extend our knowledge any further, and this is an important omission.

Most tutors did not respond to the more open-ended type of questions (i.e. 4,5,8,15,16,19 and 23). Because of the very low response rates to these questions they were not coded. Perhaps the majority of tutors did not respond because they saw these questions as being too time consuming to answer, as repeating material covered in other questions where they could indicate their preferences (via tick boxes). This may have been more true for questions 4 and 15 which asked tutors to outline their interviewing and problem solving approaches. Though the final return rate was very high, it did take a lot o f prompting and persistence to achieve. This could well support the view that fatigue and time were important considerations for tutors in deciding to complete or not complete questions. If they did feel compelled to complete the questionnaire, then ticking boxes was more time effective than responding in detail to open-ended questions.

The questions, particularly 4 and 15, assumed that tutors were able to make explicit the detail o f the interviewing and problem-solving firameworks they used. This is an assumption, given some limited support by tutors' comments written in the margin o f questionnaires, "can't answer this question as problem solving underpins everything we do", and "this list has given me some ideas for improving our course". As outlined before the use o f a less structured format might have resulted in different data being collected. The use o f semi-structured face to face interviews would have been another approach. But this too has its own difficulties (labour intensive, prone to social desirability effects).

Questions 11 and 22 used a more structured format, yet they were also not answered by the majority o f tutors. These questions were included to provide a cross check with questions 10 and 21 respectively. Questions 10 and 21 presented the five personality and three communication style descriptors, and asked tutors to rate how important each factor was to either successful interviewing or problem solving. In contrast these questions were answered by the majority of tutors. Questions 11 and 22 presented, in random order, the individual descriptor words which were used to define the personality and communication style characteristics (e.g., the characteristic of conscientiousness was defined by competence, order, dutifulness, achievement-striving, self-disciplined, deliberation). The style and format o f these sections were modelled upon work by Costa and McCrae (1989, 1992). Their research was largely conducted

within the United States of America, and not with British populations. There are potentially significant cultural differences which could have affected response rates.

Few tutors responded to these questions (i.e. 10,11,21 & 22) and as a result they were not coded. Some indication of why they were not responded to, in addition to those noted earlier, is suggested from comments written in the margin of questionnaires, for example "these are not in keeping with our philosophy", "we don't think like this", "these are too psycho-dynamic", "don't know what you mean". On hindsight and for future reference more work was needed in clarifying the purpose of such questions and, if used, then piloting formats that would increase the likelihood o f appealing to a discerning and critical audience o f educational psychology tutors.

Finally, questions 1, 2 and 12, 13 asked course tutors whether they taught interviewing and problem solving respectively and, if so, how many hours were involved. Unfortunately, 58% o f tutors who responded "other" did not then provide the exact number o f hours (as requested to do so). This limited the usefulness of the results o f this question. With hindsight this question could have left tutors to fill in the number of hours, with the responses being coded by the researcher.

3.5.2. Possible biasing effects

"You seem to assume a behavioural framework throughout - is this justified or adequate?" (Respondent 8)

"We don't teach it like this fa r too psychodynamic!" (Respondent 4)

Leedy (1980) observes that the questionnaire survey is "particularly susceptible to distortions through the introduction of bias" (p.99). This bias is often subtle, and may present itself through the types of questions asked, and how they are worded. For example, question 4, "What interviewing approach (es) do you teach on your course? (if appropriate, can you outline the steps in the approach (es) and/or include supporting material)" - however clear the wording and intent o f this question may appear it makes at least two assumptions. Firstly, it assumes that interviewing is taught in a structured

sequenced way on a course. Secondly, it assumes that the tutor is able to clearly identify and articulate the component parts of a given framework(s).

Despite a statement o f confidentiality all tutors knew that their questionnaires were going to be read and analysed at another training course. It is possible that social desirability effects could have potentially affected the validity of some of the responses. Questions, such as the one highlighted above, could have prompted some tutors to respond in ways which they perceived to be what the researcher was looking for, or in ways which were seen as being more desirable or appropriate (e.g., some tutors may have responded so as not to appear "out of touch" with recent developments in the field of problem solving and interviewing, or "old fashioned" regarding the use of technology in supporting training). Collectively these various factors could have limited the validity, reliability and ultimately the weight given to the findings.