Devising the scale

As a starting point, each of the specific skills within the Essential Skills was analysed separately and within its group. This involved careful consideration of the skill itself, what might give a pupil opportunity to develop that skill, and what might prevent or hinder that development. Statements were devised which captured the presence or absence of opportunity to learn the skill. Note that the matter in question is not the extent to which students have acquired these skills, but rather if there is an opportunity in their schooling to learn them. This consideration drew on personal experience as a teacher to produce the original set of statements. Because of the focus of the study as a whole, particular attention was paid to areas in which a learner with vision impairment might be hindered in having opportunity to learn that skill, while still ensuring that it did not limit the applicability of the instrument to the general population.

Table 5. Examples of the Essential Skills and associated items and responses

Essential Skill group

Specific skill

Students should be able to:

Items for the questionnaire Response sets

Work and

Study Skills work effectively, both independently and in groups

I work on my own, without help from others. I work as part of a group. At school I learn ways to work better in a group.

Every day, Most days, Sometimes, Almost never, Never.

Often, Sometimes, Almost never, Never.

Very True, Somewhat True, Not Very True, Not At All True Communication

Skills

argue a case clearly, logically and convincingly

I am encouraged to express my opinion.

Often, Sometimes, Almost never and Never

Social and Co- operative Skills

develop a sense of responsibility for the well-being of others and for the

environment

At school we look after each other.

I help to keep the school tidy.

Very True, Somewhat True, Not Very True, Not At All True Often, Sometimes, Almost Never, Never.

Environment

for Learning (Additional to the Essential Skills) I have enough time to complete my homework. School is a good place to be.

Often, Sometimes, Almost Never, Never

Very True, Somewhat True, Not Very True, Not At All True

Table 5 presents some of the items, showing the skill group, specific skill and answer set associated. For example the skill “work effectively, both independently and in groups” is one of the Work and Study Skills. For students to develop this skill they need to have the opportunity to work on their own and in groups. If they never work on their own or conversely never get to work in groups, their ability to do so will be hindered. This led to the item “I work on my own, without help from others” (Every day, Most days, Sometimes, Almost never, Never) and the items, “I work as part of a group”, (Often,

Sometimes, Almost never and Never) and “At school I learn ways to work better in a group”, (Very True, Somewhat True, Not Very True, Not At All True). The full final instrument is included in Appendix 7, and the items are also listed in their index groupings in Appendix 9.

In addition to the eight categories related to the eight Essential Skills, a ninth category was developed, called Environment for learning. This drew on research by Gilmore & Absalom (1995) which discussed elements of the school environment that helped or hindered learning, related to the Essential Skills of the New Zealand Curriculum. This included items such as “I have enough time to complete my homework”, and “School is a good place to be”, and addressed issues of temperature and noise in the classroom. These measures affect opportunity-to-learn in a general way. They are very similar in style and content to items used in the Improving School Effectiveness Project (ISEP), such as “I always like school” and “Teachers are always fair”. (McCall et al., 2001)

Embedded in the instrument are items related to “Press for achievement” as advocated by Oakes (1989). These include items such as “I am encouraged to work well”, and “I feel pleased with the work I do at school”. These also reflect one of the three main performance indicators proposed by Gray & Wilcox (1995), the desired aspect of “pupils’ satisfaction with their educational experiences”. Similarly there are elements corresponding to another of Gray and Wilcox’s performance indicators, namely pupil-teacher relationships. This is covered by items such as, “My teachers have time to help me”, “Things are explained to me in a way I can understand”, and “My teachers help me to learn.”

An initial set of 137 items was created. The questions were pre-tested on a small group (n=3), and the wording altered where it was unclear. The items had a variety of response scales, including “Always/Usually/Sometimes/ Never” and “Very true/ Somewhat true/ Not very true/ Not at all true”. For the pre-test and pilot study, the categories, “I don’t know” and “Doesn’t apply to me” were included to aid in the development of the instrument.

In order to establish content validity, the set of items was grouped according to the Essential Skill areas, and sent to several groups of experienced teachers and educators, including specialist teachers for learners with vision impairment. They were asked to consider, for each of the skill areas, the extent to which the questions covered the Essential Skills, – did they establish whether the learner had the opportunity to develop those skills and were there any gaps, overlaps or questions they would like to add? Overall, did they add up to the essence or intention of that particular skill set? The educators were also asked to comment on how appropriate the items would be for different age groups. The responses were very helpful in identifying ambiguous questions, repetitions and possible additions.

Analysis of the validity feedback led to the development of a set of 101 items that would make up the scales in the pilot test. To these were added questions on demographic and administrative details including gender, age, year at school, home language and whether there was a computer at home that they could use. This was then piloted, near the end of the school year, on four classes (79 useable responses) at a medium-sized high school (800 pupils) in a medium to low socio-economic area. Students were asked to think about their experiences at that school during that school year. I administered the questionnaire and noted the time taken, students’ body-language, queries and comments, in order to improve the instrument further. The questionnaire also asked how the students found the questionnaire with regard to length, ease of understanding, ease of answering and whether or not it was interesting.

The pilot study was useful in uncovering potential sources of error. For example, students often gave the current year as their year of birth, and some students confused male and female. Consequently the later instrument asked for their age as well as their date of birth, and used the terms, boy and girl. As missing values or the choice of in-between responses would cause serious difficulties in analysis, I realised it was better to glance quickly over each response booklet as it was collected and get the student to correct mistakes and omissions at the time. Responses were anonymous, and summary results were provided to the school to acknowledge the contribution of the staff and pupils.

In order to get an indication of the consistency of responses, one class (n=23) was given parts of the questionnaire to answer again about two weeks later. Unfortunately this was done in the last week of the school year and their goodwill was exhausted. Several of the students did not complete the questionnaire, or seemed to just tick down one column. On average the students gave the same response that they had given previously 53% of the time, and were different by no more than one level of response 91% of the time. Their individual correlations ranged from 0.769 to 0. Some items provided very consistent responses. These included “School is a good place to be”, “The types of sporting activities at school suit me,” and “I have to look for my own information to complete my schoolwork.” Other items were much less consistent: “I am encouraged to try new things, even if I am afraid,” (eliminated) “The teachers and students respect each other,” (eliminated), and “I look at different solutions to the same problem”. (retained)

The instrument was refined in light of the pilot study, with some items removed and some altered, leading to the “final” set of eighty-seven items. Each item was examined, along with the range of responses given. The correlations between questions that should have been related were also examined. If questions relating to the same construct did not correlate, or had negative correlations, then this indicated that one or both of the questions may have been badly worded. Questions were

removed or edited if they were confusing, ambiguous, involved multiple concepts or were not well written.(Department of Statistics, 1992)

In the pilot study there were some items asking pupils to rate how often they took part in certain activities compared with the others in their class. These were confusing as they required the students to make judgments about other people’s activities. This format was changed to a straight frequency format. Other items were too vague and did not give meaningful results. These included “I get practice speaking in various settings” and “We write for different purposes”. One item, “We read the same type of books all the time” did not correlate with any other item. This was reworded as “We read a variety of material, such as newspapers, books, magazines, fiction and non-fiction.”

An example of a question involving multiple ideas was, “I feel pleased when I have worked hard to complete a piece of work.” This has complex ideas in it: “Do I complete work?”, “Do I feel pleased when I complete work?”, “Do I feel pleased when I have worked hard?” or “Is it only when I have worked hard and completed a piece of work that I feel pleased?” The responses to this were nearly all Often (40) or Sometimes (33) with only four responding Almost Never and no one selecting Never. Similarly “When I do badly at school I feel like giving up” was intended to test for persistence, but a student who seldom does badly at school may be unable to give an answer that indicates persistence. A “Never” response could mean they never do badly, or that they never feel like giving up.

Students asked for clarification about some of the items, and these items were examined closely and altered or removed if necessary. Examples of these are “We practise using numerical patterns”, and “We use material presented in a mathematical way,” both of which are hard to understand.

The refined set of items was arranged so that items from each skill set were spread throughout the questionnaire, and items with similar response sets were grouped together. This final set, named the Essential Skills Access (ESA) test, was then administered to a sample of over 1300 students, according to the sampling scheme described below. A full copy of the ESA test is in appendix 7.