Reliability Study

Determining the test-retest reliability of the questionnaire for use in an Air Traffic Service environment is the first aim of the present study. On the whole the questionnaire is deemed reliable for use with Air Traffic Service staff, although the measurement of several variables requires further investigation.

The large majority of variables have moderate to high reliability. The independent variables of sleep length on evening shift, satisfaction with the hours of work, and the extent to which shiftwork caused sleep problems and organisational problems have only moderate reliability. Despite this the reliability coefficients were considered sufficient for these variables to remain in the study. Variables with low reliability are excluded from the remainder of the study.

The scales taken from the SSI (Barton et al, 1995) have acceptable reliability, although the GHQ is slightly low. All coping variables, social and domestic variables, and most work variables from the questionnaire constructed by Rhodes et al (1994) are found to be reliable.

Concern is noted regarding the lack of reliability for the self reported sleep length and sleep continuity variables. Although, self reporting of sleep length has been widely used in studies of sleep and shiftwork (Lavie et al , 1989; Knauth & Kieswetter, 1987; Rosa, Colligan & Lewis, 1989, Folkard & Barton, 1992). However, it is widely accepted that other methods of collecting sleep data, such as sleep diaries/logs, EEG monitoring, and actigraph devices provide more detailed and more objective data. Sleep logs have been widely used in shiftwork studies (Suvanto, Partinen, Harma & Ilmarinen, 1994; Dekker, Paley, Popkin & Tepas, 1993) but they too have specific problems. This method requires daily recording of sleep times and subsequently places a burden on participants to maintain accurate and complete records. The number of incomplete records tends to be high, yet the information gained from this method can be much more detailed than that obtained from simple sleep length

periods (Kurumanti et al, 1994). Other more objective methods include EEG monitoring and the use of actigraphs. These methods are much more accurate but there is a financial cost as well as a much more extreme commitment required by participants. In summary, it is realised that subjective reports of sleep length have problems but so do the other objective measures. Overall, it is necessary to weigh up the advantages and disadvantages of each. It is suggested from the findings of this reliability study that more objective methods, other than self reporting, should be used in gathering sleep data.

Other variables with unacceptable reliability include self reports of the number of hours of overtime worked and the adequacy of breaks during shifts. The questions regarding overtime ask for an estimate of the number of hours typically worked in the last month under (a) ‘normal’ and (b) ‘peak’ or ‘under staffed’ periods. Self reported data such as this may not be very accurate due to difficulties in recalling this information. Individuals possibly only consider the amount of overtime worked in the previous few days and inaccurately extrapolate this value over the last month. In addition, there is possibly a tendency to over estimate the amount of overtime worked in order to portray the situation as being worse than it actually is. Another concern with these questions is the possible ambiguity between what is considered ‘normal’ conditions and what is considered ‘peak’ or ‘under staffed’. A number of participants commented on their questionnaires that ‘peak’ or ‘under staffed’ was the norm in their workplace, hence the number of hours recorded is the same for each question. There is also the consideration that the question on the amount of overtime worked in ‘peak’ or ‘under staffed’ periods immediately followed the question on overtime in ‘normal’ conditions. This could easily bias individuals to inflate the number of hours of overtime worked under ‘peak’ conditions because they believe this value should

probably be greater whether in reality it may not be.

Information such as the number of hours of overtime worked should be relatively easy to obtain from more objective sources, such as work records. Obviously the full cooperation of management would be required, but this source of information would overcome the inaccuracies in participant recall and bias. With additional information

on the number of staff working and air traffic levels the distinction could be made between ‘normal’ conditions and ‘peak’ or ‘under staffed’ conditions.

A further variable that is unreliable is the reference to adequacy of breaks on each shift worked. It is likely that in answering this question an individual considers how they felt about the adequacy of their breaks only in the previous few shifts. Such feelings may not remain constant over time, hence the lack of reliability for this variable. There may also be a range of other factors which constantly change how an individual feels about their breaks. Such factors could include the number of staff working, the workload on a shift, the particular shift cycle being worked and the actual tasks an individual is performing. Therefore, like sleep data, this data may be better collected by some other means. For example, a daily log in which an individual records how they felt about their breaks at the end of each shift over a number of shift cycles. Additional comments could also be made by the individual so other trends could be identified and a fuller picture of the situation developed.

The demographic characteristics of the New Zealand Air Services staff who participated in the reliability study were very similar to the sample of Australian ATS staff who participated in the main study. The only notable differences were that a much greater proportion of Australian ATS staff had tertiary qualifications and that New Zealand ATS staff were much more likely to have worked in another country. With these differences taken into account the questionnaire is expected to be reliable when used in another country.