• No results found

BMJ Open. Consistency of the Health of the Nation Outcome Scale (HoNOS) at inpatient-to-community transition

N/A
N/A
Protected

Academic year: 2021

Share "BMJ Open. Consistency of the Health of the Nation Outcome Scale (HoNOS) at inpatient-to-community transition"

Copied!
47
0
0

Loading.... (view fulltext now)

Full text

(1)

For peer review only

Consistency of the Health of the Nation Outcome Scale (HoNOS) at inpatient-to-community transition

Journal: BMJ Open

Manuscript ID bmjopen-2015-010732 Article Type: Research

Date Submitted by the Author: 02-Dec-2015

Complete List of Authors: Luo, Wei; Deakin University, School of Information Technology

Harvey, Richard; University of Melbourne, Clinical and Biomedical Sciences Tran, Truyen; Deakin University, School of IT

Phung, Dinh; Deakin University, Australia, Centre for Pattern Recognition and Data Analytics

Venkatesh, Svetha; Deakin University, Centre for Pattern Recognition and Data Analytics

Connor, Jason; The University of Queensland <b>Primary Subject

Heading</b>: Mental health Secondary Subject Heading: Health informatics

Keywords: HoNOS, MENTAL HEALTH, Assessment

(2)

For peer review only

Consistency of the Health of the Nation Outcome Scale

(HoNOS) at inpatient-to-community transition

Wei Luo, Richard Harvey, Truyen Tran, Dinh Phung, Svetha Venkatesh, and Jason P. Connor

Abstract

Objectives: The Health of the Nation Outcome Scale (HoNOS) is a mandated outcome measure in several mental-health jurisdictions. Despite good validity reported previously, few reliability studies have been conducted and available data are inconsistent. It is unclear whether consistent interpretation of HoNOS exist across different care settings. This article assesses the consistency of HoNOS in a sample of psychiatric patients transitioned from inpatient and community.

Setting: A regional mental health service with both acute and community facilities.

Participants: 101 psychiatric patients were transferred from inpatient care to community care from years 2012 to 2014. TheirHoNOS scores of were extracted from a reporting database. Each inpatient-discharge assessment was followed by a community-intake assessment, with the median period between assessments being 4 days (range 0-14). Assessor experience and professional background were recorded.

Primary and secondary outcome measures: The difference of HoNOS at inpatient-discharge and communit-intake. Pearson correlation and Cohen’s Kappa between two total scores.

Results: Inpatient-discharge HoNOS was on average lower than community-intake HoNOS. The average HoNOS was 7.38 at discharge (median 7, range 1-22), and 11.90 at intake (median 12, range 1-25), an average increase of 4.52 (sd 6.83). Pearson correlation between two total scores was 0.073 (95% CI: 0.0950.238) and Cohen’s Kappa was 0.02 (95% CI: -0.02-0.06). No differences were observed in assessor experience or professional

background.

Conclusions: Limited support was found for consistent interpretation of the HoNOS across care settings in this patient sample.

Page 1 of 14 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(3)

For peer review only

Strengths and limitations of this study

• This study is based on the unique data from a single database covering both acute and community mental health services.

• The study site was at the forefront of implementing the HoNOS instrument; the data quality is representative.

• This is a retrospective study from a single region.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(4)

For peer review only

Introduction

Reliable measures of patient mental response to treatment is central to care planning and quality assurance. Symptom rating scales such as the Beck Depression Inventory (BDI, (1)), State-Trait Anxiety Inventory (STAI, (2)) and Positive and Negative Syndrome Scale (PANSS, (3)) are well recognised individual assessment tools that reliably measure patient mental health symptoms. Many symptom rating scales do not comprehensively assess overall patient health and social functioning. These domains are typically seen by health and government services as more critical in evaluating patient outcomes (4). To address the need for a more standardised and systematic approach at measuring health outcomes for mental health patient functioning, the Health of the Nation Outcome Scale (HoNOS) was developed (5, 6). The HoNOS has 12 items covering behaviour, impairment, symptoms and social functioning domains (6). The HoNOS is now routinely applied as a mandatory mental health outcome measure in the United Kingdom and in many Australian and New Zealand jurisdictions, and is gaining popularity in a number of other countries (7-12).

Early HoNOS psychometric validation studies were unconvincing. A six week pre-post administration analysis of the HoNOS by Bebbington, et al., (13) concluded that there were “serious problems in using the instrument as a routine measure of clinical status in busy psychiatric services.” (p. 389). They found that performance of the instrument was likely to be related to the assessors’ level of training and skill, which reflected clinical reality in most busy mental health services. Similar evaluations around this time also failed to provide compelling support for the widespread adoption of the HoNOS, primarily on psychometric grounds (e.g., (14, 15)). One of the key concerns was poor reliability (13, 15). Despite no modification to the scale itself, subsequent reliability testing has provided additional support for its utility in clinical practice, based on larger and more heterogeneous samples, and arguably more sophisticated statistical approaches to validation. The reluctance of some clinical staff to fully adopt routine HoNS assessments has been reported (16) and there remains some concerns about the utility of the scale to inform clinical, as opposed to population level change (17).

Page 3 of 14 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(5)

For peer review only

In a comprehensive qualitative systematic review of HoNOS validation studies, Pirkis, et al. (7) concluded that the scale had “good” validity (that is, it accurately measures what it was designed to measure) and had “adequate” test-retest reliability (that is, consistently measures functioning at different time points) and inter-rater reliability (high correlations between different assessors scores for the same patient). Perkis, et al. (7) did note that few HoNOS studies have formally conducted test-retest reliability (n=3), and one-third of the 12 items consistently performed poorly on this psychometric index. For example, in a sample of 100 inpatients, Orrel, et al.’s psychometric evaluation (18) found the one week test-rest reliability of HoNOS items ranged from 0.33- 0.80 (mean 0.57), with only 2 items exceeding the accepted 0.7 cut-off (19). Perkis, et al.’s review (7) also found inter-rater agreement has been identified as poor on at least 50% of the items. Orrel, et al.’s reported inter-rater item reliability ranged from -0.03 to 0.65 (mean 0.395), with only one of the items exceeding the recognised cut-off for “good” (0.60-0.74) reliability (20). In a sample of 100 older psychiatric inpatients (average age =77 years), all but two items exceed minimum test-re-test or inter-rater reliability (21). Brooks (9) also found “limited” inter-inter-rater reliability (0.50-0.65) and concluded that “the HoNOS not be implemented as a major outcome tool, until the reliability and validity of the HoNOS is clearly established.” (pp. 509-510).

Given widespread, mandated application of the HoNOS not only as a tool to evaluate patient functioning, but also as an index for comparative service performance and cost effectiveness analyses (22), scale reliability is fundamental. Although more and more patients are serviced at different care settings, both inpatient and community-based, it is a timely question to ask whether HoNOS can be interpreted consistently at different stages of care. On the basis of limited and contradictory research evidence to support consistent interpretation of the HoNOS across care settings, we examined systematic changes of HoNOS in a sample of public health service psychiatric patients transitioning from inpatient to community care.

Methods

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(6)

For peer review only

Subjects

One hundred and one adult mental health patients receiving treatment from a regional mental health service between July 2012 and June 2014 were included in the study. There were 70 males (mean age 38, SD 12.2) and 41 females (mean age 41.5, SD 13.3). The majority of the sample were being treated for Schizophrenia, schizotypal or delusional disorders (n=63).

Procedure

HoNOS were administered to inpatients on discharge by clinically trained inpatient staff, and within a maximum of 14 days by clinically trained community-based mental health staff. The median time between inpatient and community HoNOS assessments was 4 days, with 95% administered within 12 days.

Measures

The HoNOS (5, 6) consists of 12 items, each ranging from 0 (no problem) to 4 (severe problem). The 12 items are scored into four subscales: Behaviour (items 1-3), Impairment (items 4-5), Symptoms (items 6-9), and Social (items 10-12). The sum of the 12 items provides a total score (range of 0 to 48). HoNOS can be interpreted at the item, subscale or total score level. The rating period is generally the preceding two weeks for inpatients at admission, for hospital outpatients, and for all clients of community–based services.(23, 24) The exception is at discharge from acute inpatient care, in which case the rating period should generally be the preceding three days and not collected if length of stay is less than 72 hrs. A community intake rating is collected at the allocation of case manager or first care plan. (25) Therefore the second assessment period is expected to overlap with the initial assessment period.

Statistical Analysis

Both inpatient and community HoNOS scores for each patients were paired and the average difference computed. Paired exact Wilcoxon signed-rank test was performed at the three scoring levels of the HoNOS (item, substance, total score). R package exactRankTests was used for computing this test statistics. (26) both Pearson correlation and Cohen’s Kappa were calculated. Chi-square analyses examined if professional background and level of experience contributed to differences in HoNOS scores between inpatient and community assessments. Page 5 of 14 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(7)

For peer review only

Hospital and university human ethics approval was obtained (approval number 12/83).

Results

Total HoNOS score

The average inpatient discharge HoNOS score was 8.05 (median 7, range 1-22) and the average community intake HoNOS was 12.16 (median 12, range 1-25). Pearson correlation between the two scores was 0.073 (95% CI: -0.095-0.238), and Cohen’s Kappa was 0.02 (95% CI: -0.02-0.06), showing poor consistency among two scores. (The numbers are slightly higher when two assessments were done within 4 days, where Pearson correlation was 0.14 (95% CI:-0.11-0.37) and Cohen’s Kappa was 0.07 (95% CI: -0.01-0.14)). On average, the total HoNOS score increases by 4.11 (SD 6.97) from an inpatient discharge to a community intake. Exact Wilcoxon signed-rank test for matched data returned p-value < 0.001, indicating that the HoNOS increase from inpatient discharge and community intake is statistically

significant.

Schizophrenia, schizotypal and delusional disorders (F2x) were the most common diagnoses, followed by mood disorders (F3x). Given the low frequency of other diagnoses (e.g.,

adjustment disorders, acute stress reaction, anxiety disorder, substance intoxication, and dementia), an “other” category used. When diagnostic groups were considered, F2x and “Other” diagnoses demonstrated significant differences between inpatient and community assessments. There were no significant differences reported in patients with mood

disorders (Table 1).

Table 1 Change of the total HoNOS scores by diagnosis.

Average (range)

Pearson correlation

Cohen’s kappa Diagnosis Discharge Intake Change p-value (95% CI) (95% CI)

F2x: Schizophrenia, schizotypal and delusional disorders (n=63) 7.58 (1-20) 12.0 (3-25) 4.42 <0.001 0.07 (-0.18, 0.32) -0.01 (-0.06, 0.03) F3x: Mood 7.75 (1- 10.04 2.29 0.189 -0.15 0.06 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(8)

For peer review only

[affective] disorders (n=12) 18) (1-19) (-0.69, 0.49) (-0.10,0.22) Other diagnoses (n=44) 8.81 (1-22) 12.97 (2-24) 4.16 <0.001 0.11 (-0.14, 0.34) 0.04 (-0.02, 0.11)

HoNOS subscales and Individual Items

The increase was also evidenced in HoNOS subscales (2) and individual score items (3). The only score item that did not reach statistical significance was item 12: Problems with occupation and activities.

Table 2 Changes of the four HoNOS subscales.

HoNOS subscales Average (range) p-value Pearson correlation Cohen’s kappa Discharge Intake Change (95% CI) (95% CI) Behaviour 0.84 (0-5) 2.0 (0-7) 1.16 <0.001 0.24 (0.07, 0.39) 0.15 (0.07, 024) Impairment 1.33 (0-5) 2.10 (0-6) 0.77 <0.001 0.43 (0.28, 0.56) 0.08 (-0.01, 0.17) Symptoms 2.52 (0-7) 3.86 (0-10) 1.34 <0.001 0.15 (-0.02, 0.31) 0.07 (0, 0.13) Social 3.35 (0-12) 4.20 (0-13) 0.84 0.006 0.18 (0, 0.33) 0.02 (-0.04, 0.08) Page 7 of 14 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(9)

For peer review only

Table 3 Changes of the 12 HoNOS score items.

HoNOS score items Average (range) p-value Pearson correlation

Cohen’s kappa Discharge Intake Change (95% CI) (95% CI) 1. Overactive, aggressive, disruptive behaviour 0.31 (0-2) 0.73 (0-4) 0.42 <0.001 0.34 (0.19, 0.49) 0.15 (0.03, 0.27) 2. Non-accidental self-injury 0.13 (0-2) 0.44 (0-4) 0.31 <0.001 0.24 (0.08, 0.39) 0.13 (0, 0.27) 3. Problem-drinking or drug-taking 0.41 (0-3) 0.83 (0-4) 0.43 <0.001 0.31 (0.14, 0.45) 0.19 (0.08, 0.31) 4. Cognitive problems 0.71 (0-3) 1.03 (0-4) 0.32 <0.001 0.24 (0.08, 0.40) 0.05 (-0.06, 0.17) 5. Physical illness or disability problems 0.61 (0-3) 1.07 (0-4) 0.45 <0.001 0.59 (0.46, 0.69) 0.26 (0.15, 0.37) 6. Problems associated with hallucinations and delusions 0.88 (0-3) 1.21 (0-4) 0.33 <0.001 0.31 (0.15, 0.45) 0.24 (0.13, 0.36) 7. Problems with depressed mood 0.76 (0-3) 1.03 (0-4) 0.28 <0.001 0.29 (0.13, 0.44) 0.09 (-0.04, 0.21) 8. Other mental and behavioural problems 0.86 (0-3) 1.61 (0-4) 0.68 <0.001 0.26 (0.07, 0.43) 0.11 (0, 0.22) 9. Problems with relationships 1.11 (0-4) 1.70 (0-4) 0.59 <0.001 0 (-0.18, 0.16) 0.02 (-0.07, 0.12) 10.Problems with activities of daily living 0.88 (0-3) 1.34 (0-4) 0.46 <0.001 0.34 (0.19, 0.48) 0.08 (-0.03, 0.19) 11.Problems with living conditions 0.55 (0-3) 0.63 (0-4) 0.08 0.27 0.19 (0, 0.36) 0.02 (-0.12, 0.15) 12.Problems with occupation and activities 0.55 (0-3) 0.52 (0-3) -0.05 0.70 -0.04 (-0.23, 0.14) -0.05 (-0.2, 0.1)

Clinician Profession and Experience

To examine if clinical professional background and level of experience contributed to differences in HoNOS scores, the proportion of staff profession/experience between

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(10)

For peer review only

Inpatient clinicians and community clinicians was examined. Of the patients in the sample, 98 (82%) were assessed by experienced inpatient nurses and 94 (79%) were assessed by experienced community nurses, indicating equivalence across services (2x2 chi square- code experienced nurses vs. others by community or hospital 2 =0.2425, p = .622). The remaining staff included psychologists (n=15), less experienced nurses (n=2) and students (n=11). Two ANOVAs were conducted which examined potential differences in HoNOS scores within each service, by profession and experience. For the inpatient clinicians, there was no difference on HoNOS scores between experienced nurses (mean = 8.23, SD 3.0) and other professions and less experienced clinicians (mean = 8.29, SD 2.2) (F [2, 109] = 0.002, p = 0.96). Similarly, there was no difference in HoNOS scores between Community experienced nurses (mean = 12.03, SD 5.4) and other professions and less experienced clinicians (mean = 13.47, SD 6.3) (F [1, 107] = 0.888, p = 0.35).

Discussion

The HoNOS has widespread, mandated use in many public health mental health systems including the UK and Australia. It is designed to measure health outcomes for mental health patient functioning at both a clinical and population level. It has been used as key index for comparative service performance and cost effectiveness analyses (e.g., (22)), and therefore can be applied as a lever for resource allocation. Given the scope of HoNOS roll out,

reliability of measurement is critical. Perhaps surprisingly, the reliability of the HoNS has received little empirical attention. Within a publically funded mental health system, this study identified very poor total score consistency. The median delay between an inpatient-discharge HoNOS and the following community-intake HoNOS was 4 days, with 95% occurring within 12 days (range 0—14 days). All individual items on the HoNOS on this sample performed poorly. No systematic differences between assessor’s professional background or level of clinical experience were identified.

The mean difference between scores following inpatient discharge to community

assessment was 4 points, or approximately a one-third decrease in clinician assessed patient functioning. Patients with mood disorders reported less variation between assessments than those with psychotic-type diagnoses, possibility reflecting a more stable prognosis. Differences in HoNOS scores may also reflect the severity of the population under

Page 9 of 14 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(11)

For peer review only

investigation. Orrell, et al. (1999) did show average HoNOS inpatient scores to be higher than patients assessed in outpatient or community setting. Inpatient status is typically indicated for patients that cannot be managed as an outpatient or community patients, due to the severity or complexity of the condition. It is important to determine if these observed differences reflect true variation across inpatient and community functioning, or

vulnerability in reliability of measurement. Counter-intuitively, in the current study, patients assessed in the community were rated as functioning significantly more poorly than when assessed as an inpatient (within a median four day interval).

The recommended timeframe for the HoNOS assessment may provide some additional clues to poor reliability. At discharge, the period of assessment is recommended to cover the past 72 hours (24). In the community, the recommendation assessment period is within the past 2 weeks (24). The most likely clinical reason for discharge is improved functioning and/or increased symptom stability. Patients who have improved functioning over the past 3 days are among those likely to be considered for inpatient discharge. When assessed in the community, a two week period is reviewed which may encapsulate broader functioning difficulties. Functioning may have also deteriorated due to less intensive monitoring and patient management. However given the two review periods overlap and higher community ratings are consistent across all domains, differences point towards measurement problems.

The findings should be interpreted in light of some limitations. Although previous reliability studies have typically occurred in similar or smaller sample sizes (eg. (9, 18, 21)), a larger and more heterogeneous psychiatric sample across a broader age range would provide a higher powered study to more robustly detect associations between testing periods and raters. We cannot confirm all assessors received the same level of training in the HoNOS. Given training differences have been previously detected (13), there may have been systematic biases associated with either inpatient or community assessors. The interval period for test-rest reliability ranged from 0-14 days, with a median of 4 days. The

timeframe is consistent with previous HoNOS reliability studies that have re-tested up to 6 weeks (eg. (13)). The modest variation in re-testing timeframes in the current study is not psychometrically desirable, however the clear pattern of results demonstrating an upward assessment of severity across domains provides some confidence that timeframe variations

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(12)

For peer review only

did not confound findings. Finally, the score consistency was also subject to potential time biases. A central purpose of this study was to assess the ecological reliability of the HoNOS in a busy clinical mental health service. A more robust design would video record patient assessments and responses (at one point in time) and has multiple clinicians blind to the original assessment re-score the scale.

In summary, limited support was found to support consistent interpretation of the HoNOS at different settings in this psychiatric patient sample. Recognising the scale has already been widely mandated in clinical practice, further larger scale testing is considered critical to ensure these findings are not simply a result of sample or assessor bias.

Contributorship statement

WL and JPC designed the experiements. WL performed data extraction. WL, TT, DP, and SV carried out statistical analyses. WL and JPC analysed the results. All authors participated in the preparation of the manuscript.

Competing interests

The authors declare that there are no conflicts of interest.

Funding

This research is funded by an external funding source.

Data sharing statement

Dataset may be available through request to the original data guarantor.

Page 11 of 14 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(13)

For peer review only

References

1.

Beck AT, Ward CH, Mendelson M, Mock J, Erbaugh J. An inventory for

measuring depression. Archives of general psychiatry. 1961; 4: 561-71.

2.

Spielberger CD, Gorsuch RL, Lushene RE. Manual for the state-trait

anxiety inventory. 1970.

3.

Kay SR, Fiszbein A, Opler LA. The positive and negative syndrome

scale (PANSS) for schizophrenia. Schizophrenia bulletin. 1987; 13(2):

261-76.

4.

Trauer T. Outcome measurement in chronic mental illness.

International review of psychiatry (Abingdon, England). 2010; 22(2):

99-113.

5.

Wing JK, Beevor AS, Curtis RH, Park SB, Hadden S, Burns A. Health of

the Nation Outcome Scales (HoNOS). Research and development. The

British journal of psychiatry : the journal of mental science. 1998; 172:

11-8.

6.

Wing JK, Curtis R, Beevor A. HoNOS: Health of the Nation Outcome

Scales: Report on Research and Development July 1993-December 1995.

Royal College of Psychiatrists London, 1996.

7.

Pirkis JE, Burgess PM, Kirk PK, Dodson S, Coombs TJ, Williamson MK.

A review of the psychometric properties of the Health of the Nation

Outcome Scales (HoNOS) family of measures. Health and quality of life

outcomes. 2005; 3: 76.

8.

Glover G, Knight S, Melzer D, Pearce L. The development of a new

minimum data set for specialist mental health care. Health Trends. 1997;

29(2): 48-51.

9.

Brooks R. The reliability and validity of the Health of the Nation

Outcome Scales: validation in relation to patient derived measures. The

Australian and New Zealand journal of psychiatry. 2000; 34(3): 504-11;

discussion 20-1.

10.

Parabiaghi A, Rapisarda F, D'Avanzo B, Erlicher A, Lora A, Barbato A.

Measuring clinical change in routine mental health care: differences

between first time and longer term service users. The Australian and New

Zealand journal of psychiatry. 2011; 45(7): 558-68.

11.

Kortrijk HE, Mulder CL, Drukker M, Wiersma D, Duivenvoorden HJ.

Duration of assertive community treatment and the interpretation of

routine outcome data. The Australian and New Zealand journal of

psychiatry. 2012; 46(3): 240-8.

12.

Theodoridou A, Jaeger M, Ketteler D, Kawohl W, Lauber C, Hoff P, et

al. The concurrent validity and sensitivity of change of the German version

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(14)

For peer review only

of the health of the nation outcome scales in a psychiatric inpatient setting.

Psychopathology. 2011; 44(6): 391-7.

13.

Bebbington P, Brugha T, Hill T, Marsden L, Window S. Validation of

the Health of the Nation Outcome Scales. The British journal of psychiatry :

the journal of mental science. 1999; 174: 389-94.

14.

Sharma VK, Wilkinson G, Fear S. Health of the Nation Outcome Scales:

a case study in general psychiatry. The British journal of psychiatry : the

journal of mental science. 1999; 174: 395-8.

15.

Trauer T, Callaly T, Hantz P, Little J, Shields R, Smith J. Health of the

Nation Outcome Scales. Results of the Victorian field trial. The British

journal of psychiatry : the journal of mental science. 1999; 174: 380-8.

16.

Pirkis J, Burgess P, Coombs T, Clarke A, Jones-Ellis D, Dickson R.

Routine measurement of outcomes in Australia's public sector mental

health services. Australia and New Zealand health policy. 2005; 2(1): 8.

17.

Duke B. HoNOS in the consultation liaison psychiatry setting: is it

valid? Australasian psychiatry : bulletin of Royal Australian and New

Zealand College of Psychiatrists. 2010; 18(6): 547-50.

18.

Orrell M, Yard P, Handysides J, Schapira R. Validity and reliability of

the Health of the Nation Outcome Scales in psychiatric patients in the

community. The British journal of psychiatry : the journal of mental

science. 1999; 174: 409-12.

19.

Nunnally J, Bernstein I. Psychometric Theory McGraw-Hill, 1994.

20.

Fleiss JL, Levin B, Paik MC. The measurement of interrater

agreement. Statistical methods for rates and proportions. 1981; 2: 212-36.

21.

S. Shergill KKS, Knightley Seneviratna, Martin W. Orrell, Sukhwinder.

The validity and reliability of the Health of the Nation Outcome Scales

(HoNOS) in the elderly. Journal of Mental Health. 1999; 8(5): 511-21.

22.

Slade M, Byford S, Barrett B, Lloyd-Evans B, Gilburt H, Osborn DP, et

al. Alternatives to standard acute in-patient care in England: short-term

clinical outcomes and cost-effectiveness. The British journal of psychiatry

Supplement. 2010; 53: s14-9.

23.

Strategic Planning Group for Private Psychiatric Services. A national

model for the collection and analysis of a minimum data set with outcome

measures for private, hospital-based, psychiatric services. Canberra:

Commonwealth of Australia; 2000.

24.

Department of Human Services. Outcome measurement in clinical

mental health services. (ed Victorian Government Department of Human

Services). Mental Health and Drugs Division,, 2007.

25.

Barwon Health, Mental Health, Drugs and Alcohol Services. ACE

Guidelines: Assessment, Care and Evaluation Manual. Barwon Health, 2010.

26.

Hothorn T, Hornik K. exactRankTests: Exact Distributions for Rank

and Permutation Tests. 2013.

Page 13 of 14 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(15)

For peer review only

Author information:

Wei Luo, PhD, School of Information Technology, Deakin University, Australia

Richard Harvey, MD, FRANZCP, Mental Health Services, Barwon Health, Australia; School of Medicine, Deakin University, Australia; Email: [email protected]

Truyen Tran, PhD, Centre for Pattern Recognition and Data Analytics, Deakin University, Australia Dinh Phung, PhD, Centre for Pattern Recognition and Data Analytics, Deakin University, Australia Svetha Venkatesh, PhD, Centre for Pattern Recognition and Data Analytics, Deakin University, Australia

Jason P. Connor, PhD, FAPS, Centre for Youth Substance Abuse Research, The University of Queensland, Brisbane, Australia

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(16)

For peer review only

Consistency of the Health of the Nation Outcome Scales (HoNOS) at inpatient-to-community transition

Journal: BMJ Open

Manuscript ID bmjopen-2015-010732.R1 Article Type: Research

Date Submitted by the Author: 07-Mar-2016

Complete List of Authors: Luo, Wei; Deakin University, School of Information Technology

Harvey, Richard; University of Melbourne, Clinical and Biomedical Sciences Tran, Truyen; Deakin University, School of IT

Phung, Dinh; Deakin University, Australia, Centre for Pattern Recognition and Data Analytics

Venkatesh, Svetha; Deakin University, Centre for Pattern Recognition and Data Analytics

Connor, Jason; The University of Queensland, Discipline of Psychiatry and Centre for Youth Substance Abuse Research

<b>Primary Subject

Heading</b>: Mental health Secondary Subject Heading: Health informatics

Keywords: HoNOS, MENTAL HEALTH, Assessment

(17)

For peer review only

Consistency of the Health of the Nation Outcome Scales

(HoNOS) at inpatient-to-community transition

Wei Luo*, Richard Harvey, Truyen Tran, Dinh Phung, Svetha Venkatesh, and Jason P. Connor *Correspondence to: Wei Luo, Centre for Pattern Recognition and Data Analytics, School of Information Technology, Deakin University, Geelong, Victoria, Australia 3220

[email protected]

Abstract

Objectives: The Health of the Nation Outcome Scales (HoNOS) are mandated outcome-measures in many mental-health jurisdictions. When HoNOS are used in different care settings, it is important to assess if setting specific bias exists. This article examines the consistency of HoNOS in a sample of psychiatric patients transitioned from acute inpatient care and community centres.

Setting: A regional mental health service with both acute and community facilities.

Participants: 111 psychiatric patients were transferred from inpatient care to community care from 2012 to 2014. TheirHoNOS scores were extracted from a clinical database; Each inpatient-discharge assessment was followed by a community-intake assessment, with the median period between assessments being 4 days (range 0-14). Assessor experience and professional background were recorded.

Primary and secondary outcome measures: The difference of HoNOS at inpatient-discharge and community-intake were assessed with Pearson correlation, Cohen’s Kappa, and effect size.

Results: Inpatient-discharge HoNOS was on average lower than community-intake HoNOS. The average HoNOS was 8.05 at discharge (median 7, range 1-22), and 12.16 at intake (median 12, range 1-25), an average increase of 4.11 (SD 6.97). Pearson correlation between two total scores was 0.073 (95% CI: 0.0950.238) and Cohen’s Kappa was 0.02 (95% CI: -0.02-0.06). Differences did not appear to depend on assessor experience or professional background.

Conclusions: Systematic change in the HoNOS occurs at inpatient-to-community transition. Some caution should be exercised in making direct comparisons between Inpatient HoNOS and community HoNOS scores.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(18)

For peer review only

Strengths and limitations of this study

• Consistency of HoNOS scores were based on data linkage between different facilities with largely overlapping assessment timeframes.

• The data were routinely collected in a naturalistic setting with diversity in both consumer needs and clinician background, reflecting typical use of the HoNOS. • The study is limited by its retrospective nature.

• Patients in this study were primarily treated for psychosis/schizophrenia and findings many not be generalizable to other mental health conditions.

Page 2 of 15 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(19)

For peer review only

Introduction

The Health of the Nation Outcome Scales (HoNOS) are now routinely applied as a mandatory mental health outcome measure in the United Kingdom and many Australian and New Zealand jurisdictions, also gaining popularity in a number of other countries 1-6. The HoNOS serve not only as a tool to evaluate patient functioning, but also as an index for comparative service performance and cost effectiveness analyses7. Given mandated application of the HoNOS, it is a timely question to ask whether HoNOS can be interpreted consistently across different care settings. In this study, we examined stability of the HoNOS scores in a sample of public health service psychiatric consumers transitioning from

inpatient to community care.

HoNOS Background:Reliable measures of patient mental response to treatment is central to care planning and quality assurance. Symptom rating scales such as the Beck Depression Inventory (BDI, 8), State-Trait Anxiety Inventory (STAI, 9) and Positive and Negative Syndrome Scale (PANSS, 10) are well recognised individual assessment tools that reliably measure patient mental health symptoms. Many symptom rating scales do not

comprehensively assess overall patient health and social functioning. These domains are typically seen by health and government services as more critical in evaluating patient outcomes11. To address the need for a more standardised and systematic approach at measuring health outcomes for mental health patient functioning, the HoNOS was developed12 13. The HoNOS has 12 items covering behaviour, impairment, symptoms and social functioning domains 13.

Early HoNOS psychometric validation studies were unconvincing. A six week pre-post administration analysis of the HoNOS by Bebbington, et al. 14 concluded that there were “serious problems in using the instrument as a routine measure of clinical status in busy psychiatric services.” (p. 389). They found that performance of the instrument was likely to be related to the assessors’ level of training and skill, which reflected clinical reality in most busy mental health services. Similar evaluations around this time also failed to provide compelling support for the widespread adoption of the HoNOS, primarily on psychometric

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(20)

For peer review only

grounds (for example 15 16). One of the key concerns was poor reliability14 16. Despite no modification to the scale itself, subsequent reliability testing has provided additional support for its utility in clinical practice, based on larger and more heterogeneous samples, and arguably more sophisticated statistical approaches to validation. The reluctance of some clinical staff to fully adopt routine HoNOS assessments has been reported 17 and there remain some concerns about the utility of the scale to inform clinical, as opposed to

population level change 18.

In a comprehensive systematic narrative review of HoNOS validation studies, Pirkis, et al. 1 concluded that the scale had “good” validity (that is, it accurately measures what it was designed to measure) and had “adequate” test-retest reliability (that is, consistently measures functioning at different time points) and inter-rater reliability (high correlations between different assessors scores for the same patient). Pirkis, et al. 1 did note that few HoNOS studies have formally conducted test-retest reliability (n=3), and one-third of the 12 items consistently performed poorly on this psychometric index. For example, in a sample of 100 inpatients, Orrel, et al.’s psychometric evaluation 19 found the one week test-rest reliability of HoNOS items ranged from 0.33- 0.80 (mean 0.57), with only 2 items exceeding the accepted 0.7 cut-off 20. Pirkis, et al.’s review 1 also found inter-rater agreement has been identified as poor on at least 50% of the items. Orrel, et al.’s reported inter-rater item reliability ranged from -0.03 to 0.65 (mean 0.395), with only one of the items exceeding the recognised cut-off for “good” (0.60-0.74) reliability 21. In a sample of 100 older psychiatric inpatients (average age =77 years), all but two items exceed minimum test-retest or inter-rater reliability 22. Brooks 3 also found “limited” inter-rater reliability (0.50-0.65) and concluded that “the HoNOS not be implemented as a major outcome tool, until the reliability and validity of the HoNOS is clearly established.” (pp. 509-510).

HoNOS at care transition: In the Australian state of Victoria, when a consumer is discharged from an inpatient unit and subsequently admitted to a community residential unit, both an inpatient “discharge” HoNOS and a subsequent community “intake” HoNOS are completed. These data provide two assessments within a practically overlapping timeframe. Examining these data can provide important evidence on the consistency of HoNOS across mental health care settings.

Page 4 of 15 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(21)

For peer review only

Methods

Subjects

The study population includes all mental health patients from a regional mental health service who were transferred from an acute psychiatric unit to a number of community teams and received both inpatient discharge HoNOS and subsequent community intake HoNOS between July 2012 and June 2014. There were 70 males (mean age 38, SD 12.2) and 41 females (mean age 41.5, SD 13.3). The majority of the sample were being treated for Schizophrenia, schizotypal or delusional disorders (n=63).

Procedure

HoNOS were administered to inpatients on discharge by clinically trained inpatient staff, and within a maximum of 14 days by clinically trained community-based mental health staff. The median time between inpatient and community HoNOS assessments was 4 days, with 95% administered within 12 days.

Measures

The HoNOS 12 13 consists of 12 items, each ranging from 0 (no problem) to 4 (severe problem). As recommended by scale developers 12 , the 12 items are scored into four subscales12: Behaviour (items 1-3), Impairment (items 4-5), Symptoms (items 6-8), and Social (items 9-12). The sum of the 12 items provides a total score (range of 0 to 48). HoNOS can be interpreted at the item, subscale or total score level. The rating period is generally the preceding two weeks for inpatients at admission, for hospital outpatients, and for all clients of community–based services.23 24 The exception is at discharge from acute inpatient care, in which case the rating period should generally be the preceding three days and not collected if length of stay is less than 72 hrs. A community intake rating is collected at the allocation of case manager or first care plan. 25 Therefore the second assessment period is expected to overlap with the initial assessment period.

Statistical Analysis

Both inpatient and community HoNOS scores for each patients were paired and the average difference computed. Paired exact Wilcoxon signed-rank test was performed at the three scoring levels of the HoNOS (item, substance, total score). R package exactRankTests was used for computing this test statistics. 26 Both Pearson correlation and Cohen’s Kappa were

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(22)

For peer review only

calculated. Chi-square analyses examined if professional background and level of experience contributed to differences in HoNOS scores between inpatient and community assessments.

Hospital and university human ethics approval was obtained (approval number 12/83).

Results

Total HoNOS score

The average inpatient discharge HoNOS score was 8.05 (median 7, range 1-22) and the average community intake HoNOS was 12.16 (median 12, range 1-25). Pearson correlation between the two scores was 0.073 (95% CI: -0.095-0.238), and Cohen’s Kappa was 0.02 (95% CI: -0.02-0.06), showing poor consistency among two scores. (The numbers are slightly higher when two assessments were done within 4 days, where Pearson correlation was 0.14 (95% CI:-0.11-0.37) and Cohen’s Kappa was 0.07 (95% CI: -0.01-0.14)). On average, the total HoNOS score increased by 4.11 (SD 6.97) from an inpatient discharge to a community intake. The change of the total score reached the threshold of 4-8 points for clinically relevant change proposed by Parabiaghi and colleagues 4 27 and validated by Egger and colleagues28. Exact Wilcoxon signed-rank test for matched data returned p-value < 0.001, indicating that the HoNOS increase from inpatient discharge and community intake is statistically significant. Cohen’s d was 0.82 (95% CI: 0.58-1.07), indicating a large effect size (of difference). The power based on paired t-test was 1.

Schizophrenia, schizotypal and delusional disorders (F2x) were the most common diagnoses (56.8%), followed by mood disorders (F3x, 10.8%). Given the low frequency of other

diagnoses (5 cases in adjustment disorders, acute stress reaction, anxiety disorder,

substance intoxication, and dementia), an “other” category was used and it also included all episodes with no diagnosis recorded in the database. When diagnostic groups were

considered, F2x and “Other” diagnoses demonstrated significant differences between inpatient and community assessments (4.42 and 4.16, respectively). The differences between assessments also reached the Paragbiaghi’s threshold for clinically relevant change. There were no significant differences reported in patients with mood disorders (Table 1). Page 6 of 15 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(23)

For peer review only

Table 1 Change of the total HoNOS scores by diagnosis.

Average (range)

Pearson correlation

Cohen’s kappa

Diagnosis Discharge Intake Change p-value (95% CI) (95% CI)

F2x: Schizophrenia, schizotypal and delusional disorders (n=63) 7.58 (1-20) 12.0 (3-25) 4.42 <0.001 0.07 (-0.18, 0.32) -0.01 (-0.06, 0.03) F3x: Mood [affective] disorders (n=12) 7.75 (1-18) 10.04 (1-19) 2.29 0.189 -0.15 (-0.69, 0.49) 0.06 (-0.10,0.22) Other diagnoses (n=36) 8.81 (1-22) 12.97 (2-24) 4.16 <0.001 0.11 (-0.14, 0.34) 0.04 (-0.02, 0.11)

HoNOS subscales and Individual Items

The increase in scores between assessments was also evidenced in HoNOS subscales (Table 2) and individual score items (Table 3). Among Wing’s four factor subscales 12, only the change in Social did not have a p-value <0.001 and a t-test power of nearly 1. Using

Speak-Hay-Muncer’s four factor model29 and Lovaglio’s 6-item subscale30 also resulted in

statistically significant change between assessments. The only score items that did not reach statistical significance were item 11:Problems with living conditions and item 12: Problems

with occupation and activities.

Table 2 Changes of the four HoNOS subscales defined in 12 and HoNOS-6 defined in 30.

HoNOS subscales Average (range) Change Cohen’s d (95% CI) p-value (power) Pearson correlation Cohen’s kappa discha rge

Intake (95% CI) (95% CI)

Wing et al’s original 4-factor subscales Behaviour 0.84 (0-5) 2.0 (0-7) 1.16 0.75 (0.50, 1.0) <0.001 (1) 0.24 (0.07, 0.39) 0.15 (0.07, 024) Impairment 1.33 (0-5) 2.10 (0-6) 0.77 0.50 (0.26, 0.74) <0.001 (1) 0.43 (0.28, 0.56) 0.08 (-0.01, 0.17) Symptoms 2.52 (0-7) 3.86 (0-10) 1.34 0.75 (0.50, 0.99) <0.001 (1) 0.15 (-0.02, 0.31) 0.07 (0, 0.13) 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(24)

For peer review only

Social 3.35 (0-12) 4.20 (0-13) 0.84 0.33 (0.09, 0.57) 0.006 (0.97) 0.18 (0, 0.33) 0.02 (-0.04, 0.08) Speak et al.’s four factor model Personal wellbeing 2.50 (0-10) 3.69 (0-13) 1.19 0.50 (0.26, 0.74) <0.001 (1) 0.31 (0.15, 0.45) 0.05 (-0.02, 0.12) Emotional wellbeing 1.59 (0-6) 2.95 (0-11) 1.36 0.80 (0.55, 1.05) <0.001 (1) 0.27 (0.11, 0.42) 0.06 (-0.01, 0.14) Social wellbeing 2.23 (0-9) 3.47 (0-12) 1.24 0.56 (0.31, 0.80) <0.001 (1) 0.09 (-0.08, 0.25) 0.08 (0, 0.15) Severe disturbanc e 1.15 (0-4) 1.88 (0-7) 0.74 0.56 (0.31, 0.80) <0.001 (1) 0.26 (0.10, 0.42) 0.13 (0.03, 0.22) Lovaglio and Monzani’s HoNOS-6 subscales 4.21 (0-13) 6.08 (0-16) 1.87 0.58 (0.34, 0.83) <0.001 (1) 0.19 (0.02, 0.35) 0.02 (-0.03, 0.08) Page 8 of 15 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(25)

For peer review only

Table 3 Changes of the 12 HoNOS score items.

HoNOS score items Average (range) p-value Pearson

correlation

Cohen’s kappa

Discharge Intake Change (95% CI) (95% CI)

1. Overactive, aggressive, disruptive behaviour 0.31 (0-2) 0.73 (0-4) 0.42 <0.001 0.34 (0.19, 0.49) 0.15 (0.03, 0.27) 2. Non-accidental self-injury 0.13 (0-2) 0.44 (0-4) 0.31 <0.001 0.24 (0.08, 0.39) 0.13 (0, 0.27) 3. Problem-drinking or drug-taking 0.41 (0-3) 0.83 (0-4) 0.43 <0.001 0.31 (0.14, 0.45) 0.19 (0.08, 0.31) 4. Cognitive problems 0.71 (0-3) 1.03 (0-4) 0.32 <0.001 0.24 (0.08, 0.40) 0.05 (-0.06, 0.17) 5. Physical illness or disability problems 0.61 (0-3) 1.07 (0-4) 0.45 <0.001 0.59 (0.46, 0.69) 0.26 (0.15, 0.37) 6. Problems associated with hallucinations and delusions 0.88 (0-3) 1.21 (0-4) 0.33 <0.001 0.31 (0.15, 0.45) 0.24 (0.13, 0.36) 7. Problems with depressed mood 0.76 (0-3) 1.03 (0-4) 0.28 <0.001 0.29 (0.13, 0.44) 0.09 (-0.04, 0.21) 8. Other mental and behavioural problems 0.86 (0-3) 1.61 (0-4) 0.68 <0.001 0.26 (0.07, 0.43) 0.11 (0, 0.22) 9. Problems with relationships 1.11 (0-4) 1.70 (0-4) 0.59 <0.001 0 (-0.18, 0.16) 0.02 (-0.07, 0.12) 10.Problems with activities of daily living 0.88 (0-3) 1.34 (0-4) 0.46 <0.001 0.34 (0.19, 0.48) 0.08 (-0.03, 0.19) 11.Problems with living conditions 0.55 (0-3) 0.63 (0-4) 0.08 0.27 0.19 (0, 0.36) 0.02 (-0.12, 0.15) 12.Problems with occupation and activities 0.55 (0-3) 0.52 (0-3) -0.05 0.70 -0.04 (-0.23, 0.14) -0.05 (-0.2, 0.1)

Clinician Profession and Experience

To examine if clinical professional background and level of experience contributed to differences in HoNOS scores, the proportion of staff profession/experience between

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(26)

For peer review only

Inpatient clinicians and community clinicians was examined. Of the patients in the sample, 98 (82%) were assessed by experienced inpatient nurses and 94 (79%) were assessed by experienced community nurses, indicating equivalence across services (experienced nurses vs. others by community vs. hospital Chi-squared=0.2425, p = .622). The remaining staff included psychologists (n=15), less experienced nurses (n=2) and students (n=11). Two ANOVAs were conducted which examined potential differences in HoNOS scores within each service, by profession and experience. (The score changes at transition were mostly normally distributed, with Shapiro-Wilk test returning a p-value of 0.20). No evidence supported potential bias in score changes introduced by experience levels at discharge (F[2, 109]=0.11, p=0.90) or at intake (F[1,107]=0.70, p=0.50).

Discussion

The HoNOS has widespread, mandated use in many public health mental health systems including the UK and Australia. It is designed to measure health outcomes for mental health patient functioning at both a clinical and population level. It has been used as key index for comparative service performance and cost effectiveness analyses (for example7), and therefore can be applied as a lever for resource allocation. Given the scope of HoNOS roll out, reliability of measurement is critical. Perhaps surprisingly, the reliability of the HoNOS has received little empirical attention. Within a publically funded mental health system, this study identified poor total and sub-score consistency across treatment settings. The median delay between an inpatient-discharge HoNOS and the following community-intake HoNOS was 4 days, with 95% occurring within 12 days (range 0—14 days). All individual items on the HoNOS on this sample performed poorly. No systematic differences in the assessors’ professional background or level of clinical experience were identified.

The mean difference between scores following inpatient discharge to community

assessment was 4 points, or approximately a one-third decrease in clinician assessed patient functioning. This reached the 4 point cut-off of clinically relevant change suggested by Parabiaghi and colleagues 4 27 and later corroborated by Egger and colleagues28. Using the thresholds proposed by Parabiaghi 27 (13 between moderate and severe illness; 10 between moderate and mild illness), among 81 consumers deemed mild at hospital discharge, 14

Page 10 of 15 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(27)

For peer review only

would have had changed to moderate at community intake and 32 would have had changed to severe.

Patients with mood disorders reported less variation between assessments than those with psychotic-type diagnoses, possibility reflecting a more stable prognosis. Differences in HoNOS scores may also reflect the severity of the population under investigation. Orrell, et al. (1999) did show average HoNOS inpatient scores to be higher than patients assessed in outpatient or community setting. Inpatient status is typically indicated for patients that cannot be managed as an outpatient or community patients, due to the severity or complexity of the condition. It is important to determine if these observed differences reflect true variation across inpatient and community functioning, or vulnerability in

reliability of measurement. Counter-intuitively, in the current study, patients assessed in the community were rated as functioning significantly more poorly than when assessed as an inpatient with overlapping assessment windows (median four day interval).

The recommended timeframe for the HoNOS assessment may provide some additional clues to poor reliability. At discharge, the period of assessment is recommended to cover the past 72 hours 24. In the community, the recommendation assessment period is within the past 2 weeks 24. The most likely clinical reason for discharge is improved functioning and/or increased symptom stability. Patients who have improved functioning over the past 3 days are among those likely to be considered for inpatient discharge. When assessed in the community, a two-week period is reviewed which may encapsulate broader functioning difficulties. Functioning may have also deteriorated due to less intensive monitoring and patient management. However, given the two review periods overlap and higher

community ratings are consistent across all domains, differences point towards measurement problems.

The findings should be interpreted in light of some limitations. Although previous reliability studies have typically occurred in similar or smaller sample sizes (e.g., 3 19 22), a larger and more heterogeneous psychiatric sample across a broader age range would provide a higher powered study to more robustly detect associations between testing periods and assessors. Patients in this study were primarily treated for psychosis/schizophrenia and findings many not be generalizable to other mental health conditions. We cannot confirm all assessors

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(28)

For peer review only

received the same level of training in the HoNOS. Given training differences have been previously detected 14, there may have been systematic biases associated with either inpatient or community assessors. The interval period for test-rest reliability ranged from 0-14 days, with a median of 4 days. The timeframe is consistent with previous HoNOS

reliability studies that have re-tested up to 6 weeks (e.g., 14). The modest variation in re-testing timeframes in the current study is not psychometrically desirable, however the clear pattern of results demonstrating an upward assessment of severity across domains provides some confidence that timeframe variations did not confound findings. Finally, the score consistency was also subject to potential time biases. A central purpose of this study was to assess the ecological reliability of the HoNOS in a busy clinical mental health service. A more robust design would video record patient assessments and responses (at one point in time) and has multiple clinicians blind to the original assessment re-score the scale.

In summary, limited support was found to support consistent interpretation of the HoNOS at different settings in this psychiatric patient sample. Recognising the scale has already been widely mandated in clinical practice, further larger scale testing is considered critical to ensure these findings are not simply a result of sample or assessor bias.

Contributorship statement

WL, RH and JPC designed the experiments. WL performed data extraction. WL, TT, DP, and SV carried out statistical analyses. WL, RH and JPC analysed the results. All authors participated in the preparation of the manuscript.

Competing interests

The authors declare that there are no conflicts of interest.

Funding

J.P.C. is supported by a National Health and Medical Research Council (NHMRC) of Australia Career Development Fellowship (1031909).

Data sharing statement

Dataset may be available through request to the original data guarantor.

Page 12 of 15 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(29)

For peer review only

References

1. Pirkis JE, Burgess PM, Kirk PK, et al. A review of the psychometric

properties of the Health of the Nation Outcome Scales (HoNOS)

family of measures. Health and quality of life outcomes 2005;

3

:76.

2. Glover G, Knight S, Melzer D, et al. The development of a new minimum

data set for specialist mental health care. Health Trends

1997;

29

(2):48-51.

3. Brooks R. The reliability and validity of the Health of the Nation Outcome

Scales: validation in relation to patient derived measures. The

Australian and New Zealand journal of psychiatry 2000;

34

(3):504-11; discussion 20-1.

4. Parabiaghi A, Rapisarda F, D'Avanzo B, et al. Measuring clinical change in

routine mental health care: differences between first time and longer

term service users. The Australian and New Zealand journal of

psychiatry 2011;

45

(7):558-68.

5. Kortrijk HE, Mulder CL, Drukker M, et al. Duration of assertive

community treatment and the interpretation of routine outcome

data. The Australian and New Zealand journal of psychiatry

2012;

46

(3):240-8.

6. Theodoridou A, Jaeger M, Ketteler D, et al. The concurrent validity and

sensitivity of change of the German version of the health of the nation

outcome scales in a psychiatric inpatient setting. Psychopathology

2011;

44

(6):391-7.

7. Slade M, Byford S, Barrett B, et al. Alternatives to standard acute

in-patient care in England: short-term clinical outcomes and

cost-effectiveness. The British journal of psychiatry Supplement

2010;

53

:s14-9.

8. Beck AT, Ward CH, Mendelson M, et al. An inventory for measuring

depression. Archives of general psychiatry 1961;

4

:561-71.

9. Spielberger CD, Gorsuch RL, Lushene RE. Manual for the state-trait

anxiety inventory. 1970.

10. Kay SR, Fiszbein A, Opler LA. The positive and negative syndrome scale

(PANSS) for schizophrenia. Schizophrenia bulletin 1987;

13

(2):261-76.

11. Trauer T. Outcome measurement in chronic mental illness.

International review of psychiatry (Abingdon, England)

2010;

22

(2):99-113.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(30)

For peer review only

12. Wing JK, Beevor AS, Curtis RH, et al. Health of the Nation Outcome

Scales (HoNOS). Research and development. The British journal of

psychiatry : the journal of mental science 1998;

172

:11-8.

13. Wing JK, Curtis R, Beevor A. HoNOS: Health of the Nation Outcome

Scales: Report on Research and Development July 1993-December

1995: Royal College of Psychiatrists London, 1996.

14. Bebbington P, Brugha T, Hill T, et al. Validation of the Health of the

Nation Outcome Scales. The British journal of psychiatry : the journal

of mental science 1999;

174

:389-94.

15. Sharma VK, Wilkinson G, Fear S. Health of the Nation Outcome Scales: a

case study in general psychiatry. The British journal of psychiatry :

the journal of mental science 1999;

174

:395-8.

16. Trauer T, Callaly T, Hantz P, et al. Health of the Nation Outcome Scales.

Results of the Victorian field trial. The British journal of psychiatry :

the journal of mental science 1999;

174

:380-8.

17. Pirkis J, Burgess P, Coombs T, et al. Routine measurement of outcomes

in Australia's public sector mental health services. Australia and New

Zealand health policy 2005;

2

(1):8.

18. Duke B. HoNOS in the consultation liaison psychiatry setting: is it valid?

Australasian psychiatry : bulletin of Royal Australian and New

Zealand College of Psychiatrists 2010;

18

(6):547-50.

19. Orrell M, Yard P, Handysides J, et al. Validity and reliability of the Health

of the Nation Outcome Scales in psychiatric patients in the

community. The British journal of psychiatry : the journal of mental

science 1999;

174

:409-12.

20. Nunnally J, Bernstein I. Psychometric Theory 3ed. New York:

McGraw-Hill, 1994.

21. Fleiss JL, Levin B, Paik MC. The measurement of interrater agreement.

Statistical methods for rates and proportions 1981;

2

:212-36.

22. S. Shergill KKS, Knightley Seneviratna, Martin W. Orrell, Sukhwinder.

The validity and reliability of the Health of the Nation Outcome Scales

(HoNOS) in the elderly. Journal of Mental Health 1999;

8

(5):511-21.

23. Strategic Planning Group for Private Psychiatric Services. A national

model for the collection and analysis of a minimum data set with

outcome measures for private, hospital-based, psychiatric services.

Secondary A national model for the collection and analysis of a

minimum data set with outcome measures for private,

hospital-based, psychiatric services 2000.

24. Department of Human Services. Outcome measurement in clinical

mental health services. In: Victorian Government Department of

Human Services, ed.: Mental Health and Drugs Division,, 2007.

Page 14 of 15 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

(31)

For peer review only

25. Barwon Health, Mental Health, Drugs and Alcohol Services. ACE

Guidelines: Assessment, Care and Evaluation Manual. Barwon Health:

Barwon Health, 2010.

26. exactRankTests: Exact Distributions for Rank and Permutation Tests

[program], 2013.

27. Parabiaghi A, Kortrijk HE, Mulder CL. Defining multiple criteria for

meaningful outcome in routine outcome measurement using the

Health of the Nation Outcome Scales. Social psychiatry and

psychiatric epidemiology 2014;

49

(2):291-305.

28. Egger ST, Weniger G, Prinz S, et al. Health of the Nation Outcome Scales

in a psychiatric inpatient setting: assessing clinical change. Journal of

evaluation in clinical practice 2015;

21

(2):236-41.

29. Speak BL, Hay P, Muncer SJ. HoNOS - their utility for payment by results

in mental health. International journal of health care quality

assurance 2015;

28

(2):115-28.

30. Lovaglio PG, Monzani E. Health of the nation outcome scales evaluation

in a community setting population. Quality of life research : an

international journal of quality of life aspects of treatment, care and

rehabilitation 2012;

21

(9):1643-53.

Author information:

Wei Luo, PhD, School of Information Technology, Deakin University, Australia

Richard Harvey, MD, FRANZCP, Mental Health Services, Barwon Health, Australia; School of Medicine, Deakin University, Australia; Email: [email protected]

Truyen Tran, PhD, Centre for Pattern Recognition and Data Analytics, Deakin University, Australia Dinh Phung, PhD, Centre for Pattern Recognition and Data Analytics, Deakin University, Australia Svetha Venkatesh, PhD, Centre for Pattern Recognition and Data Analytics, Deakin University, Australia

Jason P. Connor, PhD, FAPS, Discipline of Psychiatry and Centre for Youth Substance Abuse Research, The University of Queensland, Brisbane, Australia

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(32)

For peer review only

Consistency of the Health of the Nation Outcome Scales (HoNOS) at inpatient-to-community transition

Journal: BMJ Open

Manuscript ID bmjopen-2015-010732.R2 Article Type: Research

Date Submitted by the Author: 06-Apr-2016

Complete List of Authors: Luo, Wei; Deakin University, School of Information Technology

Harvey, Richard; University of Melbourne, Clinical and Biomedical Sciences Tran, Truyen; Deakin University, School of IT

Phung, Dinh; Deakin University, Australia, Centre for Pattern Recognition and Data Analytics

Venkatesh, Svetha; Deakin University, Centre for Pattern Recognition and Data Analytics

Connor, Jason; The University of Queensland, Discipline of Psychiatry and Centre for Youth Substance Abuse Research

<b>Primary Subject

Heading</b>: Mental health Secondary Subject Heading: Health informatics

Keywords: HoNOS, MENTAL HEALTH, Assessment

(33)

For peer review only

Consistency of the Health of the Nation Outcome Scales

(HoNOS) at inpatient-to-community transition

Wei Luo*, Richard Harvey, Truyen Tran, Dinh Phung, Svetha Venkatesh, and Jason P. Connor *Correspondence to: Wei Luo, Centre for Pattern Recognition and Data Analytics, School of Information Technology, Deakin University, Geelong, Victoria, Australia 3220

[email protected]

Abstract

Objectives: The Health of the Nation Outcome Scales (HoNOS) are mandated outcome-measures in many mental-health jurisdictions. When HoNOS are used in different care settings, it is important to assess if setting specific bias exists. This article examines the consistency of HoNOS in a sample of psychiatric patients transitioned from acute inpatient care and community centres.

Setting: A regional mental health service with both acute and community facilities.

Participants: 111 psychiatric patients were transferred from inpatient care to community care from 2012 to 2014. TheirHoNOS scores were extracted from a clinical database; Each inpatient-discharge assessment was followed by a community-intake assessment, with the median period between assessments being 4 days (range 0-14). Assessor experience and professional background were recorded.

Primary and secondary outcome measures: The difference of HoNOS at inpatient-discharge and community-intake were assessed with Pearson correlation, Cohen’s Kappa, and effect size.

Results: Inpatient-discharge HoNOS was on average lower than community-intake HoNOS. The average HoNOS was 8.05 at discharge (median 7, range 1-22), and 12.16 at intake (median 12, range 1-25), an average increase of 4.11 (SD 6.97). Pearson correlation between two total scores was 0.073 (95% CI: 0.0950.238) and Cohen’s Kappa was 0.02 (95% CI: -0.02-0.06). Differences did not appear to depend on assessor experience or professional background.

Conclusions: Systematic change in the HoNOS occurs at inpatient-to-community transition. Some caution should be exercised in making direct comparisons between Inpatient HoNOS and community HoNOS scores.

2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59

(34)

For peer review only

Strengths and limitations of this study

• Consistency of HoNOS scores were based on data linkage between different facilities with largely overlapping assessment timeframes.

• The data were routinely collected in a naturalistic setting with diversity in both consumer needs and clinician background, reflecting typical use of the HoNOS. • The study is limited by its retrospective nature.

• Patients in this study were primarily treated for psychosis/schizophrenia and findings many not be generalizable to other mental health conditions.

Page 2 of 15 BMJ Open 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60

Figure

Table 1 Change of the total HoNOS scores by diagnosis.
Table 2 Changes of the four HoNOS subscales.
Table 3 Changes of the 12 HoNOS score items.
Table 2 Changes of the four HoNOS subscales defined in  12  and HoNOS-6 defined in  30
+4

References

Related documents

The results of this study showed that in 2010, in France, about 1 to 10 patients managed for primary lung cancer in the respiratory department of a general hospital died within 1

It was decided that with the presence of such significant red flag signs that she should undergo advanced imaging, in this case an MRI, that revealed an underlying malignancy, which

The paper assessed the challenges facing the successful operations of Public Procurement Act 2007 and the result showed that the size and complexity of public procurement,

Large babo-mutant photoreceptor cell clones were found to project their axons normally into babo heterozygous brain lobes (Fig. 3E), providing further support for the argument that

Secondary outcomes are physical tension, activation (Behavioral Activation for Depression Scale (BADS) [27]), cognitive reappraisal (Emotion Regulation Questionnaire (ERQ)

This paper is an attempt based on completely my own ideas and experiences to relate the significance of values education in the context of Bhagavad-Gita for today’s

Figure 4 The excess effect of 3 or 6 months low to moderate carbohydrate diet compared with high-carbohydrate diet on HbA1c (%) versus reported intake (Energy %) of carbohydrate in

The diagnosis of measles in the patient was confirmed by both measles virus isolation from the throat swab and also detection of measles-specific antibodies (IgM) in the serum..