Data extraction methods

Data were extracted on study design (RCT, quasi-experimental [ie: a non-randomised comparison group], non-experimental [ie: pre-post only, a before/after comparison group] and modelled), sample size and program length, authorship (private company, research institutions or government), country of origin, publication year, study characteristics (organisation size, industry type, employee target group [healthy, at risk or disease management]), and program design (single or multicomponent). Intervention focus was grouped into 3 categories: SNAPS (smoking, nutrition, alcohol, physical activity and stress), vaccination and other (dental, cancer screening). When studies reported more than one intervention arm alongside a comparator, the economic evaluation for each was considered a separate study, effectively increasing the number of studies.

Economic study metrics such as perspective, design (retrospective, prospective), time horizon (currency, time value), discount rate, comparator type (control group design, pre- post, modelled), effect measure (incremental or cost comparison), costs reported (direct, indirect), the economic form (cost benefit analysis [CBA],cost effectiveness analysis [CEA], cost utility analysis [CUA]), the calculation method, and how outcomes were measured and valued were also recorded along with economic results (reported costs, benefits and ROI). Data extraction and quality assessment (see below) were performed by one author (SB) and a 20% sample was independently coded by another (AP). Any disagreements were resolved

through discussion.

2.3.3.1 Methodological quality assessment

Studies were scored against the 36-item British Medical Journal Economic Evaluation Working Party (BMJ checklist),15,23 a guideline of methodological and essential elements to improve clarity of economic evaluations. Items referred to the study question, selection of alternatives, form of evaluation, effectiveness data, measurement and valuation of benefit, costing, modelling, adjustments for timing of costs and benefits, uncertainty and presentation of results and were all considered within three headings: study design, data collection and analysis and interpretation (Table 2.2). Each of the 36 items were given equal weighting and the items performed or reported were summed and expressed as a percentage of the total number of items applicable to each study.15 Studies were then placed into categories of methodological quality; high quality (>75%), moderate quality (50- 75%) and low quality (<50%).

Table 2.2 36-item British Medical Journal Economic Evaluation Working Party (BMJ checklist)* Study design

1. Was the research question stated?

2. Was the economic importance of the research question stated? 3. Was/were the viewpoint(s) of the analysis clearly stated and justified?

4. Was a rationale reported for the choice of the alternative programmes or interventions compared?

5. Were the alternatives being compared clearly described? 6. Was the form of economic evaluation stated?

7. Was the choice of form of economic evaluation justified in relation to the questions addressed?

Data collection

8. Was/were the source(s) of effectiveness estimates used stated?

9. Were details of the design and results of the effectiveness study given (if based on a single study)?

10. Were details of the methods of synthesis or meta-analysis of estimates given (if based on an overview of a number of effectiveness studies)?

11. Were the primary outcome measure(s) for the economic evaluation clearly stated? 12. Were the methods used to value health states and other benefits stated?

13. Were the details of the subjects from whom valuations were obtained given? 14. Were productivity changes (if included) reported separately?

15. Was the relevance of productivity changes to the study question discussed? 16. Were quantities of resources reported separately from their unit cost? 17. Were the methods for the estimation of quantities and unit costs described? 18. Were currency and price data recorded?

19. Were details of price adjustments for inflation or currency conversion given? 20. Were details of any model used given?

was based?

Analysis and interpretation of results

22. Was time horizon of cost and benefits stated? 23. Was the discount rate stated?

24. Was the choice of rate justified?

25. Was an explanation given if cost or benefits were not discounted?

26. Were the details of statistical test(s) and confidence intervals given for stochastic data? 27. Was the approach to sensitivity analysis described?

28. Was the choice of variables for sensitivity analysis justified? 29. Were the ranges over which the parameters were varied stated?

30. Were relevant alternatives compared? (i.e. Were appropriate comparisons made when conducting the incremental analysis?)

31. Was an incremental analysis reported?

32. Were major outcomes presented in a disaggregated as well as aggregated form? 33. Was the answer to the study question given?

34. Did conclusions follow from the data reported?

35. Were conclusions accompanied by the appropriate caveats? 36. Were generalisability issues addressed?

*From Systematic Reviews: CRD's guidance for undertaking reviews in health care, 3rd ed. York, UK: Centre for Reviews and Dissemination; 2009:210-211. Available at: http://www.york.ac.uk/inst/crd/index_guidance.htm. Reproduced with permission from CRD; York, UK.

2.3.3.2 Economic outcomes

The financial outcomes within each study were represented as an ROI ratio and were either extracted if an ROI was provided, or (re)calculated from reported costs and benefits of a program against a comparator. When an ROI was not reported, the costs and benefits, as measured and specific to the individual study findings for monetary value (currency), price year and discounting (if applied) were extracted and the ROI formula applied.

The formula used was ROI = (Net Benefits – Net Costs of program)/Costs of program.24 When ROI was reported, the method of ROI calculation was examined and the accepted formula was applied to the reported costs and benefits if the calculation method differed. Our chosen methodology provided a consistent comparison of financial return between studies, in addition to the ratio alleviating any costing differences arising from currency and time variances across studies. It should be noted that many employers use ROI=Benefits/Costs as the formula for Return on Investment. We compared our ROI findings against its comparative ROI (calculated from this commonly used alternative) to examine whether our ROI formula accounted for any major difference in the ROI’s we report.

The effect measure was categorised as either incremental (the calculated difference in program costs and benefits between the intervention and the comparator groups) or cost comparison (when benefits were defined as cost savings1 when pre-post analysis). Benefits included change in worker productivity, and employer health care costs.

Costs were extracted as reported, irrespective of discounting. Whether or not discounting was performed was addressed by the BMJ methodological checklist (item #23/24/25) and accounted for in the quality scoring. Studies that did not report discounting beyond a one year time horizon were penalised in score (studies ≤ 1 year time horizon received a NA [not applicable] for these items). Additionally, we did not attempt to discount long term costs; a) to avoid possible “double discounting” in cases where authors may have discounted but failed to report it; b) as often costs were not itemised over time; and c) applying a discount rate to both costs and benefits (i.e. both denominator and numerator) would not affect the ROI ratio.

2.3.3.3 Data analysis

Summary data on ROI are presented as weighted mean and standard deviation (SD) with 95% confidence intervals (CI). Because the interventions reviewed differed markedly in scope and reach, the ROI for each study was weighted by the number of employees targeted directly or indirectly by the intervention program. Mean ROIs were stratified by study characteristics (refer to 2.3.3 Data extraction methods). Results of unweighted analyses are reported for comparison. To determine if certain study characteristics predict higher ROI, linear regression methods were used against the ROI weighted by relative number of participants. All weighted data were transformed prior to analysis to remove skewness. Being represented as a ratio, the dollar value of the numerator and denominator of each ROI estimate did not require conversion to units of common purchasing power. All statistical analyses were conducted using STATA© version 12 software package (Statacorp LP, Texas, USA).

2.3.3.4 Sensitivity analysis

Sensitivity analysis of quality scoring was performed using two additional methodological quality checklists, the Consensus Health Economic Criteria list (CHEC-List),14 and the NICE study limitations checklist: economic evaluations (NICE checklist).20 Comparisons of quality scores made by the three checklists were undertaken by assessment of differences in mean scores and by using correlation coefficients to summarise stability of ranking.

Appendix 2B Tables 2B.1 and 2B.2. In this Appendix you can find the CHEC-List and the NICE study limitations checklist. Tables can be found on pages 135 and 136.