Data availability and quality

Data need to be available on a regular basis for measurement purposes so that benchmarking can be associated with continuous improvement in quality (Meissner et al. 2006). What is of an excellent standard today might show a shift in expected performance tomorrow. The reference point, or benchmark, should be reviewed regularly (Kay 2007). Therefore, data needs to be collected on a regular basis to set the benchmarks that reflect already achieved improvements. Nolte (2010) asserted that data availability could be a major challenge for benchmarking, particularly when the aim is to collect data at an international level, as each country may have its own method of or tools for collecting data. This can have impact on data availability as well as on data quality and comparability for benchmarking (Kossarova et al. 2015).

58

Within healthcare, both routine and purposeful⁴ data is used for benchmarking. A number of studies demonstrate the practicality and suitability of routine data for this purpose. For example, a US-based study by Earl et al. (2005) used administrative data from Medicare to compare the intensity of end-of-life care for cancer patients. Hermann et al. (2006) and Meehan et al. (2007) compared existing mental-health indicators taken from several healthcare organisations and verified the usability of routine data for benchmarking. However, both studies recommended considering the use of case-mix adjustment processes for fairer comparisons. While routine data is appreciated for its regular availability, issues related to the quality and comparability of such data for benchmarking have been identified (Powell et al. 2003). In order to conduct effective benchmarking, where data is suitable, complete, accurate, available and comparable, the collection of a standardised and purposeful dataset is encouraged (Nolte 2010). One example of this is the Care Quality Commission’s (CQC) project for monitoring health and social-care services across England (Care Quality Commission 2016). While ensuring the quality, regular availability and comparability of data, CQC used an extensive list of indicators to collect care-monitoring data (both quantitative and qualitative) from various sectors, including NHS acute trusts, GP practices and trusts providing mental-health services. While purposeful data collection provides a degree of control of quality and comparability of data, it also requires effective collaboration and planning for making data available for benchmarking (Ellis 2006).

4 Data collected specifically for benchmarking.

59

Data need to be complete and correct in order to provide information for a specified indicator (Campbell et al. 2003). Poor-quality data can raise a number of issues including the misinterpretation of the indicators (Raleigh 2010). Data quality issues are mostly apparent in routinely collected data, as data is not collected specifically for benchmarking. Using such data for benchmarking can be risky. For example, when routinely collected data is compared statistically, whether across time or with other care providers, variations in data are revealed. According to Powell et al. (2003: 122),

“naturally, such variations imply ranking: that the measure reflects quality and that variations in the measure reflect variations in the quality”. However, if the data is not of good quality, such variation could be misleading in indicating ranking or change, when it may be reflective simply of variations or inaccuracies in collection.

In the context of DCM, data need to be available for benchmarking purposes.

The availability of data is related to its collection by mappers and organisations and then its storage in electronic databases for benchmarking analysis. There is the opportunity for using previously collected DCM data, if it is stored in electronic databases. Based on the concept and data model of the DCM international database (Khalid et al. 2010), the University of Bradford has developed a web-based DCM database application called the arc|hive DCM database (Surr et al. 2015) that can be used as a potential resource for quality DCM data. This database provides some inbuilt data validation, as will be discussed in detail in Chapter 3. However, it still does not permit checks for quality to be conducted beyond only of actual DCM

60

codes. Further, while this system can make DCM data available more readily, it is not known how

61

frequently the system is being used. In addition, it only contains DCM data and not additional data that might be required for effective benchmarking.

While the arc|hive DCM database can technically support data availability for benchmarking, the collection of DCM data however depends on individual mappers and organisations and their regular use of DCM.

Furthermore, DCM data needs to be of good quality for benchmarking purposes. Within the context of the primary use of DCM data, the literature shows that data quality is associated with the mapper’s reliability score (e.g., inter-rater reliability; IRR) (Brooker et al. 1998; Thornton et al. 2004). The criteria for what constitutes quality DCM data for secondary analysis purposes is, however, unknown.

While in this section, the data-availability and data-quality requirements are highlighted in relation to benchmarking, the literature also underlines the significance of these requirements for other secondary uses of data (e.g., research purposes) (Weiskopf and Weng 2012). More details of data-quality and availability requirements for the secondary use of data within research context are presented in Chapter 3.

In summary, the literature highlights two important aspects of benchmarking:

the indicator selection and the effectiveness of data to feed the indicator. The data is effective if it is suitable, comparable, available and of good quality to feed the indicators for benchmarking. Examining the practicality of DCM data for benchmarking within the context of the data requirements, there is evidence that DCM provides some valid indicators (such as the WIB score) for measuring quality-of-care and quality-of-life and that DCM can produce a

62

consistent set of data within organisations and comparable data across time.

While such characteristics make DCM data suitable for internal benchmarking, there is still limited evidence that support the practicality of DCM data for external benchmarking. To do this would mean assessing the practicality of DCM indicators for comparisons, especially for identifying the best practices in care. While suitability and comparability of data are important requirements for benchmarking, the literature also indicates that DCM data should also be of good quality and available for benchmarking purposes. However, little is known about either of these areas with the existing literature.

Within any organisation, the data requirements for benchmarking also depend, in part, on the perceptions of benchmarking, which in turn influence the approaches to benchmarking, as discussed next.