Quality

The ultimate goal is to manage quality. But you cannot manage it until you have a way to measure it, and you cannot measure it until you can monitor it

Arah et al 2003: 377

Attention to quality in healthcare has become a central issue in recent years, with increasing awareness on the role quality plays in informing public policy, supporting health care management and building public awareness about the factors affecting health (CIHI 2009). As argued by WPRO (2003), hospitals, community health centres, clinics, aid posts and high-level health ministries and departments should all be concerned with the impact poor quality data has on the quality of health care provided to users. Overall, data quality is important for determining the current and future needs of patients; medico-legal responsibilities; ensuring diseases are being treated and procedures performed; measuring outcomes of health care interventions; obtaining information on the users of services; teaching health care professionals; and planning (WPRO 2003; HMN 2008).

Page | 34 In everyday language, quality represents where, on a scale of bad-good-excellent, a user may place a certain product with regard to its intended use, and also in light of comparisons with other available products (Elvers & Rosn 1997). Quality assessments generally take one of two paths: procedural or substantive. Procedural assessments are concerned with issues related to transparency, reliability and replicability – how the data was processed and analysed (Mudde & Schedler 2010; Lewin et al 2010). Substantive assessments, on the other hand, deal with

data outcomes and have criteria based on validity, accuracy and precision (ibid; ibid).

However, ensuring the quality of data is much more difficult than ensuring the quality of other raw materials, and as Tayi and Ballou (1998) argue, this difficulty is further compounded by the low priority assigned to data quality assessments.

Confidence in the quality of information produced by an agency2 _{is vital for its survival: as soon}

as any information is regarded as ‘suspect’ the credibility of an agency is called into question and their perception as being a trustworthy source is undermined (Brackstone 1999). When information in public health reports is not accurate or available when needed, potentially disruptive consequences can result, including debates becoming focused on who has the ‘right’ numbers instead of the pros and cons of public health policy (Brackstone 1999; WHO 2003). In

their work on quality, Lewin and colleagues (2010) are very clear that while local data3 _is

important in contextualising and making global data relevant; we should remain cautious about using local data alone, as it is less reliable and can be misleading. They further argue that global data is often the best starting point for making judgements on the effects,

2 _{Here, an agency could refer to a Ministry of Health, Statistics Department, District hospital or individual healthcare} clinic, for example

Page | 35 modifying factors and ways to approach and address health problems, as much local data is difficult to locate and of poor quality.

Caution over the use of local data is reflected in the long-held consensus that information and data from the Pacific is, ‘incomplete, unreliable, obsolete and of poor quality’ (Finau 1994: 163). This consensus is clearly demonstrated at an international level: while Fiji’s Annual Reports, for example, show a maternal mortality ratio (MMR) ranging between 31 and 51 during 2004 to 2008; the World Bank (2010), World Health Organization (2010), UNICEF (2010) and UNSTATS (2010) all officially report the MMR for Fiji in 2005 as 210. In the majority of these external sources no reference is made to the reported MMR provided from Fiji, and while calculations are provided for how the ‘adjusted’ or ‘modelled’ MMR was established, no justification is provided for why Fiji’s MMR (which is approximately four-times lower than the modelled data) is not included in the official statistics. Overall, it would seem that due to the perception that the quality of data being produced in the Pacific is of dubious quality, much of the information is ignored and underutilised.

The example of the limited and lessening use of data produced from within the Pacific Region is what Wang et al (1997) refer to in their seminal work on data quality as an ‘intrinsic data problem’ (Figure 3). The logic behind an intrinsic data problem is as follows:

1. Mismatches in data provided from different sources initially causes a believability

problem as users do not know which source is incorrect, only that the data conflicts

2. As information on the causes of the mismatches accumulate, evaluations on the

accuracy of the data are generated

Page | 36

4. As this reputation builds, the data are seen as having little value-add and so are used

less.

Intrinsic data problems can also stem from judgements on the data production process; for example, placing a higher value on raw data as opposed to aggregated (this is demonstrated in path (2) in Figure 3). The authors state that, ‘a reputation for poor quality can also develop with little factual basis’ (Wang et al 1997: 105). This is of heightened importance for Pacific Island Countries and Territories, as they not only have to improve the quality of their data, but

also improve the reputation of their entire HIS: an undoubtedly challenging and complex task.

Figure 3 Intrinsic data quality problem (Wang et al 1997: 105)

2.2.1 Defining quality

While literature provides a wide range of techniques to assess and improve data quality, as information systems increase their size and scope, issues of quality are becoming more

Page | 37 complex and controversial (Batini et al 2009). Due to the contextual nature of quality, there remain discrepancies in definitions of its dimensions, and no agreement on which set of

dimensions defines quality (ibid). Furthermore, assessments of data quality among qualitative

research, or ‘unstructured’ data, are virtually absent (Akkerman et al 2008; Batini et al 2009; Tayi & Ballou 1998). As discussed by Brackstone (1999), the traditional statistical concept of quality, related to measures of standard error and bias, does not adequately address the broader meaning quality has taken on in the management of organisations and systems. Here, he argues, quality refers to the ‘fitness’ of final products and services in meeting the needs of users.

However, if users’ needs are taken as the primary factor in assessing the success of products and services, and quality is taken to reflect the aspects of statistical outputs that reflect their fitness for use – the varied number and needs of users mean that we are still left without an operational definition (Brackstone 1999). Furthermore, in defining quality in terms of its ‘fitness for use’ this implies that the concept of quality is relative and that data with quality for one use might not have quality in another (Tayi & Ballou 1998); again providing no clear definition.

Other authors and agencies have attempted to define the concept of quality: Elvers and Rosn (1997), the Canadian Institute of Health Information (2009) and Wang et al (1997) similarly regard quality as a measure of how well statistics meet users’ needs and expectations. In their work on quality, Arah et al (2003) argue that performance indicators (measures to capture health and health system trends and factors) provide an operational definition of quality, as performance indicators are essentially a quantitative measure of quality. The World Health Organization (WPRO 2003) regards quality as the production and dissemination of

Page | 38 understandable information for government policy-makers, community leaders, health planners and healthcare providers. As argued by Elvers and Rosn (1997), quality has taken on a descriptive meaning, and quality assessments need to consider both the product in question, and also its purpose. The Health Metrics Network (2008) echo this sentiment over a decade later, when they describe the process of assessing existing HIS in order to understand users’ current and perceived future requirements for statistical information. They propose that such assessments must be carried out, if we are to ‘increase the availability, quality and use of health information vital for decision-making at country and global levels’ (HMN 2008a: 2).

Overall, while there remains no single definitive definition of quality, most authors agree that it lies beyond the traditional statistical concept concerned with accuracy, and that it is made- up of a number of important components or dimensions (Brackstone 1999; Elvers & Rosn 1997). Again, while there is no universal consensus on which dimensions are required to ‘produce quality’, a number of dimensions are interrelated and there is significant overlap between different authors and agencies. In their review of the literature, Batini and colleagues (2009) provide a list of what they consider the four most basic quality dimensions as used by the majority of authors on the topic: accuracy, completeness, consistency and time-related dimensions. Table 1 provides a summary of the different quality dimensions defined by various authors and agencies. In general, the different dimensions of quality assess two main features: if information on the right topics is being produced, and if the appropriate concepts of measurement are being utilised (Brackstone 1999; IHP+ 2009).

Page | 39

Table 1 Quality dimensions (ordered by number of references)

Quality Dimensions Source

Accuracy WHO 2007; Lewin et al 2010; Elvers & Rosn 1997; Brackstone 1999; WPRO 2003; WHO 2004; IMF 2006; AHIMA 2008; CIHI 2009; Batini et al 2009; Wang et al 1997

Timeliness GDDS 2003; Elvers & Rosn 1997; Brackstone 1999; WPRO 2003; WHO 2004; HMN 2008; AHIMA 2008; CIHI 2009; Batini et al 2009; Wang et al 1997

Consistency WHO 2007; GDDS 2003; WHO 2003; HMN 2008; AHIMA 2008; Batini et al

2009; Wang et al 1997

Accessibility Brackstone 1999; WPRO 2003; IMF 2006; AHIMA 2008; Wang et al 1997 Completeness WHO 2007; WPRO 2003; WHO 2004; Batini et al 2009; Wang et al 1997 Relevance Brackstone 1999; AHIMA 2008; CIHI 2009; Wang et al 1997

Comparisons WHO 2007; Elvers & Rosn 1997; CIHI 2009

Disaggregation GDDS 2003; HMN 2008; AHIMA 2008

Periodicity GDDS 2003; HMN 2008; Batini et al 2009 Representative GDDS 2003; Lewin et al 2010; HMN 2008

Security WPRO 2003; HMN 2008; Wang et al 1997

Comprehensiveness Elvers & Rosn 1997; AHIMA 2008 Interpretability Brackstone 1999; Wang et al 1997

Usability CIHI 2009; Wang et al 1997

Adequacy WHO 2004 Adjustments HMN 2008 Appropriate Lewin et al 2010 Believability Wang et al 1997 Coherence Brackstone 1999 Collection method HMN 2008 Confidentiality GDDS 2003 Coverage WHO 2007 Currency AHIMA 2008 Definition AHIMA 2008

Legible (readable) WPRO 2003

Objectivity Wang et al 1997 Precision AHIMA 2008 Reliability IMF 2006 Reputation Wang et al 1997 Serviceability IMF 2006 Usefulness WHO 2003

Out of the 31 dimensions of quality identified in this literature review; accuracy, timeliness and consistency were mentioned by a number of different authors and organisations, reflecting their elevated status in assessing data quality. While many authors support the continued use of accuracy as a single measure of quality, Tayi and Ballou (1998) highlight the limitations of ‘accuracy’: as data may be accurate, but unfit for use if untimely. Interestingly, only a limited number of authors (Brackstone 1999; CIHI 2009; AHIMA 2008; Wang et al 1997) mention the

Page | 40 component of relevance when discussing quality. All raise the question of whether the data is relevant to topical policy issues and adequately meeting the needs of users, or, as Brackstone (1999: 3) asks, if agencies are still counting ‘buggy whips’. The question of relevance seems, on face value, to be an important one to ask. However, as explicitly discussed by Elvers and Rosn (1997) and implicitly inferred by other authors and agencies in their exclusion of the dimension; relevance is not an intrinsic property of statistics. While some data may be highly relevant to certain users, for others it may have no value at all, due to their differing interests. Only users can decide the relevance of information (Elvers & Rosn 1997), and as such, it offers little practical guidance on quality assessment. Overall, despite the plethora of research on the role of quality and the need for thorough quality assessments; it remains difficult to define the concept of ‘quality’ and best describe how to assess data quality.