Definitions

Given the wide interpretation promulgated by various HSPA models, it is important to first review the definitions used in the documents that were analysed to gauge whether authors are consistent in their usage of the concept of HSPA and performance assessment in general. WHO is probably the main authority on defining HSPAs.2

It uses specific key terms to define a HSPA – country; the process of monitoring,

evaluating, communicating and reviewing; health system; indicators; goals and

strategies. Despite a lack of a standardised approach towards HSPA, these terms are

invariably included in all HSPAs produced to date and guide the construction and implementation of HSPAs across health systems. WHO also contends that a mature HSPA has a number of key attributes. It should be regular, systematic and transparent, and a whole system approach should be adopted. It should not be linked to any reform and health strategy agenda and there should be a constant revision of targets and priorities.3 However, this is not consistent with many other WHO supported HSPA processes in member states where HSPA was always closely associated with the country’s national policy agenda and health strategy (Marra, 2017). On the other hand, WHO4 continues to state that ‘the challenge lies in aligning performance assessment and accountability based on strategy, by cascading performance indicators at the macro, meso and micro levels while recognizing and adapting to the different levels

of responsibility.’ This is also the position taken by the EU and OECD.

In linking HSPA with health policy, most HSPA contenders choose not to use WHO’s rather narrow definition of health policy, where WHO contends that policy is generated through initiation and delivery of health care, but define health policies as ‘decisions, plans and actions that are undertaken to achieve specific health goals

within a society,’ irrespective of where these policies originate from.5

4.5.1.2 Methods

The document review uncovered an insipidly uniform approach in the development and implementation of HSPAs in various countries, with a few exceptions. This is rather surprising and inconsistent with individual countries’ traditional but stoic reticence to develop anything close to a common European or WHO-wide or OECD- wide HSPA framework and methodology. The document analysis also informed the methodological steps adopted for the development of Malta’s HSPA, which is congruent with many of the methodologies adopted by other countries. The analysis also confirmed that international experts are invariably brought in and consulted at the

3 _{123 WHO (Europe). The European health report 2009. 2009 pg. 142.} 4 _{123 WHO (Europe). The European health report 2009. 2009; pg. 146.}

initial stage. This was the case for Georgia, Armenia and Belgium, where several international and WHO experts were enlisted to assist in developing their respective HSPA.6,7,8 International input is also deemed important to provide oversight and technical support, especially in ascertaining an acceptable level of scientific rigour.9

Most HSPA frameworks are based upon the strategic objectives of the national health system, since the model needs to reflect the country’s needs and objectives. This was the case for the Netherlands where their model was based upon their principles of quality of care, accessibility, and affordability.10 The EU Expert Panel on Effective Ways of Investing in Health reiterate that a conceptual framework is essential to delineate the boundaries of the health system to be assessed, as well as to inform a set of dimensions and allow for the selection of appropriate performance indicators to proceed. This Expert Panel also states that ‘a conceptual model would help clarify the relationship between health system inputs, processes, outputs and impacts, as well as contextual factors influencing policy change mechanisms and the ways in which policy

changes should be prioritized and evaluated.’11

The phase of extracting and populating each dimension with performance indicators then follows. OECD, WHO and ECHI have all developed a set of criteria for choosing indicators and these criteria are increasingly being used by countries for their own identification of national indicators. Many of the indicator sets developed by international agencies are repeated iterations on the same theme and proponents in performance assessment are gradually working towards congruence and standardisation across regional and national HSPAs. There is, in fact, significant overlap in the indicator sets used by the EU (ECHI), OECD (Health at a Glance), WHO (Health for All), the Millennium Development Goals, the Parma Declaration on Environment and Health and the Comprehensive Global Monitoring Framework and Targets for the Prevention and Control of Non-Communicable Diseases.12

6 _{152 WHO (Europe). Georgia Health System Performance Assessment, 2009. 2010 pg. 15.} 7 _{150 WHO (Europe). Armenia Health System Performance Assessment, 2009. 2009, pg. 25.} 8 _{116 WHO (Europe). Case Studies on HSPA. 2012 pg.17.}

9 _{53 WHO (Europe). Pathways to Health System Performance Assessment. 2012, pg. 52.} 10 _{154 van den Berg M et al. The Dutch health care performance report. Health Research Policy &}

The presentation of the results is usually geared to initiate a policy dialogue of sorts with policy makers.13 _{This process, which was outlined in detail in the literature} review, was followed by several countries, such as the Netherlands, Belgium, Estonia, Turkey, Malta and Portugal (Rotar, et al., 2016). It is increasingly becoming apparent that international organisations need to work together and collaborate on HSPA in order to standardise the framework, definitions and also data sources. In its report regarding health investments, the EU Working Party on Public Health at Senior Level, stated that ‘this coordination will have a positive impact on Member States by reducing the burden of data collection and reporting, limiting duplication of efforts and

producing more consistent results.’14 However, the EU diplomatically and

understandably leaves each member state to follow its own route although in essence, all the routes are exceedingly similar and lead to the same outcome. In certain instances, such as in Portugal, the policy dialogue and policy cycle that invariably proceeded from the HSPA process formed an integral part of the methodology. The performance assessment of the Portuguese HSPA was driven by a set of policy imperatives on which subsequent policy recommendations were formulated on the advice of experts.15

The mode of presentation of results presents certain challenges. In most HSPAs, results are usually presented against a standard or predefined benchmark or are inserted into a trend analysis. However, standardised approaches are required to compare like-with-like and statistical methods are key to ensure validity and reliability of results. This is also important to understand variations in performance and to ensure that the observed variation is genuine. To this end, risk adjustment is often used to overcome the problem of attributability, to ensure that the outcomes are casually linked to the activity or care under measure, in what Donabedian called ‘attributional

validity’ (Donabedian, 1980 (b), p. 103). However, this is also not without difficulties.

Risk adjustment (or adjustment for case-mix), if performed on its own, may also give false results, depending upon the technique used for risk adjustment. This is evident, for instance, when producing performance ranking scores for regional health systems

13 _{116 WHO (Europe). Case Studies on HSPA. 2012.}

14 _{4 Working Party on Public Health at Senior Level - Report by sub-group 5: Health}

or hospitals. This is known as the ‘case-mix fallacy’ (Lilford, et al., 2004, p. 1148). On the other hand, the usefulness of comparative rankings is limited unless accompanied by a detailed understanding of mutable regional and intra-country contextual factors and even so, is primarily used by health authorities to spur a policy debate, rather than to take firm policy decisions.

4.5.1.3 Process in the development of HSPAs and the actors involved