Methods in developing HSPA frameworks

This section outlines the principal methodologies used to develop the various frameworks that have been elucidated upon in the previous sections. Although there is a common ultimate purpose for the development and application of HSPA frameworks, there is a multiplicity of methods in how each framework was developed. The comprehension of the methods employed was deemed important for the author to then determine and inform on the best, or rather most appropriate, method to be used for the development of Malta’s first performance assessment framework. Each model was formulated according to the contextual needs of its health system, even though there was always a common thread between most frameworks. Although not explicitly or deliberately demonstrated, the literature consistently shows three principle iterative steps in formulating a HSPA framework, with one iteration building on the previous.

These could be viewed as:

1. Determination of the model to be used.

2. Definition of the domains or dimensions that would constitute the framework.

3. Identification and generation of the performance indicators that would populate the framework.

2.5.1 Developing a model

Most processes start off with an in-depth review of existing literature and a policy analysis to identify the main policy and strategy documents and priorities that the framework would cover. This approach is common to almost all frameworks developed at national level. The Dutch HSPA framework (Ten Asbroek, et al., 2004, p. i67) first considered the development of the conceptual model and then selected the different indicator areas. The Dutch modelling emphasised that the framework must

be ‘coherently balanced, covering various performance dimensions such as

effectiveness, efficiency, quality, and equity.’ Also, the framework must be inclusive

of all areas of the health system, including health care and public health domains. Finally, a good framework needs to ensure that it is measuring the impact of the health

other proponentsargue that a health system encompasses areas that are beyond the direct reach or influence of health care organisations (Kelley, Arispe & Holmes, 2006). This argument is important when we come to discuss attributability as a general limitation of HSPA frameworks.

2.5.2 Testing the model

Once a draft model is developed, this is invariably tested. El Turabi et al (2011) used a hybrid methodology of combining the Logic Model with the Balanced Scorecard to draw up their initial model. This was then validated using interviews and focus groups, followed by piloting of the framework to test for acceptability, suitability and feasibility. The wider applicability of the framework could then be gauged by implementing the framework system-wide. El Turabi et al. (2011) also utilised an approach that is commonly used in developing HSPA frameworks, that of discussion based consensus decision taking, used primarily amongst experts with high technical knowledge of the subject and then tested through surveys or focus groups/modified Delphi Technique to obtain a wider understanding and ‘buy-in’ from relevant stakeholders.

Mainz (2003) used a similar approach in developing a ‘state of the art’ methodology.

His paper outlined two main stages – the planning phase, which included the identification of the clinical areas to be assessed and the ‘measurement team or experts’, and the ‘development phase,’ where the policy and documentation analysis is carried out, followed by the identification of indicators. Pilot testing then followed as a final step.

Together with colleagues, Mainz used this same methodology to develop the HSPA framework for the Danish health system. In their paper, Mainz et al. (2004) outlined their work on the National Indicator Project in Denmark, which initially covered specific target diseases, such as stroke, lung cancer and schizophrenia. The steps undertaken for this project were similar to other iterations in that they first identified the problem and set the priority areas, then developed the indicator base, collected and analysed the data and provided interpretive guidance to the policy makers and managers. This data was also audited and disseminated to the public.

2.5.3 Scientific basis of methodology

Whilst the methods used by various experts share a common thread, arguments have been raised from early on that some methods are unsatisfactory and not robust enough to pass the test of scientific rigour. This is mainly attributed to two overriding factors. The first is due to the nature of the framework itself, in an attempt to ‘trap’ all the key attributes that a health system should have, including quality of care, health inequalities, sustainability, efficiency measures and political acceptance. Whilst these attributes are undoubtedly important and relevant, they are not enough to ensure a comprehensive overview of the true objectives of a health system. Another common pitfall is for experts to start the process by simply choosing those indicators that are either already available or easily attainable, with the obvious conundrum of excluding other more relevant dimensions of the health system. Murray and Frenk (2000) gave an early warning shot across our bows in the primordial days of the early 2000s, where they argued that each process to develop a HSPA framework should commence with asking the most basic of questions: What are health systems for? The answer to this question will present the authors with the key goals and strategic objectives of the health system they wish to evaluate and this then paves the way to develop a robust and scientifically sound health system performance assessment framework.

Nonetheless, there is consensus that there are a number of key principles which should be respected, irrespective of which method is embarked upon. These are:

• HSPA, as the name implies, should focus upon health systems as a whole and not on the components parts of a health system.

• Whilst outcome measures are better positioned to mirror performance, structure and process indicators are also valuable.

• HSPA should form part of a regular process to measure performance and devise policy as implied within the policy cycle of a HSPA. • Although each country or health system should develop its own HSPA,

its structure and content should be comparable with that of other countries and health systems.