Content of frameworks

Each framework inevitably comprises the various dimensions or domains. The way they are depicted visually is important as it portrays the importance and relevance of the dimensions and their relationship to the objectives of the health system the model represents.

2.7.1 Outline of dimensions and their significance

The identification of the dimensions or domains within a framework is the second or intermediary step to finalising the HSPA framework. After determination of the goals, objectives and priorities of the health system and following the elucidation of the underlying conceptual model that would underpin the framework, the next step is to identify and elaborate on the various domains or dimensions that constitute the main body of the framework.

These dimensions are derived from the original strategic objectives of the health system in that they represent what the health system ‘stands for.’ Whilst in themselves they do not provide any purposeful information, their purpose is to encapsulate the various indicators that measure health system performance and hence, provide structure and form to the framework. They also ensure as comprehensive an approach as possible to include all those indicators that are of relevance for that particular health system.

2.7.2 Input or structures domains

These domains usually pertain to aspects of the health system that deal with human, capital and financial resources. Hence, any measures linked to human capital, infrastructure, equipment and beds, financial resources and the proper allocation of these resources are linked to these domains. Health system design, policies, organisational arrangements, stewardship (including leadership) and ICT systems are also regarded as domains pertaining to the inputs side of the equation.

2.7.3 Process domains

quality of care, responsiveness and choice, including patient centeredness, supply chain factors, innovation and patient safety. These domains are sometimes grouped under intermediate outcomes or goals.

2.7.4 Outcome domains

These domains deal with measuring the health outcomes of the health system and therefore include health status measures (usually at population level), patient or consumer satisfaction indices, financial and social risk protection, value for money and sustainability, as well as overall health system responsiveness.

2.7.5 Domains as reflections of the health systems they measure

As can be seen from the previous sections, the domains provide a reflection of what the health system stands for, or reflect the overriding priorities and strategic objectives of the health system. Whilst most of these domains are common to all HSPA frameworks, variations do occur since each health system has its unique characteristics and goals. Hence, a framework for a developing country may place more emphasis upon infrastructure, equity and access and capacity, whilst that of a mature health system would focus primarily upon sustainability, efficiency and quality of care. The figure in Appendix 1 is taken from work undertaken by Arah et al. (2006) and shows the various dimensions or domains within the different HSPA Frameworks.

As part of the exercise to develop a HSPA framework for Malta, the author identified the framework presented in Figure 2.5 as a template on which to build upon. This was taken from the recent work carried out by Estonian colleagues (Sotsiaal Ministerium, 2009) in the development of Estonia’s HSPA framework and will be mentioned again in the methodology part of this chapter.

External context

Demographic Economic Legal and regulatory

Levers Intermediate goals Goals

Stewardship and organisational arrangements

Equity in access

and coverage Health

Resource generation and allocation Responsiveness and choice Financial risk protection

Service provision Efficiency

Financing Quality and

effectiveness

Consumer satisfaction

Epidemiology Technical Political Sociocultural

Figure 2.5: Conceptual model of Estonia’s HSPA Framework

Source: World HealthOrganisation, Regional Office for Europe (2010b)

2.7.6 Indicators

The generation, collection and analysis of indicators represents the last step in the methodology used to develop HSPA frameworks and constitutes the ‘core business,’ so to say, of all frameworks. The literature describes a myriad of methods in the identification, definition, classification, validation, collection and analysis of indicators. The purpose of this section is not to repeat what is already available in the literature but to inform the process that could be used in the formulation and validation of indicators relevant to the Malta model.

Klazinga (2001) contended that information captured by an indicator is primarily used in processes of monitoring (control) and evaluation (planning and improvement / change) and explained that the process for development of indicators should always start with the question: ‘Who wants this indicator to do what, in relation to whom?’ This is required to ensure methodological rigour.

from the international literature and then mapping these indicators onto the domains referred to above. Unfortunately, this method is principally flawed, since the identified indicators may not be appropriate or relevant to the priorities and goals of the health system they are meant to assess. Ideally, indicators are generated de novo, seeking to identify those indicators or measures that would provide a direct or indirect (proxy) measure of the domains within the framework, assuming of course that these domains have already been purposefully chosen to reflect the objectives and priorities of the health system.

In the early development days of HSPA frameworks, Rubin, Pronovost and Diette (2001) outlined in detail the methodology for the development and testing of indicators. They contended that measures need to be meaningful, scientifically sound, generalisable and interpretable and went on to describe a 7-step process to develop a robust measure, including testing for validity and reliability. A scoring method, based on the RAND Appropriateness Method, was then devised to rate/score draft indicators, using 5 to 7 independent raters (to ensure inter-rater reliability). The next stage involved drawing up of the specifications for each indicator and then carrying out preliminary testing for reliability and validity, usually through piloting. The above described methodology outlines the most common process used to create indicators by most studies in the decade that followed.

Kristensen, Mainz and Bartels (2009) used a three-phase approach towards developing safety of care indicators for the ‘Safety Improvement for Patients in Europe’ project. These phases were classified as the Planning Phase (choice of area of study, selection of priorities and establishment of expert team), the Development Phase (review of existing evidence and rating and mapping of indicators) and the last, the Testing Phase (validation of indicators). In this last phase, indicators were scored for relevance and appropriateness (Score 1-9), validity and reliability (Score 1-9) and feasibility (Score 1-9). This scoring methodology can be found in many other studies and is also primarily based upon the RAND Appropriateness methodology (Fitch, et al., 2001).

Perera, Dowell and Crampton (2012) provides an elaboration of previous methods to identify indicators using what they term as the Systematic Indicator Development

for the purpose they have been created and the information generated from the results needs to be correctly interpreted. They contend that indicators are not always ‘axiomatically good’ since indicators developed for one purpose may not be appropriate for a different application. Sound judgments and interpretations are required to assure that indicators are not technically flawed, unreliable or worse, create controversy and perverse interpretations. As with other authors, the initial work was based on a literature review and interviews with policy makers, planners, providers and clinicians. Similar to other techniques employed by other authors, they produced a 6-step iterative process as follows:

Stage 1 Prioritisation and Selection;

Stage 2 Delineation of Intent (definition of purpose); Stage 3 Determination of Implementation Requirements; Stage 4 Development of Measure Specifications;

Stage 5 Assessment of fitness for purpose; Stage 6 Development of Targets (benchmarks).

2.7.6.1 Testing indicators for appropriateness: Validity vs feasibility

All the studies reviewed give away a certain tension between choosing indicators that are scientifically robust and sound and those that are feasible and acceptable to measure. Whilst the testing of reliability and validity is considered an essential step in all of the studies, there is an equally weighted acknowledgment that an indicator also needs to be relatively easy to measure, available and acceptable to the policy makers as well as to the wider audience and not just to the scientific community. This dilemma was amply considered in the OECD quality indicators project when devising their core set of indicators (Jee & Or, 1999). Arah et al. (2003, p. 392) consider:

… that there is a trade-off between scientific objectivity and feasibility that appears to be at work in how these effectiveness indicators are conceptualized and operationalized, just as these countries and agencies

strive to appease both purists and pragmatist.

However having said this, in a study on comparing validity / reliability with feasibility, Pena et al (2010) used two rounds of the modified Delphi process to test validity versus

method (Fitch, et al., 2001). They found a surprisingly high level of association between validity and feasibility which jars with the opinion that these two attributes run counter to each other when formulating and testing indicators. Hence, it would seem that a balance could indeed be reached in order to appease both the scientific mandarins, as well as the policy makers.