Overview of steps

The methodology selected for this process consisted of several iterative and successive steps over a span of 12 months, starting in December 2013. The methodology outlined above was also approved by the Ministry Management Board of the Ministry for Health in early December 2013 and was discussed and endorsed during the PhD upgrade panel session for the author on 14th May 2014. The ‘traditions’ used (qualitative or quantitative) at each step are outlined in italics. The final step was the production of Malta’s first health system performance assessment report.

Box 5.1: Methodological process for development of Malta’s HSPA

1. Setting up the National Expert Working Group (NEWG).

2. Analysing current international health system performance frameworks (literature review) and testing them for their appropriateness in the Maltese health care setting – Qualitative.

2a. Examination of performance frameworks from OECD, WHO, ECHI, Commonwealth Fund (USA), Canada, Australia, various EU member states. 2b. Comparative analysis of domains/dimensions and indicators contained in

each framework.

3. Developing a draft framework for Malta (Roberts, et al., 2008) that would clearly distinguish drivers (e.g. funding mechanisms), intermediate goals (e.g. quality of services) and goals of the health system (e.g. health outcomes) – Qualitative.

4. Extracting, filtering and mapping of indicators to the draft framework - qualitative with element of quantitative.

4a. Extracting of indicators from national vertical strategies.

4b. Filtering and cleaning of initial list of indicators to produce candidate indicators.

5. Short-listing of indicators - quantitative with element of qualitative.

6. Identification of gaps in the list of indicators and mapping of areas were no data is available - quantitative with element of qualitative.

7. Review and finalisation of framework model after identification and mapping of final list of indicators - qualitative with element of quantitative. 8. Data collection and analysis - production of first iteration of results.

8a. Giving a weight to each indicator using two reviewers for each indicator working independently – qualitative.

8b. Assigning owner and data collection and analysis of HSPA list of indicators, with display of results for each indicator – quantitative.

8c. Scoring each indicator in terms of time trend and comparison with international benchmarks – quantitative with element of qualitative. 8d. Scoring of each dimension – quantitative.

5.5 Results

The methods and ensuing results are presented contemporaneously over the rest of the chapter.

5.5.1 Step 1: Setting up of the National Expert Working Group

A National Expert Working Group (NEWG) was set up to oversee the development of the HSPA Framework. This group was chaired by the then a/Chief Medical Officer and consisted of national health policy experts and public health consultants, numbering eight in all, including the Chair, senior management, public and health policy expertise, European policy expertise, expertise in epidemiology and statistics as well as administrative and ICT support. A list of the members is provided in Appendix 4. The author formed part of this group and carried out most of the scientific and technical work and research on behalf of the NEWG. This served as the author’s original research contribution for this thesis.

Whilst the methodology and results described in this chapter pertain to the whole HSPA developmental process, in practice, this methodology and the ensuing results were devised, developed, guided and written up by the author, again as part of his membership on the NEWG. The NEWG served as a scientific and technical forum to discuss, debate, reach consensus and verify or scrutinise the process. This was essential, since such a methodological process cannot and indeed, should not be carried out by a sole researcher working on his or her own. As we have seen from the literature review and document analysis, this was the case for the development of HSPAs in most jurisdictions (Perić, Hofmarcher-Holzhacker & Simon (2017).

The first task was to establish the methodology to be adopted (as described above), including the sequence and content of the work processes required throughout this task. A timeline with key intervention points was established, spanning around 6 to 8 months, from December 2013 to September 2014. Areas of action were agreed amongst members of the group and the wider network of collaborators was also agreed upon. The next step was defining the terms of reference of the NEWG, including the objectives, procedures to be followed and deliverables at the end of this task. The

The main objectives of the NEWG were:

1. To develop a set of national indicators for the Maltese Health Systems Strategy.

2. To develop a national Health Performance Assessment Framework.

3. To draft Malta’s first HSPA report.

Furthermore, this process was also expected to instigate and sustain a process of policy dialogue regarding performance assessment of the health system as a whole and in specific areas of the health system. This formed part of the policy cycle that was to be developed as part of the implementation of the NHSS.

To ensure external scrutiny by an experienced third party, the Ministry for Health commissioned a WHO expert to assist the NEWG in the development of the framework and to act as an external assessor of the process and its deliverables. This expert was invited to attend the first meetings of the NEWG in mid-December 2013.

5.5.2 Step 2: Analysing current international HSPA frameworks and testing them for the Maltese health care setting.

The first task of the OWG was to review the experience of other countries in devising HSPA frameworks and ascertain whether any of these are appropriate for Malta’s needs and requirements.

5.5.2.1 Step 2a: Examination of international HSPA frameworks

The first task that was carried out was to identify and examine the various performance assessment frameworks in place internationally, especially the more established ones and those replicated by several different countries.

The purpose of this task was to review the frameworks that were already in place with a view of determining which framework or aspects of particular frameworks would be applicable to the Maltese context. This then informed the next stage, where a draft framework for Malta was developed.

The author’s role at this stage consisted of carrying out a literature review and document analysis as detailed in Chapters 2 and 4 and providing the NEWG with the relevant findings. Each framework was scrutinised for congruence to the local context, for its conceptual outline and for its content, in particular its domains or dimensions. After an extensive review of the literature, the author, together with the NEWG, considered several frameworks that may be pertinent for Malta. These included frameworks pertaining to OECD, WHO, the Commonwealth Fund in the United States, the Ontario model of Canada, the framework used in some states in Australia, and those used in some EU member states including Estonia, Portugal and the Netherlands.

The NEWG also reviewed the European Core Health Indicators project, which although not a framework in itself, was deemed relevant due to its collection of the most commonly used indicators in the European Union. An illustrative depiction of a few of the frameworks examined is provided in Appendix 5.

5.5.2.2 Step 2b: Comparative analysis of dimensions & indicators in each framework

The same literature search identified the domains or dimensions present within each framework described above. The analysis consisted of counting the occurrences of each dimension within each framework, as well as comparing these dimensions across various frameworks, looking for commonalities, divergences and reasons for inclusion.

As explained in the literature review, these dimensions provide the building blocks of each framework, upon which the indicators would be mapped. Those dimensions most relevant to the Maltese health system were discussed in detail and chosen for inclusion into the draft framework for Malta. The indicators contained within each dimension were also noted. Table 5.1 was compiled by the author to show the various domains/dimensions for several frameworks. This was used by the NEWG for this stage of the project.

173

Table 5.1 Domains/dimensions of international HSPA Frameworks

Dimension