Evidence review methodology
Stage 4: analysis and thematic coding (data extraction)
Theoretical/ conceptual models (n = 38
items; see Chapter 3)
Empirical papers (n = 172
items; see Chapters 4–6)
Meta-analyses (n = 4
items; see Chapters 4–6)
FIGURE 1 Preferred Reporting Items for Systematic Reviews and Meta-Analyses-style flow of information through stages 1–4 of the evidence synthesis.
TABLE 10 Number of studies in the academic literature relevant to each research question
Research question Number of relevant studiesa
1. Models and theories 38b
2.1. Morale 47 General 35 Health 12 2.2. Performance 42 General 36 Health 6 3. Antecedents 155 General 113 Health 42
a Because items were relevant to more than one research question, these do not add up to 214.
b These 38 items did not contain empirical data and were relevant to research question 1 only. However, the underpinning models and theories of the 172 empirical studies were also considered in addressing research question 1, as reported in Table 13.
NIHR Journals Library www.journalslibrary.nihr.ac.uk METHODOLOGY
literature was able to satisfy the peer-reviewed criterion for inclusion in the main evidence review, a review of good-quality practitioner materials was conducted in order to inform the practitioner outputs and address research question 4. This material was therefore separated from the main evidence review and is the subject of a separate review of practitioner material (see Appendix 7).
Analysis, thematic coding and synthesis
Academic literature
The purpose of this stage of the review was to examine the evidence and identify underlying themes in order to relate the findings from the various studies together to develop new insights into engagement within the workforce in general, and within the context of health care. Three members of the research team each took responsibility for one of the data analysis chapters of the report, which corresponded to the first three overarching research questions. In preparing their chapters the three team members iterated between the data extraction forms and the original full-text items to ensure the accurate capture of information.
Hannes and Lockwood58recommend adopting a pragmatic approach to synthesising evidence using a
process that‘is guided by the lines of action’ that can inform decision-making at clinical, policy or research
level, based on the argument of utility and the‘philosophy of pragmatism’ (p. 1633). While there is no
generally accepted approach to narrative synthesis, the approach adopted to synthesising our data largely
mirrors that suggested by Popay et al.,42who recommended that a narrative synthesis should seek to
explore (and interrogate) the relationships in the extracted data within and between studies, noting that these relationships are likely to emerge between characteristics of individual studies and between findings of different studies. It is at this stage that the synthesis should begin to account for the heterogeneity of the data (including types of intervention; context; sample; qualitative, quantitative and mixed methods approaches). The narrative should thus provide insights into what outcomes are attributable to particular interventions, or how conceptual frameworks can explain observed variations. The approach taken by the research team to extracting data for specific research questions and their corresponding chapters is shown in Table 11.
TABLE 11 Approaches to data extraction and analysis for research questions 1–3
Research question Specific approach to data extraction
1 l Extraction of information relating to the definition, measurement and theorisation of engagement from all the studies included in the evidence synthesis for research questions 2 and 3: a total of 172 papers
l Review of relevant information from literature reviews and conceptual papers focused on defining engagement that were identified in the second stage of the data extraction process but that either did not contain empirical data or contained empirical data that did not meet the quality threshold and so were excluded from the data extraction for research questions 2 and 3: a total of 38 papers
l Consultation of three recent academic books focusing on engagement.16,59,60
The research team identified these as being the only academic books with an exclusive focus on engagement
l Consultation of further conceptual articles focusing on defining engagement that were known among the research team or that were identified through a snowballing approach
2 l Extraction of data from original empirical papers that met the quality and relevance thresholds. A total of 89 studies. See Tables 14–19
l Summary overview of relevant meta-analyses
3 l Extraction of data from original empirical papers that met the quality and relevance thresholds. A total of 155 studies. See Tables 20–25
The project team worked collaboratively throughout these processes to identify and develop emerging themes in the data. Discussions were held to identify similarities and differences between study findings, and explore conceptual and methodological issues. The approach involved initial coding and grouping of data into clusters using descriptive rather than analytic labels in the first instance, to avoid closing themes down prematurely. The approach showed that, while the academic literature does tend to weigh
towards the idea of engagement as a psychological state, there are other emerging‘narratives’ in the data
as well, including, for example, the idea of engagement as managerial practice. Through team discussions these initial labels were developed and refined as more data were coded to reflect critical assessment and comparison between and within studies, and then between and among clusters of studies as these
expanded. This critical approach process ensured that the inclusion criteria of quality, relevance and credibility were constantly revisited and consistently applied throughout.
Grey literature
The purpose of the grey literature review was to try to include any relevant materials in this evidence synthesis to enhance rigour and overcome bias and, specifically to address research question 4, to consider what materials and tools from this wider resource might be of relevance to practitioners in the health-care context. In the end, only six sources of relevant, good-quality evidence were identified, from which 14 items describing various tools and resources were obtained. Analysis of these materials identified a number of important themes linked to engagement, including meaningfulness, senior manager effectiveness, perception of line manager, appraisals and employee voice. Although there were broad similarities between the overall themes in the academic and the grey literature concerning engagement, the review of grey material (see Appendix 7) suggests that the practitioner material focuses more on wider managerial issues (including performance management and training) rather than on psychological factors of engagement.
Reporting
The aim of this project is to summarise the evidence base on employee engagement in the form of an evidence synthesis and to make this evidence base more accessible within the NHS by disseminating findings about effective interventions, tools and resources. The dissemination strategy for the research has two strands: first, in the form of this report, which documents the overall approach and findings of the project, and, second, in the form of a series of practitioner outputs of direct relevance to NHS managers. The aim has been to ensure that these practitioner outputs are based upon and reflect the findings of a systematic, replicable and credible synthesis of the data. The practitioner outputs are set out in
Appendices 7–16.
Summary
In this chapter, we have described the methodological approach underpinning this evidence synthesis.
Following the recommendations of Briner,46we adopted a narrative approach in five stages (planning,
locating studies, evaluating material, analysis and coding, reporting).
In collaboration with the project advisory group, we refined the project protocol, detailed research questions and search terms, and conducted a series of pilot searches in order to help refine and focus our search strategy. The full search of academic literature was conducted using five databases and a wide range of grey sources. A total of 5771 studies were included in the preliminary sifting exercise whereby the abstract and/or title for each item was reviewed by two or in some cases three members of the research team.
NIHR Journals Library www.journalslibrary.nihr.ac.uk METHODOLOGY
The application of quality and relevance criteria along with the removal of non-peer-reviewed items led to the inclusion of 172 empirical articles, four meta-analyses and 38 theoretical papers in the final data extraction exercise. Items that were published in the English language after 1990, and that met the appropriate quality and relevance thresholds for the type of study, were included in the evidence synthesis. Items identified from six sources through searching the grey literature are included in the practitioner-oriented materials arising from this project.
In the next chapter, we examine the results of the evidence synthesis in relation to research question 1: what is engagement?