2.4.1 Why a systematic review?
Systematic reviews have held a privileged position in the hierarchy of evidenced based research (Greenhalgh et al. 2018; Moher et al. 2009). However, some have questioned whether explicit emphasis on auditability, reproducibility and slavish adherence to the mechanics of process have led to the production of reviews while technically excellent, present restrictive narrow aggregates of results of limited value to the body of knowledge (Greenhalgh et al. 2018;
Ioannidis, 2016). Instead, narrative reviews are proffered as an alternative means to review and interpret the literature by using purposeful academic judgement to determine relevance of inclusion of literature for review (Greenhalgh et al. 2018; MacLure 2005). However, several criticisms are levied at narrative reviews and include: risk of researcher bias; intentionally selecting literature to support a certain line of argument; purposively omitting counter-viewpoints or inadvertently omitting salient evidence because of researchers limited grasp of the literature (Grant and Booth, 2009; Greenhalgh et al. 2018). The latter consideration is a risk for novice researchers who might have a limited knowledge of the field of inquiry (Greenhalgh et al. 2018). Given that this review was undertaken by a neophyte researcher it was considered that undertaking a systematic review rather than a narrative review was preferred to minimise the risk inadvertently omitting literature and introducing bias.
2.4.2 Conduct of the review
In addition to limitations of the current evidence and review limitations already described in paper one (p. 626) it is important to consider the potential risk of bias relating to choices made during the conduct of the review. Four choices made relating to the conduct of the review are now deliberated: data sources; searching of grey literature; secondary searching of reference lists; and the use of a single screener.
2.4.2.1 Data sources
The four databases were chosen because of their ability to capture work across a variety of disciplinary perspectives reflecting multi-disciplinary working within palliative care. Medline and CINAHL capture evidence from medical and nursing peer-reviewed literature whilst Psych INFO and Behavioural Sciences data bases capture evidence from behavioural and mental health disciplines. Moreover, the NHS Knowledge Network enables these four data bases to be
55
searched simultaneous with removal of duplicates. This meant that the search was easily executed and incurred a positive time benefit and that the risk of manual errors during data management of multiple databases was minimised.
2.4.2.2 Grey literature
Searching of grey literature was not performed. Searching grey literature is advocated on the basis that studies with negative findings are less likely to be published in peer review journals and thus minimises the risk of publication bias (Hopewell et al. 2007). However, no definitive guidance exists for systematically searching for grey literature. The process is readily acknowledged as difficult with challenges relating to: selection of data repositories; ensuring completeness of search and issues around determination of data saturation; replicability of search; completeness of data and data extraction; time constraints; differentiation of evidence synthesis from primary studies; and confirming ethical compliance of studies (Adams et al.
2016). Moreover, Hartling et al.’s (2017) recent study questions the efficacy of searching grey literature for inclusion. Their recent cross-sectional examination of 129 reviews concluded that while most systematic reviews searched for unpublished studies only 6% of reviews included unpublished studies. Importantly, no significant change in the findings of the reviews occurred following removal of grey literature. The authors conclude that researchers should be reassured that if they chose to limit their searches in the interest of efficiency that this is unlikely to have impacted of the findings of their review. Admittedly, Hartling et al.’s (2017) study focussed on meta-analysis rather than heterogeneous data included in paper one. Despite this, given the time and resource limitations of the conduct of this systematic review within a PhD study the choice not to include grey literature while pragmatic was unlikely to have impacted on the review’s findings.
2.4.2.3 Secondary searching of reference lists
Secondary searching of reference lists was not undertaken. Primarily this was not adopted because the review sought to understand how distress was conceptualised in the literature with papers identified using the search term distress. Inclusion of papers through secondary searching would have shifted the lens of examination potentially compromising the primary aim of the study and thus limiting the value of the synthesis. Additionally, I consider there is a fundamental disconnect in the use of secondary searching within a systematic review for reasons which I will now explain.
56
Horsely et al.’s (2011) Cochrane review investigated the effectiveness of secondary searching of reference lists for the identification of additional relevant studies for inclusion in systematic reviews. The review team defined effectiveness as the proportion of relevant studies identified by review authors solely by checking reference lists (p.1). Horsely et al.’s review revealed significant disparities in the effectiveness of secondary searching of reference lists and reported varying yields of additional included studies ranging between 2.5% to 42.7%. From studies included in Horsely et al.’s review Greenhalgh and Peacock’s (2005) systematic review documented the highest yield of additional included studies. Greenhalgh et al. used a snow balling technique during secondary searching meaning that they obtained additional references from newly included studies in a spiralling iterative manner. Greenhalgh et al.’s method required additional time and resources costs and relied on experiential knowledge and judgement by a leading academic in the field (Horsely et al. 2011). In Greenhalgh et al.’s review given that first, the use of secondary searching was predicated on researcher expertise and second, that a significant number of additional included studies were obtained by secondary searching methods it could be argued that the final review product was actually narrative review and that use of the term systematic review was a misnomer. Thus, I assert that my earlier analysis in section 2.4.1 explaining why I excluded the use of narrative review in the conduct of paper one similarly justifies why secondary searching and snowballing of reference lists was not undertaken.
2.4.2.4 Single screener
In this review only one author screened titles and abstracts. Bias could have been introduced by my pre-conceptions in determining whether a study was eligible for inclusion during screening of titles and abstracts and subsequent full text papers (Centre for Reviews and Dissemination 2009). Using a second researcher could have minimised this risk of bias, additionally it would have provided a further check to minimise inadvertent human error during subsequent data management (Centre for Reviews and Dissemination 2009). However, there are considerable time and resource constraints involved in screening titles and abstracts and subsequent full text papers (Higgins and Green 2008). There was no specific funding available for this PhD to engage someone to do this role. An alternative strategy would have been to enlist the help of a fellow PhD student. However, my position as a part-time PhD student based in a remote and rural setting meant that my interaction with fellow PhD students at this early stage of the research process was somewhat limited. I was also keenly aware that my fellow peers had significant constraints on their time also. I thus elected not to employ an additional screener and accept this as a limitation of the review. While only a third of papers were quality
57
appraised by my two fellow authors (my PhD supervisors) high levels of agreement had occurred during the quality appraisal process.