Knowledge Based Public Health Work, Part 1

Knowledge Based Public Health Work,

Part 1

Handbook for Reviews of Published Reviews on

Interventions in the Field of Public Health

Foreword

Sweden’s National Institute of Public Health has recently been given a role as centre of competence concerning methods and strategies to foster public health. Reviews of current knowledge are an important part of this work. This paper will guide the reader through the process of compiling the results of previously published reviews on public health topics. The guide is mainly intended for people who work for the National Institute of Public Health – at regional research centres or in the central administration – and are to integrate current knowledge. It is the first in a series of reports about knowledge-based public health work. The next report will describe the step-by-step process of compiling critical reviews of primary research papers, and will be published in the spring of 2003.

After that, we plan reports that describe and discuss what knowledge-based public health work comprises, and will present and compile models for systematic planning,

implementation and follow-up within the public health sector. We will continuously collate, update and disseminate the scientific literature on the effectiveness of different types of interventions to promote health and prevent disease. This report should be seen as a draft document. The National Institute of Public Health and the authors welcome comments and questions from readers, and will incorporate changes into the next version of this handbook. Stockholm, November 2002

Gunnar Ågren Christer Hogstedt

Content

Summary ...4

Chapter 1 – Introduction ...5

Systematic reviews...5

The aim of this handbook ...6

Different work strategies within the public health sector ...7

Decisions to make when planning a review of published reviews ...8

Description of study designs...10

Chapter 2 – Formulating the review question ...13

Main components of a review question ...13

Avoid unnecessary restriction...14

Timeframe...15

Chapter 3 – Searching the literature...16

Creating a search strategy ...16

Where to search for information ...18

Reference management systems ...21

Document your search ...21

Selecting reviews ...21

Evaluating grey literature...22

Synthesis ...22

Timeframe...23

Chapter 4 – Writing up and disseminating the results ...24

The title and the authors...26

The summary ...26

The main body of the text ...26

Critical reading of the manuscript...28

Dissemination of results...29

Timeframe...29

References ...30

Appendix 1. Quality assessment protocol for review of reviews...33

Appendix 2. Fundamental search terms for articles in English...42

Sample “Review of Reviews” How can we prevent young people from taking up smoking?...46

Summary ...47 Introduction...48 Methods...49 Results...50 Discussion...53 References...60

Summary

The first part of this handbook is a step by step guide on how to conduct a systematic review of already existing reviews on public health interventions. The first step is to define the population, the interventions and the outcomes to be included in the review question. This review question is the basis for the strategy to be used in the search for published articles in databases. Databases for existing reviews and for original and review articles are described. A method for selection and quality assessment is described. An appendix to the handbook gives a protocol for the quality assessment of reviews, e.g., to assess how systematic they really are. The results of the review are then summarised in the conclusion section. Finally, the

handbook describes how the review findings can be reported and disseminated.

The second part of the handbook provides a practical example using an existing systematic review entitled How do we prevent youth from starting smoking? This review used a

systematic method of searching and judging literature on interventions used to prevent youth from starting smoking. The review reports that 47 review articles were identified. Eight of these articles provided the basis for the review. The interventions included school based, policy, mass media, and community interventions. The review concludes that the most effective way to prevent youth from starting smoking is by raising the price of cigarettes at the same time as implementing a variety of community interventions, all aimed at youth.

Chapter 1 – Introduction

Modern research generates enormous amounts of knowledge. In this deluge it can be difficult to find time to fish out the information that is relevant to one’s own field. Systematic reviews about an intervention should provide information about how effective it is, by identifying, evaluating and collating results from scientific publications and certain other sources.

Reviews are important tools when decision-makers and administrators plan programmes and allocate resources. One usually distinguishes between reviews of the literature, research overviews, and systematic reviews.

Reviews of the literature are non-systematic compilations of the published literature; they are comprised of a subjective selection of articles, and present equally subjective evaluations. Literature reviews are often compiled by experienced researchers in the field: this is both a strength and a weakness. The strength lies in the authors’ thorough knowledge of the research field. The weakness is that the authors sometimes have preconceived notions and may

overestimate the value of some articles.

Systematic reviews are systematic compilations of the scientific literature. They include both articles published in scientific journals and the so-called “grey literature”. Grey literature includes reports and documents that have not been published in peer reviewed journals or scientific publications. This grey zone includes publications directed toward a broader audience, for instance conference reports. Systematic reviews are compiled in a structured way: this means that material is collected according to a defined scheme in order to avoid bias and prevent distortion of conclusions, and also that the process is described clearly, so it can be traced, evaluated and repeated step by step.

Research overviews differ from systematic reviews in that they include only published, peer-reviewed scientific papers (no grey literature) within a specific research field.

Most of the methods used to compile systematic reviews originate from the Cochrane

Collaboration, and from the evidence-based medicine that has sprung forth as a consequence of their work. The Cochrane Collaboration, which started in 1993, is an organisation of scientists and practitioners who have taken upon themselves the task of compiling systematic state-of-the-art reviews, mainly of medical technology and therapy. The organisation was founded on the initiative of the English epidemiologist Archie Cochrane with the aim of compiling and updating systematic reviews of interventions within the health care sector, and to make the information available to physicians, policy-makers and patients.

The Cochrane Collaboration is comprised of several working groups that develop methods for systematic reviews in different health fields. One of these working groups, Cochrane Health Promotion and Public Health Field, has been focusing especially on methods for systematic review of health promotion strategies

(http://www.vichealth.vic.gov.au/cochrane/welcome/index.htm).

The Swedish Council on Technology Assessment in Health Care (SBU) is responsible for compiling the scientific knowledge needed – both by the government and by the health authorities – to evaluate health care. The working method used by SBU is founded on the systematic review process developed within the Cochrane Collaboration. SBU has published a number of reports since its inception in 1989. Prominent among these are the reports

“Preventing cardiovascular disease through population-based programs” (1997) and “Methods for smoking cessation” (1998).

The aim of this handbook

The first part of this handbook describes a method for collating already published reviews on public health work, i.e., systematic reviews of articles published in scientific journals and books and “grey literature”. These reviews are meant to address public health work in general, i.e., efforts directed towards entire populations and preferably co-ordinated outside the health care sector, and are meant to be used by public health workers and policy-makers in research and in practice.

The handbook is intended primarily as a guide for those who are compiling knowledge for the National Institute of Public Health, both those at regional research centres and central

administrators. In working with this handbook we have drawn on the expertise of

international groups who work with systematic reviews. Agreement with other handbooks can be seen from the reference list, but some descriptions of methods deserve special mention. The most important are:

• Center for Disease Control’s Guide to Community Preventive Services (Zaza et al. 2000)

• Methodological Problems in Constructing the Evidence Base in Public Health (Kelly et al. 2002)

• Undertaking Systematic Reviews of Research on Effectiveness. CRD’s guidance for those carrying out or commissioning reviews (Kahn et al. 2001)

• Cochrane’s Reviewers Handbook (Clarke & Oxman 2001)

• Draft Guidelines, The Campbell Collaboration Center. (http://www.campbellcollaboration.org/Fraguidelines.html)

• An assessment of the methods and concepts used to synthesize the evidence of effectiveness in Health Promotion: A review of 17 initiatives. (Jackson et al. 2002)

• Medical Evaluation [in Swedish]. (SBU 2001)

The first part of the handbook is arranged in the following way.

In this chapter we give a brief introduction to various methods used in public health work and describe aspects of this work. We conclude the chapter with a summary of the most commonly used study designs.

In chapter 2 we describe the process of formulating the problem to be addressed in the review.

Chapter 3 will guide the reader through the next step in the process, namely how to build a search strategy, how and where to search for previously published reviews, how to assess the quality of these articles, and how to summarise the results presented there.

Finally, in chapter 4, we suggest ways to organise and synthesise the report. We also discuss ways in which the report can be disseminated.

In the appendices we first present a quality-assessment protocol for use in evaluating published reviews (Appendix 1). Then we list search terms used frequently in various public health fields (Appendix 2).

The final part of the handbook provides an example of how the first part might be

implemented. The authors chose to compile previously published reviews on prevention of tobacco use among young people.

Different work strategies within the public health sector

“Public health work” is a general term for actions aimed at improving the health of groups of people, rather than individuals, as one does in clinical health care. Public health work uses an interdisciplinary approach and is based on scientific theories from within several different scientific paradigms. This makes it difficult to define the field, and also leads to differences of opinion concerning the methods used. One obvious difference lies between the two

fundamental perspectives on public health: pathogenesis and salutogenesis. The pathogenic perspective predominates clinical health care. In brief, it is focused on disease: how to

diagnose it and how to prevent it. The salutogenic perspective is instead focused on health and how to promote it.

Disease prevention strategies are founded in the pathogenic perspective, even if they are aimed at primary prevention, i.e., actions taken before disease sets in. Strategies for disease prevention often include identification of risk factors for a disease and attempts to reduce or eliminate risk factors in vulnerable groups. These interventions often involve a specific method and the outcome is often measured in terms of reduced prevalence of the disease or reduced prevalence of factors that contribute to disease development.

Health promotion strategies spring from a salutogenic values stance (Judd et al. 2001), where the aim is to empower groups of people or entire societies; this in turn serves to strengthen people’s ability to stay healthy. Health promotion work frequently employs combinations of methods targeting different sectors of society. It is evaluated in terms of many different outcomes including not only the ultimate goals of improved health, equity and quality of life, but also the intermediate goals of the intervention itself, such as knowledge, attitudes and motivation. In health promotion work, both the outcome and the process are frequently subject to evaluation.

Many different activities based on different approaches can coexist within public health work. Below we present some aspects of public health work that we consider important for

identification of the review questions and the focus of a planned systematic review. The rationale for this subdivision is that it simplifies the procedure of evaluating literature on public health interventions. This subdivision into different aspects is not intended as a way of defining public health work in any other context.

Decisions to make when planning a review of published reviews

As mentioned earlier, public health work can be categorised as disease prevention or health promotion; it can entail one intervention, or several; it can aim for one outcome or more. In addition, the main focus of interest when evaluating the intervention can be either the out-come itself or the road toward that goal, i.e., the process. These four main dimensions can form a framework on which to classify questions about work undertaken in the field of public health.

1. The first question to ask is whether the review concerns a strategy for disease prevention or one for health promotion. Disease prevention strategies aim to reduce disease, reduce determinants of disease, or reduce risk factors for disease. Disease prevention strategies are frequently defined as actions that target specific risk groups and risk factors that are

associated with different types of risky behaviour.

The aim of health promotion strategies is often broader than that of disease prevention strategies. According to the Ottawa Charter (1986), health promotion is “the process of enabling people to increase control over, and to improve, their health”. There are several different viewpoints about what is to be considered a successful result where health promotion is concerned. Planners, economists and policy-makers need to know how cost-effective a programme is, whereas practitioners in the field need to know what effects a programme has in real life. The people targeted by the intervention want to feel included, whereas researchers are probably mainly interested in learning what a programme shows about the relation between cause and effect (Nutbeam 2000).

Health promotion activities often give results at several different levels. One outcome model, created by Don Nutbeam (2000), is presented in Table 1. In this model, health outcomes and social outcomes represent different end-points for health promotion. They are often described in terms of quality of life and equity, or mortality, morbidity and disability. Intermediate health outcomes include determinants of the ultimate outcomes. Personal habits such as physical activity are part of a healthy life style, whereas a healthy environment is made up of physical, social and economic variables that can influence health directly in addition to contributing to a healthy life style. Access to adequate health care is also a determinant of health. The health promotion outcomes are the individual, social and structural factors that can be manipulated to have an effect on determinants of health. These outcomes are the most immediate effects of health promotion programmes, e.g., knowledge and attitudes. Social action and influence describe the results of efforts to empower groups within society to take charge of their own health. Health policy and organisational practice are the results of efforts to change structural barriers to good health, for instance through legislation, regulation and ordinances.

Example: a disease prevention strategy to reduce the risk of lung cancer and heart attack might aim to reduce smoking, for instance by having company nurses give information about the detrimental effects of smoking. The outcome would be described in terms of lower

Table 1. An outcome model for health promotion

Health promotion

actions Health promotion outcomes (intervention impact measures)

Intermediate health outcomes (modifiable determinants of health)

Health and social outcomes Education Examples include: patient education, school education, media communication Health literacy Measures include: health-related knowledge, attitudes, motivation, personal skills, self-efficacy Healthy lifestyles

Measures include: tobacco use, food choices,

physical activity, alcohol and illicit drug use

Social outcomes Measures include: quality of life, independence, equity Social mobilisation Examples include: community development, group facilitation, technical advice

Social action and influence

Measures include:

community participation, community

empowerment, social norms, public opinion

Effective health service

Measure include: provision of preventive services, access to and appropriateness of health services Health outcomes Measures include: reduced mortality disability, morbidity Advocacy Examples include: lobbying, political organisation and activism

Healthy public policy and organisational practice

Measures include: policy statements, legislation, regulation, resource allocation, organisational practices

Healthy environments

Measures include: safe physical environment, supportive economic and social conditions good food supply, restricted access to tobacco and alcohol

Example: a health promotion intervention can foster a healthy lifestyle by increasing young peoples’ insight about the strategies tobacco companies use to increase tobacco use among young people. One health promotion intervention within this area might be to involve various forces in the local community, such as sports clubs, and getting them to support smoke-free leisure time or by lobbying for regulations against smoking.

2. The second question is about the number of interventions. Both disease prevention and health promotion programmes can consist of a single intervention, e.g., giving information in schools about smoking, or of several interventions all aimed at reducing tobacco use. Usually a review will be focused on a specific health effect that can be achieved through several types of interventions. Each intervention must usually be evaluated separately, though similar programmes using the same types of interventions can be compared. If the number of interventions is made clear, the review question can be formulated more precisely and the literature search can be divided up into separate parts.

3. The third question concerns the number of outcomes. As is clear from the description, health promotion interventions are often measured in terms of several outcomes, but a scientific primary publication may focus on evaluating a single outcome. Disease prevention activities generally have a single outcome measure, but here too, several outcomes are conceivable. Outcome measures can also be intermediate – for instance insight into the importance of diet for good health, improved eating habits, cholesterol levels in blood – or ultimate outcomes, such as cardiovascular disease.

If the work with the review is to go smoothly, it is important to define the types and number of outcomes, because it is difficult to evaluate multiple outcomes in the same way. A syste-matic review can choose to evaluate several outcomes, but they must then be specified. At this stage, the outcome measure(s) should be clarified and established. You should also consider carefully whether the number of outcomes should be reduced, or if material should be written up as more than one review, each covering a particular outcome.

4. The fourth question concerns whether the review will analyse the outcome or the process. Both are of importance for policy-makers and practitioners in the public health field.

Expectations concerning results are important when an intervention is being planned, but insight into how the results are achieved is equally vital. Outcome analysis measures the results of an intervention quantitatively, e.g., in terms of units of cholesterol, units of blood pressure, number of correct answers on a test, number of participants in an activity. Process analysis studies how the results were achieved: who participated, what attitude did the various participants have towards the intervention, to what extent were they involved? For the latter analysis, qualitative methods such as focus groups and in-depth interviews are often used.

Description of study designs

Most systematic reviews follow a particular study design. Below we briefly present some of the study designs that are used most frequently. For further detail, we recommend the following literature:

Example: outcomes can include morbidity, percentage of

smokers, cotinine levels, attitudes towards smoking, knowledge of the effects of smoking, changes of policy such as restrictions on sales or advertising, shifts in public opinion, smoke-free zones, self-efficacy.

• Rothman, K.J. & Greenland, S. (1998). Modern Epidemiology. Philadelphia: Lippincott-Raven Corp.

• Green, L.W. & Lewis, F. (1986). Measurement and Evaluation in Health Education and Health Promotion. Mayfield Publishing Company, California.

• Neal, J. & Liebert, R.M. (1986). Science and Behavior. An Introduction of Methods of Research. Prentice/Hall International. New Jersey.

• Starrin, B. (1997). Along the Path of Discovery. Studentlitteratur, Sweden.

Randomised controlled trials (RCT) are studies in which the investigators compare the effects of an intervention in a study group and a control group. Allocation to study and control group is random, and the study should be double-blind, that is, neither the participants nor the investigators should know who belongs to which group. Moreover, results concerning all participants should assessed, regardless of whether they have participated for the entire duration of the study or not. If the allocation to study and control group is done in a non-random fashion, or in a way that is not truly non-random (for instance if people born on a certain day are assigned to the study group, or if every second patient in a list is assigned to the study group) the study is called a controlled clinical trial (CCT).

In a cohort study, a group of people are selected (e.g., on the basis of age or occupation) and the exposure of interest is defined. Then the cohort is followed over time and the outcome is measured at the end of the study. Comparisons of the outcome for exposed and non-exposed members of the cohort can be expressed as relative risks (RR). A cohort study can be

prospective (where the cohort is followed into the future from the day the exposure is measured until the end of the study), retrospective (where the exposure is estimated

afterwards, when the outcome is known), or ambispective (in which case the study has both prospective and retrospective elements).

In before-and-after studies, the same participants are examined before and after an

intervention. Here it is often difficult to ascertain whether any differences observed indeed result from the intervention. However, in the absence of RCT or cohort studies, before-and-after studies can provide clues about an effect, for instance when changes in health policy are to be evaluated. A related study design compares two geographical regions. The time-series study (TSS) is another design, where a phenomenon is measured sequentially over time before and after an intervention. This kind of study can give information that is more reliable than that provided by before-and-after studies. Time-series analyses can identify trends and cycles. The correlation between observations within a time-series is a statistically important attribute. When a time-series design is to be used, a stable baseline value or trend must be established before the intervention.

Qualitative research is a term that can be used to describe several different study designs (just as all the study designs presented above can be called quantitative). Qualitative research provides information on and insight into how people perceive things – what meaning they attribute to various phenomena, and how they communicate their perceptions of various events and of the world around them (Hallberg 2002). The reason these studies are called qualitative is that the results are seldom expressed numerically; nor are they evaluated statistically. Instead, the investigators gather information through focus groups or in-depth interviews, or by participating as observers.

There are many different qualitative methods for systematic accumulation and compilation of data. They can be divided into three main categories: language-oriented methods, theory-generating methods, and descriptive methods (Hallberg 2002). An example is Grounded Theory, a theory-generating method that is being used more and more within public health research (e.g., Hallberg & Jansson 1996, Dellve et al. 2001, Thunberg et al. 2001). In qualitative research, the number of informants (i.e., the study population) is often small – seldom more than thirty. Data collection and data analysis are often carried out

simultaneously. The aim is to give as broad and as varied a picture as possible of the phenomenon under study. For this reason, the investigators try to get as heterogeneous a group of informants as possible. When the investigator finds that no more new information is emerging, the material is considered “saturated” and data collection ceases. Qualitative and quantitative methods are excellent complements to each other in public health research. They should not be seen as opposites. Hallberg (2002) states that although qualitative and

quantitative research use different strategies to find answers to research questions, the

investigator’s background and perspective will influence which questions are posed, and how the data are collected and analysed, regardless of whether qualitative or quantitative methods are used. Critical evaluation of a study’s reliability, accuracy and applicability is necessary in both qualitative and quantitative research.

Yet another way to categorise different types of studies is to classify them as having an “experimental” or “non-experimental” design. Only RCTs qualify as experimental studies (see above). Non-experimental studies include “quasi-experimental” designs, such as clinical controlled trials, CCT, that do not use truly random sampling. Finally, there is a category called “observational studies”. These include cohort studies, before-and-after studies and time-series studies. Another type of study design – not very frequently used for evaluation of public health interventions – is case-referent studies.

Chapter 2 – Formulating the review question

Main components of a review question

After going through the four questions described in Chapter 1 – whether the systematic review will deal with a strategy for disease prevention or one for health promotion, how many interventions (or methods) it will include, how many outcomes it will evaluate, and what type of analysis it will make (process or outcome) – you should have a fairly clear picture of the review question, but further refinement is necessary. It is important to judiciously think through the questions posed in the review; clearly framed questions will save much effort. For instance, it is much easier to locate relevant literature if the question is defined precisely. Readers also have use for a clear statement of the research questions: it will help them judge whether they are interested in the review.

The following components should be specified in the review question.

1. What people (what participants) are to be included and from what setting? 2. What interventions are to be included?

3. What outcomes are to be included? 4. What study designs are to be included?

1. The first thing to do is decide on a target population and a setting (e.g., 13- to 19-year-olds in schools or middle-aged men in workplaces where most of the employees are women).

2. Which types of interventions to include need only be specified if the main aim of the review is to examine the effects of a specific intervention, e.g., which types of educational initiatives work in a certain setting. If the main focus of the review question is “what works” (regardless of type of intervention), obviously this should be left open and no interventions should be specified beforehand.

To formulate a clearly defined question, you must have precise answers to these questions about what to include: 1. What people (what participants) and from what

setting?

2. What interventions? 3. What outcomes? 4. What study designs?

A question that defines the first main component: Which interventions are effective in preventing children and teenagers in schools from taking up smoking?

3. Outcome has also been discussed above. In addition to defining outcome measures, you should decide how to handle combinations of outcomes. Moreover, you should decide whether unintended effects should be included.

4. In deciding which study designs to include you must consider the review question and the outcome measures. Perhaps the number of study designs needs to be limited? If several different study designs are included, they must be specified and it should be made clear precisely why they have been included. This discussion emphasises the need of competence and consensus concerning the scientific stance among the authors.

Avoid unnecessary restriction

When making a review of previously published reviews, you should not be overly restrictive in defining the review question because important reviews may be missed in the literature search. For instance, if a narrow age group is defined, some reviews may be excluded. To capture all relevant reviews, the question should be broad and general.

Once all the components in a review question have been defined, they can be used to

formulate a search strategy. When the search strategy is tested, the review question may have to be modified. This is to be expected, but major post-hoc changes in the review question should be avoided. Otherwise there is a risk that interesting articles will give a completely new perspective – new, but not pertinent to the problem being addressed.

A question that defines the fourth main component: What findings emerge from before-and-after studies of changes in policy, for example restriction of access to cigarettes or designation of more smoke-free areas? A question that defines the third main component:

Which interventions are effective in increasing motivation not to smoke and fostering positive attitudes towards non-smoking?

A question that defines the second main component: Which interventions, alone or in combination, are effective in increasing knowledge about the detrimental effects of smoking?

Timeframe

It is important to invest time in formulating a review question. It is obviously not possible to say how much time you will need to spend. Everyone involved in compiling the review should discuss the formulation of the review question.

At this stage it is also worthwhile to set up a reference group for the review. The group should include a panel of experts in the field and representatives of the target group for the review, i.e., people working in the local public health sector. The task of the reference group is to help identify relevant literature – especially grey literature – and to evaluate and suggest changes in the final review of published reviews.

Chapter 3 – Searching the literature

Creating a search strategy

You must have a strategy for how you will search for pertinent research within a field – a search strategy. The strategy should specify which databases and other sources will be searched, and which search terms and search strings (i.e., search phrases) will be used. In formulating a search strategy you set out from the key components of your review question, e.g., target group, type of intervention, study design, outcome measures (see Chapter 2). The results of the first search (the pilot search) can be useful in helping you improve your search strategy. It is extremely important to make a note of the search terms used, in what combinations they are used, and which databases are searched. Without such records you will soon begin to mix or double your searches. The search strategy you finally decide on will be described in the review.

In general, the strategy and the search terms will be influenced by how you have formulated your review question. It is important to define the question (see Chapter 2), as this facilitates the formulation of a search strategy. There are pitfalls to searching only for outcomes. The

Steps in the process of searching the literature

2. Search databases specially compiled for previously published reviews (not primary articles).

3. Search databases comprised mostly of primary articles, but that also include reviews.

4. Search for more reviews in reference lists, in the grey literature, on the Internet, and in registers of ongoing research.

5. At this point, you will have a long list of references as well as abstracts or summaries of reviews.

6. On the basis of your review question, go through the “long list” and select the reviews that appear most relevant.

7. The reviews you have selected constitute the “short list”. Order complete copies of all these articles from the libraries.

8. The articles will then be read and evaluated for quality according to a standardised protocol (see Appendix 1). At this stage you may find that some articles do not fit within the framework of the review question, or are not of sufficiently good quality. Such articles are eliminated, but are noted in a table, and the reason for their exclusion is stated.

outcome may have been described in many different ways, or it may not be mentioned in the abstract or among the key words (where the search motor looks for them), even if it has been used and measured in the study. Because of this, it is preferable not to search for outcomes. However, this can sometimes be a useful way to delimit a field if a search retrieves an unmanageable number of articles. In the fields of public health and intervention one frequently uses terms such as “outcome and process assessment”, “program effectiveness” and “program evaluation”.

Appendix 2 (from Kelly et al. 2002) describes and categorises a number of basic search terms that can be of use when you create a search strategy.

When searching specifically for published review articles, you should always include the term “review” in your search strategy; otherwise you run the risk of retrieving all the relevant original articles as well. It can also be useful to include the terms “systematic review” and “meta analysis”.

Searching only for a topic is another strategy. This strategy focuses on the subject of interest in the review question. This can, for instance, be a specific type of intervention (are school-based tobacco prevention programmes effective?), a specific health topic (what works best for promotion of safe sex?), or a specific outcome (what interventions reduce the risk of bicycle accidents?).

For some review questions, a topic search is more likely to retrieve all the relevant reviews. Another advantage is that when you are compiling a review of interventions in public health fields, it can be of importance to locate not only studies that show the effectiveness of an intervention, but also studies that shed light on whether the intervention is suitable for a specific target group, whether special circumstances have influenced the intervention, what factors might have contributed if an intervention did not have the expected effects, what difficulties must be overcome if the study is to be generalised to other populations, etc. If you restrict your search to studies that measure effectiveness, studies that provide other types of relevant information may be missed.

The disadvantage is that a search focused only on a topic will probably have very low sensitivity: in other words, only a few of the studies retrieved will be considered relevant enough to include in the review.

We recommend a pilot search to give you some idea of how many reviews there are within a certain field. If, for instance, you have decided to include literature in languages other than English and adjust your search strategy accordingly, a pilot search can tell you how much is available in other languages and suggest how to deal with translations.

The pilot search should preferably be done in the larger databases, i.e., the databases where you are most likely to find the pertinent reviews. Table 2 lists databases that only include already completed reviews.

It will usually be to your advantage to enlist the assistance of a competent librarian.

It is also important to restrict your search. If it is not feasible to have articles translated, you should restrict your search to languages you can handle. At the very least, you should search for articles in Swedish, Norwegian, Danish and English. You may also want to restrict the time period: how far back do you want to go for information? Some databases (Medline for

instance) contain studies from the 1960s onward. Psychinfo/LIT did not get started until 1987. You should also set a closing date after which you do not search for information. Because the databases are constantly being updated, there is always a chance that new relevant

information may come to light. We recommend that you stop looking for more information when you start working with the question of which studies to include in the review.

Where to search for information

When the search strategy has been defined, you should first search through databases that include reviews, and thereafter, other relevant databases. The Health Development Agency in England provides the list of databases for reviews shown in Table 2 (Kelly et al. 2002). This list has been updated and supplemented with databases that are of use for studies in Sweden. When searching for review articles, it is important not to use too narrow a search strategy. It is often sufficient to search on one subject term each for the intervention and the population of interest. If this does not retrieve enough articles, you can broaden your search to include reviews of closely related fields.

Table 2. List of systematic and other databases containing review articles. Reviews of reviews

Cochrane Library: Cochrane Database of Systematic Reviews <http://www.cochrane.de/>

DARE <http://agatha.york.ac.uk/darehp.htm> Center for Disease Control CDC Community Guide <http://www.thecommunityguide.org/home_f.html>

NICE web page (published appraisals) <http://www.nice.org.uk/nice-web/> HSTAT <http://text.nlm.nih.gov>

Evidence Base 2000, Health Development Agency, London

<http://194.83.94.80/hda/docs/evidence/eb2000/corehtm./intro.htm>

Evidence for Policy and Practice Information Co-ordinating Centre (EPPI-Centre), University of London

<http://epi.ioe.ac.uk/EPPIWeb/home.aspx?page=databases.htm>

The Combined Health Information Database (CHID), National Institute of Health and Centers for Disease Control and Prevention <http://chid.nih.gov/>

SBU – The Swedish Council on Technology Assessment in Health Care <http://www.sbu.se/admin/index.asp>

Table 3. Ongoing research

Databases including reviews in progress

CCTR (Cochrane Library) <http://www.cochrane.de/> Controlled Clinical Trials <http://controlled-trials.com>

Below, we briefly describe the databases that are most useful within the fields of disease prevention and health promotion (Medin & Alexandersson 2000):

PsycINFO (access fee). (http://www.apa.org/psycinfo/) This database covers mainly the psychological and behavioural sciences. Many databases overlap to some extent, and this one also contains articles in the fields of medicine, pharmacology and linguistics, as well as some in the nursing sciences. PsycINFO covers over 1500 journals written in many different languages, and is updated monthly. Books and book chapters are also covered, but only those in English. The database was established in 1987 and is produced by APA (The American Psychological Association).

Medline/PubMed (free of charge). (http://www.ncbi.nlm.nih.gov/PubMed/)

The Medline database covers mainly biomedical literature. Over 4000 of the world’s leading journals are included. Medline is updated once a week and can be searched from 1966

onwards. The National Library of Medicine (NLM) in Washington, D.C., USA is responsible for its publication.

Social Sciences Citation Index (SSCI, access fee).

(http://www.isinet.com/isi/products/citation/ssci/) This interdisciplinary database includes articles published in the fields of social and behavioural science since 1986. It covers over 3600 journals. It is produced by the Institute for Scientific Information. Only references are available (not full-text articles). At the same time, it is a citation index, i.e., it provides

information about who has published articles about a particular topic and which other authors have cited those articles.

ERIC (Educational Resources Information Center, free of charge).

(http://www.eric.ed.gov/searchdb/searchdb.html) This database mainly covers educational material in the form of scientific articles, books and conference reports. ERIC was established in 1966 and now contains over a million references. The database in updated once a month and is produced by Cambridge Scientific Abstracts.

Libris (free of charge). (http://www.libris.kb.se/) Libris focuses mainly on Swedish research and specialist libraries in Sweden. Libris contains over four million references. References to books and journals are updated daily; references to individual articles are updated weekly. Libris is produced by the National Library of Sweden in collaboration with seventy other libraries.

Regardless of which database you are searching, always keep the following in mind:

• read the description of what the database contains and read the help texts

• find out how the subject matter is described – interdisciplinary, specialist, popular science

• find out what kind of database it is – full-text, reference, facts

• find out what kinds of documents it covers – journal articles, conference reports, reports, books

• time covered, update frequency

• languages covered

When you have searched through all the databases you should proceed to some complementary searches:

Reference lists. A check through the reference lists of pivotal review articles can turn up additional important references that may have been missed in your search.

Handsearching. There is always a risk that a search will miss important publications, e.g., because the search strategy is not optimal or because the database is not fully indexed. If possible, you should try also to look through the journals that are most relevant to your review question in search of articles that were overlooked in the database search and the reference lists. This kind of manual search can also turn up the latest news, reports that have not yet been added to the databases or been cited in other articles.

Search the “grey literature”. Important results are sometimes published in various types of reports that are not documented in any database. Grey literature also includes, e.g., conference proceedings, term papers, case studies, theses and dissertations. Complete identification of all the relevant grey literature in a field is virtually impossible, but some of this grey literature is indexed in databases such as Spriline (http://www.spri.se/spriline/infos.htm) and SIGLE (System for Information on Grey Literature).

In this context, your reference group plays a critical role. The experts in the reference group are often well acquainted with the literature. You can also search for grey literature through various research agencies, professional networks and organisations, etc. A search for pertinent dissertations (check the database “Dissertation Abstracts”) and term papers can also be

fruitful. Conference proceedings often provide valuable information about completed and ongoing research in a field. Here again there are useful databases such as “Index to Scientific and Technical Proceedings” and “Conference Papers Index”. Some of the larger libraries have catalogues of conference reports. However, it is important to keep in mind that the summaries presented at conferences do not always agree with the final report on a study. It may therefore become necessary to contact the author directly and ask for a copy of the final report before you decide whether or not to include the information in the review.

Registers of ongoing research. There are registers that list ongoing research (not all research gets published). The web site Forskning.se

(http://www.forskning.se/om_forskningSe/default.asp) provides links to the four major Swedish research councils. Their web sites, in turn, have information on ongoing research projects. The project descriptions are sometimes linked to information from universities, colleges and other research institutions, but the completeness of that information – if it is available at all – depends on how much the scientists involved are willing to post on the database. “The National Research Register” (http://www.update-software.come/National/) is an international register of ongoing research. The research councils also maintain registers of all the research they support. On the CRD home page (Centre for Research and

Dissemination) (http://www.york.ac.uk/inst/crd/welcome.htm) there are lists of up-to-date research indexes and registers. You can also find research registers on the Internet (for instance “metaRegister of Controlled Trials”, maintained by “Current Controlled Trials”). As mentioned above, you should first contact your reference group: they probably know which research registers you should look through.

Searching the Internet. An Internet search can provide useful information about ongoing and completed research; in particular, this can give information about studies that have not yet been formally published. But Internet searching can sometimes be a discouraging task. Many of the major search engines are unable to handle a sophisticated multilevel search; one search can give thousands of hits, each of which must be examined. Yet it is possible to search the Internet in a relatively systematic way by using “meta-search engines” such as “Copernic”

and “Dogpile”. Another useful search engine is called “Scirus” (www.scirus.com). Scirus is a search engine specialised for scientific information. Scirus also has a filter that helps filter out non-scientific information.

Reference management systems

Reference management programs are exceedingly useful for keeping track of references and ensuring accurate and easy insertion into documents in preparation. With a reference

management program, you can download relevant search results electronically, directly from a database to your own reference files and save them either on your hard drive or on a disk. References located manually must be entered into the reference management system by hand. In this situation it is important to ensure that all the relevant bibliographic information is included. Several reference management programs are available: EndNote, Reference Manager and GetARef are some of the most common.

Document your search

As a general rule, the entire process from formulation of the review question to the writing-up of the paper must be clear to the reader. For the reader to be able to evaluate the final product it is necessary for you to describe clearly and in some detail how the search was done and how the studies were selected. Therefore the search process must be documented as you go along, and you should also present the rationale for any changes or additions that are made in the search and selection procedures.

Selecting reviews

When you have completed your literature search, it is time to select which reviews you will read in their entirety. Database searches generally result in long lists of references and

Important points on literature searching:

1. It is important that the search for previously published reviews is extensive and systematic, otherwise you risk drawing incorrect conclusions.

2. When you develop a search strategy you should enlist the help of a librarian with experience of electronic searching.

3. For a search to be adequate, it must include not only database searches, but also manual searches of relevant journals, and input from specialists in the field.

4. The search procedure must be documented as you go and search results must be saved.

5. A reference management system will facilitate the work and save time.

summaries/abstracts (the long list). This list is comprised of all the references that were retrieved in your searches. It is important to go through these and compare the results of the various searches, because the same reference will often be found in two different databases and the risk of ordering duplicates is substantial. Two or more readers should go through the summaries and decide on the basis of the review question which articles should be included (the short list). If there is any doubt, or if the readers disagree about whether an article should be included, they should discuss it and try to reach a consensus agreement. The readers then divide up the short list and order full-text copies of the articles. When the full article is being read and evaluated, the readers should follow a protocol (Appendix 1). Each reference is read by at least two members of the group, and if they disagree in their evaluation of the article, it should be discussed until consensus is reached. Each review is assigned to one of the three categories listed below after inclusion and exclusion criteria have been defined. Sometimes it is necessary to read the articles in two stages: first to decide whether or not the article should be included, and then once more, at which time the protocol is filled in.

1. Include in its entirety

2. Include parts, or use as background for the review

3. Exclude – in which case the rationale is noted and presented in a table as an appendix. All the articles that end up in categories 1 and 2 form together the material on which the review will be based. The next step is to evaluate the reviews that have been selected (categories 1 and 2) to determine whether they can give an exhaustive answer to the review question that has been posed. If not, the group may decide to do a completely new research overview; alternatively, it may be sufficient to supplement the review with more recent literature.

Evaluating grey literature

Grey literature in the form of reports from studies that are not published in scientific journals can be evaluated in the same way as published reviews of the literature. Other reports should be assessed by the group as a whole, and a decision made on whether the results and findings presented in this type of literature should be included in the review.

Synthesis

The aim of the synthesis is to summarise data and knowledge gleaned from the reviews. In many cases it is not possible to summarise the results quantitatively in such a way that an average can be calculated (meta analysis) because the material included is so disparate in terms of method or number of studies evaluated. If the final overview is based on a small number of published reviews, it may be possible to synthesise them as a narrative that describes each of the reviews included.

Tabular presentation of the effects examined, the number of studies included, the time-span, the method, etc., will make it easier for the reader to comprehend the synthesis. Such a table is also an excellent starting point for creating a synthesis, as it gives a summary of all the results at one glance. The tables should be presented according to type of intervention, so that each intervention has its own table. When these tables are finalised you should go through and

check which original publications the reviews are based on. The synthesis should include information on the degree to which the reviews are based on the same studies. If you find that certain studies appear to be particularly important in several reviews, they should be listed so the reader can find them.

The synthesis should also state which reviews have been included as background material, which reviews form part of the synthesis, and which have been excluded (and why). The synthesis should end with a general summary of the reviews included, and present the conclusions that have been reached.

There are several good examples of reviews of previously published reviews: a report from the University of York and the Center for Research and Dissemination, concerning tobacco prevention among young people (University of York, CRD 1999), and a couple of reports from The Effective Public Health Practice Project, Hamilton, Canada, one about the

effectiveness of school-based interventions targeting risk behaviour in teenagers (Thomas et al. 1999) and one about the effectiveness of school-based strategies for primary prevention of obesity in young people (Micucci et al. 2002).

Timeframe

Creating a search strategy for previously published reviews takes about one day. We estimate that it takes another day to go through the databases that are restricted to reviews, and yet another day to search through the other databases. This should preferably be done in

collaboration with the people who will be evaluating the literature and with the assistance of an experienced librarian.

Once these things are done, you must select and order the articles that are to be read in their entirety. This will require a whole day’s work of the person who is going to read the articles. The most time-consuming part is perusal and evaluation of the reviews that have been ordered. Here we recommend that at least two people, independently, read and evaluate each article (according to the protocol). Evaluation of each article will probably take 3 to 4 hours. Thus, the total time required for reading and evaluation will depend on how many reviews are included. As one reads, one often identifies yet other relevant articles in the reference lists; these articles should also be ordered and evaluated, further lengthening the time it takes. Finally, the synthesis described above will probably take a few more days’ work.

Chapter 4 – Writing up and disseminating the results

Writing the report is an integral part of the task of compiling a literature overview. The authors should strive to make the text accessible and informative. A clear, lucid final report enables the reader to assess the usefulness and quality of the material, and determine whether it can be used as a basis for decision-making. To make it possible to create different versions of the report, it is important that the final report be as reliable and exhaustive as possible. A report can be presented in many different formats. Which one to choose depends on the needs and wishes of the target readers. There are frequently several different versions of the final report: scientific articles, pages on web-sites, popular science summaries, etc. Below, we suggest one structure for a report (adapted from Kahn et al. 2001).

The key questions of the report:

• The way in which the report is written and presented will have impact on how the results are received, interpreted and used. The report should be clear and concise and take into consideration the needs of its intended readers.

• The summary is critical in catching the reader’s attention. It should therefore contain enough information that the reader can quickly determine the usefulness and reliability of the conclusions.

• The main body of the text should describe the systematic/scientific procedures used in preparing the review. Methods must be

described in sufficient detail. Results should be presented based on adequate measures and should describe the effectiveness and processes shown in the reviews/studies.

• Policy-makers and practitioners often do not have time to read the entire report. For this reason, the discussion section and the

conclusions should be written to make it clear that they are based on the results of the overview and not on speculation.

Suggested structure for the report Title and authors

Summary

• Introduction (background) • Objectives

• Methods/procedures (data sources, selection criteria, quality assessment, data extraction)

• Results (synthesis of the data) • Conclusions

Main body of the text • Background • Review question(s)

• Evaluation procedures (how the assessment was done) • Data sources and search strategies

• Selection criteria (inclusion and exclusion criteria) • Quality assessment of the reviews

• Data extraction • Synthesis

Information about included and excluded reviews Results of the overview

• Main conclusions about the methods

• Reliability, or estimation of how robust the results are Discussion (interpretation of the results)

Conclusions

• Recommendations (if any) to policy-makers and practitioners • Suggested future studies

Acknowledgements Conflicts of interest References

The title and the authors

The report’s title should be succinct but informative. The aim is to attract the reader’s attention and spark enough interest that he or she wants to continue reading. It can therefore be advantageous to present the main findings of the study in the title itself. At the same time you must guard against overstating the results in the title – all too easy to do when trying to be concise and informative at the same time. Make it clear at the planning stage which persons are to be listed as authors.

The summary

The information in the summary should – just like the title – catch the reader’s attention, but it should also give the reader an opportunity to form a quick impression of the quality of the review and practical applicability of the findings. A summary should generally not exceed 1000 words. The summary is the most important part of the report: many will read no more of the article than the summary (and perhaps the conclusions and recommendations). Therefore you should take pains to write a summary that does not contain too much technical detail. Instead, describe the article in a lucid, explanatory manner, without exaggerating the importance of the results.

In addition, the summary should be written in a structured manner. It should have separate sections summarising the content of the review. In this way the reader is allowed to draw his or her own conclusions about the procedures, results and conclusions. The introduction

should begin by describing how the review question is important. The next section,

objectives/aims should describe the main review question and any subsidiary questions. The

methods section presents the data sources that have been used, the criteria used for selection of studies, quality criteria and data extraction methods. The results section should present the most important findings (qualitative and/or quantitative). The conclusions should be directly related to the results. The conclusion section should also describe how the results might be put to practical use. If there are obvious gaps in current knowledge, round off the summary by recommending directions for future research.

The main body of the text

Background information. The rationale for the review can be described by discussing the problem addressed (the review question). Historic, social, economic or other perspectives can be used to describe how public health workers and decision makers need these results to discharge their daily duties. To continue to build on the background underlying the review, you might describe the target group in relation to the problem, previously used methods or interventions or relevant effects.

The review question. The main question (and the subsidiary questions, if there are any) should be described in detail from several different perspectives: population/target group, interventions, outcomes and study design.

Evaluation procedures. The methods and procedures used in the study should be described in the sections dealing with the search procedure, search strategy, inclusion and exclusion

Here you should also describe any changes that have been made in the protocol during the process of data collection. One good approach to writing this section is to keep in mind that you must give enough information that it would be possible to repeat the entire process in more or less the same way.

Details about included and excluded reviews/studies. Details concerning the selection process must be reported with great care. One way to describe this is to use a flow chart. Excluded reviews must also be listed. If possible, reasons why an article was excluded should be given here. This can often be included as an appendix.

Results. The results should be presented succinctly. Quantitative data can be presented in tables with a descriptive summary. The most important features in the reviews should also be mentioned here. This includes the target groups, interventions and outcomes that have been examined in the reviews. Information on study design and other quality aspects can also be presented in tabular form. Preparing tables and graphs is time-consuming work, but without them a reader will find it virtually impossible to understand the results.

Discussion. Readers will frequently turn to the discussion section for help on how to interpret the findings of the review. The discussion should build directly on the results; it should interpret them and discuss their practical utility.

This section should also discuss the validity of the quantitative results, possible systematic errors, the strength of the evidence, the implications of positive and/or negative effects and estimated costs. If this discussion is presented alongside results concerning qualitative aspects of the reviews and studies included in the systematic review, it may be easier for the reader to grasp the big picture and to draw his or her own conclusions.

One part of the discussion should address whether the results can be generalised. Health promotion work is almost always affected by the context: community norms and values, the people involved, organisational and economic constraints, etc. It should be made crystal-clear in the discussion that the success of a project in one setting by no means guarantees that it would be successful if carried out in another setting. Once this point has been made, the discussion can move on to the basic components of the successful projects included in the review, attempt to pinpoint the “lowest common denominator” or the “active ingredients” that make for success.

The last part of the discussion section should reason through the conclusions. Systematic reviews of a quantitative or experimental nature usually include recommendations based on the strength of evidence they have presented, ranked according to the quality of the study design, the strength of the effects observed, etc. In systematic reviews of health promotion and disease prevention interventions, this quantitative approach is often difficult. The fact that the reviews are based on many different primary studies hampers quantitative comparison of the strength of the evidence. Instead, we suggest that the evidence be described in qualitative terms; this may then lead over into cautious recommendations in some form. Besides, it is nearly impossible to compile a review of reviews of quantitative comparisons: for one thing, this would necessitate retrieval and re-evaluation of all the original studies cited in the reviews – theoretically possible, but in practice an overwhelming task.

Acknowledgements. Most reviews are the result of contributions from many individuals and organisations: they have helped with literature searches, data collection, editing, critical reading, etc. It is common courtesy to mention these contributors and state how they have been of assistance.

References and appendices. References must be complete and accurate. As mentioned above, a reference management system will be an invaluable asset (see page 21). There are several internationally accepted citation formats (e.g., APA, Vancouver, etc.). Articles within the social sciences most often use the APA format, whereas biomedical journals usually use the Vancouver format. Precise instructions on how to use the APA format are given in the American Psychological Association’s (APA) publication manual (1994). The Vancouver format is described on the University of Southampton Library web site, among others (http://www.library.soton.ac.uk/infoskills/vancouver). In addition to your reference list, you should have a list of included and excluded reviews. These can be presented in appendices. Descriptions of the search strategies, raw data from included studies, and other pertinent information can also be put in appendices.

Critical reading of the manuscript

Before the review is submitted for publication it should be read critically by the panel of experts (reference group) that was formed to assist in the reviewing process. In particular, they should comment on the review’s quality and comprehensiveness.

Summary of the main points to include in the discussion section: • Summary of the most important results

• Strengths and weaknesses of the review • Assessment of the quality of the review

• Relation between this and other reviews, especially when there are differences in quality and findings

• Implications of the results

• Strengths and weaknesses of the evidence included in the review • Effects (positive/negative, weak/strong) observed in the reviews and

studies included

• Applications for the results of the reviews • Recommendations

• Practical implications for public health workers and politicians • Remaining questions and suggestions for future research

Dissemination of results

Promotion and dissemination of the results must insofar as it is possible be tailored to each target audience. To facilitate this, it is important that reference group includes representatives from the target audience. In this way the target group can follow along in the process: can clarify their specific needs and influence how the report is presented. Obviously one cannot present the same results in the same way to all target audiences. It is frequently a good idea to consider publishing a reworked version of the compiled report in a scientific journal.

Taken together, this means that promotion and dissemination strategies will vary from review to review, and that they must be tailored to suit the aim of the project and the needs of the target group. Conceivable strategies include direct mailings to target group networks,

advertisements in professional journals, press releases, sending review copies of the article to selected journals, and publicising the report on web sites. It is also a good idea to follow up and evaluate the dissemination of the article.

Timeframe

It will probably take about two weeks to prepare a draft manuscript for evaluation by the reference group. When these experts have returned their comments, it will probably take another week or two to incorporate their suggestions and to fix the layout of the article. If several versions of the report are to be prepared, more time will be required.