An Evaluation of the ‘What Works
Centre for Crime Reduction’
Year 1: Baseline
Gillian Hunter
Alexandra Wigzell
Tiggey May
Tim McSweeney
February 2015
ii
Page
Contents
ii
Acknowledgements
iii
Summary
iv
1. Introduction
1
2. The Current Situation
12
3. Barriers, Capacity Building and Knowledge Gaps
24
4. The College and its Role in Facilitating a Research Culture
34
5. Discussion and Implications
43
References
48
Appendix A: Qualitative Interview Schedule
49
Appendix B A Breakdown of the Online Survey Results
58
iii
ACKNOWLEDGEMENTS
We are very grateful to the large number of people who gave up their time to be interviewed in person or to complete our on-line questionnaire. We should also like to thank staff at the College of Policing for their advice and support. We are especially grateful to Krisztián Pósch for his help in preparing Appendix C.
Gillian Hunter Alexandra Wigzell Tiggey May Tim McSweeney
iv
SUMMARY
This evaluation forms part of a three-year programme of work funded by the Economic and Social Research Council (ESRC), in collaboration with the College of Policing, as part of the
What Works Centre for Crime Reduction (WWCCR). This report covers the first year of the
evaluation, which sought to establish a baseline from which to measure change over the three-year programme in the understanding, use and application of research evidence in crime reduction. The emphasis throughout is on understanding not simply what works, but
how it works.
Aims of the evaluation
The overall aims of the evaluation are to:
Assess the impact of the WWCCR which is hosted by the College of Policing to determine whether it has appropriately engaged key stakeholders, produced tools and guidance that stakeholders find clear and easy to use, and improved stakeholder understanding and application of research evidence to inform practice and decision-making;
Chart outputs, modes of dissemination and user reactions about the WWCCR over the course of the evaluation;
Identify changes over time in the use of research evidence by key stakeholders, especially in strategic decision-making and resource allocation;
Use an action research model to provide feedback to the College of Policing and the academic partners over the course of the project.
Methods
In-depth interviews (N=49) were conducted with three main stakeholder groups targeted by College of Policing and the WWCCR, these were: Senior and middle management police officers (n=29); Police and Crime Commissioners (PCCs) (n=10); and Community Safety Partnership managers (CSPs) (n=10). In addition, a quantitative web-based survey was undertaken with officers of chief inspector rank and above (including civilian staff equivalents), PCCs and CSP managers. This resulted in 655 valid responses, mainly from chief inspectors (49%), superintendents (24%) and chief superintendents (9%), or civilian staff equivalents. Areas covered by the interviews and survey included:
v
(i) factors influencing decision-making and the extent to which research evidence meets practitioners’ needs;
(ii) organisational culture and the use of research evidence; (iii) familiarity with and use of research evidence;
(iv) experience of commissioning or undertaking research;
(v) extent to which College of Policing services and resources promote an evidence-based approach to policing.
The role of research evidence
Most interviewees across the three groups cited research evidence as one of many factors which affected their decision-making.
The survey findings broadly supported the qualitative analysis: around three in five (57%) respondents felt that research evidence played an important role in their day-to-day decision-making, and two-thirds (68%) reported that research evidence had changed or influenced their working practices.
Most (72%) survey respondents reported at least one occasion during the previous 12 months where research evidence had affected how they had allocated resources, although one in four said that they rarely (24%) or never (2%) looked for research evidence to inform decisions about policy or operations.
Interviewees highlighted a range of other influences which affected their decision-making, including financial considerations, national guidance, public opinion and professional judgement.
Survey respondents, identified practice-based experience/professional judgement (81%), local force guidance/publications (80%) and input from local colleagues/staff (77%) as influencing factors.
The role of professional judgement was highlighted by some police interviewees, but others noted a reduced reliance on this.
PCC interviewees spoke about the importance of public opinion in their decision-making, stressing that their elected status made them an important conduit for public opinion.
CSP interviewees noted their strategic assessment, involving local intelligence from partner agencies as central to their decision-making about strategy.
Almost three-fifths of survey respondents felt that investment in crime reduction and prevention was driven by politics rather than research evidence (56%).
vi
Experiences of commissioning, conducting and participating in research
Experience of commissioning research was widespread amongst interviewees, although the topic focus of the studies being commissioned varied widely among our three professional groups.
Between one-fifth and a third of interviewees in each of the professional groups drew upon university students to conduct research. The ‘commissioning’ of ‘cheap’ local students was seen by many interviewees as an area ripe for development in a time of fiscal constraint.
Experience of collaborating for the purposes of research rather than formal commissioning of studies was examined in the survey, with 75 per cent of respondents reporting no emphasis on collaboration with external partners.
Barriers and capacity building
Interviewees noted that the effort and time needed to access, interpret and understand the practical application of research discouraged its wider use.
Despite a large minority (44%) of survey respondents stating that evidence-based approaches were promoted by their organisation, over half felt there was no organisational emphasis on the use of research evidence to inform decision-making (51%).
Nearly half of the survey sample felt that research findings were unclear and full of jargon (48%), and a third believed research lacked clear enough messages to make it usable (34%).
Opportunities to keep abreast of emerging research
The opportunities available to survey respondents and interviewees to keep abreast of emerging findings varied. Although almost a third of survey respondents said they had accessed and read academic publications, only a few of the chief officers, PCCs or CSPs interviewees were able to identify a piece of research which had influenced their professional thinking in the year prior to interview.
Whilst interviewees considered keeping abreast of new research as crucial, it was also considered to be a luxury in their time-poor working lives.
vii
Forty-two per cent of survey respondents lacked access to research evidence through sources such as peer reviewed academic journal articles. Only one-third felt able to keep up-to-date with research evidence on policing and crime reduction (34%). Their most common source for obtaining information about research evidence was via a general web search such as Google.
Synthesising and dissemination of research findings
Synthesising and disseminating research was regarded as challenging by many interviewees because of time and resource constraints.
Whilst all the police interviewees relied on force analysts to produce short reports on police data, many were starting to disseminate through their professional networks, in-house seminars, promoting research through evidence champions and evidence-based policing sessions.
In a number of areas performance teams were tasked with horizon scanning. Interviewees tended to rely on their professional networks as both a source of information and also a place to disseminate new findings and evidence.
Knowledge gaps and needs
Two-fifths (41%) of survey respondents felt that much of what is done to tackle crime has no research evidence to justify it.
Only one in six stated that when new policies and procedures are introduced, they are made aware of the research evidence which supports them. By contrast, almost half (47%) acknowledged that some approaches to tackling crime are ignored, despite these being supported by research evidence.
More than two-fifths (45%) of survey respondents said there had been occasions where they had sought research evidence to inform policy or operations, but could not find it.
The College and its role in facilitating a culture which values research
There was a lack of knowledge about the particulars of the WW Centre amongst interviewees; Only 12 per cent of survey respondents reported being aware of the WWCCR.
viii
The most common College products accessed by survey respondents were the Police Online Knowledge Area (80%), the Knowledge Bank in POLKA (67%), Authorised Professional Practice (APP) (65%), and College of Policing published research (57%) and many felt that College services and resources had had either a minor (38%) or moderate effect (33%) on encouraging the use of evidence-based good practice in their day-to-day work.
The ‘reach’ of the College and its ability to be relevant to operational police officers as well as those holding more senior or strategic positions was mentioned as an area that should be addressed.
Various suggestions were made about improving the marketing of College services and products. These included, demonstrating local relevance when disseminating research; simplifying and increasing the accessibility of the College and a greater emphasis on the use of research as part of professional development.
Among interviewees, a key theme was the need for the College to create useable tools for applying research and to become an authoritative and ‘quality assured’ repository of evidence-based best practice in crime reduction. Coupled with this was the view that the College needed to do this over the longer-term and in a systematic way to establish its reputation and that of research as “here to stay”, rather than a fleeting “fad” for policing and crime reduction.
This evaluation forms part of a three-year programme of work funded by the Economic and Social Research Council (ESRC), in collaboration with the College of Policing (hereafter The College), as part of the What Works Centre for Crime Reduction (WWCCR). The programme intends to draw upon inter/national good practice to build on and enhance the UK’s capacity to develop, disseminate and apply evidence-based approaches to policing and crime reduction. The emphasis throughout is on understanding not simply what works, but how it works. The programme of work, involving staff at the College and a consortium of UK universities (the Commissioned Partnership), includes development of a series of systematic evidence reviews on crime reduction topics (both the collation of existing reviews and the writing of new reviews on topics identified by stakeholders), the creation of a standard system to rate and rank interventions in terms of their effectiveness and cost-savings, and training programmes to enhance professionals’ capacity and skills to appraise research evidence.
This programme of work by the Commissioned Partnership included provision for an independent evaluation, and this report presents baseline findings from this work. The focus of the evaluation is on strategic rather than tactical decision-making; and for this reason we are assessing the impact of the centre largely, but not exclusively, on middle/senior rank police officers, community safety managers, and on Police and Crime Commissioners (PCCs), rather than front-line staff and their supervisors. The evaluation includes an assessment of both impact and process and has been conceived of as action research. This means that we are providing feedback to the College of Policing and the academic partners over the course of the project in order to maximise the chances of successful outcomes.
The evaluation is particularly interested in whether there are changes in the organisational culture at senior and middle levels in the use of evidence for policy and strategic decision-making, whether this includes greater prioritisation of the creation of research evidence, and whether there are observable shifts in the allocation of resources that reflect the impact of research. In the first year of the evaluation we have sought to establish a baseline from which to measure change over the three-year programme in the understanding, use and application of research evidence in crime reduction both within policing and amongst other crime reduction practitioners.
2
1.1 The context
In March 2013 the Cabinet Office launched the ‘What Works Network’, a nationally co-ordinated initiative aimed at positioning the research evidence on ‘what works’ at the centre of public policy decision-making1. Currently there are seven research centres2 focusing on six key areas of public policy, including health, education, early intervention, well-being, ageing, local economic growth and crime reduction. These ‘research hubs’ are intended to build on existing models of delivering evidence-based policy - such as the well-established and well-funded National Institute for Health and Clinical Excellence (NICE). This provides independent evidence-based guidance to the NHS and health professionals about the targeting of funding and the most effective ways to prevent, diagnose and treat disease and ill health. The Educational Endowment Foundation has also developed a toolkit to appraise interventions in education in terms of their cost and impact.
The What Works centres are being developed in a political environment which, it is argued, is increasingly amenable to the idea of evidence-based decision-making, particularly in the context of ‘austerity’ and cost effectiveness. Government white papers, including the Cabinet Office Open Public Services (2012) and the Civil Service Reform White Paper (2012) assert government commitment to scrutiny and transparency across departments, and initiatives are in place to increase and make easier access to government administrative data for the purposes of research and evaluation (Mulgan and Puttick, 2013; UK Administrative Data Research Network, 2012). However, a recent report by the National Audit Office (2013) suggests that these ambitions are not yet embedded in practice. The NAO’s assessment of the frequency and quality of impact and cost-effectiveness evaluation across key government departments, and the use of such evidence to support resource allocation and policy development, highlighted a number of issues. These included a lack of robust impact evaluations, a lack of clarity in government decisions about what to evaluate and a failure to effectively apply learning from evaluative research.
1
This was accompanied by suggestions that future spending reviews would tie departmental funding to the extent to which policies are evidence-based. See Smith (2013).
2
National Institute for Health and Care Excellence (NICE), Sutton Trust/Educational Endowment Foundation, College of Policing What Works Centre for Crime Reduction, Early Intervention Foundation, What Works for Local Economic Growth, the Centre for Ageing Better, and the What Works Centre for Wellbeing.
3
For some years now, the College of Policing and its predecessor the National Policing
Improvement Agency (NPIA) have been promoting the importance of research evidence to
inform practice in policing and crime reduction. An NPIA action plan for improving knowledge use in policing (NPIA, 2010) presented a vision of “a police service that routinely uses good
quality knowledge to decide what to target, what action to take and what resource to deploy”
and cited a range of targets to be achieved by 2013, over which the WWCCR now take ownership. These include investing in research and developing research partnerships, quality assuring research evidence and sharing and embedding that knowledge in professional practice. There have been various practical activities organised to increase engagement with research, including for example Research Fairs and other events to match academic expertise with policing priorities.
A key area for the WWCCR, and for this evaluation, however, is not just the creation and availability of rigorous evidence on what works best to reduce crime, but also an understanding of how this evidence is adopted throughout an organisation and the extent to which it is accepted and valued within the professional culture (Ritter and Lancaster, 2013). One commonly mentioned inhibitor within policing culture, for example, is the traditional reactive ways in which police have operated, with performance targets for arrests and convictions prioritised over preventative approaches (Cherney and Head, 2011).
The most effective mechanisms for bridging the inherent gaps between research knowledge and practice are frequently debated. Recommendations focus on the accessibility and format of information, the need to involve practitioners as evidence producers so research is more firmly embedded in and relevant to frontline experience; increasing skills and capacity through training and development programmes and the ‘championing’ of research-based practice within the organisation (Sharples, 2013; Cherney and Head, 2011).
1.2 Aims of the evaluation
The overall aims of the evaluation are to:
Assess the impact of the WWCCR to determine whether it has appropriately engaged key stakeholders, produced tools and guidance that stakeholders find clear and easy to use, and improved stakeholder understanding and application of research evidence;
4
Chart outputs, modes of dissemination and user reactions over the course of the evaluation;
Identify changes over time in the use of research evidence, especially in strategic decision-making and resource allocation;
Use an action research model to provide feedback to the College and the academic partners over the course of the project.
A central issue for the evaluation is the definition of ‘research evidence’. The WWCCR is intended to stimulate greater use of evidence, and its success in so doing will obviously depend at least in part on the definitional boundaries that are placed around evidence. Inherent in the work of all the What Works centres are inter-related principles of public domain accessibility (usually through, although not limited to, publication) and quality (usually achieved through peer review prior to publication). Research evidence is closely related to, but not identical to, academic research. This report treats as research evidence any structured analysis of methods of reducing crime that (a) result in a published report following (b) some sort of independent quality assessment.
1.3 Methods
Our overall approach to the evaluation consists of a straightforward ‘before and after’ design, supplemented in the hiatus between ‘before’ and ‘after’ with research that explores consumer reactions to the products of the WWCCR. The methods thus far have comprised qualitative in-depth interviews and a quantitative web-based survey with the main stakeholder groups, which will be replicated in the final year of the evaluation. The survey covered five themes (mapped against the evaluation plan): (i) factors influencing decision-making; (ii) extent to which research evidence meets practitioners’ needs; (iii) organisational culture and the use of research evidence; (iv) familiarity with and use of research evidence; and (v) College services and resources. The in-depth interviews covered similar ground, but also covered experience of commissioning or undertaking research, perception of competency in assessing the quality of research, and ideas on how the College can better promote the use of research evidence.
We defined evidence as any published research (including 'grey’ literature such as internal reports, working papers, technical reports, conference proceedings, student dissertations and theses) on the effectiveness of a particular policy, intervention, tactic or approach which aims to reduce or prevent crime.
5
Feedback on both our proposed approach to sampling and early drafts of the interview schedule (see Appendix A) and survey were provided by the Commissioned Partnership's Academic Board and the Knowledge, Research and Practice Unit (Formerly the Research, Analysis and Information Unit) at the College. The interview schedule was piloted with two senior police officers and an online version of the survey was piloted with a small number of uniformed officers (N=6) from three force areas prior to its launch. A note of endorsement for the research from the College CEO, Chief Constable Alex Marshall, accompanied all requests for assistance.
Depth interviews
A target was set of 50 depth interviews with three main stakeholder groups targeted by the College and WWCCR (chief police officers, Police and Crime Commissioners (PCCs) and Community Safety Partnership (CSP) managers). A sampling frame was created for each group. For the chief officers this included information contained in the Police Almanac and from biographies on police force websites on age, rank, years of service, academic qualifications, including whether part of an accelerated promotion scheme, whether holding a position as National Policing Lead or any other relevant specialist area or secondment. For PCCs this included information on political affiliation, professional background, academic qualifications, age and gender. For CSPs we gathered information on geographical region and type of authority only as no other information was readily available.
Based on this information, a first selection of interviewees was informed by the need to include:
A geographic spread;
A mix of urban and rural areas (as crime reduction priorities will likely differ);
Those with and without previous academic qualifications in relevant areas (chief officers and PCCs);
A range of professional backgrounds (PCC);
Those with a specialist area or position as National Policing Lead (chief officers) ;
A mix of political affiliation/independent (PCC) ;
A range in terms of years served (chief officers); and
A gender and age mix.
Each potential interviewee was emailed a letter explaining the aims of the research and inviting them to participate in an interview. This email request was followed up with a telephone call. Where there was a refusal or an inability to participate, another interviewee
6
was selected to replicate as far as possible the selection criteria used for the original interviewee (e.g. in terms of specialism or academic qualifications).
The make-up of the final interview sample is described in Table 1 and comprised: 29 chief officers from 28 police forces; 10 PCCs across a range of professional backgrounds, including police (3); military (2); business (2); local politics/council (3) and 10 CSP managers drawn from the 10 regions of England and Wales.
Depth interviews were conducted either face-to-face or by telephone (if preferred by the interviewee) between May and September 2014.
Table 1: Job title of interview sample (N=49)
Job title/role Number
Chief Officer
Chief Constable
11
Deputy Chief Constable
8
Assistant Chief Constable
6
Assistance Commissioner
1
Commander
1
Senior Corporate Personnel
2
Police and Crime Commissioner
Conservative
6
Labour
1
Independent
3
Community Safety Partnership Manager
Unitary Authority
4
Metropolitan District
3
County Council
2
Inner London Borough
1
7
LimitationsWe struggled to recruit CSPs: three tranches of interview invitations, reaching 26 CSPs over four months were despatched before we reached our target sample of 10. We suspect that these interviewees may be unrepresentative of CSP managers nationally, their high awareness and use of research being factors that inclined them to agree to be interviewed.
A total of 18 PCCs were contacted to get our final sample of 10. It should be noted that our PCC sample was heavily biased towards Conservative Party nominees. This was due to a consistently poor response rate from those of other political affiliations, which also hindered our attempts to redress the balance.
A typology of interviewees
As part of our qualitative analysis, interviewees were coded into four broad ‘research aware’ categories:
Research savvy;
Regular users of research (which included two sub categories);
Limited users of research; and
Research indifferent
As discussed above, levels of research awareness turn on definitions of what counts as research. In both the in-depth interviews and the survey we tried to prompt participants to share our definition, whereby research evidence is any structured analysis of methods of reducing crime that (a) result in a published report following (b) some sort of quality assessment. But the concept is a loose and flexible one, and it must be recognised that some participants will have applied other definitions in answering our questions.
As analysis continues, it may be possible to develop a more elaborate typology which permits us to place people on dimensions beyond, simply, levels of research usage. However, this dimension is obviously of central interest to the evaluation. Interviewees were categorised based on our analysis about the extent of their and their organisation’s use of research (or at least their perception of this) in decision-making and resource allocation. Experience of commissioning and conducting research was also taken into consideration. However, readers should be clear that if interviewees had been coded predominantly by commissioning, for example - they would have been spread very differently across the types: for instance, the majority of chief officers would have been classified as research savvy rather than regular users. Similarly, the categories to which interviewees were assigned do
8
not necessarily reflect their level of research knowledge and awareness, as distinct from usage: some of those classed as ‘research savvy’ were open about their lack of expertise in appraising research studies (but were surrounded by those who did possess such skills), while a number of those categorised as ‘limited research users’ demonstrated some proficiency in this respect. It was also apparent that while research evidence regularly informed the decisions of those with whom we spoke – and their wider management teams – such support for research was often not perceived to be present across all departments within the organisation. This issue was noted frequently by chief officers
Survey
The finalised online survey ran for a six-week period from 16th June to 1st August 2014. Alerts providing details about the survey were distributed by the following organisations to their members:
Association of Police and Crime Commissioners;
Association of Chief Police Officers;
Police Superintendents’ Association of England and Wales;
The Association of Scottish Police Superintendents;
The Superintendents' Association of Northern Ireland;
Police Federation of England and Wales (chief inspectors only); and
Scottish Police Federation (chief inspectors only).
Alerts about the survey were also circulated on the Police Online Knowledge Area (POLKA) and disseminated within local forces via College evidence champions. In addition, Chief Constables/Commissioners for each of the UK forces (N=49) were contacted individually by email. CSP chairs and managers were contacted in England and Wales using a directory maintained by the Home Office (N=699)3. In Scotland CSPs were contacted directly by email (N=32) and via a Scottish Community Safety Network bulletin.
Each of these sources was asked for assistance in raising awareness of the survey throughout their respective organisations. For those representing police personnel, they were informed that the survey targeted officers ranked chief inspector and above (including civilian staff equivalents). They were also informed that the survey was confidential and securely located on the College website, and provided with a link to access it.
3
9
Each of these sources was contacted again one week prior to the closure of the survey, with tailored feedback on the number of responses received, and a further request for assistance in raising awareness of the survey, and encouraging any non-completers to participate. By the time the survey closed in early August 2014 we had received 993 responses4. Of the 993 responders, 157 (16%) started but failed to complete the survey. Of the 836 that completed, 90 (11%) did not provide any details of their rank or role, and there were responses from police and community support officers (PCSOs) (2), constables (22), sergeants (15) and inspectors (39).
The analyses which follows in this report focuses on the 655 respondents who completed the entire survey and belonged to one of the groups targeted by the research: PCCs, CSP managers, and senior police officers ranked chief inspector or above (including civilian staff equivalents). As described in Table 2, most survey respondents were chief inspectors (49%), superintendents (24%) and chief superintendents (9%), or civilian staff equivalents.
Responses were received from 46 police force areas5, with an average (median) of 10 returns per force (mean=13.3; range=1-58; SD=13.5). Most respondents were male (70.3%) with an average (median) age of 47 years (mean=46.6; range=21-72; SD=6.5).
Respondents had an average (median) of 22 years of service/employment (mean=21 years; range=0-48; SD=7.9). Two-thirds reported that their highest level of educational attainment was either a first degree level qualification (including foundation degrees, graduate membership of a professional Institute) (39%) or a university postgraduate/higher degree (e.g. Masters or PhD) (27%). Most (86%) reported that they were not currently studying towards a qualification
,
but chief inspectors (16%) and superintendents (17%) were more likely to report doing so than chief superintendents (10%) and other (7%) survey respondents (χ²(1, N=529)=8.16, p=0.043).The full findings from the survey are set out in Appendix B.4
Our original bid to the ESRC had set a target of 800 respondents.
5
Thirteen (2%) respondents did not indicate in which force area they worked and 32 (5%) stated “other” in response to this question.
10
Table 2: Job title of survey respondents (N=655)
Job title/role Number Percent
Chief Inspector/staff equivalent 322 49
Superintendent/staff equivalent 154 23.5
Chief Superintendent/staff equivalent 61 9
Other 33 5
Other senior police staff equivalent (e.g. staff officer) 33 5
Community Safety Partnership Manager 26 4
Community Safety Partnership Chair 7 1
Assistant Chief Constable/Commissioner 4 1
Chief Constable/Commissioner 3 0.5
Commander 3 0.5
Deputy Chief Constable/Commissioner 3 0.5
Police and Crime Commissioner 3 0.5
Police and Crime Commissioner's Office 3 0.5
Total 655 100
Analysis
The in-depth interviews were recorded and fully transcribed, with analysis undertaken using Nvivo 9. A primary coding frame was developed, based on the key areas or themes covered by the interview schedule. This was refined by further ‘sub-coding’ within each of these main areas. The survey was analysed using SPSS. The results presented here comprise largely of descriptive statistics, though techniques of exploratory and confirmatory factor analysis were also used to identify the constructs used by survey respondents in thinking about research evidence (see Appendix C).
11
Structure of report
The following chapters draw together key findings from the depth interviews and online survey, highlighting where feedback from these various groups differ or concur. Chapter 2 explores the key factors in decision-making about strategy and resource allocation, the extent to which research features in these decisions and interviewees’ experiences of commissioning and conducting research studies. Chapter 3 examines barriers and opportunities to greater use of research, the existing organisational structures for disseminating or discussing research and perceptions about key gaps in knowledge. Chapter 4 assesses the level of engagement with the College and its resources and the role envisaged for the College in facilitating a cultural shift in the value placed on research. The final chapter reviews findings against the original aims of our evaluation and presents a number of recommendations for the WWCCR.
12
2. THE CURRENT SITUATION
This chapter explores the current role of research in decision-making by interviewees and survey respondents. We start by examining the factors that influence strategic decision-making and resource allocation, including the extent to which research features in such decisions. The latter half of the chapter considers interviewees’ experiences of commissioning and conducting research studies.
2.1 Considerations that have influenced decision-making and resource
allocation
The role of research evidence
Most of the interviewees across the three groups – chief officers, PCCs and CSPs – cited research evidence as one factor in their decision-making. However, the frequency with which it was drawn upon varied widely across interviewees. We identified four broad typologies of research-users – those who had embedded it in their decision-making (the ‘research savvy’); those who used it regularly (the ‘regular users’); those who recognised its importance but drew on it infrequently (the ‘limited users’); and an outlier who operated in a near research vacuum, favouring his professional judgement instead (‘the research indifferent’). The various categories are explored further in 2.2.
The survey findings broadly support the qualitative analysis: around three in five (57%) respondents felt that research evidence played an important role in their day-to-day decision-making6, while two-thirds (68%) reported that research evidence had changed or influenced their working practices. Most (72%) survey respondents reported at least one occasion during the previous 12 months where research evidence had affected how they had allocated resources. Almost as many said that research had helped them to: develop new practice (70%); assess the impact of current practice (67%); better understand a crime problem (63%); and justify existing practice (62%).
We have recently had a conference about evidence based policing set up by the ACC. Since then I have certainly seen an increase in its use to test decision-making. (Survey#678, Chief Superintendent/civilian staff equivalent)
6 In other words they stated that they “Agree” or “Strongly agree” with the statement “Research
13
Though superintendents (77%) were more likely to report having used evidence in the last year to inform the allocation of resources than chief inspectors (69%) and chief superintendents (72%), these differences were not statistically significant (χ²(2,
N=537)=3.83, p=0.147). (None of the other survey respondents provided an answer to this
particular question.) Chief superintendents (77%) and other (79%) respondents were more likely to have reported using evidence to develop new practice than chief inspectors (66%) and superintendents (68%) (χ²(3, N=655)=8.31, p=0.040). Other respondents were also more likely to report having used research evidence in the last 12 months to justify existing practice (77%), when compared with chief inspectors (59%), superintendents (58%) and chief superintendents (53%) (χ²(3, N=655)=16.0, p=0.001). There were no significant differences observed between these different roles and the use of evidence to assess the impact of current practice, or to better understand a crime problem.
The survey asked respondents about how they access and use published research evidence. One in four practitioners said that they rarely (24%) or never (2%) seek out or
look for research evidence in order to inform decisions about policy or operations. The most
common response was for this to happen sometimes (47%), with fewer saying this occurred very often (24%) or always (3%).
A similar picture emerged in relation to the actual use of research evidence to inform decisions about policy or operations, with half (52%) saying they sometimes used research for this purpose, one in five reporting that they very often (18%) or always did (2%), but with around one in four stating that they rarely (26%) or never (3%) used research evidence for this reason.
There was considerable overlap and correlation between these responses, with four in five (82%) of these frequent users of research having also drawn upon this evidence to allocate resources in the last year (as discussed above). Similarly, 88 per cent of those who reported having sought out research evidence (sometimes, very often or always) also reported actually using this material to then inform their decision-making (again sometimes, very often or always).
We found some – but not many – indicators of the types of people from the sample who make little use of research; 35 per cent of chief inspectors rarely or never using research evidence, compared to 29 per cent of superintendents, 23 per cent of chief superintendents and 15 per cent of other survey respondents (χ²(3, N=653)=16.68, p=0.001). Those who used research rarely were also more likely to have longer lengths of service (mean 22.4 vs.
14
20.5 years, t(369.3)=-2.87; p=0.004). This is unlikely to be a product of the fact that senior staff placed more of an emphasis on research, since there were no significant differences in the average service length reported by chief inspectors (mean=21.6 years) superintendents (22.4) and chief superintendents (21.6) (F(2, 527)=.75, p=0.474). Men were more likely than women to use research evidence rarely to inform their decisions about policy or operations (31% vs. 23%; χ²(1, N=647)=3.97, p=0.046), while those currently studying towards a qualification were less likely to do so (16% vs. 30%; χ²(1, N=528)=5.79, p=0.016). Age of respondents, size of their force7, and educational level appeared unrelated to levels of research usage. As we shall discuss, levels of research usage appear to be as much a function of people’s current role as their background.
Other factors influencing decision-making
Beyond research, a broad range of other and often competing influences on decision-making were highlighted by our interviewees. These included financial considerations, national guidance, public opinion and confidence, professional judgement, practice-based evidence8, local intelligence and data analysis, local policing and partner priorities (including those of elected representatives at council and/or PCC level) and national developments (e.g. elections, legislation etc.).
Survey respondents, similarly, identified an average (median) of eight different information sources which they had reportedly used on a routine basis during the course of the previous 12 months to inform their day-to-day decision-making (mean=8.5; range=2-19; SD=3.2). The most common sources of information were practice-based experience/professional judgement (81%), local force guidance/publications (80%) and input from local colleagues/staff (77%) (see Figure 1). There was no relationship between age, gender or length of service and the number of information sources used to inform day-to-day decision-making. There were however significant differences observed based on the survey respondent’s role, with chief superintendents drawing upon a wider range of information
7
This was a binary (yes/no) variable to indicate whether the respondent belonged to a larger or smaller force, based on the number of uniformed police officers serving in 2013, and taking the mid-point within this range as the cut-off. The number of officers in the 23 larger forces responding to the survey ranged from 1,902 to 29,755. The corresponding range for the 23 smaller forces was 760 to 1,827. Those respondents belonging to one of three non-territorial forces (n=14) or reporting their force area as ‘other’ (n=32) were assigned as belonging to smaller forces.
8
We use this term to refer to the structured (and often quantitative) analysis that in-house analysts carry out. It can sometimes be of good or excellent quality, but is neither placed in the public domain through publication, or subject to independent quality assurance.
15
sources (mean=9.5) than chief inspectors (mean=8.2) (F(3, 651)=2.88, p=0.035)9. Those educated to at least first degree level also reported using a wider range of information sources (mean 8.7 vs. 8.0) (t(470.5)=-2.57; p=0.011), as did those currently studying (mean 9.3 vs. 8.3) (t(527)=-2.36; p=0.018).
Returning to our in-depth interviews, the police, PCCs and CSP managers placed emphasis on different factors in their decision-making. Chief officers tended to focus on intelligence and data analysis – including ‘demand’ with respect to resource allocation – as well as practice-based ‘evidence’ – i.e. what was going on in other areas. National guidance or directives from the Home Office and Her Majesty's Inspectorate of Constabulary (HMIC) was another key factor. This group also commonly mentioned the influence of their PCC in shaping force strategy. The role of professional judgement was expressly highlighted by some, but others emphasised that they no longer solely relied on this; nevertheless the implication was that it continued to be a salient factor.
Several of our PCC interviewees spoke about the importance of public opinion in their decision-making, stressing their elected status made them an important conduit for public opinion; some for example, mentioned their impending re-election. There were, however, conflicting views as to whether public opinion was or should be a main consideration in determining strategy, as this quote illustrates:
Things might be popular. Anti-social behaviour might be something people are particularly bothered by but actually it’s not a major issue. Anti-social behaviour in itself might cause people a lot of grief, [but] where do you invest money, those who shout the loudest or where you know there are real problems? (PCC 5)
In all CSP interviews, the key factor in decision-making was their local strategic assessment, which aggregated local intelligence from partner agencies to identify priority areas of work (e.g. domestic violence). For most, the outcome of this was for the CSP to take action to better understand the nature of the problem and explore possible solutions, which involved consulting the academic literature, commissioning further research and networking with their CSP peers around the country to identify models of good practice. A minority relied predominantly on local data, or public opinion to inform their strategies.
9
Adjustments were made for multiple testing within all ANOVA analyses using the Bonferroni correction (i.e. α = 0.05/number of tests).
16
2.2 Influence of research on decision-making and resource allocation
As detailed above, our interviewees fell into four broad categories in terms of their use of research in decision-making. For ease of understanding, we have presented the categories in the figure below:
As noted in section 1.3, our interviewees would have been categorised very differently had we used a different criterion to that of research use. To demonstrate this, we also coded the interviews on experience of commissioning only and found that the majority (n = 38) had done so on one or more occasions. Of those who had no experience of commissioning research, eight were participants in regional partnerships with universities, with whom they collaborated on research. This included supporting research students to conduct academic studies in their force or office. Only three interviewees had neither commissioned research nor engaged in research partnerships of this kind.
‘The Research Savvy’
At the top of the ladder, were ‘the research savvy’; we categorised only six interviewees in this way, so this group were very much outliers. Notably four of the six ‘research savvy’ were chief officers, whilst only one PCC and one CSP manager were so classified.
Research Savvy Limited Users Regular Users (2 sub-categories) Research Indifferent
Typology of research users
Research Savvy
- Research routinely used in decision-making
- Research integral to practice - Engagement in wider research activities e.g. funding for staff to attend conferences Research Indifferent - No use of research - Prioritisation of professional judgement in decision-making
- Apathetic about the value of research
Limited Users
- Little/no use of research evidence
- Experience of commissioning but minimal use of published academic work
- Some conflation of research evidence with data analysis/professional judgement
Problem-driven User
- Use of research limited to specific problems
Borderline Research-Savvy
- Use research as a problem-solving tool
- Good research awareness - Interest in research e.g.
17
For the ‘savvy’, research was integral to their approach to policing and crime reduction. This group routinely used research in their everyday practice, both explicitly and implicitly. There was thus both an expectation that all decisions would be grounded in the academic evidence:
It’s fundamental to me as a commissioner for my commission to use research… nothing comes to me for funding, nothing comes to me for agreement unless they can show that there's a clear body of evidence that illustrates this works or it’s worth us investing some money to see whether it will work in this particular area. (PCC 5)
And a sense that research knowledge tacitly informed discussions and the general style of policing or crime reduction practice:
I know all the Chicago stuff, reassurance policing styles, the work that the National Policing Improvement Agency [NPIA] did about four years ago on what builds confidence in terms of engagement and the sensitivities there … That’s a big area where we don’t necessarily say, ‘OK, this is what the research says …’ but that would be implicit in our discussions, not just because it’s politically attractive. That’s an example of where our accumulation of evidence over the years would lead us to a certain sort of policing style. (CO 24)
These interviewees also engaged in wider research activities, including convening research seminars for staff, funding staff to attend an annual criminology conference to build links with academics, and, in one case, establishing an independent research institute to aid their evidence-based practice.
Despite (or perhaps because of) their research knowledge, they perceived that they still had further work to do, to fully embed the use of research in decision-making and to fulfil their aspiration of creating an ‘evidence-based organisation’. In this respect they cited the same challenges to making greater use of the research as other interviewees, including the accessibility of evidence and scepticism about its value.
The survey results broadly support this conclusion, showing that just two per cent of the 655 respondents ‘always’ used research evidence to inform their decisions. However, it is important to note that the majority of survey respondents were of a lower rank than interviewees.
18
‘The Regular Users’The overwhelming majority of our interviewees in each professional group fell into this group. Interviewees were further categorised into two sub-groups: the ‘borderline research savvy’ and the ‘problem-driven research user’. The ‘borderline research savvy’ group comprised approximately a quarter of regular users. This group predominantly used the research evidence as a problem-solving tool; however they demonstrated a higher level of personal research awareness than the problem-driven research users and reported reading research in their own time. Most significantly, this group demonstrated a strong aspiration to embed research in decision-making. This ambition was both explicitly expressed by interviewees and evident from some of the examples they highlighted, which included convening academic research seminars for staff.
They’re now empowered to draw out the intelligence and the research. Accessing the research now is becoming part of the cultural norm... There’s a growing confidence. ...I’ve had seminars for middle-ranking and senior leaders and they’ve been exposed to, say, [named individual] doing a briefing about evidence-based approaches to policing. I’ve had [named individual] talking about domestic abuse and we’ve had victim workshops based around What Works...Is it [use of research] embedded fully? No. Will we have nay-sayers who try to undermine it? Yes. But a sense of momentum is building both in our force and at a national level too. (CO 3)
The ‘problem-driven research users’, which encompassed most interviewees in this regular user category, drew on the research evidence but their use of it tended to be limited to specific issues – usually for significant emerging problems or ones that had proved intractable:
We don’t obsess about it but I think it is a part of our problem-solving. Whatever problem we’re dealing with, I’m pretty confident we’ll look around widely and that means research evidence as well as local things – what works in the region, other Forces, other partners. (CO 15)
This presence of a large group of frequent research-users is borne out by the survey results. Half (52%) of respondents said that they "sometimes" used research evidence to inform decisions about policy or operations and one in five said that they used it "very often" (18%).
19
‘The Limited Users’A significant minority of interviewees (n= 16) demonstrated research awareness, but were only able to cite one or two examples where they had drawn upon the research evidence or commissioned research studies.10 Most of this group had commissioned research but made little use of published academic work. They favoured evidence-based decision-making but the main sources drawn upon were local data analysis and practice-based evidence from their organisation and others:
I’m about to police an event so part and parcel of my decision-making...was based on the available evaluation of the previous policing event and how it worked. I don’t think that will be seen in research terms at the moment, but I think it is, if you look at it in its kind of broadest terms...So I think on the operational side, our use of the grey [literature] and our own organisational learning – is kind of - on the basis of national work - good I think. In terms of the broadening out into the traditional formative academic research, it could do better. (CO 7)
While some of this group conflated data analysis and practice-based knowledge with research evidence, others appeared to have a good understanding and appreciation of research.
‘The Research Indifferent’
Only one of our interviewees appeared ‘indifferent’ to research: he made no use of the research, demonstrated no experience of commissioning studies and was apathetic about the value of research. The primary influences on this interviewee’s decision-making were his professional judgement and public opinion.
I did all the seminars but I couldn’t be bothered to do the essays to be honest. I thought I’m too busy catching crooks. I was a senior policeman by then. My role was to lead my officers and the community. It was not to take time off or spend my spare time getting some qualification that would not necessarily make me more effective. (PCC 8)
Although our interview sample located only one person who was ‘research indifferent’, the existence of this group of outliers is also evident in the survey findings: three per cent
10
When categorising interviewees we considered their use of research evidence and experience of commissioning as well as their general research awareness demonstrated in the interview.
20
reported "never" using the research evidence. As people who are indifferent to research are almost by definition unlikely to participate in evaluation research, it is reasonable to assume that we have underestimated the size of this group.
Survey findings regarding use of research evidence
In contrast to the qualitative findings, almost three-fifths of survey respondents felt that investment in crime reduction and prevention was driven by politics rather than research evidence (56%), and two-fifths agreed that while research evidence was important, it was not as important as judgement and experience in making decisions (43%). There was no relationship between the view that investment in crime reduction is politically-driven rather than research-driven and any of these variables: age, gender, rank, level of educational attainment, current involvement in studying or length of service.
Despite a large minority (44%) of survey respondents stating that evidence-based approaches were promoted by influential figures or leaders within their organisation, half felt there was no organisational emphasis on the use of research evidence to inform decision-making (51%). Only one in four felt their organisation provided sufficient support and resources to implement evidence based practice (25%), or encouraged collaboration with different research institutes in tackling crime reduction problems (25%). Other respondents (36%) were more likely to report receiving encouragement to collaborate than chief superintendents (34%), superintendents (23%) and chief inspectors (20%) (χ²(3,
N=655)=15.3, p=0.002). Force size, by contrast, had no bearing on respondents' perceptions
of encouragement to collaborate.
It's so variable from force to force and depends heavily upon the CC/ACPO team as to whether evidence-based practice is valued or not. It is not something that fits within my own force's strategy at all and has been ignored in favour of personal opinion in some significant decisions. (Survey#518, Superintendent/civilian staff equivalent).
Nobody wants to be associated with anything that can be construed as failure. Centrally we need a mechanism that provides real financial support for evaluation and makes it easier to incorporate into practical policing. We need to find a mechanism to separate evaluation from performance management and develop a culture which accepts that some initiatives do not work as well as hoped, but rather than reject them as failure, to adapt and learn from the experience. The only true failure is failing to try in the first place. (Survey#818,
21
There was no relationship between gender, age, force size or length of service and perceptions of insufficient organisational emphasis on the use of research evidence. However, chief inspectors (55%) were significantly more likely to express this view when compared with superintendents (54%), chief superintendents (46%) and other (41%) colleagues (χ²(3, N=654)=8.35, p=0.039).
2.3 Experiences of commissioning, conducting and participating in research
Experience of commissioning research was widespread amongst interviewees. Only five interviewees (one PCC, two chief officers and two CSPs) across the three groups had no experience of this. However, the nature of commissioning varied widely. First, there were distinct differences in the topic focus of commissioned work. Much of the research commissioned by PCCs centred on victim’s experiences of the criminal justice system and their needs. A number of PCCs had joined forces to fund a piece of research on this subject and several others had commissioned related studies, such as the victims of domestic violence. It is important to note that responsibility for commissioning the majority of practical and emotional support services for victims of crime was transferred from national government to PCCs on 1st October 2014 (Ministry of Justice, 2013). Therefore PCC interest in commissioning research in this area had also been influenced by this national priority. Chief officers commissioned research in a variety of areas including tactical policing (e.g. hot spots, predictive policing and use of body worn cameras), organisational matters (e.g. demand management and mobile data solutions), and police service consumers (e.g. victim satisfaction and public confidence). Similarly, the focus of CSP commissioning was research into thematic areas (e.g. evaluation of custody triage) and organisational issues (e.g. determining agency roles in preventing and addressing domestic abuse).
Chief officers were more involved than the other two groups in active academic research – usually randomised controlled trials (RCT) – and were often conducting these themselves with some technical input from universities. For example, one CSP had directly commissioned an RCT through the partnership (another CSP and one PCC had RCTs ongoing in their areas but these had not been commissioned through their organisations), whereas five of the 29 chief officers were engaged in RCTs. Notably CSPs more commonly commissioned research from independent local ‘consultants’ rather than universities. One CSP interviewee said that this was due to the relative expense of university research while several others felt that universities lacked the necessary in-depth understanding of local issues. Two CSPs reported disappointment with the university research they had
22
commissioned, citing slippage in the delivery of research outputs and failure to meet interviewee recruitment targets
.
Research collaboration rather than formal commissioning was examined in the survey and 75 per cent of respondents reported no emphasis on collaboration with external partners for research purposes. Between one-fifth and a third of interviewees in each of the professional groups drew upon university students to conduct research. This practice encompassed those at undergraduate, master’s and PhD level, including those working in the police force. In some cases – mainly amongst chief officers and some PCCs – local students were strategically matched to knowledge gaps by means of recently established local and regional research partnerships with universities and colleagues from neighbouring areas. Many of those who were not currently using university students in this way aspired to do so. The ‘commissioning’ of ‘cheap’ local students was seen by many interviewees as an area ripe for development in a time of fiscal constraint.
2.4 Ways of thinking about research evidence
The low proportion of eligible people who responded to the survey (see Methods section above) clearly creates difficulties in deriving measures of ‘research savviness’ to use in our ‘before and after’ evaluation design, because we have only limited confidence in the representativeness of our ‘before’ survey. With this in mind we have undertaken further analysis of the survey with a view to identifying the main constructs – or concepts – which respondents used in thinking about the use of research evidence in their organisations. One aim of this work was simply exploratory and interest-driven; the other was to see if an easy-to-use audit tool could be devised to detect changes across area and time in the orientation of the police and related organisations to evidence. This analysis – described in more detail in Appendix C – used the data reduction techniques of exploratory and confirmatory factor analysis to see if there were patterns in responses to the survey items that point to the main concepts that underpin respondents’ thinking on research evidence. We found clear evidence of three constructs or dimensions:
Levels of organisational support and cultural commitment to evidence-based practice
Personal skills and capacity as research users
23
These three dimensions – measured by 18 questionnaire items – are inter-correlated, but not tightly correlated. That is, people with research skills and commitment to evidence based practice may find themselves working within an unsupportive organisational culture – or vice versa.
24
3. BARRIERS, CAPACITY BUILDING AND KNOWLEDGE GAPS
In this section we examine the barriers cited by interviewees to greater use of research. In addition we report on the opportunities available to interviewees and survey respondents to keep abreast of emerging research findings and to synthesise and disseminate findings more widely within their organisations. Finally we detail the range of knowledge gaps highlighted.
3.1 Barriers to greater use of research
Interviewees were asked what barriers stood in the way of the greater use of research. A lack of time was the most commonly highlighted barrier. As one CSP told us: “It may sound
like a limp excuse but most of the time and due to budget cuts, you’re running just to keep up, never mind keep abreast of research, particularly when so much is coming out all the time”.(CSP 5) Many, however, took steps to consult research despite these challenges: “It’s something I see as being my job to do and I’ll make time to do it even when I don’t have that time” (CSP 5). The perceived lengthy time to complete and disseminate research studies
was also reported as a barrier because such timescales would fail to meet the need for immediate results:
The problem with evidence-based policing is that it’s no use running trials where it’ll be four or five years before we see any useful results. Currently the police are very task-focused and are dealing with crisis management a lot of the time: we have a problem, we put resources in, we get the fix and then we move on to the next problem. (CO 14)
This was echoed in the survey findings: while three-quarters of the crime prevention practitioners questioned felt that research evidence should be used more when allocating resources (75%), just over half (54%) stated that they lacked the time to be able to seek this research evidence out. While a higher proportion of chief inspectors (58%) reported time constraints as a barrier to sourcing evidence, relative to superintendents (53%), chief superintendents (51%) and other (47%) survey respondents, these differences were not statistically significant (χ²(3, N=655)=4.40, p=0.220).
25
Financial considerations were another frequently mentioned barrier to greater use of research. CSPs, in particular, reported that local budget reductions had significantly impinged on their ability to commission research:
...so previously we have done evaluations on new pieces of work, which we have then commissioned from local universities, so it is independent and external. Or we have traditionally, over the three years of this strategy cycle, identified key projects within that… evaluated each of those, so by the time we get to the start of the next year’s cycle we have got a really good idea about effectiveness and the outcomes… has it made a difference? Is anybody better off? What impact has that made? Now I can’t do that. Now I don’t do that. I don’t even bother asking anybody for money to do it, because I know what the answer would be.
(CSP 7)
In a similar vein, some interviewees said that budget cuts had resulted in the loss or significant reductions in staff research capacity – also discussed in Section 4. Nevertheless, a number of interviewees felt that the financial climate had created impetus to make greater use of the research since “we can’t afford to make the wrong decisions”.
A related challenge to that of time and resource capacity, was the accessibility of research. Some interviewees described “just trawling round on Google trying to find pieces of
research” and “being frustrated by my inability to find it [an academic study]”. There was
consensus that research would be more readily used if it was more accessible: “if it
[research] was handy, then I think people are much more likely to at least look at it” [CSP].
We were also told that greater use of research was hindered by difficulties in interpreting its meaning and practical relevance. Interviewees described struggling to decipher the language of academic publications:
I am kind of struggling with some of that language when you are not immersed in it on a daily basis. It is like, ‘God. What is it saying? What does this mean?
(CSP 7)
Similarly, others commented that research studies needed to provide clearer direction on the implications of their findings:
It tends to be a bit too remote. With full-on academia, someone needs to put a layer of simplification on it and say, ‘OK, here’s three good reasons why you need to do this from here on in’. (CO 10).
