4.1 Introduction
Methodology is the systematic analysis of the methods applied to a field of study.[64]
It offers the theoretical underpinning for understanding how a specific piece of research has been carried out. Many treat it as a synonym for method or body of methods.[64]
Doing this shifts it away from its true epistemological meaning and reduces it to being the procedure itself, the set of tools or the instruments that should have been its outcome. [64]
A methodology is the design process for carrying out research or the development of a procedure and is not in itself the instrument.[64]
For Guidelines, there is a standardised way of appraisal and quality assessment of methodology by the use of the AGREE tool.[65]
The Appraisal of Guidelines for Research and Evaluation (AGREE) Instrument (appendix 1) evaluates the process of practice guideline development.[65]
It is a tool that assesses the methodological rigour and transparency in which a guideline is developed and it is used internationally.[65]
The original AGREE Instrument has been updated and methodologically refined. [66]
AGREE II is not only used for appraisal but can also be used to provide a methodological strategy for the development of guidelines.[66]
In addition, it can be used to guide what information and how that information ought to be reported in guidelines.[66]
AGREE II is designed for guidelines developed by local, regional, national or international groups or affiliated governmental organizations.[67]
Amongst the medical profession there are those who question the reasoning behind a guideline developing organisation such as NICE.[61]
Some worry that guidelines produced by non-professional bodies may be intellectually suspect and a means of external control, policing, and removal of autonomy from the professional medical groups. [61]
There is a fear amongst some of the professionals that such guidelines might be a way in which politicians and health-service managers who have made use of the rationale for evidence-based guidelines will use them to make judgements that serve their own political or economic agendas.[61]
In this way they would be using evidence that proves interventions are
effective “on average” but which omit the value of experience and professional insight.[61]
Guidelines have been produced for many years by the Royal Colleges and specialist professional societies.[68]
As the field of evidence based medicine has been evolving guideline development by such organisations has in the past been characterised by problems that potentially undermine the quality and trustworthiness of the guidelines they produce.[68]
This includes lack of transparency, limitations in the process of systematic reviews, but more importantly a failure to use rigorous methodology.[68]
The rapid proliferation of guideline production often means guidelines are produced in parallel either by professional associations with an overlap in interest or by professional associations and national guideline developing bodies.[68]
This, apart from appearing unecessary and wasteful of resources, can potentially lead to confusion amongst the audience when guidelines are produced by different bodies on the same topic with differences in their recommendations.[68]
In the UK National Health Service, all doctors, nurses, pharmacists, and other health professionals now have a contractual duty to provide clinical care based on best available research evidence. [61]
Furthermore, whilst the medico-legal implications of “official” guidelines have rarely been tested in the UK, US courts have ruled that guideline developers can be held liable for faulty guidelines.[61]
In addition, the same ruling states that doctors cannot pass off their liability for poor clinical performance by claiming that adherence to guidelines corrupted their judgment.[61]
The ability to appraise the quality of a guideline irrespective of the authority that has produced it is vital.
In order to do so it is important to be able to access and assess the quality of the guideline development process by which it was produced. This is done by appraisal of the methodology.
4.2 Aim
To research the availability and quality of guideline methodology data that can inform all end users of guidelines produced by NICE on the guideline
development process for the purpose of guideline appraisal.
4.3 Methods
Reasearch on the availability of NICE methodology data was carried out by online data mining of the NICE website according to a predetermined search strategy as outlined below.
The methodology data retrieved was also reviewed for information regarding the quality strategy of the development process. It was specifically scanned for information on whether a standardised tool like the AGREE II, or a different tool or method, was used.
The NICE methodology data retrieved was also further assessed for its availability and quality by comparison with similarly retreaved methodology data from other UK guideline developing bodies.
Online Data mining
NICE methodology data
The search engine ‘google’ was used to search for available data regarding the NICE methodology. The search terms that were used were: ‘NICE’,
‘guideline’, ‘methodology’, ‘development’, ‘process’
The information produced from the search was selected if it answered the following questions:
Is this information about the NICE clinical guideline development process?
Is this information about the quality assessment of the guideline process with the AGREE tool or any other mentioned guideline quality appraisal/development tool?
Is this information about quality assessment of the development process without a standardised instrument?
An overall assessment was finally made by the reviewer regarding the ease of access to the data through the online data mining process.
Methodology data from other UK guideline producing bodies
A systematic search was performed of the information presented on the websites of all current UK surgical societies, associations, and Royal Colleges.
The list of surgical societies to be searched was drawn up by associating a society for each of the known surgical specialties and sub-specialties. One medical society was included on the list as it was felt that this medical specialty through endoscopy practice overlapped with surgical practice and should therefore be included.
Search terms relating to the associations’ titles were entered into the ‘google’
search engine and the precise title and webpage address of the society was identified. In the case that surgical subspecialties were found to have more than one associated society, all were included in the search.
Table 4.1: List of professional societies and Royal Colleges included in the search for methodology data.
Organisation Web address
Royal College of Surgeons of England www.rcseng.ac.uk
Royal College of Surgeons of Ireland www.rcsi.ie
Royal College of Physicians and Surgeons of Glasgow www.rcps.ac.uk
Royal College of Surgeons of Edinburgh www.rcsed.ac.uk
Association of Surgeons of Great Britain and Ireland ASGBI www.asgbi.org.uk Association of Coloproctology of Great Britain and Ireland ACPGBI www.acpgbi.org.uk Association of Upper Gastrointestinal Surgeons of Great Britain and
Ireland AUGIS
www.augis.org
The Vascular Society of Great Britain and Ireland www.vascularsociety.org.uk Association of Breast Surgery UK - ABS at BASO www.baso.org
British Association of Surgical Oncology BASO www.baso.org Association of Laparoscopic Surgeons of Great Britain and Ireland www.alsgbi.org British Association of Day Surgery BADS daysurgeryuk.net
British Hernia Society www.britishherniasociety.org
British Orthopaedic Association www.boa.ac.uk
British Trauma Society www.bts-org.co.uk
www.trauma.org British Association of Otorhinolaryngologists ENT-UK www.entuk.org The British Association of Urological Surgeons www.baus.org.uk British Association of Plastic, Reconstructive and Aesthetic Surgeons
BAPRAS
www.bapras.org.uk
British Association of Aesthetic Plastic Surgeons BAAPS baaps.org.uk Society for Cardiothoracic Surgery of Great Britain and Ireland www.scts.org
British Society of Gastroenterology www.bsg.org.uk
Each website was visited and carefully searched for data on guidelines and guideline methodology. The home webpage was searched as a first step followed by all linked webpages accessible through the home page option lists. All lists and linked pages were searched for data, not just those that immediately appeared relevant to guideline data such as lists referring to
‘research’ or ‘resources’.
If no information on guideline or guideline methodology was available through this route then the home webpage ‘search’ option was used and the key words ‘guideline’ and ‘methodology’ entered as separate searches not as a limited combined search.
If a society website was found to offer guideline documents but had no
information available on the website about the guideline methodology then all guideline documents were opened and their contents page searched for a methodology section. Paper versions of guideline documents were not requested and societies were not contacted via telephone or mail for
methodology data.No formal statistical software was necessary in the analysis of the results as numbers involved were small enough to analyse manually.
4.4 Results
NICE methodology data
Availability of methodology data
The methodology and all related guideline methods and tools are publicly available and easily accessible online through the NICE website and the webpage of the guideline development manual.[69]
The NICE guideline development process [69]
Referral and remit
The Department of Health asks NICE to produce a guideline on a particular topic. The topics for guidelines are based on
recommendations from topic selection consideration panels.The topic referral is also associated with a remit that identifies the broad areas to be covered.
National Collaborating Centres (NCC)
NICE commissions one of the four National Collaborating Centres (NCC) to co-ordinate the development of a guideline.
For the clinical guideline on the management of colorectal cancer this was the NCC for cancer (NCC-C).
This is responsible for developing NICE guidelines for the NHS in England and Wales on treating and caring for people with cancer. The other national collaborating centres are the centre for acute and chronic disease, the centre for women and children, and the centre for mental health.
A management board comprising representatives of relevant professional bodies oversees the work at the NCC-C. This currently includes representatives for a number of Royal
Colleges, the National Cancer Research Institute (NCRI), and other charity and academic bodies.
The management board meets regularly and among its many functions is to oversee the guideline development process for each guideline.It advises the NCC-C on negotiating with NICE on quality issues.
The guidelines team in the Centre for Clinical Practice at NICE supports and advises the NCC during the process.
Guideline Development Group (GDG)
A Guideline Development Group (GDG) is established to manage the work. The GDG is composed of health professionals who are involved in the treatment and management of patients with cancer.
It also includes at least two patient/carer representatives. GDGs usually consist of 12-15 people. NICE is not represented on the GDG.
The scope
The remit is translated into the scope. The scope provides a framework within which to conduct the guideline development work. When developing the scope key clinical issues are selected by the scoping group. This group consists of
representatives of the GDG, NCC-C technical team and NICE.
Stakeholders
Before the scope consultation takes place stakeholders are invited to a scoping workshop to discuss the key clinical issues identified by the scoping group. Stakeholder organisations are organisations with an interest in a particular guideline. They
register with NICE at the beginning of the guideline development process and contribute their views during consultation periods.
In the NICE clinical guideline development process,
stakeholders are: national patient and carer organisations, national organisations that represent healthcare professionals, companies that manufacture medicines or devices used in the clinical area covered by the guideline, providers and
commissioners of health services in England and Wales, statutory organisations including the Department of Health and research organisations.
After a consultation period the scope is finalised. The scope provides information to healthcare professionals, stakeholders and the public about the expected content of the guideline.
Review questions
The key clinical issues listed in the scope are next broken down into review questions. The exact number of review questions for each clinical guideline depends on the topic and the breadth of the scope. However, the number of review questions must be manageable for the GDG and the NCC technical team within the agreed timescale.
For standard clinical guidelines that take 10–18 months to develop (from the time the scope is signed off to submission of the draft guideline), between 15 and 20 review questions is a reasonable number. This number is based on the estimate that, on average, it is feasible for a maximum of two systematic reviews to be presented at any one GDG meeting.
Review questions are usually drafted by the NCC technical team. They are then refined and agreed by all GDG members through discussions at GDG meetings.
The different perspectives among GDG members will help to ensure that the right review questions are identified, thus enabling the literature search to be planned efficiently.
Often the main questions need refining again once the evidence has been searched, and this may generate sub-questions.
Review questions for economic analysis
Questions are selected for economic analysis as a joint decision between the health economist and the other GDG members.
The health economist is a core member of the GDG alongside the rest of the NCC technical team, and is involved at the earliest opportunity and attends all GDG meetings.
The expertise of all of the GDG members is necessary to ensure that economic evidence is underpinned by the most plausible assumptions and the best available clinical evidence. Selection is based on potential value across all key clinical issues, quality of available evidence, and time available for economic modeling.
There are likely to be large differences between clinical guideline topics in the amount, relevance and quality of the economic literature. In some topic areas there may be high-quality data that can be used in economic models, whereas in other areas there will be little information.
Defining the economic priorities for each clinical guideline starts during scoping, and proceed alongside development of the review questions.
Systematic reviewing
A systematic review is carried out for each question. Review questions are broken down into different parts and used to
devise a search strategy using the PICO (patient, intervention, comparison and outcome) framework.
This can be constructed from terms relating to the population, combined with terms relating to the interventions and
comparisons to be evaluated.
The search strategy is discussed and approved by the GDG.
The searches are then carried out by the information specialist.
Core and subject-specific databases are searched. Other sources such as registers are also included.
Before acquiring papers for assessment, the systematic reviewer sifts the evidence identified in the search in order to discard irrelevant material. Next, the remaining abstracts are scrutinised against the inclusion criteria agreed by the GDG.
Abstracts that do not meet the criteria are excluded. Any doubts about inclusion should be resolved by discussion with the GDG before the results of the study are considered.
Once the sifting is complete, full versions of the selected studies are acquired for assessment. Studies that fail to meet the
inclusion criteria once the full version has been checked are excluded; those that meet the criteria are assessed.
Because there is always a potential for error and bias in selecting the evidence, double sifting (that is, sifting by two people) of a random selection of abstracts is performed periodically.
Once a study has been selected it is assessed using a
methodology quality checklist. Data is extracted to a standard template for inclusion in an evidence table.
Meta-analysis may be needed to pool treatment estimates from different studies.
Developing recommendations
In developing recommendations the GDG must decide what the evidence means in the context of the review questions and economic questions posed.
There are many reasons why it can be difficult for a GDG to reach a decision about a recommendation. The literature search may have found no evidence. The quality of the evidence may be poor. There may be conflicting evidence. The clinical evidence may not be directly applicable to the population covered by the guideline.
The GDG may have to consider consensus methods to identify best practice. The reasoning behind all decisions are
documented and presented in the full guideline.
Final consultation and publication
At the end of the process, and after further consultation with stakeholders, NICE’s Guidance Executive signs off the guideline.
The Guidance Executive confirms that the NCC has developed the guideline in accordance with the terms of the remit from the Secretary of State for Health and the scope, and by following NICE’s process and methods.
The guideline is then published and distributed to the NHS in England and Wales.
Figure 4.1: Summary of NICE clinical guideline development process [70]
Quality of methodology data
The following information was identified in the NICE methodology manual on the NICE website relating to guideline methodology quality assessment:
“NICE methodology follows the AGREE II framework. NICE guideline development aims to be a transparent process using the principles of evidence-based medicine.” [46]
“The NICE guideline development process has been drawn on the advice of international guideline development methodology experts, internationally acceptable criteria of quality, the expertise of the clinical guidelines team in the Centre for Clinical Practice (CCP) at NICE, and the experience of the staff at the national collaborating centres (NCCs) where the guidelines are produced on behalf of NICE.”[69]
Methodology data from other UK guideline developing bodies
Availability and quality of methodology data
Table 4.2: Availability and quality of methodology data of UK surgical societies and Royal Colleges.
Society Produce guidelines Guideline methodology
Royal College of of Great Britain and Ireland ASGBI of Great Britain and Ireland AUGIS
Yes – but also refer to NICE and SIGN
No No
The Vascular Society of Great Britain and Ireland
Yes No No Great Britain and Ireland ALS-GBI
Yes No No
British Association of Day Surgery BADS
British Trauma Society No access to website without
The British Association of Urological Surgeons
Yes No No
British Association of Plastic, Reconstructive
No NA NA
and Aesthetic Surgeons Surgery of Great Britain and Ireland
21 website pages were visited and of those 19 allowed access without membership. Of those, 18 were websites of surgical societies and Royal Colleges and 1 was of a medical society of relevance to CRC
Of the 18 surgical society websites 14 produced guidelines for their members (77.8%)
No surgical society website provided information online about their guideline methodology
Of all 21 societies that produced guidelines only 1 society website provided information about their guideline methodology (5.2%). It was this same organisation that reported on their website about using the AGREE tool as a template for their guideline development and reporting
4.5 Discussion
The results show that NICE provides easily accessible information on the methodology of its guideline production. The guideline manual is an extensive and detailed document easily accessible online and clearly describing all
The results show that NICE provides easily accessible information on the methodology of its guideline production. The guideline manual is an extensive and detailed document easily accessible online and clearly describing all