Change Control and Configuration Management

54  Download (0)

Full text



Los Angeles Chapter

Change Control and Change Control and Configuration Management Configuration Management

Jerry Anderson


Bio (or,

Bio (or, ““Who is this guy?”Who is this guy?”))

 Director, Watson Corporate Computer Compliance

 24 years doing/managing IT (13 in GxP-regulated environment)

– Development of validated applications

– Network design, operations, & qualification

– System, database, & data center administration – Global Part 11 program management

 IT instructor

 BS Computer Science; MBA; ASQ CSQE  Soccer referee; Ultimate Frisbee junkie


Agenda Agenda

Configuration Management

 Change Control

 Making It Work for You

“Those who do not remember the past are condemned to repeat their mistakes.”


What is Configuration Management? What is Configuration Management?

 CM is a discipline for managing and controlling the evolution of a


– During development – In production

– During maintenance & use – Through retirement

 CM identifies and documents the functional and physical

characteristics of a system, and controls changes to those characteristics


CM: The Bottom-Line CM: The Bottom-Line

CM is the process you use to understand,

document, and control the ingredients of your


Some Relevant References Some Relevant References

 ISO/IEC 12207, Standard for IT Software Life Cycle Practices  IEEE 828, Software Configuration Management Plans

 IEEE 1042, Software Configuration Management

 BSI BS 6488, Configuration Management of Computer-Based


Why Do CM? Why Do CM?

 To manage a system well, you have to know how it’s built

 In order to know what you’ve got after a change, you have to

know what you had before the change

 To find & fix a problem, you usually have to know in some detail


Why Do CM? Why Do CM?

 Configuration management gives visibility into the state and

status of a system

 Such visibility is important during:

– Development, to end up with the right thing

– Production, to keep systems & services available

 CM answers the following questions:

– What constitutes “the system” at any point in time? – What changes have been made to get us here?


What Inspectors Will Look For What Inspectors Will Look For

From recent observations and warning letters:

“Written procedures to differentiate between revision or version changes are not employed.”

“… lack of verification that software modifications validated on the "test" system are identical to the modifications implemented later in the "live" system.”

“There is no documented evidence to indicate that noted problems in the currently distributed software version xxx have been corrected, investigated or tested …”


Configuration Management Plan Configuration Management Plan

 Specifies how you will identify and manage the elements of your


 Outlines how you will handle versioning, baselines, changes to


 If you use automated CM tools, your plan talks about HOW


Formal Configuration Management Activities Formal Configuration Management Activities

1. Configuration Identification

– What do we have, and how is it structured?

2. Configuration Control

– Another way of saying ‘change control’

3. Configuration Status Accounting

– Documents on configuration, versions, change history

4. Configuration Auditing


1. Configuration Identification 1. Configuration Identification

 Identify and uniquely name the components (“configuration

items” or “CI”) of your system

 Give version numbers to each individual CI

 Establish ‘configuration baselines’ (e.g. freezes) by identifying

the structure of how CIs interconnect to form your system

 Making revisions to your CIs or structure causes revisions to the

configuration baseline


Applied Configuration Identification Applied Configuration Identification

System A

V 1.1


V 1.2 V 1.0



V 1.0


V 1.1


V 1.2


A Note on System Baselines A Note on System Baselines

 When you baseline a system, you are ‘freezing’ it

– You’re creating a snapshot of your system

– You know the versions and functionality of all CIs – You know how all CIs interconnect

– You know the version and functionality of the system

 Baselines are good to establish just before:

– Integration testing

– Acceptance / qualification testing


2. Configuration Control 2. Configuration Control

 A process for managing changes to an established system


– Assign change roles and responsibilities – Process change requests

– Specify required change activities – Track change history and approvals – Establish new system baselines

 “Configuration Control” is another way of saying “Change


3. Configuration Status Accounting 3. Configuration Status Accounting

 Simply, a paper trail (or electronic trail) of documentation

showing the history of configuration items and changes to them

 Should include dates, component identifications, versions,

baseline descriptions, change details, information about the people involved

 May be the result of report queries if an automated CM system


4. Configuration Auditing 4. Configuration Auditing

 Also known as “Configuration Evaluation”

 Just what it sounds like: an audit process that evaluates the

effectiveness of the configuration management system

 Reviews the trail generated by Status Accounting

 Verifies that the system’s current configuration matches what


Agenda Agenda

 Configuration Management

Change Control

 Making It Work for You

"Chaos is perhaps at the bottom of everything." - George Santayana


Change Control Change Control


Change Control Change Control

A7 Corsair

DMSP Satellite Dish

True Story:

Defense Meteorological Satellite Program system is removed from ship, but antenna is left on due to CM error. Taxiing jet goes over the side...


What is Change Control? What is Change Control?

 As previously mentioned, this is also known as the

“Configuration Control” activity of CM

 Change control is the process used to request, review, plan,

approve, and implement changes to a system

 When it’s properly implemented, change control assures that

unplanned changes don’t happen, and that planned changes are well-managed


Change Control: The Bottom-Line Change Control: The Bottom-Line

Change control is the process you use to keep

your system in a known good state…

… and the documentation that PROVES you’re

doing it.


Business Drivers Business Drivers

 Change control is expensive, but it makes good business sense:

– Fewer service outages

– Ability to reliably build “add-on” services – Ability to predict the result of changes – Ease of knowledge transfer

– Better reputation for the IT group

 The FDA doesn’t worry about some of these factors, but you


Compliance Drivers Compliance Drivers

 Remember that when they look at your system, the first thing

inspectors will ask for is your validation documentation

 If you validated or qualified the system a year ago, then your

doc set is a picture of the nice, validated state of your system… a year ago

 Inspectors know this, and will want to look at the


What Inspectors Will Look For What Inspectors Will Look For

 Change control:

“ … Failure to establish and maintain a change control system for

changing documentation, procedures, specifications, or modifying the software source code.”

“Your firm fails to classify or evaluate changes, modifications… according to significance.”

“Failure to validate each significant change in the computer software…”


GAMP and Change Control GAMP and Change Control

 CM & CC guidance from various sections of GAMP 4:

– System configuration management

– Change control during system development – Operational change control

 Also references ISO 9000:2000:

– Identification and traceability

– Control of design and development changes – Control of documentation


ISO 9001-3 and Computer Change Control ISO 9001-3 and Computer Change Control

 CM & CC guidance from various sections of 9001-3:

– Configuration management used to identify software, hardware, or

documentation items in all phases of development

– Configuration management is used to control documents and data – Place implementation tools under configuration management


– Procedures must be in place to handle change requests

– Changes shall be identified, reviewed, and approved by authorized


– A configuration management system may be used to handle this


Change Control: Minimum Required Components Change Control: Minimum Required Components

 A quality system

 A written change control procedure

 A change request form

 One or more change review teams

 A computer validation group or function


Quality System Quality System

 Specifies all aspects of QA and QC in your organization

 Comprehensiveness is required because a change control

process alone doesn’t cover all the bases


Vendor management

Training requirements

Employee responsibilities

System operations



Software development


Software quality



Business continuity


Change Control Procedure Change Control Procedure

 One component of the quality system is the change control

procedure and form

– An SOP for the overall change control process – Possibly a “sub” SOP covering computer changes

 Everyone who has the capability to make changes to the

validated target (production) environment must be trained on the change control procedure


Change Review Team: Business Owner Change Review Team: Business Owner

 Decide whether changes are worth doing  Evaluate business justification

 Prioritize requests within the business function


Change Review Team: Technical Representative Change Review Team: Technical Representative

 Assure that change procedures are followed  Classify changes by type, risk

 Determine regulatory requirements and qualification


 Give the go or no-go to make a change

 Determine whether an implemented change should get final


Change Review Team: CV Change Review Team: CV

 Reviews each change to determine the impact to the validated


 Specifies how new functionality is to be tested

 Specifies how old functionality is to be re-tested

 Specifies required documentation updates

Each change to a validated system typically requires a mini revalidation / requalification effort


Change Review Team: QA Change Review Team: QA

 Assures compliance with regulatory requirements and internal

procedures (change control, validation, documentation practices, etc.)

 QA indicates that everybody did their jobs correctly, and that


Change Control Process Flow Change Control Process Flow

Change Request Business Evaluation Validation Assessment Routine? Planning Implement? Reject? Implement & Qualify Success? Reject? Implement & Document Implement? Done Y Y Y Y N N N N

Emergency? Y QA Approval Implement &Document


The Change Control Procedure The Change Control Procedure

1. Change request is submitted 2. Change is evaluated

3. Validation assessment 4. Implementation planning 5. Approval to implement

6. Implementation and qualification 7. Approval to close change

8. Special treatment of “emergency changes” 9. Special treatment of “routine changes”


1. Change Request 1. Change Request

 Anyone can submit a change request

 Information

– Unique change request number – Change requester/owner

– Description

– Reason for change

– Impact and scope of change (systems, processes, locations)

– Does the change require notification to or approval by regulatory


2. Business Evaluation of Change 2. Business Evaluation of Change

 Key stakeholders, system owners, and cognizant managers do

this, helped by the change owner and subject matter experts

– Is the change justified and feasible? – Verify impact and scope of change – Verify regulatory requirements

 “High-level” check to see if the change is worth devoting

resources to

 If yes, communicate the intent to make the change to key



3. Validation Assessment 3. Validation Assessment

 Validation and Quality groups look at the regulated systems,

data, processes in the scope of the change

– What functionality is being added, modified, retired? – What is the relative criticality of this functionality? – What other areas of functionality might be impacted?

 Output of the assessment:

– SQE and SQA activities

– Required testing (new and regression) – Required documentation updates


4. Implementation Planning 4. Implementation Planning

 Using input from the validation group, the change owner and

technical folks plan the change

– Project plan and risk management – Procurements, technology, and tools

– Assignment of technical and compliance activities – Documentation and training

– How to smoothly promote to production – How to roll back the change if necessary


5. Approval to Implement 5. Approval to Implement

 Change board and key stakeholders review the implementation


 Change board gathers input and makes a decision

– Approved for implementation – Rejected

– Go away now; come back with a better plan

 If approved for implementation, the change can be made in the

target production environment when all required activities are complete


6. Implementation and Qualification 6. Implementation and Qualification

 If not yet fully developed, technical work continues here

 Development/integration testing is completed

 IQ is executed/re-executed as required

 Change is promoted to production

 All or part of OQ/PQ re-executed as required

 Summary report of success is written (or, unsuccessful change is


7. Approval to Close Change 7. Approval to Close Change

 The change board meets again to decide:

– If the change worked, should we keep it?

– If the change failed, should we reject it or go back to the drawing

board and try again?

 If the change is to be kept:

– Approvals from change board, QA, validation – Approval, if required, from regulatory agencies


8. Emergency Changes 8. Emergency Changes

 There must be an “out” clause that allows you to make changes

required to fix things that are

– Broken

– Just about to break

 The emergency process:

– Implement and test the fix – Document the change


Agenda Agenda

 Configuration Management

 Change Control

Making It Work for You

“Insanity: Doing the same thing over and over again and expecting different results.”


Controlling Change Volume Controlling Change Volume

 More change = more risk to the system’s controlled state  Limit change by forcing the business/users to choose what

changes are implemented

– Educate users on the compliance reasons for limiting change – Implement chargeback for the cost of changes, or identify the

resources in IT, QA, CV that are available to support changes to establish the ‘change budget’

– Change requests should include a business/compliance justification,

with criteria that prevent “Because I want it!” changes

– On cross-functional (e.g. SAP) or multi-site (e.g. LIMS) systems, set

up a team of business leaders that can prioritize (and reject) changes across functions


Pre-Defined Change Risk Pre-Defined Change Risk

 Systems have a risk designation based on the risk of the process

being automated

 Many systems also have different levels of risk associated with

different system function points. Examples:

– Changing ‘standalone’ code vs. code that’s shared across modules – ERP: GxP vs. non-GxP modules and transactions

 Different types of changes have different risk/impact

 Can be difficult, complex, and time consuming to do up-front,

but will pay dividends in the future when changes force you to requalify the system


Pre-Defined System Testing Pre-Defined System Testing

 If you pre-define risk by module/transaction, you can pre-define

the testing that must occur when the module/transaction is modified

– Pre-define the requalification deliverables that will be required

based on the impact of various change types

– Partition the system OQ into mini-OQs that can be run separately

for regression testing at the right level of granularity to requalify after a change

– Note: adding new functionality always requires new test challenges

 If feasible for your organization, automated testing tools (e.g.

Mercury) can dramatically speed up qualification testing, especially when you have to execute the full OQ


Routine Changes Routine Changes

 A pre-defined list of changes that can be implemented without

‘formal’ change control

 Pre-approved by QA; need no hands-on QA or CV involvement  Typically low-risk/low-impact activities

 Examples:

– Reboot server or restart application/process

– “Like-for-like” hardware changes (e.g. disk in RAID 5 array) – Change start time of batch job


Routine Changes Routine Changes

 Caveats

– Be very clear and specific in your wording to prevent

misinterpretation (e.g. swapping servers is not a ‘like-for-like’ hardware change)

– Some routine changes must still be tested in a non-production


– Some (maybe all) require an SOP to be in place to guide the people

implementing the change

– All require some type of documentation (e.g. work request, system


– History of routine changes must be accessible – Requires periodic auditing


A Note About Change Control Documentation A Note About Change Control Documentation

 Each change is a mini development and qualification effort

 When you’re done with a change, the approved change request

plus associated documentation form a regulatory document

 System change packages should be filed along with the original

system validation/qualification documents

 This combination shows how you got and kept your system in


Get Your IT Group Thinking

Get Your IT Group Thinking “GxP“GxP!”!”

 Odds are good that the areas we’ve discussed are managed by

IT people who’ve had little exposure to CFRs or computer validation

 Now, they will at minimum be responsible for maintaining the

qualified state of your network


Get Your IT Group Thinking

Get Your IT Group Thinking “GxP“GxP!”!”

 First: Enlist IT management support  Second: Educate on the issues

– The requirement for compliant systems and networks – The risks if compliance is not achieved and maintained – What they can do to help

 Third: Training

– Basic GxPs; good documentation procedure

– Required SOPs: validation, change control, operations, security, etc.

 Fourth: Audits!

– Periodically audit IT groups to assure that new employees have


Questions? Questions?