Software
Verification and
Validation
Georgia L. Harris
Carol Hockert
NIST Office of Weights and Measures
Software V&V 2016 1
Learning Objectives
After this session, using resources
and references provided, you will
be able to
List examples of software error impact
Identify which criteria in ISO/IEC 17025
address requirements for computer
systems
Identify examples of software errors
2
Software
Verification &
Validation
What and Why
3
Software V&V 2016
What is Verification and
Validation?
Validation is ensuring "you built the right product“ (SUITABLE)
Validation is testing to confirm that it satisfies
stakeholder needs.
Examples for metrology:
Correct SOP selected, Correct equations, Results are good
Verification is ensuring "you built the product right."
(ACCURATE)
Verification is testing to confirm that a product complies
with its requirements and specifications.
Examples for metrology:
Equations match the SOP, Calculations produce the expected answer
Why do Verification and
Validation?
Ensure measurement result accuracy
Minimize impact of software errors
Ensure records are maintained as
evidence
Meet Requirements
Accreditation to ISO/IEC 17025
FDA – all software (including use of
spreadsheets) must follow “good software
engineering practices”
5
Software V&V 2016
Health Impacts
: U.S. Food &
Drug Admin.
The FDA’s analysis of 3140 medical device recalls
conducted between 1992 and 1998 reveals that 242 of
them (7.7%) are attributable to software failures. Of
those software related recalls, 192 (or 79%) were
caused by software defects that were introduced when
changes were made to the software after its initial
production and distribution. Software validation and
other related good software engineering practices
discussed in this guidance are a principal means of
avoiding such defects and resultant recalls.”
NIST OWM has seen failures in proficiency tests due to lack
of adequate software control and validation on a regular
basis.
4
Financial Impacts
: U.S. NASA
7
Automated Software Verification & Validation: An Emerging Approach for Ground Operations,
David G. Bell and Guillaume P. Brat, NASA Ames Research Center, ©2008 IEEE.
Software V&V 2016
Does it affect you?
Does it Affect You?
9
Software V&V 2016
6
17025:2005 – Sections to Consider
4.1.5.c Procedures exist to protect client’s information.
4.3.1 Procedures to control software
4.3.2.1 Quality system reviewed and approved by authorized personnel (electronic signatures)
4.3.2.2 Authorized editions of appropriate documents all locations. (Intranet, NT file Share)
4.3.3.2 Altered or new text shall be identified (electronic document)
4.3.3.4 Procedures shall describe how changes in documents, including software are controlled.
4.13.1.2 Records (electronic media) shall be stored and maintained so that they are retrievable.
4.13.1.4 Procedures to protect and back-up electronic records.
4.13.2.1 Retain records for the retention period (old versions of software also)
4.13.2.2 Observations shall be recorded at the time they are made. (electronic).
4.13.2.3 Electronic records shall avoid loss to original data (audit trails)
5.4.1 Lab shall have instructions on the use and operation of equipment (and software).
5.4.7.1 Calculations (spreadsheet) and data transfers (tables) shall be subject to checks.
5.4.7.2.a Software shall be validated
5.4.7.2.a Laboratory configurations of COTS software shall be validated.
5.4.7.2.b Procedures are established to protect data.
5.4.7.2.c Computer and automated equipment are maintained.
5.5.2 Equipment & Software shall comply with specifications.
5.5.4 Each item of equipment & software shall be uniquely identified.
5.5.5 Records shall be maintained of equipment & software.
5.5.11 When correction factors are used, procedures shall ensure software is updated.
5.5.12 Software shall be safeguarded from adjustments.
5.10.1 Reports may be issued electronically.
5.10.2.j Reports may contain electronic signatures.
5.10.7 Reports may be transmitted electronically.
11
Software V&V 2016
Laboratory Computer Systems
Assessment
Internal Audit
Quality and Technical
12 Software V&V 2016
Requirements, References
and Records
13Handbook
Requirement
Laboratory
Documentation
References
Evidence of Compliance
4.1.5. 3) or 4.1.5 c. have policiesand procedures to ensure the protection of its customers’ confidential information and proprietary rights, including procedures for protecting the electronic storage and transmission of results;
QM Section 4.2
SAP 10 “Title”
SAP 4 “Title”
Form 10, “Title”
Policy is in place; audit
shows that it is being
followed; Reviewed 3
copies of Form 10 to see
they were completely
filled out. (See
Attachment A).
No information has been
inadvertently released
Software V&V 2016
ISO/IEC 17025: Key Concepts
for Software
Documentation
Quality management system; Standard Admin. Procedure 10
Procedures for Verification & Validation
Checklist = Form A
Validation Process & Records
Data sets
Correct Updates
Calibration data (values & uncertainties)
Controls & Maintenance
Inventory, status
Security, locked cells, passwords
Approvals
It’s not just
about
data
sets!
8
Laboratory Document Control
and Approval
17025:2005, Section 4.3
Document Control
Procedures for development,
V&V, and approval
Inventory of Software (Master
List)
Technical assessment
Records of V&V
15 Software V&V 2016Document Control:
Inventory & Tracking
Sample File available:
http://www.nist.gov/pml/wmd/labme
trology/lab-resources.cfm
16
Quick Quiz:?
How many major sections of
ISO/IEC 17025 cover Software
Verification and Validation?
What is objective evidence?
17
Software V&V 2016
Software
Verification &
Validation
10
Focus: Spreadsheet
Verification & Validation
19 Spreadsheet Verification & Validation
Software QA
(SAP 10)
Requirements
(17025, FDA)
Software Engineering Life Cycle
Specs, Design, Build, Test, V&V, Use +
Documentation
Software Inventory
Form A
17025:2005 Auditing
Forms
Software V&V 2016Standard Administrative
Procedure 10 (SAP 10)
Responsibility and Authority
Software Engineering Essentials
FDA
RP 13
Risk Analysis
SSFM – NPL Best Practice Guide
Methods for Controlling and Evaluating
Software
Inventory
SAP 10, Form A
20
Software Life-Cycle
21
Figure 1. Software Life Cycle
Requirements
Design
Construction
Testing
Documentation
Installation
& Validation
& Maintenance
Operations
Retirement
Purchase
Change Control Process
Consider RISK Assessments
Time
Software V&V 2016Risk Assessment
For additional information on Risk Analysis
practices and procedures, see Validation
of software in measurement systems
(Software for Metrology Best Practice
Guide No. 1), National Physical
Laboratory (NPL),
http://www.npl.co.uk/
. January 2007.
Spreadsheets may not be an appropriate
12
Testing Phase
(SAP 10 has more…)
1.
Analysis without computer assistance
2.
Other validated computer program
3.
Experiments & tests
4.
Standard problems with known solutions
(data sets)
5.
Confirmed published data and
correlations
23
Software V&V 2016
Installation &
Acceptance
1.
Verify complete installation
2.
Ensure correct operation within system
•
Operating System updates, patches,
software updates and potential changes
3.
Document approval for use
•
Who has authority for review and
implementing new system
24
Operations &
Maintenance Phase
1.
Fix – remove errors
2.
Improve – changed specifications;
process improvements
3.
Adapt – operating environment
25
Software V&V 2016
Software
Verification &
Validation
14
Verification & Validation
Techniques
SAP 10 Descriptions
Form A Sample Items
What is evidence?
27
Software V&V 2016
SAP 10: Descriptions
Form A: Review and Evidence
A.- Software Inspection
B.- Mathematical Specification
C.- Code Review
D.- Numerical Stability
E.- Component Testing
F.- Numerical Reference Results
G.- Embedded Data Evaluation
H.- Back-to-Back Testing
I.- Analysis Without Computer
Assistance
J.- Security
Example & Demonstration
Software V&V 2016 29
References & Guides
ISO/IEC 17025:2005
Laboratory Standard Administrative Procedure 10 and Form A
NPL Guides, March 2004
“Software Support for Metrology Best Practice Guide No. 1,
Validation of Software in Measurement Systems”, Includes Risk
Assessment methods.
“Software Support for Metrology Best Practice Guide No. 7
Development and Testing of Spreadsheet Applications”, see
especially section 6, “Checklists for spreadsheet development
and testing”
“General Principles of Software Validation; Final Guidance for
Industry and FDA Staff” (FDA, January 11, 2002)
