The usual roles and responsibilities associated with the preparation of the System Test Specifications and the conducting of the associated tests should be defined in the Test Strategy, as listed below. Note however, that these roles and responsibilities may be changed or shared, according to the specific requirements of a project.
Specifically, the role of supplier may be fulfilled by a user internal function such as IT, Information Systems, Internal Support or an Engineering group.
18 Testing Computer Systems for FDA/MHRA Compliance
In addition, the contractual relationship and/or a good long term working relationship may allow the supplier to assume more of the responsibilities usually associated with the role of the user.
There is also an opposite to this situation, where there is a new working relationship, or where the Validation Plan requires the user to put additional validation activities in place in make up for deficiencies in the supplier’s quality system. In this case the user may perform more of the supplier’s traditional role, or it may require that the user conduct more tests than would usually be the case.
The key roles and responsibilities are usually assigned as summarised in Table 5.1 and in the corresponding explanatory text.
Table 5.1 – Summary of Testing Roles and Responsibilities
5.5.1 Supplier
It is the responsibility of the supplier to:
• Develop the Project Quality Plan that identifies the need for supplier specific testing.
• Develop the Hardware Test Specification (if appropriate).
• Develop the Software Module Test Specification (if appropriate).
• Develop the Software Integration Test Specification (if appropriate).
• Develop the Package Configuration Test Specification (if appropriate).
• Develop the System Acceptance Test Specification (if appropriate).
• Physically prepare for the actual tests.
• Conduct the appropriate tests (including recording the results and any retesting as required).
The Test Strategy 19
Supplier User
QA IS/IT Test PM Validation IS/IT Project PM
Team Team
5.5.2 User
It is the responsibility of the user to:
• Define the need for the various System Test Specification(s) (usually in the Validation Plan).
• Physically prepare for those tests that will be performed on site.
• Assist with those System Acceptance Tests that will be performed on site.
• Witness the System Acceptance Tests (and any others that may need to be witnessed).
This may be no more that identifying that the System Test Specification(s) are a deliverable of the supplier and the user may choose to delegate all further responsibility to the supplier. This may be acceptable in the case of a reputable supplier with whom the user has worked before.
The System Acceptance Tests are the first major test of overall functionality of the system and it is usual for the user to witness the System Acceptance Tests in order to verify that the system to be supplied meets the agreed Functional Specification.
Where a user chooses not to witness some or all of the System Acceptance Tests the following may suffice as an acceptable alternative:
• Review and/or approve the final System Acceptance Test Specification prior to the System Acceptance Tests commencing.
• Review the results of the System Acceptance Tests and associated documentation as part of the Operational Qualification.
Where the supplier audit has revealed deficiencies in the supplier’s testing regime the user may choose to review/approve other Test Specifications and/or witness additional tests (either at the premises of the supplier or on site). These may include the Software Module Tests, the Software Integration Tests, the Package Configuration Tests or the Hardware Tests.
5.5.3 Supplier Quality Assurance
It is the role of the supplier’s Quality Assurance function to ensure that:
• The System Test Specification(s) are written in accordance with the Project Quality Plan.
• The System Test Specification(s) are approved (by the supplier and, if required, the user) prior to the corresponding level of System Testing commencing.
• The System Tests are conducted in accordance with the requirements of the corresponding Test Specification, including the recording of results.
• System Tests are only performed once the prerequisite Tests have been completed and signed off.
• The completed System Tests are fully signed off by appropriate supplier (and possibly user) personnel.
5.5.4 User Compliance and Validation
It is the role of the user’s Compliance and Validation (C &V) function to ensure that:
• The Test Strategy is appropriate to the GxP criticality of the system and the size and complexity of the system
20 Testing Computer Systems for FDA/MHRA Compliance
• The need for the various System Test Specification(s) (and associated tests) are clearly defined in the Validation Plan or Test Strategy, along with the scope and outline content of the Specification (or the reasons for omitting or combining them).
• The need for and nature of reviewing and approving System Test Specification(s) and witnessing the System Tests by the user are clearly defined in the Validation Plan (the rationale for the review? Who reviews it? When do they review it? How do they review it?
How they formally reject/accept? Who witnesses the tests? Who formally accepts the results of the tests?).
• The System Test Specification(s) are traceable to the Validation Master Plan, the Project Quality Plan, the Test Strategy and the corresponding System (Design) Specification.
• The level of user involvement in conducting/signing off the System Tests is clearly defined and justified in the Validation Plan.
• The acceptable level of System Test documentation is clearly defined in the Validation Plan or Test Strategy (details required, authorised signatories allowed to sign off the tests etc).
• The need to review the System Test documentation as part of the Qualifications (IQ, OQ and PQ) and the degree of the review is clearly defined in the Validation Plan or Test Strategy (including who reviews it, when and how they review it and how they formally accept or reject it).
5.5.5 Project Manager
It is the role of the user’s and supplier’s Project Managers to ensure that:
• All of the documentation required by the user’s Validation Plan, the supplier’s Project Quality Plan and the Test Strategy is developed in a timely and properly sequenced manner and to the required standard:
– The System Test Specification(s) – The System Test Sheets(s) – The System Test Result(s) – Incident Reports (if required)
• All hold points are properly observed, and that user reviews are conducted before moving on to subsequent (dependant) phases of the project life cycle.
• The review of System Test Specification(s) is conducted prior to conducting the corresponding System Tests.
• The System Tests are conducted in a timely manner, all results are properly recorded and any necessary retests performed and signed off prior to moving onto subsequent tasks.
• Testing integrity is not compromised due to budgetary or time constraints.
5.5.6 Information Systems and Technology
It is the role of the supplier’s Information Systems and Technology function to ensure that:
• The necessary test facilities and infrastructure are available to allow the System Tests to be conducted (i.e. network infrastructure, printers, test equipment, simulation software).
• The System Tests are properly supported as required (with regards to resources, facilities, witnesses etc.).
It is the role of the user’s Information Systems and Technology function to ensure that:
The Test Strategy 21
• The necessary test facilities and infrastructure are available to allow the Site Acceptance Tests to be conducted (i.e. network infrastructure, printers, test equipment, simulation software).
5.5.7 Supplier Software Test Team
It is the role of the supplier’s software testing function (the ‘Test Team’) to ensure that:
• The System Test Specification(s) are developed in a timely manner, and in accordance with the requirements of the user’s Master Validation Plan and the supplier’s Project Quality Plan.
• The System Test Specification(s) are submitted to internal review and approval as per the supplier’s Project Quality Plan (and if required, by the user as per the user’s Validation Plan).
• The System Test Specification(s) are traceable to the corresponding System (Design) Specification, the user’s Validation Plan and the supplier’s Project Quality Plan.
• Formal System Tests are conducted in a timely manner, in accordance with the corresponding System Test Specification(s).
• The results of all formal System Tests are recorded in accordance with the requirements of the corresponding System Test Specification(s).
• Any necessary retesting is conducted in a timely manner, in accordance with the requirements of the System Test Specification(s).
• All System Tests are signed off in accordance with the requirements of the System Test Specification(s).
• Incident reports are generated for any exceptional results or circumstances that are likely to have a wider knock-on effect and will need further consideration.
Note that it is good testing practice on large projects for one set of developers or engineers to develop the System (design) Specification(s), a different team to develop the System Test Specification(s) and possibly a third, independent team to conduct the actual tests.
This ensures that the System Testing is sufficiently thorough and that the expectations and preconceptions of software designers will not impact upon the conducting of the tests.
This is not always possible on smaller projects, but the preparation of good quality System Test Specification(s) will minimise any negative impact from using the same developers/
engineers to both develop and test the functional design.
5.6 Relationships with Other Life Cycle Phases and Documents (Inputs and Outputs)