Requirements Based
Functional Testing
Traci Mapps and Mike Santos SLI Global Solutions
State Certification Testing of Voting Systems National Conference
SLI Overview
Accredited by the US National Institute of Standards and Technology as a Voting System Test Laboratory
Accredited by the United States Election Assistance Commission as a Voting System Test Laboratory In depth experience with most voting products used throughout the United States
State, Federal and International certification testing experience
Established expertise and knowledge in testing election systems to certification standards and statutory
requirements
What is Requirements Based
Functional Testing?
A testing approach in which test suites/modules are derived from requirements.
It includes functional tests and also non-functional attributes such as performance, reliability and/or usability.
Considered to be an effective testing approach, used to
significantly reduce the number of undetected discrepancies being released into production.
A testing approach which enables the determination of a necessary and sufficient set of tests which verify that the design and code of a system meets stated requirements.
What is Functional Test?
Functional Testing verifies that the system behaves correctly and functions according to the requirements which specify the intended use of the application.
Functional Testing determines if each component:
o Performs in accordance to the specifications
o Responds correctly to all conditions that may be presented by incoming events/data
o Moves data correctly from one business event to the next
o Initiates events in the order required to meet the business objectives of the system.
What is a Requirement?
A requirement is a capability or function that must
be delivered by a system component or
components.
A single documented functional need that a
particular design, product or process must be
able to perform.
It is a statement that identifies a necessary
attribute, capability, characteristic or quality of a
system for it to have value and utility to a
Characteristics of a Good Requirement
Completeness: The requirement is complete to the extent that all ofits parts are present and each party is fully developed. For example, no TBDs, no reference to nonexistent functions, inputs or outputs.
Consistency: The requirement is consistent to the extent that its provisions do not conflict with each other or with governing
specification and objectives.
Suitability: The requirement is appropriate or suitable for testing voting systems.
Feasibility: The requirement is feasible from a cost and schedule standpoint. Also can a system truly meet the requirement.
Testable: The requirement is testable to the extent that one can identify an economically feasible technique for determining whether or not the requirement has been met. Requirements must be
specific, unambiguous, and quantitative wherever possible.
Characteristics of a Good Requirement
Unambiguous: A requirement is unambiguous only if everyportion of requirement stated therein has only one interpretation
Bounded: The boundaries scope and context for the requirement should be identified
Normalized: Requirements should not overlap nor refer to other requirements or the capabilities of other requirements
Unique: Each requirement should be stated only once
Correct: A Requirement is correct only if every portion of requirement stated therein is one that the software shall meet
The Good, the Bad and the Ugly
Bad Example
The system
must be user
friendly.
How do you measure user friendliness?
8
Good Example
The user interface
shall be menu driven.
It shall provide dialog
boxes, help screens,
radio buttons,
dropdown list boxes
and spin buttons for
user inputs
Test Cycle
Requirements Definition
Requirements-based tests can be no better than the
requirements they are testing. Requirements must be correct, complete, unambiguous and logically consistent.
Functional Decomposition/Partitioning
Functional decomposition/partitioning is the breakdown of a system into its functional components or functional areas and assignment into a given partition for functional testing.Test Cycle
Test Planning
Define the scope, schedule and deliverables
Test Suite/Module Design and Validation
Write test suites and modules to validate the productTest Cycle
Traceability (Traceability Matrix)
Document the links between the requirements and the work products developed to implement and verify those requirements.
Test Suite/Module Execution
Execution of the test suites/modules against the system being tested and documentation of the results defined in the test plan.
Test Cycle
Defect Tracking
Defects detected during the testing process are written against the pertinent requirement(s) and tracked to
resolution.
Coverage Analysis
Determine the coverage of the requirements against test suites/modules and outstanding defects. Determine the percentage of the requirements that are untested,
performing to specification and not performing to specification (defects or discrepancies).
Functional Testing?
Industry research shows that the root cause of 56 percent of all errors identified in projects are introduced in the requirements phase.
Implementing formal requirements can reduce risk.
Requirements based testing can increase efficiency, reduce the risk of failure and improve overall quality.
From a Certification point of view, requirements based testing
creates a consistent baseline from which different voting systems can be consistently evaluated to the same criteria.
Complete traceability between requirements, test suites/modules, and defects provides comprehensive information to help ensure complete test coverage and suitability for certification.
The result is an effective and auditable base for certification testing.
