Quality Assurance Program

(1)

CLEARBALLOTGROUP

ClearVote 1.0

Quality Assurance

Program

Abstract: This document outlines the general quality assurance policies followed by Clear Ballot Group to test the ClearVote product.

(2)

ClearVote Quality Assurance Program

Part Number: 100059-10001

This document contains proprietary and confidential information, consisting of trade secrets of a technical and/or commercial nature. The recipient may not share, copy, or reproduce its contents without express written permission from Clear Ballot Group.

Ballot Resolver, Clear Ballot Group, ClearAccess, ClearAudit, ClearCount, ClearData, Clear Design, ClearVote, Image-to-Ballot Traceability, MatchPoint, ScanServer, ScanStation, Speed Accuracy Transparency, Visual Verification, Visualization of Voter Intent, and Vote Visualization are trademarks of Clear Ballot Group.

Other product and company names mentioned herein are the property of their respective owners.

7 Water Street, Suite 700 Boston, MA 02109

(857) 250-4957

http://www.clearballot.com

Document history

Date Description Version Authors

08/10/2015 Initial release. 1.0 Ed Smith Iris Friedman 08/20/2015 Minor corrections. 1.0.1 Ed Smith

(3)

Chapter 1. Abstract

This section defines the general description of the Quality Assurance Program at Clear Ballot Group. It contains the following sections:

l About this document l Scope of the QA Program l History and intended audience

1.1 About this document

This document outlines the general quality assurance policies that Clear Ballot Group follows in producing the ClearVote™ system.

ClearDesign™ is the first ballot design, proofing, and production system built from the ground up using the latest technological advances in a generation. ClearDesign

makes it intuitive and simple to set up and manage elections and ballots.

ClearDesign has a browser-based user interface that is consistent, streamlined, and easy to use. Election department staff can quickly create elections and then generate, modify, and proof all their ballot styles. In addition, each jurisdiction can customize the terminology used in ClearDesign to match what you they, rather than forcing their teams to learn new terminology.

ClearAccess™ is an accessible voting and ballot marking application that allows voters with sight or mobility limitations to vote in an unassisted manner. The software runs on a touchscreen computer installed with an EZ Access keypad and Origin Sip-and-Puff device. The voter may make ballot selections either by

touching the monitor, pressing buttons on the keypad, or using the Sip-and-Puff. Ballot selections may be presented on the computer's display, played back over audio headphones, or presented both on the screen and in audio format. Once the voter has finished voting, selections are printed to a paper ballot.

From its conception, ClearCount™ was designed to inspire confidence in the user. The vote and ballot visualizations that allow marginal votes to be identified and corrected by authorized officials without requiring extensive retabulation provides a new level of visibility and transparency to reviewing election results. However, those new features would not be valuable unless they were embedded in a program that had been designed, analyzed, and verified to include the highest possible level of quality.

ClearAudit™, the audit product from Clear Ballot Group, has been used in the field, and feedback from election officials and Clear Ballot Group staff has been

(6)

Chapter 1. Abstract

The Clear Ballot Group validates that ClearVote is designed and developed so that it meets functional requirements for a central-count system and is suitable for election purposes.

1.2 Scope of the Quality Assurance program at Clear Ballot Group

ClearVote combines proprietary software for ballot design, tabulation, accessible voting sessions, and results reporting with COTS hardware (scanners and laptop or desktop computers). There are many aspects to the quality program at Clear Ballot Group.

1.2.1 Items in scope of the Clear Ballot Group quality program

The major areas of the Quality Assurance rogram at Clear Ballot Group are as follows:

l The requirements definition, design, and process documentation of the ClearVote software program.

l Determining the specifications that COTS hardware must meets to provide the level of quality (workmanship) necessary for a central count system.

l The requirements and process for integrating other EMS information into ClearCount (primarily creating and validating the Ballot Definition File (BDF)). l The requirements and process for ballot layout in ClearDesign.

l The requirements and process for integrating other EMS information into the ClearVote system (primarily creating and validating the Ballot Definition File (BDF)).

l The validation and verification of the performance, and functioning and integration of the voting system, which consists of COTS hardware, BDFs (adapted from PDFs from other vendors' EMS systems), and internally developed software.

l Validation and verification of the process for installation of ClearCount and ClearAccess on COTS hardware along with the necessary steps to secure the system as would be required in a jurisdiction.

(7)

1.3 Intended audience

1.2.2 Items outside scope

The following items are out of scope of the Clear Ballot Quality Assurance Program:

l Hardware development, manufacture, and testing.

l Pre-delivery tests or maintenance records for third-party hardware. l Development of third-party Election Management Systems.

Information on the hardware manufacturers' quality programs can be found in their documentation. See the Appendix to the Technical Data Package Summary for the list of documents provided from Fujitsu (scanners), Toshiba (laptop computers), and Hewlett Packard (laptop computers).

1.3 Intended audience

Clear Ballot Group’s Quality Assurance Program has been evolving since the first Quality Engineer was hired in early 2013. The company’s internal wiki and bug-tracking system (Trac) were the first repositories of QA processes and progress. The Quality Assurance Program has been continually improved, and this document has been created and refined.

This document is intended for state election officials and their delegated Voting Systems Test Laboratory, as part of the TDP required to certify the ClearVote system for use in their state.

1.4 Reference documents

Table 1-1. Reference documents

Document Description

Software Design and Specification

Technical details on all components of ClearVote software, including coding standards, software items, databases, interfaces, and URLs.

Election Preparation and Installation Guide Election Administrator's Guide

Administrative, set up, and troubleshooting issues. An appendix contains a list of error messages for all ClearCount

components.

System Functionality Description

Describes requirements for all ClearVote functionality, including the BDF.

(8)

Chapter 1. Abstract

Document Description

System Hardware Specification

Describes requirements and validation of COTS hardware used for the ClearVote system.

Configuration Management Plan

Describes the configuration management for the ClearVote system; also lists all third party packages used.

System Security Specification

Describes the databases and their intersection with user roles and permissions.

Election Database Specification

Details each database table, the characteristics of the data within the table, and the source of the data.

Test and Verification Specification

Describes the Quality Assurance and testing processes at Clear Ballot Group

Approved Parts List Lists all components of the ClearVote system and the tested

hardware included in the Technical Data Package.

Technical Data Package Summary

(9)

2.1 ClearCount requirements

Chapter 2. ClearCount system

ClearCount performs tabulation and reporting functions within the ClearVote system.Because these functions are foundational, this section describes ClearCount in greater detail.

ClearCount system consists of several components: l ScanServer™ software

l Scripts and executables run from the ScanStation™, but located on the ScanServer

l Tabulator (runs on the ScanStation)

l Election Administration reports and other browser-based tools, including the Election Administration tool, User Administration tool, and the Ballot

Resolver™

ClearCount software makes use of some third-party open source software (such as Python, Ubuntu, MySQL, JavaScript). The full list is found in ClearVote

Configuration Management Plan.

In general, the quality verification process is the same for all components. Exceptions are noted in the following sections.

2.1 ClearCount requirements

The ClearCount System Functionality Description provides the general functional requirements for the ClearCount system. The VVSG 1.0 requirements in regards to accuracy (Volume 2, Appendix C.5) were also adapted when designing and

performing stress tests. ClearAudit™ field tests, discussions with election officials, and research into election industry best practices, also informed the process. VVSG 1.0 requirements for security, accuracy, error recovery, and integrity as described in Volume 1, Section 2.1 were taken into account in all cases.

2.2 Documentation of quality procedures

The Trac system, Clear Ballot Group’s internal ticketing system, has been the system of record for not only functional problems but also for design issues, enhancement requests, and tasks. The accompanying internal wiki contains information on functional and process standards.

As the TDP documentation took shape, information from the Trac tickets and wiki were incorporated into the System Functionality Description, the Configuration

(10)

Chapter 2. ClearCount system

As the Test and Verification Specification details, this information is also

incorporated into a series of test plans and test case reports. Each minor or major new build goes through significant regression testing to verify its acceptability. Bug fix builds have bug fixes verified by QA. The QA staff also performs smoke testing based on which components changed.

2.3 Quality responsibilities

The CEO, CTO, Director of Product Management, and the Quality Engineers are all responsible for the definition and enforcement of quality in the ClearVote system.

2.4 Tests and verification

The Test and Verification Specification lists all the test plans (describing the test approach) and test case reports (describing the test steps) for the ClearVote system. They are also listed in the appendices to this document.

Known issues present in the ClearVote build submitted for certification are listed in the Test and Verification Specification.

2.5 Documentation feedback and process review

The assembly of the TDP documentation and corresponding testing and analysis of the Clear Ballot software identifies areas where the Quality Process requires

review. The Director of Product Management convenes a series of meetings to institute additional levels of feedback and peer review into the Quality Assurance Program. These meetings recur periodically, and updated processes are

documented on the Clear Ballot Group internal wiki page as well in subsequent versions of this document.

(11)

3.1 Hardware specification review

Chapter 3. COTS hardware: laptop computers and

scanners

As described in ClearVote System Hardware Specification, system hardware components undergo extensive validation and verification.

3.1 Hardware specification review

Scanners must meet stringent requirement for reliability, ease of use, availability of support, and durability. Clear Ballot staff reviewed the specifications for all proposed hardware and conducted in-depth discussions with Fujitsu, the manufacturer of the scanners.

3.1.1 Scanner review and requirements

The major scanner requirements are:

l Scanners must be able to accept all ballot weights and paper sizes required. l The scanner must produce acceptable quality images (200 or 300 dpi) as JPGs

for correct analysis by the Tabulator.

l Scanner settings such as contrast must be configurable.

l The scanner software must be able to rename the images when provided with the target card bar code and to deposit the scanned images into a designated folder.

l Scanner performance must be acceptable for extended periods of time of continuous use.

l Scanner performance must degrade only slightly (and ideally not at all) when scanning folded ballots. Accuracy of results must not be affected.

l Scanners must have sensitive and accurate capabilities for detecting size and width irregularities, multifeeds, and so on. Recovery must be clear and

immediate.

Since the ClearVote system is designed to be scalable depending on the size of the jurisdiction, certifying different scanner models at different price points was also a requirement.

(12)

Chapter 3. COTS hardware: laptop computers and scanners

As described in the ClearCount Approved Parts List, the requirements for the laptop computers used involve the operating system, CPU, and memory. Different models are therefore selected for the ScanServer and for the ScanStation and Election Administration Station clients. The servers and clients were used in an extended series of in-house and field tests, including a million ballot performance and stress test that exercised each laptop for approximately ten hours as it

tabulated the ballot images for 100,000 ballots (ten ScanStations were used in the test).

ClearDesign and ClearAccess also utilize laptop computers. ClearAccess also utilizes all-in-one form factor computers. Selection and test of these computers follow the same process as used with ClearCount computers, with specific

ClearDesign and ClearAccess requirements added to the selection considerations.

3.1.3 Router or switch requirements

The router or switch used to connect the systems in the closed network of the ClearVote system must be gigabit router with a minimum of 4 ports.

3.2 Hardware responsibilities

The Director of Product Management bears primary responsibility for the selection of the hardware, with input and assistance from the CTO and CEO. The Quality Assurance staff assisted with the testing that verified the hardware would meet our needs.

3.3 Test plans and special tests

3.3.1 Scanner tests

The Test Case Report Scanner Behavior contains an extensive suite of test cases that were performed on each scanner model being submitted for certification (Fujitsu fi-7180, fi-6670, and fi-6800). The Test Case Report System and

Tabulation Regression contains a set of integration tests performed with larger

(13)

3.4 Future development and COTS integration

Clear Ballot Group continues to work with Fujitsu to guarantee that the functions required for the ClearVote are included in future scanner models, including them in the scanner configuration software.

Clear Ballot Group staff are committed to making ClearVote available on the latest models of COTS laptop computers with the latest operating systems and continue to test (and submit for certification as appropriate) as new versions of Microsoft Windows become available.

(14)

Chapter 4. Integration with third-party EMS

The Software Design and Specification describes the Ballot Definition File creation process. This is a customized process based on each election’s specifics, including vendor, ballot size, number of contests, choices per contest, languages on the ballot, and so on.

The Election Database Specification spreadsheet contains information on the contents of the BDF and how it corresponds to the database tables in the election database.

The following process does not apply to BDFs created through ClearDesign.

4.1 PDF review and creation of the BDF

The development staff at Clear Ballot Group is responsible for analyzing the ballot PDFs sent from the jurisdiction and translating them into the BDF zip file. Each ballot style and precinct must be analyzed and the contests and choices translated into a set of virtual coordinates required for tabulation.

Additional information, such as ballot size, may be required from the jurisdiction if the information cannot be interpreted from the PDFs.

4.2 BDFs and election creation

A BDF is required for creating an election database that contains tabulated ballot data. If the BDF does not contain all the required tables, the election database is not created.

Any test of the system that involves tabulation requires a valid BDF.

4.3 BDF testing and validation

Any given BDF is tested through election creation and tabulation. In addition, some UI tests compare the contents of reports to the data in the BDF.

(15)

4.4 Integration responsibilities

The QA staff validates the BDFs through election creation and the review of tabulation results and UI reports.

(16)

Appendix A. Test plans

The following test plans describe the test scope and approach for various

components or functions of the ClearVote system. See the Test and Verification

Specification for more information.

l Laptop Acceptance Test Plan l Scanner Acceptance Test Plan l System Test Plan

l System Error Handling Test Plan l Performance and Stress Test Plan l UI Reports Test Plan

l Ballot Adjudication and Resolution Test Plan l Tabulator Functional Test Plan

l Installation and Hardening Test Plan l User Access and Election Admin Test Plan

(17)

Appendix B. Test case reports

The test case reports contain the test steps for each ClearCount test scenario. Test instructions and a tab to place test data are also included in each

spreadsheet. See the corresponding test plan document (see Appendix A) for more information about the test approach for that system function or component. Test case reports include the following. See the Test and Verification Specification for more information.

The test case report for UI 1007 is deliberately excluded; the report was removed from the ClearCount UI.

l Test Case Report scanner behavior

l Test Case Report installation and hardening l Test Case Report Performance and stress testing l Test Case Report system and tabulation regression l Test Case Report system error handling

l Test Case Report Tabulator Regression tests l Test Case Report Tabulator error handling

l Test Case Report users election admin and access levels l Test Case Report Ballot Adjudication and Resolution l Test Case Report – UI 1001—Dashboard

l Test Case Report – UI 1002—Statement of Ballots Cast l Test Case Report – UI 1003—Statement of Votes Cast l Test Case Report – UI 1004—Contests Report

l Test Case Report – UI 1005—Statement of Votes Cast with Precincts l Test Case Report – UI 1006—Vote Visualization

l Test Case Report – UI 1008—Election Activity Log

l Test Case Report – UI 1009—Causes of Unreadable Ballots l Test Case Report – UI 1010—Ballot Images

l Test Case Report – UI 1011—ScanStation Report l Test Case Report – UI 1012—Ballot Information Page l Test Case Report – UI 1013—Annotated Ballot Page l Test Case Report – UI 1014—Box Report

(18)

Appendix B. Test case reports

l Test Case Report – UI 1016—Ballot Locator l Test Case Report – UI 1017—Human Resolutions l Test Case Report – UI 1018—Ballot Styles

l Test Case Report – UI 1019—Counter Group – Statement of Ballots Cast with Counter Groups

Quality Assurance Program

ClearVote 1.0