You are in the Configuration Management Business

(1)

YouAreInTheConfigurationManagmentBusiness-20060412.doc Last printed 4/12/2006 10:19:00 PM

You are in the

Configuration Management Business

By: John Washburn

April 12, 2006

(2)

Abstract

This paper is directed to any person involved in the testing, certification, qualification, approval or purchase of election voting equipment. The paper covers why the esoteric field of configuration management (version control) is intimately and inextricably linked to the testing, certification, qualification, approval, purchase, delivery and auditing of election voting equipment. The paper describes:

What is configuration management

Why configuration management of software is difficult What is a physical configuration audit document

What is minimum information which should be included in a physical configuration audit document How to use a physical configuration audit document to decide if your system is correct.

(3)

Abstract ... 2

What is configuration management?... 4

Why should you care? ... 5

Identification ... 8

File Fingerprints ... 10

(4)

What is configuration management?

Secretaries of state, state election boards, and other election officials are now grappling with approving,

certifying, purchasing, and auditing voting machinery. Unfortunately, most of these officials fail to realize is if you are in the business of testing, approving, qualifying, certifying, purchasing, or auditing election systems you are necessarily in the configuration management business.

What is configuration management? Configuration management is the art and practice of defining what is meant by the phrase the system . Software and any system which executes software are inherently mutable. This ability to change the functionality of a software-based system easily and cheaply is usually a virtue. For testing, approving, qualifying, certifying, purchasing, and auditing software systems this mutability though is a positive vice. Configuration management is how to reign in this inherent mutability. Configuration

management is how to precisely define the system under test (or system pending approval, qualification, certification, or purchased). Configuration management is how the system is frozen so testing, approval, qualification, or certification can be performed on a single system; not a system which is changes while under test. For those interested in fine details of configuration management, I recommend the book: Software Configuration Management by Jessica Keyes.

Election officials do not need to know the fine details of configuration management in order to understand the daunting task facing them. An electronic election system is a collection of hardware and programming.

Programming may be stored on a form of read only memory (ROM, PROM, EPROM, or EEPROM, etc.) or on traditional mutable media (hard disk, RAM, smart cards, memory packs, flash memory, etc.). Programming stored on mutable or removable media is called software. Programming stored on an immutable or hard to change media is called firmware. Both firmware and software are programming. The difference is the storage medium and the easy with which the programming on that medium can be changed.

(5)

Why should you care?

Before an elector casts a vote on an electronic election system, there are 4 stages which are intimately and inextricably linked to configuration management. These stages are NASED qualification, state certification, purchase by local election administrators and post-election auditing. Until December 2007 there is no Federal qualification process for election systems. NASED qualification is administered by the National Association of State Election Directors (NASED), and 3 vendor-funded laboratories called independent test authorities. State certification is administered by the chief election authority of the state. The chief election official varies from state to state; Secretary of State, an independent State Election Board, or an appointed position. The purchase by local election administrators also varies by state. In some states the purchase authority is at the county or parish level. In other states the purchase authority is at the municipal (City, Village, or town) level or a combination of both levels. Some but not all states require auditing election systems. But, even if auditing is not mandated every state give election officials the authority to audit election systems as part of their authority in order to insure state and or federal election laws were not violated.

How are these activities inextricably tied to configuration management? Imagine the electronic election system is represented by a CD-ROM. The directories on the CD-ROM will be used to represent the hardware used by the electronic election system. The files in a particular directory represent the programming executed on that hardware component. For this analogy there is no distinction between programming stored on traditional mutable media (software) and programming stored on read-only media (firmware). Each activity: NASED qualification, state approval, and local purchase involves 3 possibly, different systems. For the remainder of this paper the system which is NASED qualified will be represented by a CD-ROM with a blue label. The system which is certified (or approved) by the state will be represented by a CD-ROM with a green label. The systems actually purchased by a local election administrator will be represented by CD-ROM s with white labels. The system(s) selected for auditing by CD-ROM s with red labels.

Many states require the source code for approved system be held in escrow. The 2002 Voluntary Election systems Guidelines in section 4.2 of Volume I, require the source code and build tools delivered by the vendor be examined the vendor-funded independent test authority laboratory. The collection of source code and build tools will be will be represented by CD-ROM s with yellow labels.

(6)

(7)

Ideally the blue CD-ROM (NASED qualified election system) is the same as the green CD-ROM (state

approved election system) which in turn is the same as every white CD-ROM (the systems actually delivered to a local election official) is in turn the same as every red CD-ROM (the systems selected for random audits). Also, the source code given to the vendor-funded labs or to the states for escrow (the yellow CD s) should be the same and should actually create the software found on the election systems (Blue, Green, White, and Red CD s)

Configuration Management is how to know and verify all these systems are indeed the same. Configuration management is how to insure only NASED qualified systems which are state approved are sold to local election administrators to be used to aid in election administration.

Configuration management is how to determine the system submitted for state certification is NASED qualified. Configuration management is how to determine if the system purchased by a local election official is state approved. Configuration management is how to determine if all the systems purchased by a local election official are the same in all precincts. Configuration management is how to determine if all the systems presented for examination during an audit are the same systems used in a local election. Configuration management is how to define what is meant by the phrase this system is the same as that system . Configuration management is how to determine which systems are the same so testing results from one system can be applied to all systems which are the same . Configuration management is how to determine the source code examined for defects and malicious code is actually the same source code used to build a given set of system components. Configuration management is how to determine if the source code submitted to the vendor-funded laboratories for examination is the same source code held in escrow on behalf the state s chief elections officer.

(8)

Identification

The first step in configuration management is identification of the system. This is most commonly done by completely and correctly identifying all of the system components. What is the particular constellation of hardware, software, and firmware which comprise the particular system under consideration? The tool used for such system identification is the physical configuration audit document. The physical configuration audit document is a complete enumeration of the identification information for each of the following:

The collection of hardware used by the system,

The collection of firmware used for each piece of hardware,

The collection of traditional software created by the vendor for the system, The collection of common libraries used by the system,

The collection of commercial off the shelf (COTS) software used by the system.

The collection of initialization files, parameter files, scripts, stored procedures, database views, registry entries, environment variable or other files and data used by the system during the operation and used of the system.

Hardware platforms are relatively easy to identify because the circuitry of hardware changes infrequently. Because of the immutability of hardware, recording the manufacturer, model number and serial number is usually sufficient to completely and correctly identify a hardware component. For hardware, the physical configuration audit document is used primarily to note additions or deletions of hardware more than changes in hardware. Programming (traditional, firmware, COTS, common library, interpreted scripts, database

procedures, database views, etc.) is another problem altogether. Programming is so inherently mutable that recording the name of the file and the version number is absolutely inadequate to correctly and completely identify the software component. Common elements used to identify software components are name, creation date, and version number. All three of these are woefully inadequate for use in a physical configuration audit document. Files are easily renamed. The program, touch, is a utility specifically designed to change the file date to any date specified. The program, touch, is used extensively by many organizations for build

management and to assign the same file date to all software components delivered on a distribution CD. The version number is information maintained in a manner similar to any other source component of software. Most organizations leave the maintenance of version information to the developer. Very few organizations automate the assignment of version information and prevent the developer from altering the version information

manually. The result is programming (software or firmware) can be labeled with the same version number but be substantially (or completely) different.

If filename, version number, and creation date are inadequate to identify a software component, what is adequate to identify software components? At the minimum file size and a checksum are needed to identify a software component. The most common checksum used is the 32-bit cyclical redundancy checksum know as CRC-32. But even these 2 statistics, file size and CRC-32, are inadequate. The recording of these 2 statistics are used to detect mistakes. Both statistics are easily subverted if the intent is deception; passing one piece of software off as another. File size can be adjusted by padding with harmless characters within the file. Similarly if any 4 bytes within a file can be changed arbitrarily without affecting program execution, the resulting file can be assign any CRC-32 value desired.

Recording the file size and CRC-32 is adequate for informal configuration management; such as where the testing team and the development team have the same project manager. What is needed for more formal arrangements (such as independent testing labs or 3rd party certification) is a more precise and deception-resistant means to identify software components. These deception-deception-resistant identification methods are

cryptographic hashes. Cryptographic hashes are also called file fingerprints, file hashes, hash values, message digests. Sometimes and only by the most technically inept, a cryptographic hash is mistakenly referred to as a

(9)

signature, digital signature, or electronic signature. While all digital signature protocols use cryptographic hashes, digital signatures posses 2 properties which a cryptographic hash does not. A digital signature has the property of authentication. Cryptographic hashes or MAC do not. Authentication is where one knows for certain the identity of the person(s) who created the digital signature. A digital signature has the property of non-repudiation. Cryptographic hashes or MAC do not. Non-repudiation means the signer of a file cannot say

I did not sign this . In other words, non-repudiation is the assurance only the authenticated signer could have created the digital signature in question. The legal definitions of the properties of a digital signature are found in the Uniform Electronic Transactions Act the text which is incorporated into the statutes of several states such as here in Wisconsin statute.

For the remainder of this paper I will use the term file fingerprint instead of cryptographic hash. File fingerprint is a much less intimidating a term than cryptographic hash but does not lose much if any of the precision of the concept behind a cryptographic hash.

(10)

File Fingerprints

File fingerprinting (Cryptographic hashes) treat all electronic files as a collection of ones and zeros and as such can be thought of as a number of huge magnitude. File fingerprinting maps this huge number (the file contents) down into a smaller number of fixed size; usually 120, 160, 256, 384, or 512 bits (16, 20, 32, 48, or 64 bytes) in length. The exact method and details of this mapping is irrelevant to this paper but can be found here: <WIKI URL>. What is important are the following properties of a file fingerprint (cryptographic hash):

1. The same file contents always map to the same hash value. In other words if the contents of one file,

M

1 are identical to the contents of a second file,

M

2 , then the hash value of file one, H

M

1 , is

equal to the hash value of file two, H

_M

₁ . This is expressed symbolically as:

M

₁ ₂ H ₁ H ₂ .

2. The change of even a single bit in the contents of file will change the value of about half the bits in the hash value of the new (1 bit changed) file and the hash value of the original file.

3. Given a hash value of the contents of some file, it is hard to create a second file whose contents map to the same hash value.

4. Given a large number of documents it is unlikely there exist 2 files among these which are both different and have the same hash value.

Physical Configuration Audit documents must at a minimum contain the following: file name,

file location (path), file size, and

At leash one file fingerprint (cryptographic hash value) of the contents of the file. Better still is a second file fingerprint (cryptographic hash value) from another family of hashes. The more cryptographic hash values recorded the better.

(11)

Comparing Systems

Once a physical configuration audit (PCA) document is created (e.g. for the White CD-ROM s) it needs to be compared to a second physical configuration audit document (e.g. the PCA for the Green CD-ROM s). This implies a certain level of infrastructure is available. In order to perform this comparison of systems:

One needs to have tools which can create a PCA document for a given system.

One has to use those tools to create PCA documents for 2 or more systems (e.g. 2 or more CD-ROM s). One needs to have tools which can compare 2 PCA documents of 2 systems for any differences present between the 2 systems.

One needs some criterion to determine the significance of any differences found. A collection of PCA documents and tools can be found here:

http://www.washburnresearch.org/archive/FingerprintFiles.zip.

A partial PCA of the author s laptop can be in Jww-2005-10-29.txt. This PCA is an enumeration of all the found in the \Windows\ directory prior to an upgrade of the Windows XP operating system. This PCA

document was created using the PERL script, FingerprintFiles.pl, which has been compiled into a stand-alone executable file, FingerprintFiles.exe.

A second such PCA of the author s laptop can be in Jww-2005-10-31.txt. This PCA is an enumeration of all the found in the \Windows\ directory after the upgrade of the Windows XP operating system. This second PCA was created using the same tools as created the first PCA.

The lines of data in Jww-2005-10-29.txt and Jww-2005-10-31.txt are tab-delimited and can be imported into Microsoft Excel or a database. For example, from the PCA document, Jww-2005-10-29.txt, the DLL

containing the Microsoft Common Dialog boxes, comdlg32.dll, has the following identifying properties. Full Path: c:\windows\system32\comdlg32.dll

File Size: 276992 bytes

MD5 Hash value: 1EDB1BB89D021955E6F7265911175B8D)

SHA1 Hash value: 517CAD5C781E950B23B82A578CB1873206092443

The 2 PCA documents each have more than 13,000 entries. Comparing one PCA to the other would be near impossible to do manually. In order to compare one PCA manifest often a second program is used identify the differences found between 2 PCA documents.

The results of comparing the first PCA document to the second are found in Comp-20060412-1546.txt. This comparison of PCA documents was created using the PERL script, CompareManifests.pl, which has been compiled into a stand-alone executable, CompareManifests.exe. There are ? kinds of differences identified by CompareManifests.pl,

1. A file is found in one PCA document but not in the other. This indicates an addition or deletion of a file. 2. A file is found in both PCA s but the fingerprint (size, MD5 hash or SHA1 hash) is different. This

indicates the file has been upgraded or substituted.

3. Two files with different names or locations are found to have an identical fingerprint (file size, MD5 hash and SHA1 hash are the same). This indicates the file has been renamed or moved to a different location.

(12)

Here is an excerpt from Comp-20060412-1546.txt. Notice the 3 kinds of differences reported.

File, c:\windows\$NtUninstallKB891122$\spuninst\spuninst.exe, was found in jww-2005-10-31.txt, but not in jww-2005-10-29.txt File, c:\windows\$NtUninstallKB891122$\spuninst\updspapi.dll, was found in jww-2005-10-31.txt, but not in jww-2005-10-29.txt

File, c:\windows\RegisteredPackages\{30C7234B-6482-4A55-A11D-ECD9030313F2}\MSSCP.dll, in jww-2005-10-31.txt has a different fingerprint than the fingerprint in jww-2005-10-29.txt File, c:\windows\RegisteredPackages\{30C7234B-6482-4A55-A11D-ECD9030313F2}\MSWMDM.dll, in jww-2005-10-31.txt has a different fingerprint than the fingerprint in jww-2005-10-29.txt File, c:\windows\RegisteredPackages\{30C7234B-6482-4A55-A11D-ECD9030313F2}\MsPMSNSv.dll, in jww-2005-10-31.txt has a different fingerprint than the fingerprint in jww-2005-10-29.txt File, c:\windows\system32\dllcache\wmsdmod.dll, in jww-2005-10-31.txt has a different fingerprint than the fingerprint in jww-2005-10-29.txt

File, c:\windows\system32\dllcache\wmsdmoe2.dll, in jww-2005-10-31.txt has a different fingerprint than the fingerprint in jww-2005-10-29.txt File, c:\windows\system32\dllcache\wmspdmod.dll, in jww-2005-10-31.txt has a different fingerprint than the fingerprint in jww-2005-10-29.txt File, c:\windows\system32\dllcache\wmspdmoe.dll, in jww-2005-10-31.txt has a different fingerprint than the fingerprint in jww-2005-10-29.txt File, c:\windows\system32\dllcache\wmvcore.dll, in jww-2005-10-31.txt has a different fingerprint than the fingerprint in jww-2005-10-29.txt File, c:\windows\system32\dllcache\wmvdmod.dll, in jww-2005-10-31.txt has a different fingerprint than the fingerprint in jww-2005-10-29.txt File, c:\windows\system32\dllcache\wmvdmoe2.dll, in jww-2005-10-31.txt has a different fingerprint than the fingerprint in jww-2005-10-29.txt

Each of the following files has a total fingerprint of: 22240 1339635675E79B07E1CA5F840830A84F 8740BAAAC40398544A450415BEC77F0F0883D95C . c:\windows\$hf_mig$\KB890046\update\spcustom.dll

c:\windows\$hf_mig$\KB890859\update\spcustom.dll c:\windows\$hf_mig$\KB893066\update\spcustom.dll c:\windows\$hf_mig$\KB893086\update\spcustom.dll c:\windows\$hf_mig$\KB893756\update\spcustom.dll

Each of the following files has a total fingerprint of: 19555 624573F1D005ABCE2E6D9AB47ED64727 6839E52E28D68ACA917790302F840C74B24F292E . c:\windows\system32\spool\drivers\w32x86\3\HPSMAC05.GPD

c:\windows\system32\spool\drivers\w32x86\hewlett_packardhp_la7e8a_print_hpz\HPSMAC05.GPD

Each of the following files has a total fingerprint of: 9248 E7629D2374443B7E604C831DE1FEE8F1 6AEE8B0DC09F9E2E07BFA9EE4E609988EFA8E2AC . c:\windows\Fonts\ega40737.fon

c:\windows\Fonts\ega40869.fon

Each of the following files has a total fingerprint of: 36864 9D46F0008174E533872D50259F21C78F 97870CBB5A4F837A4D220D1987B11E77C9A212B5 . c:\windows\system32\dllcache\mscpxl32.dll

c:\windows\system32\mscpxl32.dLL

Each of the following files has a total fingerprint of: 3616 0BF92230027B6C3804EC90D4035938A5 4949CF698AD786E08FA5CCCA8D142D61FA306228 . c:\windows\pchealth\helpctr\OfflineCache\Professional_32#0409\00000172.query

c:\windows\pchealth\helpctr\OfflineCache\Professional_32#0409\00000173.query

Each of the following files has a total fingerprint of: 718048 3B5EAAEDB8A9D3F98DEBBDB0CFD214D5 C9E09F6F6026F928D3D6D9056AF868DF83BD44EF . c:\windows\$hf_mig$\KB890046\update\update.exe

c:\windows\$hf_mig$\KB890859\update\update.exe c:\windows\$hf_mig$\KB893066\update\update.exe c:\windows\$hf_mig$\KB893086\update\update.exe