ICS Security Configuration and Maintenance 35

Maintenance

3.2.4.1 Weak Testing Environments

CSET assessments commonly identified maintenance/testing environments as security gap areas. CSSP assessments and ICS-CERT incident response have noted poor patch management on ICS. Backup or test environments are necessary for testing patches before applying them on critical systems.

Patch management is paramount to maintaining the integrity of both IT and ICS. Unpatched software represents one of the greatest vulnerabilities to a system. Software updates on IT systems, including security patches, are typically applied in a timely fashion based on appropriate security policy and procedures. In addition, these procedures are often automated using server-based tools. Software updates on ICS cannot always be implemented on a timely basis because these updates need to be thoroughly tested by the vendor of the industrial control application and the end user of the application before being implemented. ICS outages often must be planned and scheduled days/weeks in advance. The ICS may also require revalidation as part of the update process. Another issue is that many ICS use older versions of operating systems that are no longer supported by the vendor. Consequently, available patches may not be applicable. Change management is also applicable to hardware and firmware. The change management process, when applied to ICS, requires careful assessment by ICS experts (e.g., control engineers) working in conjunction with security and IT personnel.

Vulnerabilities that have had patches available for a long time are still being seen on ICS.

Unpatched operating systems open ICS to attack through known operating system service

vulnerabilities. For example, in 2003 the Slammer worm disabled an Ohio Davis-Besse nuclear power plant safety monitoring system for nearly 5 hours. The Davis-Besse plant was in a

maintenance cycle at this time and not generating

power. According to reports, plant computer engineers had not installed the patch for the Microsoft SQL vulnerability that Slammer exploited. In fact, they did not know there was a patch, which Microsoft released 6 months before Slammer struck.5

The following are sanitized findings associated with this vulnerability from multiple assessments:

 Operating system vendor patches not applied  System computers vulnerable to operating

system service vulnerabilities  Vulnerable version of Sendmail

 Sun rpc.cmsd has an integer overflow problem in xdr_array

 Vulnerable version of RPC

 Inconsistent application of current patches on HMIs.

Recommendation: A timely patch management

process is critical to reduce vulnerabilities. Operating system patches repair vulnerabilities in the operating system that could allow an attacker to exploit the computer. The importance to system security of keeping operating system patches up- to-date cannot be over emphasized. However, patching ICS machines can present unique challenges. Among the factors to consider are system functionality, security benefit, and timeliness. This process requires elements of IT, IT security, process control engineering, and senior management and incorporates elements of an Incident Response Plan, a Disaster Recovery Plan, testbed testing, and a Configuration Management Plan. Where patching is not an option, work-arounds and defense-in-depth techniques and tactics can be used.6

Statically linked libraries need to be

independently kept up-to-date if they are different from the libraries associated with the operating system. Database software and other applications also need to be kept patched and up to date.

Guidance\references:

 Recommended Practice for Patch

36

2008, http://www.us-

cert.gov/control_systems/practices/documents/ PatchManagementRecommendedPractice_Fin al.pdf

3.2.4.2 Limited Patch Management

Abilities

Many ICS facilities, especially smaller facilities, have no test facilities, so security changes must be implemented using the live operational systems.

Recommendation: Because of the complexity of

ICS software and possible modifications to the underlying operating system, changes must undergo comprehensive regression testing. The elapsed time for such testing and subsequent distribution of updated software provides a long window of vulnerability.

Patches are additional pieces of code that have been developed to address specific problems or flaws in existing software. Vulnerabilities are flaws that can be exploited, enabling unauthorized access to IT systems or enabling users to have access to greater privileges than authorized.

A systematic approach to managing and using software patches can help organizations to

improve the overall security of their IT systems in a cost-effective way. Organizations that actively manage and use software patches can reduce the chances that the vulnerabilities in their IT systems can be exploited. In addition, they can save time and money that might be spent in responding to vulnerability-related incidents.

NIST SP 800-40 Version 2 provides guidance for organizational security managers who are responsible for designing and implementing security patch and vulnerability management programs and for testing the effectiveness of the programs in reducing vulnerabilities. The guidance is also useful to system administrators and

operations personnel who are responsible for applying and testing patches and for deploying solutions to vulnerability problems.

Requirements: The following requirements apply

to patch management:

1. Establish a testing environment for ICS.

2. Applying patches to operating system components creates another situation where significant care should be exercised in the ICS environment. Patches should be adequately tested (e.g., off-line on a comparable ICS) to determine the acceptability of side effects. Regression testing is advised. It is not uncommon for patches to have an adverse effect on other software. A patch may remove a vulnerability, but it can also introduce a greater risk from a production or safety perspective. Patching the vulnerability may also change the way the operating system or application works with control applications, causing the control application to lose some of its functionality. Another issue is that many ICS use older versions of operating systems that are no longer supported by the vendor. Consequently, available patches may not be applicable. Organizations should implement a systematic, accountable, and documented ICS patch management process for managing exposure to vulnerabilities.

a. Once the decision is made to deploy a patch, other tools can automate this process from a centralized server and can confirm that the patch has been deployed correctly. Consider separating the automated process for ICS patch

management from the automated process for non-ICS applications. Patching should be scheduled to occur during planned ICS outages.

Guidance\references:

 NIST SP 800-82Guide to Industrial Control Systems (ICS)

 NIST SP 800-40Creating a Patch and Vulnerability Management Program.

3.2.4.3 Weak Backup and Restore

Abilities

Backups, restores, and testing environments have been identified as a common issue within the industry for continuity of operations in the event of an incident. Backups are usually made, but usually not stored offsite and rarely exercised and tested.

37

Recommendation: The frequency of system

backups and the transfer rate of backup information to alternate storage sites (if so designated) are consistent with the organization's recovery time objectives and recovery point objectives. While integrity and availability are the primary concerns for system backup information, protecting backup information from unauthorized disclosure is also an important consideration depending on the type of information residing on the backup media.

Requirements: Requirements used by the CSET

self-assessment tool are summarized below: 1. The organization conducts backups of user-

level and system-level information (including system state information) contained in the system on an organization-defined frequency and protects backup information at the storage location.

2. The organization tests backup information on an organization-defined frequency to verify media reliability and information integrity. 3. The organization protects system backup

information from unauthorized modification (NIST SP800-53A, Sec CP-9).

4. The organization employs mechanisms with supporting procedures to allow the system to be recovered and reconstituted to the system’s original known secure state after a disruption or failure.

Guidance\references:

 NIST SP800-52A – Guide for assessing the Security Controls in Federal Information Systems.