Cybersecurity: Lessons Learned from DOE Projects

Cybersecurity: Lessons Learned

from DOE Projects

Mark Morgan: PNNL Ginger Armstrong: CMEEC Sue Blanchette: Groton Utilities

Scott Franklin: Exelon Paul Hartung: NOVEC

Program Outline

• Background: SGIG

• Process

• Results/Observations

• Federal Role moving forward

• Awardee perspective

“Just the facts, Ma'am”

• Smart Grid Investment Grant (SGIG) was funded by the 2009 ARRA and managed through DOE

– 400+ applied

– 99 Grants awarded

– $3.4B of federal funding, matched by $4.4B of

“Just the facts, Ma'am”

• Broad distribution of Utility Types

• Diverse range of Utilities (size) and award

values

• Broad mixture of technology types

– Transmission (10) – Distribution (13) – AMI (31) – Customer Systems (5) – Cross-cutting (40) 4

The Role of Cyber in SGIG

• Cybersecurity requirements built in from the

outset

– Pre-award: Addressed in proposal

– Post-award: Prior to work starting the awardee

was required to develop and submit a Cybersecurity Plan (CSP)

– Annually: DOE site visit with a dedicated

discussion of cyber implementation progress and alignment with approved CSP

The Cyber Team

• DOE requested PNNL

to establish a team to support the lifecycle of the program

• Drew from

Universities, National Labs, and private

industry

• Established processes

and conducted site visits

• Established resources

to aid utilities

6

Site Visit Analysis Process

• Data/Grades for 13 criteria were collected and

analyzed from the site visits using a green/yellow/red rating scale

• Demographics were collected and the awards

were classified into one of 3 categories;

PUD/Cites, Rural Electric/Cooperatives, and Transmission & Generation

Site Visit Results

• Highest performers were Transmission &

Generation followed by Cities/PUD’s and then Rural Electrics/Cooperatives

• Highest gains in performance was

demonstrated by Cities/PUD’s followed by Rural Electrics/Cooperatives and then

Transmission & Generation

Example Data Results

10.0 20.0 30.0 40.0 50.0 60.0 70.0 80.0 90.0 100.0

RE/COOP Normalized Score (%)

2012 2013

2012 Average 2013 Average Change in Average

CP 85.3 89.9 4.6

RE/COOP 81.9 84.1 2.2

T&G 95.5 94.5 (1.0)

Major Observations

• Cybersecurity “built in” not “bolt on”

– Benefits in both cost and accountability

• “White Hat”, not “Black Hat” as a future role model

• Corporate Cyber vs. Operations Cyber

• Benefits to industry and DOE

• Value of a standards based approach

• Senior Management accountability

• Vendors response

• The benefits of the CSP during and after the project

Sustaining the Momentum

• Cybersecurity is not “fire and forget”

• Large family that includes a Government role

• Examples of Federal roles:

– DOE:Electricity Subsector Cybersecurity Capability

Maturity Model (ES-C2M2)

– NIST: Framework

– DHS: US-CERT

– NERC: ES-ISAC

• Excellent non-Federal resources such as National

Awardee Perspective

Program

Project No: 09-0144

CYBER SECURITY PROJECT

Presenters: Ginger Armstrong Sue Blanchette

Who We Are

14

1

Groton Utilities provides electric service to two separate service territories: City of Groton, CT and Bozrah Light and Power (BL&P). GU also provides water and sewer to customers in the City of Groton.

2

South Norwalk Electric and Water (SNEW) provides electric and water services to customers in South Norwalk, CT

3

Jewett City Department of Public Utilities (JCDPU) provides electric and sewer services to customers in the borough of Jewett City, CT

4

Norwich Public Utilities (NPU) provides electric, water, sewer and gas services to customers in the city of Norwich, CT

5

Connecticut Municipal Electric Energy Cooperative (CMEEC) is a joint action agency that provides wholesale energy procurement, small capacity demand response generation, limited high voltage transmission resources and the management of energy efficiency programs on behalf of its wholesale

Characteristic BL&P GU NPU JCDPU SNEW CMEEC

Customers 2,700 13,496 20,900 2,200 6,300 13

Total Employees 4 119 139 9.5 45 34 IT Employees --- 7 5 --- 1.5 2

Peak Load (MW) 10 64 74 5 22 378

Annual Energy (million KWh) 44 344 348 25 100 2,006

Service Territory (Square Miles) 41 20 29 2 2 ---

Distribution Line Miles 125 152 580 11 32 ---

Technical Scope

16

ConnSMART Program

Technical

Scope ManagementProgram

Metering & Communications Interval Data Processing & Presentment Wholesale Business Intelligence Customer Demand Management Distribution Automation Security Controls and Testing Project Management DOE Grant Administration Cyber Security Planning & Compliance We are Here

Cyber Security in the Boardroom

• Executive Buy-In

• Company-wide Buy-In

• Board Education and Buy-In

• Plan approval

• Value of & Need for Awareness Training

• Incorporate as regularly reported CS metric at

Vendor Readiness

• Smart Grid hardware still maturing

• Software design basis lacking in Cyber Security

considerations

• Vendor reluctance to comply with CSPs

• Federal and Industry Standards slow to

develop, vendors unaware of requirements

• Many delays due to lack of vendor readiness

Tools that made life easier

• Frequent Cyber Security Team meetings

• Team developed Cyber Security Status

Tracking tool early on in project

• Use of outside subject matter experts

• Information exchange during DOE Site visits

• Vulnerability assessments engagements

Now what? Money & People

• Funding Cyber Security into the future

• Continued Board understanding & support

• Staffing

• Outsourcing

• Working without a net

• Managing risk, monitoring evolving threats

and advancing technologies

What would you do differently?

• Better job of management buy-in at beginning

• Tighter collaboration between the vendors

from the start

• Better understanding of resource and funding

requirements for Cyber Security

• Vendor proof of concept

Thank You

“Let us rise up and be thankful, for if we didn’t learn a lot today, at least we learned a little, and if we didn’t learn a little, at least we didn’t get sick, and if we got sick, at least we didn’t die; so,

let us all be thankful.” Buddha

Cybersecurity Lessons Learned

at NOVEC

Paul Hartung, PE

Outline

• Overview of NOVEC

• DOE Grant Background

• Cybersecurity Implementations

• Cybersecurity Roadmap

• Lessons Learned

Overview of NOVEC

• Northern Virginia Electric Cooperative (NOVEC) provides reliable electric service to more than 155,000 homes and businesses located in Clarke, Fairfax, Fauquier, Loudoun, Prince William and Stafford counties, the City of Manassas Park and the Town of Clifton, all in the state of Virginia.

• NOVEC's service territory encompasses 651 square miles with more than 6,880 miles of power lines.

• Summer peak load of ~925MW

• 53 Substations with 1200+ Intelligent Electronic Devices (IED) communicating over fiber, microwave, radio and cellular networks.

DOE Grant Goals

• Substation Automation:

– Install modern digital control equipment to better monitor and control

substation assets

– Feeder automation

– Voltage Regulator Peak Demand Voltage Reduction program – Utilize DNP 3.0

• Distribution Automation

– The VAR control project will strategically place switched capacitor banks

– Install remotely controlled switching devices consisting of intelligent electronic

reclosers and Motor Operated switches to improve reliability utilizing two-way communications for data collection and remote control.

NOVEC’s Cybersecurity Enhancements

for SCADA

SGIG Project: 2010-13

• Developed a Cybersecurity (CS) Plan

• Conducted SCADA Security Assessment

• Guiding Principle – No SCADA connection to the internet

• Electronic and physical separation from the corporate network and from the internet - SCADA Firewall and DMZ solution

• Controlled internal SCADA access

• Implemented a manual disconnect process for remote vendor access

• Enhanced access controls for vendor and internal support

• Operational audit capability - Implemented centralized logging and monitoring access to the SCADA and Substations environments

• Installed and configured Intrusion Detection technology

• Implemented DMZ architecture • Eliminated direct inbound

connections to SCADA

• Implemented more secure

remote access method

• Includes private fiber and MPLS

Log Management

• Centralized SCADA security

event logs

• Implemented Solar Winds Log

& Event Manager (LEM)

• Drafted log management

policy and procedure

• Daily log review

• Implemented SourceFire

Intrusion Detection

System at key SCADA and Substations network

Vendor Access Controls

• Vendor support now requires

pre-coordinated and a

physical connect/disconnect process instead of full time VPN access

• SCADA administrators

receive e-mail notifications for vendor logins

• Updated patches to remediate

2011 vulnerability assessment findings

• Monitor vendor patch notices

and apply as required

• Program will be formalized

under policy and procedure effort

NOVEC’s Cybersecurity Roadmap

2014 – 2017

• DOE Cybersecurity Capabilities Maturity Model Assessment (C2M2)

• Completed in April 2014 and established a maturity baseline

• Developing a road map to accomplish maturity level 2

• Cyber Training and Education

• Formalizing cyber education for all NOVEC employees

• Conduct annual Phishing Test and Cyber Survey

• Conduct Annual Vulnerability Assessment

• SCADA, Corporate, and Wireless

• Implementation Support for Metering , Outage Management and distribution line devices

• Firewall MPLS 3rd _{Party connectivity}

• Migrate OMS (Outage Management System) and MV-90 (Metering data)to DMZ

NOVEC’s Cybersecurity Roadmap

2014 – 2017

• Create & implement processes for patching

• SCADA Network

• Substation Servers

• Other substation assets to include 3rd _{party client applications}

• Continually assess vulnerabilities and Identify solutions

• New IP based radio system

• Direct connection of Distributed Automation devices to fiber

• Implement additional access controls

Lessons Learned

• Make CS integral to design

• Balance security vs operability

• DOE site visits reinforced CS

• Vulnerability testing proves the solution works

• Vendors can do more

• CS requirement the same but solutions may

differ dependent on utility size

Cyber Security Lessons Learned

Scott Franklin

Agenda

• Overview of BGE, PECO

• DOE Grant/Smart Grid Smart Meter (SGSM)

Program Overview

• Cyber Security Program Overview

• Critical Success Factors

• Challenges

• The Path Forward

BGE Overview

• Baltimore Gas and Electric (BGE) serves more than 1.2 million business and

residential electric customers and more than 655,000 gas customers located in Baltimore City and all or part of 10 Central Maryland counties

• BGE’s electric service territory encompasses 2,300 square miles with more than 26,000 miles of both overhead and underground power lines

• BGE’s gas service territory encompasses 800 square miles with more than 7,100 miles of natural gas pipeline mains

• Summer peak load of ~7,200 MW • 243 substations

PECO Overview

40

• Philadelphia Electric Company (PECO) serves more than 1.6 million business and residential electric customers and more than 500,000 gas customers located in

southeastern Pennsylvania • PECO’s electric service territory

encompasses 2,100 square miles with more than 29,000 miles of both overhead and underground power lines

• PECO’s gas service territory encompasses 800 square miles with more than 6,600 miles of natural gas pipeline mains

• Summer peak load of ~8,250 MW • 500 substations

BGE DOE Grant/SGSM Program

• SGIG Program Goals (https://www.smartgrid.gov/sites/default/files/pdfs/project_desc/09-0014-bge-project-description-final_3.pdf) – Territory-wide deployment of Advanced Metering Infrastructure (AMI) assets

including:

 575,081 Electric Smart Meters

 AMI Communications Systems

 Meter Communications Network (RF mesh, 1,250 network devices)

 Backhaul Communications (Cellular)

 Customer Care and Billing System (partially funded by the SGIG program)

 Meter Data Management System

 Customer Web Portal and Home Energy Reports

 202,906 Direct Load Control Devices

 144,482 Smart Thermostats

– Peak-Time Rebate (Default Residential Tariff)

• Overall SGSM Program Goals

– 1.2 million Electric Smart Meters – 800,000 Gas IMUs

PECO DOE Grant/SGSM Program

• SGIG Program Goals (https://www.smartgrid.gov/sites/default/files/pdfs/project_desc/PECO%20Project%20Description_0.pdf)

– Territory-wide deployment of Advanced Metering Infrastructure (AMI) and Distribution Automation

(DA) assets including:

 More than 600,000 Smart Meters

 AMI Communication Systems

 Web Portal Access

 Distribution System Automation/Upgrades  Distribution Management System/SCADA

 Intelligent Substation Upgrades

 Feeder Monitors/Indicators

 Automated Feeder Switches

 Capacitor Automation

– Dynamic Pricing Programs

 Time of Use – Pilot program offered to 120k Residential, 10k Commercial accounts

 In-Home Displays – Pilot ~200 accounts

 Customer Education and Communication

• Overall SGSM Program Goals

– 1.2 million Electric Smart Meters – 525,000 AMI Gas Module

– 4,400 MV-90 AMI Electric Meters

Cyber Security Program Overview

• Common approach for BGE and PECO

– DOE (and public-facing) Cyber Security Plans reflected existing, mature cyber security

programs and incorporated SGSM-specific requirements as needed (NISTIR-7628, ISO, CoBIT, NIST Cybersecurity Framework, ES-C2M2)

– Intelligence-driven risk management approach to cyber security

 Intelligence and security analysis (threat models)

 Comprehensive risk management program

 Configuration and vulnerability management are critical

– Secure by design

 Build security in from the start (end-to-end project management lifecycle-integrated security, work with vendors/partners)

 Cyber security requirements and SLAs embedded in vendor/partner contracts, and ENFORCED…

 Secure architecture by default

– Exhaustive cyber security vulnerability assessments (ad hoc, recurring), work with

vendors/partners to remediate any findings

– Continuous, automated security monitoring and alerting (SIEM, IDS/IPS, netflow, full

packet capture) across the SGSM infrastructure

Critical Success Factors

• Executive support (governance, budgetary) • Security as part of the corporate culture

• DOE site visits reinforced the absolute need for cyber security • Public Utility/Service Commission support for cyber security • Vendors willing to invest in cyber security and LISTEN…

– Detailed Cyber Security Requirements Matrices provided to vendors,

embedded in contracts

– Enhanced product security roadmaps

– Vendor security assessment testing (better security QA)

• Continuous, automated security monitoring and alerting • Incident response exercises

• Big data tools

Challenges

• Lack of formal SGSM security standards • So much data…big data/data analytics tools

• Multi-million node wireless network & traditional cyber security tools

(scale, protocols)

• Identifying and retaining qualified security personnel • Ever-changing threat landscape

• OT/IT collision

• SGSM infrastructure operating lifecycle versus cyber security lifecycle • Complex systems integrations with varying security capabilities

• Project versus sustain mindset • Vendor/product maturity

The Path Forward

• Continual investment in cyber security personnel and

tools

• Annual external cyber security reviews (Public

Utility/Service Commissions)

• Enhanced, multi-Utility incident response exercises

• Continuous monitoring and improvement