Proposed Cybersecurity T&E Process

(1)

M r P e t e C h r i s t e n s e n

T e s t a n d E v a l u a t i o n P o r t f o l i o M a n a g e r T h e M I T R E C o r p o r a t i o n 1 5 N o v e m b e r 2 0 1 3

W i t h g u i d a n c e a n d s u p p o r t f r o m

D r D a v e B e l l , M s S u s a n M a y, M s J e a n P e t t y D A S D D T & E : D r S t e v e n J . H u t c h i s o n , M r T o m S i m m s , M r T e r r y M u r p h y O S D D O T & E : M r D a v e A l a n d … . A n d m a n y o t h e r s

Proposed Cybersecurity T&E

Process

(2)

What, Why and How?



What do we want to accomplish?

–

Provide an overview of



Proposed OSD Cybersecurity T&E Process

–

Gather your ideas and feedback



Why is this important?

–

Threats in Cyberspace are exploiting vulnerabilities at alarming rates

–

DOD policies and procedures are changing to help DOD mitigate risks

–

Government, FFRDC, SETA and Industry partners must collaborate to

deliver operationally effective systems



How will we do it?

1.

Present proposed Cybersecurity T&E Process

2.

Gather your feedback as we go and

–

Have fun as we do it!

(3)

Bottom Line Up Front



Cyberspace is an ambiguous term

–

Liberally applied prefix to anything!



Cyber Space links Social, Information and Physical

Networks

–

Massive Attack Surface exposes “Information” to

Threats!



“Cyber” Threats exploit vulnerabilities

–

Threats exercise a “Kill Chain”



DOD SE and T&E Communities must collaborate

–

Or Mission Critical Information will remain vulnerable



Systems Acquisition and Test focus must shift

–

Assure the “Mission”



Cyber T&E must assess ability to execute Missions

–

Understand Threats, Evaluate Attack Surface and

Kill Chain to close vulnerabilities



US DOD T&E Community is working the issue

–

DT&E and DOT&E collaborated on methodology

–

Working Policy, Infrastructure and Workforce

(4)

Cybersecurity and DOD

DoD missions increasingly depend upon complex, interconnected IT environments. These environments are inherently vulnerable, providing

opportunities for adversaries to negatively impact DoD missions. A comprehensive T&E program is required to address cybersecurity, starting early in the acquisition lifecycle, to provide early discovery and

allow correction of developmental and operational issues in order to support the warfighter.

DOD Information

Network

Graphics Source: WIKIPEDIA Commons

(5)

Approved US Govt. Cybersecurity Definition



Cybersecurity

— The prevention of damage to, protection of,

and restoration of computers, electronic communications

systems, electronic communications services, wire

communication, and electronic communication. This includes

information contained therein, to ensure its availability, integrity,

authentication, confidentiality, and nonrepudiation.

–

Defined in National Security Presidential Directive-54/Homeland

Security Presidential Directive-23

(6)

Working Definition: Attack Surface

Information Domain Information Network

Cyberspace

“Exposes” Information and Data via Interconnected Social, Physical and

Information Networks

Attack Surface: A system’s exposure to reachable and exploitable cyber vulnerabilities within the system boundaries

Source: SANS Attack Surface Problem: http://www.sans.edu/research/security-laboratory/article/did-attack-surface Aug 2011: Comprehensive Experimental Analyses of

Automotive Attack Surfaces

Source: : University of California, San Diego, University of Washington Image Source: Josh O’Sullivan : MITRE Corp.

(7)

Working Definition: Cybersecurity Kill Chain and

Cyber Attack Lifecycle

Source: Mandiant APT 1 Attack Cycle MITRE: Cyber Attack Lifecycle

Cybersecurity Kill Chain: A sequence of actions performed by an adversary to execute cyber attacks with specific objectives, such as data theft.

Cyber Attack Lifecycle: A framework to understand and anticipate the moves of cyber adversaries at each stage of an attack.

(8)

Working Definition: Cybersecurity T&E



Cybersecurity T&E:

–

Examination of security measures to reduce

the attack surface and mitigate kill chain

effects in order to evaluate system resilience

in response to threat representative cyber

attacks. “



Cybersecurity T&E is not executed in a vacuum!

– In collaboration with Users, PM, Systems Engineers, Security Controls

Assessors

– Beginning prior to MS A and in conjunction with existing Systems

Engineering and Systems Security Engineering Processes

– In an incremental and iterative manner prior to

 Identify and verify baseline security requirements

 Mitigate exposed vulnerabilities and

 Assess a system’s resilience to execute Critical Operational Missions in response to threat representative cyber attacks including the ability to restore normal operations.

(9)

Challenges with DOD Cybersecurity, Capabilities

Development, Systems Acquisition and Test

Word Cloud: Google Search for “Cyber Security” 6,970,000 results (0.31 seconds)

(10)

Integrated Cybersecurity T&E Includes

SE/SSE and CIO Disciplines and Artifacts

Early Security Testing (test data) RMF Security Assessment Report/POA&M (attack surface analysis) DT Red Team “Kill Chain” Testing

Materiel Solution Analysis (MSA) DT Blue Team Testing RMF Security Plan SE Cybersecurity Requirements Validation RMF Security Assessment Plan & TEMP

T&E Step 5 Operational Cyber Vulnerability Evaluation

(11)

Cybersecurity T&E: Planned and Executed in

Collaboration: Acquisition, CIO, SE and T&E Aligned



Cybersecurity Engineering and Test must be integrated and iterative throughout the acquisition lifecycle

– Includes all communities

 Systems Engineering, IT, Security Control Assessor, etc.

–

Integrated Product Teams must align artifacts activities, within

acquisition milestones and events.

(12)

Cybersecurity T&E Process

Step 1

Understand Cybersecurity Requirements

Step 2

Characterize the Cyber Attack Surface

Step 3

Understand the Cybersecurity Kill Chain

Step 4

Cybersecurity DT&E

Beginning at Pre-MS A or Pre-EMD, with update at Milestone C: Understand Cybersecurity requirements and develop an approach for cybersecurity T&E. Requirements may be specified or implied.

Beginning at Pre-EMD: Characterize the attack surface; in the integrated environment, determine possible threat vectors.

Post CDR: Analyze and evaluate potential vulnerabilities to determine measures to improve resilience.

Prior to MS C: Cybersecurity DT&E event in a realistic mission environment, with use of cyber range, CNDSP, representative users and Cybersecurity threat representation.

Realistic developmental Cybersecurity DT&E event

Note: Steps may be iterative to resolve exposed vulnerabilities

In conjunction with MS C, operational test and evaluation event to assess residual vulnerabilities and risk. Director OT&E must approve entry to Step 6, based on resolution of vulnerabilities.

Operational Test Step 5 Operational Cyber Vulnerability Evaluation Step 6 Cyber Operational Resiliency Evaluation

Post-MS C operational test and evaluation event to assess operational capabilities to fight through Cyber Attacks

Realistic Cyber Threat Test

(13)

Example Cybersecurity T&E Example Mapped

to the Acquisition Lifecycle



Process fits but is not limited to the DoDI 5000.02 milestones

–

Steps are mapped to both milestones and design review steps

–

Programs have latitude on timing of Step activities



Process shifts “discovery” earlier within the acquisition life cycle

and builds in “fix-it” intervals



The findings in any one step may require revisiting a prior step

(14)

Experimental Analyses of Automotive Attack

Surfaces

■

Modern automobiles are

pervasively computerized



Engine, Transmission, Body, Airbag, Antilock Brakes, HVAC, Keyless Entry Control, etc.

■

Attack Surface extensive



Telematics: Blue Tooth, Cellular, Wi-Fi, Keyless Entry



Attack Surface is easily exploited

–

OBD Diagnostics

–

CD players

–

Bluetooth



Cellular radio/ Wi-Fi

–

Allow long distance vehicle

control, location tracking,

in-cabin audio exfiltration

Source: : University of California, San Diego: Stephen Checkoway, Damon McCoy, Brian Kantor, Danny Anderson, Hovav Shacham, and Stefan Savage

University of Washington: Karl Koscher, Alexei Czeskis, Franziska Roesner, and Tadayoshi Kohno

(15)

Step 1 - Understand Cybersecurity

Requirements

– Identify cybersecurity requirements for Cybersecurity T&E

- Review all available program resources

- Capabilities Document, Architectures, RFP, System Specification, Program Protection Plan,

- Identify critical operational missions and associated information systems

- Identify critical mission dependencies on hardware/software components that may be susceptible to cybersecurity intrusions

- Identify critical data exchanges and interfaces (include non-programmatic systems if applicable)

- Identify additional implied (derived) and essential requirements

– Identify cyber threat environment to be emulated in test

– Identify MAC/CL or RMF security categorization

– Identify cybersecurity test organization(s), including

- DIACAP/RMF security controls assessor

- Blue Team

- Red Team

– Identify Cybersecurity T&E resources

- Cyber range resources(e.g., National Cyber Range (NCR), DoD IA Range, Joint Information Operations Range (JIOR)) (See Backups for more detail)

- M&S or tools for cybersecurity

– Integrate cybersecurity into T&E events and/or

– Plan for dedicated cybersecurity test events as appropriate and if possible

 Plan may need to be revised and updated as understanding of Attack Surfaces and Vulnerabilities is refined

(16)

Cyber Security Requirements

 Specified requirements

– Requirements clearly identified in program documentatio

– ICDs/CDDs, CONOPs,

– Product Specifications and PPP

– DoD regulations, such a DoDI 8500.02 (DIACAP).

 DIACAP IA Controls (IACs) are identified as requirements based on a system’s MAC/CL designation

 Implied requirements

– Implied requirements are translated into technical requirements that enable the capabilities defined in CONOPS and other operational documentation

– AKA as Derived Requirements

– Requirements driven by operational capabilities

– Requirements driven by acquisition approach and/or technology choices

 Use of COTS/GOTS and free open source software (FOSS).

– Implied tasks include additional tasks the developer must accomplish to operate securely

 Includes the Cyber Threat environment

– Objective of Step 2

 Characterize the cyber attack surface to identify the additional implied cybersecurity requirements.

 Essential requirements

– Essential tasks are those that must be achieved to provide sufficient resilience to support mission accomplishment in the presence of cyber attack

– Objective of Step 3

 Analysis of potential kill chain activities to identify essential cybersecurity requirements necessary to improve resilience of the operational system to cyber attack.

(17)

Cybersecurity Testing Resources

Penetration

Testing (Red)

• “Graduation exercise”

• Exploit one or more known or suspected weaknesses

• Focus attention on specific problemor attack vector

• Both internal and external threats

• Develop an understanding of the

inherent weaknesses of a technology

• Model actions of a defined

internal or external hostile entity

• Conducted covertly with minimal staff knowledge

• May harm systems and

components and require clean up

Vulnerability

Assessment (Blue)

 Comprehensive

 Identifies any/all known

vulnerabilitiespresent in systems  Reveals systemic weaknesses in

security program

 Focuses on adequacy & implementation of technical security controls and attributes  Full knowledge and cooperation of

systems administrators

 Multiple methods used: hands-on testing, interviewing key personal, or examination of relevant artifacts  No harm to systems

 Feedback to developers and system administrators for system

remediation and mitigation

• Assesses compliance to IA controls

• Executes the Security Assessment Plan (SAP)

• Linked to the Certification and Accreditation of the system

• Based on Security Technical

Implementation Guides (STIGs) or

similar documentation

• Can be determined by multiple methods: hands-on testing, interviewing key personal, or examination of relevant artifacts

• Includes a review of operational and management security

controls

• Conducted with full knowledge and assistance of systems administrators, owner and developer

• No harm to systems

Security Controls

Assessors (SCAs)

(18)

Example Step 1: Understanding Cybersecurity

Requirements/Develop T&E Approach

Urban Assault Vehicle _{Example Requirements Resources} 1. System Threat Assessments

2. Capabilities Documents 3. Information Support Plan 4. Program Protection Plan

5. Mission Assurance Category: III 6. Confidentiality Level: Classified 7. Contract Specs

DODAF Architecture Products Cybersecurity T&E Approach

1. Early T&E involvement 2. Requirements Analysis 3. Design Reviews

4. Contactor SIL Testing

5. Blue Team DT&E Cyber Range 6. Red Team DT&E Event

7. Red Team OT&E in Field

(19)

Step 2 - Characterize the Cyber

Attack Surface

Characterize the attack surface to identify additional implied

cybersecurity requirements and possible threat vectors



In the integrated environment, determine possible threat

vectors

- Utilize cybersecurity SMEs to assist in analyzing the attack surface to determine

likely avenues of cyber attack

- Examine PPP, System Design, system architecture products (e.g., SV-1. SV-6

viewpoints) to identify interfacing systems, services, and data exchanges that may expose the system to potential threat exploits

- Examine system CONOPS to understand roles and responsibilities of a system

operators, administrators, and the computer Network Defense Service Provider (CNDSP)

- Identify host environment provisions for system protection, monitoring, access

control, system updates, etc.

- Evaluate early DIACAP/RMF and other security test artifacts

(20)

Example Step 2: Characterize the Attack

Surface

Urban Assault Vehicle Attack Surface Vehicle Attack Surface

1. Vehicle to Vehicle Comms 2. Telematics 3. Keyless Entry 4. OBD II 5. Radio 6. Anti Theft T&E Assess

1. Evaluate Contractor/SIL Security Artifacts 2. Baseline Cybersecurity posture

3. Approach to close/mitigate vulnerabilities 4. Likelihood of attack?

(21)

Step 3 – Understand the Cybersecurity

Kill Chain

Analyze and evaluate potential vulnerabilities to determine

measures to improve resilience (cyber range or lab)

-

Develop initial concept for cyber security testing activities at the component and subsystem level

• Identify test opportunities to conduct cybersecurity testing in a system of

systems context (such as JITC interoperability testing)

• Identify and integrate DIACAP/RMF security controls assessment activities into

unit testing. Functional testing, etc.

• Evaluate early DIACAP/RMF artifacts

-

Perform a vulnerability assessment using a Blue Team, to determine likely avenues of cyber attack and the most likely threat exploits

• Include or emulate the CNDSP

• Enumerate discovered vulnerabilities and supply to contractor for remediation

-

Analyze the kill chain to determine how the system would respond in the contested cyber domain

(22)

Prototype Cybersecurity Kill Chain Test

Overview

Blue Team/Red Team Portrays APT •Recon •Weaponize •Deliver •Exploit •Control •Execute •Maintain APT Objectives •Exfiltrate data

•Violate data availability •Corrupt data integrity

APT attempts multiple attacks while adjusting for success or failure Defenders attempt to analyze attacks and determine courses of action Defender Objectives

• Protect Against Intrusions

• Detect Intrusions

• React to Intrusions

• Mitigate Intrusions

• Determine Responses

• Restore After intrusion

SUT and CNDSP Portray Test Items

• Detect • Deny • Disrupt • Degrade • Deceive • Destroy • Recover Data Collection •Attacker actions •Defender detections •Defender actions •Mission activity

(23)

Example Step 3: Kill Chain Analysis

Vehicle SV-6 Systems Data Exchange Requirements

Urban Assault Vehicle Attack Surface Vehicle Attack Surface

1. Deny Vehicle/Vehicle Comms

2. Intercept Telematics

3. Clone Keyless Entry

4. Corrupt OBD II

5. Monitor Radio

6. Disable Anti Theft

T&E Activities

1. Verify/Exercise Critical Mission Threads 2. Exploit Kill Chain

3. Use Blue Team to Assess Vulnerabilities

(24)

Step 4 - Cybersecurity T&E

Evaluate system-of-systems cybersecurity in a mission

context, using realistic threat exploitation techniques



Conduct

Red Team

assessment to identify remaining

vulnerabilities

- Red Team emulates the threat adversary TTPs

- Red Team attempts to exploit the attack surface and execute cyber kill chain

activities

- Include or emulate the CNDSP - Include typical users if available



Identify exploitable threat vectors and vulnerabilities



Analyze results to determine impact to mission



Collaborate with PM and SE

to recommend corrective

actions to improve resilience

- May include non-materiel solutions such as TTP and recommendations to the

CNDSP

“Cyber resiliency” is the ability of a nation, organization, or mission or business process

(and supporting systems) to anticipate, withstand, recover from, and evolve to improve capabilities in the face of, adverse conditions, stresses, or attacks on the supporting cyber resources it needs to function.

(25)

Threat-Based Testing



Guided by a validated cyber threat assessment

–

STAR

–

Service/Component Capstone Threat Assessment



Focus testing on exploits and TTPs consistent with the

threat portrayal

–

Cyber Contested Environment portrayed by Vulnerability

Assessment Teams (

Blue

and

Red

)



How are mission functions impacted by threat adversary?

Graphic Sources: W IKIPEDIA Commons

(26)

Example Step 4: Full Up DT&E Red Team

Event

Urban Assault Vehicle Autobahn Mission

Exercise Critical Missions

1. Tx/RX Vehicle/Vehicle Comms 2. Cellular Phone Calls

3. Use Keyless Entry

4. Upload/Download OBD II Data 5. Tune Radio

6. Anti Theft

T&E Activities

1. Establish Representative Cyber Environment with Threats and Users

2. Conduct Red Team Assessment

3. Understand Mission Impacts 4. Evaluate Test Data

5. Produce DT&E Assessment

(27)

Step 5 – Operational Cyber Vulnerability Evaluation

 Step 5 - An operational cyber vulnerability assessment to determine readiness for operational evaluation

– Purpose

 OTA or a Blue Team will conduct an overt, cooperative, and comprehensive

vulnerability assessment in an operational environment

 Evaluate Configuration management, patch management, network access

controls, and system cybersecurity tools

 Leverage production-representative DT&E data to the maximum extent possible

– Provide vulnerability evaluation results and recommendations to materiel developers, as appropriate for remediation

– Vulnerability results should not be provided to Red Teams performing Step 6

– Correcting all vulnerabilities found during this step will be the entrance criteria for Step 6

– (Note) Vulnerabilities identified in Step 5 may require re-testing

– Preparation for IOT&E

– This step may also make use of available developmental test events and data as appropriate.

(28)

|28|

Example Step 5: Operational Cyber

Vulnerability Evaluation

Exercise Critical Missions 1. Tx/RX Vehicle/Vehicle Comms 2. Cellular Phone Calls

6. Anti Theft

T&E Activities

2. Conduct Blue Team Assessment

3. Evaluate Test Data

(29)

Step 6 - Cyber Operational Resiliency Evaluation

 Step 6 - A full-up operational test of the system-of-systems in a representative operational and threat environment

– Purpose

 Conduct an independent and comprehensive evaluation of protect, detect, react,

restore capabilities, to include exploitation potential, and mission impact.

 Some system information and network information may be provided to the Red

Team to facilitate the cybersecurity evaluation

– Red Team should not have access to the detailed Step 5 vulnerability evaluation

– Discover:

 How well do the system’s cybersecurity capabilities protect key/critical information

and data?

 Does the system’s ability to detect penetration and penetration attempts support the

rapid identification of hostile cyber activity?

 Does the system support rapid reaction and mitigation of penetration/exploitation?

 Does the system support reconfiguration and restoration of critical services, data,

and functions?



Systems with High/Medium risk to “CIA” for system information;

(30)

Example Step 6: Penetration Testing with

Representative Threat

Urban Assault Vehicle Autobahn Mission

Exercise Critical Missions 1. Tx/RX Vehicle/Vehicle Comms 2. Cellular Phone Calls

6. Anti Theft

T&E Activities

2. Conduct Red Team Assessment

3. Understand Mission Impacts 4. Evaluate Test Data

5. Produce OT&E Assessment

(31)

Cybersecurity T&E Key Take Aways!

 Cybersecurity T&E Process activities begin pre-Milestone A and continue throughout the Acquisition Lifecycle

– Collaborative process helps translate cybersecurity requirements, host environment, threats, etc. into testing activities

 Cybersecurity T&E process requires the development and testing of mission-driven cybersecurity requirements

– Requires systems engineering, systems security engineering and T&E expertise.

 Test and Evaluation Master Plan (TEMP) must detail

– How testing will provide the information needed to assess cybersecurity and

– Inform Systems engineering, Risk Management and Acquisition Decisions.

 Test activities must integrate

– RMF security controls assessments and

– Tests of commonly exploited and emerging vulnerabilities early

 Cybersecurity DT&E is expected to identify issues related to resilience of military capabilities before MS C

– Early developmental T&E provides data and feedback to the PM and SE Teams

– Informs requirements, facilitates change to minimize impact on cost, schedule, and performance

 Cybersecurity OT&E is expected to ensure that the system under test can

– Withstand realistic threat representative cyber-attacks and

– Return to normal operations in the event of a cyber-attack.

 Cybersecurity T&E Process represents a “shift left”

(32)

Closing



Next time someone says Cyber …..

– Stop and ask them what they really mean



Attack Surface in Cyberspace is massive

– How big is your attack surface? Is your information protected?



Cyber Threats exploit vulnerabilities

– Cyber Kill Chain must be understood and disrupted!



Current US DOD processes are being changed

– Systems Engineering, Systems Security Engineering, Developmental and Operational Test Communities are collaborating



Cyber Security must protect Mission Critical Information

– Information is the “What” Mission Assurance is the Why!



T&E seeks to ID Attack Surface and Disrupt Kill Chain!

– Close High Risk Vulnerabilities Early



US DOD T&E Community: Actively working the problem

– Methodology and Policy in work to shift discovery to left

– Cyber T&E Infrastructure and Workforce will enable and execute

Cyber Goths

(33)

“ T h e i n t e r n e t i s o n e g i g a n t i c w e l l

-s t o c k e d f r i d g e r e a d y f o r r a i d i n g ; f o r

s o m e s t r a n g e r e a s o n , p e o p l e g o u p

t h e r e a n d j u s t g i v e s t u f f a w a y

. ”

— M e g a ' Z i n e s , M a c w o r l d ( 1 9 9 5 )

Questions, Comments,

Recommendations?

Pete Christensen

T&E Portfolio Manager OSD Portfolio