This Conference brought to you by
www.ttcus.com
Linkedin/Group:
Technology Training
Corporation
Technology Training
Corporation
@Techtrain
Corporation
U S A
I t lli
d S
it C
d
U.S. Army Intelligence and Security Command
Army Intelligence and “Big Data”
29 S
t 2015
INSCOM … the Army’s Force for Dominant Intelligence
29 Sept 2015
Why?
Big Data
All Sensors need
Processing, Exploitation
and Dissemination (PED)
Unified Cloud Data (UCD): A Joint Interagency Effort
Aimed at Defining the
Unified Cloud Data (UCD)
•
Unified Cloud Data (UCD): A Joint, Interagency Effort
– Aimed at Defining the
Service Technical Approach to the ICITE Framework and Data … While Solving the
Global Processing, Exploitation & Dissemination (PED) Migration Challenges
•
UCD: Our “Big Data” Reference Implementation of a Unified Data Layer
•
All Data Work Regardless of Type of Intelligence and Security Level
•
All Data Work, Regardless of Type of Intelligence and Security Level
•
All Analytics + Enrichment Processes Run against UCD: Write Once, Use Often
•
All Indexes Make UCD Data Discoverable to Analysts
•
Security Is “Baked In”
– Security Markings Are Integrated at the Source Level,
Event / Document / Entity level, Down to Individual Attributes for an Entity
•
UCD Pilot
– Army’s Instantiation, Led by INSCOM, Assessed by Soldiers in Live
Environment – To Inform Army Programs of Record
•
Army Mission Command and Army Cyber Pilots show Benefit of
Unified Data
for Operations-Intel Convergence for the Army Operational Concept (AOC)
Extending Cloud Advanced Analytics
Extending Cloud Advanced Analytics
Open Source Information Alerts Analysts
•
Use Open Information
p
– Pilot New Analytics
y
•
Train Analysts on Sources and Capabilities
•
Sources Change; Terms Evolve
•
Stay Engaged: Frequent Changes
•
Share Information, including Concepts of Operation
P t i t C
t
t f Wh t’ K
•
Put into Context of What’s Known
•
Mature Techniques to Verify and Understand:
Who, What, Why??
•
Certify for Policy Compliance
•
Certify for Policy Compliance
•
Reinforce Mission and Legal Authorities
•
Do the Right Analysis, the Right Way
•
Leverage the Enterprise:
A
l ti C
biliti
•
Analytic Capabilities
•
Data Approach: Pay for Data Once, Use Many Ways
•
Enterprise Impact: Data Retention, Storage,
Correlation, Cyber Security…
Big Data:
U
f l f
Ti
i
C
i
Useful for Tipping + Cueing,
But Has Risks
The Enterprise is the Foundation
Foundation
Layer Backbone
Layer
Backbone
Unified Cloud Data (UCD) – Partners & Pilots
•
Demonstrated
Value:
ldi
d
i
+
Soldiers
used
UCD
+
Live
Data
+
Saved
Mission
Command
7+
months
to
pilot
Ops/Intel
Convergence,
g
,
DTRA
to
pilot
p
Constellation
on
UCD
baseline
+
Remote
Mgt +
Puppet
deployment
lets
1
SysAdmin
manage multiple UCD sites
manage
multiple
UCD
sites
+
Piloted
AWS
GovCloud for
Implementing Unified Cloud Data (UCD)
Implementing Unified Cloud Data (UCD)
Assessed UCD + Value of ‘Big Data’ Analytics to Inform Future Requirements
Good Feedback from Functional Assessment
Good Initial Feedback:
Users Want More Access + More Data
Assessed UCD + Value of Big Data Analytics to Inform Future Requirements
Improved Analyst Usability:
•
Ease of Use:
Easy Multi-INT exploitation of Unified Data with Widgets + Workflows•
Rapid Mastery:
After 3 days training, Soldiers could use UCD for mission threads•
Speed of Analysis:
Soldiers used UCD to do Country Study in 30 min (1/3 time)•
Fast Data Access:
Facial Recognition in seconds against 100Ks of recordsOperations-Intelligence Convergence:
•
Improved Situational Awareness:
BlueForce + Red Data in Common Operating Picture•
Pre-Deployment Checks:
Soldiers easily checked New Area:Know What’s Known
Enterprise Efficiencies and Security:
•
Built-In Support:
Self-Configurable Dashboards + Workflows Soldiers can share•
Info Sharing:
Built-in Reports/Report Creation (no support needed), Coalition Info Sharing•
Enterprise Operations:
Remote Admin by Fewer System Admins support multiple sitesUCD Support of Intel Functions
UCD Support of Intel Functions
Separate Data from Analytics
SensorsSeparate Data from Analytics
Security: P
UCD handles
Data Access: User Authorizations Sensors • Provenance• Security Labels • Metadata Tagging
• Extract Entities + Geo/Temporal Attributes • Metrics • more
many kinds
of data
Data Access: Match User Roles/Authorizations against Data SecurityData Ingestion
Data from Many Sources/Types •Images •Audio •Video Velocity + Content Authorizations Real Time
Community
Partners
•Video •Messages•Public Info
•Mission Command
•Etc. Analytics
Update Real-Time
Cell-Level Security
Analyst’s Conclusions
Indexes Enrich Data
•
Correlate All Data
•
Context-Based Data Navigation
•
Analysts Enrich Correlated Data:
+Know What’s Known Now
+Helps Analysts “Connect the Dots”
•
Supports ‘Big Data’ Analytics
• Map Reduce Analytics Enable Data Sharing
Context Based Data Navigation
+Beneficial to All Domains:
+Operations-Intel Convergence
+Medical Support
+Logistics Support
•
Supports ‘Big Data’ Analytics
and Multi-Discipline Fusion for
AOC environments: A2AD,
Megacities, Ad Hoc Response…
UCD Support of Intel Functions
UCD Support of Intel Functions
UCD handles
Separating Data from
Apps lets Analytics
U
S
D t
many kinds
of data
Use Same Data:
Read Once, See Data
Many Ways
UCD Support of Intel Functions
UCD Support of Intel Functions
UCD handles
Separating Data from
Apps lets Analytics
U
S
D t
Counter-Insider Threat
Security checks
many kinds
of data
Use Same Data:
Read Once, See Data
Many Ways
UCD Support of Intel Functions
UCD Support of Intel Functions
UCD handles
Separating Data from
Apps lets Analytics
U
S
D t
Counter-Insider Threat
Security checks
many kinds
of data
Use Same Data:
Read Once, See Data
Many Ways
Security checks
UCD implements full CRUD
functionality: Analysts can
UCD Support of Intel Functions
UCD Support of Intel Functions
UCD handles
Separating Data from
Apps lets Analytics
U
S
D t
Counter-Insider Threat
Security checks
many kinds
of data
Use Same Data:
Read Once, See Data
Many Ways
Security checks
UCD implements full CRUD
functionality: Analysts can
Create, Read, Update, Delete
UCD Support of Intel Functions
UCD Support of Intel Functions
UCD handles
Separating Data from
Apps lets Analytics
U
S
D t
Counter-Insider Threat
Security checks
many kinds
of data
Use Same Data:
Read Once, See Data
Many Ways
Security checks
UCD implements full CRUD
functionality: Analysts can
Create, Read, Update, Delete
UCD Unifies Data
Fine-Grained Data
Security Markings are
stored in Accumulo
UCD Support of Intel Functions
UCD Support of Intel Functions
UCD handles
Separating Data from
Apps lets Analytics
U
S
D t
Counter-Insider Threat
Security checks
many kinds
of data
Use Same Data:
Read Once, See Data
Many Ways
Security checks
UCD implements full CRUD
functionality: Analysts can
Create, Read, Update, Delete
UCD Unifies Data
Fi
G
i
d D t
Fine-Grained Data
Security Markings are
stored in Accumulo
Fast GPU-based Geospatial
and Temporal indexing
UCD Support of Intel Functions
UCD Support of Intel Functions
UCD handles
Separating Data from
Apps lets Analytics
U
S
D t
Counter-Insider Threat
Security checks
many kinds
of data
Use Same Data:
Read Once, See Data
Many Ways
Security checks
UCD implements full CRUD
functionality: Analysts can
Create, Read, Update, Delete
UCD Unifies Data
Fi
G
i
d D
Integrated Capabilities
Fine-Grained Data
Security Markings are
stored in Accumulo
g
p
such as Facial Recognition
Fast GPU-based Geospatial
and Temporal indexing
UCD Support of Intel Functions
UCD Support of Intel Functions
UCD handles
Separating Data from
Apps lets Analytics
U
S
D t
Counter-Insider Threat
Security checks
many kinds
of data
Use Same Data:
Read Once, See Data
Many Ways
Security checks
UCD implements full CRUD
functionality: Analysts can
Create, Read, Update, Delete
UCD Unifies Data
Integrated Capabilities
Fine-Grained Data
Security Markings are
stored in Accumulo
g
p
such as Facial Recognition
Can Make Workflows –
Fast GPU-based Geospatial
and Temporal indexing
Can Make Workflows
for Analyst Tasks + for
Conform to the Enterprise
Constraints We Must Handle
Constraints We Must Handle
Reduce Costs
Conform to the Enterprise
Reduce Costs
•
Software Licenses
•
Physical Footprint
•
IT Support
pp
•
Leverage Open Source / Other
Software
Increase Security
Increase Security
•
Provenance: Track Every Interaction
•
PKI and Thin Client
•
Bastion Node Construct
Expect Change
•
New Kinds of Data: OSINT/Social Media
•
Remote Administration: Separate
System Administration from the Data
Exploit Exponential
•
New Kinds of Data: OSINT/Social Media
•
New User Needs: Heat Map
•
New Capabilities from Partners: WAMI
Track Extraction from NGA
Increase in Data
•
New Technology Components: GPU
Protect the Data:
UCD Lessons Learned
UCD Lessons Learned
Protect the Data:
Build on a secure Cloud architecture with cell/object-level security, and extend security down to the weapon system to be able to Counter Insider Threat (with PKI, provenance etc.).
"Big Data” Strategy with Unified Data:
Capture and triage vast, increasing amounts of p g , g data of all typesyp , from many sources, with automated "Big Data" , y , g analytics. Analytics from different providers should leverage the same correlated data: write once, use many ways.
Enterprise-Scale Remote Administration:
Leverage automated build, delivery, install, configuration management, system administration, and monitoring to make operational use simple and scalable to the Enterprise level: automate routine tasks so humans focus on problems.
Leverage Prior Capabilities with Enterprise Platform:
Break apart legacy stovepipe capabilities from battlefield-proven Quick Reaction Capabilities (QRCs); re-host unique components on IC ITE security architecture to ensure needed capabilities endure for the future.
Transform Capabilities Acquisition:
To benefit from the innovation & speed of new capabilities, at long-term greatly reduced cost, incentivize organizations and individuals to use Open Source software, Agilebusiness models of vendors & gov’t organizations.
Encourage and Reward Partnering:
Strong INSCOM + Mission Command partnership integrated INSCOM UCD software onto MC tactical cloud servers: at NIE 15 1 showed
Intel/Ops Convergence
situational awareness DIL operations Bde TOC server consolidation NIE 15.1, showedIntel/Ops Convergence
, situational awareness, DIL operations, Bde TOC server consolidation …Future Ready for Capabilities Integration:
Future analytic capabilities that exploit Unified Data can be rapidly integrated and (done right) can inherit security.
Need New Categories of Innovative Capability:
Cyber Security and Info Assurance require new Enterprise analytics, to understand threats, attacks, system health – but we also need Innovative Capabilities + Processes to demonstrate Info Assurance (IA) and Capabilities Security.