DPI and Metadata for Cybersecurity Applications

(1)

White Paper

DPI and Metadata for

Cybersecurity Applications

How vendors can improve solutions for new market

demands by filling the gap between COTS

cybersecurity and raw data analysis

(2)

www.qosmos.com 2

Executive Summary

According to the Verizon 2013 Data Breach Investigations Report, 78% of breaches take weeks or months to discover. The authors stress the importance of a strong strategy for detection and response:

“Prevention is crucial, and we can’t lose sight of that goal. But we must accept the fact that no barrier is impenetrable, and detection/response represents an extremely critical line of defense. Let’s stop treating it like a backup plan if things go wrong, and start making it a core part of THE plan.”1

This highlights the need for situational awareness, now a necessary pillar of effective cyberdefense. In today’s world, organizations must assume that their networks will be compromised. In order to accelerate breach detection and mitigation, they need to improve their understanding and monitoring of normal network behavior. Vendors of cybersecurity solutions are the most likely sources for help. This paper offers a viable strategy for vendors to increase the effectiveness of products such as next-generation firewalls and solutions for NBAD, SIEM and DDoS attacks by using traffic metadata. It explains how metadata can be leveraged to see good from bad network behavior faster than current COTS products and raw data analysis using data logs, full packet capture and traditional Deep Packet Inspection (DPI).

Metadata strengthens cybersecurity solutions by providing behavioral context to traffic monitoring. Vendors can use metadata to:

 Establish good and bad behavior for services, and how they communicate  Rapidly define/map behavioral patterns for a host or user

 Build custom protocol “validators” to look for system exploits

 Allow analysts to query and investigate data in new and more effective ways

 Reduce data storage requirements by 10x – 150x compared to full packet capture (enabling storage of a year’s worth of metadata, which is unfeasible with full packet capture)

 Deliver products with higher quality security and fewer false positives

By enabling cybersecurity tools to search through data faster, and with fewer demands on IT

organizations to improve their situational awareness, vendors stand to increase their market relevance and value to customers as partners in cyberdefense.

(3)

www.qosmos.com 3

Vendors of cybersecurity

solutions can drive emerging

defense trends by engineering

intelligence for situational

awareness into their offerings.

Cyberdefense Trends: The Need for Situational Awareness

Networks today must support increases in data size and availability. Organizations have more network data to analyze and less certainty about what data they need to analyze. Data sets are more complex, making it harder to extract meaningful information. Cybersecurity teams can no longer afford to police all their data manually and must outsource all or part of their security analysis to third parties, where analysis can easily lose context for a specific business environment or security objective.

The volume of data and proliferation of threats change how organizations should now approach cyberdefense.

Recommendations from cybersecurity experts prioritize situational awareness and breach detection over impossible 100% prevention. Strategy should emphasize network intelligence gathering and analysis, and “smart” network monitoring as the best defense against threats.

How Conventional Security Analysis Limits Detection and Mitigation

Figure 1 shows a representative example of conventional security analysis. Billions of raw data elements collected over a period of time are screened in stages down to a few thousand investigated events.

Figure 1. Conventional Security Analysis

Within the Security Operations Center (SOC), traffic records are analyzed using conventional tools of choice. The time and resources required to validate data, examine events, identify breaches and mitigate them, as revealed in the Verizon report, can take weeks to months at substantial costs for the tools and talent.

This is largely due to a gap between conventional COTS tools and the manual analysis performed by security teams (Figure 2). The inability to search for data patterns in the context of user behavior and application usage makes useful pattern detection (and quick breach mitigation) difficult.

(5)

www.qosmos.com 5

Figure 2. Gap between COTS Tools and Raw Data Analysis

Shifting Mindsets to Situational Awareness

The use of metadata to improve the situational awareness of cybersecurity requires a shift in mindsets among vendors and their customers. Instead of preventing attacks, the premise should be that breaches will occur. Objectives should shift to detecting breaches faster by understanding the behavior and use of applications in traffic flows to recognize anomalies. Instead of thinking of cybersecurity as a discrete solution, the approach should be the integration of security with applications and web logs. Instead of relying only on protocol signatures to monitor traffic, vendors must enable products with real-time visibility into traffic patterns based on user behavior and application usage—made possible with metadata.

(6)

www.qosmos.com 6

Metadata bridge the gap

between conventional COTS

and raw data analysis,

enabling vendors to improve

their customers’ situational

awareness for cyberdefense.

The Value of Traffic Metadata

What security teams need, and what vendors should seek to provide, are capabilities to examine traffic data with the quality of full packet inspection coupled with indexed searching of protocol attributes to find meaningful user and application behavior patterns. This would improve the situational awareness of customers’ cyberdefense, and is within reach for vendors through the use of traffic metadata.

Metadata bridge the gap between conventional tools and raw analysis by enabling detection and differentiation of good and bad behavior patterns in network traffic flows. Traffic metadata provide the following advantages for vendors of cybersecurity solutions and their customers:

 Full classification and decoding of network protocols Layers 4-7, describing as many protocol and application attributes as needed

 Extraction from traffic in real-time without the need for data aggregation, formatting and database searches; they are therefore more precise, faster and easier to use than data logs  Analysis of traffic without the need to store full, raw data packets, reducing storage

requirements by a ratio of 1000:1, compared to processing packet captures and/or Syslog  Application and session aware, and capable of tracking multiple flows with a single protocol (e.g.

an FTP connection and data channels)

Figure 3 shows examples of traffic metadata. Some protocols and applications can have more than 50 metadata attributes—totaling thousands of attributes collectively—which can be selected, correlated and analyzed to provide a complete understanding of the quality and purpose of network events.

Figure 3. Examples of Application and Protocol Metadata Application Examples Typical Metadata

Email and Webmail Sender, receiver, login, subject, message, attachments, date and time Social Networking (Facebook,

Twitter, Baidu, etc.)

IM (MSN, Yahoo, Skype, QQ, etc.) Web Apps (YouTube, eBay, etc.)

User login, application URLs/activity, posting activity, tweets, search engine queries, and resulting URL clicks, chats, video streams, file attachments.

Business Apps (CRM, ERP, Citrix, MS Exchange, etc.)

User login, login / logoff date and time, data transfer sessions (type, content, time), volume (per user, IP, subnet, application)

Protocol Examples Typical Metadata

HTTP (Hypertext Transfer Protocol) URL, browser, cookies, URI, referrer,

GTP (GPRS Tunneling Protocol) Device, user location, quality of service (QoS) metrics, time, duration UDP (User Datagram Protocol) Source port, destination port, client port, server port

IP (Internet Protocol) Source address, destination address, source port, destination port, data RTSP (Real Time Streaming Protocol) Play/pause, streaming file, URL, duration

(7)

www.qosmos.com 7

Figure 4 compares records from Netflow, an industry standard for IP traffic monitoring, with and without metadata from Qosmos. Netflow alone is fast and repeatable but, because it is neither

application nor protocol aware, the standard Netflow record doesn’t disclose potential threats. Security specialists must still screen and analyze full data packets and logs manually to find behavioral context using increasingly outdated tools. Event correlations and differentiation of abnormal from normal behavior are difficult.

Figure 4. Extended Visibility into Layers 4 through 7 with Metadata

Qosmos metadata parse traffic in real time for user behavior and application usage, providing insight into what actually occurred between source and destination. In Figure 4, the Qosmos metadata additions to the Netflow record in this example reveal:

1. A referring party (chicaroo.cc) – Why would chicaroo.cc be referring users to our site?

2. A suspicious URL (http://www.golf.com/failed login.php) and no cookies – Why would anyone go directly to a failed login page without a session cookie?

3. A suspicious browser (cURL2.x) – not Internet Explorer, Firefox or Chrome, etc., but a command line version of a browser typically used in malicious scripts.

4. The server code is giving a positive result (200) despite the record’s irregularities – Is someone exploiting a vulnerability?

Without the Qosmos metadata, the Netflow record shows how much data was transferred, between what ports and when, but security specialists must still make assumptions when screening traffic data for suspicious activity. The same record enhanced with the Qosmos metadata tells security specialists what actually transpired in the communication. It enables accurate real-time traffic monitoring of both normal and abnormal behavior—i.e. situational awareness. Security specialists can work with useful pattern detection and know specifically which records to investigate. In this way, the use of metadata can reduce breach detection from weeks and months to hours and minutes.

(8)

www.qosmos.com 8

By enhancing their solutions

with metadata, firewall

vendors can enable their

customers to filter traffic

based on true protocol

recognition from Layers 1 to 7.

Use Case: Firewalls with Metadata

Firewalls historically employ five-tuple filtering of ports and Internet protocols. Basic port inspection, stateful inspection and protocol detection are typically based on predefined ports. Traffic protocols and applications cannot be identified unless ports and IP address are known.

Firewalls today need full traffic visibility independent of ports to block security breaches initiated, for example, through

social networking applications, instant messaging and email. By enhancing solutions with metadata (Figure 5), firewall vendors can enable their customers to filter traffic in real time based on true protocol detection from Layers 1 to 7, so firewalls act with application awareness. For example, they may wish to allow access to Facebook, but not allow access to Facebook game applications like Farmville.

Without metadata, vendors risk considerable time and money developing technology such as Deep Packet Inspection (DPI) and keeping up with changing protocols and applications to defend against new threats. And firewall customers must wrestle with the maintenance overhead of trying to block

applications with IP addresses and ports.

(9)

www.qosmos.com 9

Metadata improve the ability

of NBAD vendors to design

solutions that query traffic for

suspicious activity and

investigative analysis.

Use Case: NBAD with Metadata

Network Behavior Anomaly Detection (NBAD) requires visibility into normal network behavior as a baseline in order to flag abnormal behavior. To be fully effective, this should include the real-time monitoring of protocols, payloads, virus detection, bandwidth and connection rates.

Metadata improve the ability of NBAD solutions to query

traffic for suspicious activity and investigative analysis by providing information on:  Services or encrypted traffic on non-standard ports

 Hosts connecting to port 443 and not using the SSL protocol

 Hosts that started a flow with one protocol, and then changed mid stream (started with SSL then changed to something else)

 File attachments and user names in any supported protocol (Webmail, HTTP, FTP, file sharing, IM, etc.)

 A referrer or URL related to a phishing campaign

 The longest session with high error responses from the server  Sessions with traffic that falls out of protocol spec

Metadata are essential to effective cybersecurity because they provide the capability to see traffic patterns and to detect cookies that can reveal abnormal network behavior to know where a visitor is on a website at any given time. Metadata are a perfect complement to logs since they can be mixed with log information for a single, more robust view of traffic that can be indexed with data collection and indexing tools. And metadata improve investigative capabilities by requiring less storage than full packet capture, leading to faster data searches and the archiving of historical data for much longer periods of time.

Use Case: SIEM with Metadata

Security Information and Event Management (SIEM) is constantly required to process and analyze increasing quantities of data, and has a difficult time keeping up with the rising volume of network traffic. At the same time, vendors of Intrusion Protection Systems (IPS) and Intrusion Detection Systems (IDS) are trying to reduce the number of false alerts—false negatives from IPS and false positives from

(10)

www.qosmos.com 10

The use of metadata enables

IPS/IDS vendors to deliver

solutions with better

performance, stronger

capabilities and a competitive

advantage against solutions

that rely only on protocol

signature sets.

IDS—which diminish their effectiveness. Today, SIEM lacks scalability. And since most solutions use Netflow as an index to correlate events, SIEM also lacks the context of user behavior and application usage to establish situational awareness for allowing or blocking traffic accurately.

Metadata (Figures 6 and 7) enable events to be validated before going to full packet capture and provide an accurate and scalable method to build behavioral rules for more reliable alerts. For example, to better qualify IDS alerts, they can be weighted with

metadata to screen for traffic parameters such as browser type, URL length, referrer, cookies, connection time, protocol, protocol change and more. Metadata can be used to:

 Show all inbound IDS/WAF (Web Application Firewall) alerts for a cross site scripting (XSS) vulnerability hidden in “good traffic” (correlated server response of 200)

 Reveal the least common URL in traffic and any related IDS/IPS alerts

 Classify protocols on any port and process only the rules that apply for each protocol (10 to 200 rules per traffic flow instead of 2,000 rules per flow)

 Make more informed decisions on which traffic to allow and block with in-line IPS solutions Figure 6. IDS Improvement with Metadata

(11)

www.qosmos.com 11

Metadata improve visibility

and understanding of network

traffic and applications, giving

DDoS detection the ability to

clearly differentiate good from

bad traffic, especially for

application-level attacks.

Figure 7. IPS Improvement with Metadata

The use of metadata enables IPS/IDS vendors to deliver solutions with better performance, stronger capabilities and a competitive advantage against solutions that rely only on protocol signature sets. For example, metadata:

 Reduce the noise that IDS produces, and the tuning and filtering time of false positives by 50%  Reduce the false negative rate of rules in IPS by 50%, which enables customers to make more

efficient use of their IPS without dramatic changes in the way they manage it

 Detect protocol changes in traffic flow beyond the first 5-10 MB , which eliminates a huge limitation of purely signature-based IDS/IPS solutions

Use Case: DDoS Mitigation with Metadata

The use of metadata enables vendors and Managed Security Service Providers (MSSPs) to dramatically improve protection against all types of Distributed Denial of Service (DDoS) attacks—especially emerging attacks like application-level DDoS—at significantly reduced cost per customer. SYN floods are easy to detect and block for most vendors today, but application-level DDoS detection and mitigation (Figure 8) remain difficult since they require better visibility and understanding of protocol and application behavior.

(12)

www.qosmos.com 12

Vendors who use Qosmos

benefit from market-leading

traffic parsing technology that

can accelerate the delivery of

application-aware solutions.

Figure 8. Application-Level DDoS Detection and Mitigation with Metadata

Protocol decoding and metadata extraction up to Layer 7 provide complete visibility of all network traffic and applications, giving DDoS detection solutions the ability to clearly differentiate good from bad traffic, especially for application-level attacks. For example, detection can be based on sessions

connecting without a session key established over a given time, a URL, referrer, browser, etc. Even SSL renegotiation attacks become easy to identify on a per flow basis when based on metadata attributes. For vendors, the competitive advantage comes from giving customers better protection against attacks without having to increase the expertise or headcount of security specialists. The time to tune solutions is less per site and defense against DDoS is stronger with fewer chances of false alerts.

Qosmos DPI Metadata for Cybersecurity Vendors

Qosmos specializes in software libraries and tools for

cybersecurity vendors to enhance their solutions with DPI and Metadata Data Extraction. Qosmos technology enables vendors to make their solutions “applications aware” and thereby increase their value to customers by responding to today’s need for greater situational awareness in cyberdefense. As shown in Figure 9, the Qosmos ixEngine Software

Development Kit (SDK) and Qosmos Labs support services easily integrate into new or existing solutions. Vendors benefit from market-leading traffic parsing technology to accelerate the delivery of application-aware solutions. Using Qosmos reduces a vendor’s time to market with next-generation solutions;

(13)

www.qosmos.com 13

development time, cost and resource requirements; and the time to update constantly changing protocols.

Figure 9. Qosmos Embedded DPI and Metadata Technology

Figure 10. Qosmos Technology Alignment with Cybersecurity Vendors

Vendors’ Requirements Qosmos Technology

Support for Network Processors Qosmos ixEngine DPI and Metadata Extraction SDK optimized on Cavium, Intel, NetLogic, and Tilera

Performance & Scalability Handles up to 10 Gbps of traffic on a single chassis

Easy Integration C Library APIs (Can also share state table from another application )

Robust Protocol Extraction Rapidly extracts metadata from traffic flows, and can change protocol configuration on the fly. (Supports many protocols active at once.)

Wide Range of Protocols & Metadata 2,500 application protocols classified and 4,300 metadata extracted

Integration Support Quickly integrates into most systems (90 day acceleration program offered). Provides differentiating technology in security areas where security metadata are just being adopted.

Standardize the Output for Metadata Provide help in standardizing output for syslog/NetFlow/ipfix, (provide a standards framework)

(14)

www.qosmos.com 14

All next-generation

cybersecurity solutions will

leverage traffic metadata.

Figure 10 summarizes how Qosmos technology directly aligns with vendors’ needs. Qosmos offers market-leading protocol and application expertise, featuring:

 A library of plugins for classifying more than 2,500 protocols and applications  An industry-best 4,300 metadata attributes

 Advanced traffic parsing capabilities for tunneling, unidirectional flows, fragmented and partial traffic, and packet by packet inspection

 Robust architecture to handle abnormal and forged traffic  Support for core network speeds

 A custom protocol plugin SDK that allows vendors to develop and update their own protocols  On-demand protocol and application development by Qosmos

Improving Your Value Proposition to Customers

Cybersecurity experts and industry analysts recognize that not all data breaches can be prevented. Adversaries motivated by espionage, fraud, terrorism, socio-political agendas and simply mischief are too many in number and too creative with easy access to increasingly sophisticated tools. The new strategy for

effective cyberdefense is to assume some breaches will occur, but to promptly detect and mitigate them with greater situational awareness of network activity.

Traffic metadata provide vendors of cybersecurity solutions with the power to improve their value proposition to customers with solutions that increase their customers’ situational awareness. This encompasses much better capabilities to 1) differentiate good traffic from bad, and 2) detect and mitigate data breaches faster. All next generation cybersecurity solutions will leverage traffic metadata. The technology, in the form of software development kits and intelligent IP probes backed by constantly updated libraries of protocol signatures and metadata attributes, already exists through specialist companies like Qosmos. Without having to invest substantial time and resources, vendors can integrate pre-developed technology into solutions, complete with robust technical support. Many already have or are in the process of doing so.

###

Copyright © 2014 Qosmos S.A. All rights reserved. Qosmos, the Qosmos logo, Qosmos Service Aware Module, Qosmos SAM and Qosmos ixEngine are trademarks of Qosmos. Other names and brands may be claimed as the property of others.

DPI and Metadata for Cybersecurity Applications

White Paper