Total Visibility 1 1

(1)

(2)

What Is Meant by

Telemetry ?

Te·lem·e·try—

a technology that allows the remote

measurement and reporting of information of interest to the system designer or operator. The word is derived from

(3)

Check List

•  Check SNMP. Is there more you can do with it to

pull down security information?

•  Check RMON. Can you use it?

•  Check Netflow. Are you using it, can you pull

down more?

•  Check Passive DNS

(4)

Holistic Approach to

System-Wide Telemetry

Cardiologist Ophthalmologist Neurologist Nephrologist

Podiatrist

Holistic Approach to Patient Care

Uses a system-wide approach, coordinating with various specialists, resulting in the patient’s better overall health and wellbeing.

(5)

Holistic Approach to

System-Wide Telemetry

Data Center:

• Inter as well as Intra Data Center traffic

Customer Edge:

• Shared resources and

services should be available

Core:

• Performance must

not be affected _•_{Ability to trace}SP Peering:

through asymmetric traffic P P P P PE P P PE(s) L2 Agg. Broadband, Wireless (3G, 802.11), Ethernet, FTTH, Leased Line, ATM,

Frame-Relay

CPE(s)

P P

Data/Service Center

CPE/ACCESS/AGGREGATION CORE DATA/SVC PEERING

Center ISP / Alt. Carrier Listen Listen Listen Listen

(6)

Source: University of Wisconsin

Open Source Tools for NetFlow

Analysis Visualization—

FlowScan

Investigate the spike

(7)

What s NetFlow?

•  NetFlow is a form of telemetry pushed from the

network devices.

•  Netflow is best used in combination with other

technologies: IPS, vulnerability scanners, and full traffic capture.

–  Traffic capture is like a wiretap

–  NetFlow is like a phone bill

•  We can learn a lot from studying the network phone

bill!

–  Who s talking to whom? And when?

–  Over what protocols & ports?

–  How much data was transferred?

–  At what speed?

(8)

Elements of a Netflow Packet

• Packet Count! • Byte Count! Usage! QoS! Time! of Day! Application! Port! Utilization! From/To! Routing and! Peering! • Input ifIndex! • Output ifIndex! • Type of Service! • TCP Flags! • Protocol! • Start sysUpTime! • End sysUpTime!

• Source TCP/UDP Port!

• Destination TCP/UDP Port!

• Next Hop Address!

• Source AS Number!

• Dest. AS Number!

• Source Prefix Mask!

• Dest. Prefix Mask!

• Source IP Address!

• Destination IP Address!

Ingress i/f

Egress i/f

Data Flow Data Flow

(9)

Netflow Setup

•  Don t have a copy of netflow data b/c IT won t

share?

–  Many products have the ability to copy flow data off to

other destinations Collector Peakflow NetQoS Storage Export netflow data to OSU Flowtools Collector Regionalized collection to minimize WAN impact Netflow data copied to other destinations with flow-fanout

(10)

NetFlow Collection at Cisco

•  DMZ Netflow Collection (4 servers)

•  Data Center Netflow Collection (20+ servers)

•  Query/Reporting tools (OSU Flowtools, DFlow, Netflow Report Generator)

200K pps

3 ISP gateways 600GB ~ 3 months

(11)

OSU Flowtools

Netflow Collector Setup

•  Tool: OSU FlowTools!

–  - Free!!

–  - Developed by Ohio State

University!

•  Examples of capabilities!

Did 192.168.15.40 talk to

216.213.22.14?!

What hosts and ports did

192.168.15.40 talk to?!

Who s connecting to port TCP/

6667?!

Did anyone transfer data >

(12)

OSU Flowtools Example

Who s Talking?

•  Scenario: New botnet, variant undetected

–  You need to identify all systems that talked to the

botnet C&C

–  Luckily you ve deployed netflow collection at all your

PoPs

put in specific query

syntax for the example!

[mynfchost]$ head flow.acl

ip access-‐list standard bot permit host 69.50.180.3 ip access-‐list standard bot permit host 66.182.153.176

[mynfchost]$ flow-‐cat /var/local/flows/data/2007-‐02-‐12/ft* | flow-‐filter -‐Sbot -‐o -‐...

Start End Sif SrcIPaddress SrcP DIf DstIPaddress DstP 0213.08:39:49.911 0213.08:40:34.519 58 10.10.71.100 8343 98 69.50.180.3 31337 0213.08:40:33.590 0213.08:40:42.294 98 69.50.180.3 31337 58 10.10.71.100 83

flow.acl file uses familiar ACL syntax. create a

list named bot _{concatenate all}

files from Feb 12, 2007 then filter for src or dest of bot acl we ve got a host in the botnet!

(13)

Custom NetFlow Report Generator

Query by IP

(14)

Know Thy Subnets

•  Critical to providing context to an incident

–  Is the address in your DMZ? lab? remote access? desktop? data

center?

•  Make the data queryable

–  Commercial & open source products available

•  Build the data into your security devices

–  SIMS - netForensics asset groups

–  SIMS - CS-MARS network groups

–  IDS - Cisco network locale variables

variables DC_NETWORKS address 10.2.121.0-‐10.2.121.255,10.3.120.0-‐10.3.127. 255,10.4.8.0-‐10.4.15.255

variables DMZ_PROD_NETWORKS address 198.133.219.0-‐198.133.219.255 variables DMZ_LAB_NETWORKS 172.16.10.0-‐172.16.11.255

eventId=1168468372254753459 eventType=evIdsAlert hostId=xxx-‐dc-‐nms-‐4appName=sensorApp appInstanceId=6718 tmTime=1178426525155 severity=1 vLan=700 Interface=ge2_1 Protocol=tcp riskRatingValue=26 sigId=11245 sigDetails=NICK...USER" src=10.2.121.10 srcDir=DC_NETWORKS srcport=40266 dst=208.71.169.36 dstDir=OUT

data center

(15)

Network Telemetry - MRTG/RRDTool

!

•  Not just netflow, can also use SNMP to grab telemetry! •  Shows data volumes between endpoints!

You must understand your network traffic volume!

(16)

•  Network traffic data

•  Subnet information - IP address management

data

»  10.10.0.0/19 A (Active) Data Centers!

»  |-- 10.10.0.0/20 A (Active) Building 3 Data Center!

»  | |-- 10.10.0.0/25 S (Active) Windows Server Subnet!

»  | |-- 10.10.0.128/25 S (Active) Oracle 10g Subnet!

»  | |-- 10.10.1.0/26 S (Active) ESX VMWare Farm!

»  | |-- 10.10.1.64./26 S (Active) Web Application Servers!

»  10.10.0.0/16 A (Active) Indiana Campus!

»  |-- 10.10.0.0/19 A (Active) Data Centers!

»  |--10.10.32.0/19 A (Active) Site 1 Desktop Networks!

»  | |-- 10.10.32.0/24 S (Active) Building 1 1st floor!

»  | |-- 10.10.33.0/25 S (Active) Building 1 2nd floor!

Based on our design, environment, and

these aggregate traffic levels with

spikes above 400Mbps, We need

an IPS 4260

(17)

Source: UNINETT

(18)

Source: http://people.ee.ethz.ch/~oetiker/webtools/rrdtool/

Other Visualization Techniques

Using SNMP Data with

RRDTool

Anomaly for DNS Queries

Thru’put Spike RTT Spike

(19)

Source: http://www.ntop.org

Displaying RMON—ntop

Examples

Detailed

(20)

(21)

Correlating NetFlow and

Routing Data

Matching data collected from different tools

(22)

Syslog

•  De facto logging standard for hosts, network infrastructure

devices, supported in all most routers and switches

•  Many levels of logging detail available—choose the level(s)

which are appropriate for each device/situation

•  Logging of ACLs is generally contraindicated due to CPU

overhead—NetFlow provides more info, doesn’t max the box

•  Can be used in conjunction with Anycast and databases

such as MySQL (http://www.mysql.com) to provide a

scalable, robust logging infrastructure

•  Different facility numbers allows for segregation of log info

based upon device type, function, other criteria

•  Syslog-ng from

http://www.balabit.com/products/syslog_ng/ adds a lot of

useful functionality—HOW-TO located at

(23)

Benefits of Deploying NTP

•  Very valuable on a global network with network

elements in different time zones

•  Easy to correlate data from a global or a sizable

network with a consistent time stamp

•  NTP based timestamp allows to trace security

events for chronological forensic work

•  Any compromise or alteration is easy to detect as

network elements would go out of sync with the main ‘clock’

•  Did you there is an NTP MIB? Some think that we

may be able to use “NTP Jitter” to watch what is happening in the network.

(24)

Source: http://www.ethereal.com

Packet Capture Examples

Wealth of

information, L1-L7 raw data for

(25)

Total Visibility

Addendum

(26)

NetFlow—More Information

•  Cisco NetFlow Home—

http://www.cisco.com/

warp/public/732/Tech/nmp/netflow

•  Linux NetFlow Reports HOWTO—

http://

www.linuxgeek.org/netflow-howto.php

•  Arbor Networks Peakflow SP—

http://www.arbornetworks.com/

(27)

More Information about SNMP

•

Cisco SNMP Object Tracker—

http://www.cisco.com/pcgi-bin/Support/

Mibbrowser/mibinfo.pl?tab=4

•

Cisco MIBs and Trap Definitions—

http://www.cisco.com/public/sw-center/

netmgmt/cmtk/mibs.shtml

•

SNMPLink—http://www.snmplink.org/

•

SEC-1101/2102 give which SNMP

(28)

RMON—More Information

•  IETF RMON WG—

http://www.ietf.org/html.charters/

rmonmib-charter.html

•  Cisco RMON Home—

http://www.cisco.com/

en/US/tech/tk648/tk362/tk560/

tech_protocol_home.html

•  Cisco NAM Product Page—

http://www.cisco.com/en/US/products/

(29)

BGP—More Information

•  Cisco BGP Home—

http://www.cisco.com/en/US/tech/tk365/

tk80/tech_protocol_family_home.html

•  Slammer/BGP analysis—

http://www.nge.isi.edu/~masseyd/pubs/

massey_iwdc03.pdf

•  Team CYMRU BGP Tools—

(30)

Syslog—More Information

•

Syslog.org -

http://www.syslog.org/

•

Syslog Logging w/PostGres HOWTO—

http://kdough.net/projects/howto/

syslog_postgresql/

•

Agent Smith Explains Syslog—

(31)

Packet Capture—More

Information

•  tcpdump/libpcap Home—

http://www.tcpdump.org/

•  Vinayak Hegde’s Linux Gazette article—

http://www.linuxgazette.com/issue86/

(32)

Remote Triggered Black Hole

•  Remote Triggered Black Hole filtering is the

foundation for a whole series of techniques to

traceback and react to DOS/DDOS attacks on an ISP’s network.

•  Preparation does not effect ISP operations or

performance.

•  It does adds the option to an ISP’s security toolkit.

(33)

More Netflow Tools

•  NfSen - Netflow Sensor

–  http://nfsen.sourceforge.net/

•  NFDUMP

–  http://nfdump.sourceforge.net/

•  FlowCon

Total Visibility 1 1

What Is Meant by

Holistic Approach to

Holistic Approach to Patient Care

Holistic Approach to

Data Center:

Frame-Relay

CPE/ACCESS/AGGREGATION CORE DATA/SVC PEERING

Center ISP / Alt. Carrier Listen Listen Listen Listen

Open Source Tools for NetFlow

Investigate the spike

What s NetFlow?

network devices.

Elements of a Netflow Packet

Ingress i/f

Netflow Setup

NetFlow Collection at Cisco

3 ISP gateways 600GB ~ 3 months

OSU Flowtools Example

[mynfchost]$ head flow.acl

Custom NetFlow Report Generator

variables DC_NETWORKS address 10.2.121.0-­‐10.2.121.255,10.3.120.0-­‐10.3.127. 255,10.4.8.0-­‐10.4.15.255

Network Telemetry - MRTG/RRDTool

Based on our design, environment, and

Other Visualization Techniques

Anomaly for DNS Queries

Displaying RMON—ntop

Matching data collected from different tools

Syslog

Benefits of Deploying NTP

Packet Capture Examples

Wealth of

Total Visibility

More Information about SNMP

SEC-1101/2102 give which SNMP

• IETF RMON WG—

http://www.ietf.org/html.charters/

tk80/tech_protocol_family_home.html

syslog_postgresql/

More Netflow Tools

• NFDUMP

variables DC_NETWORKS address 10.2.121.0-‐10.2.121.255,10.3.120.0-‐10.3.127. 255,10.4.8.0-‐10.4.15.255

•  IETF RMON WG—

•  NFDUMP