A New Approach to Network Visibility at UBC. Presented by the Network Management Centre and Wireless Infrastructure Teams

(1)

A New Approach to

Network Visibility at UBC

(2)

• Business Drivers

• Technical Overview

• Network Packet Broker Tool

• Network Monitoring Tool

• Data Analytics and Visualization Tool

• Q&A

(3)

• Business Drivers

• Technical Overview

• Network Packet Broker Tool

• Network Monitoring Tool

• Data Analytics and Visualization Tool

• Q&A

(4)

(5)

• High availability

• High performance

• Virtualized

• Secure

(6)

Why is Visibility Needed?

NEED

VISIBILITY

Client Experience Data Centre Security Application Performance

(7)

• Life cycling needed

• Complex network with multiple paths

• Highly virtualized infrastructure

• Budget $$$$

(8)

• Business Drivers

• Technical Overview

• Network Packet Broker Tool

• Network Monitoring Tool

• Data Analytics and Visualization Tool

• Q&A

(9)

(10)

Visibility of Physical and Virtual Networks

• A virtual network spans multiple network devices

• Collect network information from multiple sources

(11)

Netflow

Many devices  Collector (Netflow Analyzer, StealthWatch)

SNMP

Many devices  Tool (Statseeker, Intermapper)

Logs

Many devices  Tool (Kibana-Elasticsearch)

Real network traffic

One or many  Tool (WireShark, Cisco NAM, WildPacket, IDS/IPS)

(12)

• Manage large number of tools and span sessions

• Separated tools and information make analysis difficult

(13)

• Manage much less number of tools and span sessions

• Single tool and information make analysis much easier

(14)

• Many-to-many port mapping for real-time adjustments of packet flow.

• Filtering of packet data based on the characteristics found in the packet

headers.

• Packet slicing and de-duplication that allows a subset of the full packet

data to be passed to the monitoring device.

• Aggregating multiple packet stream inputs into one larger stream, or

balancing one large stream into several smaller streams.

• Insertion of hardware-based time stamps that monitoring tools can use

to take more accurate measurements.

(Gartner Analyst Jonah Kowall, April 2012)

Why Network Packet Brokers?

(15)

• Business Drivers

• Technical Overview

• Network Packet Broker Tool

• Network Monitoring Tool

• Data Analytics and Visualization Tool

• Q&A

(16)

(17)

(18)

APCON: Main Panel

Network ports

(Ingress)

Tool ports

(Egress)

(19)

(20)

(21)

(22)

(23)

(24)

• Business Drivers

• Technical Overview

• Network Packet Broker Tool

• Network Monitoring Tool

• Data Analytics and Visualization Tool

• Q&A

(25)

• Challenges

• What is Statseeker?

• Advantages

• Use Cases

– Troubleshooting

– Proactive Alerting

– Baseline

(26)

• Limited visibility – What’s happening on this

part of the network?

• Troubleshooting – Is it a network issue?

• No baseline – What is normal?

(27)

Statseeker

- Commercial product

- Charts network statistics including bandwidth,

latency, utilization, errors, discards, CPU,

memory and temperature.

- Threshold and alerting

- Syslog

(28)

• Fast!

• Small footprint – 1 VM monitoring over 1000

switch stacks, and 100,000 ports

• Polls every 60 seconds

• Keeps data indefinitely with original

granularity

(29)

• Troubleshooting

• Proactive alerting

• Baseline

(30)

Troubleshooting with Network Statistics

– Does the time of the issue correlate with traffic

dips / spikes?

– Are other ports experiencing the same issue?

How about other switches?

– Track down source of traffic dip / spike

– Any errors or discards on the ports?

(31)

Unicast Storm

Example 1

(32)

High Utilization

Example 2

(33)

Compromised Server

Example 3

(34)

DOS Attack

Example 4

(35)

High Errors

Example 5

(36)

High CPU, interface down, syslog matches

Proactive Alerting

(37)

• Do we need to increase bandwidth on any

interfaces?

• Someone wants to upgrade their uplink from

1 Gbps to 10 Gbps. Do the traffic patterns

justify the upgrade?

• Able to see historical trends, and anticipate

growth requirements

(38)

Traffic Utilization over 30 days

(39)

Total traffic of multiple interfaces

Traffic Aggregation

(40)

(41)

• Business Drivers

• Technical Overview

• Network Packet Broker Tool

• Network Monitoring Tool

• Data Analytics and Visualization Tool

• Q&A

(42)

Data Analytics with ELK

(43)

By using free and open source software

• Elasticsearch – database optimized for search

• Logstash – Parse any data

• Kibana – HTML visualization frontend

ELK: How?

5 VM’s for central syslog, 2 REDIS queues, 2

logstash parsers, and 11 Elasticsearch

• All components are horizontally scalable

(44)

Logstash Forwarder (LSF) is a lightweight daemon that forwards logs from your application/server to Logstash

Logstash gets the log from LSF, or acts as a Central Syslog receiver (udp/514) from other network devices (switches, servers, etc). It sends those logs into a REDIS queue for processing

Logstash Parser pulls the logs from REDIS and parses/converts them into a format that can easily be searched by Elasticsearch

The Elasticsearch cluster contains

dedicated master nodes (esm1-3), client load balancer (esc1-2) and data nodes (es1-6). Each data node has 32GB Ram and 2TB disk.

The Kibana3 GUI and Kibana4 beta provide user access to the log data

(45)

• Input: File, syslog, udp (netflow)

• Filters: grok, mutate, GeoIP, replace, split, clone

• Output: Elasticsearch, REDIS, file

Many, many more @ https://github.com/logstash-plugins

ELK: Logstash

Common timestamp format

Easy to convert timestamps from various applications, devices,

and servers into one standard format

Data manipulation

All MAC Addresses have the same format. Any MAC’s that come

in as aabb.ccdd.eeff or aa:bb:cc:dd:ee:ff are converted to

(46)

(47)

(48)

(49)

• Business Drivers

• Technical Overview

• Network Packet Broker Tool

• Network Monitoring Tool

• Data Analytics and Visualization Tool

• Q&A

(50)

Any questions?

Q&A

(51)

Amy Osman

Network Analyst, Network Management Centre

[email protected]

Solomon Huang

Network Analyst, Network Management Centre

[email protected]

Jeremy Cohoe

Network Analyst, Wireless Infrastructure

[email protected]

Sean Wang

Network Architect, Network Management Centre

[email protected]

(52)

Slide 4: University Services

1. Erhardt, Don, The multi-purpose Franklin Lew Forum at Allard Hall., http://en.wikipedia.org/wiki/Allard_Hall

2. Baer, Rhoda, Researcher Looking Through Microscope,

http://commons.wikimedia.org/wiki/File:Researcher_looking_through_microscope.jpg

3. http://www.amsrentsline.com/vancouver-bc/apartment/shared-ubc-campus-gallery26922

4.

http://www.cisco.com/c/en/us/products/collateral/collaboration-endpoints/unified-ip-phone-7941g/product_data_sheet0900aecd802ff012.html

5. http://moementum.com/work-smart-blog/six-thoughts-on-curbing-the-disease-of-being-busy/

6. Grigoryan, Arthur, Videoconference classroom,

http://commons.wikimedia.org/wiki/File:Videoconference_classroom.jpg

7. https://it.ubc.ca/projects/new-university-data-centre-udc

8. TedxVancouver c/o Maurice Li, TEDxVancouver 2011, UBC Chan Centre,

http://commons.wikimedia.org/wiki/File:TEDxVancouver_2011,_UBC_Chan_Centre.jpg

9. Sistoiv, POS device (Italy), http://commons.wikimedia.org/wiki/File:POS_device.jpg

Slide 5: Business Requirements

1. Zammit, Jared, Blue fibre, http://www.jisc.ac.uk/network/connectivity

2. Samollov, Yuri, System Lock, https://www.flickr.com/photos/110751683@N02/13334048894/

(53)