Networking for Science + Software-Defined Networking (SDN): Hype vs. Hope

(1)

Software-Defined Networking (SDN):

Hype vs. Hope

Inder Monga

Chief Technologist and Area Lead

HEAnet Conference 2013

Athlone, Ireland

(2)

Outline

Energy Sciences Network

Networking for Science

(3)

Outline

(4)

ESnet was formed 26 years ago - 3 years after

HEAnet

(5)

26 Years as a Mission Network

• >100 Nobel Prizes

Mission of Energy Sciences Network:

Accelerate

research and discovery

for DOE

Office of Science.

Mission of DOE Office of Science:

Deliver knowledge and tools

for transforming

our understanding of the universe.

$5B/year for the US National Lab

Complex, which includes:

• world's largest collection of scientific

user facilities (32)

• supercomputers, accelerators, xray

/ neutron sources, electron

microscopes, sequencers, fusion

facilities,

(6)

Embedded in a US National Laboratory

(Berkeley Lab)

11/14/2013 6

• one of 3 CS Divisions

at Berkeley Lab

• surrounded by

scientific

collaborations,

large-scale tools, Petabytes

of data, 4000

researchers / staff

• advantages of

proximity: cafeteria and

hallway conversations

(7)

Coupled with a Major Research University

11/14/2013 7

UC Berkeley is

just downhill.

• 36,000 students,

1500 faculty

• hundreds with joint

appointments at

Berkeley Lab

(8)

Our Advisory Board

Larry Smarr

Jagdeep Singh

David Foster

Vint Cerf

Cees De Laat

Kristin Raushenbach

(9)

Log s

c

a

le

(10)

(11)

SNLL PNNL SNLA BNL AMES LLNL JGI GFDL PU Physics SUNN 10 10 100 10 10 10 1 100 100 100 10 10 10 10 10 100 10 10 10 100 100 100 100 Geographical representation is only approximate 1

The 100G Energy Sciences Network (Fall 2013)

Int’l PPPL 100 100 Cl ev . 10

SUNN STAR AofA

100G testbed

SF Bay Area Chicago New York Amsterdam

AMST U.S. R&E peerings (many) 100 U.S. commercial peerings 100 100 ESnet routers site routers 100G 10-40G 1G

Metro area circuits Site provided circuits

10 10 100 Optical only 1 100 Int’l Int’l 100 100 Int’l 100 100 100 100 100 100 100

Capability to scale

to

13.2 Tbps

(12)

ESnet Research Testbeds

100G Testbed

• High-speed protocol research

• Available since Jan 2012

• Dedicated 100G wave from

Oakland to Chicago to NYC

• Connects to 100G across Atlantic

to Amsterdam (ANA-100G)

OpenFlow Testbed

• 10G Nationwide Footprint

Dark Fiber Testbed

• Continental-scale fiber footprint for

disruptive research

Selma Jackson Houston Dallas Tulsa Kansas City St. Louis

Peoria Nashville Louisville Indianapolis Chicago South Bend Cleveland Buffalo Albany Cambridge New York Pittsburgh Washington DC Denver Goodland Albuquerque El Paso Los Angeles Phoenix Echo Springs Salt Lake City Reno Sacramento Sunnyvale Eugene Seattle Boise Raleigh Philadelphia Atlanta Charlotte Chattanooga 317 miles 205 miles 130 miles 152 miles259 miles 264 miles 295 mile s 179 miles 137 miles 22 8 m iles 25 5 m iles 2 12 miles 150 miles 22 8 m iles 275 miles 212 miles 2 76 m ile s 116 miles 95 miles 5 60 m ile s 284 miles 618 miles 31 7 m iles 42 2 miles 551 miles 325 miles 500 m iles 249 miles 863 miles 257 m iles 278 m ile s 248 m iles 172 miles 306 m iles 275 miles 147 miles 246 m iles 198 miles 460 miles 336 miles 204 miles 532 miles 138 miles

LBNL Long Haul Dark Fiber Routes 12,924 miles

BayExpres Metro Fibers: 432 miles ChiExpress Metro Fibers: 167 miles NYExpress Metro Fibers: 6 miles

74 miles Chepachet Stamford 61 miles 119 mil es Silver City 119 miles Seminary

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

2/25/13 20

ESnet 10G OpenFlow Testbed

HOUS NERSC SUNN LBNL StarLight ANL BNL NYC

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

2/25/13 3

Test% Hosts%

NERSC Test%_Hosts%

StarLight MAN

LAN

(13)

Outline

Software-Defined Networking

(14)

Network engineered for the Elephants

(15)

Sensitive Elephants, Robust Mice

> 80x reduction in data

transfer rate at

DOE-relevant distances (ANL to

NERSC) and speeds

(10Gpbs).

How to build a lossless network service?

•

Infrastructure

: ample network capacity

•

Equipment

: deep packet buffers

•

ScienceDMZ

: optimized end-site architecture

•

perfSONAR

: automatic and continual verification of

network health

•

OSCARS

: ‘fast lanes’

• 60 Mbps out / 5 Gbps in • 88 ms RTT • 122 Mbps out / 7 Gbps in • 51 ms RTT • 1 Gbps out / 9.5 Gbps in • 11 ms RTT • 7.3 Gbps out / 9.8 Gbps in • 1 ms RTT

http://www.es.net/assets/pubs_presos/sc13sciDMZ-final.pdf

(16)

“Science DMZ” Design Pattern for Data

Transfer

11/14/2013 ₁₆

Dedicated

Systems for

Data Transfer

Network

Architecture

Performance

Testing &

Measurement

Data Transfer Node

•

high performance

•

tuned for data transfer

•

proper tools

Science DMZ

•

dedicated and clean

location for DTN

•

easy to deploy - no

need to redesign the

whole network

•

additional info:

http://fasterdata.es.net/

perfSONAR

•

enables fault isolation

•

verifies correct operation

•

widely deployed in ESnet

and other networks, as well

as sites and facilities

source: Eli Dart ESnet

(17)

Prototype Science DMZ

10GE 10GE 10GE 10GE 10G Border Router WAN Science DMZ Switch/Router Enterprise Border Router/Firewall Site / Campus LAN High performance Data Transfer Node with high-speed storage

Per-service security policy control points Clean, High-bandwidth WAN path Site / Campus access to Science DMZ resources perfSONAR perfSONAR 11/14/2013 17 source: Eli Dart ESnet

(18)

Prototype Science DMZ Data Path

10GE 10GE 10GE 10GE 10G Border Router WAN Science DMZ Switch/Router Enterprise Border Router/Firewall Site / Campus LAN High performance Data Transfer Node with high-speed storage

Per-service security policy control points Clean, High-bandwidth WAN path Site / Campus access to Science DMZ resources perfSONAR perfSONAR

High Latency WAN Path Low Latency LAN Path

11/14/2013 18

source: Eli Dart ESnet

(19)

Science DMZ is

critical.

Knowledgebase:

Science DMZ:

http://fasterdata.es.net/science-dmz/

Security:

http://www.internet2.edu/presentations/tip2013/20130115-dart-science-dmz.pdf

CC-NIE:

http://www.nsf.gov/pubs/2013/nsf13530/nsf13530.htm

11/14/2013 19

(20)

Outline

Energy Sciences Network

Software-Defined Networking

(21)

http://www.tomsitpro.com/articles/sdx-software-defined-kitchen-sink,1-1085.html

SDN is everywhere!

SDN

2013

(22)

What is SDN?

Control

Software

Network

ASICs

Firmware

Network Element

Network

Monitoring

Network

Provisioning

Protocols (SNMP, TL1) Provisioning Topology Statistics

Network Virtualization

[Science]

Applications

Network Apps

[NaaS]

Protocol(s) (OpenFlow, ?)

Loose definition:

separation of

data-plane from control plane

In essence:

enables

programmability

Network

ASICs

Firmware

control

Network

ASICs

Firmware

control

Network

ASICs

Firmware

control

Network

ASICs

Firmware

control

programmable

Network Controller(OS)

Network

ASICs

Firmware

control

Network Element

Control

Software

Network

ASICs

Firmware

Cloud/End-user Applications

(23)

What is the paradigm change?

Internet today:

-

Built-in control in each layer

- Multiple management domains

SDN Approach:

-

Network-wide cross-layer view

- OpenFlow enables programmatic

access to network flows

Layer 1

Layer 2

Layer 3

Control Control Control

M

anagem

ent

La

y

er

1

La

y

er

2

La

y

er

3

Control

(Network-wide view)

M

anagem

ent

OpenFlow OpenFlow

Layer 3 Control ?

10/16/13 Inder Monga 23

(24)

Simple programming constructs

OpenFlow 1.0 standard

Switch

Port

MAC

src

MAC

dst

Eth

type

VLAN

ID

IP

Src

IP

Dst

IP

Prot

L4

sport

L4

dport

Rule

Action

Stats

1. Forward packet to zero or more ports

2. Encapsulate and forward to controller

3. Send to normal processing pipeline

4. Modify Fields

5. Any extensions you add!

+ mask what fields to match

Packet + byte counters

24

VLAN

pcp

IP

ToS

Slide courtesy Srini Seetharaman

(25)

Controller

PC

OpenFlow usage

Classic model, Simple example

OpenFlow Switch

Alice

’

s App

Decision?

OpenFlow

Protocol

Alice’s Rule

11/14/2013 Inder Monga, WLCG GDB

25

Alice

(26)

Network community is still struggling to meet

application requirements captured in 1986!

Brute force approach (add more bandwidth)

is not going to meet those requirements

First workshop report for ESnet on intersite networking, 1986

Why SDN?

(27)

(28)

www.ci.anl.gov www.ci.uchicago.edu

28

Advanced Photon Source data rates

1

10

100

1000

10000

1

-ID

-1

1

-ID

-2

1

-ID

-3

1

-ID

-4

2

-BM

2

-ID

-B

2

-ID

-E

2

-ID

-D

3

-ID

-B

3

-ID

-C

7

8

-BM

8

-ID

-I

8

-ID

-E

9

11

-ID

-B

11

-ID

-C

11

-ID

-D

12

-BM

12

-ID

-B

12

-ID

-C/D

15

-ID

20

21

-ID

21

-ID

-D

21

-ID

-E

21

-ID

-F

21

-ID

-G

22

23

-ID

-D

23

-ID

-B

30

32

-ID

-1

32

-ID

-2

34

-ID

Data Rate (expected in the next 5-10 years) MB/s

Data Rate (current) MB/s

(29)

Hope #1: Programmability will lead to greater

predictability

Science applications increasingly need

to deal with high performance,

any-any bursts of data

SDN enables

• multi-layer control – packet and

optical layer

• Control over individual flows – ex.

Route science flows around packet

bottlenecks

• Routing non-TCP flows over WAN

Many NRENs have access to fiber,

optical and packet platforms.

Burst movement of data

using PhEDex

Analysis triggered data

movement (PD2P)

(30)

Journey towards programmability

Seamless multi-layer for handling elephant flows

Layer123 SDN World Congress, Bad Homburg, October 2013

OpenFlow &

REST/JSON

OpenFlow 1.0

WDM/ OTN/ Packet

OTS

Virtualization Host A Host B OTS Config Manager L0/L1 Topology Multi-Layer Path Engine Multi-Layer Provisioning Multi-Layer Topology App

Advanced Reservation System (OSCARS)

SDN Controller Floodlight Traffic Optimization Engine

Multi-Layer

SDN Control

Layer

Infinera DTN-X

(31)

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Abstractions are important

(Scott Shenker, October 2011)

“The ability to master complexity is not the same as the

ability to extract

simplicity

”

“Abstractions key to extracting

simplicity

”

“SDN is defined precisely by these three abstractions”

•

Distribution

: centralized vs. distributed

•

Forwarding

: programming the fabric

•

Specification

: virtualization

http://opennetsummit.org/archives/oct11/shenker-tue.pdf

(32)

What is the right abstraction for a

(dynamic) collaboration?

• Set of (dynamic) point to point circuits

• Restricted & static routing policy

• Lots of meetings

(33)

Hope #2: Virtualization will

simplify

how

applications program the network

Network slice

Modeled as a

Virtual WAN

Network Element

NB API

App 1

App ‘n’

simple complex

(34)

Lawrence Berkeley National Laboratory U.S. Department of Energy | Office of Science

Journey towards programmability

Real network is too complex to program for applications

SRS, Ciena, SuperComputing 2012, Salt Lake City

Insights

•

Virtualization is

the killer-app

for SDN

(Scott

Shenker)

•

‘complexity’

pushed to the

‘network

hypervisor’

•

Architectural

simplicity –

Flow

programming

only needed at

edges of the

network, core

can be legacy

(35)

Thought experiment:

Build an N-port virtual switch for a collaboration

LHC Tier 2 Analysis Centers Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups Universities/ physics groups The LHC Open Network Environment (LHCONE)

WAN Virtual Switch

CERN →T1 mile s kms France 350 565 Italy 570 920 UK 625 1000 Netherlands 625 1000 Germany 700 1185 Spain 850 1400 Nordic 1300 2100 USA – New York 3900 6300 USA - Chicago 4400 7100 Canada – BC 5200 8400 Taiwan 6100 9850

Source: Bill

Johnston

(36)

SDN is about system optimization

When the application and network work as a system, network resource

optimization is possible

Without knowledge of flows, networking can only do coarse

characterization

Fine discrimination of flows possible with SDN, meet application needs

(37)

Hope #3: SDN enables an opportunistic

way to leverage all bandwidth without extra

investment

exploits the fact

‘In general it’s much cheaper to transport

data than to store it’

,

(38)

Is SDN ready for operations?

The innovator’s dilemma, Clayton Christensen

(39)

Challenges = Opportunities?

Provisioning Topology Statistics

[Science]

Applications

Network Apps

[NaaS]

Protocol(s) (OpenFlow, ?)

Network

ASICs

Firmware

control

Network

ASICs

Firmware

control

Network

ASICs

Firmware

control

Network

ASICs

Firmware

control

Network

ASICs

Firmware

control

Cloud/End-user Applications

1) Communication plane

can be disrupted

2) Single point of

failure or attack?

3) Responsive to

rapid topology changes?

Flapping?

4) Complexity of management

from operations on virtual to

physical reality?

5) Who do you blame?

Who do you call?

Who debugs?

6) Hardware will

never be simple, manage

capability differences

7) How does this

interoperate with the

current IP network?

(40)

Journey towards programmability

How to bridge the ‘Internet’ with SDN networks?

Treehouse, BGP over SDN infrastructure, ONS 2013 and ongoing

Insights

•

SDN networks can now peer with existing

Internet

•

New techniques need to be developed to

scale controller-based networking

•

Baby steps

(41)

The Bigger Picture: Organizational challenge

to deal with SDN

Network

(control and data plane)

Layer 0-7

Management, Tools, Measurement

Layer 8-9

People

(network engineers, sysadmins, operators)

Layer 10

Network

(API + data plane)

Network Operating System (control)

+

New tools, service plane and management

People

(network engineers**, sysadmins, operators**)

+

(software engineers/devops)

(42)

SDN Take-Away

• SDN is a journey R&E networks have been on for a while, but

recently commercially formalized

• Innovator’s dilemma gaps between established and the new ways,

industry and researcher momentum will close those gaps

• Maturity will still take some time

• Focus on the problem being solved aka hope rather than the

vendor hype

(43)

(44)

Problem: Mice and Elephant flow separation

OSCARS, 2006-2013

SDN before it was called ‘SDN’

Insights

•

abstractions are

key to success,

regardless of the

protocol

•

can only learn by

doing (lots of

naysayers)

•

Primary use will

be different than

the original

(45)

Bringing it Together:

A potential SDN R&E architecture

ESnet

NERSC

BNL

ORNL

Data Plane

Control Plane

Service Plane

R&E Network

NRM

NSI

NS I OSCARS

OF

OSCARS

SDN Ctrl.

OneWan

Switch

RON

OF

(e2e resource

broker) (e2e resource

broker)

OF

Transport

SDN

SDN only at edges,

efficient transport in core

Customer

SDN Ctrl.

Customer

SDN Ctrl.

FLA Router FLA Router FLA Router Univ.

OF

(46)

Three Inflection Points for Data-Intensive

Science



Abundant capacity (88 λ x 100Gbps)

ESnet architecture

(Science DMZ) +

NSF grants.



Campus architectures newly optimized for data mobility

(optimizing network architectures end-to-end)

(47)

What is common between modern

(48)

http://www.es.net/assets/pubs_presos/sc13sciDMZ-final.pdf

http://fasterdata.es.net/science-dmz/

http://www.internet2.edu/presentations/tip2013/20130115-dart-science-dmz.pdf

http://www.nsf.gov/pubs/2013/nsf13530/nsf13530.htm

http://www.sdncentral.com/events/brocade-infinera-esnet-sdn-demo/