Understanding Latency in Software Defined Networks

(1)

Understanding Latency in

Software Defined Networks

Keqiang He, Sourav Das,

Aditya Akella

Junaid Khalid

(2)

Latency in SDN

Centralized Controller

Mobility

100 flows

Openflow msgs

2

Some XXX msecs??

Can be as large as 10 secs!!!

(3)

Do applications care about latency?

3

Intra-DC Traffic Engineering

•

MicroTE

routes predictable traffic on time scales of 1-2s

•

Route updates can take as long as 0.5s

Latency can undermine MicroTE’s effectiveness

Fast Failover

•

Reroute the affected flows quickly in face of failures

•

Longer update time increases congestion and drops

Latency can inflate failover time by nearly 20s!

Latency is critical to many applications

Limits the applicability of SDN applications

Applications assume latency is low and constant

(4)

Factors contributing to Latency?

4

Speed of Control Programs and Network Latency

Latency in network switches

Robust Control Software Design and Distributed Controllers

(5)

Our Work

Latency in network switches

Two contributions:

•

Systematic experiments to explore latencies in

production switches

•

Design a framework to overcome the impact of

(6)

Outline

•

Motivation

•

Elements of Latency

•

Measurement Methodology

•

Inbound Latency

•

Outbound Latency

•

Insertion

•

Modification

•

Deletion

6

(7)

Outline

•

Motivation

•

Elements of Latency

•

Inbound Latency

•

Outbound Latency

•

Insertion

•

Modification

•

Deletion

(8)

Inbound Latency

Elements of Latency

PHY PHY

Switch

Controller

I1: Send to ASIC SDK

I2:

Send to OF Agent

I3:

Send to Controller

Packet 8

CPU

Memory DMA ASIC SDK OF Agent I2 CPU board

Forwarding Engine _Lookup Hardware _Tables

No Match I1 P CI Switch Fabric I3

1. Inbound Latency

(9)

Outbound Latency

Elements of Latency

PHY PHY

Switch

Controller

O1:

Parse OF Msg

O2:

Software schedules the rule

O3:

Reordering of rules in table

O4:

Rule is updated in table

Memory

CPU

ASIC SDK OF Agent O2 DMA O1

Forwarding Engine _Lookup Hardware _Tables

P CI O4 O3 Switch Fabric CPU board

1. Inbound Latency

2. Outbound Latency

(10)

Outline

•

Motivation

•

Elements of Latency

•

Inbound Latency

•

Outbound Latency

•

Insertion

•

Modification

•

Deletion

10

(11)

Latency Measurements - Setup

eth0

eth1

eth2

Control Channel

Flows IN

Flows OUT

Openflow Switch

POX

Controller

Pktgen

Libpcap

Switch

CPU

RAM

Flow table

size

Data Plane

Vendor A

1 Ghz

1 GB

896

14*10Gbps

+ 4*40Gbps

Vendor B

2 Ghz

2 GB

4096

40*10Gbps

+4*40Gbps

(12)

Outline

•

Motivation

•

Elements of Latency

•

Inbound Latency

•

Outbound Latency

•

Insertion

•

Modification

•

Deletion

12

(13)

Inbound Latency

•

Increases with flow arrival rate

•

CPU Usage is higher for higher flow arrival rates

Flow Arrival Rate

(packets/sec)

Mean Delay per

packet_in (msec)

CPU Usage

( % )

0

-

7.1

100

3.32

15.7

200

8.33

26.5

vendor A

switch

(14)

Inbound Latency

vendor A

switch 200 flows/sec

•

Increases with interference from outbound msgs

14 0 50 100 150 200 250 300 0 200 400 600 800 1000 inb oun d d ela y (ms ) flow #

(a) with flow_mod/pkt_out

0 50 100 150 200 250 300 0 200 400 600 800 1000 inb o un d dela y ( ms ) flow # (b) w/o flow_mod/pkt_out

Low Power CPU

Software Inefficiency

(15)

Outline

•

Motivation

•

Elements of Latency

•

Inbound Latency

•

Outbound Latency

•

Insertion

•

Modification

•

Deletion

(16)

Outbound Latency

•

Latency for three different flow_mod operations

Insertion

Modification

Deletion

•

Impact of key factors on these latencies

Table occupancy

Rule priority structure

(17)

Outline

•

Motivation

•

Elements of Latency

•

Inbound Latency

•

Outbound Latency

•

Insertion

•

Modification

•

Deletion

(18)

Insertion Latency – Priority Effects

Vendor B

switch

18 0 5 10 15 20 25 30 0 20 40 60 80 100 insert io n d ela y (ms ) rule #

(a) Burst size 100, same priority

0 5 10 15 20 25 30 0 20 40 60 80 100 insert io n d ela y (ms ) rule #

(19)

Vendor B

switch

(b) Burst size 100, increasing priority

100

0x0000

(20)

Vendor B

switch

100

99

0x0000

(21)

Vendor B

switch

100

99

101

0x0000 TCAM

(22)

Vendor B

switch

100

99

101

0x0000 TCAM

(23)

Vendor A

switch

0 2 4 6 8 10 0 100 200 300 400 500 600 700 800 insert io n d ela y (ms ) rule #

(24)

Vendor A

switch

24 0 2 4 6 8 10 0 100 200 300 400 500 600 700 800 insert io n d ela y (ms ) rule #

(b) Burst size 800, decreasing priority

0x0000

(25)

Vendor A

switch

0x0000 TCAM

100

101

102

103

104

(26)

Insertion Latency – Priority Effects

Vendor A

switch

0x0000 TCAM

100

101

102

103

104

99

(27)

Vendor A

switch

0x0000 TCAM

100

101

102

103

104

99

(28)

Insertion Latency – Priority Effects

Vendor A

switch

0x0000 TCAM

100

101

102

103

104

99

(29)

0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000 0 100 200 300 400 500 600 av g comp le tion tim e (m s) burst size

(b) high priority rules into a table with low priority existing rules

table 100 table 400 0 2000 4000 6000 8000 10000 12000 14000 16000 18000 20000

0

100

200

300

400

500

600

av g comp le tion tim e (m s) burst size

(a) low priority rules into a table with high priority existing rules

table 100 table 400

Insertion Latency – Table occupancy

Effects

•

Affected by the types of rules in the table

Vendor B

switch

Rule Priority Structure and Table Occupancy

TCAM Organization and # Of Slices

Switch Software Overhead

(30)

Outline

•

Motivation

•

Elements of Latency

•

Inbound Latency

•

Outbound Latency

•

Insertion

•

Modification

•

Deletion

30

(31)

Modification Latency

•

Higher than Insertion latency for vendor B

•

Not affected by rule priority but affected by table

occupancy

Vendor B

switch

0 20 40 60 80 100 120 140 160 0 20 40 60 80 100 m odifi ca tion d ely (m s) rule #

(a) 100 rules in the table

0 20 40 60 80 100 120 140 160 0 50 100 150 200 m odifi ca tion d ela y (ms ) rule # (b) 200 rules in table

(32)

Outline

•

Motivation

•

Elements of Latency

•

Measurement Methodology

•

Inbound Latency

•

Outbound Latency

•

Insertion

•

Modification

•

Deletion

32

(33)

Deletion Latency

•

Higher than Insertion latency for both vendor A and B

•

Not affected by rule priority but affected by table occupancy

Vendor B

switch

0 20 40 60 80 100 120 0 50 100 150 200 dele tio n dela y ( ms ) rule # (b) 200 rules in table 0 20 40 60 80 100 120 0 20 40 60 80 100 d ele tion d ela y (m s) rule #

(34)

Deletion Latency

•

Higher than Insertion latency for both vendor A and B

•

Not affected by rule priority but affected by table occupancy

34

Vendor A

switch

0 2 4 6 8 10 12 14 0 20 40 60 80 100 d ele tion d ela y (m s) rule #

(a) 100 rules in table

0 2 4 6 8 10 12 14 0 50 100 150 200 d ele tion d ela y (m s) rule # (b) 200 rules in table

(35)

Summary

•

Latency in SDN is

critical

to many applications

•

Assumption:

Latency is small

or

constant

•

Latency is high and variable

•

Varies with Platforms, Type of operations, Rule

priorities, Table occupancy, Concurrent operations

•

Key Factors:

TCAM Organization, Switch CPU

and

(36)

Summary

•

Latency in SDN is

critical

to many applications

•

Assumption:

Latency is small

or

constant

•

Latency is high and variable

•

Varies with Platforms, Type of operations, Rule

priorities, Table occupancy, Concurrent operations

•

Key Factors:

TCAM Organization, Switch CPU

and

inefficient

Software Implementation

(37)

(38)

Impact of Concurrent CPU jobs

38

•

Impacts insertion delay. E.g.

Polling Statistics

•

Polling stats impacts more when table occupancy is higher

0 200 400 600 800 1000 1200 0 1 2 3 4 5

flow stats polling count

table occupancy 0 table occupancy 500

Low Power CPU delays concurrent jobs