Cloud Scale Load Balancing

(1)

Cloud Scale Load Balancing

• Ananta: Cloud scale load balancing, Parveen Patel

et.al; SIGCOMM 2013

• Duet: Cloud scale Load Balancing with hardware

and soJware, Rohan Gandhi, et.al; SIGCOMM

2014

Slides taken from SIGCOMM presentaPon

What’s happening?

• Cloud Services

• Azure (MicrosoJ), AWS (Amazon), Google Cloud

PlaWorm (Google)

• MulP-‐tenant

• Huge Compute Capacity

• Needs to be shared with high utlizaPon

• Spread requests over compute/storage resources

• Load balancer in the front end

MicrosoJ

Windows Azure -‐ Some Stats

• More than 50% of Fortune 500

companies using Azure

• Nearly 1000 customers signing

up every day

• Hundreds of thousands of

servers

• We are doubling compute and

storage capacity every 6-‐9

months

• Azure Storage is Massive – over

4 trillion objects stored

MicrosoJ

Scaling

• VerPcal scaling (scale up) vs Horizontal scaling

(scale out)

• VerPcal scaling

– 

More resource/capacity to a node/switch/server

– 

Increase power of resource, bigger

– 

Reliability

• Horizontal scaling

– 

More independent elements/nodes to a system

– 

Increase resources, more of the same

– 

Distributed state

(2)

Load balancing

• RotaPng DNS

• Conﬁgure DNS to return diﬀerent IP addresses

for the same name resoluPon

• www.yahoo.com

-‐-‐-‐ x.y.z.1, x.y.z.2 ….

• Advantages, Layer 3.. Scales

• Disadvantages: DNS does not know the load or

the health of the servers

• Cannot use heterogeneous load balancing policy

• Most load balancers are switch based . Layer4-‐

Layer7

MicrosoJ

Layer 4 LB/switching

• Switch is conﬁgured to have virtual IP address (VIP)

• This is the IP address that the DNS retrurns for the

name

• Switch conﬁgured based on policy to determine which

actual server the SYN is forwarded

• Can be based on desPnaPon port# (80 for hlp, 21 for

SMTP, etc)

• Acts as a desPnaPon-‐NAT

• Return packets come to switch and the Source address

changed back to VIP

MicrosoJ

Layer 4 LB

10.1.2.3 10.1.2.4 10.1.2.5 10.1.2.6 VIP=141.1.2.6

VIP VIP-‐P DIP DP

141.1.2.6 80 10.1.2.3 80

141.1.2.6 21 10.1.2.4 21

Layer 4 LB

• Port aware

• Switching decisions based on TCP SYN

• DesPnaPon NAT

• Return packets need to come back to the

switch/LB

• Need to act as Source NAT as well

(3)

Direct path to sender

• With a NAT box technique for rewriPng,

packets need to return to the switch

• Beler to avoid going back to the LB on the

return path

• Direct return to sender

• Need IP-‐in-‐IP encapsulaPon

Layer 4 vs layer 7 load balancing

• Layer 4 LB based on TCP header only

• Layer 7 LB examines the content and then decides the server.

• Based on hlp requests.

• Layer 7 switches need to terminate TCP connecPons

• LB needs to wait for hlp request before deciding on the binding

• External Client terminates TCP connecPon at switch

• AJer applicaPon level request, binding determined to applicaPon

server.

• The client sPll need to see the same sequence numbers coming

from the applicaPon server

• Need to imitate the same connecPon as the switch.

TCP Splicing

client

_{L7 switch}

server

step1 step2 SYN(CSEQ) SYN(DSEQ) ACK(CSEQ +1) HTTPREQ(CSEQ+1) ACK(DSEQ+1) step3 step7 step8 step4 step5 step6 SYN(CSEQ) SYN(SSEQ) ACK(CSEQ+1) HTTPREQ(CSEQ+1) ACK(SSEQ +1) _DATA(SSEQ+1) ACK(CSEQ+HTTPREQ+1) DATA(DSEQ+1) ACK(CSEQ+HTTPREQ+1) ACK(DSEQ + HTTPRES + 1) ACK(SSEQ+ HTTPRES+1) _.

Ananta in a nutshell

• Is NOT hardware load balancer code running on

commodity hardware

• Is distributed, scalable architecture for Layer-‐4

load balancing and NAT

• Has been in producPon in Bing and Azure for

three years serving mulPple Tbps of traﬃc

• Key beneﬁts

– 

Scale on demand, higher reliability, lower cost,

ﬂexibility to innovate

(4)

How are load balancing and NAT used in

Azure?

Background: Inbound VIP

communicaPon

Terminology:

VIP – Virtual IP

DIP – Direct IP

Front-‐ end VM LB Front-‐ end VM Front-‐ end VM Internet

DIP = 10.0.1.1 DIP = 10.0.1.2 DIP = 10.0.1.3

LB load balances and

NATs VIP traﬃc to DIPs

Client à VIP

Client à DIP VIP = 1.2.3.4

MicrosoJ

Background: Outbound (SNAT) VIP

communicaPon

Front-‐ end VM L B Back-‐ end VM DIP = 10.0.1.1 DIP = 10.0.1.20 Front-‐ end VM LB Front-‐ end VM Front-‐ end VM

DIP = 10.0.2.1 DIP = 10.0.2.2 DIP = 10.0.2.3

Service 1 _{Service 2}

1.2.3.4 à 5.6.7.8

DIP à 5.6.7.8 VIP1 à DIP

VIP1 = 1.2.3.4 _{VIP2 = 5.6.7.8}

MicrosoJ

VIP traﬃc in a data center

DIP Traffic 56% VIP Traffic 44% Total Traffic Intra-‐ DC 70% Inter-‐ DC 16% Intern et 14% VIP Traffic Inbou nd 50% Outbo und 50% VIP Traffic MicrosoJ

(5)

Why does our world need yet another

load balancer?

TradiPonal LB/NAT design does not meet

cloud requirements

MicrosoJ

Requirem

ent Details State-‐of-‐the-‐art

Scale

•  Throughput: ~40 Tbps using 400 servers

•  100Gbps for a single VIP

•  Conﬁgure 1000s of VIPs in seconds in the event of a disaster

•  20Gbps for $80,000 •  Up to 20Gbps per VIP •  One VIP/sec conﬁguraPon rate Reliability •  N+1 redundancy •  Quick failover •  1+1 redundancy or slow failover Any service anywhere

•  Servers and LB/NAT are placed across L2 boundaries for scalability and ﬂexibility

•  NAT and Direct Server Return (DSR) supported only in the same L2 Tenant

isolaPon

•  An overloaded or abusive tenant cannot aﬀect other tenants

•  Excessive SNAT from one tenant causes complete outage VM Switch VMN Host Agent VM1

. . .

VM Switch VMN Host Agent VM1

. . .

Controlle r Controlle r

Key idea: decompose and distribute

funcPonality

Ananta Manage

r

VIP ConﬁguraPon: VIP, ports, # DIPs

MulPple xer MulPplex er

. . .

MulPplexer VM Switch VMN Host Agent VM1

. . .

SoJware router (Needs to scale to Internet bandwidth)

Hosts

(Scales naturally with # of servers)

MicrosoJ

Ananta: data plane

2nd_Tier_{: Provides} connecVon-‐level (layer-‐4) load spreading, implemented in servers. 1st Tier: Provides packet-‐level (layer-‐3) load spreading, implemented in routers via ECMP.

3rd_Tier_{: Provides}

stateful NAT implemented in the virtual switch in every server. MulPple xer MulPplex er

. . .

MulPplexer MicrosoJ VM Switch VMN Host Agent VM1

. . .

(6)

Inbound connecPons

Rout

er

Rout

er

MU

X

Host

MU

X

Rout

er

MU

X

…

Host Agent

1

2

3

VM DIP

4

5

6

7

8

Dest:

VIP

Src:

Clie

nt

Packet Headers Dest:

VIP

Dest:

DIP

Src:

M

ux

Src:

Clie

nt

MicrosoJ Dest:

Client

Src:

VIP

Packet Headers Client

Outbound (SNAT) connecPons

Packet Headers Dest: Server:80 Src: VIP:1025 MicrosoJ VIP:1025 à DIP2 Server Dest: Server:80 Src: DIP2:5555

Fastpath: forward traﬃc

Host

MUX

1

VM

…

Host Agent 1 DIP1

MUX

2

2 Host VM

…

Host Agent DIP2 Data Packets DesPnaPon VIP1 VIP2 MicrosoJ SYN SOURCE=VIP1

Fastpath: return traﬃc

Host

MUX

1

VM

…

Host Agent 1 DIP1 4

MUX

2

2 3 Host VM

…

Host Agent DIP2 Data Packets DesPnaPon VIP1 VIP2 SYN SYN-‐ACK SOURCE=VIP2 DEST=VIP1

(7)

Fastpath: redirect packets

Host

MUX

1

VM

…

Host Agent DIP1 6 7

MUX

2

Host VM

…

Host Agent DIP2 7 Redirect Packets DesPnaPon VIP1 VIP2 MicrosoJ ACK Data Packets 5

Fastpath: low latency and high bandwidth for

intra-‐DC traﬃc

Host

MUX

1

VM

…

Host Agent DIP1

MUX

2

8 Host VM

…

Host Agent DIP2 Redirect Packets Data Packets DesPnaPon VIP1 VIP2 MicrosoJ

Impact of Fastpath on Mux and Host

CPU

10

55

13

2

0

10

20

30

40

50

60 Host

Mux

% CPU

No Fastpath

Fastpath

MicrosoJ

Lessons learnt

• Centralized controllers work

– There are signiﬁcant challenges in doing per-‐ﬂow processing, e.g., SNAT

– Provide overall higher reliability and easier to manage system

• Co-‐locaPon of control plane and data plane provides faster local

recovery

– Fate sharing eliminates the need for a separate, highly-‐available management channel

• Protocol semanPcs are violated on the Internet

– Bugs in external code forced us to change network MTU

• Owning our own soJware has been a key enabler for:

– Faster turn-‐around on bugs, DoS detecPon, ﬂexibility to design new features

Cloud Scale Load Balancing