Advanced Software Engineering. Lecture 8: Data Center by Prof. Harold Liu

(1)

Advanced Software Engineering

Lecture 8: Data Center

by

(2)

Agenda

Introduction

Design and Construction

Management and Maintenance

Hot Topics

_{Real‐time traffic analysis}

_SDN

Wireless Data Center Networking

(3)

Core Elements of a Data Center

Applications

Databases – Database Management

System (DBMS) and the physical and logical

storage of data

Servers/Operating Systems

Networks

Storage Arrays

3

(4)

An Example

Consider an order processing system consisting of: _{Application for order entry.} _{Database Management System (DBMS) to store customer and product} information. _{Server/Operating System on which the Application and Database programs} are run. _{Networks that provide} _{Connectivity between Clients and the Application/Database Server} Connectivity between the Server and the Storage system. _{Storage Array} Local Area Network Storage Area Network Storage Array Client Server Application User Interface Database

(5)

An Example ..Closer Look

A customer order is entered via the Application

User Interface on a client.

Local Area Network Storage Area Network Client Server Application User Interface Storage Array

(6)

An Example ..Closer Look

A customer order is entered via the Application

User Interface on a client

The client accesses the Server over a Local Area

Network.

Storage Area Network Client Server Storage Array Local Area Network

(7)

An Example ..Closer Look

A DBMS uses the operating system on the server

to read and write this data to the physical

location on a disk.

Storage Area Network Client Server O/S and DBMS Storage Array Local Area Network

(8)

An Example ..Closer Look

A DBMS uses the operating system on the server to read and write this data to the physical location on disk. A Network provides the communication link between the server and the storage array, and transports the read/write commands and data between the server and the storage array. Storage Area Network Client Server Storage Array Local Area Network

(9)

An Example ..Closer Look

A DBMS uses the operating system on the server to read and write this data to the physical location on disk. A Network provides the communication link between the client and the server, and transports the read/write commands and data between the server and the storage array. A storage array receives the read/write commands and data from the server and performs the necessary operations to store the data on the physical disks. Storage Area Network Client Server Storage Array Local Area Network Database

(10)

Google Data Center

(11)

at Lenoir, North Carolina

http://www.google.com/about/datacenters/inside/s

(12)

Gallery

(13)

(14)

(15)

(16)

(17)

Elements of a Data Center

The Site Command Center Cable Management Network Infrastructure Terminal Servers Environmental Controls Power

(18)

(19)

(20)

(21)

Structural Aspects

a raised floor ceiling

Basement data center locations near water are not a good idea. _{Their must be a}_pathway _{for equipment to be moved in & out of}

the data center.

Make sure the floor where the data center is to be located is

(22)

Power and Environment

Dual street power supply

with UPS

When necessary, a diesel

generator as a second

backup

_{Environmental Control}

Facilities

Temperature

Moisture

dust

(23)

HVAC

(24)

(25)

Networking

3‐layer architecture

Core Layer Switch/Router

Layer‐2 Switch

Internal NIC Ethernet

connection

DLink 48ports Gbps Layer-2 Switch

(26)

Security

Physical Access

Levels of Access

(27)

Past to Future

¾

a single computer occupied the space of an entire

Data Center.

¾

That same space can be occupied by thousands

of servers today.

(28)

Let us Make it Online

HW

SW

Rack

Deployment and Testing

Maintenance

(29)

Servers

Tower Low cost, customized Space cost, not for parallel computing Blade • save space, high computing ability • suitable for large‐size data center Rack Server • small, easy to manage • need good cooling

(30)

Rack Unit (RU)

_{Unit=44.45mm (height)} _{Width=19‐inch or 23‐inch} _{1U, 2U, 4U (half rack)} 1U 4U

(31)

(32)

Data Center Topology

Internet Servers Layer-2 switch Access Data Center Layer-2/3 switch Aggregation Layer-3 router Core Top of Rack Switch

(33)

Top of

Rack (ToR)

Switch

(34)

Data Center Costs

James Hamilton published basic 2008 breakdown Servers: 45% CPU, memory, disk Infrastructure: 25% UPS, cooling, power distribution Power draw: 15% Electrical utility costs Network: 15% Switches, links, transit

(35)

Power Cost

Approximate distribution of peak power usage by

hardware subsystem in one of Google’s datacenters (circa 2007).

(36)

Data Center Challenges

Traffic load balance

Support for VM migration

Achieving bisection bandwidth

Power savings / Cooling

Network management (provisioning)

Security (dealing with multiple tenants)

(37)

Non‐Virtualized Data Centers

Too many servers for too little work High costs and infrastructure needs _Maintenance Networking Floor space _Cooling _Power _{Disaster Recovery}

(38)

(39)

(40)

What is Virtualization?

_{Run multiple OSes and user applications on the same hardware} _{e.g., run both Windows and Linux on the same laptop}

_{How is it different from dual‐boot?} _{Both OSes run simultaneously}

(41)

(42)

(43)

(44)

Reduce costs by consolidating services

onto the fewest number of physical machines

(45)

Dynamic Data Center

Virtualization helps us break the “one service per server” model Consolidate many services into a fewer number of machines when workload is low, reducing costs Conversely, as demand for a particular service

increases, we can shift more virtual machines to run that service

We can build a data center with fewer total resources,

since resources are used as needed instead of being dedicated to single services

(46)

VM Workload Multiplexing

Multiplex VMs’ workload on same physical

server

Separate VM sizing VM multiplexing s₁ s₂ s₃ We expect s₃ < s₁ + s_2. Benefit of multiplexing !

(47)

(48)

(49)

Two Types of Hypervisors (or VMM)

Hypervisor is a software layer that allows several VMs to run

on a physical machine The physical OS and hardware are called the Host VM OS and applications are called the Guest VMware ESX, Microsoft Hyper‐V, Xen Hardware Hypervisor VM1 VM2

Type 1 (bare‐metal)

Host Guest Hardware OS Process Hypervisor VM1 VM2

Type 2 (hosted)

VMware Workstation, Microsoft Virtual PC, Sun VirtualBox, QEMU, KVM Host Guest Process Process Process Process

(50)

Bare‐metal or Hosted?

_Bare‐metal _{Has complete control over hardware} _{Doesn’t have to “fight” an OS} _Hosted _{Avoid code duplication: need not code a process scheduler,}

memory management system – the OS already does that

_{Can run native processes alongside VMs}

_{Familiar environment – how much CPU and memory does a VM} take? Use top! How big is the virtual disk? ls –l

_{Easy management – stop a VM? Sure, just kill it!}

_{A combination}

_{Mostly hosted, but some parts are inside the OS kernel for} performance reasons, e.g., KVM

(51)

51 51

VM on Multi‐core CPUs

Each core can be configured for multiple VMs A Quad‐core CPU could be configured as a 32 node multi‐computer Limiting factor is often memory. Each guest OS has its own requirements (512 MB?)

(52)

Installing a Virtual machine

Base OS is Windows 7

First install Sun VirtualBox as hypervisor

Then, Guest OS will be Ubuntu 12.04.1

(53)

Installing Sun/Oracle VirtualBox

(54)

Installing VirtualBox

(55)

Installing VirtualBox

(56)

Installing VirtualBox

(57)

Installing Ubuntu VM

(58)

Installing Ubuntu VM

(59)

Installing Ubuntu VM

(60)

Installing Ubuntu VM

(61)

(62)

Installing Ubuntu VM

(63)

(64)

Installing Ubuntu VM

(65)

Data Center Topology

Internet Servers Layer-2 switch Access Data Center Layer-2/3 switch Aggregation Layer-3 router Core Top of Rack Switch

(66)

(67)

(68)

Motivation

Flow‐based traffic monitoring

Volume of processed data is reduced

Popular flow statistics tools

Cisco NetFlow

sFlow

(69)

What is sFlow?

Accurate sampling is simple enough to be performed in hardware, at wire speed. Good accuracy under heavy loads Detailed complete packet header and switching/routing information for L2‐L7 traffic flows. Scalable capable of monitoring networks at 10Gbps, 100Gbps and beyond. Thousands of devices can be monitored by a single sFlow Collector. Low Cost sFlow Agent is very simple to implement and adds negligible cost to a switch or router. Timely an up to the minute view, for real‐time controls. QoS and DoS attack.

(70)

(71)

Switching ASIC

1 in N sampling

sFlow in Operation

packet header src/dst i/f sampling parms forwarding user ID URL i/f counters

sFlow agent forwarding tables interface counters sFlow Datagram eg 128B rate pool src 802.1p/Q dst 802.1p/Q next hop src/dst mask AS path communities localPref src/dst Radius TACACS sFlow Collector & Analyzer Switch/Router

(72)

Statistical Model for Packet Sampling

N

n

c

N

c

=

⋅

Total number of frames = N Total number of samples = n Number of samples in class = c Number of frames in the class estimated by:

Relative Sampling Error

0% 25% 50% 75% 100% 1 10 100 1000 10000

Number of Samples in Class

% E rro r c %error ≤196⋅ 1 Estimating Traffic per Protocol

(73)

Even so, Data Volume is Huge!

_{Short‐term period of flow data} Massive flow data from anomaly traffic data of Internet worm and DDoS Cluster file system and cloud computing platform _{Google’s programming model, MapReduce, big table} _{Open‐source system, Hadoop} Flow data in our campus network ( /16 prefix )

# of Routers 1 Day 1 Month 1 Year

1 1.2 GB 13 GB 156 GB

5 6 GB 65 GB 780 GB

10 12 GB 130 GB 1.5 TB 200 240 GB 2.6 TB 30 TB

(74)

An Experiment: Travel Booking Website

(75)

Traditional Flow‐based Traffic Monitoring

Run on a high performance central server

Flow Data Routers High Performance Server Storage 75

(76)

Occupancy of Different Network Service Types

(77)

(78)

(79)

Software Defined Networking (SDN)

What is SDN? Background An OS for networks What is OpenFlow? How it helps SDN The current status & the future of SDN Conclusions 79

(80)

Limitations of Current Networks

80

(81)

Million of lines of source code 5400 RFCs Barrier to entry 500M gates 10Gbytes RAM Bloated Power Hungry Many complex functions baked into the infrastructure OSPF, BGP, multicast, differentiated services, Traffic Engineering, NAT, firewalls, MPLS, redundant layers, … An industry with a “mainframe‐mentality”

We have lost our way

Specialized Packet Forwarding Hardware Operating System

App App App

Routing, management, mobility management, access control, VPNs, …

(82)

Operating System Operating System

Reality

App App App Specialized Packet Forwarding Hardware Specialized Packet Forwarding Hardware Specialized Packet Forwarding Hardware Operating System

App App App

• Lack of competition means glacial innovation • Closed architecture means blurry, closed interfaces • Vertically integrated, complex, closed, proprietary • Not suitable for experimental ideas • Not good for network owners & users • Not good for researchers

(83)

Glacial process of innovation made worse

by captive standards process

Deployment Idea Standardize Wait 10 years • Driven by vendors • Consumers largely locked out • Lowest common denominator features • Glacial innovation

(84)

Old Ways to Configure a Network

Specialized Packet Forwarding Hardware

App App App

Specialized Packet Forwarding Hardware Operating System Operating System Operating System Operating System Operating System App App App

(85)

No control plane abstraction for the whole network!

It’s like old times – when there was no OS…

Wilkes with the EDSAC, 1949

(86)

Idea: An OS for Networks!!!

App App App

(87)

Idea: An OS for Networks

App App App

Network Operating System

Control Programs

(88)

Idea: An OS for Networks

Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware Simple Packet Forwarding Hardware _{Simple Packet} Forwarding Hardware Network Operating System Control Programs 88

(89)

Idea: An OS for Networks

“NOX: Towards an Operating System for Networks” Global Network View Protocols Protocols Control via forwarding interface Network Operating System Control Programs

Software‐Defined Networking (SDN)

89

(90)

Windows (OS) Windows (OS) Linux Mac OS x86 (Computer) Windows (OS) App App Linux

Linux MacMac_OS OS Virtualization layer App Controller 1 App App Controller 2 Virtualization or “Slicing” App OpenFlow Controller 1 NOX (Network OS) Controller 2 Network OS

Trend

Computer Industry

Network Industry

(91)

(92)

Outline

What is SDN? Limitations of current networks The idea of Network OS What is OpenFlow? How it helps SDN The current status & the future of SDN Conclusions 92

(93)

OpenFlow

“OpenFlow: Enabling Innovation in Campus Networks” Like hardware drivers – interface between switches and Network OS 93

(94)

OpenFlow

94

Data Path (Hardware)

Control Path (Software)

(95)

OpenFlow

95

Data Path (Hardware)

Control Path

OpenFlow

_OpenFlow

OpenFlow

Controller

_Controller

(96)

OpenFlow Protocol

SSL‐TCP

Network OS

Control Program A Control Program B

OpenFlow Basics

IBM 10 gigabit ethernet OpenFlow switch G8264, which has 48×10 GbE SFP+ ports and 4 × 40 GbE QSFP+ ports

(97)

OpenFlow Switching

97 Controller PC Hardware Layer Software Layer OpenFlow Table MAC src MAC dst IP Src IP Dst TCP sport TCP dport Action OpenFlow Client * * 5.6.7.8 * * * port 1 port 4 port 3 port 2 port 1 1.2.3.4 5.6.7.8

(98)

Control Program A Control Program B

Network OS

Flow Table

Packet Forwarding Packet Forwarding Packet Forwarding Packet Forwarding Packet Forwarding Packet Forwarding Flow Table(s) Flow Table(s) “If header = p, send to port 4” “If header = ?, send to me” “If header = q, overwrite header with r, add header s, and send to ports 5,6”

(99)

Flow Table

Rule

(exact & wildcard) Action Statistics Rule

(exact & wildcard) Action Statistics

Rule

(exact & wildcard) Default Action Statistics

Flow 1.

Flow 2.

Flow 3.

(100)

Flow Entry

Match fields Match against packets _Action Modify the action set or pipeline processing Stats Update the matching packets Match

Fields Action Stats

In Port Src MAC

Dst MAC

Eth

Type Vlan Id IP Tos

IP

Proto IP Src IP Dst

TCP Src Port

TCP Dst Port

Layer 2 Layer 3 Layer 4

1. Forward packet to port(s) 2. Encapsulate and forward to controller 3. Drop packet 4. Send to normal processing pipeline 1. Forward packet to port(s) 2. Encapsulate and forward to controller 3. Drop packet 4. Send to normal processing pipeline 1. Packet 2. Byte counters 1. Packet 2. Byte counters

(101)

Examples

Switching * Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * 00:1f:.. * * * * * * * port6 Flow Switching port3 Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action 00:20.. 00:1f.. 0800 vlan1 1.2.3.4 5.6.7.8 4 17264 80 port6 Firewall * Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * * * * * * * * 22 drop

(102)

Examples

Routing * Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action * * * * * 5.6.7.8 * * * port6 VLAN Switching * Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport Action

* * vlan1 * * * * * port6, _port7,

port9 00:1f..

(103)

OpenFlow Usage

Controller PC OpenFlow Switch OpenFlow Switch OpenFlow Switch Alice’s code Alice’s code Decision? OpenFlow Protocol Alice’s Rule Alice’s Rule Alice’s Rule Alice’s

Rule Alice’s Alice’s _RuleRule

10 3

(104)

OpenFlow Usage

Controller PC Alice’s code Alice’s code 10 4 » Alice’s code: ˃ Simple learning switch ˃ Per Flow switching ˃ Network access control/firewall ˃ Static “VLANs” ˃ Her own new routing protocol: unicast, multicast, multipath ˃ Home network manager ˃ Packet processor (in controller) ˃ IPvAlice

(105)

OpenFlow

Standard way to control flow‐tables in commercial switches and routers Just need to update firmware Essential to the implementation of SDN 105

(106)

Centralized/Distributed Control

Centralized Control OpenFlow Switch OpenFlow Switch OpenFlow Switch Controller Distributed Control OpenFlow Switch OpenFlow Switch OpenFlow Switch Controller Controller _{“Onix: A Distributed Control Platform for Large‐scale Production} Networks”, UNIX OSDI 2010.

(107)

Virtualizing OpenFlow

Network operators “Delegate” control of subsets of network hardware and/or traffic to other network operators or users Multiple controllers can talk to the same set of switches

Imagine a hypervisor for network equipments

Allow experiments to be run on the network in

(108)

Switch Based Virtualization

Normal L2/L3 Processing Flow Table Production VLANs Research VLAN 1 Controller Research VLAN 2 Flow Table Controller 108

(109)

FlowVisor

A network hypervisor developed by Stanford

A software proxy between the forwarding and

(110)

FlowVisor‐based Virtualization

OpenFlow Switch OpenFlow Protocol OpenFlow Protocol OpenFlow FlowVisor & Policy Control Craig’s Controller Heidi’s Controller Aaron’s Controller OpenFlow Switch OpenFlow Switch 110 Topology discovery is per slice OpenFlow Protocol OpenFlow Protocol

(111)

Outline

What is SDN? Limitations of current networks The idea of Network OS What is OpenFlow? How it helps SDN The current status & the future of SDN Conclusions 11 1

(112)

OpenFlow Building Blocks

Controller NOX NOX Slicing Software FlowVisor FlowVisor FlowVisor Console 112 Applications LAVI LAVI ENVI (GUI)

ENVI (GUI) _n‐Casting_n‐Casting ExpedientExpedient

NetFPGA NetFPGA Software Ref. Switch Software Ref. Switch Broadcom Ref. Switch Broadcom Ref. Switch OpenWRT OpenWRT PCEngine WiFi AP PCEngine WiFi AP Commercial Switches Stanford Provided OpenFlow Switches ONIX ONIX Stanford Provided Monitoring/ debugging tools oflops oflops oftrace

oftrace _openseer_openseer

Open vSwitch Open vSwitch HP, NEC, Pronto, Juniper.. and many more HP, NEC, Pronto, Juniper.. and many more Beacon

(113)

Current status of SDN

Hardware support 113 Ciena Coredirector NEC IP8800 More coming soon... Juniper MX‐series HP Procurve 5400 Pronto 3240/3290 WiMax (NEC) PC Engines Netgear 7324

(114)

Current status of SDN

Industry support

Google built hardware and software based on the

OpenFlow protocol

VMware purchased Nicira for $1.26 billion in 2012

IBM, HP, NEC, Cisco and Juniper also are offering

SDNs that may incorporate OpenFlow, but also have other elements that are specific to that vendor and their gear.

(115)

Future Focuses of SDN

New policies for security Programmable WLANs The placement of controllers (amount; location; centralized/distributed) Debugger for SDN 11 5

(116)

Conclusions

_{What is SDN?} _{A system‐layered abstraction} _{Programmable, flexible, and extensible} _{What is OpenFlow?} _{Interface between switches and controllers} _{Enabling SDN} _Future SDN _{Enabling innovation} 11 6

(117)

Wireless Data Center Networking

60 GHz spectrum 7 GHz (57–64 GHz) waveband data rate over 1 Gb/s 10 meters Line‐of‐Sight (LoS)