Virtualization in a Carrier Grade Environment

Virtualization in a Carrier

Grade Environment

David Beal

2

Virtualization? Oh, Virtualization!

VMs

System Virtualization

System VMs

Process VMs

System Level (ISA) Process Level (ABI)

HW Virtualization

Classic OS VM

Native, Type I Hosted, Type II

(VMware WS, KVM)

(Same ISA) (Possibly different ISA)

HW Emulation Whole System (Simics, Bochs, QEMU) Multiprogrammed Systems Dynamic Translators Multitask OS OS Virtualization Virtual Servers Virtuozzo, Solaris Zones

(Same ISA) (Possibly different ISA)

OS Translator

WABI, WINE

(==OS) (#OS)

ISA & ABI Translator FX!32 ISA & OS Translator Transitive (==OS) (#OS) High Level Language Java

Full / Native Virtualization

Transparent Para-virtualization Paravirtualized (Xen, VLX) Dynamic Binary Translation (VMware ESX,) Hw Assisted (Xen, VLX)

5

Approach A: Hardware Virtualization/Emulation

•

Bridges guests to a device via an emulated, generic device driver, the

hypervisor, and the real-device driver implemented by the hypervisor.

•

Pros: VM portability

– Guest interfaces to generic, commonly supported devices

•

Cons: Reduces I/O Throughput and RT performance

– I/O takes two (or three) steps from guest to hardware via generic driver – Custom drivers must be ported to hypervisor

• Driver optimization must occur in hypervisor

• Guest OS still sees generic device

– Hypervisor must manage all platform I/O

• Hypervisor scheduling must be properly set up for foreseeable I/O loads

6

Approach B: Hardware Partitioning

•

Dedicate hardware to a single guest for direct access/control

•

Pros: Retains native I/O performance

– I/O takes one step from OS to hardware

• Reduces CPU workload

• No emulated devices

• No additional CPU load for I/O management

– No need to port device driver to hypervisor

•

Cons: Reduces VM portability

7

VLX Approach

“Partition where we can, Virtualize where we must”

•

Virtualize the smallest set of hardware that is required for basic OS

functionality

– Interrupt controller, UART, Clocks, Timers, MMU, etc.

•

Partition as much hardware as is possible

– Memory, CPU, Devices, PCI bus

•

Provide efficient OS to OS communications mechanisms

9

Virtualization Requirements

Data Center Network Equipment

GPOS support YES YES

RTOS support (with/without MMU) NO YES

N x RTOS on multicores NO Some

GPOS & RTOS on same core NO Some

Isolation / Performance Trade-off NO Some

Memory constraints NO YES

Key Performance Metrics I/O Throughput Timing, Latency,

Determinism, I/O Throughput Communication channels Virtual Network Depend on applications Device Drivers Virtual Network, Virtual Disks Many, physical & virtual

Native Device Driver NO YES

Dedicated Devices NO YES

10

Telecom Network Segmentation

DSLAM, PABX BTS/NodeB BSC/RNC Media Gateway, Signalling Gateway

Carrier Grade Linux

cPCI, µTCA Rack Mount Servers

AdvancedTCA Billing Network Mngt Provisionning Switches, ATM Router MSC, HLR/VLR, xGSN IP/ATM Router, MGC Application Servers Web Servers Access/Edge Core Backend Services Telco Data Services

VxWorks, OSE, C5 Enterprise Linux, Solaris

Operating System Application

VLX NI – Functional Features

VirtualLogix VLX Virtual Machine Monitor

Intel VT Enabled Platform

CPU Core N

Ethernet MAC

Hard

Disk Drive Device N Ethernet MAC CPU Core #1 CPU Core #2 CPU Core #3

Run mix and match

combinations of RTOS, Open OS and their applications

Guests share Devices

RTOS #1 RTOS #3

Guests use real device drivers

Strict isolation between guest OSs

One core can be dedicated to one OS

Scalable across many cores

Operating System #M

Runs unmodified OS

Linux (OS #2)

VLX NI – Technical Features

VirtualLogix VLX Virtual Machine Monitor

Intel VT Enabled Platform

CPU Core N

Ethernet MAC

Hard

Disk Drive Device N Ethernet MAC CPU Core #1 CPU Core #2 CPU Core #3

RTOS #1 RTOS #2 Operating

System #M Linux

(OS #3)

Inter-OS communication framework

Communication, data sharing and

coordination between guest OS

Zero-copy shared memory based
High level services

The VLX Virtual MMU supports multiple modes

• Provides strict “sandboxing” of guest OS

• Allow performance optimization with respect to number of system tasks, size, and period

Advanced VM schedulers

Real-time guarantees
CPU resource reservation
Allow flexible multi-OS designs

Lightweight VMM

Performance optimized

Partitions CPUs, memory and devices

to isolate one guest OS from another

Virtualizes core platform resources

(PIC, Timer, RTC, UART)

No Host-OS or Domain-0 OS required

Legacy Applications on New Hardware

• Port existing software to new (single/multi CPU, single/multi core) hardware

with no change to existing software

• Benefits:

– Extend lifetime of legacy application code – Reduction of deployed product hardware costs – Reduction of development costs and time-to-market

Legacy and Open OS Co-existence

• Benefits:

– No need of porting applications from one environment to the other

– Enable a migration path from legacy to open OS by having both running in parallel – Allow new application code to be developed to open OS environment

– Reduce development cost and time-to-market

Linux Legacy Virtualization Virtualization Applications Applications Hardware Infrastructure

Application Isolation

• Use virtualization for deployment of software components

Dedicated resources per “purpose built” container - No contention for resources

• Benefits:

– System stability

Multi “Purpose Built” Planes Consolidation

• Benefits:

– Reduction of hardware cost, space and power consumption (CapEx and OpEx) resulting from using a single hardware to support multi-tier

applications Carrier-Grade Platform Applications Carrier-Grade Base Platform Hardware Infrastructure

Core Core Core Core

Linux

Linux

Linux

Linux

Network Throughput Benchmarks

VLX: Hardware/Software Configuration

Ethernet Controller

Ethernet Controller

Core Core Core Core

VLX NI on Intel VT

Ethernet Controller

Ethernet Controller

Ethernet Controller Ethernet Controller Ethernet Controller Ethernet Controller

Linux SMP

Network Throughput Benchmarks

Core Core Core Core

Network Throughput Benchmarks

• For standard frames (< 1518 bytes)

– VLX outperforms Xen by a factor of up to 80x

– VLX provides the same performance than Linux SMP

Network Throughput Benchmarks

• For jumbo frames (> 1512 bytes)

– Xen does not support jumbo frames

– VLX provides the same performance than Linux SMP

SPEC CPU2000 Benchmarks

SPEC Integer (Base Rate - OSware)

core 1 (rate - 1 user) core 2 (rate - 1 user) core 3 (rate - 1 user)

core 4 (rate - 1 user) cumulative rates SMP (rate - 4 users)

12,3 12,8 12,5 12,8 50,4 _49,5 0 10 20 30 40 50 60

SPEC Integer (Base Rate - Xen)

core 1 (rate - 1 user) core 2 (rate - 1 user) core 3 (rate - 1 user)

core 4 (rate - 1 user) cumulative rates SMP (rate - 4 users)

SPEC CPU2000 Benchmarks

Four Linux UP instances (kernel + applications) running on VLX utilize 99% of CPU cycles. VLX requires 1% of the CPU cycles

0 5 000 10 000 15 000 20 000 25 000 30 000 35 000 40 000 45 000 50 000

core 1 core 2 core 3 core 4

D u ra tio n ( s )

0 10 20 30 40 50 60 70 80 1 2 4 8 16 32 64 128 256 1 2 4 8 16 32 64 128 256 512B 1KB M B p s

Native Linux VLX Linux

Iometer

Sequential Read

VLX Linux over Native Linux ratio: 95%

0 10 20 30 40 50 60 70 80 1 2 4 8 16 32 64 128 256 1 2 4 8 16 32 64 128 256 512B 1KB M B p s

Native Linux VLX Linux

Iometer

Sequential Write

VLX Linux over Native Linux ratio: 93%

VLX Features

Performance Focused

– Optimized for time-critical and I/O intensive guests – Very low platform overhead

– Native (optimized) device driver re-use

Network Infrastructure Hardware

– Supporting Intel multi-core rack-mounted and ATCA hardware with Intel Virtualization Technologies

Scalable Solution

– Multi-core scaling flexibility

VLX Features

Supporting OSes for Network Equipment Providers

– MontaVista Linux CGE

– VxWorks™ and other in-house and proprietary RTOS – Inter-mixed RTOS and GPOS

• 32 & 64 bit OSes

• UP or SMP

Customizable for Customer Applications

– Per-guest memory & device allocation – Guest OS hardware access or isolation – Guest scheduling

Standard VLX Install Configurations

Quick-Install Configurations

VLX includes scripted installations for multi-Linux™, multi-VxWorks™, or Linux + VxWorks configurations

– No OS modification required

– Pre-configured virtualized core devices

– Pre-configured shared ethernet, hard disk, and UART

Other application-specific configurations can be created by either

VirtualLogix or customer

– Memory partitioning

34

For More Information