Advanced Computer Networks. Network I/O Virtualization

(1)

Tuesday 13 May 2014

1

Advanced Computer Networks

263‐3501‐00

Network I/O Virtualization

Patrick Stuedi

Spring Semester 2014

© Oriana Riva, Department of Computer Science | ETH Zürich

(2)

Outline

 _{Last week:}

 Software Defined Networking

 OpenFlow

 _Today:

 Network I/O Virtualization

 Paravirtualization

 SR-IOV

(3)

Tuesday 13 May 2014

3

Data Transfer Non-Virtualized

1) Application: syscall, e.g., socket.write() 2) OS driver: issue PCI commands

●

Set up DMA operation

3) NIC:

●

transmit data

●

raise interrupt when done

privileged instructions

OS

NIC Application

TCP/IP Driver

syscall

(4)

Virtualization and Hypervisors

Hypervisor Hardware

Guest Operation

System Guest Application

VM3

Guest Operation

System Guest Application Guest

Operation System

Guest Application

VM2

VM1

(5)

Tuesday 13 May 2014

5

Virtualization and Hypervisors

Hypervisor Hardware

Guest Operation

System Guest Application

VM3

Guest Operation

System Guest Application Guest

Operation System

Guest Application

VM2 VM1

How does

network

access work?

(6)

Option 1: Full Device Emulation

 Guest OS unaware that it is being virtualized

 Hypervisor emulates device at the lowest level

 Privileged instructions from guest driver trap into hypervisor

 Advantage: no changes to the guest OS required

 Disadvantage:

 Inefficient

 Complex

Hypervisor

Hardware Guest OS Application

Traps

Device Emulation

(7)

Tuesday 13 May 2014

7

Option 1: Paravirtualization

 Guest OS aware that it is being virtualized

 Runs special paravirtual device drivers

 Hypervisor cooperates with guest OS through paravirtual interfaces

 Advantage:

 Better performance

 Simple

 Disadvantage:

 Requires changes to the guest OS

Hypervisor

Hardware Guest OS

Application

Interfaces Device Emulation

Paravirtual Driver

(8)

Paravirtualization with VirtIO

 VirtIO: I/O virtualization framework for Linux

 Framework for developing paravirtual drivers

 Split driver model: front-end and back-end driver

 APIs for front-end and back-end to communicate KVM Hypervisor

Hardware Guest OS

Back-end drivers Device Emulation

Front-end drivers

lguest Hypervisor

Back-end drivers Device Emulation

virtio

(9)

Tuesday 13 May 2014

9

Example: KVM Hypervisor

Hypervisor Hardware Guest

Operation System

Guest Application

VM1

User Application

KVM module

Single Linux Process guest mode

(guest user, guest kernel)

user mode QEMU

kernel mode

 Based on Intel VT-x

 Additional guest execution mode

 I/O at guest OS trap into KVM (VM Exit)

 KVM schedules QEMU process to emulate I/O operation

/dev/kvm

VM Enter &

VM Exit

user mode

 Starting new guest = starting QEMU process

 QEMU process interacts with KVM through ioctl on /dev/kvm to

 Allocated memory for guest

 Start guest

 ...

(10)

VirtIO and KVM

Hypervisor

Real NIC

Guest OS

KVM module QEMU

VirtIO-Net Driver (Front-end)

tx

tap

1) VirtIO-Net driver adds packet to shared VirtIO memory 2) VirtIO-Net driver causes trap into KVM

3) KVM schedules QEMU VirtIO Back-end

4) VirtIO back-end gets packet from shared VirtIO memory and emulates I/O (via system call) 5) KVM resumes guest

rx

VirtIO Back-end

VirtIO shared memory

1 2 3

4

(11)

Tuesday 13 May 2014

11

Vhost: Improved VirtIO Backend

Hypervisor

Real NIC

Guest OS

KVM module

QEMU

VirtIO-Net

Driver

tx rx

vhost net tap

 Vhost puts VirtIO emulation code into the kernel

 Instead of performing system calls from userspace (QEMU)

(12)

Inter-VM communication

Hypervisor NIC

Guest Operation

System Guest Application

VM3

Guest Operation

System Guest Application Guest

Operation System

Guest Application

VM2

VM1

(13)

Tuesday 13 May 2014

13

Inter-VM communication

Hypervisor Hardware

Guest Operation

System Guest Application

VM3

Guest Operation

System Guest Application Guest

Operation System

Guest Application

VM2 VM1

How does inter-VM communication

work?

(14)

Switch in Hypervisor

HV NIC

Guest Operation

System Guest Application Guest

Operation System

Guest Application Guest

Operation System

Guest Application

Bridge/switch

(15)

Tuesday 13 May 2014

15

Switched Vhost in KVM

Real NIC Guest OS

KVM

VirtIO-Net Driver

vhost net

tap

bridge

Guest OS

VirtIO-Net Driver

Guest OS

VirtIO-Net Driver

vhost net tap

Guest Application

Guest

Application Guest Application

 Advantage: low latency (1 software copy)

 Disadvantage: uses host

CPU cycles

(16)

Switch Externally...

Hypervisor NIC

Guest Operation

System Guest Application Guest

Operation System

Guest Application Guest

Operation System

Guest Application

...either in

 External switch:

 Simplifies configuration:

all switching

controlled/configured by the network

 Latency = 2xDMA + 2hops

 NIC

 Latency = 2xDMA

(17)

Tuesday 13 May 2014

17

Where are we?

 Option 1: Full emulation

 No changes to guest required

 Complex

 Inefficient

 Option 2: Paravirtualization

 Requires special guest drivers

 Enhanced performance

Option 3: Passthrough

No hypervisor involvement: best performance

Problems:

(18)

Where are we?

 Option 1: Full emulation

 No changes to guest required

 Complex

 Inefficient

 Option 2: Paravirtualization

 Requires special guest drivers

 Enhanced performance

Option 3: Passthrough

No hypervisor involvement: best performance Problems:

Not good enough!

Still requires hypervisor

involvement, e.g.,

interrupt relaying

(19)

Tuesday 13 May 2014

19

Where are we?

 Option 1: Full emulation

 No changes to guest required

 Complex

 Inefficient

 Option 2: Paravirtualization

 Requires special guest drivers

 Enhanced performance

 Option 3: Passthrough

 Directly assign NIC to VM

 No hypervisor involvement: best performance

Not good enough!

Still requires hypervisor

involvement, e.g.,

interrupt relaying

(20)

Paravirtual vs Passthrough in KVM

Hypervisor

Real NIC

Guest OS

KVM module

VirtIO-Net Driver

tx rx

vhost net tap

Guest OS Application Application

Physical Driver

Real NIC

NIC exclusively assigned to VM2

VM1 VM2

(21)

Tuesday 13 May 2014

21

Challenges with Passthrough / Direct Assignment

 VM tied to specific NIC hardware

 Makes VM migration more difficult

 VM driver issues DMA requests using VM addresses

 Incorrect: VM physical addresses are host virtual addresses (!)

 Security concern: addresses may belong to other VM

 Potential solution: let VM translate it's physical addresses to real DMA addresses

- Still safety problem: exposes driver details to hypervisor, bugs in driver could result in incorrect translations

Solution: Use an IOMMU to translate/validate DMA requests from the device

 Need a different NIC for each VM

Solution: SR-IOV, emulate multiple NICs at hardware level

(22)

Challenges with Passthrough / Direct Assignment

 VM tied to specific NIC hardware

 Makes VM migration more difficult

 VM driver issues DMA requests using VM addresses

 Incorrect: VM physical addresses are host virtual addresses (!)

 Security concern: addresses may belong to other VM

 Potential solution: let VM translate it's physical addresses to real DMA addresses

- Still safety problem: exposes driver details to hypervisor, bugs in driver could result in incorrect translations

 Solution: Use an IOMMU to translate/validate DMA requests from the device

 Need a different NIC for each VM

 Solution: SR-IOV, emulate multiple NICs at hardware level

(23)

Tuesday 13 May 2014

23

Memory Address Terminology

 Virtual Address

 Address in some virtual address space in a process running in the guest OS

 Physical Address:

 Hardware address as seen by the guest OS, i.e., physical address in the virtual machine

 Machine address:

 Real hardware address on the physical machine as seen by the

Hypervisor

(24)

IOMMU

PCIe function (e.g. NIC)

IOMMU Memory

controller

Main

memory

(25)

Tuesday 13 May 2014

25

IOMMU

PCIe function (e.g. NIC)

IOMMU Memory

controller

Main memory VMM programs IOMMU

with VM-physical to machine address

translations

(26)

IOMMU

PCIe function (e.g. NIC)

IOMMU Memory

controller

Main memory VMM programs IOMMU

with VM-physical to machine address

translations

Guest OS programs NIC with VM-

physical address of

DMA

(27)

Tuesday 13 May 2014

27

IOMMU

PCIe function (e.g. NIC)

IOMMU Memory

controller

Main memory

NIC issues a DMA

request to VM physical memory

VMM programs IOMMU with VM-physical to

machine address translations

Guest OS programs NIC with VM-

physical address of

DMA

(28)

IOMMU

PCIe function (e.g. NIC)

IOMMU Memory

controller

Main memory

NIC issues a DMA

request to VM physical memory

VMM programs IOMMU with VM-physical to

machine address translations

Guest OS programs NIC with VM-

physical address of DMA

IOMMU checks and

translates to machine

(real) address for

transfer

(29)

Tuesday 13 May 2014

29

IOMMU

PCIe function (e.g. NIC)

IOMMU Memory

controller

Main memory

NIC issues a DMA

request to VM physical memory

VMM programs IOMMU with VM-physical to

machine address translations

Guest OS programs NIC with VM-

physical address of DMA

IOMMU checks and translates to machine (real) address for transfer

Memory controller

accesses memory

(30)

SR-IOV

 Single-Root I/O Virtualization

 Key idea: dynamically create new “PCI devices”

 Physical Function (PF): original device, full functionality

 Virtual Function (VF): extra device, limited functionality

 VFs created/destroyed via PF registers

 For Networking:

 Partitions a network card's resources

 With direct assignment can implement passthrough

(31)

Tuesday 13 May 2014

31

SR-IOV in Action

Hypervisor

SR-IOV NIC Guest OS

Physical Driver

Guest OS

Physical Driver

Guest OS

Physical Driver

Guest Application

Virtual Ethernet Bridge/Switch Virtual

Function Virtual

Function Physical

Function PCI

IOMMU

Physical

Driver

(32)

Advanced Computer Networks. Network I/O Virtualization