Virtualization in Linux KVM + QEMU

(1)

CS695

Topi

cs

in

Vi

rtual

iz

ati

on

and

Cl

oud

Computi

ng

Virtua

liz

ation

in

Linux

KVM

+

QE

MU

Sen th il, Puru , Pr at eek and Shash ank 1 V irt u al iz at ion in Li n u x

(2)

2 V irt u al iz at ion in Li n u x

Topics

co

ver

ed

•KVM and Q

EMU

A

rchi

tectur

e

•VT

x

sup

port

•CP

U

vi

rtu

al

iz

ati

on

in

KMV

•Memory

vi

rtual

iz

ati

on

t

echni

ques

•shado

w pag

e t

abl

e

•EP

T/NP

T

pag

e t

abl

e

•IO vi

rtu

al

iz

ati

on

in

QEMU

•KVM and Q

EMU

us

ag

e

•Vi

rtual

di

sk

cr

ea

ti

on

•Cr

ea

ti

ng vi

rtu

al

machi

nes

•Cop

y-on

-wri

te

di

sk

s

(3)

KVM + QEMU - Architecture

3 Virtualization in Linux

(4)

KVM + QEMU – Architecture

• Need for hardware support

• less privileged rings ( rings > 0) are not sufficient to

run guest – sensitive unprivileged instructions

• Should go for

• Binary instrumentation/ patching

• paravirtualization

• VTx and AMD-V

• 4 different address spaces - host physical, host

virtual, guest physical and guest virtual

(5)

Gues

t 3

Gues

t 2

Gues

t 1

Gues

t 0

Hos

t

3 Ho

st

2 Ho

st

1 Ho

st

0

KVM QEMU Gu es t K erne l Gu es t User

X86

VT

x

supp

ort

/dev/ kvm VMCS + vm x ins tr uct ions KVM QEMU Gu es t 0 -3 KVM

Comm

un

ic

ati

on

Chan

nel

s

(6)

X86 VMX Instructions

• Controls transition between VMX root and VMX

non-root

• VMX root -> VMX non-root - VM Entry

• VMX non-root -> VMX root – VM Exit

• Example instructions

• VMXON – enables VMX Operation

• VMXOFF – disable VMX Operation

• VMLAUNCH – VM Entry

• VMRESUME – VM Entry

• VMREAD – read from VMCS

• VMWRITE – write to VMCS

(7)

X86

VM

CS Structur

e

•Con

tr

ol

s

CPU

beha

vi

or

in

VT

x

non

root

mode

•4KB

structur

e

–

con

fi

gu

red

b

y

KVM

•Al

so

pr

o

vi

des

space

for gues

t

and

hos

t

regi

st

er sa

ve &

res

tor

e

•Ex

ampl

e fi

el

ds

•HL

T

exi

ti

ng

–

if 1

VM

Exi

t on HL

T

•CR3

-l

oad

exi

ti

ng

–

if 1

VM

Exi

t on

CR3

load

•Ex

cep

ti

on

Bi

tmap

–

if bi

t

ii

s set,

VM

Exi

ts on

ex

cep

ti

on

i

•VM

-en

try

in

terru

p

t

–

To del

iv

er

in

terru

p

ts

dur

in

g

VM En

try

(8)

CPU

Virtualiz

ation

in KVM

VM 1 Th ead 0 VM 1 Th ead 1 VM 2 Th ead 0 VM 2 Th ead 1 VM 2 Th ead 3 VM 2 Th ead 2 P h ysi cal CP Us P roc es s sched ul ing QEMU Proc es s thr ea ds V CP Us Gu es t OS 1 Gu es t OS 2 8 V irt u al iz at ion in Li n u x

(9)

Sh

ado

w

pag

e

table

•Pr

obl

ems

in

memory vi

rtual

iz

ati

on

•3

le

vel

s of

in

di

recti

on,

MMU

can

tr

ansl

at

e 1

le

vel

•G

V

A

->

GP

A

->

HV

A

->

HP

A

mu

st

be

achi

ev

ed

•Sol

uti

on1

-Shadow pag

e t

abl

e

•Con

tai

n

s

G

V

A

->

HP

A

. MM

U

wi

ll u

se

thi

s

in

st

ead

of

gues

t pag

e t

abl

e

•One

shado

w

tabl

e f

or eac

h

gues

t

pag

e t

abl

e

•Incr

emen

tal

ly bui

ld

(10)

Sh

ado

w

pag

e

table

bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (QEM U)

•Gues

t

w

an

ts

to cr

ea

te

a l

inea

r

mappi

ng

for a pr

ocess

•Gues

t

does pur

e de

mand

•QEMU

kno

w

s G

PA

-> HV

A mappi

ng

(

mal

loc

())

1

2

3

4

GV A 10 V irt u al iz at ion in Li n u x

(11)

Sh

ado

w

pag

e

table

bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (Q EM U) 1 2 3 1 2 3 Sha do w page t abl e G V A -> HP A Gu es t pr oc es s page t abl e G V A -> G PA ( R ead onl y) St ep 1: •Gu es t tr ie s to m ap G V A 1 -> GP A 1 •P ag e fau lt (be cau se of R O) c au ses V M ex it •KV M see s GP A as 1 b y i ns tr u cti on emul at io n /u si n g regi st er cont ents 11 V irt u al iz at ion in Li n u x

(12)

Sh

ado

w

pag

e

table

bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (Q EM U) 1 2 3 1 2 3 Sha do w page t abl e G V A -> HP A Gu es t pr oc es s page t abl e G V A -> G PA ( R ead onl y) St ep 2: •GP A 1 -> HV A 1 is ob tai ned •Thi s pos si bl e be caus e G PA -> HV A map pi ng is kno wn to QEMU /KMV 12 V irt u al iz at ion in Li n u x

(13)

Sh

ado

w

pag

e

table

bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (Q EM U) 1 2 3 1 2 3 Sha do w page t abl e G V A -> HP A Gu es t pr oc es s page t abl e G V A -> G PA ( R ead onl y) St ep 3: •KVM doe s look up on QEMU’ s page t abl e to fi nd out HV A ->HP A •KVM fi nd s out H V A 1 -> H PA C 13 V irt u al iz at ion in Li n u x

(14)

Sh

ado

w

pag

e

table

bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (Q EM U) 1 C 2 3 1 1 2 3 Sha do w page t abl e G V A -> HP A Gu es t pr oc es s page t abl e G V A -> G PA ( R ead onl y) St ep 4: •KV M up da tes sha d o w pag e tabl e wi th G V A 1 -> H PA C •KV M al so u p d at es gu es t p ag e tab le – b y em u la ti n g th e in st ru cti on whi ch tri ed to map G V A 1 -> GP A 1 •G V A -> GP A -> H V A -> H PA is d on e 14 V irt u al iz at ion in Li n u x

(15)

Sh

ado

w

pag

e

table

bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (Q EM U) 1 C 2 B 3 E 1 1 2 2 3 3 Sha do w page t abl e G V A -> HP A Gu es t pr oc es s page t abl e G V A -> G PA ( R ead onl y) St ep 5: •Si mi larl y ot her en tr ies ar e upda te as and when pag e faul t happ en s •G V A 2 -> G PA 2 -> H V A 2 -> HP A B •G V A 3 -> G PA 3 -> H V A 3 -> HP A E 15 V irt u al iz at ion in Li n u x

(16)

Sh

ado

w

pag

e

table

(ad

ditional

in

fo)

•Addi

ti

o

nal

ques

ti

ons

•

Ho

w

to i

den

ti

fy page

s

used

in pa

ge

tabl

es

t

o wri

te pr

ot

ect

them

?

•Ho

w t

o

rem

o

ve

wr

it

e

pr

ot

ecti

on when

a p

ag

e

is

not use

d i

n

an

y

page

tabl

e

?

•Wha

t

happ

ens

w

hen

pur

e

dem

and

pagi

ng

is

not

used i

.e. (

gues

t bu

ilds

the

page

tabl

e be

for

e

loadi

ng

on

CR3

) ?

•Adv

an

tag

es

•No

gu

es

t

OS

ch

ange

is

r

equi

red

•An

y

OS

c

an

be

gues

t

•No

speci

al

har

dw

ar

e

is

r

equi

red

•Di

sadv

an

tag

es

•F

or e

ver

y pag

e

tabl

e used by

gu

es

t..

Sh

ado

w

ver

si

on

has t

o

be k

ep

t.

•Shad

o

w

pag

e

tabl

e mus

t

be

consi

st

en

t

w

ith gues

t

and hos

t

•Cach

ing

shado

w

page

tabl

e

needs consi

der

abl

e

me

mory

(17)

EP

T/NP

T Basics

•Sol

uti

on2

–

EP

T/NP

T har

dw

ar

e su

pport

•EP

T/NTP

enabl

ed

MMU

can tr

ansl

at

e

tw

o l

ev

el

s

of

in

di

recti

on

.

•Fi

rs

t

one

fr

om

G

V

A

->

GP

A

and

sec

ond

fr

om

GP

A

->

HP

A

•G

V

A

->

GP

A

is ma

in

tai

ned

b

y

gues

t

and

GP

A

->

HP

A

is

mai

n

tai

ned

b

y

KVM

•KVM

does

GP

A

->

HV

A

tr

ansl

ati

on

-bec

ause

mal

loc

()

•MMU

w

al

ks

EP

T

tabl

e f

or

ev

ery GP

A

(18)

EP

T/NP

T Bu

ilding

•EP

T s

ol

uti

o

n c

ons

is

ts o

f tw

o

tabl

e

s

•GP

A

-> H

PA

-EP

T t

abl

e

•G

V

A

->

GP

A

–

gues

t pr

ocess

pag

e t

abl

e

•MMU

ac

ce

sses

these

tw

o

tabl

e

s

to c

ompl

et

e

addr

ess tr

ansl

ati

on

•Gues

t

has ful

l ri

gh

ts o

n i

ts

pag

e t

abl

e

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (QEM U)

1

2

3

4

GV A 18 V irt u al iz at ion in Li n u x

(19)

EP

T/NP

T Bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (Q EM U) 1 2 3 1 1 2 3 EP T/NTP pag e t abl e GP A -> HP A Gu es t pr oc es s page t abl e G V A -> G PA St ep 1: •Gu es t tr ie s to acc es s line ar add res s 1 •Wi ll n ot c au se p ag e fau lt , be cau se V MCS is c onf ig u red n ot to cau se p ag e fau lt VM ex it s •Gu es t OS wi ll h an d le t h is an d fi ll G V A 1 -> GP A 1 19 V irt u al iz at ion in Li n u x

(20)

EP

T/NP

T Bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (Q EM U) 1 2 3 1 1 2 3 EP T/NTP pag e t abl e GP A -> HP A Gu es t pr oc es s page t abl e G V A -> G PA St ep 2: •W he n gue st access th e line ar m em or y add ress G V A 1, th e h ar d w ar e ge ts GP A as 1 us in g gu es t p ag e tab le •And t ri e s to fi gu re ou t cor res p ond ing H PA u si n g EP T t ab le an d c au se EP T v io la ti o n 20 V irt u al iz at ion in Li n u x

(21)

EP

T/NP

T Bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (Q EM U) 1 C 2 3 1 1 2 3 EP T/NTP pag e t abl e GP A -> HP A Gu es t pr oc es s page t abl e G V A -> G PA St ep 3: •EP T vi ol ati on oc curr ed be caus e corr es pond ing H PA is not map pe d. •KVM wi ll fi ll thi s en try us ing GP A 1 -> H V A 1 -> HP A C 21 V irt u al iz at ion in Li n u x

(22)

EP

T/NP

T Bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (Q EM U) 1 C 2 3 1 1 2 3 EP T/NTP pag e t abl e GP A -> HP A Gu es t pr oc es s page t abl e G V A -> G PA St ep 4: •W he n ree xc ut es the faul ted ins tr uct ion, MM U wi ll w al k tw o tabl e in ne st ed loop to fi gur e out G V A -> HP A •i .e . f or e ver y g ue st ph ysi cal addr es s en coun ter ed b y MM U , EP T w al k wi ll be done t o fi nd GP A -> H PA 22 V irt u al iz at ion in Li n u x

(23)

EP

T/NP

T Bu

ilding

A

B

C

D

E

F

G

H

I

J

K

L

M

N

O

P

Q

R

S

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

1

2

3

4

5

6

7

8

9

10

GP A HP A H V A (Q EM U) 1 C 2 3 1 1 2 3 EP T/NTP pag e t abl e GP A -> HP A Gu es t pr oc es s page t abl e G V A -> G PA St ep 5: •Si mi larl y ev er y EP T tabl e e n try is fi lled a ft er EP T vi ol ati on f or t he c orr es pond ing GP A •si nce EP T st or es GP A -> HP A , the si ze of EP T tabl e = gu es t RAM s iz e 23 V irt u al iz at ion in Li n u x

(24)

EP

T/NP

T (

add

iti

onal

in

fo)

•Adv

an

tag

es

•No

gues

t

OS chang

e i

s

requi

red

•An

y

OS c

an be gues

t

•Need no

t t

o

tr

ap

pag

e f

aul

t upd

at

es

•Si

ze

of EP

T t

abl

e

is pr

opo

rti

onal

t

o g

ues

t

mem

or

y

si

ze

•Di

sadv

an

tag

es

•TLB mi

ss

w

oul

d c

ause

cons

ider

abl

e

o

verh

ea

d

in tr

ansl

ati

on

–

Ex

. One l

ev

el

pag

e t

abl

e

w

oul

d c

ause

3 pag

e t

abl

e

mem

or

y

ac

ce

ss

•F

or

m l

ev

el

E

P

T a

nd n l

ev

el

gues

t

pag

e t

abl

e

, EP

T sol

uti

o

n

acc

ess

mn

+ m

+ n pag

e r

ef

er

ence

s

•Ha

rdw

ar

e

suppo

rt r

equi

red

(25)

EP

T/NP

T

Scen

ario

of TLB Miss

25 V irt u al iz at ion in Li n u x 1 F 4 E 7 4 B 5 6 A 1 2 C 3 4 5 6 1 2 2 3 1 4 4 5 7 CR3 = GP A 6 GP A 5 GP A 4 EP T= HP A D H PA E H PA F R e so lv e GV A 1 -> G PA 4 : G PA 6 -> HP A A r eq ui res acc es s to HP A D , HP A E = 2 R ea d GV A 1 = G PA 4 fr om HP A A = 1 G PA 4 -> HP A B req ui res acc es s to HP A D , HP A E = 2 R ead G V A 1 = G PA 2 fr om HP A B = 1 G PA 2 -> HP A C r eq ui res acc es s to HP A D , HP A F = 2 Tot al 8 ac ces s EP T T abl e GP A -> HP A Pag e Tabl e GV A -> G PA

(26)

QEM

U

–

IO

de

vice e

mula

tion

•Basi

c

IO d

evi

ces

ar

e emul

at

ed b

y

QEMU

•Ex

ampl

e

-K

eyb

oar

d,

Mouse,

Di

spl

ay

, har

d

dri

ve

and

NIC

•De

vi

ce

ac

cess

fr

om

gues

t

is tr

apped

(

both

PIO

and

MMIO) b

y

KVM

•KVM

passes

con

tr

ol

t

o Q

EMU

to handl

e

IO

•QEMU

in

jec

ts

in

terru

p

ts

fr

om

de

vi

ces

thr

ough

KVM

•T

o emul

at

e DMA

, QEMU

uses

thr

eads

to do

the

IO

(27)

QEM

U

–

IO

de

vice e

mula

tion

-Ex

ample

•As

sum

e di

sk

dri

ve ha

vi

ng f

ol

lo

wi

ng

in

terf

ace

•r

egi

st

er

x

to speci

fy s

ect

or

nu

mber

•r

egi

st

er

y t

o r

ecei

ve

comm

and

s

(1

r

ead, 0

wr

it

e)

•r

egi

st

er

z

to r

ead/

wr

it

e da

ta

•When g

ues

t w

an

ts

t

o

read sect

or

numb

er

10

1. Gues

t d

oes

PIO

10 on r

egi

st

er

x

2. Q

EMU

sa

ves

thi

s

in

form

ati

o

n i

n

de

vi

ce

st

at

e

3. Gues

t

is

sues

read

comm

and

u

si

ng

PIO

1 on

regi

st

er

y

4. QEMU map

s se

ct

or

10 on v

ir

tual

d

is

k

fi

le

and

r

eads

nec

ess

ar

y

con

ten

t

5. is

sues

an i

n

terr

upt

6. Gues

t

reads 51

2 b

yt

es

fr

om r

egi

st

er

z

usi

ng P

IO

7. QEMU g

iv

es the d

at

a i

t r

ead fr

om VD

(28)

QEM

U

–

IO

de

vice e

mula

tion

-Ex

ample

28 V irt u al iz at ion in Li n u x P IO 10 => X Sa ves X=10 in d isk s ta te P IO 1 => Y QEMU r ea ds se ct or 10 fr om VD fi le Vi rt ual In terrup t In terrup t han dl er s rea ds da ta P IO r ea d fr om Z Da ta r ead fr om fi le Gu es t KVM QEMU