Development of Type-2 Hypervisor
for MIPS64 Based Systems
High Performance Computing and Networking Lab
Al-Khwarizmi Institute of Computer Science
Agenda
• Introduction
• Scope of Project
• Type-II Hypervisor for MIPS64 Based Systems
• Design Architecture and Execution Flow • Challenges
Virtualization
• Creating software representation of a device or resource
– E.g. server, storage device, network, OS …
• Broadly classified as
– Server Virtualization
Applications of Virtualization
• Cloud services • Security
– E.g. Cloud user isolation …
• Optimal hardware utilization
– E.g. Multiple services on a single machine
• Software development and testing
– E.g. Legacy code execution
• Dynamic load balancing in data centers
Hypervisor
• A piece of software/firmware
– Used to create Virtual Machines (VMs)
– Provides a virtual environment to run guest OS
Virtualization Solutions
• Virtualization software
– Type1
• Hyper-V, Xen, Vmware ESX
– Type 2
• Virtual PC/Server, Vmware Workstation, Virtual Box
• Available only for x86 • A Hypervisor or Virtual
Virtualization on MIPS
• Relatively less explored area
• Recent work on type-I Longsoon MIPS hypervisor
• No published work on type-II MIPS hypervisor • MIPS is typically used in embedded systems
– E.g. Switches, routers, smart TV, set-top boxes …
Important Features
• Full-Virtualization solution for MIPS • MIPS on MIPS Virtualization enabling
application
• Multiple MIPS64 guest OS on single Host machine running linux
– Multi-cores design
– Implementation of basic devices
• Developed on Cavium OCTEON family
Type-II Hypervisor
• Guest is a user level process of Host OS • Multiple VMs could be created
Multiple VMs Scenario
MIPS Hardware MIPS Linux
Hypervisor Hypervisor Hypervisor
Design
• Three major parts of hypervisor
– ISA emulation
– Memory management
– External Device implementation
• Combined implementation of all three results in a Virtual board
Virtual Board
Block Level ISA Emulation
• Basic blocks of guest instructions are extracted
– A stream of straight line code ending with a control shift instruction
• Whole block is translated to a set of safer instructions
– Translated block is executed on hardware
• For each valid block
1. Translate each
instruction of block into a set of unprivileged
instructions
2. Concatenate sets of
translated instructions to form the translated Block
Execution
• Translated block is executed directly on hardware
• Control may shift back to hypervisor due to following reasons:
– Address translation for load/Store instructions – TLB operations
– System call
– External interrupt – End of current block
Memory Management
• Physical memory • Virtual memory
• Virtual to physical memory mapping is handled by
kernel
• Simplest virtual memory view of a system
Virtual
memory map
Mapped kernel segment(kseg 2)
Unmapped uncached kernel segment (kseg1)
Unmapped cached kernel segment (kseg0)
MIPS
Memory Map
• Hypervisor provide 1-1 mapping of guest memory
– Using Mmap() system call
• GVA to HVA translation
– Identify memory segment – check for GPA in TLB
• If found, get GPA and then Get HVA from hash-map
• else, generate guest exception
Guest’s Memory Management
Guest virtual address (GVA)
Guest Physical Address (GPA)
Host Virtual Address (HVA)
External Devices
• Multi-threaded approach for devices • Implemented
– UART (Universal Asynchronous Receiver/Transmitter) – CIU (Central Interrupt Unit)
– Timer
UART
• Used for serial communication
– OS transmits and receives bytes
– Processor communicates with console & keyboard
• Implementation in hypervisor
– Receiver thread
• Continuously check arrival of new data from keyboard • Generate interrupt on receiving
• Handover the data
– Transmitter thread
Central Interrupt Unit (CIU)
• Cavium Specific Unit • Provides
communication
between devices and cores
• Routes the interrupt from devices
• Using summary and interrupt enable
CIU Implementation
• Continuously observes devices interrupt signals
– Reads corresponding bits of device registers
• Create summary registers
• If any device wants to interrupt a CPU core
On-Demand Timer
• Embedded in core thread • Register timer on guest’s
demand
• Reads time from host and presents to guest in
nanosecond resolution
Technical Challenges
• We used Cavium Octeon CN57XX as reference board
• CVM segment
• CVM specific instruction
– Customized instructions
• e.g. BBIT0, MADD, 64 bit multiplication instruction (VMULU)
• Partially documented
– Continuously pinging Cavium support staff
Technical Challenges
• Testing and verification
– Unexpected guest behavior e.g. halts, illegal instruction …
• Guest instruction treated as data on host level
– Soln. Flush Cache after block translation
• Block fetching from non-contiguous pages • Instruction emulation verification
Technical Challenges
• High Boot time
– Hurdle in development • Optimization efforts – Cache implementation – In-place execution – Structural changes • Inline functions • Macros
• Customized data structure