VMware ESX Server
Advanced Technical Design Guide
Ron Oglesby
Scott Herold
All rights reserved. No part of this book shall be repro-duced, stored in a retrieval system, or transmitted by any means, electronic, mechanical, or otherwise, without written permission from the publisher. No patent liability is assumed with respect to the use of the information contained herein. Although every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omis-sions. Neither is any liability assumed for damages resulting from the use of the information contained here-in.
International Standard Book Number (ISBN:) 0971151067
Printed in the United States of America by United Graphics, Inc., Mattoon, IL.
Brian Madden Publishing offers discounts of this book when purchased in bulk. Visit www.brianmadden.com for details.
All terms mentioned in this book that are known to be trademarks or service marks have been appropriately capitalized. Brian Madden Publishing cannot attest to the accuracy of this information. Use of a term in this book should not be regarded as affecting the validity of any trademark or service mark.
First Printing, July 2005
Scott Herold Technical Editors Stephen Beaver Michael Burke Ken Cline Copy Editor Emily Monaco Publisher Brian Madden
Section 1. Introduction to ESX Server
1. VMware Overview 17
2. ESX Architectural Overview 29
3. VMware ESX Implementation 75
Section 2. Infrastructure Design
4. Storage and Network Strategies 107 5. Designing the Guest Environment 181 6. Managing the Server with the Console OS and web interface 265
Section 3. VMware in the Enterprise
7. Security 301
8. Server Monitoring and Management 327 9. Automated Installations and Provisioning 375 10. High Availability, Backups, and Disaster Recovery 409
This book was written with you, the reader, in mind. As writers, we’re always interested in hearing your thoughts about what worked, what didn’t, what was good, and what wasn’t. We’d love to hear any feed-back or comments you might have.
Any mistakes or clarifications found in this book will be posted to www.vmguru.com/books.
For anyone wondering, neither of us work for VMware or EMC. We both work for an independent consulting company in Chicago called RapidApp.
We mention several companies, products, and vendors throughout this book. None of them paid us to be listed. These are simply com-panies whose products we’ve used and liked.
Thanks for reading, and we look forward to hearing from you at www.vmguru.com.
The VMware Solution . . . .18
So what is “virtualization?” . . . .19
The Host Machine . . . .20
The Virtualization Software . . . .21
The Virtual Machine . . . .22
The Guest Operating System . . . .22
Which VMware product which should I use? . . . .23
A 60-second Overview of the ESX Server Architecture . . . .25
2. ESX Architectural Overview . . . .31
The Console Operating System versus VMkernel . . . .32
Console Operating System . . . .32
VMkernel . . . .33
The ESX Boot Process . . . .34
LILO . . . .34
Console Operating System . . . .34
init . . . .35 /etc/rc.d/rc3.d . . . .35 S00vmkstart . . . .35 S10network . . . .35 S12syslog . . . .36 S56xinetd . . . .36 S90vmware . . . .36 S91httpd.vmware . . . .37
So why do I need to know the boot process? . . . .37
Hardware Virtualization . . . .37 System Devices . . . .38 Processors . . . .38 Network . . . .39 VLANCE . . . .39 VMXNET . . . .39 SCSI . . . .40 Hardware Allocation . . . .40 Virtual . . . .41 Console . . . .41 Shared Resources . . . .42
Modifying These Configurations . . . .42
MUI . . . .42
Console Operating System . . . .43
The Core Four Resources . . . .45
Processor . . . .45
Memory . . . .49
NUMA . . . .51
Network . . . .52
Console NIC Configuration . . . .53
VMNIC Configuration . . . .53
Virtual Switch . . . .54
Virtual Network (VMnet) Configuration . . . .55
Storage . . . .56
Virtual Disk Files for VMs . . . .56
VMFS . . . .57
Local Storage . . . .58
SAN Storage . . . .60
Other Pluggable Devices . . . .61
SCSI . . . .61
Console Operating System SCSI Device Access . . . .61
Virtual Guest SCSI Device Access . . . .62
iSCSI . . . .63
PCI . . . .63
USB/Firewire . . . .64
Parallel/Serial . . . .64
Configuring Parallel Ports . . . .64
Configuring Serial Ports . . . .65
Resource Sharing . . . .66
Processor . . . .66
Processor Share Allocation . . . .67
Specifying Min/Max Percentages . . . .68
Combination of Min/Max and Share Allocation . . . .69
Affinity . . . .70
Memory . . . .71
Transparent Page Sharing . . . .71
Idle Memory Tax . . . .71
Ballooning . . . .72
Paging . . . .72
Network . . . .73
Disk . . . .74
Summary . . . .75
3. VMware ESX Implementation . . . .77
ESX Server Hardware . . . .78
ESX Server Memory Usage . . . .79
Operating Systems Being Hosted . . . .82
Memory Standards for VMs . . . .83
Console OS Memory Requirements . . . .84
So Much Memory Per Processor . . . .85
ESX Server Processor Usage . . . .85
Real-World Processor Recommendation . . . .86
ESX Server Hard Drive Usage . . . .87
Local VMware Partitions . . . .89
Hard Drives and Controllers . . . .90
ESX Server Network Connectivity . . . .91
Console NIC Configuration . . . .92
Virtual NIC (VMNIC) Configuration . . . .92
Real-World Network Connectivity . . . .93
Real-World Sizing Strategies . . . .94
Why Should You Care About Server Sizing? . . . .94
Server Sizing Options . . . .95
Option 1. Build a Few Gigantic Servers . . . .95
Option 2. Build Many Small Servers . . . .96
Option 3. Build Servers in the Middle Ground . . . .97
Installing ESX Server . . . .98
Installation Options . . . .98
Pre-installation Steps: . . . .99
Installation Steps . . . .100
Final Web Interface Configuration (post install) . . . .105
4. Storage and Network Strategies . . . .109
Local Storage versus Remote (SAN) Storage . . . .110
Local Storage . . . .110
Remote/SAN Storage . . . .111
SAN Implementation Options . . . .113
NAS Storage . . . .118
What about my iSCSI? . . . .119
Local Storage Configuration . . . .121
Configuring Controllers . . . .121
Assigning the controller . . . .122
SAN Storage Configuration for VMFS . . . .123
SAN/LUN Sizing . . . .123
Finding and Configuring Your LUNs . . . .129
HBA Configurations . . . .132
Configuring the Fiber Channel Queue Depth . . . .132
VMFS Accessibility Modes . . . .145
ESX and the Network . . . .146
eth0 . . . .147
Speed and Duplex . . . .147
Virtual NICs / vNICs . . . .155
MAC Addresses . . . .155
Physical NICs / pNICS / VMNICs . . . .158
Speed and Duplex . . . .158
VMNIC Mapping . . . .160
Monitoring VMNIC Performance . . . .161
Virtual Switches . . . .162
Public Virtual Switches . . . .163
Private Virtual Switches . . . .165
Load balancing modes . . . .166
Beaconing . . . .170
Physical Switches . . . .171
VLAN configurations . . . .172
External Switch Tagging (EST) . . . .172
Virtual Switch Tagging (VST) . . . .173
Virtual Machine Guest Tagging (VGT) . . . .176
VMotion Networking . . . .177
Blade Networking Options . . . .178
5. Designing the Guest Environment . . . .181
Creating a Strategy for VMware . . . .182
Using VMware for Test and Development Servers . . . .184
Using VMware for Underutilized Production Servers . . . .187
Using VMware for all of your production servers . . . .190
What Makes a Good Guest? . . . .192
So what should I look at on a potential Guest? . . . .194
“Core Four” Resource Utilization . . . .195
Other Hardware Requirements / Server Roles . . . .202
Summary . . . .205
Master Installations . . . .206
What is a master installation? . . . .206
Why Master Images are Needed . . . .207
Storage . . . .209
Checkpoints . . . .210
What to put in a Master Image . . . .211
Finalizing Your Image . . . .212
Creating ISO Files . . . .214
Storing ISO Files . . . .215
Copying ISO Images to ESX Server . . . .216
Using ISO Images . . . .217
Mounting ISO Images via the Console Operating System . . . . .218
Mounting ISO Images from within Virtual Machines . . . .218
Common Issues with ISO Files . . . .219
Verifying the Checksum on Linux . . . .220
Verifying the Checksum on Windows . . . .220
Verification . . . .221
Guest Machine Networking . . . .221
Physical Network Cards and Virtual Switches . . . .222
Virtual Switches . . . .223
Virtual Network Cards . . . .226
Controlling Guest Resource Usage . . . .231
The Basics of Min/Max Settings . . . .231
Resource Shares (CPU and Memory) . . . .233
Beyond Shares and Min/Max Settings . . . .236
Managing VMDK Files . . . .239 What is a VMDK File? . . . .239 Creating VMDK Files . . . .240 VMDK File Storage . . . .241 VMDK Access Modes . . . .243 Commit Changes . . . .244 Discard Changes . . . .245 Append Changes . . . .245 Level . . . .247 Freeze . . . .248 Wait . . . .248
Adding the REDO File . . . .249
Adding a Second REDO File . . . .249
Applying the First REDO File without a Freeze . . . .249
Applying the Second REDO File with a Freeze . . . .249
Resizing VMDKs . . . .250
Opening VMDK Files Outside a VM . . . .251
Creating a Guest Virtual Machine Standard . . . .253
Creating your Guest Virtual Machines . . . .255
Manual Creation of the VMs using the Web Interface . . . .256
Scripted/Command Line Creation of Virtual Machines . . . .259
Management Concepts and Tools . . . .266
Using the Web Interface . . . .267
Using the ESX Service Console . . . .268
Using a Management Utility . . . .269
ESX Server Management Tasks . . . .271
Installing New Hardware . . . .271
Installing and configuring hardware for use by the VMkernel 272 Configuring new VMkernel hardware via the Web Interface . .272 Configuring new VMkernel hardware via the Service Console 272 Why is my hardware not detected after installation? . . . .273
Checking Disk Space . . . .274
Configuring Time Synchronization . . . .275
Letting the VMs Sync to an External Time Source . . . .276
Sync the ESX server to your Authoritative Time Source . . . .277
NTP configuration for ESX 2.1 and Greater . . . .277
Changing IP Settings for ESX . . . .280
Running Support Scripts . . . .282
Adding Users to the ESX Console . . . .283
Changing a user’s password . . . .283
Using vmkfstools . . . .284
Scanning a Fibre Adapter for new LUNs . . . .285
Create a new file system on a new LUN . . . .286
Name the newly Created File System . . . .286
Lists the files on VMFS . . . .286
Set the VMFS mode (Shared/Public) . . . .287
Recover/Unlock a VMFS file System . . . .287
Managing the Guest . . . .287
Adding Disks or Disk Space to VMs . . . .288
Creating a Second VMDK . . . .288
Expand the Existing VMDK . . . .290
Managing Permissions on a Guest . . . .291
Hard Coding a VM MAC address . . . .295
Setting the MAC address in the vmx file: . . . .296
Shutting Down a VM (vmware-cmd) . . . .296
Shutting Down or Restarting a Virtual Machine from the Command Line . . . .297
VM Power Mode Scripts . . . .298
ESX Security Architecture . . . .302
VMware MUI . . . .303
Console operating system . . . .303
VirtualCenter . . . .304
Default ESX Security . . . .304
Password Encryption (Shadow) . . . .305
Firewall . . . .306
Configuring User Access . . . .307
Role-Based Access . . . .307
Alternative to VMware’s method . . . .309
vmadmins group . . . .309
Setting up account policies . . . .312
VirtualCenter . . . .314
Read-only . . . .314
Virtual machine user . . . .314
Virtual machine administrator . . . .315
VirtualCenter administrator . . . .315
Restricting root access to ESX . . . .315
Disabling root over ssh . . . .316
Using the wheel group . . . .317
Configuring sudo . . . .318
Planning your sudo configuration . . . .318
Modifying /etc/sudoers . . . .319
Enabling External Authentication . . . .323
Uses . . . .323
Considerations . . . .324
Patching . . . .324
ESX . . . .325
Guest Operating Systems . . . .326
8. Server Monitoring and Management . . . .327
Rules Of Thumb . . . .328
VMs per Processor rule of thumb . . . .328
Quantity and Speed of Processors . . . .328
VM Configurations . . . .329
VM Workloads . . . .330
Local Storage and HBA Rules of Thumb . . . .331
Network Connectivity . . . .332
ESX Server Monitoring . . . .333
Swing Servers . . . .333
Monitoring Using Native ESX Tools . . . .334
Monitoring Using Third Party Tools . . . .347
ESX Host Monitoring . . . .347
VM Monitoring . . . .348
VMware’s VirtualCenter . . . .349
VirtualCenter Components . . . .350
VirtualCenter Licensing . . . .351
VirtualCenter Configuration Options . . . .352
Structuring your VirtualCenter Containers . . . .354
Connecting to an ESX Server . . . .358
Reviewing ESX Server Performance Data . . . .359
Reviewing VM Performance . . . .360
Host and VM Alarms . . . .361
Using VM Templates with VirtualCenter . . . .364
Scheduling Tasks . . . .372
9. Automated Installations and Provisioning . . . .375
ESX Installations . . . .376
VMware’s Scripted Installation . . . .376
Preparation . . . .377
Before you Begin . . . .378
Configuration Wizard . . . .379
PXE Deployment Tools . . . .386
Customizing your Boot Floppy . . . .387
General ks.cfg Options . . . .389
ESX-Specific Options . . . .397
%packages . . . .401
%pre . . . .401
%post . . . .401
Third Party Software Installations . . . .404
Guest Installations . . . .405
VirtualCenter Installation . . . .405
Configure the Sysprep Tools and Open Source Components . .406 Clone a VM . . . .406
Deploy from a Template . . . .406
DHCP Requirement . . . .407
PXE Installations . . . .407
10. High Availability, Backups, and Disaster Recovery . . . .411
“DR” versus “Backup and Recovery” . . . .412
Which backup software should you use? . . . .417
Backup Schedule . . . .417
Options for Backing Up the Service Console . . . .417
Virtual Machine Backup and Recovery . . . .420
Treating a VM like a Physical Server . . . .421
Treating a VM like a set of files . . . .422
Backing up using the VMware API . . . .424
High Availability within the Datacenter . . . .428
Virtual to Virtual Clusters . . . .429
Which shared storage should I use? . . . .431
Physical to Virtual Clusters . . . .433
Clustering Without Shared Storage . . . .435
How About Clustering the ESX Server? . . . .436
Disaster Recovery Considerations and Options . . . .437
VMware Disaster Recovery “Dream” Environment . . . .437
ESX Server Recovery/Deployment . . . .440
Recovering the VMs . . . .441
Recovering Using Restored VMDK Files . . . .442
Recovering Using “Normal” Backups . . . .442
Recovery Using a Combination of Methods . . . .443
Appendix & Index . . . .447
This book wouldn’t have happened without all the great guys out in the VMware Community groups. I would like to thank all those that post day after day and contribute to the overall knowledge of the community. I would like to extend a very special thanks to our tech-nical reviewers that gave of their time to correct our mistakes. As always, any mistakes or omissions are ours and not theirs. (Hopefully the 3.0 update will contain some of the additions you asked for.) I would like to thank the guys at RapidApp that had to listen—yet again—to all the talk about another book being written, and especial-ly to Scott that showed so much enthusiasm and listened as I tried to teach him about writing. He did a superb job for his first run at it. Finally, I would like to say thanks and “I’m sorry,” to my wife Dina and daughter Madison. They are the ones that really lose when I begin writing. Sorry, yet again, for the nights and weekends on the laptop. And thanks for not griping about it.
First and foremost, I have to give the most thanks to Ron for offer-ing me this opportunity and beoffer-ing my mentor throughout the entire process. Without him I would not have gotten as far as I did and you would have likely been reading small parts of this on a half kept-up website. I would also like to thank those who post in and run the VMware Community Forums. There are far too many names to men-tion, but everyone of you has helped keep the technology strong enough to warrant this book. My co-workers at RapidApp have done an unbelievable job supporting (and advertising) the book at various client sites and training classes. Without their enthusiasm for the suc-cess of the technology, we would have a lot less to write for. I would like to give a special "Thank You" to our technical reviewers who went above and beyond in making sure we provided accurate content. All of them volunteered their time for the book and their hard work is greatly appreciated. Finally, I would like to thank my mom for always being there and my loving fiancee (and toughest editor) Leah. Their support throughout the entire process has been unbelievable. Sadly, Leah can now say she knows more about VMware ESX than any other elementary teacher in the world.
It sure didn’t take long for server virtualization with VMware’s prod-ucts to catch on in a big way. Today, nearly every IT organization we work with is looking to use VMware in some capacity. At RapidApp, we have invested heavily in developing our VMware ESX technical skills and have had the opportunity to put those skills to work assist-ing several Fortune organizations to realize the benefits promised by the technology.
The success stories have been piling up. In the past 18 months we have seen many of our clients make the move from using VMware’s GSX product for their application development and testing environ-ments into their first foray using VMware ESX in their production server environment. Most of the same clients have plans for wide-spread deployment over the next 24 months, and we anticipate that within three years the use of “VMs” will become ubiquitous and a strategic component of an organizations IT infrastructure.
It’s likely if you are reading this now you recognize the benefits of the technology. The cost avoidance and TCO benefits are compelling enough on their own to make VMware a sure thing. But don’t forget the harder to quantify and sexier benefits associated with recoverabil-ity, rapid and dynamic provisioning, and that server virtualization is on the path to the utility computing model the experts are telling us is right around the corner.
This book will help you—the designer, builder, and manager of your organization’s IT infrastructur—prepare to provide the benefits of server virtualization. You won’t get far in the book before you begin to identify opportunities to use VMware to help tackle some of your key initiatives. Ron and Scott have done an excellent job compress-ing their extensive research and real world experience into this text. Use it as a field guide to help you ensure that as you begin to tacti-cally deploy VMware, you are building the right foundation for your server virtualization strategy.
Chapter
1
This book is designed to provide you with practical, real-world infor-mation about the use and design of VMware ESX Server systems. In order to study the advanced technical details, we need to ensure that you have a good baseline understanding of VMware ESX features, and VMware’s other products.
Even if you’ve already read and understand the VMware ESX admin guide or if you’re very familiar with previous versions of ESX or GSX, we still recommend that you at least skim through this chapter so that you can familiarize yourself with the specific terms and phras-ing that we’ll use throughout this book.
As you read through this book, you should keep in mind that each chapter was written separately and that it’s okay for you to read them out-of-order. Of course if you’re new to VMware and virtualization technologies, the chapter sequence will guide you through a logical progression of the components. We’ll begin in this chapter with an overview of VMware software solutions.
The VMware Solution
In case you don’t know why you’re reading this book, VMware ESX allows for the “virtualization” of Intel-based servers. This idea will be explored more in this chapter, but the basic concept is that a single piece of physical hardware will be used to host multiple logical or “virtual” servers. The ability to host multiple virtual servers on a sin-gle piece of hardware, while simple to understand in theory, can be extremely complex in execution.
When VMware technology is used, a single host allows multiple vir-tual servers to share the host’s physical resources. These resources include processor, memory, network cards, and of course local disk. (We’ll refer to these four physical resources as the “core four” throughout this book.) This architecture also gives you the ability to “partition” your host to allow for the server’s resources to be fully uti-lized while you migrate multiple physical servers to a single VMware
server running multiple virtual servers. (We’ll discuss partitioning and migration strategies in much more depth throughout this book.) So what does this mean to your environment or your business? It depends on what your business looks like and which product from VMware you’re thinking about using. Obviously if you’re reading this book then you’re at least contemplating an ESX Server implementa-tion. But even with ESX you have a number of alternatives on how to deploy and utilize the system. Also, when investigating VMware technology it is unfair to simply look at ESX and why it is used. You can only decide to use ESX instead of GSX or Workstation after you really understand what each product does and where they truly fit.
So what is “virtualization?”
Simply stated, VMware (the company) provides virtualization tech-nology. All of their products (Workstation, GSX Server, and ESX Server) work in more-or-less the same way. There is a host computer that runs a layer of software that provides or creates virtual machines that Intel operating systems can be installed to. These virtual machines are really just that—complete virtual machines. Within a virtual machine you have hardware like processors, network cards, disks, COM ports, memory, etc. This hardware is presented through a BIOS and is configurable just like physical hardware.
If we were to jump ahead a little and look at a virtual Windows serv-er, you would be able to go into device manager and view the net-work cards, disks, memory, etc., just like a physical machine. As a mat-ter of fact the whole idea is that the virtualized operating system has no idea that it is on virtual hardware. It just sees hardware as it nor-mally would.
In Chapter 2 we’ll go into detail on how virtualization actually takes place with ESX Server, but for now it’s just important to understand that from a high level there are a few components that make up any virtualization environment:
2. Virtualization software that provides and manages the virtual environment
3. The virtual machine(s) themselves (or “VM’s”—virtual hard-ware presented to the guest
4. The guest operating system that is installed on the virtual machine
Figure 1.1 The basic virtualization model
Knowing that each of these four elements must be in place in a vir-tual environment and understanding that they are distinctly different resources will allow you to understand where different virtualization software is used. Let’s examine each of these components a bit before moving into the breakdown of the different virtualization technolo-gies available from VMware.
The Host Machine
The host machine in a virtual environment provides the resources that the virtual machines will eventually have access to. Obviously the more of the resources available the more virtual machines you can host. Or put more directly, “the bigger the host machine, the more virtual machines you can run on it.” This really makes sense if you look at Figure 1.1. In Component 1 of the diagram, the host machine has a single processor, some RAM, a single disk, and a single network card. Assuming that the host is going to use some of each of these
2
Virtualization Software
3
1
Virtual Machine
Host Machine
core four resources for its own operation, you have what’s leftover available for the virtual machines you want to run on it.
For example, let’s assume that the host is using 10% of the available processor for itself. That leaves 90% of the CPU available for the vir-tual machines. If you’re only running a single virvir-tual machine (VM) then this may be more than enough. However, if you’re trying to run 10 VMs on that host then the hosts “extra” 90% CPU availability will probably lead to a bottleneck.
The other challenge is that since all of the VMs are sharing the same resource (the CPU in this case), how do you keep them from step-ping on each other? This is actually where Component 2 from Figure 1 comes in—the virtualization software.
The Virtualization Software
The virtualization software layer provides each virtual machine access to the host resources. It’s also responsible for scheduling the physical resources among the various VMs. This virtualization software is the cornerstone of the entire virtualization environment. It creates the virtual machines for use, manages the resources provided to the VMs, schedules resource usage when there is contention for a specific resource, and provides a management and configuration interface for the VMs.
Again, we can’t stress enough that this software is the backbone of the system. The more robust this virtualization software is the better it is at scheduling and sharing physical resources. This leads to more efficient virtual machines.
VMware provides three versions of this virtualization software. The first two—“VMware Workstation” and “VMware GSX Server”—are virtualization software packages that install onto an existing operat-ing system on a host computer. The third version (“VMware ESX Server”) is a full operating system in-and-of itself. We’ll explore some of the reasons to help you choose which product you should use later in this chapter. The important idea here is to understand that ESX
Server is both its own operating system and also the virtualization software (Components 1 and 2 in the model we’ve been referencing), while GSX Server and Workstation are virtualization software pack-ages that are installed on and rely upon other operating systems.
The Virtual Machine
The term “virtual machine” is often incorrectly used to describe both the virtual machine (Component 3) and the guest operating system (Component 4). For clarity in this book we will not mix the two. The virtual machine is actually the virtual hardware (or the combined vir-tual hardware and the virvir-tual BIOS) presented to the guest operating systems. It’s the software-based virtualization of physical hardware. The guest operating systems that we install into these “machines” are unaware that the hardware they see is virtual. All the guest OS knows is that it sees this type of processor, that type of network card, this much memory, etc.
It’s important to understand that the virtual machine is not the OS but instead the hardware and configurations that are presented to the guest OS.
The Guest Operating System
In case it’s not clear by now, the guest OS is the Intel-based operat-ing system (Windows, Linux, Novell, DOS, whatever) that’s runnoperat-ing on a VM. Again, understanding that the guest OS (or “guest machine” or simply “guest”) is simply the software (Component 4) that’s installed onto a VM (Component 3) will make your life easier when it comes to understanding and troubleshooting your environment. Once all four of these components are in place you’ll have a virtual environment. How this virtual environment performs, is managed, and the types of functionality available in your virtual environment are all dependent on the type of software you’re using to provide your virtual environment.
Which VMware product which should I use?
Well that’s the question isn’t it? Since this book is about ESX Server we can probably assume that you’ve already made your decision. Then again you might be using this book to help make your decision, so we’ll go ahead and look at the products and how they fit into the IT world.
Most VMware folks started out using VMware Workstation (simply called “Workstation” by VMware geeks in the know). Workstation allows us to create virtual workstations and servers on our own PC. This lets us create small test environments that we can use to test scripts, new software packages, upgrades, etc. VMware Workstation is perfect for this.
Figure 1.2: Where do the VMware products fit?
Of course Workstation has its limitations. Probably the biggest one is that VMs can only run while you’re logged into your host worksta-tion. Logoff and the VMs shutdown. Also, VMware Workstation is pretty much a local user tool which means that there is really no remote administration capabilities whatsoever. These limitations keep Workstation on the desktops of developers, engineers, and traveling salespeople who have to give multi-server demos off their laptops. No one uses Workstation for production environments.
VMware GSX Server (called “GSX” for short) is a step up from Workstation. GSX is basically a software package that installs onto an existing host operating system (either Linux or Windows). It offers
Development Test Production
Limited Production Use
ESX Server GSX Server
some remote management and remote console access to the VMs, and the various VMs can be configured to run as services without any console interaction required. Its limitation is really that it has to use resources from the host hardware through the host OS. This really limits the scalability and performance of GSX.
The reason for this is that with GSX, VMs do not have direct access to the hardware. Let’s look at an example to see how this can cause a problem. We’ll look at memory use. Let’s assume you configure 384MB of memory for a VM running on GSX Server for Windows. The “catch” here is that since GSX is “just” another Windows appli-cation, the VM doesn’t get direct access to 384MB of memory. Instead it requests 384MB of memory from Windows.
Sure you’ll see the host’s memory utilization go up by 384MB when you turn on the VM, but the guest OS has to send all memory requests to Windows. In this case you’ll have a host OS managing the “physical” memory for the guest OS. (This is on top of the guest OS managing its own memory within the VM.)
While this is just a simplified example, it points out some of the inherent limits with GSX. Does this mean GSX isn’t a good product? Not at all. It just means that it has limitations stemming from the fact the virtualization software runs on top of a host OS. GSX is still used in plenty of production environments—especially those that don’t require enterprise class scalability for their VMs, those that have a limited numbers of VMs, and those that do not require maximum performance. GSX is also frequently found in corporate test labs and is used to allow administrators to get the benefits of a “virtual” test environment without the them needing to know all the ins and outs of a full virtual server OS. Finally, a lot of companies use GSX when they don’t have the budget to buy ESX-certified hardware when the VMware champions can’t win the political “everything has to run on Windows” battle.
What makes ESX different than GSX, Workstation, or even Microsoft’s Virtual Server?
VMware ESX Server is its own operating system. Unlike GSX or Microsoft Virtual Server 2005, ESX is not a software package that installs into a host OS—ESX is the host OS. Engineered from the beginning to be nothing more than a VM host, ESX Server is com-pletely designed to give the VMs the best performance possible and to allow you (the admin) to control and shape the way the host resources are shared and utilized.
So what does using to ESX instead of GSX or Workstation get you? The answer is simple: performance and reliability.
• Performance. ESX Server provides a level of performance for your VMs that simply cannot be found in GSX or Workstation. It also allows for more advanced resource allo-cation, fine tuning of performance, a better VM-to-processor ratio, and more advanced resource sharing.
• Reliability. VMware published an ESX Server Hardware Compatibility List (HCL). If the hardware you’re using for ESX is on the HCL, then you can be confident that every-thing will work as expected. ESX also lets you get rid of any problems that exist in the host OS since host OSes don’t exist with ESX.
In short if you’re looking to implement virtual machines on an enter-prise level or if you’re looking to host a lot of production servers as VMs, then ESX is the simple choice.
A 60-second Overview of the ESX Server
Architecture
An ESX Server is made up of two core components: • The ESX Server kernel (called “VMkernel”)
• The console operating system (”COS”, also contains “VMnix”)
The term “ESX Server” is usually used to describe all of this stuff together.
Figure 1.3: ESX Server Simplified
There is quite a bit of confusion in regards to what VMnix and the VMkernel really are. VMnix is a customized linux kernel based on the Redhat 7.2 distribution. Specific kernel options are specified in VMnix that optimize the console operating system for running virtu-al machines. Various scripts are initivirtu-alized in the startup process of the console operating system which call and load the VMkernel when the system is booted into runlevel 3. Once the system has complete-ly loaded the VMkernel, resource control and virtual machine man-agement is passed from the COS to the VMkernel.
In Chapter 2 we’ll go into great (and sometimes painful) detail about the VMkernel and the console OS. For now we just want you to understand the basic architecture so you see how ESX is different from VMware’s other two main products.
Referring to Figure 1.3 you can see that the console operating system is what allows us to interact with this server. This operating systems allows us Secure Shell access, supports a web based management con-sole, allows us to FTP and copy files to and from the host. But, the Console operating system is not ESX by itself, it does not schedule resources or manage hardware access, and basically would be a sim-ple Redhat server if it wasn’t for the VMkernel.
Admin
ESX Server Simplified
Admin accessvia SSH,
HTTPS, etc. Virtual Machines
Host Hardware
Processors Memory Disk Network
Console OS
The VMkernel manages/schedules access to specific hardware resources on the host. It is the VMkernel that provides the Virtual Machines that guest operating systems can be installed to. This ker-nel is what makes ESX different from all the other software packages available. The VMkernel allows direct hardware access to the core 4 resources. It manages memory for the VMs, schedules processor time for the VMs, maintains virtual switches for VM network connectivi-ty and schedules access to local and remote storage.
This kernel has been specifically built for this task. Unlike Windows or Linux hosts that have been built to be multi-purpose servers, this kernel’s whole purpose is to share and manage access to resources. This makes it extremely light (Less than an 80-MB install package) yet extremely powerful. Overhead in VMware ESX is estimated at 3-8%, while overhead for the host in these other OS’s is generally 10-20% and sometimes as high as 30% depending on configurations. The reduction in overhead basically comes from ESX being a “bare metal” product. Unlike any of the other technologies available, ESX makes the most of your hardware and has been built from the ground up to provide superb VM performance. Contrast this to the GSX, Workstation and Microsoft Virtual Server products that are really add-ons to operating systems that are built to handle numerous tasks and are not focused on providing high end VM performance.
Chapter
2
Now that you have a basic understanding of the VMware products and a basic knowledge of ESX and how it works, its time to go into detail as to what makes it all possible. As a warning, this chapter is not intended for those with a weak heart or who are prone to falling asleep. If you want to truly understand the magic behind ESX server then this chapter is for you! Also, make sure you have an empty stom-ach, because there’s a lot of information to digest here.
The Console Operating System versus
VMkernel
One of the most difficult concepts for new VMware admins to under-stand is the difference between the Console Operating System (called “COS”) and the VMkernel. Both play extremely important roles in your ESX environment and it’s important to fully understand what each does and is capable of doing. A common misconception of ESX is that it “runs on Linux.” Let’s set the record straight once and for all: ESX is not Linux. It’s not derived from Linux, and does not run on Linux. Now that that’s out of the way, let’s get back to the COS and the VMkernel.
The easiest way to distinguish the differences between these two components is to think of the server’s console as being the “physical world” and the VMkernel as the “virtual world.” The console lets you touch and interact with it directly, and it allows access to modify con-figurations and manage the environment. The VMkernel manages everything that relates to the “virtual world” and the guests that run within the host.
Console Operating System
The COS is used to boot your system and prepare your hardware for the VMkernel. As the COS loads, it acts as the bootstrap for the VMkernel which means it prepares all the necessary resources for turnover to the VMkernel. Once the COS has loaded ESX, the VMkernel will warm boot the system, assuming the role of primary operating system. The VMkernel will then load the COS and several
other “helper worlds” as privileged VMs. The COS is responsible for quite a few things that are vital to the proper operation of ESX, including:
• User interaction with ESX. The COS is responsible for presenting the various methods to communicate with the ESX host sys-tem. It runs services that allow user interaction with the host using various methods such as:
• Direct console access
• Telnet/ssh access to the console • Web interface
• FTP
• Proc file system. The proc file system can be utilized by both the COS and the VMkernel to provide real time statistics and to change configuration options on the fly. The proc file system will be discussed in greater detail in Chapter 6.
• Authentication. There are several processes that run within the COS that provide authentication. These mechanisms deter-mine what rights a specific user ID has over the COS itself and the various guests running on the host. Chapter 7 is ded-icated to these processes and how they interact with each other, the VMkernel, and the COS.
• Running Support Applications. There are some applications that may be run within the COS to provide extended support of the host environment. Every major hardware vendor has some form of agent that will run within the COS that can detect hardware issues as they arise (voltage problems, drive failures, fans quitting, etc.) In some scenarios it may also be necessary to install a local backup client to the COS to back-up critical system files. The number of applications that are installed and run on the COS should be limited though, as the COS is really designed to support ESX and nothing else.
VMkernel
Once the OS has loaded, the VMkernel is started. At this point the VMkernel will warm boot the system and assume responsibility for all hardware management and resource scheduling is turned over to
the VMkernel for management. Even the COS gets reloaded by the VMkernel as a VM and is restricted by the VMkernel and its config-urations. (Okay, technically, the COS still manages it’s own memory and NIC, but that’s it.) The COS must follow the same rules for resource allocations and sharing as every virtual guest running on the host.
The VMkernel performs a number of functions, but one of the main jobs it has is to manage the interaction of virtual machine hardware and the physical hardware of the server. It acts as the “go-between” for scheduling resources for VMs on an as needed and as configured basis. While this may seem like a brief and simplistic description of the VMkernel, the remainder of this chapter focuses heavily on how the VMkernel works its magic and makes ESX what it is.
The ESX Boot Process
By taking a look at the boot process of an ESX host we can see how the COS and the VMkernel interact and at what point the VMkernel takes control of the system resources. There are several steps to the boot process. While we won’t cover all of them, we will highlight the ones that perform important system tasks as they relate to ESX.
LILO
LILO (or the “Linux Loader”) is a boot loader application (similar to ntloader for Windows) that the system reads when it boots from the hard drive. Based on the information contained in /etc/lilo.conf file, the system begins its boot process. The default boot option for LILO within ESX is to boot and load the VMkernel. The /etc/lilo.conf file also contains information on how the COS should be configured as it is booting. This information contains the amount of memory allo-cated to it and the devices that are configured for COS use. Many of the LILO configuration items are controlled by the vmkpcidivy com-mand, discussed later in this chapter
Console Operating System
After LILO properly initializes the boot instructions the COS begins to load. The majority of the boot process is contained in the COS.
Most of these steps are used to prepare the VMkernel to take control of the hardware resources.
init
The first process that the COS executes is init. This process reads the /etc/inittab file and determines the system runlevel that should be executed. (A Linux runlevel determines what services are started on the server and the order in which they are started.) Varying runlevels on a Linux system is comparable to the various boot options available on a Windows server such as “Safe Mode” or “Command Prompt Only.” The default system runlevel for ESX is 3, which means the sys-tem will boot and present a console for syssys-tem login. Based on this value the COS will run the scripts contained in the /etc/rc.d/rc3.d directory during the boot process.
/etc/rc.d/rc3.d
The /etc/rc.d/rc3.d directory actually contains symbolic links to start scripts in the /etc/init.d directory. By running an “ls” command in the /etc/rc.d/rc3.d directory you will see some scripts that begin with a K and some that begin with an S. Scripts that begin with a K are used to stop (or “kill”) a service during the boot process (or ensure it is not running) and scripts beginning with an S are used to start a service. You will also notice a number after the S or the K in the script’s file name. These determine the order the script is run, starting at 0 and going up to 99. The S scripts are executed in ascending number order, whereas the K scripts are executed in descending order. The order of the K or S values in the file have no meaning when it comes to what order a script runs in.
S00vmkstart
If you run an “ls –l” command in the scripts directory you’ll notice that the S00vmkstart command actually links to a script called vmkhalt. By running this script first, VMware ensures that there are no VMkernel processes running on the system during the boot process.
S10network
The network script (S10network) starts the TCP/IP services on the COS and assigns the IP address and hostname of the system.
S12syslog
The syslog script starts the daemon that processes system logs. Starting this script allows the remainder of the boot process to be logged. After the VMkernel begins to load it also provides a mecha-nism to capture the log files generated by the VMkernel for review when errors occur.
S56xinetd
The xinetd script starts the services required for the COS to handle incoming requests for access. Each application that can be started by xinetd has a configuration file in /etc/xinetd.d. If the “disable = no” flag is set in the configuration file of a particular application then xinetd starts the application. (Yeah it’s a double-negative.) The most important application that is started here is the vmware-authd appli-cation which provides a way to connect and authenticate to ESX to perform VMkernel modifications.
S90vmware
This is where the VMkernel finally begins to load. The first thing that the VMkernel does when it starts is load the proper device drivers to interact with the physical hardware of the host. You can view all the drivers that the VMkernel may utilize by looking in the /usr/lib/vmware/vmkmod directory.
Once the VMkernel has successfully loaded the proper hardware drivers it starts to run its various support scripts:
• The vmklogger sends messages to the syslog daemon and gen-erates logs the entire time the VMkernel is running.
• The vmkdump script saves any existing VMkernel dump files from the VMcore dump partition and prepares the partition in the event that the VMkernel generates unrecoverable errors.
Next the VMFS partitions (the partitions used to store all of your VM disk files) are mounted. The VMkernel simply scans the SCSI devices of the system and then automatically mounts any partition that is con-figured as VMFS. Once the VMFS partitions are mounted the
VMkernel is completely loaded and ready to start managing virtual machines.
S91httpd.vmware
One of the last steps of the boot process for the COS is to start the VMware MUI (the web interface for VMware management). At this point the VMkernel has been loaded and is running. Starting the MUI provides us with an interface used to graphically interact with ESX. Once the MUI is loaded a display plugged into the host’s local console will display a message stating everything is properly loaded and you can now access your ESX host from a web browser.
So why do I need to know the boot process?
You need to understand the basic boot process to understand that the VMkernel is a separate entity from the COS. Also if your server fails to boot or certain services or processes fail to start you’ll have a good idea of where to start looking for problems. If you’re not familiar with Linux then this is all probably very new to you. If you have some experience with Linux then this section just helped you under-stand how the VMkernel fits into the picture.
Now let’s take a look at how the VMkernel does its magic and “vir-tualizes” the hardware.
Hardware Virtualization
The whole idea behind VMware is to present a standard hardware layer as a virtual machine to the guest operating systems. These vir-tual hardware resources remain constant regardless of what physical hardware makes up the host.
The VMkernel is responsible for providing the virtual hardware layer to virtual machines. When a guest OS accesses a resource the VMkernel is then responsible for mapping the virtual request through to the physical hardware for processing. Since VMware
pres-ents standard hardware to virtual guests, you need to be familiar with this hardware. Some resources such as SCSI and the network have several options, so we need to understand when each option is used and what it changes in the environment with each.
System Devices
When ESX presents a hardware layer to a guest operating system it presents a system based on Intel’s 440BX chipset. This is a highly supported chipset and is compatible with every guest operating sys-tem that can be run within ESX. You may be wondering how we can run Pentium 4 XEON and AMD Opteron processors in a guest that has a 440BX chipset. While this sounds a little off, we’ll describe that in detail a later in this chapter. For now you just need to understand that the 440BX is what is presented to the guest and it allows for a high degree of compatibility across numerous platforms for our guest operating systems.
Processors
Assuming you are utilizing a processor that meets the requirements of ESX server, your guest will see the same type of physical proces-sor that’s installed on the host. The VMkernel is capable of accepting processor calls and handing it straight to the physical processors of the host with limited virtualization overhead. By presenting the host processor type to the guest, the VMkernel does not need to perform any conversions to ensure compatibility between the physical and hardware layer. This simply means that the processor is NOT accessed through on emulation layer. And that if your host has an MP or DP type processor then that’s what is presented to the guest. It’s important to note that not all registers of the physical CPU are presented by the VMkernel. While VMware is fairly tight-lipped about these registers, one that is known for sure is processor serial numbers. Applications that are licensed to a serial number of a processor or group of processors will not function within VMware.
Network
ESX provides us with two hardware options when presenting virtual network adapters to guest operating systems. Depending on the guest operating system, there may be a requirement for one over the other.
VLANCE
The vlance adapter is a virtualized AMD PCNET driver. This adapter has guaranteed compatibility across every guest operating system that can be run within ESX. Since it’s based on legacy hardware it will also have some limitations when utilizing it within a guest. After installing the drivers you’ll notice that the connection speed within the guest operating system shows 10Mb/sec. This is a limitation of the driver and in fact doesn’t impact the transfer rate of the hardware. The vlance adapter will utilize as much bandwidth as is available to the physical connection. There is native support for this device in every operating system that ESX has been certified for. If you’re configur-ing a DOS boot disk for a network based installation or to use a DOS based tool such as Ghost, this is the only driver that will properly function. Using this driver will require increased virtualization over-head over the other networking option available to us.
VMXNET
VMware has created a virtual network device that was designed from the ground up to interact with the VMkernel. This device is the vmxnet adapter. Because of its tight integration with the VMkernel you will receive enhanced performance when using it in a guest, especially with high speed connections. Since this device is a VMware creation there is no native support for it in any guest operating sys-tem. The only way to configure this device is to install the drivers provided by the VMware Tools installation package within the guest. Using this adapter minimizes the amount of virtualization overhead and increases the network performance for a guest OS. It’s important to note that not all operating systems will have the capability to use this device. Use of this device is based strictly on the availability of a VMware Tools installation package and vmxnet driver for the target guest.
SCSI
Like the virtual network adapter, VMware provides two different SCSI adapters that may be presented to a guest operating system. The device that’s used by your specific guest depends on the operating system that will be installed. The two options available to use are an LSI Logic adapter or a Bus Logic adapter. Each adapter has different levels of support in each of the supported operating systems. To elim-inate any error when building a guest, ESX automatically assigns the proper controller during the virtual machine configuration wizard based on operating system choice. While the default controller may be changed in some cases, it typically requires additional drivers to first be installed in the guest. It may also impact the performance of the virtual machine (more on this in Chapter 4). As a general rule of thumb the choices that VMware makes for us guarantee compatibili-ty for our guest servers.
As you can see the virtual hardware presented to the guests create a relatively flexible environment that can be used by almost any main-stream Intel OS. Now that you have a basic understanding of the vir-tual hardware as presented to the guests, let’s look a little more at how hardware is divided between the physical and virtual worlds.
Hardware Allocation
When installing and configuring ESX, you’ll see that both the COS and the VMkernel are responsible for controlling certain aspects of the hardware. There are three different settings for your hardware devices: Virtual, Console, or Shared. Devices that are allocated as “virtual” can only be accessed by the VMkernel (the virtual world). “Console” devices are those that are limited to functioning in the con-sole operating system (the physical world). The third option is a mix of the two and allows for a device to be accessed in both the COS and the VMkernel (physical and virtual worlds). There are also sev-eral different ways in which the device allocations may be changed to accommodate changing needs of the environment.
Virtual
Virtual devices, as stated above, may only be accessed by the virtual guests running on your host. The first obvious device that would be configured for virtual machines is at least one network adapter. (Later in this book we’ll discuss why you would want to strongly con-sider using multiple adapters). Configuring a network adapter for vir-tual machine use is the only way that your guests will be able to com-municate with the network outside of your host server.
In addition to network connectivity you also need a place to store the data that makes up your guest. In order to do this a SCSI adapter should also be assigned for virtual machine (which means the VMkernel). To simplify things for now, ESX also considers fiber adapters to be SCSI adapters in terms of virtual, console, or shared configuration. Depending on the size of your environment and the type of data you’ll be connecting to, you may or may not have a need for fiber HBAs or additional SCSI adapters.
Console
While “virtual” devices are only be seen by your virtual guests, “con-sole” devices, as you may have guessed, are only seen by the COS. Every ESX host has at least one network adapter that is used by the service console. (This adapter is usually, although not always, dedicat-ed to the COS.) When you communicate to the host with the MUI or connect via ssh, you’re interacting with this network interface. When you install backup or management agents on the console operating system, this adapter is also used to communicate through the net-work.
In order for the console operating system to properly boot, it needs a disk controller to be allocated to Console use. Since the COS is a standalone operating system (just like Windows) it needs a hard drive configured so it can create and use the partitions and files required to boot. This can be a physically attached hard drive, or in the case of ESX 2.5 or newer, a remote boot off of a SAN. (More on this in Chapter 4.)
We should note that you don’t need a disk controller that’s dedicated to the COS only. You just need to have a controller (either “shared” or “console”) that’s available to the COS so it can boot.
Shared Resources
Shared resources are those that can be accessed by both the VMkernel and the COS at the same time. Consider the situation we just alluded to previously where you have a system with only one SCSI controller and no SAN in your environment. In order to hold large amounts of data you purchase a SCSI drive cage that externally attaches to your ESX host. Since you only have the one SCSI adapter, you need to make sure the console has access to the internal hard drives for the installation of the COS. We also need to make sure that once ESX is installed the VMkernel will have the appropriate access to the exter-nal drive cage.
Shared devices are not limited to SCSI controllers, but may also be fiber HBAs or network adapters. In Chapter 4 we’ll introduce you to some of the advanced configurations available to you utilizing the shared bus mode.
Modifying These Configurations
During the installation process of ESX we’ll show you how to initial-ly allocate your devices in Chapter 3. As your needs change and your virtual environment grows it’s essential to know that you can modify which devices are seen in the physical and virtual worlds. Fortunately VMware provides several tools to make this possible.
MUI
Modifying device allocations utilizing the MUI (the web interface) is a relatively simple process. In order to access the configuration screen you need to log into the MUI as the root user. This will enable the “Options” tab on the main page. The top link in the left column of the options tab will be “Startup Profile.” This is the area where you can configure HyperThreading options and memory resources for the service console, and device allocations.
Device allocation configuration through the MUI is somewhat limit-ed in that the only devices that can be configurlimit-ed as “Sharlimit-ed” are SCSI and Fiber Storage Adapters. In order to share a device you must choose to allocate it to “Virtual Machines” and then select the “Share with Service Console” checkbox. You’ll notice that the network adapters installed in the system do not have this option. Configuring the network devices for shared use is an advanced configuration and not recommended unless certain conditions are met. This limitation should be sufficient for a vast majority of the implementations, but we feel it’s important to note that you cannot do everything from the MUI. Details on this can be found in Chapter 4.
Console Operating System
Modifying device allocations through the service console can be done with the vmkpcidivy command. This command can be run in two dif-ferent ways: interactive and batch mode.
Running vmkpcidivy in interactive mode is the easiest way to config-ure your devices outside of the MUI. You can run vmkpcidivy in interactive mode by accessing the service console (locally or via ssh) and using the following command:
# vmkpcidivy –i
After executing the interactive mode command you’ll be presented with a list of configurable devices in the system and how they’re cur-rently allocated. (This is shown in the Example: 2.1). The devices are presented in a list categorized by their allocations. You’ll see “Shared” devices listed twice—once under the Console section and once in the Virtual Machines section.
Example 2.1
[root@esx1 root]# vmkpcidivy -i
Checking for existing VMnix Boot Configurations. The following VMnix kernel images are defined on your system:
Boot image configuration: esx
Memory: 192M
Service Console devices:
Ethernet controller: Intel Corporation 82557 [Ethernet Pro 100] (rev 08)
RAID storage controller: Symbios Logic Inc. (formerly NCR) 53c895 (rev 02) (shared)
VM devices:
Ethernet controller: 3Com Corporation 3c905C-TX [Fast Etherlink] (rev 78)
RAID storage controller: Symbios Logic Inc. (formerly NCR) 53c895 (rev 02) (shared)
Type in the name of the boot image configuration you wish to configure or type "new" to create a new image [esx]:
Following the list of devices is a prompt asking if you would like to modify an existing configuration or create a new one. The default configuration name for ESX 2.1.1 and higher is ESX. Prior to 2.1.1 the default configuration name was VMNIX. You can tell what your default is by either paying attention to the LILO boot menu at start-up or by viewing the /etc/lilo.conf file with the following command:
# grep default /etc/lilo.conf
By choosing your default configuration you will be presented with each device and its current setting. When presented with a list of devices, the current values will be set as the “default” values. By sim-ply hitting enter the particular device that’s listed will keep its current configuration.
There are three possible values to use for allocating your devices: c, v, or s. These represent Console, Virtual Machines, and Shared, respectively. When you get to the particular device(s) you wish to modify, enter the proper value and hit enter. Once you’ve gone through the entire list of devices you’ll be prompted as to whether you want to apply the configuration changes or not. Once you’ve cho-sen to apply the changes you will need to reboot the system the changes to take effect. If you’re using vmkpcidivy for information gathering you’ll either want to break out of the application using CTRL+C or choose not to apply the changes to the configuration.
We strongly recommended that if you’re unsure of the change you’re attempting to make that you create a new configuration with a prop-er temporary name such as “esx-test.” This will require that you type “new” at the first prompt followed by your temporary configuration name at the second. When you create a new profile, the settings from your original profile are not remembered. You’ll have to pay close attention to each option presented to ensure your system comes up in a functional state after its next reboot.
The Core Four Resources
There are four resources that you need to strongly consider when you review and design your virtual environment. (These are what we’ve starting calling “Core Four.”) Properly understanding and configur-ing these resources are essential to maintainconfigur-ing a stable virtual envi-ronment. This section focuses on these “Core Four” resources and how they pertain to VMware ESX Server and its guests.
Processor
As mentioned previously, the virtualization of the processor compo-nent that is presented to virtual machines is slightly different than other devices. As we discussed, the motherboard architecture present-ed to the guest operating system is baspresent-ed on the Intel 440BX chipset which is a Pentium III-based motherboard. So how does this impact the physical processors that are installed in the host server?
This simple answer is, “it doesn’t.” The best way to describe how VMware virtualizes the processor was described to us by one of the VMware sales engineers that we frequently have the opportunity to work with. Since the system architecture as presented to the guest operating system is 440BX-based, the device manager within Windows shows you all the typical components of that “virtual moth-erboard.” The single exception in this virtual motherboard is that at the hardware level there’s a hole cut out of the motherboard where the processor belongs.
The VMkernel, based on processor family, presents the specific capa-bilities of your host processors to the guest operating system which allows full utilization of the processors installed. While there are some registers that are not virtualized, the guest has the capability to benefit from the key registers of advanced processors such as Pentium 4 XEON and AMD Opteron. Simply put, the processors are not really virtualized the same way as the other “core four” (memo-ry, disk and network) are. The processor utilization is scheduled, but what the guest sees is pretty much what it gets.
Hyper-threading
Hyper-threading is an Intel technology that allows a single processor to execute threads in parallel, which Intel claims can boost perform-ance by up to 30%. If you think about it, what Intel is really doing is presenting two processors to the OS for each physical processor installed. This idea and technology really comes from trying to make up for the poor task scheduling that Microsoft does in its Windows operating systems (but that’s another book all together).
VMware ESX 2.1 introduced support for hyper-threaded processors. The additional logical processors are packaged with the physical and are numbered adjacently. For example, processors 0 and 1 would be physical CPU 1 and its logical counterpart (and 2 and 3 would be CPU 2 and its logical counterpart, etc.). This behavior is different than that displayed in a typical x86 operating system in that all phys-ical CPUs are counted first, and then the logphys-ical CPU pairs are num-bered. It will be important to remember this numbering system if you begin to use the more advanced concepts of “pinning” a Guest VM to a specific processor for performance reasons.
The increase that a system receives from hyper-threading is depend-ent on how well the applications running on the system utilize the system cache. While a typical operating system requires that hyper-threading be enabled or disabled for the entire system, VMware has provided several mechanisms for configuring hyper-threading sharing on the system that can be configured on a per virtual machine basis: • Any. This is the default setting for virtual machines running
single processor package at the ESX level. It allows you to get the most out of enabling hyper-threading on your system, but can introduce problems where an inefficient application may impact overall performance of the other virtual machines sharing a package with it.
• Internal. This option is only supported by SMP (multiproces-sor) machines. It allows both virtual CPUs for a virtual machine to run in a single package and isolates it from any other virtual CPUs in the system. This prevents the config-ured guest form impacting other guests and protects it from other guests that may have inefficient applications. If overall system utilization allows, a guest configured with internal hyper-threading sharing can still utilize a package per virtual CPU to maximize performance.
• None. In cases where an application is known to perform poorly with hyper-threading, sharing can be disabled. This completely isolates each virtual CPU of the system to its own package. This option should only be used when suggested by VMware or an application vendor as it isolates a large amount of system resources.
Modifying the hyper-threading settings of the system may be done one of three ways. Your system must have hyper-threading enabled at the hardware level in order to view and modify these options. Also, the virtual machine must be powered off for these modifications to be possible.
• MUI. By using the MUI, the hyper-threading sharing option can be modified two ways. The first is by editing the CPU Resource Settings for the virtual machine. You’ll be present-ed with a checkbox that’s labelpresent-ed “Isolate Virtual Machine from Hyper-Threading.” The behavior of this setting depends on whether the system is a single processor system or if it has Virtual-SMP enabled. For a single processor machine this option will set the sharing value to “none.” For SMP machines, the value will be changed to “internal. The other option is to use the verbose configuration option for the guest. When presented with a list of configuration options, add (or modify) a value titled “cpu.htsharing.” Assign the value of “any, internal, or none” to this option.
• COS. You can easily set the hyper-threading sharing value by directly modifying the VMX file for the virtual machine in question. The easiest way to modify the file is to utilize the following command:
# echo cpu.htsharing = \“value\” >> /path/to/server-name.vmx
Make sure you substitute the proper value inside the escaped quota-tion marks. The escape character of “\” is required for echo to prop-erly insert the quotation marks into the configuration file. Another thing you MUST be careful of is that you use the double output sym-bol “>>”. If you use a single output symsym-bol the existing text of the file will be overwritten. It’s always recommended that you make a back-up copy of the VMX file before making any modification. If you’re familiar with the vi editor then you may use that to modify the VMX file with the following line:
cpu.htsharing = “value”
There’s one final note about using hyper-threading in your ESX envi-ronment. If you’re using a system that utilizes a NUMA memory architecture, it’s strongly recommended that you only use ESX ver-sions 2.1.2 or higher. There have been specific issues tied to these sys-tems that appear to have been fixed by the 2.1.2 updates. While there is no “official” announcement from VMware on this, internal employ-ees have confirmed that update code was included to enhance this functionality. Non-NUMA systems seem to perform extremely well with hyper-threading enabled regardless of the ESX version.
Symmetrical Multi-Processing (also known as SMP or
Virtual-SMP)
SMP is an add-on module for ESX that provides the capability to con-figure multi-processor guest operating systems. You enable Virtual SMP by plugging in a license key to your ESX host either during the install process or by modifying the licensing options afterwards. (Note that you do NOT need an SMP license to use ESX on a multi-processor host. You only need it to create VMs that use multiple physical host processors.) While SMP can provide enhanced perform-ance to your system, there are several guidelines that should be
