When SAN administrators configure a SAN array for use by an ESX host and its virtual machines, some aspects of configuration and setup are different than with other types of storage. The key considerations when planning a VMware
Infrastructure installation with SAN include:
Which SAN hardware should you select?
How should you provision LUNs to the virtual infrastructure (volume size versus number of volumes, and using VMFS versus RDM)?
Which virtual machine deployment methods should you use (cloning and patching guest operating systems)?
What storage multipathing and failover options should you use (active/passive versus active/active)?
Will VMware ESX boot from SAN?
This chapter describes the factors to consider (for both VMware Infrastructure and SAN storage administrators) when using VMware ESX with a SAN array. It also provides information on choosing from different SAN options when configuring ESX hosts to work with SAN storage.
Topics included in this chapter are the following:
“Considerations for VMware ESX System Designs” on page 72
“VMware ESX with SAN Design Basics” on page 73
“VMware ESX, VMFS, and SAN Storage Choices” on page 75
“SAN System Design Choices” on page 86
4
Considerations for VMware ESX System Designs
The types of server hosts you deploy and the amount of storage space that virtual machines require determine the level of service the infrastructure can provide and how well the environment can scale to higher service demands as your business grows. The following is a list of factors you need to consider when building your infrastructure to scale in response to workload changes:
What types of SAN configuration or topologies do you need?
♦ Do you want to use single fabric or dual fabric? VMware Infrastructure 3 supports both.
♦ How many paths to each volume are needed? It is highly recommended that at least two paths be provided to each volume for redundancy.
♦ Is there enough bandwidth for your SAN? VMware Infrastructure 3 supports both 2GFC and 4GFC.
♦ What types of array do you need? VMware Infrastructure 3 supports
active/passive, active/active, FC-AL, and direct-connect storage arrays. It is very important that you get the latest information on arrays from the
hardware compatibility list posted on VMware.com.
How many virtual machines can I install per ESX host? This determines the type of server (CPU, memory, and so on).
How big is each virtual machine’s operating system and its data disks? This determines the storage capacity now (that is, which storage array model to use, how much disk space to buy now, and how much disk space to buy in six
months). For each virtual machine, you can roughly estimate storage requirements using the following calculations:
♦ (Size of virtual machine) + (size of suspend/resume space for virtual machine)) + (size of RAM for virtual machine) + (100MB for log files per virtual machine) is the minimum space needed for each virtual machine.
NOTE: Size of suspend/resume snapshots of running virtual machines is equal to the size of the virtual machine.
♦ For example, assuming a 15GB virtual machine with 1GB virtual RAM, the calculation result is:
15GB (size of virtual machine) + 15GB (space for suspend/resume) + 1GB (virtual RAM) + 100MB
The total recommended storage requirement is approximately 31.1GB. You should also plan extra storage capacity to accommodate disk-based snapshots according to vendor recommendations.
What sorts of applications are planned for the virtual machines? Having this information helps determine the network adapter and FC bandwidth
requirements.
What rate of growth (business, data, and bandwidth) do you expect for your environment? This determines how to build your VMware and ESX infrastructure to allow room for growth while keeping disruption to a minimum.
VMware ESX with SAN Design Basics
Support for QLogic and Emulex FC HBAs allows an ESX host system to be connected to a SAN array. The virtual machines can then be stored on the SAN array volumes and can also use the SAN array volumes to store application data. Using ESX with a SAN improves flexibility, efficiency, and reliability. It also supports centralized management as well as failover and load balancing technologies.
Using a SAN with VMware ESX allows you to improve your environment’s failure resilience:
You can store data redundantly and configure multiple FC fabrics, eliminating a single point of failure.
Site Recovery Manager can extend disaster recovery (DR) capabilities provided the storage array replication software is integrated. See information on
VMware.com for solution compatibility.
ESX host systems provide multipathing by default and automatically support it for every virtual machine. See “Multipathing and Path Failover” on page 29.
Using ESX host systems extends failure resistance to the server. When you use SAN storage, all applications can instantly be restarted after host failure. See
“Designing for Server Failure” on page 82.
Using VMware ESX with a SAN makes high availability and automatic load balancing affordable for more applications than if dedicated hardware were used to provide standby services.
Because shared central storage is available, building virtual machine clusters that use MSCS becomes possible. See “Using Cluster Services” on page 83.
If virtual machines are used as standby systems for existing physical servers, shared storage is essential and a SAN is the best solution.
You can use the VMware VMotion capabilities to migrate virtual machines seamlessly from one host to another.
You can use VMware HA in conjunction with a SAN for a cold-standby solution that guarantees an immediate, automatic response.
You can use VMware DRS to automatically migrate virtual machines from one host to another for load balancing. Because storage is on a SAN array,
applications continue running seamlessly.
If you use VMware DRS clusters, you can put an ESX host into maintenance mode to have the system migrate all running virtual machines to other ESX hosts. You can then perform upgrades or other maintenance operations.
The transportability and encapsulation of VMware virtual machines complements the shared nature of SAN storage. When virtual machines are located on SAN-based storage, it becomes possible to shut down a virtual machine on one server and power it up on another server—or to suspend it on one server and resume operation on another server on the same network—in a matter of minutes. This allows you to migrate computing resources while maintaining consistent, shared access.
Use Cases for SAN Shared Storage
Using VMware ESX in conjunction with SAN is particularly useful for the following tasks:
Maintenance with zero downtime — When performing maintenance, you can use VMware DRS or VMotion to migrate virtual machines to other servers. If using shared storage, you can perform maintenance without interruption to the user.
Load balancing — You can use VMotion explicitly or use VMware DRS to migrate virtual machines to other hosts for load balancing. If using shared storage, you can perform load balancing without interruption to the user.
Storage consolidation and simplification of storage layout — Consolidating storage resources has administrative and utilization benefits in a virtual
infrastructure. You can start by reserving a large volume and then allocating portions to virtual machines as needed. Volume reservation and creation from the storage device needs to happen only once.
Disaster recovery — Having all data stored on a SAN can greatly facilitate remote storage of data backups. In addition, you can restart virtual machines on remote ESX hosts for recovery if one site is compromised.
Additional SAN Configuration Resources
In addition to this document, a number of other resources can help you configure your ESX host system in conjunction with a SAN.
VMware I/O Compatibility Guide — Lists the currently approved HBAs, HBA drivers, and driver versions. See http://www.vmware.com/support/pubs/.
VMware SAN Compatibility Guide — Lists currently approved storage arrays.
Get the latest information from http://www.vmware.com/support/pubs/.
VMware Release Notes — Provides information about known issues and workarounds. For the latest release notes, go to:
http://www.vmware.com/support/pubs
VMware Knowledge Base — Has information on common issues and workarounds. See http://www.vmware.com/kb.
Also, be sure to use your storage array vendor’s documentation to answer most setup questions. Your storage array vendor might also offer documentation on using the storage array in an ESX environment.
VMware ESX, VMFS, and SAN Storage Choices
This section discusses available ESX host, VMFS, and SAN storage choices and provides advice on how to make them.
Creating and Growing VMFS
VMFS can be deployed on a variety of SCSI-based storage devices, including Fibre Channel and iSCSI SAN equipment. A virtual disk stored on VMFS always appears to the virtual machine as a mounted SCSI device. The virtual disk hides a physical storage layer from the virtual machine’s operating system. This allows you to run even operating systems not certified for SAN inside the virtual machine.
For the operating system inside the virtual machine, VMFS preserves the guest operating system’s file system semantics, which ensure correct application behavior and data integrity for applications running in virtual machines.
You can set up VMFS-based datastores in advance on any storage device that your ESX host discovers. Select a larger volume (2TB maximum) if you plan to create multiple virtual machines on it. You can then add virtual machines dynamically without having to request additional disk space.
However, if more space is needed, you can increase the VMFS datastore size by adding extents at any time—up to 64TB. Each VMFS extent has a maximum size of 2TB.
Considerations When Creating a VMFS
You need to plan how to set up storage for your ESX host systems before you format storage devices with VMFS. It is recommended to have one VMFS partition per datastore in most configurations. You can, however, decide to use one large VMFS datastore or one that expands across multiple LUN extents. VMware ESX lets you have up to 256 LUNs per system, with the minimum volume size of 1.2GB.
You might want fewer, larger VMFS volumes for the following reasons:
More flexibility to create virtual machines without going back to the storage administrator for more space.
Simpler to resize virtual disks, create storage array snapshots, and so on.
Fewer VMFS-based datastores to manage.
You might want more, smaller storage volumes, each with a separate VMFS datastore, for the following reasons:
Less contention on each VMFS due to locking and SCSI reservation issues.
Less wasted storage space.
Different applications might need different RAID characteristics.
More flexibility, as the multipathing policy and disk shares are set per volume.
Use of Microsoft Cluster Service requires that each cluster disk resource is in its own LUN (RDM type is required for MSCS in VMware ESX environment).
Different backup policies and disk-based snapshots can be applied on an individual LUN basis.
You might decide to configure some of your servers to use fewer, larger VMFS datastores and other servers to use more, smaller VMFS datastores.
Choosing Fewer, Larger Volumes or More, Smaller Volumes
During ESX installation, you are prompted to create partitions for your system. You need to plan how to set up storage for your ESX host systems before you install. You can choose one of these approaches:
Many volumes with one VMFS datastore on each LUN.
Many volumes with one VMFS datastore spanning more than one LUN.
Fewer larger volumes with one VMFS datastore on each LUN.
Fewer larger volumes with one VMFS datastore spanning more than one LUN.
For VMware Infrastructure 3, it is recommended that you can have at most 16 VMFS extents per volume. You can, however, decide to use one large volume or multiple small volumes depending on I/O characteristics and your requirements.
Making Volume Decisions
When the storage characterization for a virtual machine is not available, there is often no simple answer when you need to decide on the volume size and number of LUNs to use. You can use a predictive or an adaptive approach for making the decision.
Predictive Scheme
In the predictive scheme, you:
Create several volumes with different storage characteristics.
Build a VMFS datastore in each volume (label each datastore according to its characteristics).
Locate each application in the appropriate RAID for its requirements.
Use disk shares to distinguish high-priority from low-priority virtual machines.
Note that disk shares are relevant only within a given ESX host. The shares assigned to virtual machines on one ESX host have no effect on virtual machines on other ESX hosts.
Adaptive Scheme
In the adaptive scheme, you:
Create a large volume (RAID 1+0 or RAID 5), with write caching enabled.
Build a VMFS datastore on that LUN.
Place four or five virtual disks on the VMFS datastore.
Run the applications and see whether or not disk performance is acceptable.
If performance is acceptable, you can place additional virtual disks on the VMFS.
If it is not, you create a new, larger volume, possibly with a different RAID level, and repeat the process. You can use cold migration so you do not lose virtual machines when recreating the volume.
Special Volume Configuration Tips
Each volume should have the right RAID level and storage characteristics for the applications in virtual machines that use the volume.
If multiple virtual machines access the same datastore, use disk shares to prioritize virtual machines.
Data Access: VMFS or RDM
By default, a virtual disk is created in a VMFS volume during virtual machine
creation. When guest operating systems issue SCSI commands to their virtual disks, the virtualization layer translates these commands to VMFS file operations.
An alternative to VMFS is using RDMs. As described earlier, RDMs are implemented using special files stored in a VMFS volume that act as a proxy for a raw device.
Using an RDM maintains many of the same advantages as creating a virtual disk in the VMFS but gains the advantage of benefits similar to those of direct access to a physical device.
Benefits of RDM Implementation in VMware ESX
Raw device mapping provides a number of benefits as listed below.
User-Friendly Persistent Names — RDM provides a user-friendly name for a mapped device. When you use a mapping, you do not need to refer to the device by its device name. Instead, you refer to it by the name of the mapping file. For example:
/
vmfs/volumes/myVolume/myVMDirectory/myRawDisk.vmdk Dynamic Name Resolution — RDM stores unique identification information for each mapped device. The VMFS file system resolves each mapping to its current SCSI device, regardless of changes in the physical configuration of the server due to adapter hardware changes, path changes, device relocation, and so forth.
Distributed File Locking — RDM makes it possible to use VMFS distributed locking for raw SCSI devices. Distributed locking on a raw device mapping makes it safe to use a shared raw volume without losing data when two virtual machines on different servers try to access the same LUN.
File Permissions — RDM makes it possible to set up file permissions. The permissions of the mapping file are enforced at file open time to protect the mapped volume.
File System Operations — RDM makes it possible to use file system utilities to work with a mapped volume, using the mapping file as a proxy. Most operations that are valid for an ordinary file can be applied to the mapping file and are redirected to operate on the mapped device.
Snapshots — RDM makes it possible to use virtual machine storage array snapshots on a mapped volume.
NOTE: Snapshots are not available when raw device mapping is used in physical compatibility mode. See “Virtual and Physical Compatibility Modes” on page 61.
VMotion — RDM lets you migrate a virtual machine using VMotion. When you use RDM, the mapping file acts as a proxy to allow VirtualCenter to migrate the virtual machine using the same mechanism that exists for migrating virtual disk files. See Figure 4-1.
Figure 4-1. VMotion of a Virtual Machine Using an RDM
SAN Management Agents — RDM makes it possible to run some SAN
management agents inside a virtual machine. Similarly, any software that needs to access a device using hardware-specific SCSI commands can be run inside a virtual machine. This kind of software is called “SCSI target-based software.”
NOTE: When you use SAN management agents, you need to select physical compatibility mode for the mapping file.
See Chapter 6 for more information on viewing and configuring datastores and managing RDMs using the VI Client.
VMware works with vendors of storage management software to ensure that their software functions correctly in environments that include VMware ESX. Some of these applications are:
SAN management software
Storage resource management (SRM) software
Storage array snapshot software
Replication software
Such software uses physical compatibility mode for RDMs so that the software can access SCSI devices directly.
Various management products are best run centrally (not on the ESX host), while others run well in the service console or in the virtual machines themselves. VMware does not certify or provide a compatibility matrix for these types of applications. To find out whether a SAN management application is supported in an ESX
environment, contact the SAN management software provider.
Limitations of RDM in VMware ESX
When planning to use RDM, consider the following:
RDM is not available for devices that do not support the export of serial numbers —RDM (in the current implementation) uses a SCSI serial number to identify the mapped device. Thus these devices (also known as block devices that connect directly to the cciss device driver or a tape device) cannot be used in RDMs.
RDM is available with VMFS-2 and VMFS-3 volumes only — RDM requires the VMFS-2 or VMFS-3 format. In VMware ESX 3, the VMFS-2 file system is read-only. You need to upgrade the file system to VMFS-3 to be able to use the files it stores.
RDM does not allow use of VMware snapshots in physical compatibility mode — The term snapshot here applies to the ESX host feature and not the snapshot feature in storage array data replication technologies. If you are using RDM in physical compatibility mode, you cannot use a snapshot with the disk.
Physical compatibility mode allows the virtual machine to manage its own snapshot or mirroring operations.
For more information on compatibility modes, see “Virtual and Physical
Compatibility Modes” on page 61. For the support of snapshots or similar data replication features inherent in storage arrays, contact the specific array vendor for support.
No partition mapping — RDM requires the mapped device to be a whole volume presented from a storage array. Mapping to a partition is not supported.
Using RDM to deploy LUNs — This can require many more LUNs than is used in the typical shared VMFS configuration. The maximum number of LUNs
supported by VMware ESX 3.x is 256.
Sharing Diagnostic Partitions
VMware ESX hosts collect debugging data in the form of a core dump, similar to most other operating systems. The location of this core dump can be specified as local storage, on a SAN volume, or on a dedicated partition. If your ESX host has a local disk, that disk is most appropriately used for the diagnostic partition, rather
VMware ESX hosts collect debugging data in the form of a core dump, similar to most other operating systems. The location of this core dump can be specified as local storage, on a SAN volume, or on a dedicated partition. If your ESX host has a local disk, that disk is most appropriately used for the diagnostic partition, rather