Chapter 3. General practices for VMware
3.6 Virtual Machine File System
Virtual Machine File System (VMFS) is a cluster file system optimized for virtual machines. A virtual machine consists of a small set of files that are stored in a virtual machine folder. VMFS is the default file system for physical SCSI disks and partitions. VMFS is supported on
allowing shared access to allow multiple ESXi hosts to concurrently read and write to the same storage. VMFS can expand dynamically, allowing an increase in the size of the VMFS without downtime. In vSphere 5.0, a new VMFS version was introduced, stepping up from VMFS3 to VMFS5. A VMFS datastore can span multiple LUNS (extents), but the
recommended implementation is to have a one-to-one relationship. When we refer to a datastore, we refer to a VMware container that is viewed on the VMware side, whereas if we refer to a logical unit number (LUN), we talk about the storage array volume:
VMFS5, by default, supports the use of hardware-assisted locking or atomic test and set (ATS) locking if supported by your storage array.
VMFS5 allows for reclaiming physical storage space on thin provisioned storage LUNs.
VMFS maximum numbers:
Volumes per host: 256 Hosts per volume: 64
Powered-on virtual machines per VMFS volume: 2048
Concurrent datastore operations:
vMotion operations per datastore: 128 Storage vMotion operations per datastore: 8 Storage vMotion operations per host: 2
Non-vMotion provisioning operations per host: 8
Table 3-2 shows the maximum parameters compared between VMFS3 and VMFS5.
Table 3-2 Maximum values for VMFS3 and VMFS5
Although VMFS3 allowed 64 TB datastores (volumes), it did not allow 64 TB LUNs. 2 TB multiplied by 32 maximum extents equals a 64 TB datastore. Now with VMFS5, a single LUN
Item VMFS 3 maximum VMFS 5 maximum
Volume size 64 TBa
a. For VMFS3 volumes with 1 MB block size, the maximum is 50 TB.
64 TBb
b. The actual maximum will depend on the RAID controller or maximum size of the LUN that is supported by the storage access driver (FC, iSCSI) that is being used. Contact your vendor to find the maximums.
Raw device mapping size (virtual compatibility)
2 TB minus 512 bytes 2 TB minus 512 bytes
Raw device mapping size (physical compatibility)
2 TB minus 512 bytesc
c. If the presented LUN is greater than 2 TB.
64 TB
Block size 8 MB 1 MBd
d. 1 MB is the default block size. Upgraded VMFS5 volumes will inherit the VMFS3 block size value.
File size (1 MB block size) 256 GB 2 TB minus 512 bytese
e. The maximum file size for an upgraded VMFS5 is 2 TB minus 512 bytes, irrespective of the file-system block size.
File size (2 MB block size) 512 GB File size (4 MB block size) 1 TB File size (8 MB block size) 2 TB
can actually be up to 64 TB (if your storage array supports it). However, this is the maximum, and larger LUNs will give you less performance than smaller LUNs. Finding the correct LUN size is finding the best compromise between performance and fitting a good number of VMs into the datastore.
Another consideration about large LUNs is that if you lose the large LUN for whatever reason, you will need to restore the data. Obviously, to restore a large LUN or its contents takes longer than a smaller LUN.
3.6.1 VMFS extents
Extents is the merge of multiple LUNs (extents) to form a VMFS datastore. A common recommendation is not to use extents at all. Now with larger supported LUNs with VMFS5 you can expand your LUNs to a size that was previously only supported by extents with VMFS3.
However, regular virtual machine virtual disks are still only supported to a size of 2 TB minus 512 bytes. Using Raw Device Mapping (RDM) in physical compatibility mode allows you now to use LUNs with up to 64 TB. When required, virtual machines with virtual disks larger than 2 TB minus 512 bytes RDM in physical compatibility mode is your only option.
3.6.2 Disk alignment
For maximum performance for storage I/O transactions, it is important that all storage layers are aligned. What this means is that if a partition is not aligned, I/O will cross track boundaries and cause additional I/O and result in a penalty on throughput and latency. VMware sets a starting block of 128 on VMFS3 and 2056 on VMFS5 when creating the partition using the vSphere Client/web client or the API in general, but not if you use the command line fdisk tool or vmkfstools.
Newer operating systems such as Windows 2008 and Windows Vista autoalign their partitions as well. But if you create a partition with a mechanism that is using XP or DOS to partition the virtual disk, it will not be automatically aligned. It is important to have the alignment correct when you deploy virtual machines. It is not a configuration setting that does the alignment, creating the partition results in the alignment.
When you have the guest partitions misaligned, you cannot easily fix this. In general, you have to create a new partition and copy the data from the misaligned partition to the new, aligned partition. This is difficult with the operating system (OS) partition since you cannot copy the data because it is being used. You would need to use a separate OS that mounts the disks and copies that data. This is usually a longer downtime. There are some tools that have automated this process under VMware, but it is still with some risks and is disruptive. Ensure that your templates are aligned, or if you are using another process to deploy virtual
machines, that this process does the alignment correctly.
The SVC, Storwize V7000, and V3700 recommended an extent size of 256 KB, which is a multiplier of 8 KB. The SVC, Storwize V7000, and V3700 offset is always 0. The VMFS3 with a starting block of 128 and VMFS5 with a starting block of 2048 are aligned. VMFS2 had, by default, its starting block as 63. If the starting block was 63, it would not be aligned. If you have never used VMFS2 and not used fdisk or vmkfstools, you should not expect to see your SAN VMFS datastores in an unaligned state.
more likely. Windows 2008, for example, sets the partition starting offset to 1024 KB (1,048,576 bytes) for disks larger than 4 GB.
In ESX versions prior to vSphere 5.0, the command for checking the starting block was
fdisk -lu, but is deprecated because it does not handle GUID Partition Table (GPT) partitions. It still works on VMFS3 or upgraded VMFS LUNs.
Example 3-3 shows an example of how to check the starting block of a LUN.
Example 3-3 Checking the LUN start block #first you want to find the disk names ls /vmfs/devices/disks/ eui.0020c24000106086 eui.0020c24000106086:1 eui.0020c24001106086 naa.600605b0025e8420ff0005a9567454af naa.600605b0025e8420ff0005a9567454af:1
#Then you can get the information on the disk using the disk name
More information about the partedUtil command can be found at the following website:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC &externalId=1036609
Example 3-4 shows how to check Windows basic disks for disk alignment.
Example 3-4 Check basic Windows disks
C:\>wmic partition get BlockSize, StartingOffset, Name, Index BlockSize Index Name StartingOffset 512 0 Disk #0, Partition #0 1048576
512 1 Disk #0, Partition #1 105906176
Partition 0 starting offset is 1,048,576 bytes or 1024 KB. Partition 1 starting offset is at 105,906,176 bytes which can be divided by 8, which means this partition is also aligned. For dynamic disks in Windows 2003, use the dmdiag.exe -v command, and in
Windows 2008, the diskdiag.exe -v command.
This KB article explains how to align a Windows 2003 partition:
http://technet.microsoft.com/en-us/library/aa995867.aspx
For more information about Windows alignment, see the following website:
http://download.microsoft.com/download/C/E/7/CE7DA506-CEDF-43DB-8179-D73DA13668C5/ DiskPartitionAlignment.docx
On Linux, you can check the starting offset with the fdisk -lu command. In the resulting output of this command if the starting offset is, for example, 2048, because that is divisible by 8 this means the disks are aligned.
More information about alignment can be found at the following website:
3.6.3 Virtual machine files
A virtual machine consists of multiple files that are stored in a folder called the
home folder
. The home folder has the same name as the virtual machine name. The virtual machine home folder is stored in the root directory of the VMFS datastore.NFS datastores do not use the VMFS file system, instead the regular NFS file system is used but the structure is the same and the file names are the same.
Table 3-3 lists all VM files and their purpose.
Table 3-3 Different virtual machine files