vMotion fails with the error: Migrate_SetFailure: Failed waiting for data. Error bad0004. Busy
vMotion fails with the error: Migrate_SetFailure: Failed
waiting for data. Error bad0004. Busy
Symptoms
Unable to vMotion the virtual machine
When you attempt to vMotion the virtual machine, the task begins, but fails at 10% In the vmware.log file of the virtual machine, you see messages similar to:
Dec 06 16:41:29.167: vmx| Migrate_SetFailure: Failed waiting for data. Error bad0004. Busy.
Dec 06 16:41:29.167: vmx| Migrate: cleaning up migration state.
Dec 06 16:41:29.167: vmx| MigrateSetState: Transitioning from state 11 to 0.
Dec 06 16:41:29.167: vmx| Msg_Post: Error Dec 06 16:41:29.167: vmx| [vob.vmotion.resolve.swap.all.files.failed] vMotion migration
[c0a8027c:1291653666313618] failed to find a valid swapfile. Dec 06 16:41:29.167: vmx| [msg.moduletable.powerOnFailed] Module Migrate power on failed.
Resolution
This issue may occur due to the slight difference in the type of swap files in ESX 4.1 and earlier ESX versions.
To resolve this issue, you must modify the swap type setting on both the source and target host.
To modify the setting:
1.
Launch the vSphere Client and log in to your vCenter Server.2.
Select the source ESX host and then click the Configuration tab.3.
Click Software > Advanced Settings > Migrate.4.
Under the Migrate options, locate the line containing Migrate.VMotionResolveSwapType. It will be set to the default value of 1.5.
Change the value to 0.6.
Click OK.7.
Repeat Steps 2 and 3 for the destination hostVMotion fails after a third-party security tool performs a
port scan of the ESX/ESXi hosts
Symptoms
VMotion fails after a third-party security tool (such as IBM Internet Security Systems) performs a port scan of the ESX or ESXi hosts.
cpu1:1086) Migrate: 2250 Error with migration listen socket, shutting down: I/O error.
A general system error occurred; timed out waiting for migration data
The vmkernel.log contains messages similar to:
May 1 21:20:45 ESXsrvr vmkernel: 11:22:29:39.358 cpu13:1915)World: vm 1923: 901: Starting world migSendHelper-1916 with flags 1
May 1 21:20:45 ESXsrvr vmkernel: 11:22:29:39.358 cpu13:1915)World: vm 1924: 901: Starting world migRecvHelper-1916 with flags 1
May 1 21:20:45 ESXsrvr vmkernel: 11:22:29:39.364 cpu1:1086)MigrateNet: vm 1086: 854: Accepted connection from <xxx.xxx.xxx.xxx>
May 1 21:21:05 ESXsrvr vmkernel: 11:22:29:59.642 cpu12:1916)Migrate: 7309: 1241227232551280: Another pre-copy iteration needed with 30737 modified pages (last = -1)
May 1 21:21:07 ESXsrvr vmkernel: 11:22:30:02.092 cpu10:1916)Migrate: 7309: 1241227232551280: Another pre-copy iteration needed with 17783 modified pages (last = 30737)
May 1 21:21:09 ESXsrvr vmkernel: 11:22:30:03.938 cpu9:1916)Migrate: 7304: 1241227232551280: Stopping pre-copy: Not enough forward progress (Modified pages 17783 vs. 22217) - stopping pre-copy
May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)MigrateNet: vm 1086: 854: Accepted connection from <xxx.xxx.xxx.xxx>
May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)WARNING: MigrateNet: vm 1086: 865: Couldn't set nodelay option on socket May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)ALERT: Migrate: 2250: Error with migration listen socket, shutting down: I/O error
May 1 23:32:52 ESXsrvr vmkernel: 12:00:41:45.964 cpu1:1086)Migrate: 2312: Exit requested...
The Hostd.log contains messages similar:
[2009-05-01 23:31:41.190 'App' 22911920 error] SSLStreamImpl::BIORead ( A6A2D10) failed: Connection reset by peer
[2009-05-01 23:31:41.190 'App' 22911920 error]
SSLStreamImpl::DoServerHandshake ( A6A2D10) SSL_accept failed with BIO Error
[2009-05-01 23:31:41.190 'Proxysvc' 22911920 warning] SSL Handshake on client connection failed for peer , error=SSL Exception: BIO Error [2009-05-01 23:32:22.994 'App' 21588912 error]
SSLStreamImpl::DoServerHandshake ( A6B9AA8) SSL_accept failed with Unexpected EOF
[2009-05-01 23:32:22.994 'Proxysvc' 21588912 warning] SSL Handshake on client connection failed for peer <xxx.xxx.xxx.xxx>, error=SSL
Exception: Unexpected EOF
[2009-05-01 23:32:52.085 'ha-eventmgr' 130374576 info] Event 271 : Issue detected on ESXsrvr.mydomain.com in ha-datacenter: Migrate: 2250: Error with migration listen socket, shutting down: I/O error
(12:00:41:45.964 cpu1:1086)
Resolution
This issue is resolved in ESX/ESXi 4.0 Update 2. For more information, see vSphere 4 download page. This issue is resolved in ESX and ESXi 3.5. For more details, see KB 1026126 (ESX) at
This issue might occur if a network port-scanner-process attempts to engage VMotion migration port (8000) on the ESX or ESXi host. On ESX/ESXi 3.5.x, you must disable and then re-enable VMotion on the ESX/ESXi host.
The workaround provided to resolve the issue was:
To disable and re-enable VMotion:
1.
Select the ESX/ESXi host in the VI Client.2.
Select Configuration > Advanced Settings > Migrate > Migrate.enabled.3.
Change the value of Migrate.enabled setting from 1 to 0.4.
Click OK.5.
Select Configuration > Advanced Settings > Migrate > Migrate.enabled.6.
Change the Migrate.enabled setting from 0 to 1.7.
Click OK.To prevent VMotion from failing, you must exclude port 8000 in your port scanning software. Note: A VMotion network should never be accessible by untrusted sources. You must isolate the management network as described in the VMware Infrastructure 3 Security Hardening Guide
Failed vMotion, host busy, White Spaces and Directory Names.
So recently I came across and issue where a "space" at the end of a VM's directory causes
it to fail to vMotion.
The error is typically cryptic stating that the host is too busy.
If you try to browse the directory using the vsphere client it looks empty. You also can't
rename the directory using the vsphere.
For us the problem came about when we were creating VM's. We typically created quite
long machine names in vsphere. "machine name - OS - description.
Ask you can see we tend to add Tue description of the vm to the name. Now what
vSphere does is take the first 32 characters and use that as the directory name. If the
32nd character is a "space"... well you get the picture.
I solved the issue by using the vMA (this can also be done using the ESX console). I use
the vMA a fair amount and have all of our nfs shares mounted in /mnt. I would
Anyway the fix:
1. Shutdown and remove the VM from inventory.( Don't delete from disk!!!).
2. Using the vMA browse to the location of the folder. Rename through folder using the
mv command. mv "folder with space " "folder with space".
3. Renregister Tue vm with sphere or the ESX server.
If anybody else has had this problem and solved it drop me a line.
vMotion fails with general system error
Posted by Monique on July 11, 2011Not long ago I wanted to vMotion a virtual machine, it failed with a general system error at 80%. First thing that crossed my mind was, it must have a CD/DVD device attached. Wich is usually the reason for failing vMotion. So I editted the virtual machine settings from “client device” to “host device” and back. Checked to make sure there were no VMware tools upgrades that got stuck and tried it again. Still, I got the same error. I vMotionned some other machine off the same host, just to make sure vMotion was still viable. That worked, so vMotion on the cluster was still functional. It had to be something else related to this particular virtual machine.
I checked all it’s settings, but didn’t notice anything unusual. I then checked the vmx log files of the virtual machine and the hostd log files, to see if I could find anything out of the ordinary in the logs. Among other things I found these entrances in the logs, wich struck me as odd:
Vmon_MigrateWriteGotData: Limit Exceeded MigrateWrite failed
Could be an instance of a bug 49917
When I Googled for “migration write data limit exceeded” I found this VMware KB, VMware KB1011971. It states that this issue may occur on ESX(i) clusters earlier than 4.0 update 1, when the Video RAM of a virtual machine is greater than 30 MB. Since this machine was running on one of our (soon to be extinct) ESX 3.5 clusters, I gathered this might be it. I checked the virtual machine settings again. This time for the Video RAM settings and it turned out the machine had 32 MB of Video RAM. After checking with the client, I brought the VM down, editted it’s Video RAM and brought it back online. This time it vMotionned without any problem.
Once again, one of life’s little annoyances resolved
---
Creating a snapshot for a virtual machine fails with the
error: File is larger than maximum file size supported
Symptoms
In the vSphere Client, you see the error:
File is larger than the maximum size supported by datastore
In the
hostdlog file, you see the error:
Snapshot guest failed: The file is too big for the filesystem.
In the
vmware.logfile of the virtual machine, you see an error similar to:
vmx| FILE: File_VMFSSupportsFileSize: Requested file size
(554051831808) larger than maximum supported filesystem file size (274877906944)
vmx| DiskLibCreateCustom: if your disk is on VMFS, you may consider increasing the block size.
vmx| DISKLIB-LIB : Failed to create link: The destination file system does not support large files (12)
vmx| SNAPSHOT: BranchDisk: Failed to create child disk
'/vmfs/volumes/uuid/vmname/vmname-000001.vmdk' : The destination file system does not support large files (12)
vmx| SNAPSHOT: SnapshotBranch failed: The destination file system does not support large files (5).
vmx| [msg.checkpoint.save.fail2.std3] Error encountered while saving snapshot.
vmx| The destination file system does not support large files.
Cause
This failure occurs when the snapshot file at its maximum size would be unable to fit into a
datastore. Starting with version 4.0, ESX and ESXi will compare the maximum size of a
snapshot redolog file with the maximum size of files on the datastore. The redolog file may not
work correctly once it reaches the maximum size of the datastore. If the file could grow beyond
the maximum size, ESX cancels the Create Snapshot operation and displays this error instead.
Note: This check does not occur on ESX/ESXi 3.5 and earlier. On these versions, a snapshot will be
created even if there is insufficient space to store a full-size redolog.
Resolution
Maximum file size
Compare the base disk size of the virtual machine against the block size of the datastore which
contains the working directory of the virtual machine. By default, the working directory contains
the virtual machine's .vmx configuration file. The maximum file size differs among versions of
ESX/ESXi, and among version of VMFS.
If you experience this error even after confirming that the snapshot files can fit on the datastore,
proceed to the Calculating the overhead required by snapshot files section in this article.
Note: A virtual machine on NFS or VMFS has a maximum virtual disk size of 2048 GB - 512
Bytes, the same as the maximum in each of these tables.
ESX/ESXi 4.0 with VMFS3
On ESX/ESXi 4.0, the maximum file size corresponds to the block size of the VMFS3 datastore:
Block Size
Maximum File Size
1 MB
256 GB - 512 Bytes
2 MB
512 GB - 512 Bytes
4 MB
1024 GB - 512 Bytes
8 MB
2048 GB - 512 Bytes
ESX/ESXi 4.1 and ESXi 5.0 with VMFS3, or VMFS3 upgraded to VMFS5
On ESX/ESXi 4.1 and ESXi 5.0 using a VMFS3 datastore, or an ESXi 5.0 host using a VMFS5 datastore
upgraded from VMFS3, the maximum file size corresponds to the block size of the VMFS datastore:
Block Size
Maximum File Size
1 MB
256 GB
2 MB
512 GB
4 MB
1024 GB
ESXi 5.0 with VMFS5
On ESXi 5.0 and newly formatted VMFS5, a standard 1 MB block size is available:
Block Size
Maximum File Size
1 MB
2048 GB - 512 Byte
Moving files to accommodate space requirements
To resolve this issue you can either change the location of the virtual machine configuration files
or change the
workingDirto a datastore with enough block size.
The
workingDiris the location where the snapshots are created, By default,
workingDircontains the virtual machine's
.vmxconfiguration file. To change the
workingDirdirectory to a
datastore with enough block size, see
Creating snapshots in a different location than default
virtual machine directory (1002929)
.
To move the virtual machine's disks and/or configuration files you can use Storage vMotion or
Cold migration with relocation of files. For more information, see:
vSphere 5.0: Migrating Virtual Machines section of the vCenter Server and Host Management
guide.
vSphere 4.1: Migrating Virtual Machines section of the vSphere Datacenter Administration
guide.
vSphere 4.0: Migrating Virtual Machines section of the vSphere Basic System Administration
guide.
If the virtual machine already has snapshots, some procedures may not work or may try to create a
snapshot. The following table lists the requirement for the various procedures:
Procedure
Requirements
have snapshots on ESXi 5.0 and later.
Cold migration with
relocation of files
Virtual machine may have snapshots. The source and destination hosts must be
running ESX/ESXi 3.5 or later.
Change
workingDirVirtual machine may have snapshots. When new snapshots are created, new
redologs will be placed in the
workingDirdirectory.
Hot clone
Virtual machine may have snapshots, but the snapshot hierarchy must be fewer
than 31 snapshots deep. Hot clone of a virtual machine creates a snapshot on the
source at the beginning of the process, and deletes the snapshot when complete.
Cold clone
Virtual machine may have snapshots. Cloning the virtual machine creates a new
virtual machine with the same content as the original virtual machine, but without
snapshots.
vMotion to ESX/ESXi
3.5
Virtual machine may have snapshots. Virtual machine must use hardware version
4. ESX/ESXi 3.5 does not perform the check described here and allows creation of
snapshots on virtual machines.
Calculating the overhead required by snapshot files
The failure depends on the size of the virtual disk. All virtual machines having disks with a maximum
supported size by VMFS may experience this error. Overhead for the snapshot is roughly about 2GB for
a disk size of 256GB. If snapshots are to be used, consider the overhead while deciding the size of the
disks.
Maximum VMDK size
Maximum Overhead
Maximum size less overhead
256 GB - 512 B
~ 2 GB
254 GB
512 GB - 512 B
~ 4 GB
508 GB
1024 GB - 512 B
~ 8 GB
1016 GB
Note: VMware recommends that you to create virtual disks of size less than the maximum minus
the overhead, to enable use of features like Snapshot, Clone, and Independent-nonpersistent
disks.
Additional Information
When performing a storage vMotion, you may encounter the error:
Moving a virtual machine that has snapshots is not supported when the virtual machine has disks placed outside of its home datastore.
For more information on the maximum file size per VMFS block size, see the Configuration Maximums
document for your version of ESX/ESXi.
Tags
cannot-take-snapshot snapshot snapshot-fails snapshots take-snapshot create-datastore
---
Committing snapshots when there are no snapshot entries in
the snapshot manager
Details
The snapshot manager shows no snapshots but there are delta files present. One or more sets of -00000X.vmdk and -00000X-delta.vmdk files are in the directory with the virtual disk. The .vmx file points to one of the -00000X.vmdk files, usually the highest numbered file, indicating that the snapshot file is in use.
Note: This article applies only to ESX/ESXi 3.x and 4.x. For information about ESXi 5.0, see
Consolidating snapshots in vSphere 5 (2003638).
Solution
Most of the snapshot related issues have improved with changes to the Delete All snapshot process in the patch releases for ESX/ESXi 3.5 and 4.0. For more information, see:
VMware ESX 3.5, Patch ESX350-201006401-SG: Updates VMkernel, VMware Tools, hostd, VMX, VMnix (1022899)
VMware ESXi 3.5, Patch ESXe350-201006401-I-SG: Updates firmware (1020052)
VMware ESX 4.0, Patch ESX400-201006201-UG: Updates the VMware ESX 4.0 Core and CIM components (1017721)
VMware ESXi 4.0, Patch ESXi400-201006201-UG: Updates Firmware(1017739)
Resolution
Confirm that the virtual machine is not pointing to the base disk.Open the virtual machine configuration file ( .vmx) or edit the settings of the virtual machine and see if any of the virtual disks are using a -00000X.vmdk file. If no disks are using --00000X.vmdk, this virtual machine is not using any of these files. Although unlikely, it is possible that another virtual machine is storing its snapshots in this directory. Check the other virtual machines. If none of them refer to these files, they can be safely erased.
Typically, the file is in use by the virtual machine. When a snapshot is deleted, any additional files in the hierarchy that are not identified by the snapshot manager are included in the commit process. Creating a new snapshot and deleting it clears the entire hierarchy.This means that all snapshot files on the virtual machine are committed, then deleted.
A bit of free space is required to create the new snapshots. If the virtual machine needs to remain running, more space must be allowed for, as the new snapshot grows (accepting new changes to the virtual disks) as the older snapshots commit.
To commit all snapshots by using the Virtual Infrastructure Client:
1.
Take a Snapshot. For more information, see the Take a Snapshotsection of the VMware vSphere Online Library.2.
Delete all Snapshots. For more information, see the Delete all Snapshotssection of the VMware vSphere Online Library.On an ESXi host, to commit all snapshots using the command line:
1.
Log in to the ESXi host as root via the console or an SSH session. For more information about SSH, see see Tech Support Mode for Emergency Support (1003677) or Using Tech Support Mode in ESXi 4.1 (1017910).Note: The following commands can also be executed remotely using the vSphere Command Line for both ESX and ESXi hosts. For more information, see vSphere Command Line Interface documentation.
2.
Run this command to get a list of virtual machines and the VMID for each virtual machine: vim-cmd vmsvc/getallvmsOutput similar to the following is displayed:
Vmid Name File Guest OS Version Annotation
1 vm1 [datastore1] vm1/vm1.vmx windows7Server64Guest vmx-08 3 testvm [iscsi1] testvm/testvm.vmx winNetDatacenterGuest vmx-08 Make a note of the VMID for the specific virtual machine.
3.
To verify if the snapshot exists, run this command and check the Snapshot Name, Snapshot Created On, and Snapshot State:vim-cmd vmsvc/snapshot.get [VMID] You see an output similar to:
Get Snapshot: |-ROOT
--Snapshot Name : Test --Snapshot Desciption :
--Snapshot Created On : 7/27/2011 13:49:55 --Snapshot State : powered on
4.
Run this command to create a new snapshot:vim-cmd vmsvc/snapshot.create [VmId] [snapshotName] [snapshotDescription] [includeMemory] [quiesced] For example, to create a snapshot on VM testvm:
vim-cmd vmsvc/snapshot.create 3 snapshot1 snapshot 0 0
Note: You can use any name you like. The name appears in the snapshot manager.
5.
Run this command to remove all snapshots:vim-cmd vmsvc/snapshot.removeall [VMID]
On an ESX host, to commit all snapshots by using the command line:
1.
Log in to the ESX host as root via the console or an SSH session. For more information about SSH, see Unable to connect to an ESX host using Secure Shell (SSH) (1003807).Note: The following commands can also be executed remotely using the vSphere Command Line for both ESX and ESXi hosts. For more information, see vSphere Command Line Interface documentation.
2.
Type vmware-cmd -l and press Enter. The output appears similar to:/vmfs/volumes/UUID/VMNAME/VMNAME.vmx
3.
Type vmware-cmd /vmfs/volumes/UUID/VMNAME/VMNAME.vmx hassnapshot and press Enter to confirm that there is a snapshot. If the output displays a value of 1, a snapshot is present. If the output displays a value of 0, there is no snapshot present.4.
Type vmware-cmd /vmfs/volumes/UUID/VMNAME/VMNAME.vmx createsnapshot <name> <description> <quiesce> <memory> and press Enter to create a new snapshot. For example, the command vmware-cmd /vmfs/volumes/UUID/VMNAME/VMNAME.vmx createsnapshot "test" "" 0 0 makes a snapshot without memory, quiescing, or a description called test .information about the syntax of the vmware-cmd command, see vSphere Command Line Interface documentation.
5.
Type vmware-cmd /vmfs/volumes/UUID/VMNAME/VMNAME.vmx removesnapshots and press Enter to remove the snapshot.Additional Information
The remove snapshot process can take a long time to complete if the snapshots are large. A file . vmsdmay interfere with the creation of the snapshot in step 4 if the memory snapshot was
left behind. For more information, seeUnderstanding virtual machine snapshots in VMware ESX (1015180)
.
If you entered the removesnapshots command via an SSH tool (such as PuTTY or secureCRT) the SSH session must be left open. Closing the SSH program aborts the process. If leaving an SSH session open for an extended time is unacceptable, run the command from the physical console.
The commit process has no progress that you can follow. As long as the date on the files continues to update, the process is working. Also, if the virtual machine is off, you can use the file * command to see if any files are in use. If any of the files return the error message can't read `filename' (Device or resource busy)they are locked by the VMkernel. The commit process is actively occurring to those files.
When the commit has completed successfully, there are no -00000X.vmdk or -00000X-delta.vmdk files left unless they were not part of the snapshot tree. These files can be deleted. To confirm the commit succeeded, view the .vmx file and verify that virtual disks are now pointing to a base disk (-flat.vmdk ).
If the .vmx contains a disk that is still pointing to a snapshot file, the commit process failed. If the attempt was made with the virtual machine running, plan an outage and try again with the virtual machine off. If that does not work, the virtual disk must be cloned using vmkfstools -i. The source file name is the current active -00000X.vmdk as identified in the .vmx file. When the clone is complete, point the virtual machine to use the newly cloned disk. The original base disk and snapshot tree can be deleted. For more information on consolidating a snapshot from the command line using cloning, see Consolidating
Snapshots (1007849).
As an alternative to using the command line, you can use VMware Infrastructure (VI) Client to perform the clone operation on the virtual machine. The cloned virtual machine retains the content of the associated snapshot disks at the time of cloning. When the virtual machine has been cloned successfully, test the operation of the resulting cloned virtual machine and decommission the previous virtual machine. For more information on cloning a virtual machine using VI Client, see Cloning Virtual Machines.
Note
:
Cloning using VI Client requires VirtualCenter and all applicable licenses. Cloning a virtual machine using VI Client does not allow you to select individual virtual disks connected to the virtual machine. Relative to using the vmkfstools command to perform the same operation on a single virtual disk, consolidating snapshots using VI Client clones all disks and may require more disk space.Workaround
If you are still not able to consolidate the snapshots and if a restore from backup is not possible, you can use VMware vCenter Converter Standalone to convert the virtual machine into a new virtual machine:
1.
Review the Release Notes.2.
Check the System Requirements as outlined in the Converter Standalone User's Guide.3.
Within the virtual machine, stop all I/O intensive processes.4.
Install VMware vCenter Converter Standalone into the virtual machine.5.
Start VMware vCenter Converter Standalone.6.
In the source dialog select Physical Machine and follow the instructions on the screen. For a detailed description review the Converter Standalone User's Guide.7.
Once the conversation finished successfully, you should test the virtual machine created:8.
a.
Check the virtual machine configuration by right clicking on the virtual machine and click Edit Settings.b.
Make sure that the network cards are not connected.c.
Power on the virtual machine and verify that everything is proper.9.
If everything is proper, power off the virtual machine with the snapshots.10.
In the new virtual machine, set the network cards to Connected at power on.Note: This works because the Converter Standalone does not know the virtual disk structure. It sees only that what the operating system sees like a physical machine. You should remember that the virtual machine conversion takes a long time because the conversion is done over the network.
See also
Consolidating snapshots (1007849)
Determining if there are leftover delta files or snapshots that VMware vSphere or Infrastructure Client cannot detect (1005049)
For translated versions of this article, see:
Español: Cómo hacer Commit a snapshots cuando estos no aparecen en el snapshot manager (1000636)
ESXi 4.1 virtual machines in an invalid state fail to power on
with the error: FoundryVMDirectlyOpenSocketToVMX:
Failed to create socket pair
Symptoms
When using VMware ESXi 4.1, you may experience one or more of these symptoms:
Unable to connect to the MKS
Unable to power on virtual machines
Powering on virtual machines reports some virtual machines in an invalid state
Power state of the virtual machine is reported incorrectly in the vCenter Server/ESX inventory
If you restart hostd, virtual machines that were previously not in an invalid state may appear as
When a virtual machine is in an invalid state, you see these errors:
vmx| VmdbPipeStreamsOvlError: write failed, draining reads vmx| VmdbPipeStreamsOvlError: Couldn't initiate write vmx| Redirecting stdin/stdout/stderr to /dev/null.
vmx| SOCKET 1 (91) send error 12: Cannot allocate memory
vmx| Vix: [177936 mainDispatch.c:2472]: VMAutomation: Connection Error (1) on connection 0.
vmx| SOCKET 1 (91) send error 12: Cannot allocate memory -> not able to create socket,no mem available.
mks| SOCKET 2 (92) recv error 104: Connection reset by peer mks| SOCKET 2 (92) destroying VNC backend on socket error: 1
The messages log contains entries similar to:
o sfcb-CIMXML-Processor[9857708]: SendMsg sending to 7 9857708-9 Bad file descriptor
sfcb-CIMXML-Processor[9857708]: spSendMsg sending to 7 9857708-9 Bad file descriptor
sfcb-CIMXML-Processor[9857708]: --- spSendReq/spSendMsg failed to send on 7 (-1)
root: sfcbd-watchdog:Restarting SFCB! Log a bug!!! root: sfcbd-watchdog:stopping sfcbd
root: sfcbd Stopping sfcbd
root: sfcbd-watchdog:starting sfcbd root: sfcbd Starting sfcbd
sfcb-sfcb[9849840]: --- Using /etc/sfcb/sfcb.cfg
o FoundryVMDirectlyOpenSocketToVMX: Failed to create socket pair.
The hostd logs may report this error when you try to power on the virtual machine:
Cannot connect to
/var/run/vmware/root_0/1299674323658606_59831734/testAutomation-fd: File not found
vMotion fails intermittently at a random percentage
Connecting to the remote console via the vSphere Client fails with the error:
Unable to connect to the MKS: There is no VMware process running for config file
Unable to retrieve any files from the datastore via Datastore Browser
The vm-support diagnostic information gathering utility is unresponsive
The Health Status tab may not load correctly
Cause
This issue occurs due to exhaustion of VMkernel socket resources on ESXi hosts. It occurs more
frequently on hosts where OEM CIM providers have been installed, which may be caused by additional
CIM Providers loaded under sfcbd. To ensure the stability of the OEM CIM providers, ensure that they
are updated to the latest release.
Note: This issue only occurs on ESXi. It does not occur on ESX. On ESX, sfcbd runs as a process in the
service console rather than from a world under the VMkernel.
Resolution
Resolution
This issue is resolved by VMware ESXi 4.1 Patch ESXi410-201107401-BG. For more information, see
VMware ESXi 4.1 Patch ESXi410-201107401-BG: Updates Firmware (2000609).
Note: A system which is found to be in a bad state must be rebooted prior to applying the patch.
Workaround
To work around this issue, stop the sfcbd hardware monitoring agent on the ESXi host. When sfcbd is
disabled, hardware status information for the ESXi host will be unavailable.
To stop sfcbd:
1. Log in to the VMware ESXi host as the root user. For more information, see Using Tech Support
Mode in ESXi 4.1 (1017910).
2. Run the command:
/etc/init.d/sfcbd-watchdog stop
3. To make this change persistent on reboot, run these commands:
chkconfig sfcbd-watchdog offchkconfig sfcbd off
If hardware monitoring is an environmental requirement, you can extend the amount of time before the
issue re-occurs by changing the configuration of sfcbd:
1. From the ESXi shell, edit the
/etc/sfcb/sfcb.cfgfile using a text editor.
2. Search for the entry
provProcs: 16, and change the value from
16to
12.
3. Restart
sfcbdfor the changes to take effect using the command:
/etc/init.d/sfcbd-watchdog restart
If the symptoms persist after applying this patch or workaround, collect logs from the environment and
contact VMware Support. For more information, see:
Filing a Support Request (1021619)
How to Submit a Support Request
Collecting diagnostic information for VMware ESXi Server (1010705)
Unable to create snapshot: Operation failed because file
already exists or Cannot complete the operation because the
file or folder [DatastoreName] VMname/VMname.vmx
already exists
Symptoms
While creating a virtual machine snapshot from the VI or vSphere Client, or from the command
line, you may receive any of these errors depending on your product version:
ESX/ESXi 3.5 or VirtualCenter 2.5 VI Client:
Operation failed because file already exists
ESX/ESXi 3.5 Service Console
VMControl error -3: Invalid arguments
Example:
# vmware-cmd VMname.vmx createsnapshot name desc 0 0
VMControl error -3: Invalid arguments
ESX/ESXi/vCenter Server 4.0 vSphere Client
Cannot complete the operation because the file or folder
[DatastoreName] VMname/VMname.vmx already exists
ESX/ESXi/vCenter Server 4.1 vSphere Client
Cannot complete the operation because the file or folder
<unspecified filename> already exists
ESX/ESXi 4.0 Service Console
# vmware-cmd
/vmfs/volumes/47ecbe04-acfca740-564c-001a4be960e0/VMname/VMname.vmx createsnapshot name desc 0 0
Task reported error: (vim.fault.FileAlreadyExists) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'Cannot complete the operation because the file or
folder [DatastoreName] VMname/VMname.vmx already exists',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
file = '[DatastoreName] VMname/VMname.vmx'
}
createsnapshot(name, desc, 0, 0) = 0
ESX/ESXi 4.1 Service Console
# vmware-cmd
/vmfs/volumes/47ecbe04-acfca740-564c-001a4be960e0/VMname/VMname.vmx createsnapshot name desc 0 0
Task reported error: (vim.fault.FileAlreadyExists) {
dynamicType = <unset>,
dynamicProperty = (vmodl.DynamicProperty) [],
msg = 'Cannot complete the operation because the file or
folder <unspecified filename> already exists',
faultCause = <unset>,
faultMessage = (vmodl.LocalizableMessage) [],
file = '[DatastoreName] VMname/VMname.vmx'
}
createsnapshot(name, desc, 0, 0) = 0
ESX/ESXi 3.5 hostd.log file contains these entries:
'BaseLibs' 51723184 info] SNAPSHOT: SnapshotBranchDisk
failed: The file already exists (39).
'BaseLibs' 51723184 info] SNAPSHOT: SnapshotBranch failed:
The file already exists (5).
'BaseLibs' 51723184 info] DISKLIB-LINK : File
'/vmfs/volumes/4c0f81ad-b85976a8-21d1-001cc493ce16/VMname/VMname-000002-delta.vmdk' already
exists.
'BaseLibs' 51723184 info] DISKLIB-LIB : Failed to create
link: The file already exists (39)
'BaseLibs' 51723184 info] SNAPSHOT: SnapshotBranchDisk
failed: The file already exists (39).
'BaseLibs' 51723184 info] SNAPSHOT: SnapshotBranch failed:
The file already exists (5).
ESX/ESXi 4.x hostd.log file contains these entries:
already exists (5).
info 'Vmsvc'] Failed to do Snapshot op: Error: (12) The
file already exists
info 'DiskLib'] DISKLIB-LINK : File
'/vmfs/volumes/47ecbe04-acfca740-564c-001a4be960e0/VMname/VMname-000004-delta.vmdk' already
exists.
info 'DiskLib'] DISKLIB-LIB : Failed to create link: The
file already exists (39)
info 'DiskLib'] DISKLIB-LIB : Failed to create link: The
file already exists (39)
info 'Libs'] SNAPSHOT: SnapshotBranch failed: The file
already exists (5).
info 'Vmsvc'] Failed to do Snapshot op: Error: (12) The
file already exists
Resolution
This issue occurs when VMware ESX or ESXi tried to create a new
vDisknamedelta.vmdk redolog file, but the generated filename already exists. This can occur if the
-delta.vmdk redolog file does not have an associated descriptor file.
Every virtual disk is composed of two files, a "flat" or "delta" file containing the actual data and
a descriptor file containing geometry and topology information. For example,
vDiskname.vmdk and vDiskname-flat.vmdk or vDiskname-delta.vmdk. For
more information, see
Understanding virtual machine snapshots in VMware ESX (1015180)
.
When creating a snapshot, the host uses the lowest file number that is not in use by the chain of
snapshots and does not exist on the directory. The host calculates this by checking through the
descriptor files.
When the host tries to create the snapshot, it first creates the new descriptor file, then creates the
associated -delta.vmdk file. If the host detects that a -delta.vmdk file with that name
already exists the snapshot operation fails and the virtual machine reverts to its previous state.
To resolve this issue, locate and manually remove the -delta.vmdk file that does not have an
associated descriptor file.
1. Connect to the ESX or ESXi host using the vSphere Client.
2. Locate the affected virtual machine in the Inventory.
3. Determine which datastore(s) the virtual machine is stored on.
4. Open the Datastore Browser.
5. Navigate to the working directory containing the Virtual Machine configuration files. For
more information, see
Creating snapshots in a different location than default virtual
machine directory (1002929)
.
6. Confirm that remnant redolog delta.vmdk files from previous snapshots exist. For
example:
o
VM is currently pointed to vmname-000001.vmdk
o
vmname-000001.vmdk 's parent is the VM base disk, vmname.vmdk
oA file named vmname-000002-delta.vmdk exists in the VM directory.
oThe .vmx file does not reference the file named
vmname-000002-delta.vmdk.
o
The .vmsd file does not reference the file named
vmname-000002-delta.vmdk.
7. Rename or move the remnant file aside. For example, move the file
vmname-000002-delta.vmdk into a backup directory.
8. Retry the Create Snapshot operation, it should complete successfully.
Note: If the memory state is captured for a snapshot, a .vmsn file is created before the new
.vmdk files. If the snapshot operation fails unexpectedly, the process halts and the .vmsn file is
left behind. The .vmsn files can be safely removed if the virtual machine has no snapshots. If
there are any snapshots, look in the .vmsd file to identify the which .vmsn files are referenced
by the VM and remove the rest.
For more information, see
Creating a virtual machine snapshot fails with the error: File already
exists (1002190)
.
Understanding virtual machine snapshots in VMware ESX
Symptoms
This article may be helpful when you encounter these issues:
Virtual machines are not responding or cannot start due to broken parent and child virtual disk
dependencies.
Virtual machines are not responding or do not start due to redo logs residing on datastores that
do not have free space.
Snapshot creation takes too long when specifying the memory snapshot option.
Snapshot delete or remove operations result in the vSphere or VMware Infrastructure (VI) Client
timing out.
Purpose
This article provides information about virtual machine snapshots.
Resolution
What is a snapshot?
A snapshot preserves the state and data of a virtual machine at a specific point in time.
The state includes the virtual machine’s power state (for example, powered-on, powered-off,
suspended).
The data includes all of the files that make up the virtual machine. This includes disks, memory,
and other devices, such as virtual network interface cards.
A virtual machine provides several operations for creating and managing snapshots and snapshot
chains. These operations let you create snapshots, revert to any snapshot in the chain, and
remove snapshots. You can create extensive snapshot trees.
In VI3 and vSphere 4.x, the virtual machine snapshot delete operation combines the
consolidation of the data and the deletion of the file. This caused issues when the snapshot files
are removed from the Snapshot Manager, but the consolidation failed. This left the VM still
running on snapshots, and the user may not notice until the datastore is full.
In vSphere 4.x, an alarm can be created to indicate if a VM was running in snapshot mode. For
more information, see
Configuring VMware vCenter Server to send alarms when virtual
machines are running from snapshots (1018029)
.
In vSphere 5.0, enhancements have been made to the snapshot removal. In vSphere 5.0, you are
informed via the UI if the consolidation part of a RemoveSnapshot or RemoveAllSnapshots
operation has failed. A new option, Consolidate, is available via the Snapshot menu to restart the
consolidation.
When creating a snapshot, there are several options you can specify:
Name: This is used to identify the snapshot.
Description: This is used to describe the snapshot.
Memory: If the
<memory>flag is 1 or true, a dump of the internal state of the virtual machine is
included in the snapshot. Memory snapshots take longer to create.
Quiesce:If the
<quiesce>flag is 1 or true, and the virtual machine is powered on when the
snapshot is taken, VMware Tools is used to quiesce the file system in the virtual machine.
Quiescing a file system is a process of bringing the on-disk data of a physical or virtual computer
into a state suitable for backups. This process might include such operations as flushing dirty
buffers from the operating systems in-memory cache to disk, or other higher-level
application-specific tasks.
Note: Quiescing indicates pausing or altering the state of running processes on a computer,
particularly those that might modify information stored on disk during a backup, to guarantee a
consistent and usable backup.
When a snapshot is created, it is comprised of these files:
<vm>-<number>.vmdk
and
<vm>-<number>-delta.vmdkA collection of
.vmdkand
-delta.vmdkfiles for each virtual disk is connected to the virtual
machine at the time of the snapshot. These files can be referred to as child disks, redo logs, or
delta links. These child disks can later be considered parent disks for future child disks. From the
original parent disk, each child constitutes a redo log pointing back from the present state of the
virtual disk, one step at a time, to the original.
Note: The
<number>value may not be consistent across all child disks from the same snapshot.
The file names are chosen based on filename availability.
<vm>.vmsd
The
.vmsdfile is a database of the virtual machine's snapshot information and the primary
source of information for the snapshot manager. The file contains line entries which define the
relationships between snapshots as well as the child disks for each snapshot.
<vm>Snapshot<number>.vmsn
These files are the memory state at the time of the snapshot.
What products use the snapshot feature?
In addition to being able to use snapshot manager to create snapshots, snapshots are used by
many VMware and third-party products and features. Some VMware products that use snapshots
extensively are:
VMware Data Recovery
VMware Lab Manager
VMware vCenter and the VMware Infrastructure Client (Snapshot Manager, Storage vMotion)
Note: This is not an exhaustive list.
How do snapshots work?
Our VMware API allows VMware and third-party products to perform operations with virtual
machines and their snapshots. This is a list of common operations that can be performed on
virtual machines and snapshots using our API:
CreateSnapshot: Creates a new snapshot of a virtual machine. As a side effect, this updates the
current snapshot.
RemoveSnapshot: Removes a snapshot and deletes any associated storage.
RemoveAllSnapshots: Remove all snapshots associated with a virtual machine. If a virtual
machine does not have any snapshots, then this operation simply returns successfully.
RevertToSnapshot: Changes the execution state of a virtual machine to the state of this
snapshot.
(vSphere 5.0 only) Consolidate: Merges the hierarchy of redo logs.
This is a high-level overview of how to create, remove, or revert snapshot requests that are
processed within the VMware environment:
1. A request to create, remove, or revert a snapshot for a virtual machine is sent from the client to
the server using the VMware API.
2. The request is forwarded to the VMware ESX host that is currently hosting the virtual machine in
question.
Note: This only occurs if the original request was sent to a different server, such as vCenter,
which is managing the ESX host.
3. If the snapshot includes the memory option, the ESX host writes the memory of the virtual
machine to disk.
Note: The length of time the ESX host takes to write the memory onto the disk is relative to the
amount of memory the virtual machine is configured to use.
4. If the snapshot includes the quiesce option, the ESX host requests the guest operating system to
quiesce the disks via VMware Tools.
Note: Depending on the guest operating system, the quiescing operation can be done by the
sync driver, the vmsync module, or Microsoft's Volume Shadow Copy (VSS) service. For more
information on quiescing, see Troubleshooting Volume Shadow Copy (VSS) quiesce related
issues (1007696) for VSS or A virtual machine can freeze under load when you take quiesced
snapshots or use custom quiescing scripts (5962168) for the SYNC driver.
5. The ESX host makes the appropriate changes to the virtual machine's snapshot database (
.vmsdfile) and the changes are reflected in the snapshot manager of the virtual machine.
Note: When removing a snapshot, the snapshot entity in the snapshot manager is removed
before the changes are made to the child disks. The snapshot manager does not contain any
snapshot entries while the virtual machine continues to run from the child disk. For more
information, see Committing snapshots when there are no snapshot entries in the snapshot
manager (1002310).
6. The ESX host calls a function similar to the Virtual Disk API functions to make changes to the
child disks (
-delta.vmdkand
.vmdkfiles) and the disk chain.
Note: During a snapshot removal, if the child disks are large in size, the operation may take a
long time. This can result in a timeout error message from either VirtualCenter or the VMware
Infrastructure Client. For more information about timeout error messages, see vCenter
operation times out with the error: Operation failed since another task is in progress (1004790).
The child disk
The child disk, which is created with a snapshot, is a sparse disk. Sparse disks employ the
copy-on-write (COW) mechanism, in which the virtual disk contains no data in places, until copied
there by a write. This optimization saves storage space. The grain is the unit of measure in which
the sparse disk uses the copy-on-write mechanism. Each grain is a block of sectors containing
virtual disk data. The default size is 128 sectors or 64KB.
Child disks and disk usage
It is important to note these points regarding the space utilization of child disks:
If a virtual machine is running off of a snapshot, it is making changes to a child or sparse disk.
The more write operations made to this disk, the larger it grows.
The space requirements of the child disk are in addition to the parent disk on which it depends.
If a virtual machine has a 10 GB disk with a child disk, the space used will be 10 GB + the child
disk size.
Child disks have been known to grow large enough to fill an entire datastore.
The speed at which child disks grow is directly dependent on the amount of I/O being done to
the disk.
The size of the child disk has a direct impact on the length of time it takes to delete the snapshot
associated to the child disk.
These Knowledge Base articles touch on the topic of child disks and disk usage:
No more space for the redo log error when attempting to start a virtual machine (1002103)
Why snapshot removal can stop a virtual machine for long time (1002836)
Troubleshooting a datastore or VMFS volume that is full or near capacity (1003412)
The disk chain
Generally, when you create a snapshot for the first time, the first child disk is created from the
parent disk. Successive snapshots generate new child disks from the last child disk on the chain.
The relationship can change if you have multiple branches in the snapshot chain.
This diagram is an example of a snapshot chain. Each square represents a block of data or a grain
as described above:
Caution: Manually manipulating the individual child disks or any of the snapshot configuration
files may compromise the disk chain. VMware does not recommend manually modifying the
disk chain as it may result in data loss. For more information, see
Consolidating snapshots
(1007849)
.
Additional Information
To determine if a virtual machine is running on snapshots, see Determining if a virtual machine
is using snapshots (1004343).
There are specific considerations when hosting a Microsoft Active Directory controller in a
virtual environment. For a full list of considerations, see Microsoft Knowledge Base article
888794.
Note: The preceding link was valid as of November 9, 2009. If you find the link to be broken,
provide feedback on the article and a VMware employee will update the article as necessary.
Time-sensitive applications may be impacted by reverting to a previous snapshot. Reverting the
snapshot will revert the virtual machine to the point in time when the snapshot was created.
This includes any operations conducted by the time-sensitive service or application in the guest
operating system.
Reverting virtual machines to a snapshot causes all settings configured in the guest operating
system since that snapshot to be reverted. The configuration which is reverted includes, but is
not limited to, previous IP addresses, DNS names, UUIDs, guest OS patch versions, etc.
A snapshot operation should not be performed on a virtual machine which uses third-party iSCSI
software initiators and is running in VMware Infrastructure 3. You can perform a snapshot
operation in a vSphere environment, but it requires additional steps. For more information, see
Using a Microsoft or 3rd party software iSCSI Initiator in a VMware environment (1010547) or
Running a Third-Party iSCSI initiator in the Virtual Machine section of:
o
VMware Infrasturcture 3: SAN System Design and Deployment Guide
oVMware vSphere 4: SAN System Design and Deployment Guide
There have been changes to the Delete All snapshot process in the patch releases for ESX/ESXi
3.5 and 4.0. For more information, see:
o
VMware ESX 3.5, Patch ESX350-201006401-SG: Updates VMkernel, VMware Tools,
hostd, VMX, VMnix (1022899)
o
VMware ESXi 3.5, Patch ESXe350-201006401-I-SG: Updates firmware (1020052)
oVMware ESX 4.0, Patch ESX400-201006201-UG: Updates the VMware ESX 4.0 Core and
CIM components (1017721)
o
VMware ESXi 4.0, Patch ESXi400-201006201-UG: Updates Firmware (1017739)
For versions prior to VMware ESX 4.0 Update-2, the task of consolidating all snapshots (Remove
All Snapshots task) caused unique changes stored only in the second snapshot delta disk to be
copied upwards through the snapshot chain and into the first snapshot, or its "parent.". This
effect is recursive for each preceding parent file.
Example: You have a base disk of size 8 GB and 2 levels of snapshots, each of 4 GB each. During
a Remove All Snapshot Tasks, the first snapshot delta disk file can grow, worse-case scenario, to
8GB, as all new blocks from the second snapshot are written. Any common changes stored in
both snapshot levels do not require additional space.
From ESX4.0 Update 2 onwards, the snapshot mechanism has changed. VMware ESX now
incorporates improved consolidation procedures which lessen the demand of free space. You
are able to consolidate virtual machine delta disks even while minimal free space on your
datastore is available.
Domain login fails if snapshot is reverted
Symptoms
The virtual machine is joined to a Windows domain
The snapshot was reverted to a point more than 30 days in the past You are unable to log in to a virtual machine running Windows You do a system restore to a point more than 30 days in the past A Windows Logon error message appears:
Windows cannot connect to the domain, either because the domain
account was not found. Please try again later. If this message
continues to appear, contact your system administrator for assistance.
Resolution
This issue occurs because the machine account password changes automatically every 30 days when a computer is joined to a Windows domain. When the computer's snapshot is reverted to a point in the past, or a system restore is run, the password in the virtual machine can revert to a version that does not match the version on the domain. When the passwords do not match the domain cannot authenticate the
machine's system account credentials nor any user logins from that machine.
To resolve this issue, follow the steps in the Microsoft Knowledge Base article Issues with domain membership after a system restore.
Note: To prevent this issue from happening, disable machine account password updates. For more information, see the Microsoft Knowledge Base article How to disable automatic machine account password changes .
---
Resolving the CID mismatch error: The parent virtual disk
has been modified since the child was created
Symptoms
You are unable to power on a virtual machine or consolidate snapshots if a parent file has been modified after the child has been created. This will include base disks and snapshot delta files. Depending on the modification or caused mismatch, any of the following error messages can be
observed:
Failed to open '<virtual machine disk>' with flags 0xe (The parent virtual disk has been modified since the child was created).
Failed to open (The parent virtual disk has been modified since the child was created).
Failed to open '<virtual machine disk>': The parent virtual disk has been modified since the child was created (18).
DISKLIB-LINK : Attach: Content ID mismatch (7b7644b2 != 4f5a6761). DISKLIB-LINK : Attach: the capacity of each link is different (83886080 != 46399652).
You see this error in the vSphere Client:
Cannot open the disk
'/vmfs/volumes/4a365b5d-eceda1-19-439b-000cfc0086f3/examplevm/examplevm-000001.vmdk' or one of the snapshot disks it depends on.
Reason: The parent virtual disk has been modified since the child was created.
The virtual machine log contains information similar to:
Aug 12 16:36:12.755: vmx| DISKLIB-LINK : Attach: Content ID mismatch (d0fdb25b != ef4854ee).
Aug 12 16:36:12.765: vmx| DISKLIB-CHAIN : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm.vmdk" : failed to open (The parent virtual disk has been modified since the child was
created).
Aug 12 16:36:12.766: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002-delta.vmdk" : closed.
Aug 12 16:36:12.769: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000001-delta.vmdk" : closed.
Aug 12 16:36:12.770: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-flat.vmdk" : closed. Aug 12 16:36:12.771: vmx| DISKLIB-LIB : Failed to open
'/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk' with flags 0xa (The parent virtual disk has been modified since the child was created).
Aug 12 16:36:12.772: vmx| DISK: Cannot open disk
"/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk": The parent virtual disk has been modified since the child was created (18).
Aug 12 16:36:12.773: vmx| Msg_Post: Error
Aug 12 16:36:12.774: vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk' or one of the snapshot disks it depends on.
Aug 12 16:36:12.774: vmx| [msg.disk.configureDiskError] Reason: The parent virtual disk has been modified since the child was created.---
Aug 12 16:36:12.787: vmx| Module DiskEarly power on failed.
o The virtual machine shuts down abruptly during snapshot removal, deletion, or commit.
o The virtual machine shuts down abruptly during or immediately following a Storage
vMotion or Migration.
o You see the following error when attempting to power on a virtual machine: A general system error occurred: Internal error
.
Performing a snapshot removal or powering on the virtual machine generates the error: Content ID mismatch
Purpose
This document provides information and troubleshooting steps to diagnose Content ID mismatch conditions between two or more virtual disk files for a virtual machine.
THE CONTENT OF THIS ARTICLE IS PROVIDED "AS-IS," AND TO THE MAXIMUM EXTENT PERMITTED BY APPLICABLE LAW, VMWARE DISCLAIMS ALL OTHER REPRESENTATIONS AND WARRANTIES, EXPRESS OR IMPLIED, REGARDING THIS CONTENT, INCLUDING THEIR FITNESS FOR A PARTICULAR PURPOSE, THEIR MERCHANTABILITY, OR THEIR NONINFRINGEMENT. VMWARE SHALL NOT BE LIABLE FOR ANY DAMAGES ARISING OUT OF OR IN CONNECTION WITH THE USE OF THIS CONTENT, INCLUDING DIRECT, INDIRECT, CONSEQUENTIAL DAMAGES,
LOSS OF BUSINESS PROFITS OR SPECIAL DAMAGES, EVEN IF VMWARE HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
Resolution
Overview
The CID value of a virtual machine disk descriptor file aids in the goal of ensuring content in a parent virtual disk file, such as a flat or base disk, is retained in a consistent state.
The child delta disks which derive from that base disk's snapshot contain all further writes and changes; these changes depend on the source disk to remain intact.
A virtual machine disk descriptor file details the basic geometry, format, or otherwise identification and handling for a virtual disk or virtual disk delta file. A CID resides in each virtual machine's disk descriptor file for integrity or state tracking, per above.
Example descriptor file for a base disk examplevm.vmdk:
# Disk DescriptorFile version=1 CID=7b7644b2 parentCID=ffffffff createType="vmfs" # Extent description RW 20971520 VMFS "examplevm-flat.vmdk"
# The Disk Data Base #DDB ddb.toolsVersion = "0" ddb.adapterType = "lsilogic" ddb.geometry.sectors = "63" ddb.geometry.heads = "255" ddb.geometry.cylinders = "1305" ddb.uuid = "60 00 C2 9f ae de ba e9-95 4e a7 a6 4e e9-95 c1 c1" ddb.virtualHWVersion = "4"
Example descriptor file for a delta disk examplevm-000001.vmdk:
# Disk DescriptorFile version=1 CID=69a1c662 parentCID=7b7644b2 createType="vmfsSparse" parentFileNameHint="examplevm.vmdk" # Extent description RW 20971520 VMFSSPARSE "examplevm-000001-delta.vmdk"
# The Disk Data Base #DDB
Note: examplevm-000001.vmdk refers to, and in another sense,
depends on, examplevm.vmdk.
When the virtual machine references a virtual disk, it cites either the base disk's descriptor file, or a snapshot delta's descriptor file. In this example, the virtual machine configuration file, or
examplevm.vmx, refers to the delta disk descriptor file:
scsi0:0.present = "true"
scsi0:0.fileName = "examplevm-000001.vmdk" scsi0:0.deviceType = "scsi-hardDisk"
Any time a virtual machine is powered on, the referenced base or delta disk descriptor file's CID value is changed (see CID highlighted in blue, above):
examplevm-000001.vmdk before power-on:
CID=69a1c662
parentCID=7b7644b2
examplevm-000001.vmdk after power-on:
CID=6aff3ba2
parentCID=7b7644b2
All of the above details a virtual machine in good running condition. A mismatch can be found here, which prevents tasks to succeed for this virtual machine:
examplevm.vmdk:
CID=12a9ffab parentCID=ffffffff examplevm-000001.vmdk:
CID=69a1c662 parentCID=7b7644b2
In effect, a CID mismatch ensures that deviance from the original disk state results in all dependent child delta content being invalidated. This protects stored data from further potential corruption.
Cause
Content ID mismatch conditions are triggered by interruptions to major virtual machine migrations such as Storage vMotion or Migration, VMware software error, or user action.
These Content IDs are only used for virtual machine disks with snapshots. For additional information on snapshots, see Understanding virtual machine snapshots in VMware ESX (1015180).
Some scenarios to avoid in particular include:
Interrupting a virtual machine migration or Storage vMotion. For more information, see After cold migration on ESX Server, virtual disk with snapshot has the wrong CID (1005228).
Adding snapshotted virtual machine disks to other virtual machines and powering them on.
Expanding, enlarging, or modifying virtual machine disks with existing snapshots. For more
information, see A virtual machine cannot boot after extending a base virtual disk that is part of a snapshot hierarchy (1646892).
Troubleshooting
When there is a CID mismatch, the virtual machine name is provided in the error message, but you must identify:
What virtual machine disk, or disks are affected.
What specific disk descriptor files are affected.
The cause of the mismatch, or what changes occurred. Identifying the virtual machine disk and descriptor files affected
There are several methods to log into an ESX host to review content of utilized datastores, depending on the version of ESX utilized. For more information, see Editing configuration files in VMware ESX
(1017022).
Notes:
The datastore browser provided in the VMware vSphere Client or VMware Infrastructure Client do not distinguish between virtual machine descriptor (1) and flat or delta files (2). They are
collapsed into singular entities to make datastore browsing simpler. As you need to distinguish between the two files, use the additional access methods provided in the referenced article.
Due to the nature of the problem experienced, the quickest method for resolving the issue is with
the Command-Line Interface available on the individual ESX host. Utilize this method if you have sufficient background knowledge on command-line usage.Alternatively, you can use the VMware vSphere Command Line Interface (CLI) or VMware vSphere Management Assistant appliance (vMA) to obtain the virtual machine disk descriptor files for review.
If you are unable proceed, see Filing a Support Request (1021619) to seek technical assistance from VMware.
After locating the virtual machine's files and directory:
1.
The virtual machine's vmware.log file contains information which identifies the specific disk chain affected. Review the logs and note the location and files affected. For example:Aug 12 16:36:12.755: vmx| DISKLIB-LINK : Attach: Content ID mismatch (d0fdb25b != ef4854ee).
Aug 12 16:36:12.765: vmx| DISKLIB-CHAIN : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm.vmdk" : failed to open (The parent virtual disk has been modified since the child was
created).
Aug 12 16:36:12.766: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002-delta.vmdk" : closed.
Aug 12 16:36:12.769: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000001-delta.vmdk" : closed.
Aug 12 16:36:12.770: vmx| DISKLIB-VMFS : "/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-flat.vmdk" : closed. Aug 12 16:36:12.771: vmx| DISKLIB-LIB : Failed to open
'/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk' with flags 0xa (The parent virtual disk has been modified since the child was created).
Aug 12 16:36:12.772: vmx| DISK: Cannot open disk
"/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk": The parent virtual disk has been modified since the child was created (18).
Aug 12 16:36:12.773: vmx| Msg_Post: Error
Aug 12 16:36:12.774: vmx| [msg.disk.noBackEnd] Cannot open the disk '/vmfs/volumes/4a365b5d-eceda119-439b-000cfc0086f3/examplevm/examplevm-000002.vmdk' or one of the snapshot disks it depends on.
Aug 12 16:36:12.774: vmx| [msg.disk.configureDiskError] Reason: The parent virtual disk has been modified since the child was created.---
Aug 12 16:36:12.787: vmx| Module DiskEarly power on failed.
Note: This indicates that the file examplevm-000002.vmdk references its parent (which in itself references another parent file), one of which has been modified some time after examplevm-000002.vmdk was created. You must take corrective measures on any of these files: examplevm.vmdk, examplevm-000001.vmdk, and examplevm-000002.vmdk.
2.
With the problem point (or points) determined, make backup copies of the disk descriptor files that require corrections or editing. In the above example, backups ofexamplevm.vmdk, examplevm-000001.vmdk, andexamplevm-000002.vmdkare required.3.
Review the contents of each affected descriptor file and compare the mismatching values. For example: examplevm.vmdk: CID=12a9ffab parentCID=ffffffff examplevm-000001.vmdk: CID=69a1c662 parentCID=7b7644b2 examplevm-000002.vmdk: CID=59fab513 parentCID=69a1c6624.
Disk examplevm-000002.vmdk links to examplevm-000001.vmdk without error. However, the base disk examplevm.vmdk has been modified, causing the error.
Correcting the Content ID mismatch
At this point the problem point has been identified, the virtual machine's files have backups, and corrections must be applied.
To correct the Content ID mismatch:
