LazyMethodsEnabler tool

Following hypervisor.fr’s post Storage vMotion : the method is disabled, we work today for a new customer whom had too many some virtual machine with disabled methods (and increasing on a daily basis thanks to backup jobs!) :

Without blaming existing yellow and black crappy backup tool (although it is quite tempting!), there are too many affected virtual machines to fit any of these VMware KB workarounds. As a reminder, workarounds are:

• VM unregister/register (bye bye statistics, custom fields and history as MoRef is no longer the same > NoWay)
• vCenter’s database’s manual purge (this operation needs vCenter service to be stopped, so it’s not really a production workaround if it occurs often > NoWay)
• Restart a backup job (that is viable on a small scale, but in our case, the number of VM with this problem increased by about 60 daily ! > NoWay)

Pending the migration to a real backup solution (the transition to Veeam Backup being expected), we still had to find a cleaner solution :p Assuming that a restart of the VM backup is likely to resolve the issue, we wondered what is causing the error. The KB explains:

When a VM-level backup begins, the backup system informs vCenter to disable Storage vMotion for that VM to ensure that the backups can complete successfully

The interesting part is *the backup system informs vCenter*. After some research, we finally found this is done through Virtual Disk Development Kit (VDDK) calls. In order to have a better understanding of backup and VMDK files handling by third party tools, we started to read the Virtual Disk Programming Guide looking for some answers.

Besides the extremely interesting part of this SDK (we also recommand you to take a look just to see how amazing it is), what we were looking for is located in Advanced Transport APIs on Prepare For Access and End Access.

Here is the excerpt answering our question:

The VixDiskLib_PrepareForAccess() function notifies a vCenter-managed host that a virtual machine’s disks are being opened, probably for backup, so the host should postpone virtual machine operations that might interfere with virtual disk access. Call this function before creating a snapshot on a virtual machine. Internally, this function disables the vSphere API method RelocateVM_Task.

The VixDiskLib_EndAccess() function notifies the host that a virtual machine’s disks have been closed, so operations that rely on the virtual disks to be closed, such as vMotion, can now be allowed. Call this function after closing all the virtual disks, and after deleting the virtual machine snapshot. Normally this function is called after previously calling VixDiskLib_PrepareForAccess, but you can call it to clean up after a crash. Internally, this function re-enables the vSphere API method RelocateVM_Task.

So it’s all about VixDiskLib_PrepareForAccess() and VixDiskLib_EndAccess()  calls who notifies vCenter to handle the disabled methods (restraining for instance to order a Storage vMotion during a backup).

Regarding our issue, restarting a backup job perform the following operations PrepareForAccess>Backup>EndAccess and therefore is supposed to fix the issue thanks to the EndAccess() call. On this basis, we thought that a viable large-scale solution (other than switching to real backup solution) would be to have a program that does the following operations PrepareForAccess>EndAccess (and hence would be much faster than a full backup of each VM).

As VDDK is a collection of C/C++ libraries, we had to bring back some memories on C++ developement skills (we are indeed far from C# or PowerShell!). So we develop a program that will help reactivate the methods disabled by a f! tool

This tool is standalone for now, so you must use the VM’s MoRef in argument (yes I know it’s not user friendly, but hey what did you expect, it’s C++ !). It takes the following parameters:

lazyMethodEnabler.exe -host 10.69.69.69 -user "vmdude" -password "esxi4ever" -vm "moid=vm-69"

• -host host IP : vCenter server IP
• -user vcenter Username : username used to connect to vCenter
• -password vcenter Password : username password
• -vm moid=moref : VM’s MoRef

We added an optionnal parameter -disableMethods which allow to disable the RelocateVM_Task method. Used with this parameter, the program will only call VixDiskLib_PrepareForAccess().

It will not call VixDiskLib_EndAccess() and will set a VM with the disabled method.

The Visual Studio 2012 (Visual C++) sources are available here:

Since this program is based on VDDK calls, the SDK must be installed on the same computer as lazyMethodEnabler will be run. Tests were made with VDDK 5.0 (available here : http://www.vmware.com/download/download.do?downloadGroup=VDDK50U2) on a Windows 2008 R2 server.

PS: Sorry Samir it took so long to publish this requested post :p Mea Culpa !