Locked orphans VMDK files

We encounter some strange behavior with our pal zozor during a datastore maintenance (understand migration).

Since we still have not upgrade to ESXi5 and datastore maintenance mode, we used the datastore maintenance script from hypervisor.fr

No problem with it (it’s really awesome for 4.x branches), except that after few check, we found orphans vmdk still in the source datastore.

After check, thieses vmdk weren’t used by VM, the VM vmdk files were indeed in the destination datastore (the migration went well).

After a double check (you better do with theses case) in the .vmx configuration file and in the .vmdk descriptor, we decide to delete them as they were no longuer used.

And here were the errors :

We got the same result trying to delete them directly trought SSH on ESXi servers ! With a closer look, we noticed some error message during delete tries in the /var/log/message file :

On the step n°1, we see the rm command in order to delete the -flat.vmdk file (the one remaining after datastore migration).

On the step n°2, message about the owner, but no error

On the step n°3, we have a locking file try, and after that we got the same owner message than the one in step 2 but now with an error : is not free

After theses error reading, we can understand that the owner saw in the step 2 won’t unlock the file we try to delete = not f**g cool !

Start with that, we got several options. If we had just a few ESXi servers, we can connect on each of them and try to execute the delete command hoping that it is the one !

Otherwise, assuming that from 2 servers, we wanted to automate things, we must find a way to find the right server straight.

In order to do this, we take the owner ID see in step 2 :

Basing ton the VMware KB 1008728, we understand that the last part (the red one) of this ID match the MAC address of the ESXi server.

Finally, we had a lead finding the rebellious esxisme :p Now, in order to find the server with this MAC address, we can use the aweome network report script from hypervisor.fr (not published yet, but it’ll soon be) or we can use this little PowerCLI OneLiner :

Get-VMHost | Get-VMHostNetworkAdapter | ?{$_.Mac -eq "3c:4a:92🇩🇪1b:44"}

And voilà, we got the gilty server, we can connect directly to it and delete the remaining orphans vmdk files, and it goes well :p = cool