Skip to main content

HOW TO Recover from failed Storage VMotion

 HOW-TO: Recover from failed Storage VMotion

A while ago I received a request from the storage department to move a whole ESX cluster to another storage I/O-Group. This would be a disruptive action.
I was wondering if storage VMotion would help me out here if they assigned me some new storage on the other I/O-Group instead of moving the current LUNs.

If you are going to use sVMotion I would strongly suggest using the sVMotion plug-in from lostcreations.

Well sVMotion made it possible to move the cluster to the other I/O-Group online. This would save me a lot of time explaining to the customer why the entire cluster had to go offline. We are talking off 175+ vms here.

So that was the path I chose, and it worked out fine, but I got so excited on sVMotion that I didn’t paid attention to the available storage space left on my new LUNs. So after a while the LUN filled up and the sVMotion process failed.

Whenever a sVMotion fails you probably end up in a situation where the config files are moved to the new location and the .vmdk files and their accompanying snapshot files (sVMotion creates a snapshot in order to copy the .vmdk files) are still in the old location.

Issuing another sVMotion will generate this error: “ERROR: A specified parameter was not correct. spec” and if you turn off the vm you get an extra option “Complete Migration”. This option actually makes a copy of the .vmdk files to the same LUN, and hence it requires twice the space of the .vmdks on your LUN and most importantly it requires downtime.

Here’s what I did to resolve this split:

  • Create a snapshot of the vm. Since it’s not available via vCenter GUI in this state, you have to do this in the COS or connect your VIC directly to the ESX host.

    • Through SSH console session:

      • find the config_file_path of the VM
        vmware-cmd -l
      • Create a snapshot of the vm
        vmware-cmd <config_file_path> createsnapshot snapshot_name snapshot_description 1 1

    • Through VIC:
      • Use GUI as normal.

  • Remove (Commit) the snapshots:
    vmware-cmd <config_file_path> removesnapshots
    This will remove the newly created snapshot AND the snapshot created by sVMotion.

  • vCenter still thinks the vm is in dmotion state so you can’t edit settings, perform VMotion or anything else via vCenter. To fix this we need to clear the DMotionParent parameters in the .vmx file with the following commands from the COS:
    vmware-cmd <config_file_path> setconfig scsi0:0.DMotionParent ""
    vmware-cmd <config_file_path> setconfig scsi0:1.DMotionParent ""
    Do this for every DMotionParent entry in the .vmx file, so be sure to check your .vmx file to get the right SCSI IDs. Note that editing the .vmx file directly will not trigger a reload of the .vmx config file!

  • Now Perform a new storage migration to move back the .vmx configuration file to its original location.

  • Clean up destination LUN and remove any files/folders created by the failed sVMotion. We’re done and back in business again without downtime!
    We can retry the sVMotion now.

Offcourse I didn’t found out all this by myself. All credits go to Argyle from theVMTN Forum. You can read his original thread here.

Someone would probably say “Don’t try this at home”, but if you’re curious and do want to try this at home, use the following procedure to reproduce this split situation:

  • perform a sVMotion of a TEST vm
  • on the COS of the ESX host issue”:
    service mgmt-vmware restart
  • Have fun!!


Popular posts from this blog

Question كيفية عمل share للـ outlook conntact لكل الـ Domain Users

  الحل بسيط جدا عايز الكونتاكت تتحدث دايما بحيث انك لما تضيف يوزر جديد يسمع في الكونتاكت اول حاجه بتدخل علي in office 2003 tools --- email account ---- add address book --- internet directory service (LDAP) type your server name then login info . mark this server require me to logon type any user on active directory and its password then save and close outlook and open it again now you will find all your active directory users in address book

3 things has to be done for better performance

  Tips from Goutham: 3 things has to be done for better performance: By default, XP displays extra graphic objects for menu items which can slow down your display. 1. To turn off these selectively... Right click My Computer Select Properties Click Advanced tab Under Performance, click Settings button To turn them all off, select Adjust for best performance Preference is to leave them all off except for Show shadows under mouse pointer and Show window contents while dragging 2. To speed up the display of the Start Menu Items, turn off the menu shadow. Right click open area of the Desktop Select Properties Click Appearance tab Click Effects button Uncheck Show shadows under menus 3. You can increase system performance by loading more of the system into memory. DO NOT attempt this with less then 512MBs of ram. Your system will become unstable. Click Start Click Run Enter regedit Click OK Go to HKEY_LOCAL_MACHINE SYSTEM CurrentControlSet Control Session Manager Memory Management Double cli

The difference between DNS and NDS

  Novell Directory Services(NDS) - Novell Directory Services (NDS) is a popular software product for managing access to computer resources and keeping track of the users of a  network , such as a company's  intranet , from a single point of administration. Using NDS, a network administrator can set up and control a  database  of users and manage them using a  directory  with an easy-to-use graphical user interface ( GUI ). Users of computers at remote locations can be added, updated, and managed centrally. Applications can be distributed electronically and maintained centrally. NDS can be installed to run under  Windows NT , Sun Microsystem's Solaris, and IBM's  OS/390  as well as under Novell's own  NetWare  so that it can be used to control a multi-platform network. NDS is generally considered an industry  benchmark  against which other products, such as Microsoft's Active Directory, must compete. Lucent Technologies plans to integrate NDS into its own QIP product