Skip to main content

Accurately recreating the partition table of a VMFS volume

 This article need an expert to use, Please reach to VMware support to not run into more complex situation.


 Details
  • You cannot mount a VMFS volume which you were previously able to mount.
  • The partition table of the volume no longer contains any partitions.
  • When recreating a partition table on a volume, this error appears in /var/log/vmkernel:

    WARNING: LVM: 1650: [vmhba1:0:2:1] Device size mismatch (actual 16777083 blocks, stored 16777088 blocks)
 Solution
In some cases, the original partition table was not created at the end of the volume. You must calculate what the end boundary of the lost partition table entry is, and re-create the new one with the same ending.

Notes:
The procedure in this article uses this sample:

00110000 0d d0 01 c0 03 00 00 00 10 00 00 00 00 0a 00 00 |................|
00110010 00 03 56 4d 77 61 72 65 2c 20 56 4d 77 61 72 65 |..VMware, VMware|
00110020 20 56 69 72 74 75 61 6c 20 53 31 2e 30 20 76 6d | Virtual S1.0 vm|
00110030 68 62 61 30 3a 31 3a 30 00 00 00 00 00 00 00 00 |hba0:1:0........|
00110040 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00110050 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 00 |................|
00110060 ff 7f 00 00 00 00 01 00 00 00 07 00 00 00 06 00 |................|
00110070 00 00 09 00 00 00 00 00 00 00 00 00 10 01 00 00 |................|
00110080 00 00 d8 9b 6f 47 42 5d 03 db c8 6d 00 0c 29 05 |....oGB]...m..).|
00110090 e9 39 e7 8a c6 c3 06 42 04 00 39 3a c7 c3 06 42 |.9.....B..9:...B|
001100a0 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
001100b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|



Perform these steps:
 
  1. Collect the first two (2) megabytes of the volume using the dd command:

    # dd if=/dev/sdX of=/root/sdX.dump bs=1M count=2

    The file sdX.dump is created in /root. If copying a file from the ESX console is a problem, substitute /root with /vmfs/volumes/volume_name/sdX.dump.

    You can download the file using the VMware Infrastructure / vSphere Client datastore browser.

    Note: You can view the information live with this command:

    # hexdump -C -n 2048000 /dev/sdX
     
  2. Identify the necessary values:
     
    1. To view the 2 MB dump, run the command:

      # hexdump -C sdX.dump | less
       
    2. Search for the magic number (it marks the beginning offset of the LVM device header offset): 0d d0 01 c0

      In the sample, it is located at 0x00110000 or 1114112.

      Note: The conversion of the 0x00110000 (Hex) to 1114112 (Decimal) can be done using the Windows Calculator in Programmer Mode. Activate Programmer mode, input in HEX format the value "0x00110000" and change to decimal in the Calculator to get the converted value.

      The starting sector is calculated by:

      (<LVM device header offset in decimal> - 1048576) / 512

      In this sample, the starting sector = (1114112 - 1048576) / 512 = 128

      Note: 1048576: This number is always the same (1048576 <=> 1MB). It corresponds to the 1 MB padding (zeros) at the beginning of a VMFS partition. This is true for vmfs3 and vmfs5.
       
    3. To get the total size (in bytes) of the volume, extract from position 0x5e (94) the next 8 bytes (00 00 ff 7f 00 00 00 00).

      As the value is encoded in little endian, it resolves to 0x000000007fff0000 or 2147418112 bytes.

      Note: To find position 0x5e, go to byte 0x110000 + 0x5e, or find 110050 in the first column and then go right 0xe (15) bytes.

      Little endian: 00 00 ff 7f 00 00 00 00
      Big endian: 00 00 00 00 7f ff 00 00
      Bits: 00000000 00000000 00000000 00000000 01111111 11111111 00000000 00000000
      Decimal: 0*2^0 + 0*2^1 + ... + 1*2^16 + ... + 0*2^47 = 2147418112
       
    4. Calculate the ending block:

      ((<Total size in bytes> + (<Starting offset> * 512)) / 512) -1

      In this sample, the ending sector = ((2147418112 + (128 * 512)) / 512) - 1 = 4194303
       
  3. Use the fdisk utility to correct the partition:
     
    1. Start fdisk on the device using the -u option to be able to set the starting block:

      # fdisk -u /dev/sdX
       
    2. If the partition exists, remove it:
      1. Press d and press Enter.
      2. Press 1 and press Enter.
         
    3. Create the partition:
      1. Press n and press Enter.
      2. Enter the calculated value of the starting offset (in this sample: 128) and press Enter.
      3. Enter the calculated value of the ending block (in this sample: 4194303) and press Enter.
         
    4. Change the partition type to FB:
      1. Press t and press Enter.
      2. Press 1 and press Enter.
      3. Enter fb and press Enter.
         
    5. Save the changes and exit. Press w and press Enter.
       
  4. Rescan the existing volumes to see if the datastores can be mounted with the command:

    # vmkfstools -V

    The missing volume should be mounted now. If not, check /var/log/vmkernel for further evidence of:

    WARNING: LVM: 1650: [vmhba1:0:2:1] Device size mismatch (actual 16777083 blocks, stored 16777088 blocks)

    Verify your numbers and that the partition was created correctly.

Note: For large LUNs, fdisk seems to have a display issue (that is, it displays the ending block as a negative number). In this case, you can use parted or another tool to create the partition.

If this is not an option and you are still getting Device size mismatch errors, you can determine the ending with this calculation:

<Adjusted End> = -(abs(<Previous End>) + (<Actual> - <Stored> + <Starting Offset>) )

Where:
  • <Adjusted End> is the new ending block adjusted to work around the fdisk bug.
  • <Previous End> is the ending block fdisk displayed for the volume that generated the above error.
 Confidential or Internal Information

A VMFS5 sample for reference:

00200000 0d d0 01 c0 05 00 00 00 15 00 00 00 02 16 04 00 |................|
00200010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00200020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 60 03 |..............`.|
00200030 0d 90 1b 3c 19 02 4a 3d 96 45 96 1a 91 43 56 69 |...<..J=.E...CVi|
00200040 72 74 75 61 00 00 00 00 00 00 00 00 00 00 00 00 |rtua............|
00200050 00 00 00 00 00 00 00 00 00 00 00 02 00 00 00 be |................|
00200060 ef ff 76 01 00 00 01 00 00 00 6f 17 00 00 6e 17 |..v.......o...n.|
00200070 00 00 05 00 00 00 00 00 00 00 00 00 10 01 00 00 |................|
00200080 00 00 c7 fc 2c 51 50 ab d5 bb 3d 3a 18 a9 05 5e |....,QP...=:...^|
00200090 33 00 40 0a 89 f9 a1 cb 04 00 78 89 1b 6e a9 d6 |3.@.......x..n..|
002000a0 04 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
002000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 |................|

002000c0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
 

In this example,
the magic number can be found on the line starting 00200000 , the total size is located on the line starting 00200050 and the little endian value is: 00 be ef ff 76 01 00 00 .

Convert the little endian value to big endian: 00 00 01 76 ff ef be 00 .
Convert Hex to Decimal: 1610611670528
The size of the volume is: 1610611670528 bytes = ~1500GB


Collect the first two (2) megabytes of the volume using the dd command:

# dd if=/dev/sdX of=/root/sdX.dump bs=1M count=2

The file sdX.dump is created in /root . If copying a file from the ESX console is a problem, substitute /root with /vmfs/volumes/volume_name/sdX.dump.

We can also use /vmfs/devices/disks/naa.X for this command This is valid for 5.x and 6.x

 

 

 


Recreating the volume with partedUtil
 
For VMFS3, using MBR:
partedUtil set /path/to/device "1 <starting offset> <ending offset> 251 0"
 
In this example: partedUtil set /dev/sdX "1 128 4194303 251 0"
 
 
For VMFS5, using GPT:
Note: You can use GPT partition type if the partition size is more than 2TB
partedUtil setptbl /path/to/device gpt "1 <starting offset> <ending offset> AA31E02A400F11DB9590000C2911D1B8 0"
 
example:
partedUtil setptbl /vmfs/devices/disks/naa.6f01faf000dcedf3000005a6558a589f gpt "1 2048 35156901854 AA31E02A400F11DB9590000C2911D1B8 0"
 

For VMFS 6 
SR = 19126886203 

Example Calculation goes as below 
 

LVM header :- 0d d0 01 c0

Shows this offset 00200000 = 2048 = VMFS 6 

00 02 e0 ff ff 05 00 00 = Hex 

000005ffffe00200 = Decimal Value :- 6597067670016


As per the Calculation and formula :- ((<Total size in bytes> + (<Starting offset> * 512)) / 512) -1

6597067670016 + (2048*512)/512 – 1 = 12884899840

12884899841 - 1 = 12884899840

Partition Table of the Datastore 'partedUtil getptbl' output below which matches the calculation :

802048 255 63 12884901888
1 2048 12884899840 AA31E02A400F11DB9590000C2911D1B8 vmfs 0


Hexdump of the LVM header output of datastore :

00200000 0d d0 01 c0 06 00 00 00 15 00 00 00 01 26 03 00 |.............&..|
00200010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00200020 00 00 00 00 00 00 00 00 00 00 00 00 00 00 62 31 |..............b1|
00200030 37 37 62 33 30 64 37 61 37 62 36 39 35 31 36 63 |77b30d7a7b69516c|
00200040 39 63 65 39 30 30 35 64 66 33 64 33 39 66 53 65 |9ce9005df3d39fSe|
00200050 72 76 65 72 00 00 00 00 00 00 00 02 00 00 00 02 |rver............|

00200060 e0 ff ff 05 00 00 01 00 00 00 02 00 00 00 01 00 |................|
00200070 00 00 07 00 00 00 00 00 00 00 00 00 10 01 00 00 |................|
00200080 00 00 77 b4 43 5b 6c e2 7c a0 45 98 00 ca fe 01 |..w.C[l.|.E.....|
00200090 00 85 9c 9a 72 d8 95 70 05 00 f8 b8 5c 30 ed 84 |....r..p....\0..|
002000a0 05 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
002000b0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
002000c0 00 00 00 00 00 00 00 00 00 00 00 10 00 00 ff 5f |..............._|
002000d0 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|

Comments

Popular posts from this blog

Recreating a missing VMFS datastore partition in VMware vSphere 5.x and 6.x

    Symptoms A datastore has become inaccessible. A VMFS partition table is missing.   Purpose The partition table is required only during a rescan. This means that the datastore may become inaccessible on a host during a rescan if the VMFS partition was deleted after the last rescan. The partition table is physically located on the LUN, so all vSphere hosts that have access to this LUN can see the change has taken place. However, only the hosts that do a rescan will be affected.   This article provides information on: Determining whether this is the same problem Resolving the problem   Cause This issue occurs because the VMFS partition can be deleted by deleting the datastore from the vSphere Client. This is prevented by the software, if the datastore is in use. It can also happen if a physical server has access to the LUN on the SAN and does an install, for example.   Resolution To resolve this issue: Run the  partedUtil  command on the host with the issues and verify if your output

ما هى ال FSMO Roles

  بأختصار ال FSMO Roles هى اختصار ل Flexible Single Operation Master و هى عباره عن 5 Roles فى ال Active Directory و هما بينقسموا لقسمين A - Forest Roles 1- Schema Master Role و هى ال Role اللى بتتحكم فى ال schema و بيكون فى Schema Master Role واحد فى ال Forest بيكون موجود على Domain Controller و بيتم التحكم فيها من خلال ال Active Directory Schema Snap in in MMC بس بعد ما يتعمل Schema Register بواسطه الامر التالى من ال Cmd regsvr32 schmmgmt.dll 2-Domin Naming Master و هى ال Role المسئوله عن تسميه ال Domains و بتتأكد ان مفيش 2 Domain ليهم نفس الاسم فى ال Forest و بيتم التحكم فيها من خلال ال Active Directory Domains & Trusts B- Domain Roles 1-PDC Emulator و هى ال Role اللى بتتحكم فى ال Password change فى ال domain و بتتحكم فى ال time synchronization و هى تعتبر المكان الافتراضى لل GPO's و هى تعتبر Domain Role مش زى الاتنين الاولانيين و بيتم التحكم فيها من خلال ال Active directory Users & Computers عن طريق عمل كليك يمين على اسم الدومين و نختار operations master فى تاب ال PDC Emu

Unlock the VMware VM vmdk file

  Unlock the VMware VM vmdk file Kill -9 PID Sometimes a file or set of files in a VMFS become locked and any attempts to edit them or delete will give a device or resource busy error, even though the vm associated with the files is not running. If the vm is running then you would need to stop the vm to manipulate the files. If you know that the vm is stopped then you need to find the ESX server that has the files locked and then stop the process that is locking the file(s). 1. Logon to the ESX host where the VM was last known to be running. 2.  vmkfstools -D /vmfs/volumes/path/to/file  to dump information on the file into /var/log/vmkernel 3.  less /var/log/vmkernel  and scroll to the bottom, you will see output like below: a. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)FS3: 130: &lt;START vmware-16.log&gt; b. Nov 29 15:49:17 vm22 vmkernel: 2:00:15:18.435 cpu6:1038)Lock [type 10c00001 offset 30439424 v 21, hb offset 4154368 c. Nov 29 15:49:17 vm22 vmkernel: gen 664