.ultrageek. Computers, Music, and Thoughts

19Aug/1415

Converting Physical Drives or Partitions to Vmware vmdk (P2V)

Note: This assumes you are trying to virtualize a Windows OS.

What if you had a perfectly good working hard drive but the machine that was running it was dead, DOA, not working, or had been upgraded with a new hard drive and OS? Connecting that to a machine and getting the files is easy enough but what about if you wanted to boot the OS back up? The Virtual Option is going to be your best bet but how? Sure there are plenty of options like Ghost that will keep your machine backed up with a VMDK option but how to accomplish this without spending money? Now that is a challenge...challenge accepted!

The process is generally referred to as Physical To Virtual or P2v for short. In doing some research on how to do this I've worked out a process that as long as you don't mind forking over a working email address (man these sites are so clever!) you can download a couple tools to make this possible.

To accomplish this task you will need to complete the following steps:

  1. Create the virtual drive image (vmdk) file to use with VM Ware Workstation 10.x or VM Player 6.0
  2. Create the VMX File used by the converter
  3. Tranform the image with the appropriate drivers to boot (HAL, HD, etc.)

Depending on what machines you have at your disposal here is one method.

Step One - Download Tools

  1. Disk2VHD
    1. Use this to pull the image off the drive in to one VHD file
    2. Download Link: http://technet.microsoft.com/en-us/sysinternals/ee656415.aspx
  2. StarWind V2V Image Converter
    1. Used to convert the VHD to VMDK
    2. You will have to give them a working email address after which they will email you the link to the download...so sneaky
    3. Download Link: http://www.starwindsoftware.com/converter
  3. VM Ware Converter
    1. Used to load the appropriate drivers and make the VMDK ready for use, otherwise you will get blue screens or boot device errors
    2. You will have to have a free vmware account to access this download...even more sneaky!
    3. Download Link: http://www.vmware.com/products/converter

Step Two - Clean up your source!

Do yourself a favor and delete the following files:

  1. Hibernate and Pagefile files. Turn on show hidden, find these large guys, and delete them.
  2. Use "Tree Size" free edition and determine where else you can trim unneeded files: http://www.jam-software.com/treesize_free/

Step Three - Create your image

  1. Fire up Disk2VHD
    1. Uncheck "Use Vhdx", Starwind will complain that it is not a valid image
    2. Check "Prepare for use in Vitual PC" if you are running on XP or Windows Server 2003. This will save you the "VMware Converter" step.
    3. Check the partition you wish to backup
    4. If Windows Server 2008, Windows 7, or higher also backup the hidden system partition that starts with "\\?\Volume{12393df90-...". You will need this.
    5. Save to a location that has twice the space as DIsk2VHD recommends as you will need space for the VMDK in the next step.
    6. If you get Shadow Copy errors, just uncheck that box. It is a convenience, not a deal breaker. It's used more (I believe) for copying hard drives on machines that are still running AKA your boot drive.
    7. Create!
  2. Fire Up StarWind V2V Image Converter
    1. Run this as administrator
    2. Select the VHD
    3. I recommend "VMWare growable image" so that it doesn't take up space for empty drive space. You can always use the vmware tools to change this.
    4. Pick IDE
    5. Select the Save Spot
    6. Next starts the process...now do something else as unless your drive is small or your machine is insanely fast it is going to take a bit to crunch
  3. Fire up VM Player
    1. We need to create the VMX file so that the Converter can process the VMDK file. If you are handy with creating these by hand then skip this step but with the latest version of VM Player, it will create new VM's from scratch which is something the earlier Pre-5.0 versions would not.
    2. Go ahead and create the directory for where you want the vmx and vmdk output. Vmware images work much better silo'd and segmented from other files.
    3. Note: If you have VM Workstation you can use the custom set up of the new VM to select an existing VMDK
    4. Click "Create a New Virtual Machine"
    5. Select "I will install the operating system later"
    6. Pick the appropriate OS
    7. Use the default location. We are going to move the VMX to the directory you created on step 3.2
    8. Pick the Size Disk and Single/Multi File, does not really matter. We are just trying to get the VMX created
    9. The hardware for CPU, RAM, etc. doesn't matter as much, you will edit these on VM Player once you have converted the image.
    10. Edit the VMX File
    11. Make sure somewhere in the file is the following lines:
      1. ide1:0.present = "TRUE"
      2. ide1:0.fileName = "C:\VM\Old Machine\OLD-PC.vmdk"
    12. Where filename equals the path to your VMDK
    13. Save this file to the directory you created in step 3.2
    14. This should work...
  4. Fire Up Vmware Converter
    1. You do not have to run this as administrator though it does not hurt
    2. Go ahead and create ANOTHER directory for where you want the FINAL vmx and vmdk output. This is a different directory from step 3.2.
    3. Click the 'Convert Machine'
    4. From the "Select Source Type:" drop down select
    5. Select your vmx file
    6. Click Next. If Converter has issues with the VMX this is where it will puke and stop.
    7. Select the product you're going to use the VMDK on, Player or Workstation and the directory. This will need to be the directory you created for step 4.2 and not the directory you created in step 3.2
    8. Review the final details and Next
    9. Finish to start the process
  5. Errors with Server 2008, Windows 7, or newer OS (Unable to Find the System Volume, reconfiguration is not possible)
    1. First, you need to make sure you got the 'System Reserved' partition in the image that you made from the source disk. This is the boot loader partition and without it things are pretty hopeless though I'm sure someone has figured out a workaround.
    2. Second, you will need either the original or similar in system version ISO of the OS you are trying to recover as you will need it to boot up and run several commands.
    3. Third, you will need to use VM Player or workstation to boot up the source VMDK. You will need to add the ISO referenced in step 5.2 and you will probably need to boot in to the VM bios in order to make the CD ISO bootable. There is a handy option now to boot straight in to the bios, saving you the effort of having to bang the keyboard before the post finishes and boots in to the OS.
    4. Once you have booted in to the recovery/setup, you will need to follow these steps outlines in this webpage: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1037507
      1. If you do a "bcdedit' and the partition says 'unknown' use this
      2. If your OS is on 'D:' use step 10 and not 8, this is what I had to do for my Server 2008 R2 image
      3. I did not have to do the regedit, just the bcdedit commands. This fixed the 'unknown' issues

If this seems like a lot of steps, it is but I tried to skip obvious steps/defaults. The challenge was mostly getting the right tools, understanding the order of operation, and then the new boot loader Microsoft uses for its latest OS's. If someone actually uses this and needs help, finds a missing step, dead link,  or just a general question please leave a comment and I'll do my best, thanks for the read and hope this has been helpful.

Here are the pages that I used to help kick start this:

 

Filed under: Knowledge, VMware 15 Comments
20Sep/111

Backing up VM Server using the CLI

I do work for a company that has a VM Server with multiple hosts running on it. DNS, Mail, Web, these tasks generally work best on dedicated hosts so that when they go bump in the night they don't take everything with it or complain when you have to reboot or restart services to fix them. Isolation worked before VM's and it has only gained ground afterwards. The question comes, how to back them up on the cheap? Shutting them down, zipping and/or copying them is not an option as it takes a bit of time (30-120 minutes) to get copies of the vmdk's.

So what I'm about to share with you is not a new idea as it was obtained from the usual amount of Google searching. What I will show though is an end to end process for admins that have VM Server 2.x running on a Microsoft Windows platform to back up, while the machine is running, to a file that can then be backed up either through FTP or other means if you have physical access to the box. It will basically snapshot, copy, and then zip up the vmdk files. Experts can skip to the end where the final example is assembled, the rest please trudge along so that the "why" and "how" is answered.

Step One - Preparation

  • Storage - Determine where you are going to house the backups. Make sure it has at least twice or three times as much as space as the VM currently takes on your storage.
  • VMware Command Line Tools - You will need to download and install the following package in order to use the "VMrun" command called by the script you will create used to list your VM's, create and delete snapshots, and use to perform other command line maintenance chores. Here is the link to the VMware site to download: http://www.vmware.com/support/developer/vix-api/. You can also Google "VMrun" for additional info.
  • Compression - You will need a command line compression program in order to take the files that you have copied and archive them to a single compressed image for backup and storage. My recommendation for this problem is to download and use the latest version of 7zip which is open source. The good news is that you will not have to download a specific command line version, just install the standard package and make sure that the folder is in the system path so that you can call it from a batch file without having to explicitly call where the EXE is located. Here is the 7zip website: http://www.7-zip.org/
  • Robocopy - You can use anything to copy this but I have found that through the years robocopy is tried and true for this type of copying. I would recommend using the Windows 2003 Resource Kit for the latest version: http://www.microsoft.com/download/en/details.aspx?displaylang=en&id=17657. I'm sure there are other sources but would recommend getting it from Microsoft if possible. Don't worry about the age, command copying is pretty old technology, not a lot has changed though robocopy does have some flexibility not found in your standard xcopy toolset. This is a good reference: http://ss64.com/nt/robocopy.html
  • Text Editor - Make sure you know how to use a text editor such as notepad or Editpad Lite (plug). Would highly recommend that your text editor of choice resides in your "Send To" menu when right clicking a file. If that concept is greek to you, we might be in trouble.

Step Two - Batch File Set up

  • Directory - Create a specific directory to store all of your backups. For this example we're going to assume that you have a second drive called "D:" and that we have created a folder underneath called Backup. This is where we will be copying all the VM files as well as creating the archive folders.
  • Batch File - Create a batch file called "backup.bat". You can do this from the command line by navigating to the directory and going "notepad backup.bat" and notepad will ask if you would like to create the file. You can create a text file and rename. When you rename or create the file and view it in explorer it should look like an EXE and not a text file icon otherwise you have your extensions hidden and all you have done is created a backup.bat.txt file. If this hangs you up, drop to a command line (CMD.EXE) and do either a move backup.bat.txt backup.bat or substitute move for ren, either works.
  • Backup Log - We will be piping console output to a file called "Backup.log" in order to debug and check the execution of our backup job. You do not have to create this file, when we run the batch file it will automatically get created.

Step Three - Batch File Creation

  • Beer - Get one or two, you might need it if this makes your brain hurt. For others, continue on, once you do one host it is just a matter of rubber stamping out the rest of the hosts.
  • VM List - Drop to a command line (start --> run --> CMD) and type the following "vmrun -T server -h https://localhost:8333/sdk -u username -p password list" without the quotes of course. This will give you a list of VM's running that hopefully makes sense to you. You will want to copy and paste that list in to notepad for future reference.
  • Batch File Lines
    • Start Log: Echo %date% - %time% - Starting Email Back Process >> backup.log
      • This gives you an idea of when the process started which coupled with the stop log line you can get the delta of time it took to complete the task. Your server might run slow while this is happening but at least your VM's did not have to be suspended or stopped to complete the backup task.
    • Create Snapshot: vmrun -T server -h https://localhost:8333/sdk -u username -p password snapshot "[Standard] Virtual Machine.vmx" Backup
      • -T is for the type of VMware server.
      • Snapshot is the action
      • Backup is the name of the snapshot and can be named anything
    • Copy: robocopy "C:\Virtual Machine" "D:\Backup\email" /E /R:0 /W:0 /XF *00001.vmdk* /XF *.vmsn /XD *.lck* /XF *.vmem
      • /e for subfolders
      • /r:0 sets retries to zero
      • /w:0 sets the time between retries to zero
      • /XF excludes files, /XD excludes directories
    • Delete Snapshot: vmrun -T server -h https://localhost:8333/sdk -u username -p password deletesnapshot "[Standard] Virtual Machine.vmx" Backup
    • Compress File: 7z a -r -oD:\backup -t7z %date:~4,2%%date:~7,2%%date:~10,4%\email_%date:~4,2%%date:~7,2%%date:~10,4%.7z -v2g D:\Backup\mail\*.* -mmt -mx9
      • a and -r sets the program to archive and recursive
      • -o is the directory
      • -tyz sets the archive type to 7zip, this can be just zip
      • %date:~4,2%%date:~7,2%%date:~10,4%\email_%date:~4,2%%date:~7,2%%date:~10,4%.7z - This creates a sub directory with today's date as well as the archive with today's date
      • -v2g - Sets the volume archive size to 2gigs
      • D:\Backup\mail\*.* - Source
      • -mmt - Turns on multi-threading. This can put your machine on its knee's so determine if speed or CPU power is more important, might be a none starter if done after hours. This will default to 2 threads I believe, some of the other compression methods support more cores like bzip2
      • -mx9 - Compression level, 9 being the highest
    • Stop Log: Echo %date% - %time% - Finished Email Backup Process >> backup.log

Step Four - Putting it all together (or the EZ example)

Echo %date% - %time% - Starting Email Back Process >> backup.log
vmrun -T server -h https://localhost:8333/sdk -u username -p password snapshot "[Standard] Virtual Machine.vmx" Backup
robocopy "C:\Virtual Machine" "D:\Backup\email" /E /R:0 /W:0 /XF *00001.vmdk* /XF *.vmsn /XD *.lck* /XF *.vmem
7z a -r -oD:\backup -t7z %date:~4,2%%date:~7,2%%date:~10,4%\email_%date:~4,2%%date:~7,2%%date:~10,4%.7z -v2g D:\Backup\mail\*.* -mmt -mx9
Stop Log: Echo %date% - %time% - Finished Email Backup Process >> backup.log

You will want to make sure all of your commands are in your environment path, this might require some tweaking to get this to run with task schedule, and you might want to tweak some of the 7zip settings as needed. I had the source page that I used years ago to create this script but it is gone, even from google cache. If someone wants to email a source that closely matches this I will be more than happy to give credit where do. I've seen a lot of posts of scripts that are pretty fancy but a lot involve suspending the machine which is just not an option when the job could easily take 30-120 minutes to complete.