Alexander’s Blog

We all know that it is fairly easy to host servers in virtual environments these days. In fact, it is so easy that people can often overlook the implications of hosting Active Directory Domain Controllers (DCs) in a virtual environment. Because Active Directory doesn’t support any method that restores snapshots of the operating system due to the fact that it can cause an Update Sequence Number (USN) rollback, one can argue that Domain Controllers are not really supported in a virtual environment.

Although I wouldn’t go as far as stating that DCs are not supported in a virtualized environment but clearly snapshots are not supported for virtualized DCs because of the way Active Directory was designed. If you decide to run DCs in a virtualized environment, here are some solutions that are documented in Microsoft’s KB article 888794. This is only a partial list. For the complete listing check out the KB article.

If the virtual hosting environment software correctly supports a SCSI emulation mode that supports forced unit access (FUA), un-buffered writes that Active Directory performs in this environment are passed to the host operating system. If forced unit access is not supported, you must disable the write cache on all volumes of the guest operating system that host the Active Directory database, the logs, and the checkpoint file.

Make sure that all the domain controllers perform inbound replication on all locally held Active Directory partitions according to the schedule defined on site links and connection objects, especially in the number of days that is specified by the tombstone lifetime attribute.

When a domain controller runs in a virtual hosting environment, do not pause the domain controller for long periods of time before you resume the operating system image. If you do pause the domain controller for a long time, replication may stop and cause lingering objects.

An Active Directory domain controller requires regular system state backups to recover from user, hardware, software, or environmental problems. The default useful life of a system state backup is 60 or 180 days, depending on the operating system version and the service pack revision at play during the installation. This useful life is controlled by the tombstone lifetime attribute in Active Directory. At least one domain controller in every domain in the forest should be backed up every tombstone lifetime number of days. In a production environment, you should make system state backups from two different DCs on a daily basis.

In an effort to boot with the latest zone contents, the Microsoft DNS Server service waits 15 or more minutes for Active Directory to inbound replicate before loading an AD-integrated DNS zone. Configuring DC guests to point to themselves as primary for name resolution causes domain controllers to hang while applying network connections during OS startup. Virtualized domain controllers should point to one or two reliable off-box DNS Servers to insure faster OS startup. Similarly, virtual host computers should point to one or two off-box DNS Servers for name resolution. Virtual host computers should not point to virtualized DNS Server running on the local virtual host computer.

To roll back the contents of Active Directory to a previous point in time, restore a valid system state backup. A system state backup can be restored up to the tombstone lifetime number of days after the backup was performed. The backup must have also been made on the same operating system installation as the operating system that you are restoring. Active Directory does not support other methods to roll back the contents of Active Directory. In particular, Active Directory does not support any method that restores a “snapshot” of the operating system or the disk volume the operating system resides on. This kind of method causes a rollback in the update sequence number (USN) used to track changes in Active Directory. When a USN rollback occurs, the contents of the Active Directory databases on the improperly restored domain controller and its replication partners may be permanently inconsistent.

If you are running servers, or plan to run servers, in a virtual environment it is good for you to know the support policy documented by Microsoft in the KB article 897615: Support policy for Microsoft software running in non-Microsoft hardware virtualization software.

The vSphere Command-Line Interface (vSphere CLI) command set allows you to run common system administration commands against ESX/ESXi systems from any machine with network access to those systems. vSphere CLI commands are especially useful for ESXi hosts because ESXi does not include a service console. You can use vicfg-cfgbackup.pl within vSphere CLI to backup or restore your VMware ESXi 4.1 server configuration or to reset the host to factory settings.

vicfg-cfgbackup

The vicfg-cfgbackup command backs up and restores ESXi configuration data. You can back up the host configuration, restore the configuration to the host, force the restore of the configuration, and reset the host to factory settings.

WARNING! This command is supported for ESXi hosts but not for ESX hosts.

vicfg-cfgbackup
<conn_options>
[--force |
--help |
--load <backupfile> |
--reset |
--save <backupfile>]

You can use the following options with vicfg-cfgbackup.

conn_options
Specifies the target server and authentication information if required. Run vicfg-cfgbackup –help for a list of all connection options.

–force | -f
Forces the restore of the configuration.

–help
Prints a help message for each command-specific and each connection option. Calling the script with no arguments or with –help has the same effect.

–load | -l <backupfile>
Restores configuration from <backupfile> onto the host.

–save | -s <backupfile>
Backs up the host configuration.

Include the number of the build that is running on the host that you are backing up in the backup filename. If you are running the vSphere CLI from vMA, the backup file is saved locally on vMA. Local storage for backup files is safe because vMA is stored in the /vmfs/volumes/<datastore> directory, which is separate from the ESXi image and configuration files.

--reset | -r
Resets the host to factory settings.

–quiet | -q
Performs all operations without prompting for confirmation.

Step-by-Step Instructions

Follow the instructions below for backing up and restoring your ESXi 4.1 server configuration from your Windows computer.

Backing Up the Configuration

1. Install vSphere CLI on your Windows computer. I like to install it on Windows 7 and manage my ESXi server using vSphere Client GUI and vSphere CLI. You can download vSphere CLI here.
2. Go to Start -> All Programs -> VMware -> VMware vSphere Client -> Command Prompt.
3. Change directory to the folder where the vicfg-cfgbackup.pl is located. This will typically be in the C:\Program Files (x86)\VMware\VMware vSphere CLI\bin folder. Obviously, if you add the path to your system path you can run the command from any folder.
4. Use the following syntax at the vCLI prompt:
   vicfg-cfgbackup.pl –server ip-address –username root –password your-password –save filename
   For example, type the following command and press Enter:
   vicfg-cfgbackup.pl –server 192.168.1.200 –username root –password MySecretPassword –save esxi41.bak
   Where 192.1681.200 is your ESXi server’s IP address, MySecretPassword is your password, and esxi41.bak is the name of the backup file that will contain the server configuration settings. Note that there is a space after each double dash (–) but not after the double dash.
5. After you run the command you should see the following notification and your file will be backed in the folder where you ran the command, e.g. in the VMware vSphere Client\bin folder.
   Saving firmware configuration to esxi4.1.bak …
   Make sure you verify that the file exists by running the dir *.bak command at the command prompt.

Restoring the Configuration

To restore your VMware ESXi 4.1 server configuration use the same vicfg-cfgbackup.pl command.

1. Go to Start -> All Programs -> VMware -> VMware vSphere Client -> Command Prompt.
2. Change directory to the folder where the vicfg-cfgbackup.pl is located. This will typically be in the C:\Program Files (x86)\VMware\VMware vSphere CLI\bin folder.
3. Use the following syntax at the vCLI prompt:
   vicfg-cfgbackup.pl –server ip-address  –username root –password your-password –load esxi41.bak -f -q
   For example, type the following command and press Enter:
   vicfg-cfgbackup.pl –server 192.168.1.200 –username root –password MySecretPassword –load esxi41.bak -f -q
   Where 192.1681.200 is your ESXi server’s IP address, MySecretPassword is your password, and esxi41.bak was the name of the backup file that contains the server configuration settings. Note that there is a space after each double dash (–) but not after the double dashes.

In order to restore your configuration, you need to place your new ESXi 4.1 server into evaluation mode. This may not be necessary if you have recently installed your server and have not added your license key. If you have already added your license key then simply use the vSphere Client and set the server to evaluation mode. Another thing to keep in mind if your restore fails is to use the -f option, which forces a restore.

Helpful Links

1. More information and documentation for vSphere CLI is available here.
2. A complete reference for vSphere CLI is available here.
3. vSphere CLI is available for download here.

Microsoft has posted a comprehensive list of Hyper-V updates for Windows Server 2008 R2 that you should check it out if you work with Hyper-V.

The posted table shows the list of software updates and hotfixes for Hyper-V in Windows Server 2008 R2. Updates that are available on Windows Update are indicated, as well as the download location for those that are available at the Microsoft Download Center. These updates and hotfixes can help avoid some known issues and may save you a support call.

Keep in mind that some updates are required only under certain circumstances, as noted in the table provided by Microsoft.

To upgrade VMware ESXi 4.0 host to ESXi 4.1, first you need to download the upgrade zip file from VMware. There are several methods to upgrade your ESXi but in my opinion using the vSphere CLI tool is by far the easiest. The versions listed in this post are the latest version available at the time of writing.

If you don’t have the vSphere CLI v4.1.0 installed, you can download it from VMware. Make sure that you download vSphere CLI, not PowerCLI. You will need the vSphere CLI utility to upgrade your ESXi host. You can install vSphere CLI utility on Windows XP and Vista. At the time of writing, Windows 7 is not supported. I installed it on Windows 7 but had problems running the utility. I used a Windows XP computer and it worked just fine for me.

When you start the utility, you may notice that it is starting in VMware\VMware vSphere CLI folder. You need to be in the Perl\bin folder where the utility resides. One solution is to add the Perl\bin folder to the path and then close and re-open the command prompt. If you have added the VMware\VMware vSphere CLI\Perl\bin to the path and are unable to execute the vihostupdate.pl utility, just simply change the directory to VMware\VMware vSphere CLI\Perl\bin and then execute the command. You can type vihostupdate.pl –help for help with the syntax.

Execute the following command to upgrade ESXi4.0 host to ESXi 4.1.

NOTE: In the blog post what may look like a hyphen is actually two dashes. Make sure you type two hyphens before the words server, install and bundle. For some reason my WordPress blog is converting the double hyphens “- -” to one solid dash “–”.

vihostupdate.pl –server Server_IP_Address –install –bundle Full_Path_to_Zip_File

For example, enter the following command.

vihostupdate.pl –server 10.0.1.200 –install –bundle “C:\Source Files\VMware\ESXi 4.1\upgrade-from-ESXi4.0-to-4.1.0-0.0.260247-release.zip”

where 10.10.1.200 is the IP address of the ESXi server. If the path includes spaces then use quotes, as shown in the example above. Do not enter the two dashes “–” before the path. Your command will look something like this.

C:\Program Files\VMware\VMware vSphere CLI\Perl\bin>vihostupdate.pl –server 10.0.1.200 –install –bundle “C:\Source Files\VMware\ESXi 4.1\upgrade-from-ESXi4.0-to-4.1.0-0.0.260247-release.zip”
Enter username: root
Enter password:
Please wait patch installation is in progress …
The update completed successfully, but the system needs to be rebooted for the changes to be effective.

Note: Your password won’t be displayed. After you press enter it can take a very long time to install the software so be patient.

Verify the bulletins are installed on the ESXi host by running the following command.

vihostupdate.pl –server 10.0.1.200 –query

C:\Program Files\VMware\VMware vSphere CLI\Perl\bin>vihostupdate.pl –server 10.0.1.200 –query
Enter username: root
Enter password:
———Bulletin ID——— —–Installed—– —————-Summary—————–
ESXi410-GA                    2010-12-11T02:24:28 ESXi upgrade Bulletin
ESXi410-GA-esxupdate          2010-12-11T02:24:28 ESXi pre-upgrade Bulletin

You will be prompted to enter the username and password again.

Reboot the ESXi host by using the reboot command.

vicfg-hostops.pl –server 10.0.1.200 –o reboot

The letter before reboot is O, not zero. You will be notified that the host rebooted successfully.

Congratulations! You have upgraded your ESXi 4.0 host to ESXi 4.1 successfully.

Copyright ©2010 Zubair Alexander. All rights reserved.

Snapshots must be one of the nicest features in Hyper-V. Unfortunately, too many people are confused…..justifiably, in my opinion…. about how exactly the snapshots work in Hyper-V. I am not talking about taking snapshots and restoring to a previous snapshot. That’s simple enough even for a ten year old kid. I am talking about how differencing disks work, why sometimes when you delete snapshots the disk is merged while other times it is not, what exactly is saved in a snapshot folder, how to save an image so it consists of only one portable VHD file, what should you do if you want to save an image and load it on another computer but the image has one or more snapshots, what happens to the virtual machine’s settings and how can you back them up…….and a gazillion other similar questions that come to mind when working with Hyper-V virtual machines.

Well, I can’t cover all the possible scenarios in this article so I will discuss some fundamental concepts and explain how snapshots work behind the scenes and how you can deal with an image that have snapshots attached to them, especially if you intend to backup and load that image to another computer.

Under the Covers

Microsoft Virtualization Program Manager Ben Armstrong has documented how snapshots work “under the covers.” I have the pleasure of having Ben speak at my Windows Networking User Group in Seattle. Known in the industry as the Virtual PC guy, he is a recognized authority on virtualization. Here’s is what he has documented in his blog.

When you take a snapshot of a running virtual machine:

1. We pause the virtual machine.
2. We create a new differencing disk for each virtual hard disk and hook it up to the virtual machine.
3. We make a copy of the virtual machine’s configuration file.
4. We resume the virtual machine (note that this happens so quickly that you do not know that the virtual machine was paused at all.
5. Once the virtual machine is running again we start saving the contents of the virtual machine’s memory to disk.
6. While this is happening we monitor memory activity inside the virtual machine and if the the guest operating system attempts to modify memory that we have not copied, we intercept the write attempt and copy the original memory contents then allow the write to go through.

Once the snapshot is completed we have a set of the virtual machine configuration file, the virtual machine saved state files and the snapshot differencing disks (.AVHDs) stored in a folder under the virtual machine’s snapshot directory.

This is now treated as a read-only “point in time” image of a virtual machine.  You can apply a virtual machine, in which case we set the virtual machine to start from a copy of the snapshot.  You can change the virtual machine settings after applying a snapshot – but you cannot change the snapshot settings themselves.

You can also delete a snapshot.  If you delete a snapshot that has no descendants (snapshot with differencing disks that reference the snapshot being deleted) then the files associated with the snapshot will just be deleted.  If you delete a snapshot with only one descendant the configuration and saved state files for the snapshot will be deleted and the snapshot differencing disks will be merged with those of it’s descendant.  If you delete a snapshot with more than one descendant the snapshot configuration and saved state files will be deleted – but the differencing disks will not be merged until the number of descendant snapshots is reduced to one.

A Closer Look at the Snapshots

To elaborate on what Ben has documented, let’s look at an example. Let’s say you have a virtual machine with several snapshots. You would like to make a backup of this virtual machine and load it on another computer. The folder structure of the virtual image looks like this.

Notice there is a Snapshots folder and a Virtual Machines folder. There is one .VHD file that has a date of 9/3/2009 (more than a year old). There are  seven .AVHD (snapshot differencing disks) files, each with a date that corresponds to the time when the snapshot was taken. The Snapshots folder looks like this.

This structure contains a bunch of XML files and additional folders labeled with GUID numbers. If you look inside the folders they contain .BIN and .VSV files.

Finally, the Virtual Machines folder looks like this.

This folder structure looks very similar to the Snapshots folder except that it only has one XML file and one GUID folder that contains a single .BIN and .VSV file.

At this point if you were to delete the oldest snapshot and then turn off the virtual machine, you will notice that the size and the date of the.VHD file that you need to export to another machine will still have the old date. The reason being you deleted a snapshot that has more than one descendant. Although the snapshot that you deleted will have its configuration and saved state files deleted but the differencing disk is not going to be merged with the .VHD file until you are left with only one snapshot. Even if you delete the second, third, fourth, fifth and sixth snapshot, the differencing disks (.AVHD files) will still be there until the merge takes place. Therefore, if you want to get rid of all the snapshots (.AVHD files) and be left with only one .VHD file, you will have to delete all the snapshots and then turn the virtual machine off.

NOTE: Even if you delete all the snapshots and then shut down the virtual machine, your differencing disks (.AVHD files) will not be merged. You must TURN OFF the virtual machine, not shut it down.

Once you have deleted all the snapshots and then turn off the virtual machine, the differencing disks will be merged at that time. Depending on the size of the differencing disks this may take several hours. The status in the Hyper-V console will display the percentage of completion.

Exporting the Image

To make a copy of the virtual machine so you can load it on another computer, you can export the virtual machine. The virtual machine must be turned off before you can export the image. Even if you have a bunch of snapshot, you can still turn the computer off and export it. The entire file structure, including the snapshots, will be backed up in the exported folder. For example, create a folder called Exported Hyper-V Images. Do not create sub-folders in this folder because when you export an image it automatically creates a folder for the image with its name. You will an XML file in the root of the folder structure plus three other folders.

1. Snapshots (contains snapshots, unless you don’t have any, in which case this folder will be empty)
2. Virtual Hard Disks (contains a .VHD file)
3. Virtual Machines (contains a .EXP file)

You can simply copy this entire folder structure to another computer and load the image. All your snapshots will be in tact, along with the virtual machine settings.

#CLUSTER-INVARIANT#:{GUID}

On a side note, you may have noticed a strange entry in the notes in the Name section of your Hyper-V settings. It’s not something that you added. The entry adds the #CLUSTER-INVARIANT# after the notes so it looks something like this:

Corporate Server image used for training and testing.#CLUSTER-INVARIANT#:{abba09d4a-e956-4367-8357-f46bd7ea6d4}

DO NOT delete or edit this entry after the notes. The Virtual Machine Manager (VMM) adds these values to the notes field so that the system can associate a specific virtual machine with VMM. If you move a VMM from one host to another without VMM having any knowledge of the move, these fields are used to reassociate the VM.

Hopefully, you have a better understanding of how snapshots work behind the scenes and how to better manage them. There’s much more to managing Hyper-V images. I plan to address other issues at a later time, such as how to manage the networking aspects of Hyper-V.