VMware VM migration (compute & storage)

Updated: Jul 20, 2022

vMotion Process Explained:

A migration specification is created that contains the following information:

The virtual machine that is being live-migrated

Configuration of that virtual machine (virtual hardware, VM options, etc.)

Source ESXi host

Destination ESXi host

vMotion network details

The migration specification is shared with the source and destination ESXi hosts by the vCenter Server instance, making sure that all necessary information is exchanged to start the migration process. The vCenter Server communicates with ESXi hosts using the Virtual Provisioning X Daemon (VPXD) that calls out the Virtual Provisioning X Agent (VPXA) that is running on the ESXi hosts. VPXA listens to messages from VPXD, it receives the migration spec and passes that on to the VMX process via host-d. The Host Daemon (host-d) maintains host-specific information and access for management including virtual machine telemetry like the VM state. When a migration is started, host puts the virtual machine in an intermediate state so the virtual machine its configuration cannot be changed during the migration.

Iterative Memory Pre-Copy

Page tracing is a continuous cycle. It will work towards memory pre-copy convergence by using multiple iterations. The first iteration (precopy phase-1) copies the virtual machine memory. The following iterations (precopy phase 0 to n) work on copying the dirtymemory pages.

Prepare Copy to the Pre-Copy Phase

By now, all processes and communication paths are ready for the live migration to start. The prepare phase is all about making sure that the destination ESXi host pre-allocates the compute resources for the to-be migrated virtual machine. Also, the virtualmachine is created on the destination ESXi host, but it is masked. All the information about the virtual machine configuration is already know as that is included in the migration spec.

With the prepare phase done, the process moves to the pre-copy phasewhere the memory is transferred from the source to the destination ESXi host. There is a need to trace all the virtual machine memory pages on the source ESXi host. By doing that, the vMotion process knows what memory pages are overwritten during migration, referred to as dirty pages, as it needs to re-send these memory pages to the destination host

Page Tracing

During the pre-copy phase, the vCPU’s, in use by the virtual machine, are briefly stunned to install the page tracers. The VMkernel migration module now asks VMM to start the page tracing as VMM owns the memory page table state of the virtual machine. The following diagram shows what happens when the guest OS is writing data to memory during a vMotion.

Iterative Memory Pre-Copy

As the memory pages are copied from the source to the destination ESXi host, we need to determine when all memory is copied to its destination. VMM asks the VMkernel if the pre-copy process can be terminated after each iteration. This is only possible when all memory changes (dirty pages) are copied to the destination host. Part of the iterative memory pre-copy algorithm is to match all destination memory pages to its source. Starting at page zero all the way to the maximum or last memory page number, all memory pages are sequentially checked to see if the destination pages are in sync with the source pages.

Switchover

With the memory pre-copy migration terminated by VMM, all memory pages reside on the destination ESXi host. VMM now sends a remote procedure call (RPC) to VMX that it can suspend the source virtual machine. VMX will enter the checkpoint phase where it suspends the virtual machine and sends the checkpoint data to the destination ESXi host.

In the process, the virtual machine on the destination ESXi host will be de-masked, and the state is restored using the checkpoint data. What basically happens is that the virtual machine on the destination is powered on, but the boot process is interrupted and pointed to the memory pages that are migrated from the source ESXi host. All this typically happens in 100-200ms. That is the stun time in which the virtual machine is not in running state. The duration of the stun time depends on a variety of factors like host hardware and dynamic guest workloads.

The virtual machine is now live-migrated!

VMotion Migration Compute & Storage

To successfully migrate VMs between clusters with different CPU generations, the Vmware EVC feature must be enabled to safely move powered on VMs.