KVM: migrating a VM

---

Introduction

KVM currently supports savevm/loadvm and offline or live migration
Migration commands are given when in qemu-monitor (Alt-Ctrl-2).
Upon successful completion, the migrated VM continues to run on the destination host.

Note

You can migrate a guest between an AMD host to an Intel host and back. Naturally, a 64-bit guest can only be migrated to a 64-bit host, but a 32-bit guest can be migrated at will.

There are some older Intel processors which don't support NX (or XD), which may cause problems in a cluster which includes NX-supporting hosts. We may add a feature to hide NX if this proves to be a problem in actual deployments.

Requirements

• The VM image is accessible on both source and destination hosts (located on a shared storage, e.g. using nfs).
  • It is recommended an images-directory would be found on the same path on both hosts (for migrations of a copy-on-write image -- an image created on top of a base-image using "qemu-image create -b ...")
• The src and dst hosts must be on the same subnet (keeping guest's network when tap is used).
• Do not use -snapshot qemu command line option.
• For tcp:// migration protocol
  • the guest on the destination must be started the same way it was started on the source.
• For ssh:// migration protocol
  • it is recommended to use vnc (not SDL)
  • redirect nothing to/from stdin/stdout.
  • ssh-keys have been exchanged between src and dst hosts (otherwise the user must type the password).
  • kvm/qemu executable (e.g. /usr/bin/kvm) and data-directory (e.g. /usr/kvm/share/qemu) can be found on both hosts using the same (full) path.

highlights / merits

• Almost unnoticeable guest down time

• Guest is not involved (unique to KVM Live Migration 1)

• Capability to tunnel VM state through an external program (unique to KVM Live Migration 1)

  • ssh/gzip/bzip2/gpg/your own

• Built-in security using ssh (unique to KVM Live Migration 1)

• Upon success guest continues to run on destination host, upon failure guest continues to run on source host (with one exception)
• Short and Simple
• Easy to enhance
• Hardware independence (almost).
• Support for migration of stopped (paused) VMs.
• Open

1 These features are unique to KVM Live Migration as far as I know. If you know of other hypervisor that support any of them please update this page or let me (Uri) know.

User Interface

The user interface is through the qemu monitor (alt-ctrl-2 on the SDL window)

Management

migrate [-d] <URI>
migrate_cancel

The '-d' will let you query the status of the migration.

With no '-d' the monitor prompt returns when the migration completes. URI can be one of 'exec:<command>', 'ssh://<ip>' or tcp://<ip:port>

Status

info migration

Migration Parameters

migrate_set_speed <speed>   set bandwidth control parameters (max transfer rate per second)

Example / HOWTO

• A is the source host, B is the destination host:

  • TCP example:

    1. Start the VM on B with the exact same parameters as the VM on A, in migration-listen mode (-incoming tcp://0:4444 (or other PORT))
    2. Start the migration (always on the source host):

       A: migrate -d tcp://B:4444 (or other PORT)
       

    3. Check the status (on A only):

       info migration
       

  • SSH example:

    • Note: It is recommended not to use SDL when using SSH.
    • Start the migration

      A: migrate ssh://B
      

  • Offline example:

    1. unlimit bandwidth used for migration:

       A: migrate_set_speed 1g
       

    2. stop the guest:

       A: stop
       

    3. continue with either TCP or SSH migration as described above.

Problems / Todo

• TSC offset on the new host must be set in such a way that the guest sees a monotonically increasing TSC, otherwise the guest may hang indefinitely after migration.
• usbdevice tablet complains after migration.
• handle migration completion better (especially when network problems occur).
• More informative status.
• Migration does not work while CPU real-mode/protected mode are still changing.

savevm/loadvm to an external state file (using pseudo-migration)

• let the state file be /saved_images/VM.STATE
• savevm (qemu monitor):

  • stop
    migrate file:///saved_images/VM.STATE
    

• loadvm (qemu command line and qemu monitor):

  • <qemu-command-line> -incoming file:///saved_images/VM.STATE
    cont
    

more exec: options

• To be supported directly by Migration Protocols, but until then...
• Save VM state into a compressed file
  • Save

    • stop
      migrate_set_speed 4095m
      migrate "exec:gzip -c > STATEFILE.gz"
      

  • Load

    • gzip -c -d STATEFILE.gz | <qemu-command-line> -incoming stdio
      

• Save VM State into an encrypted file (assumes KEY has already been generated)
  • Save

    • stop
      migrate_set_speed 4095m
      migrate "exec:gpg -q -e -r KEY -o STATFILE.gpg"
      

  • Load VM state from an encrypted file

    • gpg -q -d -r KEY STATEFILE.gpg | <qemu-command-line> -incoming stdio
      

• Send encrypted VM state from host A to host B (Note: ssh is probably better, this is just for show)
  • on host A

    • migrate "exec:gpg -q -e -r KEY | nc B 4444"
      

  • on host B

    • nc -l 4444 | gpg -q -d -r KEY | <qemu-command-line> -incoming stdio
      

Algorithm (the short version)

1. Setup
   • Start guest on destination, connect, enable dirty page logging and more
2. Transfer Memory
   • Guest continues to run
   • Bandwidth limitation (controlled by the user)
   • First transfer the whole memory
   • Iteratively transfer all dirty pages (pages that were written to by the guest).
3. Stop the guest
   • And sync VM image(s) (guest's hard drives).
4. Transfer State
   • As fast as possible (no bandwidth limitation)
   • All VM devices' state and dirty pages yet to be transferred
5. Continue the guest
   • On destination upon success
     • Broadcast "I'm over here" Ethernet packet to announce new location of NIC(s).
   • On source upon failure (with one exception).

Instructions for kvm-13 and below: MigrationQemu0.8.2.