1===================== 2VFIO device Migration 3===================== 4 5Migration of virtual machine involves saving the state for each device that 6the guest is running on source host and restoring this saved state on the 7destination host. This document details how saving and restoring of VFIO 8devices is done in QEMU. 9 10Migration of VFIO devices consists of two phases: the optional pre-copy phase, 11and the stop-and-copy phase. The pre-copy phase is iterative and allows to 12accommodate VFIO devices that have a large amount of data that needs to be 13transferred. The iterative pre-copy phase of migration allows for the guest to 14continue whilst the VFIO device state is transferred to the destination, this 15helps to reduce the total downtime of the VM. VFIO devices can choose to skip 16the pre-copy phase of migration by returning pending_bytes as zero during the 17pre-copy phase. 18 19A detailed description of the UAPI for VFIO device migration can be found in 20the comment for the ``vfio_device_migration_info`` structure in the header 21file linux-headers/linux/vfio.h. 22 23VFIO implements the device hooks for the iterative approach as follows: 24 25* A ``save_setup`` function that sets up the migration region and sets _SAVING 26 flag in the VFIO device state. 27 28* A ``load_setup`` function that sets up the migration region on the 29 destination and sets _RESUMING flag in the VFIO device state. 30 31* A ``save_live_pending`` function that reads pending_bytes from the vendor 32 driver, which indicates the amount of data that the vendor driver has yet to 33 save for the VFIO device. 34 35* A ``save_live_iterate`` function that reads the VFIO device's data from the 36 vendor driver through the migration region during iterative phase. 37 38* A ``save_state`` function to save the device config space if it is present. 39 40* A ``save_live_complete_precopy`` function that resets _RUNNING flag from the 41 VFIO device state and iteratively copies the remaining data for the VFIO 42 device until the vendor driver indicates that no data remains (pending bytes 43 is zero). 44 45* A ``load_state`` function that loads the config section and the data 46 sections that are generated by the save functions above 47 48* ``cleanup`` functions for both save and load that perform any migration 49 related cleanup, including unmapping the migration region 50 51 52The VFIO migration code uses a VM state change handler to change the VFIO 53device state when the VM state changes from running to not-running, and 54vice versa. 55 56Similarly, a migration state change handler is used to trigger a transition of 57the VFIO device state when certain changes of the migration state occur. For 58example, the VFIO device state is transitioned back to _RUNNING in case a 59migration failed or was canceled. 60 61System memory dirty pages tracking 62---------------------------------- 63 64A ``log_global_start`` and ``log_global_stop`` memory listener callback informs 65the VFIO IOMMU module to start and stop dirty page tracking. A ``log_sync`` 66memory listener callback marks those system memory pages as dirty which are 67used for DMA by the VFIO device. The dirty pages bitmap is queried per 68container. All pages pinned by the vendor driver through external APIs have to 69be marked as dirty during migration. When there are CPU writes, CPU dirty page 70tracking can identify dirtied pages, but any page pinned by the vendor driver 71can also be written by the device. There is currently no device or IOMMU 72support for dirty page tracking in hardware. 73 74By default, dirty pages are tracked when the device is in pre-copy as well as 75stop-and-copy phase. So, a page pinned by the vendor driver will be copied to 76the destination in both phases. Copying dirty pages in pre-copy phase helps 77QEMU to predict if it can achieve its downtime tolerances. If QEMU during 78pre-copy phase keeps finding dirty pages continuously, then it understands 79that even in stop-and-copy phase, it is likely to find dirty pages and can 80predict the downtime accordingly. 81 82QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking`` 83which disables querying the dirty bitmap during pre-copy phase. If it is set to 84off, all dirty pages will be copied to the destination in stop-and-copy phase 85only. 86 87System memory dirty pages tracking when vIOMMU is enabled 88--------------------------------------------------------- 89 90With vIOMMU, an IO virtual address range can get unmapped while in pre-copy 91phase of migration. In that case, the unmap ioctl returns any dirty pages in 92that range and QEMU reports corresponding guest physical pages dirty. During 93stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped 94pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those 95mapped ranges. 96 97Flow of state changes during Live migration 98=========================================== 99 100Below is the flow of state change during live migration. 101The values in the brackets represent the VM state, the migration state, and 102the VFIO device state, respectively. 103 104Live migration save path 105------------------------ 106 107:: 108 109 QEMU normal running state 110 (RUNNING, _NONE, _RUNNING) 111 | 112 migrate_init spawns migration_thread 113 Migration thread then calls each device's .save_setup() 114 (RUNNING, _SETUP, _RUNNING|_SAVING) 115 | 116 (RUNNING, _ACTIVE, _RUNNING|_SAVING) 117 If device is active, get pending_bytes by .save_live_pending() 118 If total pending_bytes >= threshold_size, call .save_live_iterate() 119 Data of VFIO device for pre-copy phase is copied 120 Iterate till total pending bytes converge and are less than threshold 121 | 122 On migration completion, vCPU stops and calls .save_live_complete_precopy for 123 each active device. The VFIO device is then transitioned into _SAVING state 124 (FINISH_MIGRATE, _DEVICE, _SAVING) 125 | 126 For the VFIO device, iterate in .save_live_complete_precopy until 127 pending data is 0 128 (FINISH_MIGRATE, _DEVICE, _STOPPED) 129 | 130 (FINISH_MIGRATE, _COMPLETED, _STOPPED) 131 Migraton thread schedules cleanup bottom half and exits 132 133Live migration resume path 134-------------------------- 135 136:: 137 138 Incoming migration calls .load_setup for each device 139 (RESTORE_VM, _ACTIVE, _STOPPED) 140 | 141 For each device, .load_state is called for that device section data 142 (RESTORE_VM, _ACTIVE, _RESUMING) 143 | 144 At the end, .load_cleanup is called for each device and vCPUs are started 145 (RUNNING, _NONE, _RUNNING) 146 147Postcopy 148======== 149 150Postcopy migration is currently not supported for VFIO devices. 151