release-notes.md

- [v54.0](#v540)
    - [Live Migration Version Compatibility](#live-migration-version-compatibility)
    - [VFIO Device Live Migration](#vfio-device-live-migration)
    - [Live Migration Cancellation](#live-migration-cancellation)
    - [Event Monitor Improvements](#event-monitor-improvements)
    - [Emulated ARM SMMUv3 (Experimental)](#emulated-arm-smmuv3-experimental)
    - [Flat VMDK Disk Image Support](#flat-vmdk-disk-image-support)
    - [Balloon Statistics](#balloon-statistics)
    - [`virtio-console` Over a Unix Socket](#virtio-console-over-a-unix-socket)
    - [AArch64 Boot Improvements](#aarch64-boot-improvements)
    - [Faster Memory Prefault](#faster-memory-prefault)
    - [VM Generation ID Device](#vm-generation-id-device)
    - [Snapshot/Restore and Live Migration Improvements](#snapshotrestore-and-live-migration-improvements)
    - [Device and Platform Improvements](#device-and-platform-improvements)
    - [New Command-Line and API Options](#new-command-line-and-api-options)
    - [Changed Default Behavior](#changed-default-behavior)
    - [Deprecations and Removals](#deprecations-and-removals)
    - [Security Hardening](#security-hardening)
    - [Notable Bug Fixes](#notable-bug-fixes)
    - [Contributors](#contributors)
- [v53.0](#v530)
    - [Snapshot/Restore and Live Migration Improvements](#snapshotrestore-and-live-migration-improvements-1)
    - [Nested Hyper-V (Windows Guest) Support](#nested-hyper-v-windows-guest-support)
    - [`virtio-rtc` Device Support](#virtio-rtc-device-support)
    - [Guest Clock Updates Across Snapshot/Restore and Migration](#guest-clock-updates-across-snapshotrestore-and-migration)
    - [Buffered Serial Output for Late-Connecting Clients](#buffered-serial-output-for-late-connecting-clients)
    - [New Command-Line and API Options](#new-command-line-and-api-options-1)
    - [Deprecations](#deprecations)
    - [Notable Bug Fixes](#notable-bug-fixes-1)
    - [Contributors](#contributors-1)
- [v52.0](#v520)
    - [Security Fixes](#security-fixes)
    - [Confidential VM Support: KVM SEV-SNP](#confidential-vm-support-kvm-sev-snp)
    - [VFIO Device Passthrough via `iommufd` and `vfio-cdev`](#vfio-device-passthrough-via-iommufd-and-vfio-cdev)
    - [Multi-Connection TCP Live Migration](#multi-connection-tcp-live-migration)
    - [Userfaultfd Demand-Paged Snapshot Restore](#userfaultfd-demand-paged-snapshot-restore)
    - [Configurable PCI BDFs for Devices](#configurable-pci-bdfs-for-devices)
    - [AIO Block Backend `write_zeroes` and `punch_hole` Support](#aio-block-backend-write_zeroes-and-punch_hole-support)
    - [Asynchronous QCOW2 Backend with `io_uring`](#asynchronous-qcow2-backend-with-io_uring)
    - [Generic `vhost-user` Device Support](#generic-vhost-user-device-support)
    - [Core Scheduling for vCPU Threads](#core-scheduling-for-vcpu-threads)
    - [Snapshot/Restore and Live Migration Improvements](#snapshotrestore-and-live-migration-improvements-2)
    - [New Command-Line and API Options](#new-command-line-and-api-options-2)
    - [MSHV Support Improvements](#mshv-support-improvements)
    - [Deprecations](#deprecations-1)
    - [Notable Bug Fixes](#notable-bug-fixes-2)
    - [Contributors](#contributors-2)
- [v51.0](#v510)
    - [Security Fixes](#security-fixes-1)
    - [Significant QCOW2 v3 Improvements](#significant-qcow2-v3-improvements)
    - [ACPI Generic Initiator Support](#acpi-generic-initiator-support)
    - [Block Device DISCARD and WRITE\_ZEROES Support](#block-device-discard-and-write_zeroes-support)
    - [Notable Performance Improvements](#notable-performance-improvements)
    - [MSHV Support Improvements](#mshv-support-improvements-1)
    - [Notable Bug Fixes](#notable-bug-fixes-3)
    - [Contributors](#contributors-3)
- [v50.0](#v500)
    - [Configurable Nested Virtualization Option on x86\_64](#configurable-nested-virtualization-option-on-x86_64)
    - [Compression Support for QCOW2](#compression-support-for-qcow2)
    - [Notable Performance Improvements](#notable-performance-improvements-1)
    - [Live Disk Resizing Support for Raw Images](#live-disk-resizing-support-for-raw-images)
    - [Developer Experience Improvements](#developer-experience-improvements)
    - [Improved File-level Locking Support](#improved-file-level-locking-support)
    - [Logging Improvements](#logging-improvements)
    - [Notable Bug Fixes](#notable-bug-fixes-4)
    - [Contributors](#contributors-4)
- [v49.0](#v490)
    - [MSHV Support Improvements](#mshv-support-improvements-2)
    - [Logging Improvements](#logging-improvements-1)
    - [Removed Default IP and Mask for `virtio-net` Devices](#removed-default-ip-and-mask-for-virtio-net-devices)
    - [Notable Bug Fixes](#notable-bug-fixes-5)
    - [Contributors](#contributors-5)
- [v48.0](#v480)
    - [Experimental `fw_cfg` Device Support](#experimental-fw_cfg-device-support)
    - [Experimental `ivshmem` Device Support](#experimental-ivshmem-device-support)
    - [Firmware Boot Support on `riscv64`](#firmware-boot-support-on-riscv64)
    - [Increased vCPU Limit on x86\_64/kvm](#increased-vcpu-limit-on-x86_64kvm)
    - [Improved Block Performance with Small Block Sizes](#improved-block-performance-with-small-block-sizes)
    - [Faster VM Pause Operation](#faster-vm-pause-operation)
    - [Updated Documentation on Windows Guest Support](#updated-documentation-on-windows-guest-support)
    - [Policy on AI Generated Code](#policy-on-ai-generated-code)
    - [Removed SGX Support](#removed-sgx-support)
    - [Notable Bug Fixes](#notable-bug-fixes-6)
    - [Contributors](#contributors-6)
- [v47.0](#v470)
    - [Block Device Error Reporting to the Guest](#block-device-error-reporting-to-the-guest)
    - [Nice Error Messages on Exit](#nice-error-messages-on-exit)
    - [Alphabetically Sorted CLI Options for ch-remote](#alphabetically-sorted-cli-options-for-ch-remote)
    - [Notable Bug Fixes](#notable-bug-fixes-7)
    - [Deprecations](#deprecations-2)
    - [Contributors](#contributors-7)
- [v46.0](#v460)
    - [File-level Locking Support with `--disk`](#file-level-locking-support-with---disk)
    - [Improved Error Reporting with VM Resizing](#improved-error-reporting-with-vm-resizing)
    - [IPv6 Address Support with `--net`](#ipv6-address-support-with---net)
    - [Experimental AArch64 Support with the MSHV Hypervisor](#experimental-aarch64-support-with-the-mshv-hypervisor)
    - [Deprecated SGX Support](#deprecated-sgx-support)
    - [Notable Bug Fixes](#notable-bug-fixes-8)
    - [Contributors](#contributors-8)
- [v45.0](#v450)
    - [Experimental `riscv64` Architecture Support](#experimental-riscv64-architecture-support)
    - [Alphabetically Sorted CLI Options](#alphabetically-sorted-cli-options)
    - [Improved Downtime of VM Live Migration](#improved-downtime-of-vm-live-migration)
    - [Notable Bug Fixes](#notable-bug-fixes-9)
    - [Contributors](#contributors-9)
- [v44.0](#v440)
    - [Configurable `virtio-iommu` Address Width](#configurable-virtio-iommu-address-width)
    - [Notable Performance Improvements](#notable-performance-improvements-2)
    - [New Fuzzers](#new-fuzzers)
    - [Notable Bug Fixes](#notable-bug-fixes-10)
    - [Contributors](#contributors-10)
- [v43.0](#v430)
    - [Live Migration over TCP Connections](#live-migration-over-tcp-connections)
    - [Notable Performance Improvements](#notable-performance-improvements-3)
    - [Notable Bug Fixes](#notable-bug-fixes-11)
    - [Contributors](#contributors-11)
- [v42.0](#v420)
    - [SVE/SVE2 Support on AArch64](#svesve2-support-on-aarch64)
    - [Notable Bug Fixes](#notable-bug-fixes-12)
    - [Sponsorships](#sponsorships)
    - [Contributors](#contributors-12)
- [v41.0](#v410)
    - [Experimental "Pvmemcontrol" Support](#experimental-pvmemcontrol-support)
    - [Sandboxing With Landlock Support](#sandboxing-with-landlock-support)
    - [Notable Performance Improvements](#notable-performance-improvements-4)
    - [Notable Bug Fixes](#notable-bug-fixes-13)
    - [Contributors](#contributors-13)
- [v40.0](#v400)
    - [Support for Restoring File Descriptor Backed Network Devices](#support-for-restoring-file-descriptor-backed-network-devices)
    - [Notable Bug Fixes](#notable-bug-fixes-14)
    - [Contributors](#contributors-14)
- [v39.0](#v390)
    - [Variable Sizing of PCI Apertures for Segments](#variable-sizing-of-pci-apertures-for-segments)
    - [Direct Booting with bzImages](#direct-booting-with-bzimages)
    - [Support for NVIDIA GPUDirect P2P Support](#support-for-nvidia-gpudirect-p2p-support)
    - [Guest NMI Injection Support](#guest-nmi-injection-support)
    - [Notable Bug Fixes](#notable-bug-fixes-15)
    - [Contributors](#contributors-15)
- [v38.0](#v380)
    - [Group Rate Limiter on Block Devices](#group-rate-limiter-on-block-devices)
    - [CPU Pinning Support for Block Device Worker Thread](#cpu-pinning-support-for-block-device-worker-thread)
    - [Optimized Boot Time with Parallel Memory Prefault](#optimized-boot-time-with-parallel-memory-prefault)
    - [New 'debug-console' Device](#new-debug-console-device)
    - [Improved VFIO Device Support](#improved-vfio-device-support)
    - [Extended CPU Affinity Support](#extended-cpu-affinity-support)
    - [Notable Bug Fixes](#notable-bug-fixes-16)
    - [Contributors](#contributors-16)
- [v37.0](#v370)
    - [Long Term Support (LTS) Release](#long-term-support-lts-release)
    - [Multiple PCI segments Support for 32-bit VFIO devices](#multiple-pci-segments-support-for-32-bit-vfio-devices)
    - [Configurable Named TAP Devices](#configurable-named-tap-devices)
    - [TTY Output from Both Serial Device and Virtio Console](#tty-output-from-both-serial-device-and-virtio-console)
    - [Faster VM Restoration from Snapshots](#faster-vm-restoration-from-snapshots)
    - [Notable Bug Fixes](#notable-bug-fixes-17)
    - [Contributors](#contributors-17)
- [v36.0](#v360)
    - [Command Line Changes](#command-line-changes)
    - [Enabled Features Reported via API Endpoint and CLI](#enabled-features-reported-via-api-endpoint-and-cli)
    - [NUMA Support for PCI segments](#numa-support-for-pci-segments)
    - [CPU Topology Support on AMD Platforms](#cpu-topology-support-on-amd-platforms)
    - [Unix Socket Backend for Serial Port](#unix-socket-backend-for-serial-port)
    - [AIO Backend for Block Devices](#aio-backend-for-block-devices)
    - [Documentation Improvements](#documentation-improvements)
    - [Notable Bug Fixes](#notable-bug-fixes-18)
    - [Contributors](#contributors-18)
- [v35.0](#v350)
    - [`virtio-vsock` Support for Linux Guest Kernel v6.3+](#virtio-vsock-support-for-linux-guest-kernel-v63)
    - [User Specified Serial Number for `virtio-block`](#user-specified-serial-number-for-virtio-block)
    - [vCPU TSC Frequency Included in Migration State](#vcpu-tsc-frequency-included-in-migration-state)
    - [Notable Bug Fixes](#notable-bug-fixes-19)
    - [Contributors](#contributors-19)
- [v34.0](#v340)
    - [Paravirtualised Panic Device Support](#paravirtualised-panic-device-support)
    - [Improvements to VM Core Dump](#improvements-to-vm-core-dump)
    - [QCOW2 Support for Backing Files](#qcow2-support-for-backing-files)
    - [Minimum Host Kernel Bump](#minimum-host-kernel-bump)
    - [Notable Bug Fixes](#notable-bug-fixes-20)
    - [Contributors](#contributors-20)
- [v33.0](#v330)
    - [D-Bus based API](#d-bus-based-api)
    - [Expose Host CPU Cache Details for AArch64](#expose-host-cpu-cache-details-for-aarch64)
    - [Notable Bug Fixes](#notable-bug-fixes-21)
    - [Contributors](#contributors-21)
- [v32.0](#v320)
    - [Increased PCI Segment Limit](#increased-pci-segment-limit)
    - [API Changes](#api-changes)
    - [Notable Bug Fixes](#notable-bug-fixes-22)
    - [Contributors](#contributors-22)
- [v31.1](#v311)
- [v31.0](#v310)
    - [Update to Latest `acpi_tables`](#update-to-latest-acpi_tables)
    - [Update Reference Kernel to 6.2](#update-reference-kernel-to-62)
    - [Improvements on Console `SIGWINCH` Handler](#improvements-on-console-sigwinch-handler)
    - [Remove Directory Support from `MemoryZoneConfig::file`](#remove-directory-support-from-memoryzoneconfigfile)
    - [Documentation Improvements](#documentation-improvements-1)
    - [Notable Bug Fixes](#notable-bug-fixes-23)
    - [Contributors](#contributors-23)
- [v30.0](#v300)
    - [Command Line Changes for Reduced Binary Size](#command-line-changes-for-reduced-binary-size)
    - [Basic vfio-user Server Support](#basic-vfio-user-server-support)
    - [Heap Profiling Support](#heap-profiling-support)
    - [Documentation Improvements](#documentation-improvements-2)
    - [Notable Bug Fixes](#notable-bug-fixes-24)
    - [Contributors](#contributors-24)
- [v28.2](#v282)
- [v29.0](#v290)
    - [Release Binary Supports Both MSHV and KVM](#release-binary-supports-both-mshv-and-kvm)
    - [Snapshot/Restore and Live Migration Improvements](#snapshotrestore-and-live-migration-improvements-3)
    - [Heap Allocation Improvements](#heap-allocation-improvements)
    - [`ch-remote` Improvements](#ch-remote-improvements)
    - [`AArch64` Documentation Integration](#aarch64-documentation-integration)
    - [`virtio-block` Counters Enhancement](#virtio-block-counters-enhancement)
    - [TCP Offload Control](#tcp-offload-control)
    - [Notable Bug Fixes](#notable-bug-fixes-25)
    - [Removals](#removals)
    - [Deprecations](#deprecations-3)
    - [Contributors](#contributors-25)
- [v28.1](#v281)
- [v28.0](#v280)
    - [Community Engagement (Reminder)](#community-engagement-reminder)
    - [Long Term Support (LTS) Release](#long-term-support-lts-release-1)
    - [Virtualised TPM Support](#virtualised-tpm-support)
    - [Transparent Huge Page Support](#transparent-huge-page-support)
    - [README Quick Start Improved](#readme-quick-start-improved)
    - [Notable Bug Fixes](#notable-bug-fixes-26)
    - [Removals](#removals-1)
    - [Contributors](#contributors-26)
- [v27.0](#v270)
    - [Community Engagement](#community-engagement)
    - [Prebuilt Packages](#prebuilt-packages)
    - [Network Device MTU Exposed to Guest](#network-device-mtu-exposed-to-guest)
    - [Boot Tracing](#boot-tracing)
    - [Simplified Build Feature Flags](#simplified-build-feature-flags)
    - [Asynchronous Kernel Loading](#asynchronous-kernel-loading)
    - [GDB Support for AArch64](#gdb-support-for-aarch64)
    - [Notable Bug Fixes](#notable-bug-fixes-27)
    - [Deprecations](#deprecations-4)
    - [Contributors](#contributors-27)
- [v26.0](#v260)
    - [SMBIOS Improvements via `--platform`](#smbios-improvements-via---platform)
    - [Unified Binary MSHV and KVM Support](#unified-binary-mshv-and-kvm-support)
    - [Notable Bug Fixes](#notable-bug-fixes-28)
    - [Deprecations](#deprecations-5)
    - [Removals](#removals-2)
    - [Contributors](#contributors-28)
- [v25.0](#v250)
    - [`ch-remote` Improvements](#ch-remote-improvements-1)
    - [VM "Coredump" Support](#vm-coredump-support)
    - [Notable Bug Fixes](#notable-bug-fixes-29)
    - [Removals](#removals-3)
    - [Contributors](#contributors-29)
- [v24.0](#v240)
    - [Bypass Mode for `virtio-iommu`](#bypass-mode-for-virtio-iommu)
    - [Ensure Identifiers Uniqueness](#ensure-identifiers-uniqueness)
    - [Sparse Mmap support](#sparse-mmap-support)
    - [Expose Platform Serial Number](#expose-platform-serial-number)
    - [Notable Bug Fixes](#notable-bug-fixes-30)
    - [Notable Improvements](#notable-improvements)
    - [Deprecations](#deprecations-6)
    - [New on the Website](#new-on-the-website)
    - [Contributors](#contributors-30)
- [v23.1](#v231)
- [v23.0](#v230)
    - [vDPA Support](#vdpa-support)
    - [Updated OS Support list](#updated-os-support-list)
    - [`AArch64` Memory Map Improvements](#aarch64-memory-map-improvements)
    - [`AMX` Support](#amx-support)
    - [Notable Bug Fixes](#notable-bug-fixes-31)
    - [Deprecations](#deprecations-7)
    - [Contributors](#contributors-31)
- [v22.1](#v221)
- [v22.0](#v220)
    - [GDB Debug Stub Support](#gdb-debug-stub-support)
    - [`virtio-iommu` Backed Segments](#virtio-iommu-backed-segments)
    - [Before Boot Configuration Changes](#before-boot-configuration-changes)
    - [`virtio-balloon` Free Page Reporting](#virtio-balloon-free-page-reporting)
    - [Support for Direct Kernel Booting with TDX](#support-for-direct-kernel-booting-with-tdx)
    - [PMU Support for AArch64](#pmu-support-for-aarch64)
    - [Documentation Under CC-BY-4.0 License](#documentation-under-cc-by-40-license)
    - [Deprecation of "Classic" `virtiofsd`](#deprecation-of-classic-virtiofsd)
    - [Notable Bug Fixes](#notable-bug-fixes-32)
    - [Contributors](#contributors-32)
- [v21.0](#v210)
    - [Efficient Local Live Migration (for Live Upgrade)](#efficient-local-live-migration-for-live-upgrade)
    - [Recommended Kernel is Now 5.15](#recommended-kernel-is-now-515)
    - [Notable Bug fixes](#notable-bug-fixes-33)
    - [Contributors](#contributors-33)
- [v20.2](#v202)
- [v20.1](#v201)
- [v20.0](#v200)
    - [Multiple PCI segments support](#multiple-pci-segments-support)
    - [CPU pinning](#cpu-pinning)
    - [Improved VFIO support](#improved-vfio-support)
    - [Safer code](#safer-code)
    - [Extended documentation](#extended-documentation)
    - [Notable bug fixes](#notable-bug-fixes-34)
    - [Contributors](#contributors-34)
- [v19.0](#v190)
    - [Improved PTY handling for serial and `virtio-console`](#improved-pty-handling-for-serial-and-virtio-console)
    - [PCI boot time optimisations](#pci-boot-time-optimisations)
    - [Improved TDX support](#improved-tdx-support)
    - [Live migration enhancements](#live-migration-enhancements)
    - [`virtio-mem` support with `vfio-user`](#virtio-mem-support-with-vfio-user)
    - [AArch64 for `virtio-iommu`](#aarch64-for-virtio-iommu)
    - [Notable bug fixes](#notable-bug-fixes-35)
    - [Contributors](#contributors-35)
- [v18.0](#v180)
    - [Experimental User Device (`vfio-user`) support](#experimental-user-device-vfio-user-support)
    - [Migration support for `vhost-user` devices](#migration-support-for-vhost-user-devices)
    - [VHDX disk image support](#vhdx-disk-image-support)
    - [Device pass through on MSHV hypervisor](#device-pass-through-on-mshv-hypervisor)
    - [AArch64 for support `virtio-mem`](#aarch64-for-support-virtio-mem)
    - [Live migration on MSHV hypervisor](#live-migration-on-mshv-hypervisor)
    - [AArch64 CPU topology support](#aarch64-cpu-topology-support)
    - [Power button support on AArch64](#power-button-support-on-aarch64)
    - [Notable bug fixes](#notable-bug-fixes-36)
    - [Contributors](#contributors-36)
- [v17.0](#v170)
    - [ARM64 NUMA support using ACPI](#arm64-numa-support-using-acpi)
    - [`Seccomp` support for MSHV backend](#seccomp-support-for-mshv-backend)
    - [Hotplug of `macvtap` devices](#hotplug-of-macvtap-devices)
    - [Improved SGX support](#improved-sgx-support)
    - [Inflight tracking for `vhost-user` devices](#inflight-tracking-for-vhost-user-devices)
    - [Notable bug fixes](#notable-bug-fixes-37)
    - [Contributors](#contributors-37)
- [v16.0](#v160)
    - [Improved live migration support](#improved-live-migration-support)
    - [Improved `vhost-user` support](#improved-vhost-user-support)
    - [ARM64 ACPI and UEFI support](#arm64-acpi-and-uefi-support)
    - [Notable bug fixes](#notable-bug-fixes-38)
    - [Removed functionality](#removed-functionality)
    - [Contributors](#contributors-38)
- [v15.0](#v150)
    - [Version numbering and stability guarantees](#version-numbering-and-stability-guarantees)
    - [Network device rate limiting](#network-device-rate-limiting)
    - [Support for runtime control of `virtio-net` guest offload](#support-for-runtime-control-of-virtio-net-guest-offload)
    - [`--api-socket` supports file descriptor parameter](#--api-socket-supports-file-descriptor-parameter)
    - [Bug fixes](#bug-fixes)
    - [Deprecations](#deprecations-8)
    - [Contributors](#contributors-39)
- [v0.14.1](#v0141)
- [v0.14.0](#v0140)
    - [Structured event monitoring](#structured-event-monitoring)
    - [MSHV improvements](#mshv-improvements)
    - [Improved aarch64 platform](#improved-aarch64-platform)
    - [Updated hotplug documentation](#updated-hotplug-documentation)
    - [PTY control for serial and `virtio-console`](#pty-control-for-serial-and-virtio-console)
    - [Block device rate limiting](#block-device-rate-limiting)
    - [Deprecations](#deprecations-9)
    - [Contributors](#contributors-40)
- [v0.13.0](#v0130)
    - [Wider VFIO device support](#wider-vfio-device-support)
    - [Improved huge page support](#improved-huge-page-support)
    - [MACvTAP support](#macvtap-support)
    - [VHD disk image support](#vhd-disk-image-support)
    - [Improved Virtio device threading](#improved-virtio-device-threading)
    - [Clean shutdown support via synthetic power button](#clean-shutdown-support-via-synthetic-power-button)
    - [Contributors](#contributors-41)
- [v0.12.0](#v0120)
    - [ARM64 enhancements](#arm64-enhancements)
    - [Removal of `vhost-user-net` and `vhost-user-block` self spawning](#removal-of-vhost-user-net-and-vhost-user-block-self-spawning)
    - [Migration of `vhost-user-fs` backend](#migration-of-vhost-user-fs-backend)
    - [Enhanced "info" API](#enhanced-info-api)
    - [Contributors](#contributors-42)
- [v0.11.0](#v0110)
    - [`io_uring` support by default for `virtio-block`](#io_uring-support-by-default-for-virtio-block)
    - [Windows Guest Support](#windows-guest-support)
    - [`vhost-user` "Self Spawning" Deprecation](#vhost-user-self-spawning-deprecation)
    - [`virtio-mmio` Removal](#virtio-mmio-removal)
    - [Snapshot/Restore support for ARM64](#snapshotrestore-support-for-arm64)
    - [Improved Linux Boot Time](#improved-linux-boot-time)
    - [`SIGTERM/SIGINT` Interrupt Signal Handling](#sigtermsigint-interrupt-signal-handling)
    - [Default Log Level Changed](#default-log-level-changed)
    - [New `--balloon` Parameter Added](#new---balloon-parameter-added)
    - [Experimental `virtio-watchdog` Support](#experimental-virtio-watchdog-support)
    - [Notable Bug Fixes](#notable-bug-fixes-39)
    - [Contributors](#contributors-43)
- [v0.10.0](#v0100)
    - [`virtio-block` Support for Multiple Descriptors](#virtio-block-support-for-multiple-descriptors)
    - [Memory Zones](#memory-zones)
    - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements)
    - [Preliminary KVM HyperV Emulation Control](#preliminary-kvm-hyperv-emulation-control)
    - [Notable Bug Fixes](#notable-bug-fixes-40)
    - [Contributors](#contributors-44)
- [v0.9.0](#v090)
    - [`io_uring` Based Block Device Support](#io_uring-based-block-device-support)
    - [Block and Network Device Statistics](#block-and-network-device-statistics)
    - [HTTP API Responses](#http-api-responses)
    - [CPU Topology](#cpu-topology)
    - [Release Build Optimization](#release-build-optimization)
    - [Hypervisor Abstraction](#hypervisor-abstraction)
    - [Snapshot/Restore Improvements](#snapshotrestore-improvements)
    - [Virtio Memory Ballooning Support](#virtio-memory-ballooning-support)
    - [Enhancements to ARM64 Support](#enhancements-to-arm64-support)
    - [Intel SGX Support](#intel-sgx-support)
    - [`Seccomp` Sandbox Improvements](#seccomp-sandbox-improvements-1)
    - [Notable Bug Fixes](#notable-bug-fixes-41)
    - [Contributors](#contributors-45)
- [v0.8.0](#v080)
    - [Experimental Snapshot and Restore Support](#experimental-snapshot-and-restore-support)
    - [Experimental ARM64 Support](#experimental-arm64-support)
    - [Support for Using 5-level Paging in Guests](#support-for-using-5-level-paging-in-guests)
    - [Virtio Device Interrupt Suppression for Network Devices](#virtio-device-interrupt-suppression-for-network-devices)
    - [`vhost_user_fs` Improvements](#vhost_user_fs-improvements)
    - [Notable Bug Fixes](#notable-bug-fixes-42)
    - [Command Line and API Changes](#command-line-and-api-changes)
    - [Contributors](#contributors-46)
- [v0.7.0](#v070)
    - [Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug](#block-network-persistent-memory-pmem-virtiofs-and-vsock-hotplug)
    - [Alternative `libc` Support](#alternative-libc-support)
    - [Multithreaded Multi Queued `vhost-user` Backends](#multithreaded-multi-queued-vhost-user-backends)
    - [Initial RamFS Support](#initial-ramfs-support)
    - [Alternative Memory Hotplug: `virtio-mem`](#alternative-memory-hotplug-virtio-mem)
    - [`Seccomp` Sandboxing](#seccomp-sandboxing)
    - [Updated Distribution Support](#updated-distribution-support)
    - [Command Line and API Changes](#command-line-and-api-changes-1)
    - [Contributors](#contributors-47)
- [v0.6.0](#v060)
    - [Directly Assigned Devices Hotplug](#directly-assigned-devices-hotplug)
    - [Shared Filesystem Improvements](#shared-filesystem-improvements)
    - [Block and Networking IO Self Offloading](#block-and-networking-io-self-offloading)
    - [Command Line Interface](#command-line-interface)
    - [PVH Boot](#pvh-boot)
    - [Contributors](#contributors-48)
- [v0.5.1](#v051)
- [v0.5.0](#v050)
    - [Virtual Machine Dynamic Resizing](#virtual-machine-dynamic-resizing)
    - [Multi-Queue, Multi-Threaded Paravirtualization](#multi-queue-multi-threaded-paravirtualization)
    - [New Interrupt Management Framework](#new-interrupt-management-framework)
    - [Development Tools](#development-tools)
    - [Kata Containers Integration](#kata-containers-integration)
    - [Contributors](#contributors-49)
- [v0.4.0](#v040)
    - [Dynamic virtual CPUs addition](#dynamic-virtual-cpus-addition)
    - [Programmatic firmware tables generation](#programmatic-firmware-tables-generation)
    - [Filesystem and block devices vhost-user backends](#filesystem-and-block-devices-vhost-user-backends)
    - [Guest pause and resume](#guest-pause-and-resume)
    - [Userspace IOAPIC by default](#userspace-ioapic-by-default)
    - [PCI BAR reprogramming](#pci-bar-reprogramming)
    - [New `cloud-hypervisor` organization](#new-cloud-hypervisor-organization)
    - [Contributors](#contributors-50)
- [v0.3.0](#v030)
    - [Block device offloading](#block-device-offloading)
    - [Network device backend](#network-device-backend)
    - [Virtual sockets](#virtual-sockets)
    - [HTTP based API](#http-based-api)
    - [Memory mapped virtio transport](#memory-mapped-virtio-transport)
    - [Paravirtualized IOMMU](#paravirtualized-iommu)
    - [Ubuntu 19.10](#ubuntu-1910)
    - [Large memory guests](#large-memory-guests)
- [v0.2.0](#v020)
    - [Network device offloading](#network-device-offloading)
    - [Minimal hardware-reduced ACPI](#minimal-hardware-reduced-acpi)
    - [Debug I/O port](#debug-io-port)
    - [Improved direct device assignment](#improved-direct-device-assignment)
    - [Improved shared filesystem](#improved-shared-filesystem)
    - [Ubuntu bionic based CI](#ubuntu-bionic-based-ci)
- [v0.1.0](#v010)
    - [Shared filesystem](#shared-filesystem)
    - [Initial direct device assignment support](#initial-direct-device-assignment-support)
    - [Userspace IOAPIC](#userspace-ioapic)
    - [Virtual persistent memory](#virtual-persistent-memory)
    - [Linux kernel bzImage](#linux-kernel-bzimage)
    - [Console over virtio](#console-over-virtio)
    - [Unit testing](#unit-testing)
    - [Integration tests parallelization](#integration-tests-parallelization)

# v54.0

This release has been tracked in [v54.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/6?filterQuery=release%3A%22Release+54%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Live Migration Version Compatibility

Starting with this release, Cloud Hypervisor guarantees that live migrations
from the two previous releases are accepted ("n-2"). For example, v54.0 accepts
migrations from v52.0 and v53.0. Upgrades spanning more releases might work,
but incremental upgrades, for example from v54.0 to v56.0 to v58.0, are
strongly recommended. New functionality is added in a way that keeps VMs from
older releases working, so only newly created VMs can use it. The migration
protocol version is now an internal detail (#8751, #8787).

**Migrating back to an older release is not officially supported.** It might
work, depending on the VMM and VM configuration, but there is no guarantee.

This is another important step on our live migration roadmap. This effort
is tracked in
[#7111](https://github.com/cloud-hypervisor/cloud-hypervisor/issues/7111).

### VFIO Device Live Migration

VMs with migratable VFIO devices, for example ConnectX VFs bound to
`mlx5_vfio_pci`, can now be live migrated. The device state is transferred
together with the guest, and memory written by the device is tracked during the
migration. This requires `--platform iommufd=on` and a host driver with
migration support. On the destination, pre-opened VFIO device and `iommufd`
file descriptors can be passed through the new `vfio_fds` and `iommufd_fd`
options of `receive-migration` and `restore`. VFIO devices behind a virtual
IOMMU cannot be migrated yet (#8528).

### Live Migration Cancellation

An ongoing live migration can now be cancelled with the new
`vm.cancel-migration` API endpoint or the `ch-remote cancel-migration` command.
This lets management software abort a migration that does not converge, whose
destination is no longer the best choice, or that puts too much load on the
network. The VM keeps running on the source and a `vm.migration-cancelled`
event is emitted. After a failed or cancelled migration, Cloud Hypervisor
attempts to restore the source VM to its pre-migration state if it has not
already moved to the destination (#8455, #8935).

### Event Monitor Improvements

The event monitor (`--event-monitor`) is now properly documented, including its
format and all events (#8888). We aim to keep the events stable, so that
management software can rely on them. A new `vmm.started` event signals that
the VMM is ready to serve API requests (#8888). The live migration events have
been refined: the new `vm.migration-starting` and
`vm.migration-receive-starting` events are emitted when a migration attempt
begins, `vm.migration-started` now means that the handshake succeeded, and
`vm.migration-receive-failed` is emitted for every failure (#8850).

### Emulated ARM SMMUv3 (Experimental)

On AArch64, passthrough devices can now be placed behind an emulated ARM SMMUv3
with `--device ...,iommu=smmuv3`. Unlike with virtio-iommu, the guest uses its
stock `arm-smmu-v3` driver, and the translation is offloaded to the physical
SMMUv3 through `iommufd`. This makes it possible to assign devices that use the
IOMMU themselves, such as NVIDIA Grace-Blackwell GPUs. The feature requires
KVM, `--platform iommufd=on` and a host kernel with nested translation support
for ARM SMMUv3. Hotplug, snapshot/restore and live migration are not supported,
and `iommu=smmuv3` cannot be mixed with `iommu=virtio` (#8955, #8805, #8919,
#8790).

### Flat VMDK Disk Image Support

Flat VMDK disk images (`monolithicFlat` and `twoGbMaxExtentFlat`) are now
supported through `--disk image_type=vmdk`. A VMDK image consists of a
descriptor file that references one or more extent files holding the data.
Because the descriptor refers to other files by path, VMDK images are only
opened when `backing_files=on` marks the image as trusted, and relative extent
paths are confined to the descriptor's directory. Only synchronous I/O is
supported, and with Landlock the extent paths must be allowed through
`--landlock-rules` (#8599, #8714, #8886, #8885, #8691).

### Balloon Statistics

Statistics reported by the guest's `virtio-balloon` driver can now be queried
through the new `vm.balloon-stats` API endpoint, the `ch-remote balloon-stats`
command and the D-Bus API. They give management software insight into the
guest's memory usage, for example to size the balloon. The guest driver must
support balloon statistics for this to work (#8603).

### `virtio-console` Over a Unix Socket

The `virtio-console` device can now be exposed over a Unix socket with
`--console socket=<path>`. It works the same way as `--serial socket=<path>`,
and one client can be connected at a time. Configurations that pass a socket
path without selecting socket mode, or the other way round, are now rejected
for both `--serial` and `--console` (#8687).

### AArch64 Boot Improvements

AArch64 now supports firmware-aided direct kernel boot. When both `--firmware`
and `--kernel` are given together with `--fw-cfg-config`, the firmware receives
the kernel, command line and initramfs through the `fw_cfg` interface and boots
the kernel. This requires a build with the `fw_cfg` feature (#8910). Direct
kernel boot can also use ACPI using EFI tables supplied by Cloud Hypervisor
when `acpi=on` is added to the kernel command line. This enables features such
as PCI hotplug without a firmware, and a new ACPI Time and Alarm Device
provides the RTC to such guests (requires Linux 7.1 or newer with
`CONFIG_ACPI_TAD`) (#9013). Finally, AArch64 guests now receive SMBIOS tables
including the `--platform` values, which requires an updated EDK2 `CloudHv`
firmware (#8786).

### Faster Memory Prefault

Prefaulting guest memory (`prefault=on`) is now considerably faster on large
hosts. Memory zones are now prefaulted in parallel on their host NUMA nodes,
which reduced the prefault time of a 3 TiB guest on a two-socket host from 51.8
s to 7.4 s. This applies to both VM boot and snapshot restore. In addition,
prefaulted memory is now backed by transparent huge pages when the host THP
policy is `madvise` (#8785, #8962, #8915).

### VM Generation ID Device

On x86_64 and AArch64, VMs now have a VM Generation ID device whose value
changes on every restore and incoming live migration. Before, all VMs restored
from the same snapshot started with the same random number generator state and
produced the same `/dev/urandom` output until something reseeded it. Guest
kernels built with `CONFIG_VMGENID` now reseed after a restore. The device is
always present and needs no configuration. With direct kernel boot on AArch64,
the guest needs `acpi=on` on the kernel command line. Applications that seeded
their own random number generator keep that state and have to draw fresh
randomness themselves after a restore (#8907).

### Snapshot/Restore and Live Migration Improvements

* Saving and restoring vCPU state is now up to 5-9 times faster for VMs with
  many vCPUs, which shortens snapshot, restore and live migration downtime
  (#8961).
* Live migration downtime and total migration time have been reduced further,
  in particular for VMs with QCOW2 disks and for VMs with shared file-backed
  memory (#8898, #8943, #8982, #8589).
* Memory zones can be remapped to different host NUMA nodes on
  `receive-migration` and `restore` through the new `zone_updates` option
  (#8370).
* A new `memory_restore_mode=copyonwrite` maps the snapshot memory file
  copy-on-write, so VMs restored from the same snapshot share the page cache
  (#8581).
* A new `preserve_source=on` option of `send-migration` leaves the source VM
  paused instead of shutting it down after an offload snapshot (#8623).
* Serialized VM state is about 29% smaller for a VM with 16 disks. The
  previous PCI configuration state representation is still accepted when
  restoring snapshots (#8904).

### Device and Platform Improvements

* SEV-SNP VMs with VFIO devices now use roughly half as much host memory.
  Memory and CPU hotplug, huge pages and a virtual IOMMU are now rejected for
  SEV-SNP guests (#8501).
* On x86_64, the AMD HyperTransport range (1012 GiB to 1 TiB) is now reserved
  in the guest memory layout, so VFIO DMA mappings no longer fail for guests
  with more than 1011 GiB of memory (#8902).
* A PCI host bridge is now only created on PCI segment 0, which allows
  endpoints on other segments to be assigned PCI device 0 as needed by NVIDIA
  Grace GPUs (#8747).
* `vhost-user-net` devices now use the MAC address and MTU from the backend's
  configuration space when the backend provides them, and the reference
  backend gained a `mac=` option (#8684).
* VMs with passthrough GPUs start faster, by about 400 ms per GPU (#8975).

### New Command-Line and API Options

* A `guest_block_size` option has been added to `--disk` to override the
  logical block size advertised to the guest, for example to expose a 512-byte
  image stored on 4K-native storage (#8595).
* `reserve` now defaults to `on` for `--memory` and `--memory-zone` when
  `hugepages=on` is set, so a shortage of huge pages fails at VM creation
  instead of causing a `SIGBUS` at runtime. An explicit `reserve=off` is
  still honoured (#8770).

### Changed Default Behavior

* Nested virtualization is now disabled by default on all architectures. It
  widens the attack surface of the host kernel, and most users do not need it.
  Guests that use hardware virtualization themselves now need `--cpus nested=on`
  (#9045).k
* The `virtio-net` offloads (`offload_csum`, `offload_tso` and `offload_ufo`)
  are now disabled by default, because they make the host do more work for
  the guest's packets. They can be enabled again with
  `--net ...,offload_csum=on,offload_tso=on,offload_ufo=on` (#9041).
* Both new defaults apply to the command line and the API. VMs restored from
  a snapshot or received through live migration keep their configured
  values.

### Deprecations and Removals

* Disk image type autodetection, deprecated since v52.0, has been removed.
  `image_type` must now be specified for every disk that is not a
  `vhost-user` disk, and it must match the image format (#8794, #8798).
* The `memory_mode` option of `receive-migration` has been removed. The
  sender now announces the memory mode through the migration protocol
  (#8819).
* The `local=on` option of `send-migration` is deprecated in favor of
  `memory_mode=memfds` (#8820).
* The per-device `iommu=on` option is deprecated in favor of `iommu=virtio`.
  In the API, the `iommu` fields change from a boolean to an
  `"Off"`, `"Virtio"` or `"Smmuv3"` value. Booleans are still accepted but
  deprecated, as is the top-level `iommu` field of `VmConfig` (#8955).
* Unaligned requests to raw and fixed VHD disks with `direct=on` are no
  longer emulated with read-modify-write cycles, which caused bugs and
  security issues. Such requests now fail with an I/O error. Linux and EDK2
  guests are not affected, because their requests are multiples of the
  advertised logical block size, which now also accounts for the direct I/O
  alignment of host block devices. `guest_block_size` values below the
  backing block size are now also rejected for read-only raw and fixed VHD
  disks (#9064).

### Security Hardening

* A security policy and a threat model describing trusted and untrusted
  entities have been added (#8384).
* A malicious `vhost-user` backend can no longer crash the VMM by resizing the
  guest memory files (#8841).
* Validate `vfio-user` sparse mmap areas against the region size, and enforce
  domain ownership on virtio-iommu detach (#8840).
* Validate that `KVM_HC_MAP_GPA_RANGE` hypercalls from SEV-SNP guests stay
  within guest memory (#8992).
* Fix guest-triggerable VMM panics in TPM register reads (#8816) and in VFIO
  DMA ranges spanning sparse mmap areas (#8734).
* Reads from unbacked PIO/MMIO addresses now return zeros instead of stale data
  (#8817), and `fw_cfg` DMA requests are decoded correctly and no longer allow
  a guest-controlled out-of-bounds offset (#8994).
* Harden QCOW2, VHD, VHDX and VMDK image parsing against malformed images
  (#8679, #8799).
* Release `vsock` local ports only to their owner (#8612).
* Bump `rustls` to fix RUSTSEC-2026-0285, which affects TLS-encrypted live
  migration (#8883).
* The main thread now runs under its own seccomp filter once it has started
  the worker threads. Before, it ran without a filter (#9062).
* A warning is now logged when TCP live migration runs without TLS
  (`tls_dir`). Plain TCP migration is unencrypted and unauthenticated, so
  anyone who can reach the destination can send it a VM. The documentation
  now uses TLS in all TCP migration examples (#9066).

### Notable Bug Fixes

* Fix several QCOW2 data integrity bugs: data loss from a stale punch-hole
  (#8624), double allocations after a failed L2 relocation or a failed write
  (#8637, #8748), and L2 table refcount ordering after a failed write (#8721).
* Fix unbounded QCOW2 image growth and space leaks (#8597, #8951), and use the
  whole refcount table the header declares (#8952).
* Flush cached QCOW2 metadata on device pause so snapshots and shared-disk
  migrations do not miss completed writes (#8607). Only clear the QCOW2 dirty
  bit when the last user of the image is destroyed (#8647), and open QCOW2
  backing files with `O_DIRECT` when `direct=on` (#8673).
* Fix VHDX sector size handling and reject images with 4K logical sectors
  (#8565), and fsync the image on guest flush (#8719).
* Fall back to `write` when `fallocate` returns `EOPNOTSUPP`, for example on
  tmpfs (#8604).
* Make a disk resize by a `vhost-user-blk` backend visible to the guest
  (#8824).
* Correctly snapshot the AArch64 GIC redistributor and ICC state of every
  vCPU (#8638), restore nested virtualization state in the right order
  (#8996), and save and restore the guest shadow stack pointer (#8924).
* Preserve queue indices across device resume (#8626), and fix a panic when
  restoring a VM with an unused virtio queue (#8702).
* Fix on-demand restore with THP (#8694) and of fully file-backed VMs (#8849),
  and poison guest pages that a failed on-demand restore never served (#8928).
* Fix restoring a VM with TAP networking without `CAP_NET_ADMIN` (#8605).
* Fix short reads and writes across the codebase, which could truncate
  snapshot files or hang a restore (#8897).
* Fix silent data corruption after a live migration: disk reads and received
  network frames that were written into guest memory during pre-copy could
  arrive with stale content on the destination (#9015).
* Fix hangs of TCP live migration on worker failure (#8795), detect dead
  connections (#8660), and clean up receiver threads after a failed migration
  (#8656).
* Don't resume the source VM after a failed postcopy migration when the guest
  already runs on the destination (#8890), and reject postcopy and on-demand
  restore for VMs with VFIO, vDPA, `vfio-user` or `vhost-user` devices
  (#9020).
* Allow `vhost-user` devices to be migrated again after a migration (#8920).
* Add missing CPUID compatibility checks for live migration and restore
  (leaf 0x7 sub-leaves, AMX and XSAVE state components). This may reject
  migrations to hosts that previously appeared to work (#8659, #8726, #8854).
* Reinstate the gap between boot RAM and the hotplug memory region (#8780).
* Exclude the COM1 and `fw_cfg` ports from the PCI I/O window, fixing Windows
  guests that did not start their virtio devices (#8980), and accept balloon
  requests with more than 256 pages, fixing the balloon with Windows guests
  (#9007).
* Set the virtio PCI ISR configuration bit on configuration changes (#9011).
* Fix the MSI-X pending bit array for vectors 31 to 63 of every 64-vector
  block, where setting or clearing a pending bit changed the wrong bits
  (#9058).
* Only attach the negotiated `virtio-net` queue pairs, avoiding dropped frames
  (#8772), and drop TX frames instead of exiting the VM when the TAP device
  returns `EIO` (#8723).
* Don't reply to `vsock` RST packets with an RST (#8692).
* Recognize masked PCI BAR size probes, fixing OpenBSD boot (#8598), and fix
  16-bit writes to the PCI configuration address port (#8818).
* Fix the guest cache topology for multi-package guests on x86_64 (#8731) and
  AArch64 (#8617).
* Set the PMU interrupt in the AArch64 ACPI MADT (#8861), fix the ACPI SRAT
  Generic Initiator handle for PCI devices (#8766), and propagate clean
  shutdowns on AArch64 (#8710).
* Handle `SystemEvent` exits on riscv64 (#8941).
* Fix SEV-ES/SEV-SNP boot on newer host kernels (#8884).
* Reclaim balloon memory with private file-backed memory zones (#8909).
* Improve `vhost-user` reconnection by retrying the complete handshake,
  replaying in-order queues and aborting the retry on device teardown (#8613,
  #8592, #8759, #8463).
* Make device removal retryable (#8739), reject the removal of devices
  attached to the virtio-iommu (#8650), and only eject the vCPU the guest is
  expected to eject (#8611).
* Fix CPU affinity with 128 or more vCPUs on musl (#8972), and reject CPU
  affinity values beyond `CPU_SETSIZE` instead of panicking (#8843).
* Raise the HTTP API request body limit from 50 KiB to 4 MiB (#8752).
* Allow several missing syscalls in the seccomp filters that killed the VMM
  with `SIGSYS` (#8681, #8682, #8779, #8808, #8814, #8832, #8954, #8957).
* Return correct HTTP status codes: 400 or 409 for invalid configurations
  (#8665, #8946), 404 for unknown devices or disks (#8640), and 409 for
  requests conflicting with the VM state (#8851).
* Fix the OpenAPI specification for `generic_vhost_user` (#8674) and the
  `BreakPoint` VM state (#8675), and omit `virtio-block` latency counters
  before the first request (#8878).
* Clean up a stale serial socket under a lock before binding (#8733), and
  reuse the serial socket listener across reboot (#8823).
* Update the MSHV crates, fixing boot on older Microsoft Hypervisor versions
  (#8809).

### Contributors

Many thanks to everyone who has contributed to our release:

* aiqubits <aiqubits@hotmail.com>
* Alexander Lvov <alexander.lvov.git@gmail.com>
* Alyssa Ross <hi@alyssa.is>
* Anatol Belski <anbelski@linux.microsoft.com>
* Andrew Onyshchuk <oandrew@meta.com>
* Atish Patra <atishp@meta.com>
* Bo Chen <bchen@crusoe.ai>
* Chinmoy <daschinmoyy21@gmail.com>
* CMGS <ilskdw@gmail.com>
* Demi Marie Obenour <demiobenour@gmail.com>
* doge <me@crackerben.com>
* Evan Almloff <evanalmloff@gmail.com>
* Gauravpawar101 <Gauravpawar2522@gmail.com>
* Goran Draganić <goran@daytona.io>
* Isaku Yamahata <iyamahata@crusoe.ai>
* Jan Vokas <jan.vokas@zerops.io>
* Julian Schindel <julian.schindel@cyberus-technology.de>
* Leander Kohler <leander.kohler@cyberus-technology.de>
* leo03164 <leo03164@gmail.com>
* Leo Li <cheerleaderleo@outlook.com>
* Lily Goscha <lilygoscha@gmail.com>
* Max Makarov <maxpain@linux.com>
* Muminul Islam <muislam@microsoft.com>
* Oliver Anderson <oliver.anderson@cyberus-technology.de>
* Pascal Scholz <pascal.scholz@cyberus-technology.de>
* Peter Delevoryas <pdel@meta.com>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Rob Bradford <rbradford@meta.com>
* Rogelio Ruiz <rruiz@techxagon.io>
* Ruben Hakobyan <hruben@meta.com>
* Saravanan D <saravanand@crusoe.ai>
* Sebastien Boeuf <sboeuf@meta.com>
* Shatrughan Rai <polyglot.dev@outlook.com>
* Simon Lucido <simonlucido@meta.com>
* Stepan Rabotkin <epicstyt@gmail.com>
* Sumedh Alok Sharma <sumsharma@microsoft.com>
* tonic <tonicbupt@gmail.com>
* Tynan Daly <tynan.s.daly@gmail.com>
* Umang Pokhriyal <umangpokhriyall@gmail.com>
* vans163 <vans554@gmail.com>
* Victor Vieux <vieux@repl.it>
* Wei Liu <liuwe@microsoft.com>
* windsornguyen <win@dedaluslabs.ai>
* Yilei Zheng <trilith.zheng@gmail.com>
* Yi Wang <foxywang@tencent.com>
* ZenAlexa <zimingwang945@gmail.com>

# v53.0

This release has been tracked in [v53.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/6?filterQuery=release%3A%22Release+53%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Snapshot/Restore and Live Migration Improvements

* An offloaded snapshot/restore daemon has been introduced, allowing snapshot
  and restore to be carried out by an external process. The live migration
  protocol has been extended so page faults can be serviced from the source.
  This enables *postcopy* for remote live migration and *on-demand paging* for
  the offload daemon, both built on `userfaultfd`, bringing the offload daemon
  to parity with the internal snapshot/restore implementation (#8403, #8264,
  #8556).
* Snapshot pages can now be prefaulted in the background after a
  `userfaultfd`-based restore, combining a fast restore-to-resume with fully
  faulted-in memory shortly afterwards. The number of prefault threads scales
  with the available parallelism (#8150, #8538, #8484).
* Live migration data can now be encrypted using mutual TLS (#8053).
* VFIO migration v2 is now supported for same-host snapshot and restore of
  migratable VFIO devices (for example ConnectX VFs bound to `mlx5_vfio_pci`)
  (#8303).
* Migrated guests now issue post-migration network announcements so that
  connectivity on the new host is refreshed within milliseconds rather than
  taking up to 20 seconds to recover (#8263).
* The `vm.send-migration` API endpoint now returns immediately rather than
  blocking for the duration of the migration (#8021).

### Nested Hyper-V (Windows Guest) Support

Windows guests can now run nested Hyper-V as used by WSL2 on the KVM backend
(#8481).

### `virtio-rtc` Device Support

Support for `virtio-rtc` devices has been added (via `--rtc`), providing a
Real Time Clock to the guest. This is primarily useful for, and only enabled by
default on, MSHV guests that do not have access to `kvmclock` (#7795).

### Guest Clock Updates Across Snapshot/Restore and Migration

The guest clock is now advanced to account for the elapsed wall-clock time when
a VM is resumed after a snapshot-restore or a live migration so the guest sees
the correct time of day immediately rather than lagging behind by the downtime.
On x86 the kvmclock realtime flag is preserved so that the kernel adjusts the
clock automatically (#8256). On ARM64 the guest counter is advanced using the
`KVM_REG_ARM_TIMER_CNT` interface (#8343). vCPU TSC offsets are also
re-synchronized on restore (#8437).

### Buffered Serial Output for Late-Connecting Clients

In socket serial mode (`--serial socket=...`) the output produced before a
client connects is now buffered so a console client that attaches after boot
sees the earlier output (#8322).

### New Command-Line and API Options

* A `reserve=on|off` option has been added to `--memory` and
  `--memory-zone` (default `off`). When enabled, guest RAM is mapped
  without `MAP_NORESERVE` so an overcommitted configuration fails cleanly at VM
  creation with `ENOMEM` (#8350).
* The SMBIOS/DMI fields exposed via `--platform` have been extended and scoped
  to specific SMBIOS structures, for workloads that consume this data for
  licensing, inventory, or host identification (#8259).
* The VMM can now consume pre-opened VFIO `cdev` and `iommufd` file descriptors
  supplied by the caller, letting a privileged orchestrator open the device
  nodes once and hand the FDs to an unprivileged VMM at boot or at runtime.
  This is also a prerequisite for live migration of VMs using VFIO devices
  (#8287).
* A "glog style" log format has been added, with a single-character level and
  local or UTC timestamps (#8265).
* A new `--seccomp=errno` option has been added that will result in filtered
  syscalls to set `errno` to `EPERM`. This can be used when the killing of the
  process with `SIGSYS` is not desired (#8578).

### Deprecations

* The legacy `--platform` SMBIOS keys `serial_number` and `uuid` are
  deprecated in favor of the new SMBIOS-structure-scoped keys. The
  legacy keys continue to be accepted, with a warning, and will be
  removed in a future release (#8259).
* The generic `vhost-user` device's `virtio_id` command-line
  parameter is deprecated in favor of `device_type`, matching the API
  and configuration naming. `virtio_id` is retained as an alias that
  logs a warning and will be removed in a later release (#8564).

### Notable Bug Fixes

* Revert `create memfd for private mappings`, which caused a serious memory
  consumption regression without hugepages or `shared=on` (#8246).
* Advertise the mandatory Hyper-V partition privileges so Windows
  guests enable the reference TSC page instead of taking a VM exit on
  every clock read (#8380).
* Correctly save and restore ARM64 SVE registers (#8268).
* Set the guest physical address space size correctly on AMD (#8291).
* Sparse align the initial PCI BAR placement to avoid a Windows rebalancing
  deadlock (#8205).
* Various TPM fixes, including for Windows guests (#8305, #8312).
* Preserve `WRITE_ZEROES` on QCOW2 images with backing files (#8245)
  and fix `WriteZeroes` sector arithmetic overflow (#8388).
* Improve `O_DIRECT` handling so that `direct=on` is usable on hosts
  requiring 4K alignment, including opening fixed VHD images (#8335).
* Resolve relative QCOW2 backing file paths correctly (#8196).
* Drain block I/O before the device is paused (#8356) and retry
  locking when interrupted by `EINTR` (#8402).
* Fix VHDX region-table overlap detection (#8483) and an incomplete
  bounds check in the VHDX synchronous I/O worker (#8529).
* Fix a guest deadlock by signalling `DEVICE_NEEDS_RESET` and
  releasing the activation barrier when a virtio device fails to
  activate (#8295).
* Handle short and misaligned MMIO/PIO accesses without panicking:
  validate the MMIO access size (#8506), return all-ones for
  unregistered MMIO/PIO reads (#8237), correct `fw_cfg` short and long
  reads (#8507), respect the PCI CFG capability length for BAR access
  (#8272, #8238), and reject incorrect MSI-X table and PBA reads (#8417).
* Translate virtio-iommu buffers that span adjacent mappings (#8520),
  and add centralized virtio descriptor-range validation (#8232).
* Handle a guest/host half-close on vsock connections (#8372).
* Avoid a guest-triggerable VMM panic by replacing asserts on ACPI
  device accesses with checks (#8405).
* Emit a complete ACPI `_S5_` sleep package, fixing an early-boot
  fault on OpenBSD guests (#8426).
* Transfer post-snapshot guest memory changes during live migration to
  avoid corruption with `vhost-user` devices (#8361).
* Improve configuration validation to return errors instead of
  panicking on malformed input (#8511), require the `ch-remote`
  snapshot/restore config arguments instead of panicking (#8331),
  reject resizing below the boot memory size (#8428), reject removal of
  an already-removed device (#8559), and fix generic `vhost-user`
  parsing (#8564).
* Don't leak a file descriptor when ejecting a VFIO device (#8367),
  and fix a PCI device hotplug race by deferring device visibility
  (#8369).
* Fix PCI segment ID validation (#8377) and make the PCI BDF
  configurable for the balloon and ivshmem devices (#8310).
* Return HTTP 404 for API requests against a VM that has not been
  created (#8320), and omit unset optional fields from API responses so
  that strict client generators accept them (#8321).
* Clean up a stale API socket under a lock before binding (#8325).
* Cache the local timezone before installing seccomp filters (#8327).
* Enhance the ACPI PPTT with cache topology information (#7893).
* Correct several OpenAPI specification entries: the
  `iommu_address_width` field name (#8244), integer schema formats
  (#8278), `GenericVhostUserConfig` properties (#8543), a parameter
  name (#8544), and the `vmm.nmi` to `vm.nmi` endpoint (#8571).
* Deduplicate `VmConfig::preserved_fds` (#8271).

### Contributors

Many thanks to everyone who has contributed to our release:

* Aastha Rawat <aastharawat@microsoft.com>
* Adel-Ayoub <adelayoub.maaziz@gmail.com>
* Alexander Lvov <alexander.lvov.git@gmail.com>
* Alyssa Ross <hi@alyssa.is>
* Anatol Belski <anbelski@linux.microsoft.com>
* Andrei Vagin <avagin@google.com>
* Anirudh Rayabharam <anrayabh@microsoft.com>
* Atish Patra <atishp@meta.com>
* Bo Chen <bchen@crusoe.ai>
* Cameron Baird <cameronbaird@microsoft.com>
* Chris Webb <chris@arachsys.com>
* CMGS <ilskdw@gmail.com>
* Damian Barabonkov <dbctl@pm.me>
* Daniel Verkamp <drv@meta.com>
* doge <me@crackerben.com>
* Dylan Reid <dgreid@fb.com>
* Gauthier Jolly <contact@gjolly.fr>
* Henry Hrvoje Tonkovac <htonkovac@gmail.com>
* Ian Klemm <hi@ianklemm.de>
* Julian Schindel <julian.schindel@cyberus-technology.de>
* Leander Kohler <leander.kohler@cyberus-technology.de>
* Max Makarov <maxpain@linux.com>
* Meng Zhuo <mengzhuo@iscas.ac.cn>
* Muminul Islam <muislam@microsoft.com>
* Nikolas Kyx <55556836+nyx191@users.noreply.github.com>
* Oliver Anderson <oliver.anderson@cyberus-technology.de>
* Pascal Scholz <pascal.scholz@cyberus-technology.de>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Rob Bradford <rbradford@meta.com>
* Roman <roman@vanesyan.com>
* Rowen-Ye <rowenye1@gmail.com>
* Ruben Hakobyan <hruben@meta.com>
* Saravanan D <saravanand@crusoe.ai>
* Sayed Kaif <metsw24@gmail.com>
* Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
* Sebastien Boeuf <sboeuf@meta.com>
* tchaton <thomas.chaton.ai@gmail.com>
* Thomas Prescher <thomas.prescher@cyberus-technology.de>
* tonic <tonicbupt@gmail.com>
* Tushar Khatri <hello@tusharkhatri.in>
* wangyf0611 <wangyufeng@iscas.ac.cn>
* Wei Liu <liuwe@microsoft.com>
* wuxinyue <wuxinyue.wxy@antgroup.com>
* yanjianqing <yanjianqing@kylinos.cn>

# v52.0

This release has been tracked in [v52.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/6?filterQuery=release%3A%22Release+52%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Security Fixes

This release fixes a use-after-free vulnerability in the
`virtio-block` async I/O completion path (#8220). Details can be found
in GHSA-f47p-p25q-83rh (CVE-2026-45782).

### Confidential VM Support: KVM SEV-SNP

AMD SEV-SNP confidential VMs can now be launched on KVM in addition to
MSHV. Cloud Hypervisor uses `guest_memfd` to back private memory and
loads an IGVM-packaged firmware (e.g. Oak stage0) to bring up an
SEV-SNP guest on real hardware (#7942).

Measured-boot support has been completed for KVM SEV-SNP guests,
bringing parity with the QEMU launch flow: the kernel, command line,
and initrd are reflected in the launch measurement, and a signed SNP
ID block can be passed so the guest (or a remote attestor) can verify
the launch (#8123).

### VFIO Device Passthrough via `iommufd` and `vfio-cdev`

Cloud Hypervisor now supports the modern Linux VFIO device-access
model based on the `iommufd` subsystem and per-device `vfio-cdev`
character devices (available since Linux v6.6). This unlocks the
advanced IOMMU features exposed by modern hardware and enables fully
accelerated IOMMU support inside the guest. The legacy
`container/group`-based path continues to be supported (#7981).

In addition, several VFIO/PCI improvements have landed:

* A new `host_mmap_bars` option allows selective mapping of VFIO BARs
  into the guest, so devices that expose registers that should not be
  fast-pathed can be selectively trapped (#7991).
* Sub-page VFIO BARs are now expanded to a full page for `mmap()`,
  avoiding spurious failures on devices with small BARs (#7939).
* `VfioMsix::cap` and `VfioMsix::bar` are now kept synchronized to
  avoid stale MSI-X state (#7993).
* VFIO MMIO regions are now reliably cleared in `DeviceManager::drop`
  (#8063).
* Deferred PCI BAR reprogramming state is now part of the saved
  device state (#7945).
* GSIs for MSI/MSI-X interrupt vectors are now allocated lazily,
  reducing GSI consumption for VMs with many devices (#7940).

### Multi-Connection TCP Live Migration

The `send-migration` HTTP command now accepts a `connections`
parameter (defaults to `1`) that controls how many parallel TCP
connections are used for the live migration data transfer. On
servers with high-bandwidth (e.g. 100G) network links, multiple
connections significantly increase migration throughput (#7669).

### Userfaultfd Demand-Paged Snapshot Restore

A new `memory_restore_mode` option on the restore path allows guest
memory to be populated lazily via `userfaultfd` instead of reading
the entire memory snapshot before the VM resumes. This dramatically
reduces restore-to-resume latency for large guests, at the cost of
faulting pages in on demand after resume (#7800).

### Configurable PCI BDFs for Devices

Users now can configure the PCI slot/device number, the middle
part of the PCI BDF, through the CLI and API; bus and function
remain fixed at `0`. This covers virtio devices (except balloon),
VFIO devices, vfio-user device, and generic vhost-user devices
(#8175, #7965, #8174, #8163).

### AIO Block Backend `write_zeroes` and `punch_hole` Support

The AIO block backend now implements `write_zeroes` and `punch_hole`,
fixing a v51.0 regression that affected guests on hosts with
`io_uring` disabled (e.g. RHEL 9 / CentOS Stream 9, where
`kernel.io_uring_disabled=2` is a common hardening default)
(#7817).

### Asynchronous QCOW2 Backend with `io_uring`

A new `QcowDiskAsync` backend uses `io_uring` for data I/O on QCOW2
images and is automatically selected when `io_uring` is available
(falling back to the synchronous backend otherwise). Each virtio
queue gets its own ring and its own duplicated file descriptor,
enabling parallel data I/O across queues; QCOW2 metadata is shared
behind an `Arc<RwLock<_>>` (#7882).

### Generic `vhost-user` Device Support

A new generic `vhost-user-generic` device type has been added.
All information about the device (number of queues, queue size,
feature bits, etc.) is provided to Cloud Hypervisor via the
command line or API, allowing arbitrary vhost-user backends to be
attached without device-type-specific support in the VMM (#7221).

### Core Scheduling for vCPU Threads

A new `core_scheduling` option on `--cpus` controls how vCPU threads
are placed on SMT siblings via the kernel's core-scheduling feature,
which is useful for mitigating cross-thread side-channel attacks
without disabling SMT outright. Three modes are supported: `vm`
(default — all vCPU threads share one cookie), one-cookie-per-vCPU,
and a configurable grouping mode (#7747).

### Snapshot/Restore and Live Migration Improvements

* Snapshot and restore now treat the memory backing file as a sparse
  file: on snapshot the backing FD is sought with
  `SEEK_DATA`/`SEEK_HOLE` to skip unallocated regions, and on restore
  the on-disk file is read as sparse. Pages the guest never touched
  remain as holes in the backing file, substantially reducing both
  snapshot size and restore time (#8113).
* Paused VMs can now be migrated (#8099).
* User-configurable downtime and timeout parameters for live
  migration (#7835), and improved downtime observability (#7979).
* A new option to automatically resume the VM on restore, useful
  when restoring from the VMM command line without an API socket
  (#7857).
* Snapshot/restore support for `vhost-user` devices has been filled
  out (#7908), including migration support for `virtio-fs` (#7937).
* Vsock connections are now reset on snapshot restore to avoid stale
  half-open connections on the guest side (#7958).
* The KVM clock is now restored before vCPUs are resumed (#7932),
  and `notify_guest_clock_paused` is called for Hyper-V guests
  (#7933), eliminating clock jumps observed after restore.
* `vhost-user` devices receive a guest interrupt on resume so that
  in-flight I/O is not stalled across pause/resume (#7851), and
  activated queue eventfds are signaled on resume for all virtio
  devices (#8004).
* `UFFDIO_WAKE` and `UFFD_FEATURE_MISSING_HUGETLBFS` are now handled
  correctly (#8016).

### New Command-Line and API Options

* `--no-shutdown` keeps the VMM process running when the guest
  triggers a shutdown. This is useful for higher-level management
  layers that need full control over the VMM process (#8025).
* fw_cfg now supports inline string items via
  `--fw-cfg name=...,string=...`, matching QEMU's syntax (#7825).
* A new per-zone `mergeable` option on `--memory-zone` lets KSM
  merging be controlled per memory zone instead of only globally
  (#7983).
* Disk lock granularity is now configurable via `--disk` (#7763).
* `--cpus threads_per_core` is now enforced to be at most two
  (#8078), matching the hardware constraint.
* `--log-format` controls the log output format, complementing
  existing log-level controls (#8172). Wallclock log timestamps
  now use UTC (#8193).
* The HTTP API now logs error responses (#8181), and the
  `user_devices` configuration is included in the OpenAPI spec
  (#8072).

### MSHV Support Improvements

* `SVM_EXIT_CPUID`, `SVM_EXIT_MSR`, and `GHCB_MSR_TERM_REQ` handlers
  have been added to the SEV-SNP exit path (#7934).
* Guest MTRR MSRs are now preserved on KVM (#7944).
* Nested-virtualization control fixes on AMD (#7783).

### Deprecations

Auto-detection of disk image types is now deprecated and will be
removed in a future release. Specify the image type explicitly via
the `--disk image_type=...` CLI option or the `image_type` field
on `DiskConfig` in the HTTP/D-Bus API to avoid the deprecation
warning (#8219).

### Notable Bug Fixes

* Fix a guest-triggerable VMM panic via an out-of-bounds
  `queue_select` in the MSI-X vector access path on the virtio PCI
  common config (#7918).
* Saturate instead of underflowing on a malformed balloon `actual`
  value reported by the guest (#7903).
* IOMMU address translation now returns errors instead of panicking
  on malformed guest input (#8023).
* Several virtio device error paths now reset queues instead of
  panicking or exiting the worker (#8128); additional device
  manager paths avoid guest-induced panics (#8186).
* Virtio barriers are now sized from the number of queues actually
  activated by the guest, fixing pause hangs (#7881).
* Spec compliance fixes for virtio device config layouts (#7805).
  and other small spec deviations (#8066). Console descriptor-chain
  processing has been fixed (#7802). dword writes to the MSI-X
  control register are now handled (#7956).
* PCI segment ACPI `_UID` values now match the segment ID (#7992).
* Fix VHDX multiqueue data corruption (#7746).
* Fix block-device resize-disk (#7948) and the writeback-mode
  update flow (#7949).
* Fix vdpa RX failure after device reset by always using base 0
  (#7964).
* Fix rebooting with landlock and a pty console (#7911).
* Reject rate limiting with `vhost-user` (#8135).
* Fix a deadlock between `CpuManager::pause()` and MMIO writes to
  the ACPI CPU hotplug controller (#7990, #8092).
* Cancel `vhost-user` reconnection if the device has been unplugged
  (#8182).
* Remove the API socket path when `start_vmm` fails (#8018).
* Add missing ID validation for the virtio-rng device (#9a46affc3).
* Fix UB in `MemoryRangeTable::read_from` (#7828) and `load_igvm()`
  (#8129).
* `memory_actual_size` reflects hot-plug state, removing a slow
  recomputation path (#7969).
* DIO alignment is now queried directly from the underlying file
  rather than assumed (#7767).
* QCOW2 metadata access is now thread-safe with `num_queues > 1`
  (#7744).
* Block devices unconditionally assume sparse support, matching
  QEMU and avoiding a non-existent probe ioctl (#7757).
* Multi-value CLI options now use `.action(ArgAction::Append)` so
  that repeating an option appends rather than replacing (#7778).
* AIO completions are batched in `next_completed_request` for the
  block backend, reducing per-completion overhead (#7864).
* Roll back the PCI BAR address on a failed `move_bar` so that an
  out-of-range MMIO address programmed by the guest leaves the
  device functional rather than crashing the VMM (#7950).
* Fix O_DIRECT writes failing with `EINVAL` on the synchronous
  QCOW2 backend (#8012).
* Fix O_DIRECT reads failing with `EINVAL` on the async `io_uring`
  QCOW2 read path (#8051).
* Move QCOW2 compressed-cluster decompression out of the metadata
  lock, removing a bottleneck for compressed images (#8061).
* Clamp the balloon's reported range to its region before
  processing unused pages (#8098).
* Validate the requested balloon size against total guest RAM
  (#8116).
* Reject `--memory-zone` configurations that set both `mergeable`
  and `shared` (#8126).
* Disable sector 0 writes for autodetected VHD images (#8218).

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Anatol Belski <anbelski@linux.microsoft.com>
* Andrei Vagin <avagin@google.com>
* Anirudh Rayabharam <anrayabh@microsoft.com>
* Bo Chen <bchen@crusoe.ai>
* CMGS <ilskdw@gmail.com>
* Changyuan Lyu <changyuanl@google.com>
* Chinmoy <daschinmoyy21@gmail.com>
* Damian Barabonkov <dbctl@pm.me>
* Demi Marie Obenour <demiobenour@gmail.com>
* Dylan Reid <dgreid@dylanreid.com>
* Dylan Reid <dgreid@fb.com>
* Emir Beganovic <beganovic.emir@gmail.com>
* JP Kobryn <inwardvessel@gmail.com>
* Jared White <git@jaredwhite.dev>
* Julian Schindel <julian.schindel@cyberus-technology.de>
* Julian Schindel <mail@arctic-alpaca.de>
* Keith Adler <kadler@cloudflare.com>
* Kevin Hui <kevinhui@meta.com>
* Leander Kohler <leander.kohler@cyberus-technology.de>
* Max Makarov <maxpain@linux.com>
* Muminul Islam <muislam@microsoft.com>
* Nguyen Dinh Phi <phind.uet@gmail.com>
* Pascal Scholz <pascal.scholz@cyberus-technology.de>
* Peter Oskolkov <posk@google.com>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Rob Bradford <rbradford@meta.com>
* Roberto Campesato <render@metalabs.org>
* Ruben Hakobyan <hruben@meta.com>
* Saravanan D <saravanand@crusoe.ai>
* Sebastian Eydam <sebastian.eydam@cyberus-technology.de>
* Sebastian Walz <sebastian.walz@secunet.com>
* Shayon Mukherjee <shayonj@gmail.com>
* Souradeep <schakrabarti@microsoft.com>
* Souradeep Chakrabarti <schakrabarti@microsoft.com>
* Stepan Rabotkin <epicstyt@gmail.com>
* Thomas Prescher <thomas.prescher@cyberus-technology.de>
* Tobias Kässer <t.kaesser@gmail.com>
* Victor Vieux <vieux@repl.it>
* Vincent Thomas <vincent@v-thomas.com>
* Wei Liu <liuwe@microsoft.com>
* Zhiheng Tao <junchuan.tzh@antgroup.com>
* dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* sohakpt <sohakpt@outlook.com>
* wuxinyue <wuxinyue.wxy@antgroup.com>

# v51.0

This release has been tracked in [v51.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/6?filterQuery=release%3A%22Release+51%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Security Fixes

This release fixes a security vulnerability in disk image handling.
Details can be found in
[GHSA-jmr4-g2hv-mjj6](https://github.com/cloud-hypervisor/cloud-hypervisor/security/advisories/GHSA-jmr4-g2hv-mjj6).

* A new `backing_files=on|off` option has been added to `--disk` to
  explicitly control whether QCOW2 backing files are permitted. This
  defaults to `off` to prevent the loading of backing files entirely.
  (#7685)
* Explicit image type specification via the user interface, removing
  reliance on format autodetection (#7728).
* Prevent sector-zero writes for autodetected raw images (#7728).

### Significant QCOW2 v3 Improvements

A large number of QCOW2 v3 specification features have been implemented:

* RAW backing file support for QCOW2 overlays (#7570)
* Zero bit in L2 entries (#7627)
* Incompatible feature bit validation (#7612)
* Dirty bit support (#7636)
* Variable refcount widths (1 to 64-bit) (#7633)
* Corrupt bit detection and marking (#7639)
* Autoclear feature bits handling (#7648)
* Thread safety fix for multiple virtio queues (`num_queues > 1`)
  (#7661)
* Correct zero-fill for reads beyond backing file size (#7678)
* Live disk resize support (#7687)

### ACPI Generic Initiator Support

ACPI Generic Initiator Affinity (SRAT Type 5) support has been added
to associate VFIO-PCI devices with dedicated memory/CPU-less NUMA
nodes. This enables the guest OS to make NUMA-aware memory allocation
decisions for device workloads. A new `device_id` parameter has been
added to `--numa` for specifying VFIO devices. (#7626)

### Block Device DISCARD and WRITE_ZEROES Support

The `virtio-blk` device now supports `DISCARD` and `WRITE_ZEROES`
operations for QCOW2 and RAW image formats. This enables thin
provisioning and efficient space reclamation when guests trim
filesystems. A new `sparse=on|off` option has been added to `--disk` to
control disk space management: `sparse=on` (default) enables thin
provisioning with space reclamation, while `sparse=off` provides thick
provisioning with consistent I/O latency. (#7666)

### Notable Performance Improvements

* Transparent Huge Pages (THP) support has been extended to cover
  anonymous shared memory (`shared=on`) via `madvise`. Previously, THP
  was only used for non-shared memory. (#7646)
* The `vhost-user-net` device now uses the default set of vhost-user
  virtio features, including `VIRTIO_F_RING_INDIRECT_DESC`, which
  provides a performance improvement. (#7653)

### MSHV Support Improvements

* Optimize CPU state update after emulation by only updating special
  registers when changed (#7603)
* Enable SMT for guests with `threads_per_core > 1` (#7668)
* Stub `save_data_tables()` to unblock VM pause/resume (#7692)
* Handle `GHCB_INFO_SPECIAL_DBGPRINT` VMG exit in SEV-SNP guest exit
  handler (#7703)
* Fix CVM boot failure on MSHV (#7548)
* Fix CPU topology detection for multithreaded configurations (#7576)

### Notable Bug Fixes

* Fix VFIO device hot-remove leaving group and container file
  descriptors open, preventing re-add (#7676)
* Fix snapshot restore when backing file is on read-only storage with
  `shared=false` (#7674)
* Enforce `VIRTIO_BLK_F_RO` even if guest does not negotiate it
  (#7705)
* Fix read-only block device FLUSH requests from OVMF preventing VMs
  from booting (#7706)
* Fix vhost-user device not properly dropping unowned file descriptors
  (#7679)
* Fix `vhost-user-block` `get_config` interoperability (#7617)
* Fix vsock TOCTOU race condition by copying packet header from guest
  memory before processing (#7530)
* Fix vsock handling of large TX packets spanning multiple data
  descriptors (#7680)
* Add `gettid()` to all seccomp filters (#7596)
* Fix MAC address parsing that wrongly allowed `+` instead of hex
  characters (#7579)
* Improve UUID parse error message and `--net` fd help text (#7702)
* Fix various inconsistencies in our OpenAPI specification file
  (#7716, #7726)
* Various documentation fixes (#7602, #7606)

### Contributors

Many thanks to everyone who has contributed to our release:

* Aastha Rawat <aastharawat@microsoft.com>
* Alyssa Ross <hi@alyssa.is>
* Anatol Belski <anbelski@linux.microsoft.com>
* Anirudh Rayabharam <anrayabh@microsoft.com>
* Bo Chen <bchen@crusoe.ai>
* Champ-Goblem <cameron@northflank.com>
* Changyuan Lyu <changyuanl@google.com>
* Damian Barabonkov <dbctl@pm.me>
* Demi Marie Obenour <demiobenour@gmail.com>
* Leander Kohler <leander.kohler@cyberus-technology.de>
* Muminul Islam <muislam@microsoft.com>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Rob Bradford <rbradford@meta.com>
* Rowen-Ye <rowenye1@gmail.com>
* Saravanan D <saravanand@crusoe.ai>
* Stanislav Kinsburskii <skinsburskii@linux.microsoft.com>
* Thomas Leroy <thomas.leroy.mp@gmail.com>
* Wei Liu <liuwe@microsoft.com>
* Yi Wang <foxywang@tencent.com>
* Zhibin Li <banlu.lzb@antgroup.com>
* stevenhorsman <steven@uk.ibm.com>

# v50.0

This release has been tracked in [v50.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/6?filterQuery=release%3A%22Release+50%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Configurable Nested Virtualization Option on x86_64

The `nested=on|off` option has been added to `--cpu` to allow users
to configure nested virtualization support in the guest on x86_64
hosts (for both KVM and MSHV). The default value is `on` to maintain
consistency with existing behavior. (#7408)

### Compression Support for QCOW2

QCOW2 support has been extended to handle compression clusters based on
zlib and zstd. (#7462)

### Notable Performance Improvements

Performance of live migration has been improved via an optimized
implementation of dirty bitmap maintenance. (#7468)

### Live Disk Resizing Support for Raw Images

The `/vm.resize-disk` API has been introduced to allow users to resize block
devices backed by raw images while a guest is running. (#7476)

### Developer Experience Improvements

Significant improvements have been made to developer experience and
productivity. These include a simplified root manifest, codified and
tightened Clippy lints, and streamlined workflows for `cargo clippy` and
`cargo test`. (#7489)

### Improved File-level Locking Support

Block devices now use byte-range advisory locks instead of whole-file
locks. While both approaches prevent multiple Cloud Hypervisor instances
from simultaneously accessing the same disk image with write
permissions, byte-range locks provide better compatibility with network
storage backends. (#7494)

### Logging Improvements

Logs now include event information generated by the event-monitor
module. (#7512)

### Notable Bug Fixes

* Fix several issues around CPUID in the guest (#7485, #7495, #7508)
* Fix snapshot/restore for Windows Guest (#7492)
* Respect queue size in block performance tests (#7515)
* Fix several Serial Manager issues (#7502)
* Fix several seccomp violation issues (#7477, #7497, #7518)
* Fix various issues around block and qcow (#7526, #7528, #7537, #7546,
  #7549)
* Retrieve MSRs list correctly on MSHV (#7543)
* Fix live migration (and snapshot/restore) with AMX state (#7534)

### Contributors

Many thanks to everyone who has contributed to our release:

* Anatol Belski <anbelski@linux.microsoft.com>
* Anirudh Rayabharam <anrayabh@microsoft.com>
* Bo Chen <bchen@crusoe.ai>
* Changyuan Lyu <changyuanl@google.com>
* Chengyu Fu <chengyu.fu@linux.alibaba.com>
* Connor Brewster <cbrewster@hey.com>
* Demi Marie Obenour <demiobenour@gmail.com>
* Eugene Korenevsky <ekorenevsky@aliyun.com>
* Julian Stecklina <julian.stecklina@cyberus-technology.de>
* Matt Moriarity <matt@mattmoriarity.com>
* Muminul Islam <muislam@microsoft.com>
* Oliver Anderson <oliver.anderson@cyberus-technology.de>
* Pascal Scholz <pascal.scholz@cyberus-technology.de>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Praveen K Paladugu <prapal@linux.microsoft.com>
* Rob Bradford <rbradford@rivosinc.com>
* Sebastien Boeuf <seb@rivosinc.com>
* Songqian Li <sionli@tencent.com>
* Stefan Nürnberger <stefan.nuernberger@cyberus-technology.de>
* Thomas Prescher <thomas.prescher@cyberus-technology.de>
* Wei Liu <liuwe@microsoft.com>

# v49.0

This release has been tracked in [v49.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+49%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### MSHV Support Improvements

On AArch64 with MSHV, firmware boot (#7391) and graceful guest shutdown
(#7354) are now supported, and CI coverage has been added (#7381).

### Logging Improvements

Logs now use an improved timestamp format (#7355), emit an explicit
message on graceful shutdown (#7346), and reduce noisy warnings with
newer guest kernels (#7423).

### Removed Default IP and Mask for `virtio-net` Devices

The implicit default IP (192.168.249.1) and netmask (255.255.255.0) for
`virtio-net` devices have been removed, as previously announced in the
deprecation notice two releases ago (#7365).  Users now can create
`virtio-net` devices with no IP and netmask assigned.

### Notable Bug Fixes

* Prevent crash on malformed vsock connect requests (#7310)
* Fix vCPU hotplug when the vCPU count exceeds 255 (#7349)
* Drop the VFIO container when no VFIO devices are active (#7364)
* Gracefully close preserved file descriptors on hot device removal (#7371)
* Fix race condition in pause–resume cycles (#7397)
* Add NetConfig offload parameters to the OpenAPI spec file (#7445)
* Fix seccomp filter violation from the http-server thread (#7454)

### Contributors

Many thanks to everyone who has contributed to our release:

* Aastha Rawat <aastharawat@microsoft.com>
* Alyssa Ross <hi@alyssa.is>
* Anirudh Rayabharam <anrayabh@microsoft.com>
* Ariel Chenet <apchenet@gmail.com>
* Bo Chen <bchen@crusoe.ai>
* Can Zhang <icloud9957@gmail.com>
* Eugene Korenevsky <ekorenevsky@aliyun.com>
* Maximilian Güntner <code@mguentner.de>
* Muminul Islam <muislam@microsoft.com>
* Oliver Anderson <oliver.anderson@cyberus-technology.de>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Rob Bradford <rbradford@rivosinc.com>
* Ruoqing He <heruoqing@iscas.ac.cn>
* Wei Liu <liuwe@microsoft.com>
* Yi Wang <foxywang@tencent.com>

# v48.0

This release has been tracked in [v48.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+48%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Experimental `fw_cfg` Device Support

This feature enables passing configuration data and files, such as VM
boot configurations (kernel, kernel cmdline, e820 memory map, and ACPI
tables), from the host to the guest. (#7117)

### Experimental `ivshmem` Device Support

Support for inter-VM shared memory has been added. For more information,
please refer to the [ivshmem documentation](docs/ivshmem.md). (#6703)

### Firmware Boot Support on `riscv64`

In addition to direct kernel boot, firmware boot support has been added
on `riscv64` hosts. (#7249)

### Increased vCPU Limit on x86_64/kvm

The maximum number of supported vCPUs on x86_64 hosts using KVM has been
raised from 254 to 8192. (#7299)

### Improved Block Performance with Small Block Sizes

Performance for `virtio-blk` with small block sizes (16KB and below)
is enhanced via submitting async IO requests in batches. (#7146)

### Faster VM Pause Operation

The VM pause operation now is significantly faster particularly for VMs
with a large number of vCPUs. (#7290)

### Updated Documentation on Windows Guest Support

Our Windows documentation now includes instructions to run Windows 11
guests, in addition to Windows Server guests. (#7218)

### Policy on AI Generated Code

We will decline any contributions known to contain contents generated or
derived from using Large Language Models (LLMs). Details can be found
in our [contributing documentation](CONTRIBUTING.md). (#7162)

### Removed SGX Support

The SGX support has been removed, as announced in the deprecation notice two
release cycles ago. (#7093)

### Notable Bug Fixes

* Seccomp filter fixes with glibc v2.42 (#7327)
* Various fixes related to  (#7331, #7334, #7335)

### Contributors

Many thanks to everyone who has contributed to our release:

*  Alex Orozco <alexorozco@google.com>
*  Alyssa Ross <hi@alyssa.is>
*  Anirudh Rayabharam <anrayabh@microsoft.com>
*  Bo Chen <bchen@crusoe.ai>
*  Demi Marie Obenour <demiobenour@gmail.com>
*  Lucas Grosche <lucas.grosche@cyberus-technology.de>
*  Muminul Islam <muislam@microsoft.com>
*  Oliver Anderson <oliver.anderson@cyberus-technology.de>
*  Peter Oskolkov <posk@google.com>
*  Philipp Schuster <philipp.schuster@cyberus-technology.de>
*  Ruoqing He <heruoqing@iscas.ac.cn>
*  Shubham Chakrawar <schakrawar@crusoe.ai>
*  Songqian Li <sionli@tencent.com>
*  Wei Liu <liuwe@microsoft.com>

# v47.0

This release has been tracked in [v47.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+47%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Block Device Error Reporting to the Guest

Instead of exiting on I/O errors, the `virtio-block` device now reports
errors to the guest using `VIRTIO_BLK_S_IOERR`. It improves the user
experience particularly when the guest rootfs is not backed by the
affected block device. (#7107)

### Nice Error Messages on Exit

We now have the chain of errors being reported and printed nicely, when
Cloud Hypervisor or ch-remote exits on errors. (#7066)

### Alphabetically Sorted CLI Options for ch-remote

To improve readability, ch-remote now prints help information in
alphabetical order. (#7130)

### Notable Bug Fixes

* Error out early when block device serial is too long (#7124)
* Fix partial commands being discarded for `virtio-vsock` (#7195)
* Disable the broken interrupt support for the `rtc_pl031` device to
  prevent spurious guest interrupts (#7199)

### Deprecations

* A default IP (`192.168.249.1`) and mask (`255.255.255.0`) are
 currently assigned  to the `virtio-net` device if no value is specified
 by users. Such behavior is now deprecated.  Users of this behavior will
 receive a warning message and should make adjustments. The behavior
 will be removed in two release cycles (v49.0).

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Bo Chen <bchen@crusoe.ai>
* Demi Marie Obenour <demiobenour@gmail.com>
* Gauthier Jolly <contact@gjolly.fr>
* Hengqi Chen <hengqi.chen@gmail.com>
* Jinank Jain <jinankjain@microsoft.com>
* Jinrong Liang <cloudliang@tencent.com>
* Jean-Philippe Brucker <jean-philippe@linaro.org>
* Maximilian Güntner <code@mguentner.de>
* Muminul Islam <muislam@microsoft.com>
* Nuno Das Neves <nunodasneves@linux.microsoft.com>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Ruoqing He <heruoqing@iscas.ac.cn>
* Songqian Li <sionli@tencent.com>
* Wei Liu <liuwe@microsoft.com>
* Yi Wang <foxywang@tencent.com>
* ninollei <ninollx@hotmail.com>

# v46.0

This release has been tracked in [v46.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+46%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### File-level Locking Support with `--disk`

Now file-level locking is enforced for disk images, provided by users
with `--disk`. This ensures that only a single Cloud Hypervisor instance
can obtain write access to a given disk image at any time, preventing
misconfiguration and avoiding potential data corruption. (#6974)

### Improved Error Reporting with VM Resizing

Instead of returning a generic error `400` (e.g. `BadRequest`), users
now get a more specific error `429` (e.g. `TooManyRequests`) when a
pending VM resizing is not completed. This allows users to better handle
different errors, say retrying the request when applicable. (#7043)

### IPv6 Address Support with `--net`

It is now possible to specify an IPv6 address and mask when creating a
network interface with `--net`. (#7048)

### Experimental AArch64 Support with the MSHV Hypervisor

It is now possible to start VMs on AArch64 platforms when using MSHV
hypervisor. (#7055)

### Deprecated SGX Support

The SGX support now is deprecated with a warning message if it being
used, with the intention to remove its support from our code base in two
release cycles (e.g. v48.0). (#7090)

### Notable Bug Fixes

* Remove `path` as required for `DiskConfig` from the OpenAPI spec file
  (#7017)
* Properly parse PCI capabilities (#7018)
* Reprogram PCI device BAR when its MSE bit is set (#7063)
* Update IOMMU mappings of MMIO regions with BAR reprogram for VFIO
  devices (#7064)
* Avoid resizing VMs to zero vCPUs (#7086)
* Fix extended topology enumeration leaf exposed to the guest (#7087)

### Contributors

Many thanks to everyone who has contributed to our release:

* Bingxin Li <bl497@cam.ac.uk>
* Bo Chen <bchen@crusoe.ai>
* Fabiano Fidêncio <fidencio@northflank.com>
* Gregory Anders <ganders@cloudflare.com>
* Jinank Jain <jinankjain@microsoft.com>
* Julian Stecklina <julian.stecklina@cyberus-technology.de>
* Muminul Islam <muislam@microsoft.com>
* Paolo Bonzini <pbonzini@redhat.com>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Rob Bradford <rbradford@rivosinc.com>
* Ruoqing He <heruoqing@iscas.ac.cn>
* Thomas Prescher <thomas.prescher@cyberus-technology.de>
* abm-77 <andrewmiller77@protonmail.com>

# v45.0

This release has been tracked in [v45.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v44.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Experimental `riscv64` Architecture Support

Cloud Hypervisor now has experimental `riscv64` architecture
support. Details can be found from the [riscv
documentation](docs/riscv.md).

### Alphabetically Sorted CLI Options

To improve the readability of CLI options, the output of the `--help`
now is alphabetically sorted. (#6988)

### Improved Downtime of VM Live Migration

The downtime of VM live migration is reduced via delaying some of the
tearing down process of the source VM after the destination VM is up and
running. (#6987)

### Notable Bug Fixes

* Fix seccomp filters related to http-api thread (#6967)
* Handle cross-page access in the emulator for mshv (#6968)

### Contributors

Many thanks to everyone who has contributed to our release:

* Anatol Belski <anbelski@linux.microsoft.com>
* Andrew Consroe <aconz2@gmail.com>
* Bo Chen <bchen@crusoe.ai>
* Jinank Jain <jinankjain@microsoft.com>
* Jinrong Liang <cloudliang@tencent.com>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Ruoqing He <heruoqing@iscas.ac.cn>
* Stefan Kober <stefan.kober@cyberus-technology.de>
* Wei Liu <liuwe@microsoft.com>

# v44.0

This release has been tracked in [v44.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v44.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Configurable `virtio-iommu` Address Width

The `iommu_address_width` option has been added to `--platform` to allow users
to limit the `virtio-iommu` address space in the guest. (#6900)

### Notable Performance Improvements

The `VIRTIO_BLK_F_SEG_MAX` feature has been enabled for `virtio-block` devices,
which brings significant performance improvements on throughput. (#6885)

The `io_uring` entries are no longer forced to use async helper workers,
delegating the decision to the kernel. This change resolved the issue of having
excessive amount of worker threads when `io_uring` is being used, which is
expected to improve performance, such as reducing memory usage and reduce CPU
contention.

### New Fuzzers

Our continuous fuzzing infrastructure is augmented with two new fuzzers to cover
x86 instruction emulator and `virtio-vsock`.

### Notable Bug Fixes

* Fix short read and short write that impact QCOW and VHDX support. (#6884)
* Various bug fixes on VHDX support. (#6890, #6899)

### Contributors

Many thanks to everyone who has contributed to our release:

* Arvind Vasudev <avasudev@crusoeenergy.com>
* Bo Chen <bo.arvin.chen@gmail.com>
* Nikolay Edigaryev <edigaryev@gmail.com>
* Rob Bradford <rbradford@rivosinc.com>
* Ruoqing He <heruoqing@iscas.ac.cn>
* Wei Liu <liuwe@microsoft.com>

# v43.0

This release has been tracked in [v43.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v43.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Live Migration over TCP Connections

Support has been added to enable direct live migration from two hosts via TCP
connections. This supplements the existing support for migrating over a UNIX
socket which can then be tunnelled as desired. The documentation has been
updated. (#6850)

### Notable Performance Improvements

The `VIRTIO_RING_F_INDIRECT_DESC` feature has been enabled for `virtio-block`
devices. This significantly increases the throughput of the devices with a
small negative impact on latency. (#6826)

### Notable Bug Fixes

* Cloud Hypervisor now accepts VFIO devices that use I/O PCI BARs on non x86-64
  architectures. Whether they function depends on the host PCI host bridge
  support - previously they would be rejected even if the driver did not use
  these BARs. (#6871)
* Command line groups were adjusted to ensure that at least one payload
  parameter was provided if any other VM parameters provided. (#6832)

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Andrew Consroe <aconz2@gmail.com>
* Bo Chen <bo.arvin.chen@gmail.com>
* Jinrong Liang <cloudliang@tencent.com>
* Julian Stecklina <julian.stecklina@cyberus-technology.de>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <rbradford@rivosinc.com>
* Ruoqing He <heruoqing@iscas.ac.cn>
* Wojtek Czekalski <wczekalski@me.com>

# v42.0

This release has been tracked in [v42.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v42.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### SVE/SVE2 Support on AArch64

The SVE and SVE2 feature bits are now propagated through to the guest on
AArch64. (#6678, #6691)

### Notable Bug Fixes

* Reduce latency notification when rate limited (#6672)
* Fix `virtio-console` resizing (#6704)
* Fix resizing when console uses TTY (#6754)
* Avoid deadlock in PCI BAR reprogramming that can occur when adding a new
  `virtio` device to a VM that has been restored (#6775)
* Fix console resizing after VM restore (#6748)
* Fix memory resize error due to incorrect bounds checks (#6736)

### Sponsorships

During this release cycle a new VFIO CI worker has been provided by Crusoe
Energy and a new ARM64 CI worker has been provided by Ubicloud.

### Contributors

Many thanks to everyone who has contributed to our release:

* Alexandru Matei <alexandru.matei@uipath.com>
* Alyssa Ross <hi@alyssa.is>
* Anirudh Rayabharam <anrayabh@microsoft.com>
* BharatNarasimman <bharatn@microsoft.com>
* Bo Chen <chen.bo@intel.com>
* Jinank Jain <jinankjain@microsoft.com>
* Jonas Scholz <Jonas.Scholz@bbscholz.de>
* Nuno Das Neves <nudasnev@microsoft.com>
* Praveen K Paladugu <prapal@linux.microsoft.com>
* Purna Pavan Chandra <paekkaladevi@microsoft.com>
* Rob Bradford <rbradford@rivosinc.com>
* Ruoqing He <heruoqing@iscas.ac.cn>
* Songqian Li <sionli@tencent.com>
* Tom Dohrmann <erbse.13@gmx.de>
* Wei Liu <liuwe@microsoft.com>
* Wenyu Huang <huangwenyuu@outlook.com>
* Yuhong Zhong <yz@cs.columbia.edu>
* wuxinyue <wuxinyue.wxy@antgroup.com>

# v41.0

This release has been tracked in [v41.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v41.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Experimental "Pvmemcontrol" Support

VMM support has been added for this experimental functionality (requires
currently out of tree Linux kernel patches) to allow guests to control its
physical memory properties to allow optimisations and security features.
(#6318, #6467)

### Sandboxing With Landlock Support

Support for restricting the VMM process using the Linux kernel "Landlock" API
has been added  - this can be used to restrict the files (and the read/write
permissions) that the VMM process can access. This adds another layer of
security alongside the existing sycall filters (`seccomp`) - this can be
enabled with `--landlock` and [fully documentated](docs/landlock.md). (#5170)

### Notable Performance Improvements

* Reduced heap allocations in `virtio-net` via the use of a cache of `Iovec`
  structures (#6636)
* Notification suppression ("`EVENT_IDX`") support has been added to
  `virtio-block` giving a 60% improvement in single queue block throughput and
  IOPs performance (#6580)
* Correct size used for `status` field in `virtio-block` state (#6586)

### Notable Bug Fixes

* Avoid panic on out-of-bounds PCI MSI-X access (#6657)
* Fix undefined behaviour on AArch64 leading to wrong optimisation on KVM API
  access (#6647)
* Rust v1.80.0 added use of `fcntl` syscall on debug assertions so this is now
  included in the virtio-device seccomp filters for tests that use this (#6648)
* Short reads are now handled correctly in the `virtio-vsock` device (#6621)
* Fix undefined behaviour on TTY ioctl leading to wrong optimisation (#6568)

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Bo Chen <chen.bo@intel.com>
* Changyuan Lyu <changyuanl@google.com>
* Jinank Jain <jinankjain@microsoft.com>
* Julian Stecklina <julian.stecklina@cyberus-technology.de>
* Muminul Islam <muislam@microsoft.com>
* Nuno Das Neves <nudasnev@microsoft.com>
* Praveen K Paladugu <prapal@linux.microsoft.com>
* Rob Bradford <rbradford@rivosinc.com>
* Songqian Li <sionli@tencent.com>
* Wei Liu <liuwe@microsoft.com>
* Yuanchu Xie <yuanchu@google.com>
* ihciah <ihciah@gmail.com>
* wuxinyue <wuxinyue.wxy@antgroup.com>

# v40.0

This release has been tracked in [v40.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v40.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Support for Restoring File Descriptor Backed Network Devices

It is now possible to pass file descriptors over the HTTP API (and using
`ch-remote`) when restoring to update the file descriptors for network devices.
This enables snapshot & restore functionality for guests using `macvtap` or
other file descriptor backed network devices. (#6286)

### Notable Bug Fixes

* Default values have been removed from required fields in the OpenAPI metadata
  (#6495)
* The help syntax of `ch-remote remove-device` has been improved (#6456)
* A double close of file descriptors has been fixed when using `--serial`
  (#6486)
* To prevent loops a limit on the nesting level for QCOW2 backing files has
  been introduced (#6482)
* Boot time performance has been improved with multiple cores by avoiding
  `cpuid` instructions and by seeding the in kernel file descriptor table
  (#6498, #6478)
* L1 cache details are more likely to be propagated into the guest (#6523)
* The default topology for guests now uses multiple cores rather than sockets
  (#6504)

### Contributors

Many thanks to everyone who has contributed to our release:

* Alexandru Matei <alexandru.matei@uipath.com>
* Bo Chen <chen.bo@intel.com>
* Jinank Jain <jinankjain@microsoft.com>
* Josh Soref <2119212+jsoref@users.noreply.github.com>
* Muminul Islam <muislam@microsoft.com>
* Nuno Das Neves <nudasnev@microsoft.com>
* Omer Faruk Bayram <omer.faruk@sartura.hr>
* Praveen K Paladugu <prapal@linux.microsoft.com>
* Purna Pavan Chandra <paekkaladevi@linux.microsoft.com>
* Rob Bradford <rbradford@rivosinc.com>
* SamrutGadde <samrut.gadde@gmail.com>
* Sean Banko <sbanko@crusoe.ai>
* Songqian Li <sionli@tencent.com>
* Wei Liu <liuwe@microsoft.com>
* Yi Wang <foxywang@tencent.com>
* Yu Li <liyu.yukiteru@bytedance.com>
*

# v39.0

This release has been tracked in [v39.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v39.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Variable Sizing of PCI Apertures for Segments

It is now possible to use `--pci-segment` to adjust the aperture size that
devices 32-bit and 64-bit PCI device BARs will be allocated from. Previously
the address space was equally distributed across all the segments which may
leave insufficient space for devices that require a large 32-bit space. With
this change the weighting per segment can be adjusted. (#6387)

### Direct Booting with bzImages

Support for directly booting Linux from bzImages has been added.(#6200)

### Support for NVIDIA GPUDirect P2P Support

The `x_nv_gpudirect_clique` option was added to `--device` to allow the
configuration of device P2P support with NVIDIA GPUs. (#6235)

### Guest NMI Injection Support

A new API endpoint and `ch-remote` option added for injecting an NMI into the
guest. (#6047)

### Notable Bug Fixes

* Workaround for kernel bug affecting guest IRQ masking on AMD (#6353)
* Correctly cleanup `sigwinch_listener` process (#6208)
* Graceful shutdown of HTTP API thread (#6248, #6247)
* Fix `queue_affinity` option in OpenAPI metadata (#6268)
* Fix documentation to indicate only stream mode is supported by `virtio-vsock`
  (#6306)
* Fix `virtio-fs` tag validation (#6358, #6359)
* Add missing `pvpanic` device to OpenAPI metadata (#6372)
* Fixes for nested virtualization with VFIO devices (#6110, #6298, #6297,
  #6319)
* Fix for backing file for `virtio-mem` regions with snapshot/restore (#6337,
  #6338)
* Explicitly mark FDs used for network devices as invalid across
  snapshot/restore (#6332, #6286)
* Improve `event-monitor` events around reboot (#6277, #6274)
* Fix potential deadlock around paused devices during live migration (#6293)
* Fix panic when running `ch-remote` with no subcommand (#6230)
* Fix hotplug of `virtio` devices after snapshot/restore and live migration
  (#6326, #6265)

### Contributors

Many thanks to everyone who has contributed to our release:

* Alexandru Matei <alexandru.matei@uipath.com>
* Andrew Carp <acarp@crusoeenergy.com>
* Bo Chen <chen.bo@intel.com>
* Bouke van der Bijl <i@bou.ke>
* Chris Webb <chris@arachsys.com>
* Jinank Jain <jinankjain@microsoft.com>
* Lucas Jacques <contact@lucasjacques.com>
* Muminul Islam <muislam@microsoft.com>
* Nuno Das Neves <nudasnev@microsoft.com>
* Ravi kumar Veeramally <ravikumar.veeramally@intel.com>
* Rob Bradford <rbradford@rivosinc.com>
* Ruslan Mstoi <ruslan.mstoi@intel.com>
* Stefan Nuernberger <stefan.nuernberger@cyberus-technology.de>
* Thomas Barrett <tbarrett@crusoeenergy.com>
* Wei Liu <liuwe@microsoft.com>
* Yi Wang <foxywang@tencent.com>

# v38.0

This release has been tracked in [v38.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v38.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Group Rate Limiter on Block Devices

Users now can throttle a group of block devices with the new
`--rate-limiter-group` option. Details can be found from the [I/O
Throttling documentation](docs/io_throttling.md)

### CPU Pinning Support for Block Device Worker Thread

Users now have the option to pin virt-queue threads for block devices
to specific host cpus.

### Optimized Boot Time with Parallel Memory Prefault

The boot time with `prefault` option enabled is optimized via parallel
memory prefault.

### New 'debug-console' Device

A 'debug-console' device is added to provide a user-configurable debug
port for logging guest information. Details can be found from the [Debug
IO Ports documentation](docs/debug-port.md).

### Improved VFIO Device Support

All non-emulated MMIO regions of VFIO devices are now mapped to the VFIO
container, allowing PCIe P2P between all VFIO devices on the same
VM. This is required for a wide variety of multi-GPU workloads involving
GPUDirect P2P (DMA between two GPUs), GPUDirect RDMA (DMA between a GPU
and an IB device).

### Extended CPU Affinity Support

Users now can set the vcpu affinity to a host CPU with index larger
than 255.

### Notable Bug Fixes

* Enable HTT flag to avoid crashing cpu topology enumeration software
such as hwloc in the guest (#6146)
* Fix several security advisories from dependencies (#6134, #6141)
* Handle non-power-of-two CPU topology properly (#6062)
* Various bug fixes around `virtio-vsock`(#6080, #6091, #6095)
* Enable nested virtualization on AMD if supported (#6106)
* Align VFIO devices PCI BARs naturally (#6196)

### Contributors

Many thanks to everyone who has contributed to our release:
* Alyssa Ross <hi@alyssa.is>
* Bo Chen <chen.bo@intel.com>
* Daniel Farina <daniel@ubicloud.com>
* Jinank Jain <jinankjain@microsoft.com>
* Muminul Islam <muislam@microsoft.com>
* Peteris Rudzusiks <rye@stripe.com>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Ravi kumar Veeramally <ravikumar.veeramally@intel.com>
* Rob Bradford <rbradford@rivosinc.com>
* Ruslan Mstoi <ruslan.mstoi@intel.com>
* Sean Banko <sbanko@crusoeenergy.com>
* Thomas Barrett <tbarrett@crusoeenergy.com>
* Wei Liu <liuwe@microsoft.com>
* Yi Wang <foxywang@tencent.com>
* acarp <acarp@crusoeenergy.com>

# v37.0

This release has been tracked in [v37.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v37.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Long Term Support (LTS) Release

This release is a LTS release. Point releases for bug fixes will be made
for the next 18 months; live migration and live upgrade will be
supported between the point releases of the LTS.

### Multiple PCI segments Support for 32-bit VFIO devices

Now VFIO devices with 32-bit memory BARs can be attached to non-zero PCI
segments on the guest, allowing users to have more 32-bit devices and
assign such devices to appropriate NUMA nodes for better performance.

### Configurable Named TAP Devices

Named TAP devices now accepts IP configuration from users, such as IP
and MAC address, as long as the named TAP device is created by Cloud
Hypervisor (e.g. not existing TAP devices).

### TTY Output from Both Serial Device and Virtio Console

Now legacy serial device and virtio console can be set as TTY mode as
the same time. This allows users to capture early boot logs with the
legacy serial device without losing performance benefits of using
virtio-console, when appropriate kernel configuration is used (such as
using kernel command-line `console=hvc0 earlyprintk=ttyS0` on x86).

### Faster VM Restoration from Snapshots

The speed of VM restoration from snapshots is improved with a better
implementation of deserializing JSON files.

### Notable Bug Fixes

* Fix aio backend behavior for block devices when writeback cache
  disabled (#5930)
* Fix PvPanic device PCI BAR alignment (#5956)
* Bug fix to OpenAPI specification file (#5967)
* Error out early for live migration when TDX is enabled (#6025)

### Contributors

Many thanks to everyone who has contributed to our release:

* Bo Chen <chen.bo@intel.com>
* Jinank Jain <jinankjain@microsoft.com>
* Markus Sütter <markus.suetter@secunet.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <rbradford@rivosinc.com>
* Rui Chang <rui.chang@arm.com>
* Ruslan Mstoi <ruslan.mstoi@intel.com>
* Thomas Barrett <tbarrett@crusoeenergy.com>
* Wei Liu <liuwe@microsoft.com>
* Yi Wang <foxywang@tencent.com>
* Yong He <alexyonghe@tencent.com>

# v36.0

This release has been tracked in [v36.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v36.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Command Line Changes

We switched back to use the `clap` crate to create our command line,
since the `argh` crate is barely maintained. There were several syntax
changes:

* All `--option value` commands now are `--option=value`.
* The `--disk DISK1 --disk DISK2` command now is `--disk DISK1 DISK2`.
* The `-v -v -v`command now is `-vvv`.

Note: the released binary size increased around 0.3M due to this change.

### Enabled Features Reported via API Endpoint and CLI

Now the enabled (Cargo) features of the running Cloud Hypervisor
instance can be queried via API endpoint (`/vmm.ping`) and CLI
(`--version -v`).

### NUMA Support for PCI segments

The `--numa` command is augmented with a new option `pci_segment=`, so
that users can define the relationship between PCI segments and NUMA
nodes. Examples can be found from the [memory documentation](docs/memory.md)

### CPU Topology Support on AMD Platforms

Now the CPU topology on x86_64 platforms supports multiple vendors.

### Unix Socket Backend for Serial Port

The `--serial` command is augmented with a new option `socket=`, allowing
users to access the serial port using a Unix socket.

### AIO Backend for Block Devices

An AIO backend is added for `virtio-block` devices to improve block
device performance when the `io_uring` feature is not supported by the
host Operating System.

### Documentation Improvements

* New [documentation](docs/coverage.md) for collecting coverage data
* Various typo fixes

### Notable Bug Fixes

* Fix a deadlock when TDX is enabled (#5845)
* Only advertise AMX feature bits to guest when the AMX cpu feature is
  enabled (#5834)
* Correct default value for vCPU topology on AArch64 (#5893)

### Contributors

Many thanks to everyone who has contributed to our release:

*  Anatol Belski <anbelski@linux.microsoft.com>
*  Bo Chen <chen.bo@intel.com>
*  Dario Nieuwenhuis <dirbaio@dirbaio.net>
*  Jianyong Wu <jianyong.wu@arm.com>
*  Jinank Jain <jinankjain@microsoft.com>
*  Muminul Islam <muislam@microsoft.com>
*  Praveen K Paladugu <prapal@linux.microsoft.com>
*  Ravi kumar Veeramally <ravikumar.veeramally@intel.com>
*  Rob Bradford <rbradford@rivosinc.com>
*  Thomas Barrett <tbarrett@crusoeenergy.com>
*  Wei Liu <liuwe@microsoft.com>
*  Yi Wang <foxywang@tencent.com>
*  dom.song <dom.song@amperecomputing.com>

# v35.0

This release has been tracked in [v35.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v35.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### `virtio-vsock` Support for Linux Guest Kernel v6.3+

Since kernel v6.3, a vsock packet can be included in a single descriptor,
instead of being split over two descriptors. Our `virtio-vsock` implementation
now support both situations.

### User Specified Serial Number for `virtio-block`

A new option `serial` is added to the `--block` command that allows users to
specify a serial number for block devices which will be guest visible.

### vCPU TSC Frequency Included in Migration State

This ensures migration works correctly between hosts that have different TSC
frequencies if the guest is running with TSC as the source of timekeeping.

### Notable Bug Fixes

* Disallow concurrent CPU resizing (#5668)
* Handle APIC EOI message for MSHV (#5681)
* Use wrapping add for memory offset from instruction emulator (#5719)
* Add global spell check via the 'typos' GitHub action (#5720)
* Ensure probing reads are block size aligned (#5727)
* Multiple bug fixes around the latency counter for block devices
  (#5712, #5750, #5762, #5763)
* Replace unsound `static mut` with `once_cell` (#5772)

### Contributors

Many thanks to everyone who has contributed to our release:

*  Alyssa Ross <hi@alyssa.is>
*  Anatol Belski <anbelski@linux.microsoft.com>
*  Bo Chen <chen.bo@intel.com>
*  Christian Blichmann <cblichmann@google.com>
*  Jianyong Wu <jianyong.wu@arm.com>
*  Jinank Jain <jinankjain@microsoft.com>
*  Julian Stecklina <julian.stecklina@cyberus-technology.de>
*  Omer Faruk Bayram <omer.faruk@sartura.hr>
*  Philipp Schuster <philipp.schuster@cyberus-technology.de>
*  Rob Bradford <rbradford@rivosinc.com>
*  Ruslan Mstoi <ruslan.mstoi@intel.com>
*  Thomas Barrett <tbarrett@crusoeenergy.com>
*  Wei Liu <liuwe@microsoft.com>
*  Yi Wang <foxywang@tencent.com>
*  zhongbingnan <zhongbingnan@bytedance.com>

# v34.0

This release has been tracked in [v34.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v34.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Paravirtualised Panic Device Support

A new device has been added that can communicate when the guest kernel has
panicked and share those details with the VMM. This is controlled with a new
`--pvpanic` command line option and JSON API change equivalent. (#5526)

### Improvements to VM Core Dump

Requesting to dump the guest memory as core dump will now transparently pause
the VM if required; returning to the original state after. (#5604)

### QCOW2 Support for Backing Files

The support for QCOW2 files has been enhanced to include support for using
backing files. (#5573)

### Minimum Host Kernel Bump

The minimum supported host kernel is now 5.13 in order to incorporate a bug fix
for `KVM_FEATURE_ASYNC_PF_INT` functionality. (#5626)

### Notable Bug Fixes

* The x86 emulator is only compiled in if MSHV is compiled in (the kernel
  carries out this job with KVM) (#5561).
* A regression has been fixed in VFIO support for devices that use MSI rather
  than MSI-X (#5658).
* When triggering a VM shutdown or reset via I/O the vCPU thread will be
  blocked until that asynchronous event has been received (#5645).
* Pausing a VM is now a synchronous action with the request only completing
  when all vCPUs are paused (#5611).
* Event monitor support now correctly supports concurrent access (#5633).
* Bug fixes for the QCOW2 file support (#5573).

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Anatol Belski <anbelski@linux.microsoft.com>
* Bo Chen <chen.bo@intel.com>
* Changyuan Lyu <changyuanl@google.com>
* Christian Blichmann <cblichmann@google.com>
* Manish Goregaokar <manishsmail@gmail.com>
* Omer Faruk Bayram <omer.faruk@sartura.hr>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Praveen K Paladugu <prapal@linux.microsoft.com>
* Rob Bradford <rbradford@rivosinc.com>
* Ruslan Mstoi <ruslan.mstoi@intel.com>
* Yi Wang <foxywang@tencent.com>
* Yong He <alexyonghe@tencent.com>
* Yu Li <liyu.yukiteru@bytedance.com>
* dom.song <dom.song@amperecomputing.com>

# v33.0

This release has been tracked in [v33.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v33.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### D-Bus based API

A D-Bus based API has been added as an alternative to the existing REST
API. This feature is gated by the `dbus_api` feature. Details can be
found in the [API documentation](docs/api.md).

### Expose Host CPU Cache Details for AArch64

Now the CPU cache information on the host is properly exposed to the
guest on AArch64.

### Notable Bug Fixes

* Report errors explicitly to users when VM failed to boot (#5453)
* Fix VFIO on platforms with non-4k page size (#5450, #5469)
* Fix TDX initialization (#5454)
* Ensure all guest memory regions are page-size aligned (#5496)
* Fix seccomp filter lists related to virtio-console, serial and pty
  (#5506, #5524)
* Populate APIC ID properly (#5512)
* Ignore and warn TAP FDs in more situations (#5522)

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Anatol Belski <anbelski@linux.microsoft.com>
* Bo Chen <chen.bo@intel.com>
* Jianyong Wu <jianyong.wu@arm.com>
* Omer Faruk Bayram <omer.faruk@sartura.hr>
* Rafael Mendonca <rafaelmendsr@gmail.com>
* Ravi kumar Veeramally <ravikumar.veeramally@intel.com>
* Rob Bradford <rbradford@rivosinc.com>
* Ruslan Mstoi <ruslan.mstoi@intel.com>
* Yu Li <liyu.yukiteru@bytedance.com>
* zhongbingnan <zhongbingnan@bytedance.com>

# v32.0

This release has been tracked in [v32.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v32.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Increased PCI Segment Limit

The maximum number of PCI segments that can be used is now 96 (up from 16).

### API Changes

* The VmmPingResponse now includes the PID as well as the build details.
  (#5348)

### Notable Bug Fixes

* Ignore and warn TAP FDs sent via the HTTP request body (#5350)
* Properly preserve and close valid FDs for TAP devices (#5373)
* Only use `KVM_ARM_VCPU_PMU_V3` if available (#5360)
* Only touch the tty flags if it's being used (#5343)
* Fix seccomp filter lists for vhost-user devices (#5361)
* The number of vCPUs is capped at the hypervisor maximum (#5357)
* Fixes for TTY reset (#5414)
* CPU topology fixes on MSHV (#5325)
* Seccomp fixes for older distributions (#5397)

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Anatol Belski <anbelski@linux.microsoft.com>
* Bo Chen <chen.bo@intel.com>
* Hao Xu <howeyxu@tencent.com>
* Muminul Islam <muislam@microsoft.com>
* Omer Faruk Bayram <omer.faruk@sartura.hr>
* Rafael Mendonca <rafaelmendsr@gmail.com>
* Rob Bradford <rbradford@rivosinc.com>
* Ruslan Mstoi <ruslan.mstoi@intel.com>
* Smit Gardhariya <gardhariya.smit@gmail.com>
* Wei Liu <liuwe@microsoft.com>

# v31.1

This is a bug fix release. The following issues have been addressed:

* Ignore and warn TAP FDs sent via the HTTP request body (#5350)
* Properly preserve and close valid FDs for TAP devices (#5373)
* Only use `KVM_ARM_VCPU_PMU_V3` if available (#5360)
* Only touch the tty flags if it's being used (#5343)
* Fix seccomp filter lists for vhost-user devices (#5361)

# v31.0

This release has been tracked in [v31.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v31.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Update to Latest `acpi_tables`

Adapted to the latest [acpi_tables](https://github.com/rust-vmm/acpi_tables).
There has been significant API changes in the crate.

### Update Reference Kernel to 6.2

Updated the recommended guest kernel version from 6.1.6 to 6.2.

### Improvements on Console `SIGWINCH` Handler

A separate thread had been created to capture the `SIGWINCH` signal and resize
the guest console. Now the thread is skipped if the console is not resizable.

Two completely different code paths existed for handling console resizing, one
for `tty` and the other for `pty`. That makes the understanding of the console
handling code unnecessarily complicated. Now the code paths are unified. Both
`tty` and `pty` are supported in single `SIGWINCH` handler. And the new handler
can works with kernel versions earlier than v5.5.

### Remove Directory Support from `MemoryZoneConfig::file`

Setting a directory to `MemoryZoneConfig::file` is no longer supported.

Before this change, user can set a directory to `file` of the `--memory-zone`
option. In that case, a temporary file will be created as the backing file for
the `mmap(2)` operation. This functionality has been unnecessary since we had
the native support for hugepages and allocating anonymous shared memory.

### Documentation Improvements

* Various improvements in API document
* Improvements in Doc comments
* Updated Slack channel information in README

### Notable Bug Fixes

* Fixed the offset setting while removing the entire mapping of `vhost-user` FS
  client.
* Fixed the `ShutdownVmm` and `Shutdown` commands to call the correct API
  endpoint.

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Bo Chen <chen.bo@intel.com>
* Daniel Farina <daniel@fdr.io>
* Dom <peng6662001@163.com>
* Hao Xu <howeyxu@tencent.com>
* Muminul Islam <muislam@microsoft.com>
* Omer Faruk Bayram <omer.faruk@sartura.hr>
* Ravi kumar Veeramally <ravikumar.veeramally@intel.com>
* Rob Bradford <rbradford@rivosinc.com>
* Ruslan Mstoi <ruslan.mstoi@intel.com>
* Smit Gardhariya <gardhariya.smit@gmail.com>
* Yang <ailin.yang@intel.com>
* Yong He <alexyonghe@tencent.com>

# v30.0

This release has been tracked in [v30.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v30.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Command Line Changes for Reduced Binary Size

The `clap` crate was replaced by the `argh` crate to create our command
line, which reduced our release binary size from 3.6MB to 3.3MB. There
were several syntax changes:

* All `--option=value` commands now are `--option value`.
* The `--disk DISK1 DISK2` command now is `--disk DISK1 --disk DISK2`.
* The  `-vvv` command now is `-v -v -v`

### Basic vfio-user Server Support

Our `vfio-user` crate is extended to provide basic server side support
with an example of gpio vfio-user device. This crate now is moved to [its
own repository](https://github.com/rust-vmm/vfio-user) under the
`rust-vmm` organization.

### Heap Profiling Support

A new building target is added for profiling purposes with examples of
heap profiling using `dhat` gated by the `dhat-heap` feature.

### Documentation Improvements

The documentation on Intel TDX is expanded with details of the building
and using [TD-Shim](https://github.com/confidential-containers/td-shim),
references to [TDX Linux](https://github.com/intel/tdx-linux), and
version information of guest/host kernel/TDVF/TDShim being tested. Also,
a new 'heap profiling' documentation is added with improvements on the
existing 'profiling' documentation.

### Notable Bug Fixes

* Close FDs for TAP devices that are provided to VM (#5199, #5206)
* Set vcpu thread status properly and signal `exit_evt` upon thread exit (#5211)
* Populate CPUID leaf 0x4000_0010 (TSC frequency) (#5178, #5179)
* Inform the TPM guest driver upon failed TPM requests on the host (#5151)
* Bug fix to OpenAPI specification file (#5186)

### Contributors

Many thanks to everyone who has contributed to our release:

* Anatol Belski <anbelski@linux.microsoft.com>
* Anirudh Rayabharam <anrayabh@linux.microsoft.com>
* Bo Chen <chen.bo@intel.com>
* Jinank Jain <jinankjain@microsoft.com>
* Kaihang Zhang <kaihang.zhang@smartx.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Praveen K Paladugu <prapal@linux.microsoft.com>
* Ravi kumar Veeramally <ravikumar.veeramally@intel.com>
* Rob Bradford <robert.bradford@intel.com>
* Ruslan Mstoi <ruslan.mstoi@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>
* Yong He <alexyonghe@tencent.com>
* Yu Li <liyu.yukiteru@bytedance.com>

# v28.2
This is a bug fix release. The following issues have been addressed:

* Fix QCOW2 refcount table size (#5034)
* Fix unpause support on MSHV in dual binary (#5037)
* Threads inside `virtio` devices are now shutdown on reboot (#5095)

# v29.0

This release has been tracked in [v29.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v29.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Release Binary Supports Both MSHV and KVM

On `x86-64` the binary included in releases supports both the KVM and MSHV
hypervisor with runtime detection to identify the correct hypervisor to use.

### Snapshot/Restore and Live Migration Improvements

Improvements have been made to the data structures used for both live migration
and snapshot/restore. Unfortunately this has broken compatibility with older
versions (support for migrating between major versions is not yet officially
supported but has worked for some versions.)

### Heap Allocation Improvements

Improvements have been made to the volume of heap allocations when running with
`virtio-block` devices along with a reduction in the peak heap size.

### `ch-remote` Improvements

Support for "pinging" the VMM and shutting the VMM down have been added to
`ch-remote`.

### `AArch64` Documentation Integration

The documentation for `AArch64` support has been integrated into the main
README.

### `virtio-block` Counters Enhancement

The counters for the `virtio-block` device has extended to include min/mean/max
latency counters.

### TCP Offload Control

The `virtio-net` device has gained support for controlling the enabling of
checksum and offloading. This allows the device to be used in environments
where the hardware lacks support for the offloading.

### Notable Bug Fixes

* Update dependencies including a version of `linux-loader` that addresses an
  infinite loop issue ([details](https://github.com/rust-vmm/linux-loader/security/advisories/GHSA-52h2-m2cf-9jh6))
* Fix bugs related to `virtio-net` including an integer overflow issue
  (#4924, #4949)
* Use host `cpuid` information for L2 cache for older KVM on x86 (#4920)
* Memory mapped into the guest is now marked as non-dumpable which prevents large core files (#5016)
* Fix QCOW2 refcount table size (#5034)
* Fix unpause support on MSHV in dual binary (#5037)
* Threads inside `virtio` devices are now shutdown on reboot (#5095)

### Removals

No functionality has been removed in this release.

### Deprecations

* Support for specifying a directory with `MemoryZoneConfig::file` or
  `MemoryConfig::file` has been deprecated. This was originally used for
  supporting huge pages or shared memory backing which is now natively supported
  (#5085)

### Contributors

Many thanks to everyone who has contributed to our release:

* Bo Chen <chen.bo@intel.com>
* Claudio Fontana <claudio.fontana@gmail.com>
* Hao Xu <howeyxu@tencent.com>
* Henry Wang <Henry.Wang@arm.com>
* Jinank Jain <jinankjain@microsoft.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Philipp Schuster <philipp.schuster@cyberus-technology.de>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Shuaiyi Zhang <zhangsy28@lenovo.com>
* Wei Liu <liuwe@microsoft.com>
* Yong He <alexyonghe@tencent.com>
* Yuji Hagiwara <yuuzi41@gmail.com>


# v28.1
This is a bug fix release. The following issues have been addressed:

* Update dependencies including a version of `linux-loader` that
addresses an infinite loop issue ([details](https://github.com/rust-vmm/linux-loader/security/advisories/GHSA-52h2-m2cf-9jh6))
* Fix bugs related to `virtio-net` including an integer overflow issue
  (#4924, #4949)
* Use host `cpuid` information for L2 cache for older KVM on x86 (#4920)
* Improve README and documentation

# v28.0

This release has been tracked in [v28.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v28.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Community Engagement (Reminder)

Just a reminder that we have a new mailing list to support broader community
discussions.  Please consider
[subscribing](https://lists.cloudhypervisor.org/g/dev/). We plan to use
this to announce a regular meeting for those interested in talking about Cloud
Hypervisor development.

### Long Term Support (LTS) Release

This is the first version of Cloud Hypervisor to be released under the LTS
release process. Point releases for bug fixes will be made for the next 18
months; live migration and live upgrade will be supported between the point
releases of the LTS.

### Virtualised TPM Support

Support for adding an emulated CRB TPM has been added. This has its own [TPM
documentation](docs/tpm.md).

### Transparent Huge Page Support

By default, but controllable through `--memory thp=off` if it possible to back
the guest memory with Transparent Huge Pages (no file backing/`shared=off`)
then this will be used resulting in improved boot performance.

### README Quick Start Improved

The README has been refreshed with the quick start guide updated to reflect the
different firmware options and to recommend the use of pre-built binaries.

### Notable Bug Fixes

* Inappropriate Copy-on-Write of pinned pages (e.g. VFIO) leading to higher
  memory consumption (#4835)
* Multiple `virtio` device bug fixes found through fuzzing (#4859, #4799)
* Large QCOW files (> 4TiB) are now supported (#4767)
* Support for > 31 vCPUS on aarch64 (#4863)
* Multiple fixes to OpenAPI specification file (#4720, #4811)
* Programming of the MSI-X table has been optimised leading to faster boot on
  newer Linux kernels (#4744)
* Error on reboot from race to close TAP devices (#4871)
* Non-spec compliant virtio-block read-only support (#4888)

### Removals

The following functionality has been removed:

* Support for non-PVH firmware booting has been removed (#4511)
* I/O ports used for older versions of firmware have been removed (#3926)
* Deprecated API options for kernel/cmdline/initramfs have been removed (#4737)

### Contributors

Many thanks to everyone who has contributed to our release:

* Anatol Belski <anbelski@linux.microsoft.com>
* Bo Chen <chen.bo@intel.com>
* Fabiano Fidêncio <fabiano.fidencio@intel.com>
* Jianyong Wu <jianyong.wu@arm.com>
* Jinank Jain <jinankjain@microsoft.com>
* Jinrong Liang <cloudliang@tencent.com>
* lv.mengzhao <lv.mengzhao@zte.com.cn>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Praveen K Paladugu <prapal@linux.microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>

# v27.0

This release has been tracked in [v27.0
group](https://github.com/orgs/cloud-hypervisor/projects/6/views/4?filterQuery=release%3A%22Release+v27.0%22)
of our [roadmap project](https://github.com/orgs/cloud-hypervisor/projects/6/).

### Community Engagement

A new mailing list has been created to support broader community discussions.
Please consider [subscribing](https://lists.cloudhypervisor.org/g/dev/); an
announcement of a regular meeting will be announced via this list shortly.

### Prebuilt Packages

Prebuilt packages are now available. Please see [this
document](https://github.com/cloud-hypervisor/obs-packaging/blob/main/README.md)
on how to install. These packages also include packages for the different
firmware options available.

### Network Device MTU Exposed to Guest

The MTU for the TAP device associated with a `virtio-net` device is now exposed
to the guest. If the user provides a MTU with `--net mtu=..` then that MTU is
applied to created TAP interfaces. This functionality is also exposed for
`vhost-user-net` devices including those created with the reference backend
(#4658, #4676.)

### Boot Tracing

Support for generating a trace report for the boot time has been added
including a script for generating an SVG from that trace (#4659.)

### Simplified Build Feature Flags

The set of feature flags, for e.g. experimental features, have been simplified:

* `msvh` and `kvm` features provide support for those specific hypervisors
  (with `kvm` enabled by default),
* `tdx` provides support for Intel TDX; and although there is no MSHV support
  now it is now possible to compile with the `mshv` feature (#4696,)
* `tracing` adds support for boot tracing,
* `guest_debug` now covers both support for gdbing a guest (formerly `gdb`
  feature) and dumping guest memory.

The following feature flags were removed as the functionality was enabled by
default: `amx`, `fwdebug`, `cmos` and `common` (#4679, #4632.)

### Asynchronous Kernel Loading

AArch64 has gained support for loading the guest kernel asynchronously like
x86-64. (#4538)

### GDB Support for AArch64

GDB stub support (accessed through `--gdb` under `guest_debug` feature) is now
available on AArch64 as well as as x86-64.

### Notable Bug Fixes

* This version incorporates a version of `virtio-queue` that addresses an issue
  where a rogue guest can potentially DoS the VMM (rust-vmm/vm-virtio#196.)
* Improvements around PTY handling for `virtio-console` and serial devices
  (#4520, #4533, #4535.)
* Improved error handling in virtio devices (#4626, #4605, #4509, #4631, #4697)

### Deprecations

Deprecated features will be removed in a subsequent release and users should
plan to use alternatives.

* Booting legacy firmware (compiled without a PVH header) has been deprecated.
  All the firmware options (Cloud Hypervisor OVMF and Rust Hypervisor Firmware)
  support booting with PVH so support for loading firmware in a legacy mode is no
  longer needed. This functionality will be removed in the next release.

### Contributors

Many thanks to everyone who has contributed to our release:

* Anatol Belski <anbelski@linux.microsoft.com>
* Bo Chen <chen.bo@intel.com>
* James O. D. Hunt <james.o.hunt@intel.com>
* Jianyong Wu <jianyong.wu@arm.com>
* Markus Napierkowski <markus.napierkowski@cyberus-technology.de>
* Michael Zhao <michael.zhao@arm.com>
* Nuno Das Neves <nudasnev@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Smit Gardhariya <sgardhariya@microsoft.com>
* Wei Liu <liuwe@microsoft.com>

# v26.0

### SMBIOS Improvements via `--platform`

`--platform` and the appropriate API structure has gained support for supplying
OEM strings (primarily used to communicate metadata to systemd in the guest)
(#4319, #4446) and support for specifying the UUID (#4389.)

### Unified Binary MSHV and KVM Support

Support for both the MSHV and KVM hypervisors can be compiled into the same
binary with the detection of the hypervisor to use made at runtime.

### Notable Bug Fixes

* The prefetchable flag is preserved on BARs for VFIO devices (#4353, #4454)
* PCI Express capabilities for functionality we do not support are now filtered
  out (#4456)
* GDB breakpoint support is more reliable (#4354, #4363)
* `SIGINT` and `SIGTERM` signals are now handled before the VM has booted
  (#4269, #4293)
* Multiple API event loop handling bug fixes (#4309, #4362)
* Incorrect assumptions in virtio queue numbering were addressed, allowing
  the`virtio-fs` driver in OVMF to be used (#4341, #4314)
* VHDX file format header fix (#4291)
* The same VFIO device cannot be added twice (#4453, #4463)
* SMBIOS tables were being incorrectly generated (#4442)

### Deprecations

Deprecated features will be removed in a subsequent release and users should
plan to use alternatives.

* The top-level `kernel` and `initramfs` members on the `VmConfig` have been
  moved inside a `PayloadConfig` as the `payload` member. The OpenAPI document
  has been updated to reflect the change and the old API members continue to
  function and are mapped to the new version. The expectation is that these old
  versions will be removed in the v28.0 release.

### Removals

The following functionality has been removed:

* The unused `poll_queue` parameter has been removed from `--disk` and
  equivalent. This was residual from the removal of the `vhost-user-block`
  spawning feature (#4402.)

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Anatol Belski <ab@php.net>
* Archana Shinde <archana.m.shinde@intel.com>
* Bo Chen <chen.bo@intel.com>
* lizhaoxin1 <Lxiaoyouling@163.com>
* Maximilian Nitsch <maximilian.nitsch@d3tn.com>
* Michael Zhao <michael.zhao@arm.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Steven Dake <sdake@lambdal.com>
* Wei Liu <liuwe@microsoft.com>

# v25.0

### `ch-remote` Improvements

The `ch-remote` command has gained support for creating the VM from a JSON
config and support for booting and deleting the VM from the VMM.

### VM "Coredump" Support

Under the `guest_debug` feature flag it is now possible to extract the memory
of the guest for use in debugging with e.g. the `crash` utility. (#4012)

### Notable Bug Fixes

* Always restore console mode on exit (#4249, #4248)
* Restore vCPUs in numerical order which fixes aarch64 snapshot/restore (#4244)
* Don't try and configure `IFF_RUNNING` on TAP devices (#4279)
* Propagate configured queue size through to vhost-user backend (#4286)
* Always Program vCPU CPUID before running the vCPU to fix running on Linux
  5.16 (#4156)
* Enable ACPI MADT "Online Capable" flag for hotpluggable vCPUs to fix newer
  Linux guest

### Removals

The following functionality has been removed:

* The `mergeable` option from the `virtio-pmem` support has been removed
  (#3968)
* The `dax` option from the `virtio-fs` support has been removed (#3889)

### Contributors

Many thanks to everyone who has contributed to our release:

* Dylan Bargatze <dbargatz@users.noreply.github.com>
* Jinank Jain <jinankjain@microsoft.com>
* Michael Zhao <michael.zhao@arm.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>
* Yi Wang <wang.yi59@zte.com.cn>

# v24.0

### Bypass Mode for `virtio-iommu`

`virtio-iommu` specification describes how a device can be attached by default
to a bypass domain. This feature is particularly helpful for booting a VM with
guest software which doesn't support `virtio-iommu` but still need to access
the device. Now that Cloud Hypervisor supports this feature, it can boot a VM
with Rust Hypervisor Firmware or OVMF even if the `virtio-block` device exposing
the disk image is placed behind a virtual IOMMU.

### Ensure Identifiers Uniqueness

Multiple checks have been added to the code to prevent devices with identical
identifiers from being created, and therefore avoid unexpected behaviors at boot
or whenever a device was hot plugged into the VM.

### Sparse Mmap support

Sparse mmap support has been added to both VFIO and vfio-user devices. This
allows the device regions that are not fully mappable to be partially mapped.
And the more a device region can be mapped into the guest address space, the
fewer VM exits will be generated when this device is accessed. This directly
impacts the performance related to this device.

### Expose Platform Serial Number

A new `serial_number` option has been added to `--platform`, allowing a user to
set a specific serial number for the platform. This number is exposed to the
guest through the SMBIOS.

### Notable Bug Fixes

* Fix loading RAW firmware (#4072)
* Reject compressed QCOW images (#4055)
* Reject virtio-mem resize if device is not activated (#4003)
* Fix potential mmap leaks from VFIO/vfio-user MMIO regions (#4069)
* Fix algorithm finding HOB memory resources (#3983)

### Notable Improvements

* Refactor interrupt handling (#4083)
* Load kernel asynchronously (#4022)
* Only create ACPI memory manager DSDT when resizable (#4013)

### Deprecations

Deprecated features will be removed in a subsequent release and users should
plan to use alternatives

* The `mergeable` option from the `virtio-pmem` support has been deprecated
  (#3968)
* The `dax` option from the `virtio-fs` support has been deprecated (#3889)

### New on the Website

A new blog post [Achieving Bare Metal Performance Within a Virtual
Machine](https://www.cloudhypervisor.org/blog/achieving-bare-metal-performance-within-a-virtual-machine)
has been added to the Cloud Hypervisor website.

### Contributors

Many thanks to everyone who has contributed to our release:

* Anatol Belski <anbelski@linux.microsoft.com>
* Bo Chen <chen.bo@intel.com>
* Fabiano Fidêncio <fabiano.fidencio@intel.com>
* LiHui <andrewli@kubesphere.io>
* Maksym Pavlenko <pavlenko.maksym@gmail.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Steven Dake <steven.dake@gmail.com>
* Vincent Batts <vbatts@hashbangbash.com>
* Wei Liu <liuwe@microsoft.com>

# v23.1

This is a bug fix release. The following issues have been addressed:

* Add some missing seccomp rules
* Remove `virtio-fs` filesystem entries from config on removal
* Do not delete API socket on API server start (#4026)
* Reject `virtio-mem` resize if the guest doesn't activate the device
* Fix OpenAPI naming of I/O throttling knobs

# v23.0

### vDPA Support

A vDPA device has a datapath that complies with the virtio specification but
with a vendor specific control path. The addition of `--vdpa` and the REST API
equivalent allows the use of these devices with Cloud Hypervisor.

### Updated OS Support list

The list of officially supported and tested OS versions has been updated to
include Ubuntu "jammy" 22.04 and EOLed versions removed.

### `AArch64` Memory Map Improvements

The memory map when running on `AArch64` has been improved for the handling of
the UEFI region which means that the booted guest OS now has full access to its
allocated RAM. (#3938)

### `AMX` Support

Under a compile time gate of `amx` it is possible compile in support for the
`AMX` instruction set extension for guest use. This also requires runtime
enabling with `--cpu features=amx`.

### Notable Bug Fixes

* Generate error when incorrect HTTP method used for some API endpoints (#3887)
* CMOS based reset is now available to support rebooting on "jammy" (#3914)
* ACPI tables are not produced for memory hotplug when running with
  `virtio-mem` (#3883)
* `virtio-iommu` backed PCI segments are now comprehensively placed behind the
  vIOMMU (#3870)
* Seccomp rules have been extended for `virtio-fs` to support direct access
  (#3848)

### Deprecations

Deprecated features will be removed in a subsequent release and users should
plan to use alternatives

* The `mergeable` option from the `virtio-pmem` support has been deprecated
  (#3968)
* The `dax` option from the `virtio-fs` support has been deprecated (#3889)

### Contributors

Many thanks to everyone who has contributed to our release:

* Bo Chen <chen.bo@intel.com>
* Fabiano Fidêncio <fabiano.fidencio@intel.com>
* Henry Wang <Henry.Wang@arm.com>
* Jianyong Wu <jianyong.wu@arm.com>
* LiHui <andrewli@kubesphere.io>
* Michael Zhao <michael.zhao@arm.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>
* William Douglas <william.douglas@intel.com>
* Yi Wang <wang.yi59@zte.com.cn>

# v22.1

This is a bug fix release. The following issues have been addressed:

* VFIO ioctl reordering to fix MSI on AMD platforms (#3827)
* Fix `virtio-net` control queue (#3829)

# v22.0

### GDB Debug Stub Support

Cloud Hypervisor can now be used as debug target with GDB. This is controlled
by the `gdb` compile time feature and details of how to use it can be found in
the [gdb
documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/gdb.md).

### `virtio-iommu` Backed Segments

In order to facilitate hotplug devices that require being behind an IOMMU (e.g.
QAT) there is a new option `--platform iommu_segments=<list_of_segments>` that
will place all the specified segments behind the IOMMU.

### Before Boot Configuration Changes

It is now possible to change the VM configuration (e.g. add or remove devices,
resize) before the VM is booted.

### `virtio-balloon` Free Page Reporting

If `--balloon free_page_reporting=on` is used then the guest can report pages
that is it not using to the VMM. The VMM will then notify the host OS that
those pages are no longer in use and can be freed. This can result in improved
memory density.

### Support for Direct Kernel Booting with TDX

Through the use of `TD-Shim` lightweight firmware it is now possible to
directly boot into the kernel with TDX. The [TDX
documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/main/docs/intel_tdx.md#tdshim)
has been updated for this usage.

### PMU Support for AArch64

A PMU is now available on AArch64 for guest performance profiling. This will be
exposed automatically if available from the host.

### Documentation Under CC-BY-4.0 License

The documentation is now licensed under the "Creative Commons Attribution 4.0
International" license which is aligned with the project charter under the
Linux Foundation.

### Deprecation of "Classic" `virtiofsd`

The use of the Rust based [virtiofsd](https://gitlab.com/virtio-fs/virtiofsd)
is now recommended and we are no longer testing against the C based "classic"
version.

### Notable Bug Fixes

* Can now be used on kernels without `AF_INET` support (#3785)
* `virtio-balloon` size is now validated against guest RAM size (#3689)
* Ensure that I/O related KVM VM Exits are correctly handled (#3677)
* Multiple TAP file descriptors can be used for `virtio-net` device hotplug (#3607)
* Minor API improvements and fixes (#3756, #3766, #3647, #3578)
* Fix sporadic seccomp violation from glibc memory freeing (#3610, #3609)
* Fix Windows 11 on AArch64 due to wider MSI-X register accesses (#3714, #3720)
* Ensure `vhost-user` features are correct across migration (#3737)
* Improved vCPU topology on AArch64 (#3735, #3733)

### Contributors

Many thanks to everyone who has contributed to our release:

* Akira Moroo <retrage01@gmail.com>
* Barret Rhoden <brho@google.com>
* Bo Chen <chen.bo@intel.com>
* Fabiano Fidêncio <fabiano.fidencio@intel.com>
* Feng Ye <yefeng@smartx.com>
* Henry Wang <Henry.Wang@arm.com>
* Jianyong Wu <jianyong.wu@arm.com>
* lizhaoxin1 <Lxiaoyouling@163.com>
* Michael Zhao <michael.zhao@arm.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>

# v21.0

### Efficient Local Live Migration (for Live Upgrade)

In order to support fast live upgrade of the VMM an optimised path has been
added in which the memory for the VM is not compared from source to
destination. This is activated by passing `--local` to the `ch-remote
send-migration` command. This means that the live upgrade can complete in the
order of 50ms vs 3s. (#3566)

### Recommended Kernel is Now 5.15

Due to an issue in the `virtio-net` code in 5.14 the recommended Linux kernel
is now 5.15. (#3530)

### Notable Bug fixes

* Multiple fixes were made to the OpenAPI YAML file to match the implementation (#3555,#3562)
* Avoid live migration deadlock when triggered during the kernel boot (#3585)
* Support live migration within firmware (#3586)
* Validate the `virtio-net` descriptor chain (#3548)
* `direct=on` (`O_DIRECT`) can now be used with a guest that makes unaligned accesses (e.g. firmware) (#3587)

### Contributors

Many thanks to everyone who has contributed to our release:

* Anatol Belski <anbelski@linux.microsoft.com>
* Barret Rhoden <brho@google.com>
* Bo Chen <chen.bo@intel.com>
* Fabiano Fidêncio <fabiano.fidencio@intel.com>
* Henry Wang <Henry.Wang@arm.com>
* Liang Zhou <zhoul110@chinatelecom.cn>
* Michael Zhao <michael.zhao@arm.com>
* Muhammad Falak R Wani <falakreyaz@gmail.com>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>
* Ziye Yang <ziye.yang@intel.com>

# v20.2

This is a bug fix release. The following issues have been addressed:

* Don't error out when setting up the SIGWINCH handler (for console resize)
  when this fails due to older kernel (#3456)
* Seccomp rules were refined to remove syscalls that are now unused
* Fix reboot on older host kernels when SIGWINCH handler was not initialised
  (#3496)
* Fix virtio-vsock blocking issue (#3497)

# v20.1

This is a bug fix release. The following issues have been addressed:

* Networking performance regression with `virtio-net` (#3450)
* Limit file descriptors sent in `vfio-user` support (#3401)
* Fully advertise PCI MMIO config regions in ACPI tables (#3432)
* Set the TSS and KVM identity maps so they don't overlap with firmware RAM
* Correctly update the `DeviceTree` on restore

# v20.0

### Multiple PCI segments support

Cloud Hypervisor is no longer limited to 31 PCI devices. For both `x86_64` and
`aarch64` architectures, it is now possible to create up to 16 PCI segments,
increasing the total amount of supported PCI devices to 496.

### CPU pinning

For each vCPU, the user can define a limited set of host CPUs on which it is
allowed to run. This can be useful when assigning a 1:1 mapping between host and
guest resources, or when running a VM on a specific NUMA node.

### Improved VFIO support

Based on VFIO region capabilities, all regions can be memory mapped, limiting
the amount of triggered VM exits, and therefore increasing the performance of
the passthrough device.

### Safer code

Several sections containing unsafe Rust code have been replaced with safe
alternatives, and multiple comments have been added to clarify why the remaining
unsafe sections are safe to use.

### Extended documentation

The documentation related to VFIO has been updated while some new documents have
been introduced to cover the usage of `--cpus` parameter as well as how to run
Cloud Hypervisor on Intel TDX.

### Notable bug fixes

* Naturally align PCI BARs on relocation (#3244)
* Fix panic in SIGWINCH listener thread when no seccomp filter set (#3338)
* Use the tty raw mode implementation from libc (#3344)
* Fix the emulation of register D for CMOS/RTC device (#3393)

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Bo Chen <chen.bo@intel.com>
* Fabiano Fidêncio <fabiano.fidencio@intel.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>
* Willen Yang <willenyang@gmail.com>
* William Douglas <william.douglas@intel.com>
* Ziye Yang <ziye.yang@intel.com>

# v19.0

### Improved PTY handling for serial and `virtio-console`

The PTY support for serial has been enhanced with improved buffering when the
the PTY is not yet connected to. Using `virtio-console` with PTY now results in
the console being resized if the PTY window is also resized.

### PCI boot time optimisations

Multiple optimisations have been made to the PCI handling resulting in
significant improvements in the boot time of the guest.

### Improved TDX support

When using the latest TDVF firmware the ACPI tables created by the VMM are now
exposed via the firmware to the guest.

### Live migration enhancements

Live migration support has been enhanced to support migration with `virtio-mem`
based memory hotplug and the `virtio-balloon` device now supports live
migration.

### `virtio-mem` support with `vfio-user`

The use of `vfio-user` userspaces devices can now be used in conjunction with
`virtio-mem` based memory hotplug and unplug.

### AArch64 for `virtio-iommu`

A paravirtualised IOMMU can now be used on the AArch64 platform.

### Notable bug fixes

* ACPI hotplugged memory is correctly restored after a live migration or
  snapshot/restore (#3165)
* Multiple devices from the same IOMMU group can be passed through via VFIO
  (#3078 #3113)
* Live migration with large blocks of memory was buggy due to an in issue in
  the underlying crate (#3157)

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Bo Chen <chen.bo@intel.com>
* Henry Wang <Henry.Wang@arm.com>
* Hui Zhu <teawater@antfin.com>
* Jianyong Wu <jianyong.wu@arm.com>
* Li Yu <liyu.yukiteru@bytedance.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>
* William Douglas <william.douglas@intel.com>
* Yu Li <liyu.yukiteru@bytedance.com>

# v18.0

### Experimental User Device (`vfio-user`) support

Experimental support for running PCI devices in userspace via `vfio-user`
has been included. This allows the use of the SPDK NVMe `vfio-user` controller
with Cloud Hypervisor. This is enabled by `--user-device` on the command line.

### Migration support for `vhost-user` devices

Devices exposed into the VM via `vhost-user` can now be migrated using the live
migration support. This requires support from the backend however the commonly
used DPDK `vhost-user` backend does support this.

### VHDX disk image support

Images using the VHDX disk image format can now be used with Cloud Hypervisor.

### Device pass through on MSHV hypervisor

When running on the MSHV hypervisor it is possible to pass through devices from
the host through to the guest (e.g with `--device`)

### AArch64 for support `virtio-mem`

The reference Linux kernel we recommend for using with Cloud Hypervisor now supports `virtio-mem` on AArch64.

### Live migration on MSHV hypervisor

Live migration is now supported when running on the MSHV hypervisor including
efficient tracking of dirty pages.

### AArch64 CPU topology support

The CPU topology (as configured through `--cpu topology=`) can now be
configured on AArch64 platforms and is conveyed through either ACPI or device
tree.

### Power button support on AArch64

Use of the ACPI power button (e.g `ch-remote --api-socket=<API socket> power-button`)
is now supported when running on AArch64.

### Notable bug fixes

* Using two PTY outputs e.g. `--serial pty --console pty` now works correctly (#3012)
* TTY input is now always sent to the correct destination (#3005)
* The boot is no longer blocked when using a unattached PTY on the serial console (#3004)
* Live migration is now supported on AArch64 (#3049)
* Ensure signal handlers are run on the correct thread (#3069)

### Contributors

Many thanks to everyone who has contributed to our release:

* Alyssa Ross <hi@alyssa.is>
* Anatol Belski <anbelski@linux.microsoft.com>
* Arafatms <arafatms@outlook.com>
* Bo Chen <chen.bo@intel.com>
* Fazla Mehrab <akm.fazla.mehrab@vt.edu>
* Henry Wang <Henry.Wang@arm.com>
* Jianyong Wu <jianyong.wu@arm.com>
* Jiaqi Gao <jiaqi.gao@intel.com>
* Markus Theil <markus.theil@tu-ilmenau.de>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>
* Yu Li <liyu.yukiteru@bytedance.com>

# v17.0

### ARM64 NUMA support using ACPI

The support for ACPI on ARM64 has been enhanced to include support for
specifying a NUMA configuration using the existing control options.

### `Seccomp` support for MSHV backend

The `seccomp` rules have now been extended to support running against the MSHV
hypervisor backend.

### Hotplug of `macvtap` devices

Hotplug of `macvtap` devices is now supported with the file descriptor for the
network device if opened by the user and passed to the VMM. The `ch-remote`
tool supports this functionality when adding a network device.

### Improved SGX support

The SGX support has been updated to match the latest Linux kernel support and
now supports SGX provisioning and associating EPC sections to NUMA nodes.

### Inflight tracking for `vhost-user` devices

Support for handling inflight tracking of I/O requests has been added to the
`vhost-user` devices allowing recovery after device reconnection.

### Notable bug fixes

* VFIO PCI BAR calculation code now correctly handles I/O BARs (#2821).
* The VMM side of `vhost-user` devices no longer advertise the
  `VIRTIO_F_RING_PACKED` feature as they are not yet supported in the VMM
(#2833).
* On ARM64 VMs can be created with more than 16 vCPUs (#2763).

### Contributors

Many thanks to everyone who has contributed to our release:

* Anatol Belski <anbelski@linux.microsoft.com>
* Arafatms <arafatms@outlook.com>
* Bo Chen <chen.bo@intel.com>
* Fei Li <lifei.shirley@bytedance.com>
* Henry Wang <Henry.Wang@arm.com>
* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
* Jianyong Wu <jianyong.wu@arm.com>
* Li Hangjing <lihangjing@bytedance.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>
* Yukiteru <wfly1998@sina.com>

# v16.0

### Improved live migration support

The live migration support inside Cloud Hypervisor has been improved with the addition of the tracking of dirty pages written by the VMM to complement the tracking of dirty pages made by the guest itself. Further the internal state of the VMM now is versioned which allows the safe migration of VMs from one version of the VMM to a newer one. However further testing is required so this should be done with care. See the [live migration documentation](docs/live_migration.md) for more details.

### Improved `vhost-user` support

When using `vhost-user` to access devices implemented in different processes there is now support for reconnection of those devices in the case of a restart of the backend. In addition it is now possible to operate with the direction of the `vhost-user-net` connection reversed with the server in the VMM and the client in the backend. This is aligns with the default approach recommended by Open vSwitch.

### ARM64 ACPI and UEFI support

Cloud Hypervisor now supports using ACPI and booting from a UEFI image on ARM64. This allows the use of stock OS images without direct kernel boot.

### Notable bug fixes

* Activating fewer `virtio-net` queues than advertised is now supported. This appeared when using OVMF with an MQ enabled device (#2578).
* When using MQ with `virtio` devices Cloud Hypervisor now enforces a minimum vCPU count which ensures that the user will not see adverse guest performance (#2563).
* The KVM clock is now correctly handled during live migration / snapshot & restore.

### Removed functionality

The following formerly deprecated features have been removed:

* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage`
  binaries has been deprecated. When using direct boot users should configure
  their kernel with `CONFIG_PVH=y`.

### Contributors

Many thanks to everyone who has contributed to our release including some new faces.

* Anatol Belski <anbelski@linux.microsoft.com>
* Bo Chen <chen.bo@intel.com>
* Dayu Liu <liu.dayu@zte.com.cn>
* Henry Wang <Henry.Wang@arm.com>
* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
* Jianyong Wu <jianyong.wu@arm.com>
* Michael Zhao <michael.zhao@arm.com>
* Mikko Ylinen <mikko.ylinen@intel.com>
* Muminul Islam <muislam@microsoft.com>
* Ren Lei <ren.lei4@zte.com.cn>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>
* Yi Wang <wang.yi59@zte.com.cn>

# v15.0

Highlights for `cloud-hypervisor` version v15.0 include:

### Version numbering and stability guarantees

This release is the first in a new version numbering scheme to represent that
we believe Cloud Hypervisor is maturing and entering a period of stability.
With this new release we are beginning our new stability guarantees:

* The API (including command line options) will not be removed or changed in a
  breaking way without a minimum of 2 releases notice. Where possible warnings
  will be given about the use of deprecated functionality and the deprecations
  will be documented in the release notes.
* Point releases will be made between individual releases where there are
  substantial bug fixes or security issues that need to be fixed.

Currently the following items are **not** guaranteed across updates:

* Snapshot/restore is not supported across different versions
* Live migration is not supported across different versions
* The following features are considered experimental and may change
  substantially between releases: TDX, SGX.

### Network device rate limiting

Building on our existing support for rate limiting block activity the network
device also now supports rate limiting. Full details of the controls are in the
[IO throttling documentation.](docs/io_throttling.md)

### Support for runtime control of `virtio-net` guest offload

The guest is now able to change the offload settings for the `virtio-net`
device. As well as providing a useful control this mitigates an issue in the
Linux kernel where the guest will attempt to reprogram the offload settings
even if they are not advertised as configurable (#2528).

### `--api-socket` supports file descriptor parameter

The `--api-socket` can now take an `fd=` parameter to specify an existing file
descriptor to use. This is particularly beneficial for frameworks that need to
programmatically control Cloud Hypervisor.

### Bug fixes

* A workaround has been put in place to mitigate a Linux kernel issues that
  results in the CPU thread spinning at 100% when using `virtio-pmem` (#2277).
* PCI BARs are now correctly aligned removing the need for the guest to
  reprogram them (#1797,#1798)
* Handle TAP interface not being writable within virtio-net (due to the buffer
  exhaustion on the host) (#2517)
* The recommended Linux kernel is now v5.12.0 as it contains a fix that
  prevents snapshot & restore working (#2535)

### Deprecations

Deprecated features will be removed in a subsequent release and users should plan to use alternatives

* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage`
  binaries has been deprecated. When using direct boot users should configure
  their kernel with `CONFIG_PVH=y`. Will be removed in v16.0.

### Contributors

Many thanks to everyone who has contributed to our release including some new faces.

* Alyssa Ross <hi@alyssa.is>
* Anatol Belski <anbelski@linux.microsoft.com>
* Bo Chen <chen.bo@intel.com>
* Gaelan Steele <gbs@canishe.com>
* Jianyong Wu <jianyong.wu@arm.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>
* William Douglas <william.douglas@intel.com>

# v0.14.1

Bug fix release branched off the v0.14.0 release. The following bugs were fixed
in this release:

* CPU hotplug on Windows failed due to misreported CPU state information and
  the lack of HyperV CPUID bit enabled (#2437, #2449, #2436)
* A seccomp rule was missing that was triggered on CPU unplug (#2455)
* A bounds check in VIRTIO queue validation was erroneously generating
  DescriptorChainTooShort errors in certain circumstances (#2450, #2424)

# v0.14.0

Highlights for `cloud-hypervisor` version 0.14.0 include:

### Structured event monitoring

A new option was added to the VMM `--event-monitor` which reports structured
events (JSON) over a file or file descriptor at key events in the lifecycle of
the VM. The list of events is limited at the moment but will be further
extended over subsequent releases. The events exposed form part of the Cloud
Hypervisor API surface.

### MSHV improvements

Basic support has been added for running Windows guests atop the MSHV
hypervisor as an alternative to KVM and further improvements have been made to
the MSHV support.

### Improved aarch64 platform

The aarch64 platform has been enhanced with more devices exposed to the running
VM including an enhanced serial UART.

### Updated hotplug documentation

The documentation for the hotplug support has been updated to reflect the use
of the `ch-remote` tool and to include details of `virtio-mem` based hotplug as
well as documenting hotplug of paravirtualised and VFIO devices.

### PTY control for serial and `virtio-console`

The `--serial` and `--console` parameters can now direct the console to a PTY
allowing programmatic control of the console from another process through the
PTY subsystem.

### Block device rate limiting

The block device performance can now be constrained as part of the VM
configuration allowing rate limiting. Full details of the controls are in the
[IO throttling documentation.](docs/io_throttling.md)


### Deprecations

Deprecated features will be removed in a subsequent release and users should plan to use alternatives

* Support for booting with the "LinuxBoot" protocol for ELF and `bzImage`
  binaries has been deprecated. When using direct boot users should configure
  their kernel with `CONFIG_PVH=y`.


### Contributors

Many thanks to everyone who has contributed to our 0.14.0 release including
some new faces.

* Bo Chen <chen.bo@intel.com>
* Henry Wang <Henry.Wang@arm.com>
* Iggy Jackson <iggy@theiggy.com>
* Jiachen Zhang <zhangjiachen.jaycee@bytedance.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Penny Zheng <Penny.Zheng@arm.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Vineeth Pillai <viremana@linux.microsoft.com>
* Wei Liu <liuwe@microsoft.com>
* William Douglas <william.r.douglas@gmail.com>
* Zide Chen <zide.chen@intel.com>

# v0.13.0

Highlights for `cloud-hypervisor` version 0.13.0 include:

### Wider VFIO device support

It is now possible to use Cloud Hypervisor's VFIO support to passthrough PCI
devices that do not support MSI or MSI-X and instead rely on INTx interrupts.
Most notably this widens the support to most NVIDIA cards with the proprietary
drivers.

### Improved huge page support

Through the addition of `hugepage_size` on `--memory` it is now possible to
specify the desired size of the huge pages used when allocating the guest
memory. The user is required to ensure they have sufficient pages of the
desired size in their pool.

### MACvTAP support

It is now possible to provide file descriptors using the `fd` parameter to
`--net` which point at TAP devices that have already been opened by the user.
This aids integration with `libvirt` but also permits the use of MACvTAP
support. This is documented in dedicated [macvtap documentation](docs/macvtap-bridge.md).

### VHD disk image support

It is now possible to use VHD (fixed) disk images as well as QCOWv2 and raw
disk image with Cloud Hypervisor.

### Improved Virtio device threading

Device threads are now derived from the main VMM thread which allows more
restrictive seccomp filters to be applied to them. The threads also have a
predictable name derived from the device id.

### Clean shutdown support via synthetic power button

It is now possible to request that the guest VM shut itself down by triggering
a synthetic ACPI power button press from the VMM. If the guest is listening for
such an event (e.g. using systemd) then it will process the event and cleanly
shut down. This functionality is exposed through the HTTP API and can be
triggered via `ch-remote --api-socket=<API socket> power-button`.

### Contributors

Many thanks to everyone who has contributed to our 0.13.0 release including
some new faces.

* Bo Chen <chen.bo@intel.com>
* Mikko Ylinen <mikko.ylinen@intel.com>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Samuel Ortiz <sameo@linux.intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Vineeth Pillai <viremana@linux.microsoft.com>
* Wei Liu <liuwe@microsoft.com>
* William Douglas <william.r.douglas@gmail.com>
* Xie Yongji <xieyongji@bytedance.com>

# v0.12.0

Highlights for `cloud-hypervisor` version 0.12.0 include:

### ARM64 enhancements

The use of `--watchdog` is now fully supported as is the ability to reboot the
VM from within the guest when running Cloud Hypervisor on an ARM64 system.

### Removal of `vhost-user-net` and `vhost-user-block` self spawning

In order to use `vhost-user-net` or `vhost-user-block` backends the user is now
responsible for starting the backend and providing the socket for the VMM to
use. This functionality was deprecated in the last release and how now been
removed.

### Migration of `vhost-user-fs` backend

The `vhost-user-fs` backend is no longer included in Cloud Hypervisor and it is
instead hosted in [its own
repository](https://gitlab.com/virtio-fs/virtiofsd-rs)

### Enhanced "info" API

The `vm.info` HTTP API endpoint has been extended to include the details of the
devices used by the VM including any VFIO devices used.

### Contributors

Many thanks to everyone who has contributed to our 0.12.0 release:

* Anatol Belski <anbelski@linux.microsoft.com>
* Julio Montes <julio.montes@intel.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Samuel Ortiz <sameo@linux.intel.com>
* Wei Liu <liuwe@microsoft.com>

# v0.11.0

Highlights for `cloud-hypervisor` version 0.11.0 include:

### `io_uring` support by default for `virtio-block`

Provided that the host OS supports it (Linux kernel 5.8+) then `io_uring` will
be used for a significantly higher performance block device.

### Windows Guest Support

This is the first release where we officially support Windows running as a
guest. Full details of how to setup the image and run Cloud Hypervisor with a
Windows guest can be found in the dedicated [Windows
documentation](docs/windows.md).

### `vhost-user` "Self Spawning" Deprecation

Automatically spawning a `vhost-user-net` or `vhost-user-block` backend is now
deprecated. Users of this functionality will receive a warning and should make
adjustments. The functionality will be removed in the next release.

### `virtio-mmio` Removal

Support for using the `virtio-mmio` transport, rather than using PCI, has been
removed. This has been to simplify the code and significantly
reduce the testing burden of the project.

### Snapshot/Restore support for ARM64

When running on the ARM64 architecture snapshot and restore has now been
implemented.

### Improved Linux Boot Time

The time to boot the Linux kernel has been significantly improved by the
identifying some areas of delays around PCI bus probing, IOAPIC programming and
MPTABLE issues. Full details can be seen in #1728.

### `SIGTERM/SIGINT` Interrupt Signal Handling

When the VMM process receives the `SIGTERM` or `SIGINT` signals then it will
trigger the VMM process to cleanly deallocate resources before exiting. The
guest VM will not be cleanly shutdown but the VMM process will clean up its
resources.

### Default Log Level Changed

The default logging level was changed to include warnings which should make it
easier to see potential issues. New [logging
documentation](docs/logging.md) was also added.

### New `--balloon` Parameter Added

Control of the setup of `virtio-balloon` has been moved from `--memory` to its
own dedicated parameter. This makes it easier to add more balloon specific
controls without overloading `--memory`.

### Experimental `virtio-watchdog` Support

Support for using a new `virtio-watchdog` has been added which can be used to
have the VMM reboot the guest if the guest userspace fails to ping the
watchdog. This is enabled with `--watchdog` and requires kernel support.

### Notable Bug Fixes

* MTRR bit was missing from CPUID advertised to guest
* "Return" key could not be used under `CMD.EXE` under Windows SAC (#1170)
* CPU identification string is now exposed to the guest
* `virtio-pmem` with`discard_writes=on` no longer marks the guest memory as
  read only so avoids excessive VM exits (#1795)
* PCI device hotplug after an unplug was fixed (#1802)
* When using the ACPI method to resize the guest memory the full reserved size
  can be used (#1803)
* Snapshot and restore followed by a second snapshot and restore now works
  correctly
* Snapshot and restore of VMs with more than 2GiB in one region now work
  correctly

### Contributors

Many thanks to everyone who has contributed to our 0.11.0 release including some new faces.

* Anatol Belski <anbelski@linux.microsoft.com>
* Bo Chen <chen.bo@intel.com>
* Daniel Verkamp <dverkamp@chromium.org>
* Henry Wang <Henry.Wang@arm.com>
* Hui Zhu <teawater@antfin.com>
* Jiangbo Wu <jiangbo.wu@intel.com>
* Josh Soref <jsoref@users.noreply.github.com>
* Julio Montes <julio.montes@intel.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* pierwill <19642016+pierwill@users.noreply.github.com>
* Praveen Paladugu <prapal@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>


# v0.10.0

Highlights for `cloud-hypervisor` version 0.10.0 include:

### `virtio-block` Support for Multiple Descriptors

Some `virtio-block` device drivers may generate requests with multiple descriptors and support has been added for those drivers.

### Memory Zones

Support has been added for fine-grained control of memory allocation for the guest. This includes controlling the backing of sections of guest memory, assigning to specific host NUMA nodes and assigning memory and vCPUs to specific memory nodes inside the guest. Full details of this can be found in the [memory documentation](docs/memory.md).

### `Seccomp` Sandbox Improvements

All the remaining threads and devices are now isolated within their own `seccomp` filters. This provides a layer of sandboxing and enhances the security model of `cloud-hypervisor`.

### Preliminary KVM HyperV Emulation Control

A new option (`kvm_hyperv`) has been added to `--cpus` to provide an option to toggle on KVM's HyperV emulation support. This enables progress towards booting Windows without adding extra emulated devices.

### Notable Bug Fixes

- When using `ch-remote` to resize the VM parameter now accepts the standard sizes suffices (#1596)
- `cloud-hypervisor` no longer panics when started with `--memory hotplug_method=virtio-mem` and no `hotplug_size` (#1564)
- After a reboot memory can remove when using `--memory hotplug_method=virtio-mem` (#1593)
- `--version` shows the version for released binaries (#1669)
- Errors generated by worker threads for `virtio` devices are now printed out (#1551)

### Contributors

Many thanks to everyone who has contributed to our 0.10.0 release including some new faces.

* Alyssa Ross <hi@alyssa.is>
* Amey Narkhede <ameynarkhede02@gmail.com>
* Anatol Belski <ab@php.net>
* Bo Chen <chen.bo@intel.com>
* Hui Zhu <teawater@antfin.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Samuel Ortiz <sameo@linux.intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Wei Liu <liuwe@microsoft.com>

# v0.9.0

Highlights for `cloud-hypervisor` version 0.9.0 include:

### `io_uring` Based Block Device Support

If the `io_uring` feature is enabled and the host kernel supports it then `io_uring` will be used for block devices. This results a very significant performance improvement.

### Block and Network Device Statistics

Statistics for activity of the `virtio` network and block devices is now exposed through a new `vm.counters` HTTP API entry point. These take the form of simple counters which can be used to observe the activity of the VM.

### HTTP API Responses

The HTTP API for adding devices now responds with the name that was assigned to the device as well the PCI BDF.

### CPU Topology

A `topology` parameter has been added to `--cpus` which allows the configuration of the guest CPU topology allowing the user to specify the numbers of sockets, packages per socket, cores per package and threads per core.

### Release Build Optimization

Our release build is now built with LTO (*Link Time Optimization*) which results in a ~20% reduction in the binary size.

### Hypervisor Abstraction

A new abstraction has been introduced, in the form of a `hypervisor` crate so as to enable the support of additional hypervisors beyond `KVM`.

### Snapshot/Restore Improvements

Multiple improvements have been made to the VM snapshot/restore support that was added in the last release. This includes persisting more vCPU state and in particular preserving the guest paravirtualized clock in order to avoid vCPU hangs inside the guest when running with multiple vCPUs.

### Virtio Memory Ballooning Support

A `virtio-balloon` device has been added, controlled through the `resize` control, which allows the reclamation of host memory by resizing a memory balloon inside the guest.

### Enhancements to ARM64 Support

The ARM64 support introduced in the last release has been further enhanced with support for using PCI for exposing devices into the guest as well as multiple bug fixes. It also now supports using an initramfs when booting.

### Intel SGX Support

The guest can now use Intel SGX if the host supports it. Details can be found in the dedicated [SGX documentation](docs/intel_sgx.md).

### `Seccomp` Sandbox Improvements

The most frequently used virtio devices are now isolated with their own `seccomp` filters. It is also now possible to pass `--seccomp=log` which result in the logging of requests that would have otherwise been denied to further aid development.

### Notable Bug Fixes

* Our `virtio-vsock` implementation has been resynced with the implementation from Firecracker and includes multiple bug fixes.
* CPU hotplug has been fixed so that it is now possible to add, remove, and re-add vCPUs (#1338)
* A workaround is now in place for when KVM reports MSRs available MSRs that are in fact unreadable preventing snapshot/restore from working correctly (#1543).
* `virtio-mmio` based devices are now more widely tested (#275).
* Multiple issues have been fixed with virtio device configuration (#1217)
* Console input was wrongly consumed by both `virtio-console` and the serial. (#1521)

### Contributors

Many thanks to everyone who has contributed to our 0.9.0 release including some new faces.

* Anatol Belski <ab@php.net>
* Bo Chen <chen.bo@intel.com>
* Dr. David Alan Gilbert <dgilbert@redhat.com>
* Henry Wang <Henry.Wang@arm.com>
* Howard Zhang <howard.zhang@arm.com>
* Hui Zhu <teawater@antfin.com>
* Jianyong Wu <jianyong.wu@arm.com>
* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
* LiYa'nan <oliverliyn@gmail.com>
* Michael Zhao <michael.zhao@arm.com>
* Muminul Islam <muislam@microsoft.com>
* Praveen Paladugu <prapal@microsoft.com>
* Ricardo Koller <ricarkol@gmail.com>
* Rob Bradford <robert.bradford@intel.com>
* Samuel Ortiz <sameo@linux.intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Stefano Garzarella <sgarzare@redhat.com>
* Wei Liu <liuwe@microsoft.com>


# v0.8.0

Highlights for `cloud-hypervisor` version 0.8.0 include:

### Experimental Snapshot and Restore Support

This release includes the first version of the snapshot and restore feature.
This allows a VM to be paused and then subsequently snapshotted. At a later
point that snapshot may be restored into a new running VM identical to the
original VM at the point it was paused.

This feature can be used for offline migration from one VM host to another, to
allow the upgrading or rebooting of the host machine transparently to the guest
or for templating the VM. This is an experimental feature and cannot be used on
a VM using passthrough (VFIO) devices. Issues with SMP have also been observed
(#1176).

### Experimental ARM64 Support

Included in this release is experimental support for running on ARM64.
Currently only `virtio-mmio` devices and a serial port are supported. Full
details can be found in the [ARM64 documentation](https://github.com/cloud-hypervisor/cloud-hypervisor/blob/8ab15b9a984a448006f97b1211498c1bb583de3b/docs/arm64.md).

### Support for Using 5-level Paging in Guests

If the host supports it the guest is now enabled for 5-level paging (aka LA57).
This works when booting the Linux kernel with a vmlinux, bzImage or firmware
based boot. However booting an ELF kernel built with `CONFIG_PVH=y` does not
work due to current limitations in the PVH boot process.

### Virtio Device Interrupt Suppression for Network Devices

With `virtio-net` and `vhost-user-net` devices the guest can suppress
interrupts from the VMM by using the `VIRTIO_RING_F_EVENT_IDX` feature. This
can lead to an improvement in performance by reducing the number of interrupts
the guest must service.

### `vhost_user_fs` Improvements

The implementation in Cloud Hypervisor of the VirtioFS server now supports sandboxing itself with `seccomp`.


### Notable Bug Fixes

* VMs that have not yet been booted can now be deleted (#1110).
* By creating the `tap` device ahead of creating the VM it is not required to
  run the `cloud-hypervisor` binary with `CAP_NET_ADMIN` (#1273).
* Block I/O via `virtio-block` or `vhost-user-block` now correctly adheres to
  the specification and synchronizes to the underlying filesystem as required
  based on guest feature negotiation. This avoids potential data loss (#399,
  #1216).
* When booting with a large number of vCPUs then the ACPI table would be
  overwritten by the SMP `MPTABLE`. When compiled with the `acpi` feature the
  `MPTABLE` will no longer be generated (#1132).
* Shutting down VMs that have been paused is now supported (#816).
* Created socket files are deleted on shutdown (#1083).
* Trying to use passthrough devices (VFIO) will be rejected on `mmio` builds
  (#751).

### Command Line and API Changes

This is non exhaustive list of HTTP API and command line changes:

* All user visible socket parameters are now consistently called `socket`
  rather than `sock` in some cases.
* The `ch-remote` tool now shows any error message generated by the VMM
* The `wce` parameter has been removed from `--disk` as the feature is always
  offered for negotiation.
* `--net` has gained a `host_mac` option that allows the setting of the MAC
  address for the `tap` device on the host.

### Contributors

Many thanks to everyone who has contributed to our 0.8.0 release including some new faces.

* Anatol Belski <ab@php.net>
* Arron Wang <arron.wang@intel.com>
* Bo Chen <chen.bo@intel.com>
* Dr. David Alan Gilbert <dgilbert@redhat.com>
* Henry Wang <Henry.Wang@arm.com>
* Hui Zhu <teawater@antfin.com>
* LiYa'nan <oliverliyn@gmail.com>
* Michael Zhao <michael.zhao@arm.com>
* Rob Bradford <robert.bradford@intel.com>
* Samuel Ortiz <sameo@linux.intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Sergio Lopez <slp@redhat.com>

# v0.7.0

Highlights for `cloud-hypervisor` version 0.7.0 include:

### Block, Network, Persistent Memory (PMEM), VirtioFS and Vsock hotplug

Further to our effort to support modifying a running guest we now support
hotplug and unplug of the following virtio backed devices: block, network,
pmem, virtio-fs and vsock. This functionality is available on the (default) PCI
based transport and is exposed through the HTTP API. The `ch-remote` utility
provides a CLI for adding or removing these device types after the VM has
booted. User can use the `id` parameter on the devices to choose names for
devices to ease their removal.

### Alternative `libc` Support

Cloud Hypervisor can now be compiled with the `musl` C library and this release
contains a static binary compiled using that toolchain.

### Multithreaded Multi Queued `vhost-user` Backends

The `vhost-user` backends for network and block support that are shipped by
Cloud Hypervisor have been enhanced to support multiple threads and queues to
improve throughput. These backends are used automatically if `vhost_user=true`
is passed when the devices are created.

### Initial RamFS Support

By passing the `--initramfs` command line option the user can specify a file to
be loaded into the guest memory to be used as the kernel initial filesystem.
This is usually used to allow the loading of drivers needed to be able to
access the real root filesystem but it can also be used standalone for a very
minimal image.

### Alternative Memory Hotplug: `virtio-mem`

As well as supporting ACPI based hotplug Cloud Hypervisor now supports using
the `virtio-mem` hotplug alternative. This can be controlled by the
`hotplug_method` parameter on the `--memory` command line option. It currently
requires kernel patches to be able to support it.

### `Seccomp` Sandboxing

Cloud Hypervisor now has support for restricting the system calls that the
process can use via the `seccomp` security API. This on by default and is
controlled by the `--seccomp` command line option.

### Updated Distribution Support

With the release of Ubuntu 20.04 we have added that to the list of supported
distributions and is part of our regular testing programme.

### Command Line and API Changes

This is non exhaustive list of HTTP API and command line changes

* New `id` fields added for devices to allow them to be named to ease removal.
  If no name is specified the VMM chooses one.
* Use `--memory`'s `shared` and `hugepages` controls for determining backing
  memory instead of providing a path.
* The `--vsock` parameter only takes one device as the Linux kernel only
  supports a single Vsock device. The REST API has removed the vector for this
  option and replaced it with a single optional field.
* There is enhanced validation of the command line and API provided
  configurations to ensure that the provided options are compatible e.g. that
  shared memory is in use if any attempt is made to used a `vhost-user` backed
  device.
* `ch-remote` has added `add-disk`, `add-fs`, `add-net`, `add-pmem` and
  `add-vsock` subcommands. For removal `remove-device` is used. The REST API
  has appropriate new HTTP endpoints too.
* Specifying a `size` with `--pmem` is no longer required and instead the size
  will be obtained from the file. A `discard_writes` option has also been added
  to provide the equivalent of a read-only file.
* The parameters to `--block-backend` have been changed to more closely align
  with those used by `--disk`.

### Contributors

Many thanks to everyone who has contributed to our 0.7.0 release including some new faces.

* Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
* Bo Chen <chen.bo@intel.com>
* Cathy Zhang <cathy.zhang@intel.com>
* Damjan Georgievski <gdamjan@gmail.com>
* Dean Sheather <dean@coder.com>
* Eryu Guan <eguan@linux.alibaba.com>
* Hui Zhu <teawater@antfin.com>
* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
* Martin Xu <martin.xu@intel.com>
* Muminul Islam <muislam@microsoft.com>
* Rob Bradford <robert.bradford@intel.com>
* Samuel Ortiz <sameo@linux.intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Sergio Lopez <slp@redhat.com>
* Yang Zhong <yang.zhong@intel.com>
* Yi Sun <yi.y.sun@linux.intel.com>

# v0.6.0

Highlights for `cloud-hypervisor` version 0.6.0 include:

### Directly Assigned Devices Hotplug

We continued our efforts around supporting dynamically changing the guest
resources. After adding support for CPU and memory hotplug, Cloud Hypervisor
now supports hot plugging and hot unplugging directly assigned (a.k.a. `VFIO`)
devices into an already running guest. This closes the features gap for
providing a complete Kata Containers workloads support with Cloud Hypervisor.

### Shared Filesystem Improvements

We enhanced our shared filesystem support through many `virtio-fs` improvements.
By adding support for DAX, parallel processing of multiple requests, `FS_IO`,
`LSEEK` and the `MMIO` virtio transport layer to our `vhost_user_fs` daemon, we
improved our filesystem sharing performance, but also made it more stable and
compatible with other `virtio-fs` implementations.

### Block and Networking IO Self Offloading

When choosing to offload the paravirtualized block and networking I/O to an
external process (through the `vhost-user` protocol), Cloud Hypervisor now
automatically spawns its default `vhost-user-blk` and `vhost-user-net` backends
into their own, separate processes.
This provides a seamless paravirtualized I/O user experience for those who want
to run their guest I/O into separate executions contexts.

### Command Line Interface

More and more Cloud Hypervisor services are exposed through the
[Rest API](vmm/src/api/openapi/cloud-hypervisor.yaml) and thus only
accessible via relatively cumbersome HTTP calls. In order to abstract
those calls into a more user friendly tool, we created a Cloud Hypervisor
Command Line Interface (CLI) called `ch-remote`.  The `ch-remote` binary
is created with each build and available e.g. at
`cloud-hypervisor/target/debug/ch-remote` when doing a debug build.

Please check `ch-remote --help` for a complete description of all available
commands.

### PVH Boot

In addition to the traditional Linux boot protocol, Cloud Hypervisor now
supports direct kernel booting through the [PVH ABI](https://xenbits.xen.org/docs/unstable/misc/pvh.html).

### Contributors

With the 0.6.0 release, we are welcoming a few new contributors. Many thanks
to them and to everyone that contributed to this release:

* Alejandro Jimenez <alejandro.j.jimenez@oracle.com>
* Arron Wang <arron.wang@intel.com>
* Bin Liu <liubin0329@gmail.com>
* Bo Chen <chen.bo@intel.com>
* Cathy Zhang <cathy.zhang@intel.com>
* Eryu Guan <eguan@linux.alibaba.com>
* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
* Liu Bo <bo.liu@linux.alibaba.com>
* Qiu Wenbo <qiuwenbo@phytium.com.cn>
* Rob Bradford <robert.bradford@intel.com>
* Samuel Ortiz <sameo@linux.intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Sergio Lopez <slp@redhat.com>

# v0.5.1

This is a bugfix release branched off v0.5.0. It contains the following fixes:

* Update DiskConfig to contain missing disk control features (#790) - Samuel Ortiz and Sergio Lopez
* Prevent memory overcommit via virtio-fs (#763) - Sebastien Boeuf
* Fixed error reporting for resize command - Samuel Ortiz
* Double reboot workaround (#783) - Rob Bradford
* Various CI and development tooling fixes - Sebastien Boeuf, Samuel Ortiz, Rob Bradford

# v0.5.0

Highlights for `cloud-hypervisor` version 0.5.0 include:

### Virtual Machine Dynamic Resizing

With 0.4.0 we added support for CPU hot plug, and 0.5.0 adds CPU hot unplug and
memory hot plug as well. This allows to dynamically resize Cloud Hypervisor
guests which is needed for e.g. Kubernetes related use cases.
The memory hot plug implementation is based on the same framework as the CPU hot
plug/unplug one, i.e. hardware-reduced ACPI notifications to the guest.

Next on our VM resizing roadmap is the PCI devices hotplug feature.

### Multi-Queue, Multi-Threaded Paravirtualization

We enhanced our virtio networking and block support by having both devices use
multiple I/O queues handled by multiple threads. This improves our default
paravirtualized networking and block devices throughput.

### New Interrupt Management Framework

We improved our interrupt management implementation by introducing an Interrupt
Manager framework, based on the currently on-going [rust-vmm vm-device](https://github.com/rust-vmm/vm-device)
crates discussions. This move made the code significantly cleaner, and allowed
us to remove several KVM related dependencies from crates like the PCI and
virtio ones.

### Development Tools

In order to provide a better developer experience, we worked on improving our
build, development and testing tools.
Somehow similar to the excellent
[Firecracker's devtool](https://github.com/firecracker-microvm/firecracker/blob/master/tools/devtool),
we now provide a [dev_cli script](scripts/dev_cli.sh).

With this new tool, our users and contributors will be able to build and test
Cloud Hypervisor through a containerized environment.

### Kata Containers Integration

We spent some significant time and efforts debugging and fixing our integration
with the [Kata Containers](https://github.com/kata-containers) project. Cloud
Hypervisor is now a fully supported Kata Containers hypervisor, and is
integrated into the project's CI.

### Contributors

Many thanks to everyone that contributed to the 0.5.0 release:

* Bo Chen <chen.bo@intel.com>
* Cathy Zhang <cathy.zhang@intel.com>
* Qiu Wenbo <qiuwenbo@phytium.com.cn>
* Rob Bradford <robert.bradford@intel.com>
* Samuel Ortiz <sameo@linux.intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Sergio Lopez <slp@redhat.com>
* Yang Zhong <yang.zhong@intel.com>

# v0.4.0

Highlights for `cloud-hypervisor` version 0.4.0 include:

### Dynamic virtual CPUs addition

As a way to vertically scale Cloud Hypervisor guests, we now support dynamically
adding virtual CPUs to the guests, a mechanism also known as CPU hot plug.
Through hardware-reduced ACPI notifications, Cloud Hypervisor can now add CPUs
to an already running guest and the high level operations for that process are
documented [here](docs/hotplug.md)

During the next release cycles we are planning to extend Cloud Hypervisor
hot plug framework to other resources, namely PCI devices and memory.

### Programmatic firmware tables generation

As part of the CPU hot plug feature enablement, and as a requirement for hot
plugging other resources like devices or RAM, we added support for
programmatically generating the needed ACPI tables. Through a dedicated
`acpi-tables` crate, we now have a flexible and clean way of generating those
tables based on the VMM device model and topology.

### Filesystem and block devices vhost-user backends

Our objective of running all Cloud Hypervisor paravirtualized I/O to a
vhost-user based framework is getting closer as we've added Rust based
implementations for vhost-user-blk and virtiofs backends. Together with the
vhost-user-net backend that came with the 0.3.0 release, this will form the
default Cloud Hypervisor I/O architecture.

### Guest pause and resume

As an initial requirement for enabling live migration, we added support for
pausing and resuming any VMM components. As an intermediate step towards live
migration, the upcoming guest snapshotting feature will be based on the pause
and resume capabilities.

### Userspace IOAPIC by default

As a way to simplify our device manager implementation, but also in order to
stay away from privileged rings as often as possible, any device that relies on
pin based interrupts will be using the userspace IOAPIC implementation by
default.

### PCI BAR reprogramming

In order to allow for a more flexible device model, and also support guests
that would want to move PCI devices, we added support for PCI devices BAR
reprogramming.

### New `cloud-hypervisor` organization

As we wanted to be more flexible on how we manage the Cloud Hypervisor project,
we decided to move it under a [dedicated GitHub organization](https://github.com/cloud-hypervisor).
Together with the [cloud-hypervisor](https://github.com/cloud-hypervisor/cloud-hypervisor)
project, this new organization also now hosts our [kernel](https://github.com/cloud-hypervisor/linux)
and [firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware)
repositories. We may also use it to host any rust-vmm that we'd need to
temporarily fork.
Thanks to GitHub's seamless repository redirections, the move is completely
transparent to all Cloud Hypervisor contributors, users and followers.

### Contributors

Many thanks to everyone that contributed to the 0.4.0 release:

* Cathy Zhang <cathy.zhang@intel.com>
* Emin Ghuliev <drmint80@gmail.com>
* Jose Carlos Venegas Munoz <jose.carlos.venegas.munoz@intel.com>
* Qiu Wenbo <qiuwenbo@phytium.com.cn>
* Rob Bradford <robert.bradford@intel.com>
* Samuel Ortiz <sameo@linux.intel.com>
* Sebastien Boeuf <sebastien.boeuf@intel.com>
* Sergio Lopez <slp@redhat.com>
* Wu Zongyong <wuzongyong@linux.alibaba.com>

# v0.3.0

Highlights for `cloud-hypervisor` version 0.3.0 include:

### Block device offloading

We continue to work on offloading paravirtualized I/O to external processes,
and we added support for
[vhost-user-blk](https://access.redhat.com/solutions/3394851) backends.
This enables `cloud-hypervisor` users to plug a `vhost-user` based block device
like [SPDK](https://spdk.io)) into the VMM as their paravirtualized storage
backend.

### Network device backend

The previous release provided support for
[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. Now we
also provide a TAP based vhost-user-net backend, implemented in Rust. Together
with the vhost-user-net device implementation, this will eventually become the
Cloud Hypervisor default paravirtualized networking architecture.

### Virtual sockets

In order to more efficiently and securely communicate between host and guest,
we added an hybrid implementation of the
[VSOCK](http://man7.org/linux/man-pages/man7/vsock.7.html) socket address
family over virtio. Credits go to the
[Firecracker](https://github.com/firecracker-microvm/firecracker/blob/master/docs/vsock.md)
project as our implementation is a copy of theirs.

### HTTP based API

In anticipation of the need to support asynchronous operations to Cloud
Hypervisor guests (e.g. resources hotplug and guest migration), we added a HTTP
based API to the VMM. The API will be more extensively documented during the
next release cycle.

### Memory mapped virtio transport

In order to support potential PCI-free use cases, we added support for the
[virtio MMIO](https://docs.oasis-open.org/virtio/virtio/v1.1/cs01/virtio-v1.1-cs01.html#x1-1440002)
transport layer. This will allow us to support simple, minimal guest
configurations that do not require a PCI bus emulation.

### Paravirtualized IOMMU

As we want to improve our nested guests support, we added support for exposing
a [paravirtualized IOMMU](docs/iommu.md) device through virtio. This allows
for a safer nested virtio and directly assigned devices support.

To add the IOMMU support, we had to make some CLI changes for Cloud Hypervisor
users to be able to specify if devices had to be handled through this virtual
IOMMU or not. In particular, the `--disk` option now expects disk paths to be
prefixed with a `path=` string, and supports an optional `iommu=[on|off]`
setting.

### Ubuntu 19.10

With the latest [hypervisor firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware),
we can now support the latest
[Ubuntu 19.10 (Eoan Ermine)](http://releases.ubuntu.com/19.10/) cloud images.

### Large memory guests

After simplifying and changing our guest address space handling, we can now
support guests with large amount of memory (more than 64GB).

# v0.2.0

Highlights for `cloud-hypervisor` version 0.2.0 include:

### Network device offloading

As part of our general effort to offload paravirtualized I/O to external
processes, we added support for
[vhost-user-net](https://access.redhat.com/solutions/3394851) backends. This
enables `cloud-hypervisor` users to plug a `vhost-user` based networking device
(e.g. [DPDK](https://www.dpdk.org)) into the VMM as their virtio network backend.

### Minimal hardware-reduced ACPI

In order to properly implement and guest reset and shutdown, we implemented
a minimal version of the hardware-reduced ACPI specification. Together with
a tiny I/O port based ACPI device, this allows `cloud-hypervisor` guests to
cleanly reboot and shutdown.

The ACPI implementation is a `cloud-hypervisor` build time option that is
enabled by default.

### Debug I/O port

Based on the Firecracker idea of using a dedicated I/O port to measure guest
boot times, we added support for logging guest events through the
[0x80](https://web.archive.org/web/20211028033025/https://www.intel.com/content/www/us/en/support/articles/000005500/boards-and-kits.html)
PC debug port. This allows, among other things, for granular guest boot time
measurements. See our [debug port documentation](docs/debug-port.md) for more
details.

### Improved direct device assignment

We fixed a major performance issue with our initial VFIO implementation: When
enabling VT-d through the KVM and VFIO APIs, our guest memory writes and reads
were (in many cases) not cached. After correctly tagging the guest memory from
`cloud-hypervisor` we're now able to reach the expected performance from
directly assigned devices.

### Improved shared filesystem

We added shared memory region with [DAX](https://www.kernel.org/doc/Documentation/filesystems/dax.txt)
support to our [virtio-fs](https://virtio-fs.gitlab.io/) shared file system.
This provides better shared filesystem IO performance with a smaller guest
memory footprint.

### Ubuntu bionic based CI

Thanks to our [simple KVM firmware](https://github.com/cloud-hypervisor/rust-hypervisor-firmware)
improvements, we are now able to boot Ubuntu bionic images. We added those to
our CI pipeline.

# v0.1.0

Highlights for `cloud-hypervisor` version 0.1.0 include:

### Shared filesystem

We added support for the [virtio-fs](https://virtio-fs.gitlab.io/) shared file
system, allowing for an efficient and reliable way of sharing a filesystem
between the host and the `cloud-hypervisor` guest.

See our [filesystem sharing](docs/fs.md) documentation for more details on how
to use virtio-fs with `cloud-hypervisor`.

### Initial direct device assignment support

VFIO (Virtual Function I/O) is a kernel framework that exposes direct device
access to userspace. `cloud-hypervisor` uses VFIO to directly assign host
physical devices into its guest.

See our [VFIO](docs/vfio.md) documentation for more detail on how to directly
assign host devices to `cloud-hypervisor` guests.

### Userspace IOAPIC

`cloud-hypervisor` supports a so-called split IRQ chip implementation by
implementing support for the [IOAPIC](https://wiki.osdev.org/IOAPIC).
By moving part of the IRQ chip implementation from kernel space to user space,
the IRQ chip emulation does not always run in a fully privileged mode.

### Virtual persistent memory

The `virtio-pmem` implementation emulates a virtual persistent memory device
that `cloud-hypervisor` can e.g. boot from. Booting from a `virtio-pmem` device
allows to bypass the guest page cache and improve the guest memory footprint.

### Linux kernel bzImage

The `cloud-hypervisor` linux kernel loader now supports direct kernel boot from
`bzImage` kernel images, which is usually the format that Linux distributions
use to ship their kernels. For example, this allows for booting from the host
distribution kernel image.

### Console over virtio

`cloud-hypervisor` now exposes a `virtio-console` device to the guest. Although
using this device as a guest console can potentially cut some early boot
messages, it can reduce the guest boot time and provides a complete console
implementation.

The `virtio-console` device is enabled by default for the guest console.
Switching back to the legacy serial port is done by selecting
`--serial tty --console off` from the command line.

### Unit testing

We now run all unit tests from all our crates directly from our CI.

### Integration tests parallelization

The CI cycle run time has been significantly reduced by refactoring our
integration tests; allowing them to all be run in parallel.