Website: femu-ose.github.io · Join the FEMU community on Discord for FEMU-related discussions, questions, and ideas. Everyone is welcome!
______ ______ __ __ _ _
| ____| ____| \/ | | | |
| |__ | |__ | \ / | | | |
| __| | __| | |\/| | | | |
| | | |____| | | | |__| |
|_| |______|_| |_|\____/ -- A fast, accurate, scalable, and extensible NVMe SSD Emulator
FEMU is a fast, accurate, scalable, and extensible NVMe SSD emulator based on QEMU/KVM. It enables full-system evaluation of storage systems and supports multiple SSD architectures for systems research.
New to FEMU? Start with The FEMU Manual (PDF). One document covers building and running FEMU, its architecture and the design of each component, every mode and feature, every parameter, measuring, troubleshooting and contributing, with diagrams throughout.
FEMU is supported by the U.S. National Science Foundation through NSF POSE award #2550145, Toward a Community-Driven Fast Emulator (FEMU) Ecosystem for Next-Generation Storage Systems Research and Innovation.
Consolidation in progress (2026). Features that were previously maintained in separate FEMU-based repositories are being ported into this repository; CXL SSD emulation is already merged as the
femu-cxl-ssddevice. FEMU is also being made easier to configure, script, and drive with AI coding agents. Much of this work is AI-assisted. We believe that careful use of AI-assisted coding, with every change built and regression-tested in CI and reviewed by the maintainers, will help FEMU reach a more organized code structure and better efficiency. A more thorough review is under way in parallel; in the meantime, some existing behavior may change or regress. Please report bugs, regressions, and feature requests through GitHub Issues or Discord. Contributions made with your own coding agents are welcome too: feel free to submit pull requests.
Overview · Features · Architecture · Requirements · Installation · Quick Start · Where to Go Next · Usage · Configuration · Development · Troubleshooting · Citation · Contributing · Support · License · Acknowledgments
The full documentation starts at the doc map, and the same pages are collected in the FEMU Manual (PDF). What changed since the last release is in the changelog.
FEMU bridges the gap between SSD hardware platforms and SSD simulators. It runs the full system stack (applications, OS and the NVMe interface) on emulated SSDs of several architectures, each with configurable parameters.
- Fast: NoSSD mode completes I/O in a few microseconds to tens of microseconds, depending on the host (NoSSD).
- Accurate: the SSD modes charge NAND, channel and garbage collection time from a configurable timing model.
- Scalable: several devices and namespaces per VM, within the host sizing limits.
- Extensible: each mode is a separate backend under
hw/femu/(code structure).
| Mode or feature | Use it for | Turn it on with | Guest kernel | Guest tools | Host needs | Launcher | Checked |
|---|---|---|---|---|---|---|---|
| NoSSD | fast NVMe device in DRAM, no flash timing | femu_mode=2 (the default) |
any with the NVMe driver | nvme-cli, fio | none beyond the common ones | run-nossd.sh |
CI: realize, Identify, write and read back |
| BlackBox SSD (BBSSD) | a commercial SSD: device FTL, GC, NAND timing | femu_mode=1 |
any with the NVMe driver | nvme-cli, fio | about 17 GiB free RAM for the launcher's 12 GiB device | run-blackbox.sh |
CI: realize, Identify, write and read back; guest: quick start, run end to end |
| Zoned Namespace (ZNS) | zoned storage research | femu_mode=3 |
5.9 or newer with CONFIG_BLK_DEV_ZONED=y; 4 KiB guest pages |
nvme-cli 1.12 or newer for nvme zns |
none beyond the common ones | run-zns.sh |
CI: realize, Identify, write and read back |
| Open-Channel SSD 1.2 | host-managed FTL research | femu_mode=0,lver=1 |
4.16 to 5.14 (LightNVM was removed in 5.15) | LightNVM tools, or SPDK on newer kernels | none beyond the common ones | run-whitebox.sh |
CI: realize, Identify |
| Open-Channel SSD 2.0 | host-managed FTL research | femu_mode=0 (lver=2 is the default) |
4.17 to 5.14 (LightNVM was removed in 5.15) | LightNVM tools, or SPDK on newer kernels | none beyond the common ones | run-whitebox.sh |
CI: realize, Identify |
| Key-value SSD (KV) | key-value store research | femu_mode=5 |
6.0 or newer; no block device, the namespace is /dev/ngXnY |
nvme-cli io-passthru, hw/femu/scripts/kv-probe.c |
none beyond the common ones | run-kvssd.sh |
CI: realize, Identify, store and retrieve |
| Computational storage (CSD) | running programs next to the data | femu_mode=4,fdm_size=<MiB> |
any with the NVMe driver | hw/femu/tests/csd tools |
csd_program_dir for shared-library programs; --enable-csd-ubpf build for eBPF programs |
run-csd.sh |
CI: realize, Identify, write and read back |
| Flexible Data Placement (FDP) | placement hints on a BBSSD | femu-subsys,fdp=on,fdp.nruh=<n> and femu,femu_mode=1,subsys=<id> |
any with the NVMe driver; placement hints need passthrough or io_uring commands | nvme-cli with nvme fdp |
none beyond the common ones | run-blackbox-fdp.sh |
CI: realize, Identify, write and read back |
| Multiple namespaces | several namespaces, each with its own mode | namespaces=<n>, optionally namespace_sizes and namespace_modes |
any with the NVMe driver (ZNS namespaces need what ZNS needs) | nvme-cli | none beyond the common ones | none | CI: realize, Identify, write and read back |
| Namespace management | create, delete and attach namespaces at run time | ns_mgmt=on on a NoSSD or BBSSD controller; femu-subsys,ns_mgmt=on to share namespaces |
any with the NVMe driver | nvme-cli create-ns, attach-ns |
none beyond the common ones | none | CI: realize, Identify, write and read back |
| Metadata and protection information | per-block metadata, PI types 1 to 3 | meta=<bytes>,mc=<mask>, plus pi=on with meta of 8 or more |
CONFIG_BLK_DEV_INTEGRITY=y to use metadata formats through the block layer |
nvme-cli format |
none beyond the common ones | none | CI: realize, Identify, write and read back |
CXL SSD, der=off |
CXL memory backed by flash, all accesses trapped | femu-cxl-ssd below pxb-cxl and cxl-rp on -machine q35,cxl=on |
CONFIG_CXL_BUS, CXL_PCI, CXL_ACPI, CXL_MEM, CXL_PORT, CXL_REGION, CXL_REGION_INVALIDATION_TEST (in a VM), DEV_DAX, DEV_DAX_CXL, DEV_DAX_KMEM |
cxl-cli, daxctl, ndctl |
a build with CONFIG_CXL_MEM_DEVICE |
run-cxlssd.sh |
CI: realize |
CXL SSD, der=memslot |
cached pages mapped into the guest as KVM memory slots | der=memslot on femu-cxl-ssd |
as for der=off |
as for der=off |
KVM (TCG is refused) | run-cxlssd.sh |
CI: realize |
CXL SSD, der=cylon |
cached pages mapped by a Cylon host kernel | der=cylon,cylon-kernel-ack=on on femu-cxl-ssd |
as for der=off |
as for der=off |
Cylon host kernel; KVM with EPT A/D bits and the TDP MMU; 4 KiB host pages; a shared, preallocated hugetlb backend. Without them the device warns and uses MMIO | run-cxlssd.sh |
CI: realize |
| CXL caching API (CCA) | guest pins, unpins and invalidates cached pages | cca=on on femu-cxl-ssd |
as for der=off; a devdax region |
hw/femu/tools/cca (ccactl, cca-test), run as root |
as for der=off |
run-cxlssd.sh |
CI: realize |
| NVMe front end on a CXL SSD | the same media as CXL memory and as an NVMe namespace | femu,bus=pcie.0,femu_mode=1,cxl_ssd=<id> after the femu-cxl-ssd |
as for der=off, plus the NVMe driver |
as for der=off, plus nvme-cli |
as for der=off |
none | CI: realize, Identify, write and read back |
When femu_mode is not set, the device runs in NoSSD mode (2). Flexible Data
Placement is not a separate mode: it is BlackBox with fdp=on set on the
subsystem. The CXL SSD is not an NVMe mode either: femu-cxl-ssd is a CXL
Type-3 memory device whose DRAM page cache sits in front of the BlackBox FTL.
OpenChannel needs a host that speaks it; LightNVM was removed from Linux in
5.15. Choosing a mode has the full
decision table.
+----------------------------------------------------------+
| VM / Guest OS |
| NVMe block device CXL memory |
| (nvme-cli, fio, ...) (devdax / kmem) |
+------------^^-----------------------------^^-------------+
|| ||
PCIe / NVMe CXL.mem (Type 3)
|| ||
+-------------------vv--------------------+ +------vv-------------+
| FEMU NVMe SSD controller | | femu-cxl-ssd |
| +------------+ +----------+ +---------+ | | |
| | BlackBox | | WhiteBox | | ZNS | | | DRAM page cache |
| | (BBSSD) | | (OCSSD) | | (ZNSSD) | | | (FIFO/LIFO/CLOCK/ |
| | + FDP | | | | | | | S3-FIFO) |
| +------------+ +----------+ +---------+ | | + direct mapping |
| +------------+ +----------+ +---------+ | | into the guest |
| | NoSSD | | CSD | | KVSSD | | | |
| | (ultra-low | | (compute | | (key- | | | misses and dirty |
| | latency) | | storage)| | value) | | | evictions go to |
| +------------+ +----------+ +---------+ | | the BlackBox FTL |
+-----------------------------------------+ +---------------------+
| FTL and NAND flash timing model (all modes except NoSSD) |
+-----------------------------------------------------------------+
| QEMU/KVM |
+-----------------------------------------------------------------+
| Host Linux |
+-----------------------------------------------------------------+
The NVMe controller (NVMe 1.4, reported as version 1.4.0) hands each command to the mode backend that owns the namespace. The modes with flash share the FTL and NAND timing model, and the emulated medium lives in host DRAM. Architecture walks through each layer, its source files and its threads.
An x86_64 Linux host with KVM, Python >= 3.9 and GLib >= 2.66 (Ubuntu 22.04 or 24.04; CI builds on both). The emulated SSD lives in host DRAM, so the default BBSSD launcher needs about 17 GiB of free RAM. Full details, including the guest kernel each mode needs: requirements.md.
OCSSD needs a guest kernel older than 5.15 and ZNS needs 5.9 or newer.
git clone https://github.com/MoatLab/FEMU.git
cd FEMU && mkdir build-femu && cd build-femu
cp ../femu-scripts/femu-copy-scripts.sh . && ./femu-copy-scripts.sh
sudo ./pkgdep.sh # Debian/Ubuntu dependencies
./femu-compile.sh # builds build-femu/qemu-system-x86_64Dependencies, optional features (CSD uBPF, CXL SSD), debug builds and common build errors: build.md.
From build-femu/:
./make-guest-image.sh # Ubuntu 24.04 guest in ~/images/u20s.qcow2
./run-blackbox.sh # terminal 1: boot the guest with a BBSSD
./run-guest-ssh.sh sudo nvme list # terminal 2: the emulated SSD is /dev/nvme0n1
./run-guest-ssh.sh sudo poweroffThe full walk-through, with fio and the write amplification factor, is in quick-start.md. Other ways to get a guest image are in guest-image.md.
The emulated SSD lives in memory: nothing written to it survives shutting the VM down.
| I want to | Read |
|---|---|
| Read everything in one document | The FEMU Manual (PDF) |
| Pick a mode for my experiment | Choosing a mode |
| Set up one mode or feature | the guide linked from the Features table |
| Look up a property, counter or script | properties, runtime properties, log pages and counters, scripts |
| Understand how FEMU works | architecture, timing model, security and limits |
| Measure or tune | measuring, performance tuning |
| Fix a problem | troubleshooting and FAQ, debugging |
| Change FEMU | code structure, testing, CONTRIBUTING.md |
| See what changed | changelog |
Everything else is in the doc map.
Each mode has its own guide, linked from the Features table and below.
CSD mode is derived from CEMU. We thank the CEMU authors, Qiuyang Zhang, Jiapin Wang, You Zhou, Peng Xu, Kai Lu, Jiguang Wan, Fei Wu and Tao Lu, and Emilio (@Emilio597), who ported it to FEMU in #188. If you use the CSD mode, please also cite CEMU (BibTeX).
Every device property, with its type, default and meaning, is in
properties.md, and the QOM counters in
runtime-properties.md. Both
are generated from the binary, and CI fails when they fall out of date.
./qemu-system-x86_64 -device femu,help prints the same descriptions.
All FEMU code, scripts and docs live under hw/femu/. A top-level
femu-scripts link points to hw/femu/scripts/.
The troubleshooting and FAQ page answers the questions asked most often in the issue tracker; build.md covers build errors.
- Check the doc map and the FAQ.
- Search Issues for similar problems.
- Ask in GitHub Discussions or on Discord.
- Contact the maintainers for research collaboration.
FEMU has been used in systems research published at ASPLOS, OSDI, SOSP, FAST, SIGCOMM, HPCA, DAC, DATE and other venues. See the growing list of research papers using FEMU.
If you use FEMU in your research, please cite our FAST 2018 paper. The same entry is in CITATION.cff, which GitHub shows as "Cite this repository":
@inproceedings{Li+18-FEMU,
author = {Huaicheng Li and Mingzhe Hao and Michael Hao Tong and
Swaminathan Sundararaman and Matias Bj{\o}rling and Haryadi S. Gunawi},
title = {{The CASE of FEMU: Cheap, Accurate, Scalable and Extensible Flash Emulator}},
booktitle = {16th USENIX Conference on File and Storage Technologies (FAST 18)},
year = {2018},
}If you use one of these modes, also cite the paper it comes from:
- FDP: WARP, Characterizing and Emulating FDP SSDs with WARP (FAST '26).
- CXL SSD (
femu-cxl-ssd): Cylon, Cylon: Fast and Accurate Full-System Emulation of CXL-SSDs (FAST '26). - CSD: CEMU, CEMU: Enabling Full-System Emulation of Computational Storage Beyond Hardware Limits (ASPLOS '26).
We welcome contributions from the community! FEMU is actively used in systems research worldwide.
CONTRIBUTING.md covers style (scripts/checkpatch.pl),
tests, sign-off and pull requests. Document new properties and features under
hw/femu/docs/, with usage examples, and add an
entry to the changelog.
We welcome research collaborations: joint paper development, access to advanced FEMU features, and performance optimization consulting. Email huaicheng@cs.vt.edu with your research area, institution and timeline.
GitHub Issues for bugs and feature requests, GitHub Discussions and Discord for questions, and the doc map.
For research institutions and industry partners: custom FEMU development and consulting, performance optimization services, training workshops and tutorials, and priority technical support. Contact Huaicheng Li, Virginia Tech.
For a bug, include the host OS and kernel version, the FEMU commit, the full QEMU command line, the complete error messages or logs, the steps to reproduce, and what you expected; reporting a bug has the full list. For a feature request, describe the use case and motivation, the technical requirements, and an implementation approach if you have one; consider contributing it.
FEMU is released under the GNU General Public License v2.0 or later.
Copyright (C) 2018-2024 Virginia Tech and Contributors
This program is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
Full license text: GPL-2.0
FEMU incorporates code from QEMU (machine emulator and virtualizer, GPL v2.0), the QEMU NVMe controller, LightNVM (OpenChannel SSD support) and Linux kernel headers and interface definitions (GPL v2.0). See individual file headers for specific attribution details.
FEMU development is supported by the U.S. National Science Foundation (NSF POSE award #2550145), Virginia Tech (primary development and maintenance), research collaborators (algorithm contributions and validation) and the systems community (feedback, bug reports and improvements).
Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
FEMU builds upon QEMU/KVM (virtualization infrastructure), ideas from SSD simulators (SSDSim, FlashSim, VSSIM), hardware platforms (OpenSSD, DFC), and the NVMe, OpenChannel and ZNS specifications.
We thank all contributors who have helped improve FEMU: algorithm developers and performance optimizers, platform porting and compatibility testing, documentation improvements and examples, bug reports and feature suggestions.