Disk-only clone

TL;DR

fcvm's existing snapshot commands clone a running VM's memory + disk and resume it mid-execution via a UFFD page server. Disk-only clone captures only the disk at a consistent point and cold-boots brand-new VMs from it — no memory image, no UFFD, no shared state. Each clone is a fully independent machine that boots its kernel from scratch and re-launches the container against the prepared filesystem.

# capture only the disk of a running VM (paused briefly for a consistent reflink)
fcvm snapshot create --pid <vm_pid> --tag base --disk-only

# cold-boot independent copies — memory/cpu/hugepages chosen freely at run time
fcvm snapshot run --tag base --name c1 --hugepages --memory 8192 --cpu 4
fcvm snapshot run --tag base --name c2          # another fresh, independent copy

Proposed CLI. This work introduces --disk-only on snapshot create and promotes --tag + --memory/--cpu/--hugepages/--env onto snapshot run (today's run uses --snapshot with cpu/mem internal-only — see P4).

The headline architectural finding: this feature already exists as the empty cell of a 2×2 matrix. The cold-boot path and the snapshot-restore path already share the same disk-reflink machinery (DiskManager::create_cow_disk). Disk-only clone is (captured disk) × (cold boot) — a combination the current code never wires up, but every piece of which is already built and tested. The design's job is to make that matrix explicit instead of adding a fourth ad-hoc path.

The problem & the two asks

From #625, verbatim:

Why this is useful

Boot a VM, do expensive one-time work inside it — install packages, warm an image cache, seed a database, run a build — then capture the disk once and fan out N cold copies that each start from that prepared state. Unlike a memory clone, the copies are not frozen mid-execution: they boot fresh, so they can run a different command, get different resources, and have no shared memory pages or in-flight state to corrupt.

The second ask falls out for free

“Start the clone with hugepages even if the original wasn't” is hard for a memory clone: Firecracker rejects the file-backed memory backend for hugepage snapshots, so hugepages force the UFFD restore path and the backing must match the snapshot (src/commands/snapshot.rs:1001, 1016). A disk-only clone has no memory image to restore, so that entire constraint evaporates — the clone allocates fresh guest RAM and can request --hugepages, any --memory, any --cpu with zero coupling to the source. See Hugepages.

How it differs from the existing snapshot/clone

The proper abstraction

Today there are two boot paths that look unrelated but in fact share their disk plumbing and differ along exactly two orthogonal axes. The design makes those axes first-class instead of bolting on a third special case.

Axis A — RootfsSource: where does rootfs.raw come from?

DiskManager::new(vm_id, base_rootfs, vm_dir) reflinks whatever base_rootfs points at into the per-VM rootfs.raw (src/storage/disk.rs:26,41). It is already source-agnostic:

• BaseLayer — the content-addressed layer2-{sha}.raw from ensure_rootfs(...) (prepare_vm, src/commands/podman/mod.rs:297). Used by podman run.
• CapturedDisk(path) — a snapshot's disk.raw. Already used today by snapshot-restore, which calls DiskManager::new(vm_id, restore_config.source_disk_path, vm_dir) (src/commands/common.rs:886).

Axis B — BootMode: how does the guest come up?

• ColdBoot — fresh kernel boot; fc-agent reads the MMDS boot-plan and launches the container.
• MemoryRestore { uffd, vmstate } — resume a paused process from a memory image.

The matrix — disk-only clone is the empty cell

Both occupied cells already share create_cow_disk. Disk-only clone = the existing CapturedDisk reflink (from the UFFD cell) composed with the existing ColdBoot machinery (from the podman run cell). Almost no new mechanism — just a new wiring of two proven ones.

Axis C — three policies the matrix exposes

Placing the empty cell next to its neighbours reveals that a disk-only clone is not just “cold boot with a different disk” — three smaller policies become explicit. The first review of this design collapsed them into one coarse StoragePolicy; an adversarial pass showed that hides two data-loss bugs, so they are split out.

C1 · QuiescePolicy (capture side)

What state must be flushed to the disk before the reflink. A memory snapshot is consistent only because it also captures the dirty page cache; a disk-only capture has no such safety net, so it must sync/fsfreeze (or stop the workload) first. Detailed in the capture path.

C2 · StoragePolicy (guest side): provision vs. preserve

On a normal cold boot fc-agent provisions storage — it wipes podman state and (re)installs the image. On a disk-only cold boot the storage is already prepared inside the captured disk, so fc-agent must preserve it:

C3 · ContainerStatePolicy (guest side)

The subtlety an adversarial review surfaced: fc-agent always issues podman run --name fcvm-container with no --replace (fc-agent/src/container.rs:948). A captured VM already has an fcvm-container record and writable layer, so a naïve re-run either collides on the name or, if pre-removed, deletes the prepared layer. The clone must choose explicitly:

v1 defaults to StartCaptured: it is the most faithful reading of “start fresh copies from that cloned disk state” — everything written inside the source (including the container's own filesystem) reappears in each copy — and it sidesteps the name collision because podman start fcvm-container resumes an existing record rather than creating a duplicate. This requires fc-agent to branch from the current unconditional podman run --name fcvm-container (container.rs:948) to podman start for the captured container. FreshContainer is the opt-in for “same image/volumes, clean container.” The P4 “file written inside the container layer survives” test exercises the StartCaptured default.

Reuse seams (verified file:line)

Threading rootfs_override: Option<PathBuf> + StoragePolicy through the existing path touches a remarkably small surface, because everything downstream is already parameterized on base_rootfs and args.

The dispatch wrapper (cmd_snapshot_run on a DiskOnly tag) synthesizes a RunArgs from the snapshot's metadata plus CLI overrides, sets rootfs_override = Some(disk.raw) + storage_policy = Preserve + no_snapshot = true, then calls the ordinary prepare_vm → run_vm_loop → cleanup_vm_context trio (mod.rs:155 / :1060 / :1205). All network / vsock / volume / listener / health wiring is reused verbatim.

fc-agent: the Preserve storage policy

On a normal boot the guest agent destroys podman storage and rebuilds it. A disk-only clone must not — the captured disk already contains the writable container layer and (for some image modes) the image itself. This is the highest-risk part of the feature.

Branch points a Preserve plan must guard

On the wire this is #[serde(default)] storage_policy/container_state_policy fields on the Plan struct (fc-agent/src/types.rs), defaulting to today's behavior — so existing plans deserialize unchanged (proven by the minimal-plan test). Changing fc-agent requires rebuilding the initrd (content-addressed by the fc-agent binary SHA).

The image-delivery wrinkle (refined after review)

The first draft said “reject overlay mode.” An adversarial pass showed that conflates two different things — what matters is where the image physically lives, which is determined by delivery, not the image_mode label:

So the v1 guard must key on the effective delivery — “was a separate read-only overlay store device attached?” — which is exactly what MMDS records: it emits image_mode only alongside an image_device (config.rs:377). It must not key on the raw resolve_image_mode(), which defaults to Overlay even for remote pulls (mod.rs:77) that have no device and bake into the rootfs; and not on “an image_device exists,” because btrfs/archive also attach a device (vm_config.rs:481) yet podman load it into the rootfs. The reject condition is precisely overlay-mode AND a store device was attached. In every mode the container's writable layer lives in /var/lib/containers on rootfs.raw; that is what Preserve + ContainerStatePolicy protect.

The capture path

Disk-only capture is create_snapshot_core (src/commands/common.rs:1553) with the memory half removed and a guest quiesce added. The factored create_disk_only_snapshot_core keeps the pause/resume bracket, adds the quiesce, and drops the Firecracker memory dump.

QuiescePolicy — flush the guest before the reflink

Before reflinking, fcvm runs a quiesce command inside the guest over the existing exec vsock channel — the same path fcvm exec uses (host src/commands/exec.rs:146 → fc-agent exec server agent.rs:120 → guest command fc-agent/src/exec.rs:377). No new control channel is needed, and because it's a runtime command (not the boot-time Plan), capture can use it without any fc-agent build change.

Ordering: freeze (exec vsock) → vCPU pause → reflink → vCPU resume → unfreeze (exec vsock). Freeze is the floor because Sync alone leaves a window before the pause. This is the gap codex's first review flagged ("needs disk quiescing").

Metadata the DiskOnly config must carry

SnapshotMetadata (src/storage/snapshot.rs:66–112) already records image, vcpu, memory, network, volumes, hugepages, extra_disks, user, ports, forward-localhost, network-mode, ipv6-prefix, tty, interactive. To reconstruct a cold-boot plan it additionally needs fields that are currently not persisted anywhere:

Hugepages & resources come free

The second ask — “start the clone with hugepages even if the original wasn't” — needs no new mechanism. It is purely a consequence of cold-booting.

// src/commands/snapshot.rs:1016
if hugepages || env("FCVM_FORCE_UFFD") {
    // implicit in-process UFFD server …
}

// src/commands/podman/vm_config.rs:927
MachineConfig {
  vcpu_count: args.cpu,
  mem_size_mib: args.mem,
  huge_pages: if args.hugepages { Some("2M") } else { None },
}

--hugepages, --memory, and --cpu are already implemented for the fresh-boot path; the disk-only run just exposes them on snapshot run and lets them flow through unchanged. The only existing guard that still applies is the 2 MiB memory-alignment check for hugepages (mod.rs:162).

Top risks & mitigations

Ranked; the first three are data-loss / silent-corruption class and gate the design.

1. Torn filesystem — pause alone doesn't flush the page cache. (codex-flagged) Without a memory snapshot, dirty guest pages never reach the disk. Mitigation: QuiescePolicy — sync/fsfreeze (or stop the workload) before the reflink; freeze brackets the copy.
2. fc-agent destroys the captured storage. Four resets plus a truncate+mkfs btrfs setup. Mitigation: StoragePolicy::Preserve guards all of them (container.rs:64/424/451/1103 + 357/377); Provision stays the default. Covered by the “file inside the container layer survives the clone” e2e test.
3. Container name collision / writable-layer loss. (codex-flagged) podman run --name fcvm-container has no --replace. Mitigation: ContainerStatePolicy — v1 StartCaptured (podman start, preserves the writable layer, no collision); FreshContainer opt-in.
4. Clone needs the original host image tag. (codex-flagged) prepare_vm always resolves/exports the image. Mitigation: gate image-resolution + localhost-export (mod.rs:320,451) on rootfs_override — a rootfs-baked clone needs neither.
5. Separate-store (localhost-overlay) image not in the rootfs. Mitigation: v1 rejects only when a separate read-only overlay store device was actually attached (the effective delivery, recorded in metadata) — never the raw resolve_image_mode(), which defaults to Overlay even for remote pulls that bake into the rootfs; v2 re-attaches the content-addressed store.
6. Wrong kernel for a nested/btrfs-profile source. Mitigation: persist kernel_profile in metadata and cold-boot under it; rootfs_type alone is insufficient.
7. Secrets leaking into a world-readable snapshot. Mitigation: env vars are never persisted; re-supplied at run time via --env.
8. Footgun: --pid (UFFD) on a DiskOnly tag. Mitigation: dispatch rejects it; a DiskOnly tag has no memory image to serve.

Stacked PR sequence — green at every step

The work is decomposed so each PR compiles, passes tests locally, and is independently mergeable. Early PRs are purely additive (new fields default to existing behavior), so nothing changes until the final wiring PR turns the feature on. Each branch is stacked on the previous one (main → P1 → P2 → …).

1. P1 — metadata foundation (no behavior change). Add SnapshotKind { Full, DiskOnly } with #[serde(default)] on SnapshotConfig; persist all fields the later PRs read — effective image delivery (overlay-store-attached vs rootfs-baked), kernel_profile, container_cmd, privileged, rootfs_type, non_blocking_output — into VmConfig (src/state/types.rs:102) and mirror into SnapshotMetadata (built at common.rs:1395). Update all SnapshotConfig{…} literals (prod builder + ~6 test fixtures) to set kind: Full. unit tests for serde round-trip + back-compat (old config deserializes as Full); existing snapshot suite unchanged.
2. P2 — disk-only capture (+ guard the old run/serve paths). --disk-only on snapshot create; factor create_disk_only_snapshot_core (freeze via exec vsock → pause → reflink → resume → unfreeze, with unfreeze on every exit path; skip memory dump + memory disk-check); reject only separate-store capture (overlay mode with an attached store device — the effective delivery, not raw resolve_image_mode). Quiesce uses the existing exec vsock (exec.rs:146), so it needs no fc-agent change and can land here. Reject a DiskOnly tag in the existing cmd_snapshot_run (snapshot.rs:707,1076) AND snapshot serve (snapshot.rs:291) with a clear error, so P2 leaves no crashing path on main. capture from a live VM → disk.raw present, no memory.bin/vmstate.bin, kind=DiskOnly; freeze invoked + unfreeze-on-failure; Overlay rejection; `snapshot run/serve` on a DiskOnly tag fail with a clear message, not a missing-file panic.
3. P3 — fc-agent StoragePolicy + ContainerStatePolicy. Add #[serde(default)] storage_policy + container_state_policy to the boot Plan (defaults = today's behavior). Guard all destroyers behind Preserve — the three resets (container.rs:64/424/451), the create_vm_user reset (container.rs:1103), the destructive btrfs truncate/mkfs (container.rs:357/377) while still mounting the existing btrfs.img, early-storage-conf, image-prep; add the Preserve arm (verify image present, run from local storage). Implement StartCaptured (podman start the captured container — v1 default, no name collision, preserves the writable layer) and FreshContainer (remove-then-run). (Quiesce is NOT here — it's a runtime exec command from P2, not a boot-Plan field.) Rebuild initrd. additive (defaults = today); fc-agent unit tests for Plan deserialization + each guard; full existing host suite still green.
4. P4 — cold-boot run wiring + the end-to-end test. RunArgs.rootfs_override + policies through prepare_vm; gate the snapshot-cache diversion and the image-resolution/localhost-export (mod.rs:320,451) on rootfs_override; promote --memory/--cpu/--hugepages/--env to real args on snapshot run; dispatch cmd_snapshot_run on kind == DiskOnly → synthesize RunArgs from metadata (incl. kernel_profile), reject --pid; set process_type=Clone + snapshot_name. e2e: a file written inside the container layer of the source survives the clone & the container runs; hugepage upgrade (source without → clone with, healthy); two clones independent; clone works after the original host image tag is removed; --pid-on-DiskOnly rejected.
5. P5 — docs & published design. README + .claude/CLAUDE.md + DESIGN.md coherence; this page on GitHub Pages. (The design page itself lands first, ahead of code, as this PR.) Pages workflow deploys; doc links resolve; lint/format clean.

Testing plan

Following the repo's rule that a regression test must fail without the fix and pass with it, and that new tests are run locally, not merely compiled:

P4's tests require a rebuilt initrd (from P3). All run via make test-root FILTER=… locally before push; CI runs the full host/container × arm64/x64 × snapshot matrix.

Review gates (issue success criteria)

• /code-review on each PR; findings addressed before merge.
• Local codex adversarial review on each diff; findings addressed.
• Docs kept coherent (README / CLAUDE.md / DESIGN.md) as part of the feature, not after.

Fit with the hypervisor abstraction

fcvm has a separate RFC to become hypervisor-agnostic (Firecracker + Cloud Hypervisor — docs/hypervisor-abstraction.md, epic #632). Disk-only clone is designed to land now on the Firecracker path while mapping cleanly onto those future neutral types:

Because a disk-only clone needs no snapshot format, UFFD, or drive-retarget — the three things that differ most between Firecracker and Cloud Hypervisor — it is the most portable clone variant and a natural early win for the CH backend (which the RFC schedules as “P1: cold boot + run a container, no snapshots”).