Modal swaps virtio-fs for block devices to speed up VM Sandbox filesystems

Fast to boot, slow to use

Modal recently shipped VM Sandboxes, which it says support more sophisticated agentic workloads at scale with the same performance expectations as its gVisor-based platform. The filesystem is a core building block, since everything in a Sandbox depends on its speed.

The company's container filesystem is ImageFS, shipped in April 2022. According to Modal, it pulls only the minimum metadata upfront, mounts it through a Rust-based FUSE server, and lazily fetches file contents from a tiered content-addressed cache that runs from local memory to regional CDNs to global blob storage. Modal credits ImageFS as a large part of why its containers cold-start in under a second. Writes are handled with OverlayFS, which merges the read-only ImageFS layer with a writable upper layer.

Containers share the host kernel and can use ImageFS directly. VMs run a full guest kernel that cannot see the host's filesystems, so Modal first bridged the gap with virtio-fs, which forwards guest filesystem operations to a daemon on the host called virtiofsd. VMs booted from the same modal.Image objects with the expected cold-start performance.

Day-to-day use was a different story. Modal reports that a recursive chown went from four seconds to over a minute, that real-world workloads such as pnpm install and bun install looked thrice as slow, and that ripgrep performance fell off a cliff.

Why chown was so costly

Modal broke down the syscalls from its bun and pnpm workloads to build synthetic microbenchmarks. It compared ImageFS used directly on the host, behind gVisor, and over virtio-fs in a VM. In those tests, stat was 2x slower under virtio-fs, while chown was over 6x worse.

Tracing virtiofsd showed that a single chown decomposed into many operations. The guest kernel sent a FUSE_GETXATTR to check and clear file capabilities, a FUSE_GETATTR to check and clear setuid bits, and finally a FUSE_SETATTR for the ownership change. In total, that meant three round trips from guest to host and six host syscalls by virtiofsd, each landing on ImageFS. Modal says one chown over virtio-fs took 99 µs, even with virtiofsd tuned to cache and coalesce operations as much as possible.

The underlying issue, in Modal's framing, is that virtio-fs is a shared filesystem. Metadata has an authority, and virtio-fs places that authority on the host side of the VM boundary, so every metadata operation has to cross it, even when the host never needs to read the VM's root filesystem.

Speaking in blocks

Modal's answer was to let the filesystem live entirely inside the guest. The host then only holds bytes and returns them on request, which is what a block device does. Filesystems like ext4 translate between files and blocks, and the guest kernel already treats each block device as exclusively owned, so no operation has to ask someone else about the state of the world.

With virtio-blk, the same chown stays inside the guest. The ext4 driver checks capabilities and setuid bits locally, makes the ownership change and logs it to its write-ahead journal. The host learns about it later, when ext4 flushes a batch of writes. According to Modal, this spreads the cost of hundreds or thousands of file operations over a single block command and happens asynchronously. Modal puts chown over virtio-blk at about 2.1 µs per call.

The trade-off is that the host can no longer work with the VM's filesystem, since it only sees blocks.

ImageFS-blk: keeping lazy loading

Block devices solved speed but not compatibility with ImageFS's lazy loading and modal.Image support. Running mkfs.ext4 -d to build a disk would require all image content upfront, which defeats the lazy design.

Instead, Modal pulls only minimal file metadata for a requested image and uses it to write ext4's inode records itself. Each inode holds a file's permissions, size and a list of extents, the block ranges that represent its data. The rest of the ext4 image is left sparse, and actual blocks are supplied only when the guest reads them. Because Modal wrote the ext4 metadata, it can map block requests to ImageFS file bytes and back. A host daemon named ImageFS-blk handles this.

The source text provided ends while describing that daemon, so further details of its implementation are not covered here.

What it means for Sandbox users

For people running workloads in VM Sandboxes, the point of the work is that package installs, searches and metadata-heavy operations should not pay a penalty for the VM boundary. Modal's own measurements show the gap it was closing: 99 µs versus about 2.1 µs for a single chown. The company says the goal is to keep VM Sandboxes at the same performance expectations as its container platform.