# Inside Replit’s New Sandbox Engine, Part 1: The Path to MicroVMs
How Replit rebuilt its sandbox engine with microVMs for stronger isolation, bottomless storage, and fast startup across hundreds of millions of projects.
- Published: 2026-11-04T20:00:00.000Z
- Authors: Luis Héctor Chávez, Jason Dellaluce, Adnan Abdulhussein
- Canonical: https://frontier.replit.com/sandbox-engine-microvms-part-1/
---
Agent sandboxes are all the rage. A new sandbox startup is announced every couple of weeks. At Replit, we’ve been running sandboxes for nearly a decade. For years, we've run our custom sandbox infrastructure to make it easy for anyone to code online, and now Agents use these sandboxes to build applications without requiring any coding at all.
In the past few months, we finished rolling out a complete rewrite of our sandbox engine to improve security, capabilities, and portability across clouds. Our Bring Your Own Cloud offering brings Replit into customers’ AWS, Google Cloud, or Azure environments. As with most infrastructure work, we had to pull this off without anyone even noticing. We migrated hundreds of millions of projects smoothly, without data loss. In this post, we’ll dive into how we changed the isolation boundary, kept sandboxes fast, and made the new engine work at scale.
In this post, we'll dive into how we changed the isolation boundary and kept our sandboxes fast. In Part 2, we cover how we orchestrate tens of thousands of microVMs at once on Kubernetes.
## Why our work matters
Security, alignment, and containment are among the hottest topics in the AI industry. OpenAI’s Hugging Face incident and Anthropic’s evaluation incidents showed the world that autonomous agents can reach and harm real systems beyond their intended scope. Although these evaluations might have had reduced model safeguards, the clear takeaway is that traditional security boundaries are no longer sufficient in the age of AI. NIST’s agent standards initiative and congressional scrutiny put isolation, identity, and monitoring firmly on the industry’s agenda.
Sandboxing was not a new category for Replit. It was infrastructure our customers had depended on since the early days, and a foundation of our defense-in-depth approach. Early on, we realized that as AI agents become increasingly autonomous and widely adopted, the environment in which they operate needs to evolve too. The industry is now converging on the same requirements. DeepSeek’s DSec reinforces the case for elastic sandboxed compute infrastructure, and extra validation comes from open-source initiatives such as Kubernetes Agent Sandbox or Google’s AX and Agent Substrate. Early this year, selected agent-infrastructure startup rounds announced since 2025 totaled **$59.3 million** in funding across Daytona, E2B, Runloop, and Blaxel.
In November 2025, we set out to rebuild our sandbox infrastructure for more autonomous agents, persistent workspaces, and much higher concurrency. By February, we were already using it internally. We spent the following months hardening it and rolling it out.
## The Path to MicroVMs
Our sandbox engine started out simple: stateless code execution in Docker, spread across Google Compute Engine VMs. Over the years, we added persistent workspaces, public endpoints, cached Nix packages, regional clusters, and bottomless storage with instant disk forks. As we set out to chart the path towards microVMs, we realized it would be much easier to start from scratch than to port the existing runtime from Docker to microVMs. This also allowed us to prototype and develop rapidly without risking the existing system.
A microVM is a lightweight virtual machine with its own Linux kernel and a minimal set of virtual devices. We use KVM to separate each sandbox from the host and from other customers’ workloads.
Containers still separate processes inside each sandbox. Hardware-assisted virtualization now separates the sandboxes themselves.
First, we needed to pick a virtual machine monitor (VMM) to run each microVM. We chose Cloud Hypervisor, which shares reusable rust-vmm components with Firecracker and includes features such as virtiofs, transparent huge pages, device hotplug, and live migration. Nested virtualization also lets us run development environments that launch VMs of their own. Hotplug and memory snapshots were particularly useful for our design, as they allowed attaching customers’ disks on demand to running warm-pooled VMs. We also patched Cloud Hypervisor to improve snapshot portability across our supported bare-metal and virtualized environments.
A minimal init process inside each microVM starts, stops, and monitors its containers. The host communicates with it over gRPC using vsock, a host-to-guest channel that does not require an IP network.
With all the primitives in place, we needed a single supervisor to set up the sandbox: start the microVM, configure its network, launch the containers, and run the servers clients use to interact with it through our river protocol.
These are general-purpose Linux VMs, not environments tied to a single app-building workflow. We use the same foundation for our self-driving company: agents investigate incidents, review code, analyze business data, and triage support tickets. In late January, we combined microVMs and remote filesystems with access policies, token proxies, audit logging, and our ZeroTrust network so teams could run agents in parallel against the systems they actually use. People set the goals. Isolated environments give the agents somewhere to do the work.
## Security and isolation
Over the past year, there have been a large number of Linux kernel CVEs that can lead to Local Privilege Escalation (LPE): Copy Fail, Dirty Frag (xfrm-ESP), Dirty Frag (RxRPC), Fragnesia, DirtyClone, epoll race-condition PE bug, Bad Epoll, GhostLock, FUSE filesystem driver flaw. Many of these bugs have been around for years, but developments in AI have increased the rate at which these are being discovered.
Our previous runtime used Linux containers, which share the host kernel. A reachable vulnerability in that kernel can let an attacker escape a container and reach other workloads, credentials, or network endpoints on the host. Not every kernel CVE is exploitable in every configuration, but sharing the kernel makes a successful escape much more consequential.
With microVMs, compromising a sandbox’s guest kernel does not by itself compromise the host kernel. An attacker still has to cross the virtualization boundary.
Giving each sandbox its own kernel also lets us support tools such as Docker, which require privileges we could not safely expose in the shared-kernel runtime.
### Minimized privileges and kernel surface
In our new design, a sandbox consists of a microVM with minimal runc containers running inside. Alongside the development environment, supporting services handle PostgreSQL, git backups, and secure request proxying. We considered a microVM for each component, but decided the extra overhead was not worth it and did not meaningfully change the tenant isolation model. Containers still provide separate namespaces, resource limits, and role-specific permissions inside the microVM guest.
The virtualization boundary does not make every process inside it equally trusted. We configure namespaces, seccomp, device access, and capabilities by role, so a user process does not inherit a monitoring agent’s permissions. Docker gets the additional privileges it needs inside the guest, and never reaches the physical host.
The VMM itself runs under an unprivileged identity with no Linux capabilities.
### Smaller kernel, smaller attack surface
Our guest kernel starts from `tinyconfig`, not a standard distribution configuration. We leave out hardware subsystems a microVM does not need, including Bluetooth, wireless, USB, and display drivers. Your coding agent definitely doesn’t need a Nintendo Switch controller. Required drivers are built in, and guest initialization disables module loading. Given the increasing rate of kernel CVE disclosures this year, our minimal kernel build gives us peace of mind by entirely avoiding most of the affected code areas.
Daily automation keeps our kernel builds moving forward, with autonomous agents built on Replit handling the update workflow as part of our self-driving company.
We audited **6,211 Linux-CNA CVEs** published from June 1 through November 4, 2026, using records from CVEProject and kernel.org. **We avoided 5,969 of them (96.1%)** by promptly incorporating upstream fixes or by excluding the affected code. 4,375 fixes (70.4%) landed even before disclosure, and **676 (10.9%) additional CVEs were excluded by configuration.** At least **15** of these are explicitly described by the Linux CNA or Red Hat as enabling or potentially enabling LPE.
A recent example is CVE-2026-98432, the Linux TCP flaw Anthropic demonstrated could give a remote attacker root access. We incorporated the fixed kernel on October 12, before the CVE was published and more than two weeks before Anthropic described its exploit.
### Never trusting the workload
eBPF programs observe process execution, file access, and network activity inside each sandbox. Linux Security Modules (LSM) provide hooks for enforcing restrictions at the kernel boundary. Together, these mechanisms give us detailed runtime visibility and a way to detect and block specific forbidden operations.
The system determining abuse and compromise is self-driving as well. Runtime telemetry feeds autonomous agents that continuously evolve our known detections and forbidden patterns. We will write more about how we automate threat research at Replit, but think of an always-on security camera that blocks harmful behavior immediately when recognized as such, and improves every time it discovers something new.
There is one security monitor for each sandbox, which operates on both the host and guest sides of the microVM. This component is supervised and is able to self-heal and restart after a crash or tampering, or when rolling out a new version. To guarantee always-on security, we optimized our continuous delivery pipeline so that the security component can be updated and hot-reloaded across all sandboxes running in production less than 15 minutes after changes are merged. Since we operate at the kernel level with eBPF and LSM, this also empowers us to move quickly with ad hoc remediations in response to zero-day kernel exploits even before a fix is available in mainline.
When eMudhra’s intermediate CA keys were compromised, certificate trust became an urgent security problem across the industry. Our sandbox harness automatically injected an updated CA bundle into all running sandboxes, letting us respond through an existing control path rather than asking customers to update each environment.
### Isolate the network
Every sandbox can use the same private IP addresses because it has its own network namespace and routing table. Linux virtual routing and forwarding (VRF) keeps those overlapping networks separate, and connection tracking preserves the routing identity as traffic leaves and returns. Baseline firewall rules restrict access to cloud metadata and infrastructure address ranges, with explicit allowances where private connectivity is required.
For sandboxes with an egress policy, enforcement runs outside the guest. A host-side proxy checks TLS handshake hostnames without decrypting application payloads, filters DNS, and rejects traffic outside the permitted path. The guest cannot disable those host-side rules. Private endpoint tunnels similarly give the workload a local socket while keeping their credentials on the host.
Virtual-NIC rate limiters also enforce configured inbound and outbound bandwidth limits outside the guest, allowing bounded bursts while limiting sustained traffic. Changing the guest’s network configuration does not remove the VMM’s limiter. Bandwidth control contains resource abuse. It is separate from deciding which destinations are allowed.
### Securing credentials and supply chain
Our managed connectors use credential brokers that forward requests to upstream services, such as GitHub and Google Docs. The broker attaches the upstream credential after performing authorization, so the workload can use the service without receiving its account API key. It removes caller-supplied authentication headers and strips sensitive response headers and trailers.
Package managers such as npm and pip also route their requests through a registry firewall. Any attempt to install known malicious dependencies and packages is blocked right away, preventing our users and their agents from getting harmful content into their sandboxes in the first place. Our coding agent is aware of this and autonomously takes care of replacing or removing blocked dependencies for our users to improve supply-chain security.
## Bottomless storage
The filesystem of each sandbox lives on our remote block-storage service, which fetches data on demand. Starting a large workspace does not mean downloading its entire filesystem first.
To integrate with our custom block device service, we wrote a vhost-user virtio-blk backend. This process runs alongside Cloud Hypervisor and handles block I/O for the guest, making our remote disks appear as regular virtual block devices. It also moves away from managing hundreds of network block devices in one host kernel, which caused us lots of pain in the past due to unexpected deadlocks and node failures. This was also one of our first completely AI-written components back in the Claude Sonnet 4 era, going through five to seven iterations before consolidating to the current sync/async hybrid model running in production, which was what maximized throughput while keeping latency low.
Copy-on-write storage also makes forks cheap. A new disk can reference existing blocks and diverge as it writes. We also implemented graceful storage reconnection, so that rollouts can drain outstanding I/O, persist the disk, and reconnect without restarting the guest.
## Keeping sandboxes fast
To spin up a sandbox, we need to boot a Linux kernel, pull images, start containers, and bring up guest agents. Our minimal guest kernel boots quickly, and warm pools let us do most of the remaining work ahead of time. Acquiring a sandbox takes roughly 500 ms, including attaching and mounting the customer’s volumes and configuring supporting services in parallel. That’s the handoff with disks mounted, as booting an empty VM takes much less.
Doing this for tens of thousands of sandboxes at once took a lot more than warm pools. In Part 2, we’ll cover how we orchestrate it all on Kubernetes.
## What comes next
The system has been running healthy in production for a few months. We migrated hundreds of millions of projects without any disruption, and came out with faster boot time, stronger isolation, and evolved security primitives. We’ll keep investing in security and share more about the choices behind our Kubernetes orchestration and automated security research. In the meantime, go build something on Replit.
## Acknowledgements
Authored by Connor Brewster, Jason Dellaluce, Adnan Abdulhussein, Luis Hector Chavez, and the rest of the Compute Platform team at Replit. If you want to work with us on cutting-edge infrastructure, our team is hiring.
Containers still separate processes inside each sandbox. Hardware-assisted virtualization now separates the sandboxes themselves.
First, we needed to pick a virtual machine monitor (VMM) to run each microVM. We chose Cloud Hypervisor, which shares reusable rust-vmm components with Firecracker and includes features such as virtiofs, transparent huge pages, device hotplug, and live migration. Nested virtualization also lets us run development environments that launch VMs of their own. Hotplug and memory snapshots were particularly useful for our design, as they allowed attaching customers’ disks on demand to running warm-pooled VMs. We also patched Cloud Hypervisor to improve snapshot portability across our supported bare-metal and virtualized environments.
A minimal init process inside each microVM starts, stops, and monitors its containers. The host communicates with it over gRPC using vsock, a host-to-guest channel that does not require an IP network.
With all the primitives in place, we needed a single supervisor to set up the sandbox: start the microVM, configure its network, launch the containers, and run the servers clients use to interact with it through our river protocol.
These are general-purpose Linux VMs, not environments tied to a single app-building workflow. We use the same foundation for our self-driving company: agents investigate incidents, review code, analyze business data, and triage support tickets. In late January, we combined microVMs and remote filesystems with access policies, token proxies, audit logging, and our ZeroTrust network so teams could run agents in parallel against the systems they actually use. People set the goals. Isolated environments give the agents somewhere to do the work.
## Security and isolation
Over the past year, there have been a large number of Linux kernel CVEs that can lead to Local Privilege Escalation (LPE): Copy Fail, Dirty Frag (xfrm-ESP), Dirty Frag (RxRPC), Fragnesia, DirtyClone, epoll race-condition PE bug, Bad Epoll, GhostLock, FUSE filesystem driver flaw. Many of these bugs have been around for years, but developments in AI have increased the rate at which these are being discovered.
Our previous runtime used Linux containers, which share the host kernel. A reachable vulnerability in that kernel can let an attacker escape a container and reach other workloads, credentials, or network endpoints on the host. Not every kernel CVE is exploitable in every configuration, but sharing the kernel makes a successful escape much more consequential.
With microVMs, compromising a sandbox’s guest kernel does not by itself compromise the host kernel. An attacker still has to cross the virtualization boundary.
Giving each sandbox its own kernel also lets us support tools such as Docker, which require privileges we could not safely expose in the shared-kernel runtime.
### Minimized privileges and kernel surface
In our new design, a sandbox consists of a microVM with minimal runc containers running inside. Alongside the development environment, supporting services handle PostgreSQL, git backups, and secure request proxying. We considered a microVM for each component, but decided the extra overhead was not worth it and did not meaningfully change the tenant isolation model. Containers still provide separate namespaces, resource limits, and role-specific permissions inside the microVM guest.
The virtualization boundary does not make every process inside it equally trusted. We configure namespaces, seccomp, device access, and capabilities by role, so a user process does not inherit a monitoring agent’s permissions. Docker gets the additional privileges it needs inside the guest, and never reaches the physical host.
The VMM itself runs under an unprivileged identity with no Linux capabilities.
### Smaller kernel, smaller attack surface
Our guest kernel starts from `tinyconfig`, not a standard distribution configuration. We leave out hardware subsystems a microVM does not need, including Bluetooth, wireless, USB, and display drivers. Your coding agent definitely doesn’t need a Nintendo Switch controller. Required drivers are built in, and guest initialization disables module loading. Given the increasing rate of kernel CVE disclosures this year, our minimal kernel build gives us peace of mind by entirely avoiding most of the affected code areas.
Daily automation keeps our kernel builds moving forward, with autonomous agents built on Replit handling the update workflow as part of our self-driving company.
We audited **6,211 Linux-CNA CVEs** published from June 1 through November 4, 2026, using records from CVEProject and kernel.org. **We avoided 5,969 of them (96.1%)** by promptly incorporating upstream fixes or by excluding the affected code. 4,375 fixes (70.4%) landed even before disclosure, and **676 (10.9%) additional CVEs were excluded by configuration.** At least **15** of these are explicitly described by the Linux CNA or Red Hat as enabling or potentially enabling LPE.
A recent example is CVE-2026-98432, the Linux TCP flaw Anthropic demonstrated could give a remote attacker root access. We incorporated the fixed kernel on October 12, before the CVE was published and more than two weeks before Anthropic described its exploit.
### Never trusting the workload
eBPF programs observe process execution, file access, and network activity inside each sandbox. Linux Security Modules (LSM) provide hooks for enforcing restrictions at the kernel boundary. Together, these mechanisms give us detailed runtime visibility and a way to detect and block specific forbidden operations.
The system determining abuse and compromise is self-driving as well. Runtime telemetry feeds autonomous agents that continuously evolve our known detections and forbidden patterns. We will write more about how we automate threat research at Replit, but think of an always-on security camera that blocks harmful behavior immediately when recognized as such, and improves every time it discovers something new.
There is one security monitor for each sandbox, which operates on both the host and guest sides of the microVM. This component is supervised and is able to self-heal and restart after a crash or tampering, or when rolling out a new version. To guarantee always-on security, we optimized our continuous delivery pipeline so that the security component can be updated and hot-reloaded across all sandboxes running in production less than 15 minutes after changes are merged. Since we operate at the kernel level with eBPF and LSM, this also empowers us to move quickly with ad hoc remediations in response to zero-day kernel exploits even before a fix is available in mainline.
When eMudhra’s intermediate CA keys were compromised, certificate trust became an urgent security problem across the industry. Our sandbox harness automatically injected an updated CA bundle into all running sandboxes, letting us respond through an existing control path rather than asking customers to update each environment.
### Isolate the network
Every sandbox can use the same private IP addresses because it has its own network namespace and routing table. Linux virtual routing and forwarding (VRF) keeps those overlapping networks separate, and connection tracking preserves the routing identity as traffic leaves and returns. Baseline firewall rules restrict access to cloud metadata and infrastructure address ranges, with explicit allowances where private connectivity is required.
For sandboxes with an egress policy, enforcement runs outside the guest. A host-side proxy checks TLS handshake hostnames without decrypting application payloads, filters DNS, and rejects traffic outside the permitted path. The guest cannot disable those host-side rules. Private endpoint tunnels similarly give the workload a local socket while keeping their credentials on the host.
Virtual-NIC rate limiters also enforce configured inbound and outbound bandwidth limits outside the guest, allowing bounded bursts while limiting sustained traffic. Changing the guest’s network configuration does not remove the VMM’s limiter. Bandwidth control contains resource abuse. It is separate from deciding which destinations are allowed.
### Securing credentials and supply chain
Our managed connectors use credential brokers that forward requests to upstream services, such as GitHub and Google Docs. The broker attaches the upstream credential after performing authorization, so the workload can use the service without receiving its account API key. It removes caller-supplied authentication headers and strips sensitive response headers and trailers.
Package managers such as npm and pip also route their requests through a registry firewall. Any attempt to install known malicious dependencies and packages is blocked right away, preventing our users and their agents from getting harmful content into their sandboxes in the first place. Our coding agent is aware of this and autonomously takes care of replacing or removing blocked dependencies for our users to improve supply-chain security.
## Bottomless storage
The filesystem of each sandbox lives on our remote block-storage service, which fetches data on demand. Starting a large workspace does not mean downloading its entire filesystem first.
To integrate with our custom block device service, we wrote a vhost-user virtio-blk backend. This process runs alongside Cloud Hypervisor and handles block I/O for the guest, making our remote disks appear as regular virtual block devices. It also moves away from managing hundreds of network block devices in one host kernel, which caused us lots of pain in the past due to unexpected deadlocks and node failures. This was also one of our first completely AI-written components back in the Claude Sonnet 4 era, going through five to seven iterations before consolidating to the current sync/async hybrid model running in production, which was what maximized throughput while keeping latency low.
Copy-on-write storage also makes forks cheap. A new disk can reference existing blocks and diverge as it writes. We also implemented graceful storage reconnection, so that rollouts can drain outstanding I/O, persist the disk, and reconnect without restarting the guest.
## Keeping sandboxes fast
To spin up a sandbox, we need to boot a Linux kernel, pull images, start containers, and bring up guest agents. Our minimal guest kernel boots quickly, and warm pools let us do most of the remaining work ahead of time. Acquiring a sandbox takes roughly 500 ms, including attaching and mounting the customer’s volumes and configuring supporting services in parallel. That’s the handoff with disks mounted, as booting an empty VM takes much less.
Doing this for tens of thousands of sandboxes at once took a lot more than warm pools. In Part 2, we’ll cover how we orchestrate it all on Kubernetes.
## What comes next
The system has been running healthy in production for a few months. We migrated hundreds of millions of projects without any disruption, and came out with faster boot time, stronger isolation, and evolved security primitives. We’ll keep investing in security and share more about the choices behind our Kubernetes orchestration and automated security research. In the meantime, go build something on Replit.
## Acknowledgements
Authored by Connor Brewster, Jason Dellaluce, Adnan Abdulhussein, Luis Hector Chavez, and the rest of the Compute Platform team at Replit. If you want to work with us on cutting-edge infrastructure, our team is hiring.