Fire2, distributed Firecracker sandbox platform

Overview

Fire2 provisions disposable sandboxes for code that cannot be trusted. An orchestrator distributes requests across workers, and each worker controls its own Firecracker microVMs. I designed and built the system alone in about 7,800 lines of Go.

Reliability under failure

Sandbox creation takes several seconds and can fail at any point. I modeled the workflow as a state machine with pending, scheduling, assigned, starting and running states. PostgreSQL and River provide at-least-once job delivery. Each transition is atomic and idempotent, with conditional updates that prevent a retry from creating a second VM or overwriting a newer state.

A smaller attack surface

Workers execute untrusted code, so they never receive PostgreSQL credentials. They communicate with the orchestrator over gRPC on a private Tailscale network provisioned with Ansible. This limits what a compromised worker can reach and avoids one database connection per added worker.

Scheduling and recovery

The scheduler uses weighted random selection based on worker heartbeats, with 70 percent of the score from available CPU and 30 percent from available memory. On restart, a worker inspects proc, finds orphaned Firecracker processes and removes them before accepting new work.

Testing and delivery

Integration tests run against disposable PostgreSQL instances with testcontainers. Controlled gRPC failures and latency exercise each recovery path. The platform includes authentication, per-user sandbox ownership, VM sizing, deletion endpoints and a React interface. Deployment uses Ansible and templated systemd units.