Cover image for OVERVIEW
GitHub-hosted runners served us initially,

OVERVIEW GitHub-hosted runners served us initially, but as CI/CD workloads grew to 10,000+ pipeline executions per month, we needed custom compute: specific AMIs with pre-installed tooling, Docker-in-Docker with privileged access, and control over scaling behavior. I worked on building a self-hosted runner platform that gave us full ownership of the compute layer. The platform consists of three interconnected systems: a Python autoscaler consuming GitHub webhooks and driving ASG capacity via Redis state, instance orchestrators running on each machine managing runner lifecycle with ephemeral detach-and-terminate patterns, and resource monitors collecting CPU/memory/disk/network telemetry for Datadog. The result: 500+ concurrent runners, 10K+ monthly pipelines, 60% cost reduction vs GitHub-hosted, and sub-60-second queue times. The entire fleet runs in a dedicated AWS account with isolated networking, Packer-built AMIs, and Atlantis-managed Terraform. CHALLENGE & SOLUTION The Problem: GitHub-hosted runners cost $0.008/minute (Linux) and $0.016/minute (macOS). At 10K+ executions/month averaging 8 minutes each, we were spending over $6,400/month, growing 15% month-over-month. Beyond cost, specific builds needed Docker-in-Docker with privileged access, 32GB+ RAM for monorepo compilations, and custom kernel configurations for security scanning tools. Queue times during peak hours exceeded 5 minutes. GitHub's shared fleet couldn't prioritize our workloads or guarantee capacity. We needed elastic compute that matched our demand patterns without the overhead of managing individual runner processes. The challenge wasn't just deploying EC2 instances, it was building an intelligent autoscaler that could react to GitHub's webhook events in real-time, scale ASGs up and down based on actual queue depth, and ensure instances were ephemeral (no state left behind between jobs). The Solution: The autoscaler is a Python service (FastAPI) that consumes GitHub workflow_job webhook events and drives ASG desired capacity. It maintains queue and rate signals in Redis, using a reactive-plus-predictive sizing model: queue depth, job arrival rate, estimated runner startup time, and a configurable multiplier. The controller runs on an APScheduler loop that reconciles ASG capacity against Redis state, with throttling-aware intervals to avoid AWS API limits. The instance orchestrator runs on each machine. On startup, it loads AWS config via boto3, constructs the host abstraction (with atexit Terminate() for failure cleanup), initializes a Docker client and systemd-backed runner service. It pulls CI credentials from Secrets Manager, randomly selects from multiple service accounts for rate-limit distribution, authenticates with container registries, then starts the runner. When a job begins, the orchestrator detaches the instance from the ASG (preventing new job assignments) and when the job completes, it terminates the instance — true ephemeral runners. The resource monitor is a sidecar Python process using psutil that collects CPU, memory, disk, and network metrics, submitting them to Datadog via the datadog library. It provides fleet-wide visibility into resource utilization and abuse patterns. The AMI layer uses Packer templates with a centralized version file as the single source of truth for runner binaries. AMI build commits are recorded in version-tracking files, and Terraform references these SHAs for immutable, rollback-friendly deployments. The CI pipeline bakes AMIs, commits version bumps, and Atlantis autoplan triggers for staging/production rollout. ARCHITECTURE: HOW IT WORKS Autoscaler, Event-Driven Capacity Management The autoscaler's architecture: Webhook handler (validates GitHub payloads via hmac verification) → Queue monitor (Redis-backed state using redis-py) → Controller (APScheduler loop that reconciles desired capacity). The controller lists tagged ASGs via boto3, computes desired capacity using documented formulas (startup time, predictive multiplier, arrival rate window), and calls set_desired_capacity. Orphan cleanup runs via EC2 describe_instances. Separation of concerns: webhook transport (FastAPI routes) vs scaling domain (controller module) vs AWS client (boto3 wrappers) vs metrics emission (datadog library). Interface-driven AWS boundaries with dependency injection for isolated unit tests using moto. Instance Orchestrator, Ephemeral Runner Lifecycle: The orchestrator's lifecycle: Init (boto3 config, Docker client, Secrets Manager pull) → Authenticate (random service account selection for rate-limit distribution) → Run (systemd runner service) → Detach (remove from ASG on job start) → Terminate (instance shutdown on job completion or failure). The random service account selection is deliberate: it distributes token usage across multiple accounts, avoiding rate limits on any single one. The host abstraction uses a protocol class to abstract cloud-specific operations (IMDS, ASG, Secrets Manager) for testability. AMI Pipeline & Version Pinning Packer templates with centralized version file. When AMIs rebuild (via CI with corporate build controls), the pipeline commits updated SHAs to version-tracking files. Terraform references these SHAs, so 'rollback' = 'restore previous SHA and atlantis apply'. Deployment guardrails include labels for tested configurations and concurrency notes across runner flavors (stable, docker, docker-arm, slim-arm). Terraform Infrastructure Terraform projects per runner flavor/environment: stable, stable-staging, docker, docker-arm, slim-arm, plus VPC and IAM modules, plus shared module code. All managed via Atlantis with per-directory plan/apply. The dedicated AWS account has isolated networking, and runners can only communicate with public-facing assets, no access to the internal production network.

Rishikesh Sonawane

AWS DevOps | Terraform | CI/CD | Cloud Infrastructure | AI