Self-Hosted LLM Orchestration Stack

Every AI coding tool I used defaulted to the most expensive available model, and I had no visibility into what I was spending or why. Inference was being treated as a convenience rather than a scarce resource.

A self-hosted orchestration layer on LiteLLM with an 8-tier cost hierarchy across 12+ providers. The bottom tier is permanently free, the next is subscription-covered, and paid frontier models are reserved for tasks that genuinely need them. Every tier has an ordered fallback chain, and provider-level daily budget guardrails route around exhausted providers automatically.

Context-window exhaustion routes separately from general provider failure. Running out of context is a structurally different problem from an outage: you do not want the next cheapest model, you want the largest-context one. So context errors escalate to a 2M-context model, while the general chains keep handling rate limits, outages, and budget exhaustion.

14 role-typed agents sit on top of the router, each with tuned temperature, a constraining system prompt, and a model sequence chosen for what each fallback is actually good at rather than just the next available option. The critic agent deliberately runs on a free model, because adversarial brainstorming benefits from volume and independence more than raw capability. The code reviewer gets a 1M-context model, because full-codebase audits actually need it.

Docker Compose locally, a VPS on AWS Lightsail behind Caddy for remote HTTPS access.

Grafana and Loki for observability, plus persistent vector memory across sessions via Mem0 and Qdrant.

GitHub Actions CI/CD with preflight config validation and commit-level rollback. Wiping persistent volumes is explicitly forbidden in the pipeline, so a routine deploy can never destroy state.

Unverified model IDs ship flagged with VERIFY comments, because a wrong ID silently falls through to a different provider and burns its budget invisibly.

For routine tasks the workflow now runs effectively free. Measured against defaulting to a frontier model, that is a 30 to 150x cost reduction on the same work.