Live demo: [Deploy URL]

LLM APIs fail, rate-limit, and drift in latency in ways application code shouldn't have to know about. SentinelAI is the layer that absorbs that: every chat request goes through a semantic cache first, then a circuit-breaker-guarded provider chain (Groq primary, Gemini fallback), with cost and latency recorded on every single call. The goal isn't to make LLM calls smarter — it's to make them boring: predictable latency, bounded blast radius when a provider degrades, and a real accounting of what every request cost.

A typical integration calls groq.chat.completions.create(...) directly from the request handler. Three things happen in production that don't happen in the demo:

A provider has a bad day. Groq returns 503s for ten minutes during a capacity event. Every request that hits it now waits out the full timeout before failing — your p99 latency goes from 2s to 20s, and every one of those calls returns an error to the end user. There is no fallback path because the code never had one.

The same questions get asked over and over. A support bot answers "what's your refund policy" 400 times a day with 400 near-identical prompts. Each one is a full paid inference call at ~1.5-2.5s of latency, even though the answer hasn't changed since the last time someone asked. At scale this is a linear cost curve for what is, semantically, a cache-hit workload.

Nobody can answer "why was it slow yesterday." When latency spikes or a provider starts erroring, there's no per-request record of which provider handled it, how long it took, whether it fell back, or what it cost — just application logs that weren't built to answer infrastructure questions.

SentinelAI addresses all three with one gateway: cache what's already been asked, fail over when a provider is down, and log enough about every request to answer the "why" after the fact.

Multi-provider routing with circuit breaker failover — Every /v1/chat request tries Groq first. If Groq is unavailable (timeout, 5xx, or its circuit is already open), the request transparently retries against Gemini before ever reaching the caller. Each provider has its own in-memory circuit (services/circuit_breaker.py): three consecutive failures trips it to OPEN for 60 seconds, after which one test request is allowed through (HALF_OPEN) to probe recovery. Outcome: a provider outage degrades to "slower, served by the other provider" instead of "every request fails."

Semantic cache with pgvector HNSW indexing — Prompts are embedded with all-MiniLM-L6-v2 (384 dimensions, normalized) and matched against stored responses by cosine similarity, not exact string match. A rephrased question still hits the cache. Lookup is two-stage: an O(1) SHA-256 hash check for exact repeats, then an HNSW-indexed nearest-neighbor search in Postgres for near-duplicates above a 0.92 similarity threshold. Outcome: cache hits return in ~15-50ms instead of the 1.5-2.5s a live provider call takes, at zero marginal cost.

Async request pipeline (FastAPI + Celery + Redis) — The response is built and returned to the client before the request is logged to Postgres or written into the cache. Those two writes are queued as a single Celery task (post_process_task) over a Redis broker and executed by a separate worker process. Outcome: request-serving latency is never coupled to database write latency.

Real-time cost tracking per request — Every response carries usage.cost_usd, computed from actual input/output token counts against a per-model pricing table (services/cost.py). Cache hits report $0.00 and accrue against saved_cost_usd on the cache entry instead. Outcome: cost is a first-class field on every logged request, not an estimate reconstructed later from provider invoices.

Observability dashboard with latency, cache, and provider health — The Next.js dashboard polls /v1/metrics, /v1/logs, /v1/cache/stats, /v1/circuit/states, and /v1/worker/stats every 15 seconds and renders p50/p95/p99 latency, cache hit/miss ratio, per-provider request share and error rate, live circuit breaker state, and a per-request pipeline trace (cache check → provider call → response). Outcome: an operator can see a provider degrading or a circuit opening within 15 seconds, without grepping logs.

Intelligent provider health scoring — Circuit state isn't just up/down: CircuitBreakerRegistry tracks a rolling failure count and last-failure timestamp per provider and exposes it at /v1/circuit/states and /health, so routing decisions and dashboard alerts are driven by the same source of truth.

Loading

Postgres over SQLite — config.py still carries a database_url SQLite fallback from an earlier iteration, but the live engine (db/database.py) is wired to postgres_url. Celery workers write to the database from a separate process than the API server; SQLite's single-writer lock model doesn't hold up once writes are concurrent and out-of-process. Postgres's MVCC gives every worker and every request its own transaction without blocking the others.

pgvector HNSW over a Python-side cosine scan — The naive approach — pull every cached embedding into Python and compute cosine similarity in a loop — is O(n) per lookup and gets slower as the cache grows, which is exactly backwards for a cache. Migration 0002_requests_cache builds an HNSW index (m=16, ef_construction=64) on cache_entries.embedding, so the ORDER BY embedding <=> :vec LIMIT 1 query in check_cache() resolves in Postgres via approximate nearest-neighbor search — O(log n) — instead of a full table scan.

Celery async over synchronous DB writes — /v1/chat returns as soon as it has a response; the DB write and cache-store happen after, via post_process_task.delay(...) over Redis. If those writes were awaited inline, every request's latency would include Postgres round-trip time and, for cache misses, an embedding computation — both irrelevant to what the caller is waiting for.

Circuit breaker over retry-with-backoff — Retrying a failing provider still spends the full timeout window on every failed attempt, which is what caused the p99 blowup described above. CircuitBreakerRegistry.is_available() short-circuits to False instantly once a provider has failed three times in a row, so failed requests stop paying the timeout tax and go straight to the fallback provider. The 60-second HALF_OPEN probe means recovery is detected without a human resetting anything.

Semantic cache over exact-match cache — An exact-match cache (hash the prompt, look it up) only catches identical strings. "What's your refund policy?" and "can you explain the refund policy" are different strings with the same answer. Embedding the prompt and comparing by cosine similarity catches paraphrases; the exact-hash check stays as a fast-path O(1) short-circuit for genuinely identical repeats before falling through to the vector search.

0.92 cosine similarity threshold — Set in services/cache.py as SIMILARITY_THRESHOLD. Below ~0.90, unrelated-but-topically-adjacent prompts ("summarize this contract" vs. "review this contract") start collapsing onto the same cache entry, returning a wrong answer with high confidence. 0.92 was chosen to bias toward precision — a false cache-hit is a silently wrong answer served to a user, which is a worse failure mode than an avoidable cache miss that just costs one extra provider call.

Circuit breaker state in Redis, not in-process memory — An in-process dict only produces correct circuit state for one replica; with more than one backend instance, each would independently decide whether a provider is healthy, defeating the point of a shared circuit. State is now a Redis hash per provider (circuit:{provider}), with the failure counter incremented via HINCRBY (atomic — correct under concurrent requests across replicas, unlike a read-increment-write) and a short-lived SET NX lock around the OPEN transition so concurrent replicas don't all fire duplicate webhook notifications. Falls back to a local in-memory copy if Redis is unreachable, so a Redis outage degrades circuit accuracy but never blocks provider calls — the added latency is negligible either way, since these checks only run on cache misses, which already cost 1.5-2.5s for the LLM call itself.

No load test has been run against this deployment yet — the table below is the schema to fill in after running one (see backend/tests/ for where to add a load-test script; test_gateway.py is currently a placeholder).

Metric Value Cache hit rate TBD Avg latency (cache hit) TBD Avg latency (cache miss / live provider call) TBD Latency reduction (hit vs. miss) TBD Cost saved per 500 requests TBD p95 latency @ 20 concurrent users TBD Failed requests during simulated provider outage TBD

Component Technology Why API server FastAPI + Uvicorn Async request handling; the cache check and provider calls are I/O-bound and benefit from async/await end to end Async workers Celery (solo pool) Decouples DB writes and cache stores from the request/response cycle; solo pool avoids Windows multiprocessing issues in dev, swapped for prefork in production Message broker Redis Backs the Celery task queue (db 0) and result backend (db 1); also used directly for /v1/worker/stats queue-depth checks Primary database PostgreSQL (asyncpg driver) Concurrent writes from the API process and Celery worker process require a real multi-writer database Vector search pgvector (HNSW index) Stores embeddings as a native Postgres column type and answers nearest-neighbor queries with an index instead of a client-side scan Embedding model sentence-transformers — all-MiniLM-L6-v2 384-dim, CPU-friendly, normalized embeddings suitable for cosine similarity; loaded once per process (optionally warmed at startup) LLM providers Groq (llama-3.1-8b-instant), Gemini (gemini-2.5-flash) Free-tier-friendly, OpenAI-compatible (Groq) and REST (Gemini) APIs used as primary/fallback pair Frontend Next.js 14 (App Router) + Recharts Single dashboard page polling the gateway's own observability endpoints; no separate backend-for-frontend Containerization Docker + Docker Compose backend and frontend each ship their own Dockerfile; see setup notes below on what compose does and doesn't start

Docker Desktop

Python 3.12 with the backend's dependencies installed (pip install -r backend/requirements.txt) — the bootstrap script runs Alembic and mints a key directly, outside any container

Free API keys: Groq and Gemini

This single command: checks Docker is running, creates backend/.env from .env.example if it doesn't exist yet, starts Postgres + Redis (docker-compose up -d postgres redis), waits for Postgres to report healthy, runs alembic upgrade head (creates api_keys, requests, cache_entries, and the cache's HNSW index — the full schema, in one step), and mints a default API key — printed once.

After it finishes, open backend/.env and set GROQ_API_KEY and GEMINI_API_KEY (the gateway can't call either provider without them).

Service Port What it runs postgres 5432 PostgreSQL 16 + pgvector redis 6379 Celery broker/backend, API key cache, rate limiting backend 8000 uvicorn app.main:app — FastAPI gateway worker — celery -A app.worker worker --pool=solo — log writes, cache stores, webhook delivery frontend 3000 next start — dashboard

Schema is entirely Alembic-managed (api_keys, requests, cache_entries, plus the cache's HNSW index) — init_db() on backend startup only ensures the vector extension exists, as a safety net.

Returns HTTP 200 with "status": "healthy" once Postgres, Redis, and the providers are all reachable — see API reference for the full response shape, and HTTP 503 with "status": "unhealthy" if the database or Redis can't be reached.

Then open http://localhost:3000 for the dashboard, and confirm the worker status badge shows LIVE.

To view logs or stop everything: docker-compose logs -f / docker-compose down.

frontend/app/page.tsx calls same-origin /api/gateway/*, which frontend/app/api/gateway/[...path]/route.ts (a Next.js server-side route handler, not a next.config.js rewrite — a rewrite can't add headers) proxies to the backend, injecting Authorization: Bearer $SENTINEL_API_KEY there. The browser never receives that header, so it never shows up in DevTools' network tab or the client JS bundle. Configure it in frontend/.env.local (see frontend/.env.example) — use a low-rate-limit tenant key minted via POST /v1/keys, not the master key, since the dashboard only ever needs read access to metrics/logs.

tests/conftest.py spins up a real Postgres+pgvector container via testcontainers (needs Docker) and runs alembic upgrade head against it before any test runs, so integration tests exercise the real schema — not a mock. Redis is faked (fakeredis), and Groq/Gemini calls are monkeypatched (conftest.mock_providers) so the suite has no external dependencies or cost.

tests/unit/ — circuit breaker state transitions (and that webhooks fire only on CLOSED→OPEN/→CLOSED, not every failure), rate limiter windowing and Redis-down degradation, API key hashing/rotation/cache invalidation, cost calculation, webhook HMAC signing, health check status rollup rules.

tests/integration/ — /v1/chat (cache hit/miss, provider fallback, all-down, auth, rate limiting), /v1/keys CRUD + rotation, /v1/circuit/*, /health*, /v1/webhook/* — all through the real HTTP layer via httpx.ASGITransport.

ruff check . lints the same way CI does. Both run automatically in .github/workflows/ci.yml on every push/PR — the CI job needs no external services either, for the same reason (testcontainers + fakeredis).

The README's benchmark table below reflects real runs of this suite. Three scenarios, each answering a different question:

1. Realistic mix, real providers — validates the cache/cost value proposition with real latency numbers. Kept to low concurrency to stay under Groq/Gemini's free-tier rate limits.

2. Infra stress test, mock providers — finds the gateway's own ceiling (DB pool, Redis connections), not Groq's. Requires the mock-provider override:

3. Automated failover — forces the mock Groq into 100% failure mid-run and asserts zero failed requests (Gemini serves everything) plus a confirmed circuit-breaker transition. Requires the same docker-compose.loadtest.yml stack as above:

For a quick single-number check (cache hit rate, p95, cost saved) without installing Locust, tests/load_test.py is a smaller standalone script — see its docstring.

All endpoints except /health, /health/live, and /health/ready require Authorization: Bearer <token>. The token is either the master admin key (API_KEY in .env — always valid, required for /v1/keys/*) or a per-tenant key minted via POST /v1/keys (each with its own requests-per-minute limit). See /docs for the full OpenAPI reference, including /v1/keys/* (key management) and /v1/webhook/* (circuit-breaker webhook config/test).

Send a chat request through the gateway (cache → Groq → Gemini).

Optional "bypass_cache": true skips the semantic cache check.

Paginated request log. Query params: page (default 1), limit (default 50, max 200), status (success | error | fallback), provider (groq | gemini).

Aggregated metrics for a time window (1h | 6h | 24h | 7d, default 24h).

Marks every cache entry as stale (does not delete rows). Useful for demos and testing.

Manually force a circuit back to CLOSED — useful when demoing failover recovery without waiting out the 60s timeout.

Celery/Redis connectivity and queue depth, used by the dashboard's worker status badge.

Returns the static per-1M-token pricing table used for cost calculation.

No auth required. Runs the database, Redis, Celery, and provider checks concurrently and returns HTTP 503 if the database or Redis is unreachable ("status": "unhealthy") — built for uptime monitors that check status code, not just body. GET /health/live is a bare liveness probe (always 200 if the process is up); GET /health/ready returns the same payload as /health and is meant for a Kubernetes readiness probe.

Set in backend/.env (see backend/.env.example).

Variable Required Default Description GROQ_API_KEY Yes "" API key for the primary provider (Groq) GEMINI_API_KEY Yes "" API key for the fallback provider (Gemini) POSTGRES_URL Yes postgresql+asyncpg://sentinel:sentinel_dev_pass@localhost:5432/sentinelai Async SQLAlchemy connection string; this is what the engine actually uses DATABASE_URL No sqlite+aiosqlite:///./sentinelai.db Legacy SQLite URL from an earlier iteration; not used by the active engine API_KEY Yes sentinel-dev-key-123 Master admin key — always valid, required for /v1/keys/*; per-tenant keys are the normal /v1/chat auth path ENVIRONMENT No development Informational — not currently branched on in code CORS_ALLOWED_ORIGINS No http://localhost:3000,http://127.0.0.1:3000 Comma-separated browser origins allowed to call the API GROQ_BASE_URL / GEMINI_BASE_URL No real provider endpoints Override to point at tests/mock_provider.py for load testing LOG_LEVEL No INFO Standard Python logging level; all logs are JSON lines to stdout PRELOAD_EMBEDDING_MODEL No false If true, loads the SentenceTransformer model at process startup instead of on first use LOG_STAGE_TIMINGS No false If true, logs per-stage timing breakdowns (cache_check_ms, groq_ms, gemini_ms, ...) for each /v1/chat call

See backend/.env.example for the full list, including the API key cache TTL, default per-key rate limit, circuit-breaker webhook config, and health-check timeouts — every one of them now lives in Settings (app/config.py) rather than being hardcoded in a service file.

Postgres → managed (RDS, Neon, or Railway Postgres) with pgvector enabled, instead of a local container

Redis → managed (ElastiCache or Upstash) — already just a REDIS_URL/CELERY_BROKER_URL change, no code changes needed

Multiple Celery workers with the prefork pool (the current solo pool is a Windows-dev workaround, single-threaded by design)

Key rotation reminders / expiry — POST /v1/keys/{id}/rotate exists, but nothing currently prompts a tenant to rotate an old key

CI auto-deploy — .github/workflows/ci.yml runs lint, tests, and an image build check on every push; it doesn't push images or deploy anywhere yet, since no deploy target exists

Two items that used to live in this section are actually shipped now: circuit breaker state is Redis-backed (services/circuit_breaker.py) with a same-process in-memory fallback if Redis is down, so it's correct across multiple backend replicas; and the full schema (api_keys, requests, cache_entries, the HNSW index) is Alembic-managed — alembic upgrade head is the only way the schema gets created or changed, Base.metadata.create_all() is gone entirely.