
60-second walkthrough of the real operations dashboard, persisted trace, human-review queue, and verified failure behavior.
Auditable AI support triage with policy-gated actions
Business problem
A fluent AI answer is not enough for a support operation. Operators need to know which policy supported it, why a write action was approved or blocked, whether provider failure was mistaken for missing evidence, and whether one customer's data can ever reach another customer's run.
Delivery
• FastAPI workflow with explicit classification, retrieval, citation, policy, and action stages
• Credential-derived tenant isolation before retrieval or tool access
• Versioned knowledge, inspectable retrieval, and exact-quote citation validation
• Separate retry, repair, deadline, call-count, and cost ceilings around model calls
• Human review for risky writes, conflicting evidence, low confidence, tool failure, and injection
• Content-free metrics and persisted traces for operational review
• TypeScript/Next.js operations console for metrics, run traces, evidence, knowledge versions, and review decisions, with the tenant credential kept behind the server boundary
Verified evidence
• 49 backend tests, 2 frontend contract tests, and 3 Chromium E2E tests pass across contracts, failure boundaries, tenant isolation, injection filtering, pagination, and reviewer workflows
• A scripted failure demonstration passes 5/5 scenarios
• The clean Docker Compose path passes 3/3 public API checks with PostgreSQL, and all four operations-console routes return HTTP 200
• The deterministic end-to-end evaluation passes 8/8 scenarios
• Four seeded defects are detected, including fabricated citations and an extra retry
• The measured optional hybrid retrieval path reaches Recall@4 1.00, MRR 0.95, and unsupported-query empty-result accuracy 1.00 on the checked-in evaluation set
Client fit
The deliverable is not merely a prompt or an API. It gives an operator one inspectable path from metrics to a run's evidence and tool trace, then through a human decision without exposing the tenant credential to the browser. The same design applies to ticketing, CRM, onboarding, and other AI-assisted operations. This is a portfolio demonstration; it does not claim paid-provider answer quality or production traffic.
Completed work
https://github.com/yuten0901/ai-support-triage
Need a production-ready AI support system? My Reliable RAG & AI Support Automation with Evaluation service covers grounded retrieval, approval gates, evaluation, observability, and deployment handoff.
https://contra.com/s/26UqocOO-reliable-rag-and-ai-support-automation-with-evaluation
0 likes · 2 views · Aug 26, 2026
Built an auditable FastAPI + Next.js AI support console with tenant isolation, grounded traces, human approval, PostgreSQL, Docker, and browser E2E tests.