Tested before launch: AI voice agent QA and security

The challenge An AI agent can sound perfect in a demo and still fail the one call that matters: the emergency, the angry customer, or the caller trying to trick it. Most agents go live after a few test calls.

My role I designed and built the whole testing and security suite, and the dashboard that shows the results.

What I built • 38 simulated callers per run (angry, rushed, off-topic, emergencies), each scenario run 3 times: 114 calls for about $0.25 • Deterministic checks first, an LLM judge only where code can't decide • Metrics that matter: containment, correct vs unnecessary transfers, missed handoffs, hallucinations • A security track based on the OWASP Top 10 for LLM Applications: 21 prompt-injection and data-leakage attacks, including Arabic and hidden instructions in CRM notes • Every change runs through the suite in CI and is blocked if it regresses • Results in a React + TypeScript dashboard built test-first (88 tests, 99% coverage)

The result • The first run caught a plumbing agent missing 5 of 41 emergency handoffs, including a gas leak, before it ever went live • Giving each agent only the caller's own record cut leaks of other customers' details from 14 calls to 0 • Injection resistance rose from 57% to 94%

See it Live results: https://voicedesk-console.vercel.app Full write-up: https://adelatya.vercel.app/case-studies/voice-agent-security-evals/