Backend Architecture for an Inbound Voice AI Line

Overview

I designed and built the backend behind a regulated SOS phone line in Brazil. The service helps callers with account access problems, suspected fraud, errors, and other urgent support issues. It runs in production and receives approximately one new call every 15 seconds during peak traffic.

The voice experience uses a Speaker–Thinker architecture that I proposed. The Speaker maintains the live conversation, while the Thinker is an existing internal AI agent with the specialized logic needed to resolve support cases. My work made those capabilities securely available inside a phone call without rebuilding them in the voice layer.

Speaker–Thinker without blocking the conversation

The Thinker can take longer to respond than a natural pause in a voice conversation. A conventional blocking tool call would leave the caller waiting in silence.

The backend request remains synchronous, but the Speaker continues talking while it is in progress. When the response arrives, the system injects the result into the Speaker's active context. The agent can then answer with the new information without treating the backend latency as a frozen conversational turn.

Secure, reusable tool gateway

I built a shared multi-tenant Voice Tool Gateway in Go and a specialized connector in Python. The gateway validates OAuth client credentials, derives tenant identity from the authenticated token, resolves each allowed tool through reviewed configuration, and forwards the request to the correct internal destination. mTLS protects internal service communication.

The specialized connector handles the Thinker's protocol, session context, concurrency constraints, and response presentation. Keeping those responsibilities outside the gateway lets the core remain small and reusable. Adding another tenant or standard HTTP tool is mainly a configuration and infrastructure-permission change, rather than a modification to the gateway itself.

Observability separates authentication and routing failures from transport errors and functional backend responses. Telemetry covers latency and status without logging tokens or conversation bodies.

Adaptive authentication for an SOS line

Authentication cannot assume that every caller still has access to their account or WhatsApp. The agent asks what the person can use and selects the available verification path from the conversation. When email is the viable channel, the system sends a PIN and continues the validation flow by phone.

This fallback is especially important for callers whose reason for contacting the SOS line is the loss of another access channel.

Voice CSAT and analytics pipeline

I also developed the voice CSAT flow. At the end of the conversation, the agent asks the caller for a rating from 1 to 5. The score reaches the backend through the post-call webhook, is published to a queue, and is consumed asynchronously by analytics services.

The queue keeps analytics processing outside the critical path of the live call and provides a structured signal for later service-quality analysis.

My role

I proposed the Speaker–Thinker architecture and owned the backend implementation: the Go gateway, the Python connector, OAuth and mTLS controls, the non-blocking conversational tool behavior, and the voice CSAT pipeline.

I was not the owner of the Twilio IVR. I participated in the design and integration decision forums needed to connect that layer with the voice agent and backend platform.

Production status

The system is live on the regulated inbound line in Brazil and handles real peak traffic of approximately one new call every 15 seconds. The current evidence supports production operation and capacity; outcome and satisfaction metrics are not yet consolidated.