Speaker-Aware AI Voice Assistant Development
Vince Swu
Speaker-Aware AI Voice Assistant
Overview
I built a real-time voice assistant that combines speaker identification, speech recognition, LLM-based responses, and text-to-speech into a single Python application.
The system can identify a known speaker from their voice, transcribe what they say, send the speaker and transcript to an LLM, and generate a spoken response.
The Challenge
Most speech assistants focus on understanding what was said.
This project adds another layer: identifying who said it.
The system needed to continuously process microphone audio, detect speech, compare the speaker against known voice samples, transcribe recognized speech, generate a response, and convert that response back into audio.
What I Built
Speaker Identification
I implemented speaker recognition using Pyannote speaker embeddings.
Reference
.wav files are loaded from a local speaker directory and converted into embeddings. Incoming speech is processed in the same way, then compared against the stored embeddings using cosine similarity.A similarity threshold determines whether the speaker is recognized. Speech that does not match a known speaker is ignored.
Real-Time Speech Processing
Microphone input is captured using
sounddevice at a 16 kHz sample rate.The audio pipeline uses:
WebRTC voice activity detection
WebRTC noise suppression
3-second audio chunks
Overlapping audio windows
Background transcription processing
Only chunks containing detected voice activity are sent for further processing.
Speech-to-Text
Recognized speech is transcribed using Faster-Whisper.
The Whisper model is configured for CUDA when running the application, allowing GPU acceleration where available.
The transcription worker runs independently from microphone capture and places recognized queries into the main processing pipeline.
Speaker-Aware AI Responses
After identifying the speaker and transcribing their speech, the application sends both pieces of information to Google Gemini.
The response layer extracts:
Response language
Plain response text
A transcripted version of the response
This information determines how the assistant generates its spoken response.
Text-to-Speech
The project supports two voice-generation paths.
For configured languages, Coqui XTTS v2 can generate speech using a reference voice sample.
If a local voice is unavailable, the system falls back to Gemini's TTS model.
Generated WAV files are placed into an audio queue and played through the system's audio output.
Architecture
The application uses Python threads, queues, locks, and condition variables to coordinate microphone capture, transcription, and audio playback.
Technical Implementation
AI / ML
Pyannote Audio
Speaker embeddings
Faster-Whisper
PyTorch
Coqui XTTS v2
Generative AI
Google Gemini 2.5 Flash
Gemini TTS
Speaker-aware prompting
Language-aware responses
Audio
SoundDevice
WebRTC Audio Processing
Voice activity detection
Noise suppression
WAV processing
Audio buffering
Engineering
Python
Multithreading
Thread-safe queues
Temporary workspaces
Environment-based API configuration
Result
The result is a working voice interaction pipeline that connects speaker recognition, speech-to-text, conversational AI, and speech synthesis.
Instead of treating every voice input as anonymous, the system first determines whether the speaker matches a known voice and then carries that speaker identity into the conversational layer.
Like this project
Posted Aug 11, 2026
Built speaker-recognition, an open-source Python system that identifies and verifies speakers from audio using deep learning voice embeddings.
