Real Time AI Video Avatar Chat Widget

I handled building the frontend of a real time AI powered video chat using HeyGen Avatar streaming tech with website integration and multi language support. Check it out here: https://www.digital-ateam.de/vertriebsbutler/.

Widget integration: JS snippet enables the app to be integrated into any website with full UI and avatar customisation.

State management: implemented coordinated API calls across multiple services. Used React Query for managing and synchronising all the data in the app.

Integrated with WebSocket (GraphQL Subscriptions) for real time updates, and HTTP/HTTPS for queries.

LiveKit integration: video streaming from HeyGen with connection management.

Audio processing pipeline: Custom Web Audio API implementation with silence detection, chunk management and multilingual support for speech to text input.

Silence detection: RMS-based audio analysis with configurable thresholds and timing

Real-time transcription: Streaming audio chunks to Amazon Transcribe with base64 encoding optimisation.

Voice activity detection: State management preventing conflicts during avatar speech

Lazy loading architecture: Progressive component initialisation with short load times

Mobile optimisation: platform-specific UI adaptations

Smart UI transitions: Framer Motion animations with spring physics

Context-aware controls: Dynamic button states based on avatar speaking status and connection state

Multi-language support: support for any language based on client needs with dynamic locale switching

Browser language detection: Automatic language selection based on user preferences

Real-time language switching: No page reload required for language changes

Comprehensive error boundaries: Graceful handling of API failures, network issues, and browser constraints

Automatic reconnection: Retry logic with exponential backoff for session recovery

Fallback mechanisms: Degradation from video → audio → text-only modes

Timeline: 5-month development cycle with iterative releases

Responsibility: Full frontend ownership including architecture decisions and implementation

Separation of concerns: Clean abstraction layers for API, state, and UI logic

Reusable patterns: Custom hooks for audio recording, avatar management, and session handling