Avatoon: Animated 3D Avatars with Real-Time Lip-Sync by Khaled AlamAvatoon: Animated 3D Avatars with Real-Time Lip-Sync by Khaled Alam

Avatoon: Animated 3D Avatars with Real-Time Lip-Sync

Portfolio project
Khaled Alam
View completed work

🧠 Avatoon

Avatoon is a lightweight React Three Fiber component for rendering animated 3D avatars with real-time viseme-driven lip-sync. It supports lifelike head movements, morph target control, and optional goal-based gestures (e.g., flexing, sleeping), making it ideal for voice assistants, interactive characters, or storytelling apps.

✨ Features

🎤 Real-time lip-sync using phoneme-viseme mapping
🧍 Subtle head motion animation while talking
🌐 GLTF model support via useGLTF
⚛️ Plug-and-play with React Three Fiber + Drei
🎯 Goal-based gestures like "Muscle" or "Sleep"

📦 Installation

npm install avatoon

🚀 Usage

import { Avatoon } from "avatoon";

export default App = () => {
    const visemeJson = {
        visemes: [{ 
            { "time": 0, "viseme": "X" },
            { "time": 1.3, "viseme": "A" },
            { "time": 1.367, "viseme": "C" },
         }],
        audio_base64: '',
    };

    return (
        <Avatoon
            glbUrl='https://raw.githubusercontent.com/khaledalam/avatoon/main/test/assets/placeholder-avatar.glb'
            goal={"Normal"}
            onRenderComplete={() => console.log("Render Completed!")}
            visemeJson={visemeJson}
        />
    );
}

To run example:

npm run example

🧩 API

Avatoon(props)

Prop Type Default Description glbUrl string (required) URL to the .glb avatar file (T1 or T2) goal string "Normal" Goal-based motion preset: "Muscle", "Sleep", etc. onRenderComplete () => void undefined Callback fired when avatar finishes rendering visemeJson VisemeData undefined JSON structure for syncing visemes with audio playback showPlayVoiceButton boolean false If true, renders a play/stop voice button in the scene

👤 Avatar Types

T1 (Static Face - Realistic)
Most photorealistic
No facial morphing
Lightweight
T2 (Blendshape Face - Expressive)
Separate eyeballs and mouth
Supports morph targets / ARKit visemes
Slightly less realistic but animatable

📘 Types

interface VisemeData {
  visemes: Array<{ time: number; viseme: string | null }>;
  audio_base64?: string;
}

🤝 Contribution

Pull requests are welcome! If you'd like to suggest improvements or fix issues, feel free to fork and submit.

🛡️ Author

Khaled Alam

0 likes · 4 views · Jul 14, 2025

Developed Avatoon, a React component for animated 3D avatars with real-time lip-sync.

Tags