Avatoon: Animated 3D Avatars with Real-Time Lip-Sync by Khaled AlamAvatoon: Animated 3D Avatars with Real-Time Lip-Sync by Khaled Alam

Avatoon: Animated 3D Avatars with Real-Time Lip-Sync

Khaled Alam

Completed work

Software Engineer

glTF Viewer

Computer Software

🧠 Avatoon

Avatoon is a lightweight React Three Fiber component for rendering animated 3D avatars with real-time viseme-driven lip-sync. It supports lifelike head movements, morph target control, and optional goal-based gestures (e.g., flexing, sleeping), making it ideal for voice assistants, interactive characters, or storytelling apps.

demo video

✨ Features

🎤 Real-time lip-sync using phoneme-viseme mapping

🧍 Subtle head motion animation while talking

🌐 GLTF model support via useGLTF

⚛️ Plug-and-play with React Three Fiber + Drei

🎯 Goal-based gestures like "Muscle" or "Sleep"

📦 Installation

npm install avatoon

🚀 Usage

import { Avatoon } from "avatoon";

export default App = () => {
    const visemeJson = {
        visemes: [{ 
            { "time": 0, "viseme": "X" },
            { "time": 1.3, "viseme": "A" },
            { "time": 1.367, "viseme": "C" },
         }],
        audio_base64: '',
    };

    return (
        <Avatoon
            glbUrl='https://raw.githubusercontent.com/khaledalam/avatoon/main/test/assets/placeholder-avatar.glb'
            goal={"Normal"}
            onRenderComplete={() => console.log("Render Completed!")}
            visemeJson={visemeJson}
        />
    );
}

To run example:

npm run example

🧩 API

`Avatoon(props)`

Prop Type Default Description glbUrl string (required) URL to the .glb avatar file (T1 or T2) goal string "Normal" Goal-based motion preset: "Muscle", "Sleep", etc. onRenderComplete () => void undefined Callback fired when avatar finishes rendering visemeJson VisemeData undefined JSON structure for syncing visemes with audio playback showPlayVoiceButton boolean false If true, renders a play/stop voice button in the scene

👤 Avatar Types

T1 (Static Face - Realistic)

Most photorealistic

No facial morphing

Lightweight

T2 (Blendshape Face - Expressive)

Separate eyeballs and mouth

Supports morph targets / ARKit visemes

Slightly less realistic but animatable

📘 Types

interface VisemeData {
  visemes: Array<{ time: number; viseme: string | null }>;
  audio_base64?: string;
}

🤝 Contribution

Pull requests are welcome! If you'd like to suggest improvements or fix issues, feel free to fork and submit.

🛡️ Author

Khaled Alam

📧 khaledalam.net@gmail.com 🌍 Website | LinkedIn | X(Twitter)

Like this project