SherpaNova: a real time AI avatar product

SherpaNova is an AI cooking companion. A real time avatar appears on screen and talks you through a recipe while you cook, holding its place across steps and waiting when you need it to.

Most AI products are judged on whether the answer is correct. In a live conversation, correctness is not enough. An answer that arrives two seconds late is a wrong answer, because the person has already moved on, repeated themselves, or given up on you.

That single constraint shapes the entire architecture. Latency budgets stop being something you measure at the end and become the thing every decision is made against.

The avatar guides users through a recipe with live video and voice, navigating chapter by chapter through the steps. It keeps conversational state so it knows where in the recipe you are, and it handles interruption, which is what people actually do when their hands are covered in flour and they need something repeated.

The hardest part was not the model. Getting a real time avatar SDK to render properly inside a custom application, driven by application state rather than a demo page, meant hitting failure modes that were not documented anywhere.

What worked was stripping everything back to the smallest thing that would render at all, getting that stable, then layering one piece at a time: video, then voice, then the chapter state driving the conversation. Two weeks of that.

Next.js and TypeScript, Simli for the real time avatar, WebRTC for the live media transport, AI voice for the conversational layer.