An LLM coach in production: LangGraph, evals, cost control

A demo answers one question well. A product answers the same question ten thousand times a day, for users who are paying, on inputs nobody anticipated, at a token bill that has to stay smaller than the subscription.

This is the AI layer of F/AI, an AI fitness trainer with 340,000+ registered users. Three LLM features, all of them shipped and all of them load-bearing.

Plan generation and editing. The model does not write a workout from a blank prompt. It works over the user's own history — what they lifted, what they skipped, what they said hurt — and produces a plan that can be edited in conversation. Retrieval over the user's own data, not over a document dump.

Body scan. A computer-vision pass over a single photo returns body type, estimated composition and measurements, which then feed back into the plan.

Chat coach. Multimodal: photograph a machine you have never seen in a gym and it tells you what it is, what it trains and which exercises to do on it. The screenshot on this project is that feature.

Over 176,000 messages have gone to the AI coach — the counter is public at fitgpt.pro/stats.

LangGraph, not a prompt in a loop. Explicit graphs with named nodes, so a failure has a location. Retries and fallbacks live in the graph, not scattered through the calling code.

Evaluation from the start. A reference set of real inputs, scored on every change to a prompt or a model. Without it you cannot tell a prompt improvement from a regression — you only find out from the reviews, weeks later.

Observability with Langfuse. Every trace is inspectable: what was retrieved, what was sent, what came back, what it cost, how long it took. When a user says "it gave me nonsense on Tuesday", Tuesday is findable.

Cost control. Model routing by task, aggressive caching of what does not need to be regenerated, and per-user budget ceilings. Token spend is a line item in a subscription business, not a rounding error.

Failure handling. Timeouts, malformed output, refusals and provider outages all have a defined behaviour that is not a crash and not a blank screen.

The order of work is almost always the same: make the output measurable, make the failures visible, make the cost predictable — and only then make it smarter. Most AI features that "work in the demo and fall apart with real users" are missing the first three, not the fourth.