AI-Powered Video-to-Recipe Backend Development
Jongeum Kim
Oishii — AI-Powered Recipe Extraction from Cooking Videos
A mobile application that transforms YouTube cooking videos into structured, usable recipes through multimodal AI analysis.
Overview
Cooking videos contain great recipes, but the useful information is scattered across narration, visual demonstrations, captions, ingredient labels, and on-screen text.
Oishii transforms that unstructured video content into structured recipe data containing ingredients, cooking steps, timings, temperatures, and practical tips.
The application is being developed by a four-person developer team. I own the backend architecture, AI integration, database design, operational analytics, and deployment.
My Role
Backend & AI Engineer
I designed and implemented:
FastAPI backend architecture
WebSocket communication
Google Gemini video analysis
Prompt design and structured AI output
Redis-based quota control
PostgreSQL operational analytics
AI API cost tracking
Error classification and request monitoring
Backend deployment
Tech Stack
Python · FastAPI · WebSocket · Google Gemini · Redis · Lua · PostgreSQL · SQLAlchemy · React Native · Render
The Challenge
Turning a YouTube cooking video into a recipe sounds simple:
YouTube Video → Gemini → Recipe
But integrating a long-running AI service into a real application introduced a different set of engineering problems.
A single Gemini analysis can take 20–50 seconds.
Requests can fail because of upstream rate limits or temporary service issues.
Every request has a monetary cost.
Multiple simultaneous requests can race against usage quotas.
And the mobile client may disconnect while processing is underway.
The backend therefore needed to do much more than simply call an AI API.
What I Built
I designed the backend as a lightweight AI orchestration layer between the React Native application and Google Gemini.
The mobile application sends a YouTube video for analysis through a WebSocket connection.
The backend then:
Atomically reserves usage quota in Redis.
Sends the video to Gemini for multimodal analysis.
Receives a structured recipe containing ingredients, steps, timings, temperatures, and tips.
Records latency, token usage, cost, and request status in PostgreSQL.
Commits or rolls back the quota reservation depending on the result.
Returns the structured recipe or an explicit error to the mobile client.
This keeps AI-specific operational concerns out of the mobile application while avoiding unnecessary backend complexity.
Handling Long-Running AI Requests
Video understanding behaves very differently from a typical CRUD request.
Because Gemini analysis can take tens of seconds, I chose WebSocket communication for the analysis endpoint.
The connection remains open while the backend communicates with Gemini and eventually returns either the completed recipe or an explicit failure state.
The protocol can also support richer status events in the future without requiring a redesign of the API contract.
Race-Safe Quota Control with Redis
AI requests cost money, so usage control is part of the backend architecture rather than an afterthought.
Oishii uses both per-client and global quotas.
A simple "check quota, then decrement" implementation creates a race condition when multiple requests arrive concurrently.
I solved this using Redis Lua scripts that perform quota validation and reservation atomically.
The lifecycle becomes:
Reserve → Execute Gemini Request → Commit or Rollback
If Gemini succeeds, the reservation is committed.
If Gemini fails, the reservation is rolled back so the user does not lose quota because of an upstream failure.
This provides correct quota accounting even under concurrent requests.
AI Cost & Operational Analytics
For every Gemini request, the backend records operational data including:
Input and output tokens
Cached and reasoning tokens
Request latency
Estimated API cost
HTTP status
Error category
Request timestamp
Gemini usage metadata is combined with model-specific pricing rules to estimate the cost of each analysis.
The calculated cost is persisted when the request occurs, allowing the system to answer questions such as:
How much does a typical recipe analysis cost?
How often does Gemini fail?
Which errors occur most frequently?
How long does video analysis normally take?
A lightweight administrative interface exposes these metrics for monitoring.
Designing Failure Ownership
I initially considered automatic backend retries for failed Gemini requests.
But that seemingly simple feature introduced much larger architectural questions:
Who owns a pending job if the mobile app closes?
Where should its result be stored?
How should the client reconnect and retrieve it?
How long should the backend retain unfinished jobs?
Instead of introducing durable queues and user-specific job state prematurely, I chose a simpler ownership boundary:
The backend owns one request attempt.
The mobile client owns the overall user interaction and retry decision.
This keeps the architecture appropriate for the project's current scale while leaving durable background processing as a future extension if product requirements justify it.
Choosing the Right Storage for Each Type of State
The backend deliberately separates temporary and durable state.
Redis handles:
Usage quotas
Reservation markers
TTL-based state
PostgreSQL stores:
Analysis history
Gemini request history
Token usage
API cost
Latency
Error information
This keeps high-frequency temporary state separate from durable operational history.
Result
Oishii evolved from a simple AI integration into a practical backend for operating a long-running, paid external AI service.
The current architecture provides:
Structured multimodal recipe extraction
Long-running WebSocket communication
Atomic quota enforcement
Safe quota rollback on upstream failures
AI token and cost tracking
Structured error classification
Operational analytics
Lightweight managed deployment
Most importantly, the project reinforced a broader engineering principle:
Integrating AI into a real product involves much more than obtaining a valid model response.
Failure ownership, concurrency, cost control, temporary state, persistence, and client lifecycle all become part of the system design.
Like this project
Posted Sep 20, 2026
Designed and implemented AI backend for transforming videos into recipe data.
Likes
0
Views
0
Timeline
Jul 13, 2026 - Ongoing
