AI Agents Multimodal Video Pipeline by Ugo ChukwuAI Agents Multimodal Video Pipeline by Ugo Chukwu

AI Agents Multimodal Video Pipeline

Ugo Chukwu

Completed work

AI Agent Engineer

AI Developer

AI Engineer

Python

React

TypeScript

Advertising

Multimodal Video Pipeline — Designing Video Generation as an Operable System

Role: Founding Lead Engineer Company context: Peerless Public reconstruction: Multimodal Video Pipeline by Etherlabs Repository: https://github.com/Etherlabs-dev/multimodal-video-pipeline Delivery: Tier 1 runnable repository; local and CI verified

This is a fresh, sanitized reconstruction of video-production systems I designed and built during my work for Peerless. The public repository uses new code, synthetic media, deterministic adapters, and no proprietary prompts, production credentials, client identities, paid provider calls, live storage, or advertising actions.

The outcome

I designed and built a production-control product that turns an approved storyboard into a durable tenant-scoped scene graph, routes each scene using dated capability evidence, records every attempt, reconciles signed callbacks, supports one bounded compatible fallback, assembles four exact output manifests, and locks reviewer approval to a version and checksum.

The product demonstrates the less visible engineering that makes generative video operable: asynchronous state, multimodal provenance, recovery, idempotency, exact assembly, and safe delivery.

Northstar Athletics, every storyboard, asset, job, timing, failure, clip, and output shown here is synthetic.

The problem: video generation is not one model call

A campaign can contain hooks, shared body scenes, product details, proof, start plates, motion references, captions, and multiple output variants. Different scenes may need different routes. Jobs can fail after minutes. Callbacks can be duplicated, altered, or lost. A plausible output may still contain the wrong segment version.

The core product problem was therefore not “how do I call a video model?” It was “how do I make an asynchronous multimodal production workflow inspectable, recoverable, and safe to approve?”

My role

As Founding Lead Engineer at Peerless CMO, I owned architecture, prioritization, end-to-end delivery, testing, production iteration, client rollout, and later technical direction. I was the sole engineer through July 2026 before the engineering team expanded.

For the video system, that included scene orchestration, provider boundaries, asset resolution, callback and polling behavior, bounded recovery, finalization, composition, review workflow, security controls, and operational failure states.

System architecture

The public reconstruction combines a responsive React/Vite/TypeScript control room with FastAPI, Pydantic, and versioned SQLite state. Signed synthetic sessions establish tenant and role. Deterministic media adapters exercise the complete workflow without credentials or paid calls.

Every scene retains input asset roles, route, model profile, attempts, state, optimistic version, failure reason, and recovery history. Only terminal scene graphs can be assembled.

Storyboard to durable scene graph

The approved storyboard expands into ordered hook and body scenes. Required assets are assigned semantic roles such as start plate, product image, motion reference, or source edit.

Routing uses capability and compatibility metadata rather than a hard-coded favorite provider. The model registry separates three facts:

dated capability evidence;

whether the profile is enabled and verified; and

whether the public repository has an adapter.

Verified catalog entries do not imply provider traffic. Seedance 2.5 remains an explicitly disabled watchlist entry; Sora 2 is unavailable and disabled. The public adapters are deterministic local implementations.

Signed callbacks and reconciliation

Provider-style callbacks are authenticated over the raw body. Receipt identities are idempotent. If the same receipt arrives with different content, the service rejects the replay.

Completion is not inferred from a callback alone. Durable attempt and scene state govern reconciliation and finalization.

Failure is a product state

The control room shows a failed scene, its route decision, required assets, attempt history, quality rules, and downstream output impact. Recovery is explicit rather than silently automatic.

One compatible fallback may preserve the start plate, duration, tenant scope, caption, and storyboard lineage. The original failed attempt remains immutable.

This prevents uncontrolled retries, accidental route drift, and loss of failure evidence.

Exact assembly and review

Finalization waits for all required scenes to reach terminal state. It emits four exact hook manifests once, preserving the shared body sequence. Each output receives deterministic composition metadata and a checksum.

The review surface plays a real 31-second synthetic video and exposes clip lineage, captions-safe-area checks, resolution, duration, segment count, and checksum.

A production reviewer approves that exact artifact. Packaging is idempotent, local-only, and records published: false.

Tenant and role boundaries

Producers inspect scenes, advance deterministic runs, reconcile callbacks, and choose bounded recovery.

Reviewers approve exact outputs and create local delivery packages.

Every read and mutation is tenant-filtered.

Stale versions fail rather than overwrite current state.

Verification

The August 12, 2026 publication gate includes:

7 passing web interaction and focus tests;

14 passing API and security tests;

13/13 deterministic evaluations;

two consecutive reset/migration runs;

passing frontend and Python lint, production build, dependency audits, and secret scan;

local API health and actual 31-second media playback;

desktop/mobile browser acceptance with zero final console warnings or errors; and

GitHub Actions run 31537140989, passing every gate at commit 875c80b8aad75e384f5de0c98919089e3f70406c.

These results prove deterministic workflow behavior—not provider quality, production volume, campaign performance, uptime, or hosted availability.

The 13 evaluation invariants

The suite checks catalog states, tenant-scoped scene expansion, route provenance, cross-tenant hiding, one compatible fallback, terminal-state finalization, four exact hook manifests, idempotent finalization, checksum-locked approval, local-only packaging, and callback receipt idempotency.

They do not score visual quality or claim that any provider was called.

Model registry as an evidence system

The model catalog is designed to resist a common failure in fast-moving AI products: treating marketing claims, implemented adapters, and observed usage as the same thing.

Each profile records dated source evidence and a public status. Verified and routable means the deterministic system has enough supported capability metadata to select the profile. Watchlist means the product exists in the catalog but remains disabled. Unavailable means the route must not be selected. None of those states independently proves a paid API request.

This gives operators an explainable answer to “why this route?” while keeping claims falsifiable as providers evolve.

Operational state and recovery design

The scene timeline makes dependencies visible. A failed shared body scene blocks all four hook variants; a failed isolated hook affects a narrower output set. The recovery dialog therefore shows output impact before the producer acts.

The system preserves the original failed attempt, then creates a separate fallback attempt. That lineage is important for debugging, cost review, quality comparison, and later provider-policy changes. Replacing the failed record with the successful retry would make the run look cleaner and make the system less explainable.

Security and concurrency boundaries

The public API tests forged and missing sessions, cross-tenant access, producer/reviewer violations, stale scene versions, callback signature failure, altered receipt replay, premature finalization, and checksum-locked approval.

Optimistic versions prevent two operators from recovering or approving against different states. HMAC validation prevents unauthenticated callback mutation. Receipt hashes prevent an attacker or faulty provider from reusing an accepted identifier for different content.

The local delivery package is also a security boundary: it proves the manifest and checksum workflow without placing credentials or external side effects in the public repository.

Why the repository is intentionally unhosted

The essential engineering claims are reproducible locally and in CI. Hosting would require identity replacement, managed tenant storage, object storage, TLS, restricted CORS, external signing secrets, provider billing, and operational monitoring. Creating a thin hosted shell would not prove those production concerns.

The Tier 1 repository instead makes its deterministic orchestration, synthetic playback, failures, recovery, assembly, review, and package artifacts inspectable without ongoing infrastructure cost.

Historical engineering context

A dated private-source review confirmed an active video subsystem spanning storyboard bridging, multimodal asset preparation, explainable routing, asynchronous dispatch, callback and polling completion, bounded recovery, hook composition, captions, durable outputs, and production QA.

Reachable provider integrations prove architecture, not current live traffic or volume. Proprietary prompts, private assets, internal names, provider configuration, and client results remain excluded.

Mobile production hierarchy

Mobile reorders the workflow into Runs, Scenes, and Recovery. It preserves inspection and core action without squeezing the desktop control room into horizontal scroll.

What I learned

The output video is only the visible edge of the system. Operability comes from the records around it: scene identity, input roles, routing rationale, attempt history, callback receipts, failure state, recovery bounds, exact manifests, checksums, and reviewer authority.

Generative video becomes trustworthy when the system can answer not only “what did we create?” but also “from which inputs, through which route, after which failures, assembled from which exact clips, and approved by whom?”