Event-Driven Serverless Video Transcoding Pipeline

View completed worktcoder-web.cloudflare-c49.workers.dev

TCoder - Event Driven Serverless Transcoding Pipeline

Implementation of an event-driven, serverless video transcoding pipeline. Composes Cloudflare Workers for orchestration, R2 for object storage, Upstash Redis for state management, and Fly.io Machines for compute. Built with TypeScript, Hono, and Effect.

This is a personal side project built to explore architectural tradeoffs in distributed orchestration, backpressure handling, and worker lifecycle management. It is not intended for production use.

Demo Site: https://tcoder-web.cloudflare-c49.workers.dev/

Blog Post: I tried to understand how large-scale video pipelines work, so I built a serverless transcoding system

Architecture

Three-layer separation: control plane, state store, and compute plane. Each layer has distinct responsibilities and failure boundaries.

Layer Component Responsibility Control Plane Cloudflare Worker API endpoints, admission control, machine lifecycle management State Store Upstash Redis Job queue, machine pool tracking, job status Compute Plane Fly.io Machines FFmpeg transcoding, R2 I/O operations

System Design

Event-Driven Design

R2 object creation events drive the pipeline. Uploads trigger queue messages, which update Redis state and trigger machine provisioning decisions.

Upload: Client requests presigned URL from Worker API. Uploads video directly to R2 input bucket.

Event Notification: R2 emits object-created event to Cloudflare Queue (tcoder-events).

Queue Processing: Worker queue handler receives batch, extracts job ID from object key, updates Redis job status to pending, enqueues job in sorted set.

Admission Control: Worker checks Redis machine pool for available capacity. If under limit, attempts to start stopped machine or spawn new Fly.io Machine.

Job Processing: Fly.io Machine polls Redis for jobs using ZPOPMIN. On job assignment, downloads input from R2, runs FFmpeg transcoding, uploads outputs to R2 output bucket.

Completion: Machine sends webhook to Worker API with job results. Worker updates Redis job status. Client polls status endpoint or receives webhook callback.

Machine Lifecycle: Idle machines remain in pool. Scheduled cron job stops machines idle beyond threshold and recovers stuck uploading jobs. Stopped machines can be restarted for new jobs.

Quick Start

Prerequisites

Bun runtime

Cloudflare account (Workers, R2, Queues)

Upstash account (Redis)

Fly.io account

Setup

Install dependencies:

bun install

Create R2 buckets and queue:

bunx wrangler r2 bucket create tcoder-input && \
bunx wrangler r2 bucket create tcoder-output && \
bunx wrangler queues create tcoder-events && \
bunx wrangler r2 bucket notification create tcoder-input --event-type object-create --queue tcoder-events

Set Cloudflare Worker secrets:

bunx wrangler secret bulk .env

Required secrets (see env.local.example):

UPSTASH_REDIS_REST_URL, UPSTASH_REDIS_REST_TOKEN

FLY_API_TOKEN, FLY_APP_NAME, FLY_REGION

R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY

R2_INPUT_BUCKET_NAME, R2_OUTPUT_BUCKET_NAME

WEBHOOK_BASE_URL

Initialize Fly.io app:

bun run fly:first-launch

Set Fly secrets (see fly/README.md).

Deploy:

bun run deploy        # Cloudflare Worker
bun run fly:deploy    # Fly.io image

Project Structure

tcoder/
├── src/
│   ├── index.ts              # Worker entry, queue + cron handlers
│   ├── api/                  # Hono API routes
│   ├── r2/                   # Presigned URLs, event handling
│   ├── redis/                # Upstash client and schema
│   └── orchestration/        # Admission, spawner, machine pool
├── fly/
│   ├── ffmpeg-worker/        # Fly Machine worker code
│   └── Dockerfile            # Worker container
├── packages/
│   └── tcoder-client/        # TypeScript SDK
└── design/
    └── architecture/         # PlantUML diagrams

Documentation

API Usage Guide - CURL examples and API reference

Local Development - Docker Compose setup

Fly.io Workers - Worker details, debugging

TypeScript SDK - Client library

Scripts

bun run dev              # Local development
bun run deploy           # Deploy Cloudflare Worker
bun run fly:deploy       # Deploy Fly.io image
bun run fly:logs         # View Fly.io logs
bun run test             # Run tests

Related Projects

TCoder Middleware: Bunny.net edge middleware that protects /premium/ paths by verifying JWT tokens.

0 likes · 5 views · Jan 25, 2026

Implemented an event-driven, serverless transcoding pipeline with Cloudflare Workers and Fly.io.