Self-Healing System Development for MOS
Atul Ranjan
Verified
MOS Self-Healing Production System
1. Overview
The self-healing system is the recovery layer inside the Blonde Waterfall MOS backend (
mos-backend). MOS is a distributed content pipeline: a job fans out into shots, each shot into a chain of generative tasks (image_edit → frame_expand → video_generate → shot_reorder → upscale), which converge on an After Effects render on a Windows EC2 fleet, then delivery.Every hop in that pipeline is a place where work can be lost rather than failed. A Fargate Spot reclamation kills a worker mid-task. An SQS message dead-letters after three deliveries into dying workers. A delayed render-poll message evaporates. A provider returns a 524. A render server fills its C: drive with After Effects temp files and starts emitting black frames.
None of those produce an error anybody sees. They produce silence — a task stuck
running forever, a job pinned at processing, a customer's calendar entry that never becomes a video.The self-healing system exists to make silence impossible. It has three missions:
Detect — continuously scan for state that cannot make progress on its own, using per-step staleness TTLs rather than blanket timeouts.
Recover — re-enqueue, re-poll, restart, or clean up automatically, with bounded attempts and idempotent writes, so a transient infrastructure failure never becomes a permanent customer incident.
Escalate — when automatic recovery is exhausted or the failure is deterministic, classify it, stamp a
productionIncident on the job and calendar entry, and route it to a human with the failure class already named — instead of letting it age silently as "overdue editor work."It runs as two always-on ECS services (
mos-scheduler, mos-worker-nexrender-healthcheck), a drain hook inside every SQS worker, a read-only cross-system audit API, and a set of dry-run-first repair CLIs.2. Problem It Solves
Before this layer existed, recovery was a human with a terminal. Someone noticed a job hadn't moved, ran a
_diag-* script, read CloudWatch, and hand-retried. Two base_clip_batch jobs sat stuck at 176/180 and 178/180 tasks with nothing in the system capable of finishing them.Problem Solution this system provides Worker killed mid-task (Spot/OOM/SIGKILL) leaves the task
running forever Two-layer orphaned-task recovery: drain-reset on SIGTERM + scheduler reaper for hard kills. A worker death is recoverable, not permanent. SQS message dead-letters after maxReceiveCount — nothing will ever re-run it Reaper sends a brand-new message to the main queue (receive count reset to 1) without ever touching the DLQ. Retrying a live task would duplicate work TTLs are always larger than that queue's visibility timeout, so a message merely being redelivered is never reaped. Jobs stranded pending/waiting mid-chain leak a concurrency slot; enough of them drive availableSlots to 0 and the entire backlog stalls recoverStalledProcessingJobs fails them after a 3h no-activity TTL, before the concurrency count runs, so slots free up in the same tick. Render submitted, poll message lost — render task sits running while nexrender may already be done recoverStaleActiveRenders re-enqueues the poll first; only past the resubmit window does it ask nexrender directly, and it resubmits only when nexrender reports failure or no longer has the job. Retrying every failure forever burns credits on deterministic breakage An 8-class failure taxonomy splits retryable from operator_required; non-retryable classes are never auto-looped. Render server disk fills → silent black renders Health-check worker drives PowerShell over SSM: routine cleanup at <35 GB free, worker-stopping pressure cleanup at <15 GB. Render worker scheduled task dies on a Windows box Per-instance ASG sweep via SSM detects and restarts NexrenderWorker, publishing NexrenderWorkersRestarted. Each service healthy, but the systems disagree (credits vs. calendar vs. jobs vs. reviews vs. delivered media) Reliability control plane audits 6 named invariants across the whole customer workflow and reports contradictions. Failed production ages silently in the editor queue as overdue work Failures become a production_incident state, excluded from editor due/overdue metrics and surfaced in a separate Production Issues section.Who it serves
The pipeline itself — the primary consumer. Most recoveries complete with no human involved.
Backend engineers — get a classified failure with a correlation ID instead of a log dive.
Ops / support — get a bounded queue of
operator_required incidents, not an unbounded queue of stuck jobs.Editors — never see work that production can't deliver.
Customers — a Spot reclamation costs minutes, not a missing video.
3. Core Mechanisms & Workflows
Layer 1 — Drain-reset (graceful kills) · packages/workers/src/base.worker.ts
packages/workers/src/base.worker.tsEvery SQS worker tracks
receiptHandle → taskId for in-flight messages. On SIGTERM/SIGINT (deploy, scale-in, Spot's 2-minute notice), stop() resets each in-flight task running → queued before releasing the message with visibility 0, so the next worker's running write wins the race. Only re-runnable generative steps are reset; render/render-poll/deliver have their own recovery. Entirely best-effort — a failure is logged and never blocks shutdown. Paired with ECS task scale-in protection, toggled on when a worker goes busy and off when it goes idle.Layer 2 — Orphaned-task reaper (hard kills + dead-letters) · scheduler.worker.ts
scheduler.worker.tsreapStuckTasks() runs every 5-minute tick, in two passes:running pass — now - startedAt >= TTL(stepType).step TTL queue visibility
image_edit 12 min 5 min frame_expand 12 min 5 min video_generate 40 min 30 min shot_reorder 8 min 2 min upscale 15 min 5 minqueued pass — tasks that never got a startedAt and crossed their pickup TTL: the worker never claimed it.It rebuilds the exact worker message shape, re-derives the queue URL, resets the task to
queued, bumps reapCount, and stamps a fresh startedAt so the TTL clock restarts. Re-running is safe because outputs are content-addressed (processedAssets cache + chainNextTask ordering) — a reap either cache-hits or regenerates. Bounded at 3 re-enqueues, 300 scanned and 50 acted on per tick; past the limit the task is failed and cascaded to the job via cascadeShotFailure / cascadeInputChainFailure and recorded through jobErrorService. It also normalizes legacy kebab-case step names, so old tasks can't hide from every TTL.AE render recovery · dag-resolver.service.ts
dag-resolver.service.tsrecoverStrandedRenders — jobs flagged needsAeRerender where every shot is settled but the render task never got enqueued (lost the one-shot completion-callback race). Normalizes output-ready shots, completes their tasks, re-enqueues the render. Idempotent, no TTL.recoverStaleActiveRenders — render tasks running/pending with no poll activity for STALE_AE_RENDER_POLL_MS (3 min). Re-enqueues the poll message first. Elapsed time alone never kills or duplicates a render; a fresh submission happens only past the resubmit window and only when nexrender explicitly reports failure or has lost the job.recoverStalledProcessingJobs — the backstop for the gap nothing else covers. A job qualifies only when all hold: no task activity for STALLED_JOB_TTL_MS (3h), no running/queued task (that's the reaper's domain — never race it), and not needsAeRerender. It fails the job and its stranded tasks with a reason naming the exact shape (stranded pending/waiting vs. all tasks terminal but job never transitioned).Automatic production-failure recovery · production-failure-recovery.service.ts
production-failure-recovery.service.tsclassifyProductionFailure maps an error string onto 8 classes:class retryable
infrastructure (ENOSPC, no space left on device) ✅ template_transfer (S3 copy, ECONNRESET, ETIMEDOUT) ✅ provider_timeout (5xx, 429, quota, 524, poll-count exceeded) ✅ stranded_pipeline (worker never claimed it, non-transitioning job) ✅ template_unavailable (bundle has no .aep) ❌ provider_rejection (safety settings, sensitive content, insufficient balance, HTTP 400) ❌ media_or_template (After Effects error, could not read from source) ❌ unknown ❌Classification is context-aware: a
template_transfer failure against a template that is missing or unpublished is re-classified as template_unavailable — retrying it would never succeed.recoverRetryableFailedJobs then applies the guard rails before touching anything: the entry must be funded, not cancelled/delivered/approved, the review must exist and not be hidden, and the owner must still exist. Attempts are capped at 2 automatic recoveries with a 5-minute cooldown, and each attempt is reserved inside a Firestore transaction so two ticks can't double-retry the same job. Only the four transient classes may reset an exhausted retry budget.retryFailedProductionJob does the repair in one atomic batch: failed/dead tasks reset to pending — or waiting when the shot has unsatisfied dependencies — errors cleared, shot statuses restored dependency-aware, the job returned to pending, and the linked calendar entry and review re-pointed to queued. Whatever happens, a productionIncident is stamped on both job and entry with failureClass, attemptCount, detectedAt, lastAttemptAt and a status of retry_pending / retrying / exhausted / operator_required.Infrastructure self-healing · nexrender-healthcheck.worker.ts
nexrender-healthcheck.worker.tsEvery 5 minutes: enumerate
InService ASG instances, check each in parallel over SSM PowerShell for NexrenderWorker task state, node process, and free disk.Restart the scheduled task when it's down.
Run disk cleanup on the per-render
Windows\Temp\nexrender child directories (cleaning the parent misses them — it's continuously touched). Routine pass at 120-min staleness; under pressure, stop the worker, terminate aerender.exe/AfterFX.exe, and clean at 30-min staleness.Prune superseded nexrender attempts, but never one in
picked/running — nexrender's DELETE removes the record without cancelling the job.Publish
NexrenderWorkersHealthy/Unhealthy/Restarted, NexrenderDiskFreeGbMinimum, NexrenderStuckJobs, ProductionHealthy, HealthCheckHeartbeat to CloudWatch.Persist
infraStatus/production-health with consecutiveFailures, statusChangedAt, lastHealthyAt — and fire a Slack alert only on status transition, including the recovery ("Production recovered: …").Even total discovery failure self-reports: a missing ASG, a missing config, or zero instances writes a
down health doc with a customer-legible message ("No After Effects workers are online. Jobs remain queued.").Workflow-level sweeps
The same tick also runs
autoApproveDueTrialHeroes, materializeDueCalendarEntries, materializeDueLocalTours, sweepStuckDeferredCalendarGen(10 min) — for listings whose calendar generation was deferred on photo classification but whose post-batch trigger never fired — and the flag-gated sweepUnfundedEntriesToBlocked. Separately, GET /api/listings/:id/strategy self-heals: if the strategy doc is missing but the listing has a calendar, it returns regenerating: true and rebuilds in the background.Reliability control plane (detect-only) · reliability-audit.service.ts
reliability-audit.service.tsA customer incident exists whenever subsystems disagree, even when each is individually healthy. The audit never re-derives access or credits from frontend labels — it reads canonical authorities (
getWorkspaceState, canonical billing usage, calendar, jobs, reviews, delivered media) and checks 6 invariants:User and active-subscription status agree.
Used + reserved + available credits equal the active plan limit.
Unfunded calendar work does not enter the editor queue or production.
Active production and revisions do not stall beyond 72 hours.
Delivered entries always have downloadable media.
Actionable editor reviews retain a valid calendar relationship or their own rendered media.
Findings are typed (
SUBSCRIPTION_STATUS_DIVERGENCE, CREDIT_ALLOCATION_DIVERGENCE, UNFUNDED_WORK_IN_EDITOR_QUEUE, DELIVERED_WITHOUT_MEDIA, STALLED_PRODUCTION, ORPHANED_EDITOR_REVIEW, …) with critical/warning/info severity. Surfaced via GET /api/admin/reliability/incidents, GET /api/admin/reliability/users/:uid, scripts/audit-production-reliability.ts, and /internal/admin/reliability. All read-only by design — repairs are added one incident class at a time, only after false-positive rates are understood.Health roll-up
GET /api/admin/health composes ALB, scaling, render-server, and alarm sub-checks plus /infra-status, and returns worst-of-all as the overall status. Each evaluate* is a pure reducer, so the rules are unit-tested without the AWS SDK. Tuning lives in one HEALTH_THRESHOLDS object: queue depth 100 → yellow / 1000 → red, heartbeat stale at 5 min, a service stuck desired != running for 10 min → red.4. Technologies Used
Language & runtime
TypeScript 5 (strict) on Node.js 20 LTS; pnpm workspace monorepo (
api / workers / shared)Single multi-stage Docker image; ECS overrides
CMD per serviceOrchestration & compute
AWS ECS Fargate + Fargate Spot — worker fleet;
mos-scheduler pinned at exactly 1 taskECS task scale-in protection — toggled per worker on busy/idle transitions
EC2 Auto Scaling Group (Windows Server 2022) — After Effects render fleet, ASG 1–8
AWS SSM Send Command — PowerShell remote execution for health checks, worker restart, and disk recovery (no SSH/RDP)
Messaging & state
Amazon SQS — Standard queues with per-queue visibility timeouts and DLQs; recovery deliberately re-enqueues to the main queue
Firestore (Admin SDK) — jobs, tasks, calendar entries, reviews,
jobErrorEvents, infraStatus/production-health; transactions for recovery-attempt leases, batched writes for atomic repairObservability & alerting
CloudWatch Metrics —
MOS/Nexrender and MOS/Errors namespacesCloudWatch Logs — per-service log groups, deep-linked from the admin UI
Slack incoming webhook — transition-only alerts (degradation and recovery)
Winston structured logging with
AsyncLocalStorage correlation-ID propagation; SQS publishers stamp MessageAttributes, workers re-bind in base.worker.tsError model
Typed
AppError hierarchy (ExternalServiceError → VeoError / GeminiError / NexrenderError / …, plus PipelineStateError, ConflictError, CreditError), with classifyUnknown() as the single wrapping chokepointjobErrorService.record() as the single failure-recording chokepoint — classify → severity → jobErrorEvents → job counter → CloudWatch → deduped notification, wrapped so it can never break the callerAPI & ops surfaces
Express admin API with capability-gated RBAC (
view_infra_status, view_users)Dry-run-first CLIs:
audit-production-reliability.ts (read-only), repair-editor-production-incidents.ts (--apply opt-in), reconcile-credit-lifecycle.ts, reconcile-editor-queue-lifecycle.ts, audit-editor-queue-health.tsTesting
Jest unit suites:
task-reaper, production-failure-recovery, dag-resolver-stalled-jobs, nexrender-stale-recovery, nexrender-disk-recovery, reliability-audit.service, funding-sweep, health-aggregate-rollupLive e2e case runner (
tests/e2e-live) — UC13 covers strategy self-heal against a real deployment5. Architecture at a Glance
Where the code lives
packages/workers/src/scheduler.worker.ts — the tick; every pass in its own try/catch, all before the production-pause gate, because recovery must run even while new dispatch is pausedpackages/workers/src/base.worker.ts — drain-reset, heartbeat, task protectionpackages/workers/src/utils/task-reaper.ts — TTL table + pure isTaskStale / step-name normalizationpackages/workers/src/nexrender-healthcheck.worker.ts + utils/nexrender-disk-recovery.tspackages/shared/src/services/dag-resolver.service.ts — the three render/job sweepspackages/shared/src/services/production-failure-recovery.service.ts — taxonomy + bounded retrypackages/api/src/services/reliability-audit.service.ts, health-aggregate.service.tsDocs:
docs/observability/orphan-task-recovery.md, docs/production-reliability-control-plane.md, docs/editor-production-incident-recovery.md6. Failure Taxonomy & the Escalation Ladder
The design principle is that a failure should climb exactly as far as it needs to, and no further:
Graceful shutdown → drain-reset, seconds. Invisible.
Hard kill / DLQ → reaper re-enqueue, ≤ one step TTL. Bounded at 3.
Transient production failure → classified retry, ≤ 2 attempts, 5-min cooldown, transactional lease.
Deterministic failure → no retry loop.
productionIncident with status: operator_required and the exact class.Unrecoverable stall → job failed with a reason naming the shape, concurrency slot released.
Cross-system contradiction → reliability finding, typed and severity-ranked, surfaced to admins.
Infrastructure down → health doc + Slack transition alert + customer-legible message.
The rules that keep it from doing harm are as important as the recoveries themselves:
Never race yourself. Each sweep explicitly excludes the states another sweep owns (
hasRunningOrQueued → reaper's; needsAeRerender → stranded-render's; render/deliver → their own).Time alone never destroys work. The stale-render sweep asks nexrender before replacing an attempt; TTLs sit above visibility timeouts so live redelivery is never mistaken for death.
Every automatic action is idempotent and audited, because outputs are content-addressed and every write stamps attempt counts and timestamps.
No generic "repair account" button. Per the repair policy, every repair must name one violated invariant, identify the authoritative source, support dry-run, be idempotent, write an audit log, refuse ambiguous records, and ship with a regression test for the original customer incident.
7. Success Criteria
A worker death is recoverable, not permanent — Spot reclamation, OOM, SIGKILL, and dead-lettering all have a defined path back into the pipeline.
No task stays orphaned past its step TTL — 8–40 minutes worst case, versus indefinitely before.
The reaper is a no-op on a healthy fleet — defaults reproduce existing behavior for any task inside its TTL.
No job holds a concurrency slot for more than 3 hours without progress, so
availableSlots can never be starved to 0 by stalled work.Automatic retries are bounded and classified — ≤2 attempts on 4 transient classes only; deterministic failures are never looped.
No failed production record silently ages as editor-overdue work — it becomes a typed
production_incident, excluded from due/overdue metrics.Recovery runs even while dispatch is paused — incident response never blocks incident recovery.
Operators get a classification, not a log dive — failure class, attempt history, and correlation ID are on the job document before anyone opens CloudWatch.
Detection ships before repair — the reliability control plane is read-only until each repair policy is confirmed with its billing and production owner.
Like this project
Posted Sep 20, 2026
Detects silently stalled pipeline work, orphaned tasks, lost renders, stalled jobs and recovers it with bounded, idempotent retries before escalating.
Likes
0
Views
0
Clients
Blonde Waterfall
