Example: Distributed video transcoding

Muhkoo Theater is a personal, end-to-end-encrypted media library: you add a video, it gets transcoded to adaptive HLS, and it streams back to any of your devices. The interesting part is that there is no transcode backend. Your browser and any spare machine running a native worker sign in as the same user, join the same Space, and drain the same client.db queue as peers. Add a machine, add throughput; close every tab, the queue survives and resumes.

This page is the worked example behind the reusable Distributed background jobs pattern. Read that for the step-by-step build; read this for how the pieces fit, the job state machine, and how a 30 GB 4K file moves through the system without ever sitting in memory.

Note

Everything here is owner-scoped: every worker is a device signed in as the same user. It spreads your work across your machines, not a public compute market — so contention is low and trust is implicit.

Architecture

Four surfaces you already have, wired into a self-organizing pool. Nothing is a dedicated “jobs” service.

        add video                                              stream HLS
           │                                                       ▲
           ▼                                                       │
 ┌───────────────────┐                                 ┌───────────────────────┐
 │  Browser (peer)   │        both are just            │ Native worker (peer)  │
 │  ffmpeg.wasm      │        signed-in devices        │ system ffmpeg (HW)    │
 │  score ≈ cores    │◀──────────────────────────────▶│ score = cores × 30    │
 └─────────┬─────────┘                                 └───────────┬───────────┘
           │  claim + heartbeat lease   │   claim + heartbeat lease │
           │                            ▼                           │
           │            ┌──────────────────────────────┐           │
           ├───────────▶│   jobs table  (client.db)    │◀──────────┤   durable
           │            │  SOURCE OF TRUTH · owner-scoped│           │   floor
           │            │  queued·processing·done·error │           │
           │            └──────────────────────────────┘           │
           │            ┌──────────────────────────────┐           │
           ├───────────▶│  per-user Space (client.space)│◀──────────┤   real-time
           │            │  jobsig: new·progress·hello·  │           │   coordination
           │            │  control  (ephemeral)         │           │
           │            └──────────────────────────────┘           │
           │            ┌──────────────────────────────┐           │
           └───────────▶│  encrypted shards (client.storage)◀───────┘   payload =
                        │  P2P (WebRTC) ↔ origin (R2)   │               manifest
                        │  raw in · HLS segments out    │               (read cap)
                        └──────────────────────────────┘

• jobs table — the durable source of truth. A row is the job’s state machine. It survives reloads and is visible to every device.
• A per-user Space — ephemeral, real-time coordination (jobsig messages): wake idle workers, mirror progress across devices, announce presence + power, and pause/cancel. The DB poll is the floor; the Space makes it instant.
• Encrypted shards — the raw upload is stored once as content-addressed shards. The job row carries only the manifest (an opaque read capability), so any worker fetches the bytes peer-to-peer (WebRTC between co-present devices) with origin (R2) as the fallback. The HLS output lands as shards too.
• A media record — the finished job writes a row the library reads to play.

The job state machine

A job row moves through five states. The columns that drive it are status, the lease (worker_id + heartbeat_at), and the payload pointer (input_manifest).

State	Meaning
queued	Enqueued and waiting. Claimable once its payload is fetchable.
processing	A worker holds the lease and is transcoding. Kept alive by a heartbeat.
done	Output stored; result_id points at the media row.
error	A genuine processing failure (bad input, ffmpeg crash).
paused	User-suspended; excluded from claiming until resumed.

                     insert (enqueue)
                          │
                          ▼
        ┌──────────────────────────────────┐
   ┌───▶│              queued              │◀─── release  (claimed but can't
   │    └──────────────────────────────────┘     fetch payload → give it back)
   │        │                     ▲
   │        │ claim               │ reclaim
   │        │ (write worker_id,   │ (heartbeat stale > 45s:
   │        │  re-read to confirm)│  the worker died)
   │        ▼                     │
   │    ┌──────────────────────────────────┐
   │    │            processing            │──── heartbeat every ~12s
   │    └──────────────────────────────────┘
   │   retry   │            │            │
   ├───────────┘            │            │
   │  pause                 ▼            ▼
   │ (control)         ┌─────────┐   ┌─────────┐
   └──── paused        │  done   │   │  error  │
                       └─────────┘   └─────────┘

Transitions:

• claim — a worker writes its worker_id onto a claimable row, then re-reads to confirm it won (a cheap optimistic lock; a rare double-claim is wasteful, not wrong). Claimable = queued, or processing with a stale heartbeat.
• heartbeat — while processing, the worker bumps heartbeat_at every ~12 s (throttled, independent of progress ticks). This is the liveness signal.
• reclaim — if a worker dies mid-job, its heartbeat stops; after STALE_MS (45 s) another worker treats the job as claimable again. This is crash recovery.
• release — a worker that claims a job but can’t fetch the payload returns it (status: "queued", worker_id: null) instead of failing it. Never claim-and-error.
• pause / cancel — a control signal over the Space; whichever device runs the job aborts. retry just flips a failed/paused row back to queued.

Who runs the job: grading by power (and size)

Every device is a worker, but they are not equal. Each announces a rough capability score in its hello presence signal (a browser ≈ core count; the native worker ≈ cores × 30, because native ffmpeg with hardware acceleration crushes ffmpeg.wasm). A weaker worker defers a claimable job to a stronger one for a short window, stepping in only if the job looks stuck.

const MY_SCORE = navigator.hardwareConcurrency || 4;   // native worker: cores × 30
const strongerOnline = () => [...peers.values()]
  .some((w) => Date.now() - w.ts < 30_000 && w.score > MY_SCORE);

// deferral, inside the claim check:
if (job.input_manifest && strongerOnline() && Date.now() - job.created_at < BACKSTOP_MS)
  return null;                                          // let the strong worker take it

Two guards keep it smooth: a startup grace (hold off claiming right after joining until you’ve heard peers), and a backstop (take a deferred job anyway if it’s sat unclaimed too long — the strong worker is overwhelmed or gone).

Capability includes size

Power is one axis; can this worker even hold the payload is the other. The in-browser transcoder is ffmpeg.wasm, capped at ~2 GB of WASM memory, and reading a File whole hits the browser’s ~2 GB ArrayBuffer limit. So a 4K movie cannot be transcoded in a tab. The browser refuses to claim oversized jobs and lets them fall to a native worker:

const MAX_LOCAL_BYTES = 1_000_000_000; // ~1 GB ceiling for the in-browser tool
const canDo = (j) =>
  (haveLocalCopy(j) || j.input_manifest) && (j.input_size ?? 0) <= MAX_LOCAL_BYTES;

The UI shows “too large for this browser — waiting for the native worker,” so an oversized job reads as routed, not stuck. Skipping this check is exactly what produces a “the file could not be read” error on a big upload.

The raw payload: store once, stream back

The payload is the crux for large files. Two rules keep memory flat regardless of size.

Storing streams. space.putFile slices the source per chunk as it encrypts and uploads, so enqueuing even a 30 GB file never materializes it in memory. The job row gets the resulting manifest once the store lands:

const { manifest } = await space.putFile(file, { name: file.name, type: file.type });
await jobs().update(id, { input_manifest: manifest, updated_at: Date.now() });

Reading back streams too — on the native worker. A single Uint8Array can’t hold a multi-GB file (Node caps ~4 GB), so instead of getFile, the worker uses space.getFileStream(manifest) and pipes decrypted chunks straight to a temp file. ffmpeg then reads the file directly; nothing large ever lives in RAM:

import { createWriteStream } from "node:fs";
import { once } from "node:events";

const out = createWriteStream(tmpPath);
for await (const chunk of space.getFileStream(manifest)) {   // P2P → origin, decrypted
  if (!out.write(chunk)) await once(out, "drain");           // backpressure
}
await new Promise((res, rej) => out.end((e) => (e ? rej(e) : res())));

const { hls_index } = await transcodeToHls({ inputPath: tmpPath /* ffmpeg reads it */ });
await jobs().update(job._id, { status: "done", result_id: media._id });

The HLS output goes back through storage the same way: each .ts segment is a small putFile, and an index manifest ties the variants together. Playback then pulls segments by manifest (peer or origin) and feeds them to hls.js.

End to end

1. Add a video. The browser inserts a queued job (input_size set) and starts storing the raw as shards in the background. It posts jobsig: new so idle workers wake.
2. Route by capability. For a small file the browser may transcode it locally; for a large one it declines (size ceiling) and the manifest is what matters.
3. A worker claims it. The native worker (higher score) wins the lease, writes worker_id, and starts. Its heartbeat keeps the lease; browsers mirror its progress signals live even though they aren’t running the job.
4. Transcode. The worker streams the raw to disk and runs hardware ffmpeg into HLS renditions, storing each segment as shards.
5. Finish. It writes the media row, sets the job done + result_id, releases the raw shards (the HLS output is the durable copy), and posts jobsig: done.
6. Play. Any device streams the HLS variants from shards — adaptive, encrypted, no origin round-trip when a peer has the bytes.

Run a native worker

The native worker is the same claim loop as a Node process. It logs in with your ZK identity, joins the same Space, and drains the same table — it just has native ffmpeg and a high score:

npx @muhkoo/theater-transcoder@latest    # point a spare machine at your queue

Nothing in the queue changes; a native worker is simply a peer that can hold big payloads and encode fast. It needs a system ffmpeg with the input codec (e.g. HEVC) and a little free temp-disk for the streamed raw.

Where to go next

• Distributed background jobs — the reusable pattern, built step by step (the jobs table, the lease, the Space, grading).
• client.space — getFileStream, sendMessage / onMessage, presence.
• client.storage & Offline — content-addressed shards, manifests as read capabilities, peer transfer.
• client.db — the queue table, queries, the owner-scoping rule.