Running FFmpeg in the browser

The first thing to understand about running FFmpeg in a browser tab is that FFmpeg was never written for one. ffmpeg.wasm is the same C code, cross-compiled to WebAssembly, running inside the sandbox a browser gives every page. Inside that sandbox it decodes and encodes video exactly the way the native binary does. What changes is the machine underneath it, and a browser tab has no threads by default. Video encoding without threads is slow enough that a real-time editor built on it wouldn't feel real-time.

ffmpeg.wasm ships two builds to deal with that: a single-threaded core and a multi-threaded one, core-mt. The multi-threaded core is what Studio, BeemMeUp's in-browser FFmpeg node-graph editor, wants by default. It's also the one that needs SharedArrayBuffer.

Why threads need SharedArrayBuffer

Worker threads in a browser don't share memory with the main thread by default. They message-pass. FFmpeg's multi-threaded encoder wants several workers touching the same frame buffer without copying it between them on every step, which means it needs real shared memory: a SharedArrayBuffer, readable and writable by more than one thread at once.

Concurrent shared memory paired with a high-resolution timer is also the shape of a timing side-channel attack. That's why browsers don't hand SharedArrayBuffer to any page that asks for it. They hand it to pages that have proven no other origin's code is sharing the same process: pages that are cross-origin isolated.

What isolation costs

Getting crossOriginIsolated to read true takes two response headers on the document. Cross-Origin-Opener-Policy: same-origin stops other origins' windows from holding a reference to this one. Cross-Origin-Embedder-Policy: require-corp stops the page from embedding anything that hasn't opted in.

require-corp is the one that costs you something. Once it's on, every subresource the page loads (an <img>, a <video>, a fetch) has to be same-origin, fetched in CORS mode, or carry a Cross-Origin-Resource-Policy header saying it's fine to share. Anything else gets blocked, and the failure is often silent: a broken image, a video element that never fires, no banner pointing at the real cause.

That's the trade-off in one sentence. Multi-threaded FFmpeg in the browser needs isolation, and isolation blocks any cross-origin media that hasn't explicitly opted in. Turn the headers on for the whole app and you can break thumbnails and players that have nothing to do with Studio.

Measure before you flip the switch

So the work started with an inventory before any headers changed. If the storage provider couldn't serve media to an isolated page, that would change the design.

The inventory ran headless Chromium against the live app and checked two things on every route: whether crossOriginIsolated read true, and whether any network request failed. It found one real break: a bare <img src> or <video src> pointed at a cross-origin host with no Cross-Origin-Resource-Policy header gets blocked under require-corp, and neither GCS nor Supabase Storage, where BeemMeUp's video and thumbnails live, sends one.

hls.js needed no change at all, because it fetches its manifest and segments with fetch() in CORS mode rather than through a bare tag, and a CORS-mode request already carries the information require-corp wants. Only cross-origin media loaded the plain HTML way was at risk.

That distinction is what the fix became: a <StudioMedia> wrapper that forces crossorigin="anonymous" on every image and video element inside the tool, so a thumbnail component that renders everywhere else in the app doesn't quietly render nothing in the one place isolation is on.

Scoping the headers to one route

crossOriginIsolated is decided once, at the document's initial response. A client-side route change can't flip it later, so the headers only need to be on the responses for the pages that want isolation:

{
  source: '/ffmpeg-studio/:path*',
  headers: [
    { key: 'Cross-Origin-Opener-Policy', value: 'same-origin' },
    { key: 'Cross-Origin-Embedder-Policy', value: 'require-corp' },
  ],
},

Only requests landing on /ffmpeg-studio get COOP and COEP. Every other route keeps loading cross-origin media the ordinary way, because it never asked for isolation in the first place.

Memory is the other cost

The multi-threaded core buys speed at a fixed price. Its heap is capped at 1 GB with growth disabled, where the single-threaded core grows toward wasm32's roughly 2 GB ceiling.

Faster and more likely to run out of memory on a large file are the same trade. Studio ships core-mt by default and falls back to the single-threaded core automatically whenever crossOriginIsolated reads false, whether that's the deliberate route scoping or a browser that doesn't grant isolation at all. Neither core is strictly better. Each is a different answer to how much memory a video file is allowed to need before something has to give.