All posts
FFmpegEngineering12 Oct 2026 · 4 min read

Reading a video file byte by byte

Open an MP4 in a hex viewer and the first thing you see is four bytes of length, then four ASCII letters: ftyp. Everything in the file follows the same pattern. A box is usually a 32-bit size (a size of 1 means a 64-bit size follows), a four-character type, then a payload that may itself be a run of more boxes. That is the whole container format at the byte level, and once you can see it, a lot of "why won't this play" questions answer themselves.

Boxes, then NAL units

The top level of an MP4 is usually three boxes. ftyp declares the brand, the kind of file this claims to be. moov is the index: tracks, timing, and a table saying where every sample lives. mdat is the payload, the encoded samples themselves, with nothing in it that says where one ends and the next begins. The index is the only thing that knows.

Inside mdat, an H.264 sample is a run of NAL units. Each one starts with a header byte whose low five bits give its type: 7 is a sequence parameter set, 8 a picture parameter set, 6 is SEI, 5 is a keyframe slice (IDR), 1 is an ordinary slice. How the units are separated depends on the container. A raw .h264 file or an MPEG-TS stream marks each unit with a start code, 00 00 01 or 00 00 00 01. MP4 does it differently: each unit is preceded by a length, and the width of that length (usually 4 bytes) is recorded once in the avcC box. The parameter sets live in avcC too, so the decoder has them before it reads a single frame.

00000000  00 00 00 20 66 74 79 70 69 73 6f 6d 00 00 02 00  ... ftypisom....
          ^size=32    ^type

Same video, two framings. A tool that assumes start codes will see garbage in an MP4, and one that assumes length prefixes will see garbage in a transport stream.

What Analyze shows you

The What the FFMPEG tool has an Analyze view with an Inspect panel, and its Bitstream subtab shows both layers for the file you loaded. At the top is a fact grid: NAL unit count (taken over the first 900 frames and capped, so a long file shows 6000+), the length prefix (shown as 4 bytes (AVCC) for a typical MP4), counts of IDR, slice and SEI units, and whether SPS and PPS also appear in-band or only in avcC. Below that is a strip of up to 400 NAL units sampled from that set, coloured by type, with each width scaled to the square root of its size so small SEI units don't vanish next to a large keyframe. Hover one and the readout gives its name, size, sample number and byte offset.

Under the strip is a hex dump in the layout you would expect: an eight-digit offset, 16 bytes per row, and an ASCII column where anything outside 0x20 to 0x7E becomes a dot. Select a frame in the Frames subtab and the dump jumps to that frame's sample at its absolute file offset. With no frame selected it shows the head of the file, and it never renders more than 1 KiB (64 rows), saying so when it truncates. Row 0 is ftyp.

Codec plus container

The Codec subtab turns the same probe into a table: one row per video and audio stream, plus the container, against four browsers (Chrome, Firefox, Safari, Edge). Each cell reads Yes, Partial or No.

The rules behind it are short. H.264 with yuv420p and a Baseline, Main or High profile is Yes everywhere. HEVC is Yes in Safari and Partial elsewhere, because the other three depend on hardware decode. VP9 and AV1 are Yes in Chromium browsers and Firefox and Partial in Safari. A video stream whose pixel format is anything other than plain yuv420p makes every cell No, whatever the codec. The container gets its own row: MP4 is Yes everywhere. QuickTime MOV is Yes in Safari and Partial elsewhere. Inspect reads MP4 and MOV, so those are the only two container rows you will see.

The matrix is a hand-written snapshot, not a live compatibility lookup, and the code comment says so. That is also why codec and container are two separate questions. A browser needs to demux the box structure and decode the NAL units inside, and either half can be the one that fails.

Getting the data out

The Inspect header has an Export data menu with JSON and CSV. The export is not a separate formatter: it calls the same builder as the in-browser ffprobe -show_format -show_streams run, with -of json or -of csv, so the file you download matches that in-browser ffprobe run for the same probe. The extension is dropped and .probe.json or .probe.csv added, so clip.mp4 becomes clip.probe.json, and if there is nothing real to export it downloads nothing rather than an empty file.

All of it runs against the bytes already in your browser. Nothing is uploaded.