Test a video pipeline with the 401-file video set

The video set is large enough that browsing by file type is the slow route. Purpose pages group the fixtures by the job: /for/video-codec-testing for codec and container work, /for/streaming-manifest-testing for HLS and DASH, /for/caption-encoding and /for/subtitle-testing for timed text, /for/media-accessibility for multi-track audio and subtitles, /for/media-metadata-testing for chapters and container tags, and a purpose page for each AI-video task — upscaling, denoise, matting, interpolation, depth, stabilisation, restoration, segmentation, and colourisation.

Each fixture still carries a spec sheet, and for video the sheet is where the useful detail lives: the codec and encoder settings, the pixel format, the frame cadence, whether a rotation flag is set, and — for the paired fixtures — the identifier of the reference clip. Read it before you run anything, because half the value of a fixture is knowing exactly what was done to it.

The codec set is deliberately controlled. Fourteen codecs cover the common and the awkward, seven pixel formats cover subsampling and bit depth, and — the part that matters most — seven containers are stream-copied from a single encode rather than re-encoded per container. Only the wrapper differs. If your tool handles the MP4 and fails on the MKV, the problem is unambiguously container handling, because the video bitstream inside them is byte-identical.

Around that sit the variations that break players and transcoders in practice: GOP length and B-frame structure, faststart versus not, fragmented MP4, interlaced content, rotation metadata that a naive player ignores, and VP9 with an alpha channel. That last one is worth a warning, because probing it wrongly is the expected mistake — WebM stores VP9 alpha in a block-additional element signalled by a flag, and some decoders ignore it and report an opaque pixel format. The fixture documents this so a false alarm does not cost you an afternoon.

Streaming fixtures come in two shapes. Some are parser-only manifests — an HLS playlist or a DASH MPD you feed to a manifest parser without needing the media to exist. Others are complete, playable packages: the manifest, the initialisation segment, and the media segments, ready for an actual player.

For those to work in a browser player they have to be fetchable cross-origin, so streaming fixtures are the single documented exception to the library's download-only rule: they are served inline with permissive CORS, and Safari gets a correct MIME type for the .m3u8. Everything else about the sandboxing stays in place. The practical upshot is that you can paste a manifest URL into your player and it will load, rather than downloading a file you then have to serve yourself.

Ad-signalling fixtures sit alongside them — VAST, VMAP, and SCTE-35 — for testing an insertion pipeline reads the markers rather than skipping past them.

Caption bugs are almost always encoding bugs, and almost never reproduce on the tidy UTF-8 file a developer tests with. The timed-text set exists to supply the untidy ones: UTF-8 with and without a byte-order mark, UTF-16 in both byte orders, CRLF and LF and files that mix the two, cp1252, and scripts that expose assumptions — CJK, Arabic, Hebrew, Cyrillic, and emoji. SCC fixtures carry computed odd parity rather than plausible-looking bytes, so a parser that ignores parity is caught.

There is also a set of intentionally-corrupt captions, which are the fixtures worth keeping in a regression suite. A caption parser that throws on a malformed cue is fine; one that silently drops it and renders a video with missing subtitles is the failure you want to catch before a viewer does.

A multi-track fixture is only useful if the tracks are actually labelled, and this is where most hand-made test files fall short: the language tags are placeholders and the dispositions are unset, so a player that ignores them passes. The accessibility fixtures set them for real — language tags on each subtitle and audio track, and default, forced, hearing-impaired, visual-impaired, and comment dispositions applied where they belong.

That lets you test the behaviours that matter to a viewer who needs them: does your player select the forced track for foreign dialogue without being asked, does it offer the SDH subtitle track distinctly from the ordinary one, and does it surface the audio-description track at all. Chapter fixtures in six formats and container-metadata fixtures cover the surrounding metadata handling.

This is the part that turns the set from a pile of clips into a measurement tool. For ten AI-video tasks, each degraded input names a pristine clip it was derived from: a downscaled clip for upscaling, a noisy one for denoise, a frame-decimated one for interpolation, a desaturated one for colourisation, a shaky one for stabilisation, a heavily compressed one for restoration, and per-frame alpha or mask ground truth for matting and segmentation.

Run the degraded clip through your model, then compare the output to the reference rather than to your impression of it. The encodes are two-tier by design — ground truths are encoded at a higher quality than the inputs — so the measurement floor sits below the thing you are measuring, and codec artefacts in the reference do not become the limit of what you can detect.

A practical caution: match the fixture to the claim you are testing. An upscaler evaluated only on a clean downscale will look better than it is, because real footage arrives with compression damage as well as missing resolution. Reach for the restoration fixtures too if that is what your users will feed it.

You do not need many. A container pair to catch wrapper handling, one caption file with an awkward encoding, one intentionally-corrupt container, one streaming package, and one ground-truth pair for whatever your pipeline claims to improve will catch most of what breaks.

Because everything is generated deterministically, the same fixture is byte-identical every time you download it. A newly failing test means your code changed, not that the fixture did — which is the property that makes a regression suite worth having at all. The reference at Read a spec sheet and use what it tells you covers reading the properties, and Test with clean and noisy pairs covers the paired-fixture pattern in general.