Generated, not collected: what a test file’s provenance decides

Almost every codebase that touches files has a fixture nobody can account for. It is called something like sample.png or test-doc.pdf, it was downloaded during an afternoon several years ago, and it has been the input to a test ever since. Nobody knows its bit depth, whether it carries a colour profile, or whether its alpha channel is actually doing anything. Nobody knows who took the photograph inside it. And nobody could produce a second one like it if it were deleted tomorrow.

That file is not neutral. A test result is a statement about two things — the code and the input — and you only get to attribute the result to the code if you already know the input. When the input is a mystery, every surprising result has two suspects and no way to eliminate one of them. This is the argument behind a design decision in Novus Examples that sounds like a footnote and is actually the whole point: every file in the library is generated to an exact specification rather than collected from somewhere. This piece is about what that buys, and about how to get the same benefit in test data you build yourself.

The first is what is inside it. Formats are containers with a great deal of latitude, and two files with the same extension can differ in ways that decide the outcome of your test. A PNG can be 8-bit or 16-bit, palettised or truecolour, carrying an ICC profile or not, with an alpha channel that is present but entirely opaque. A CSV can be UTF-8 or cp1252, comma or semicolon delimited, with quoted fields containing embedded newlines or without. When your pipeline behaves oddly on such a file, you cannot say whether the code is wrong or the input was simply unusual — and the effort of finding out is often larger than the bug.

The second is whether you are allowed to keep it. A fixture committed to a repository is distributed with that repository, to everyone who clones it, for as long as the history exists. A photograph pulled from a search result is somebody’s photograph. This is the quiet risk in scrounged test data, because it never surfaces during development and only becomes a question at exactly the wrong moment — an audit, an acquisition, a customer’s legal review.

The third is whether you could make it again. Fixtures go missing, and needs change. When you want the same file at a different size, or with one property altered so you can isolate a variable, an unreproducible sample gives you nothing to work from. You go hunting again, find something slightly different, and quietly break the comparability of every result you had before.

In Novus Examples nothing was downloaded from anywhere. Each file is the deterministic output of a written specification, and three consequences follow from that one decision — one for each of the unknowns above, which is not a coincidence, because those unknowns are what the decision was made to remove.

Licensing stops being a question. A file authored purely to serve as a sample has no photographer, no author, and no prior rights holder, so there is nothing in the catalog whose provenance anyone has to establish later. The example files carry no usage restriction, and the visual templates likewise ask for no attribution. That is not a permissive licence stretched over uncertain origins; it is what remains when there are no origins to be uncertain about.

Contents stop being a question, because each file arrives with its own sheet of stated properties — how much noise was mixed into an image, which timestamps and text encodings a document was written with, what sample rate an audio clip was rendered at. The value of that shows up at the exact moment your pipeline misbehaves. You already have the second half of the equation written down, so the odd behaviour has only one possible owner left, and that owner is the code.

And reproducibility stops being a question, because a specification plus a seed is a recipe rather than a souvenir. The library’s own editor demonstrates it: hand it a JSON tone specification and it renders genuine 16-bit PCM WAV from a seeded generator, so the identical spec yields the identical bytes on any machine, on any day. A fixture built that way is a fixed reference point. A file someone once downloaded is a snapshot of one afternoon nobody recorded.

As of the most recent release gate, that discipline runs across 2,558 files in 17 categories and 221 extensions, navigable by format or by 123 curated testing purposes, alongside 504 editable visual templates across 42 categories. The live catalog is the canonical count, since it keeps growing. But the size is the least interesting thing about it — a large pile of files of unknown provenance would be worse than a small set you understand.

There is a difference between a fixture that is noisy and a fixture that states its noise level, and it is the difference between an impression and a measurement. Novus Examples takes the second option everywhere it can: a denoise fixture states its noise level, an interpolation fixture states which frames were removed. A fixture’s job is not to look like production data. Its job is to be completely described.

The practical payoff shows up the moment you have to tell somebody else what you found. "The upscaler looked a bit soft on a noisy clip" is not a bug report; it is a feeling. "The upscaler loses detail above this stated noise level, on this fixture, against this reference" is a report someone can reproduce, argue with, and fix. Adjectives do not survive being passed between people. Numbers do.

The same reasoning produced a detail in the video set that is easy to skim past and worth stealing outright. Reference clips are encoded at a deliberately higher quality than the degraded versions derived from them — two tiers, not one. The purpose is to keep the reference cleaner than any difference you might be hunting for. Encode the reference at the same quality as the input and its own compression artefacts become the smallest thing you are capable of detecting, at which point the number you report describes your fixture rather than the software you were evaluating.

The sharpest illustration of this in the library is the video container set, described in full in Novus Examples grows up: 401 video fixtures, 480 templates, and OBS-compatible scenes: seven containers carrying one bitstream that was stream-copied rather than re-encoded, so the wrapper is the only thing that differs between them. What matters for your own work is the reasoning rather than the example. If those seven files had each been encoded separately, a failure on one of them would have had two candidate explanations, and you would have had to spend time eliminating the second before the test told you anything at all.

That is the line between a fixture set and a pile of files. A pile has variety. A set has controlled variation. Variety feels more thorough and reliably produces ambiguous failures; controlled variation looks repetitive on a listing page and produces failures that resolve themselves. It is why keeping a plain photo and a transparent PNG as separate entries, an sRGB image and a CMYK one, a clean CSV and a deliberately messy one, is not catalog padding. It is the guarantee that when one passes and its neighbour fails, exactly one known thing separates them.

The cost of getting this wrong is not a failed test — it is a test whose failure nobody can act on, which in practice means a test that gets muted. Ambiguity is how suites quietly die.

A large part of the catalog does not ship as single files at all, but as matched twos: tidy alongside deliberately mangled, colour alongside greyscale, a born-digital document alongside a scan of the same thing. The reason is that a transformation cannot be judged from one endpoint. OCR, colourisation, visual diffing, format migration — each takes a file from one known state to another, and grading the result means holding both states at once. One file lets you say the output looks plausible. Two let you say what changed.

The video set takes the idea to its logical end: ten AI-video task suites in which every degraded input records the identifier of its own pristine reference, with both sides sharing a group so you can navigate from either. If you work on upscaling, denoising, frame interpolation, matting, or depth estimation, that pairing is the difference between a demonstration and a measurement — a lone clip can only be judged by eye, and an eye grades on a curve that shifts with how much you want the result to be good.

The general form is worth adopting even when no model is involved. Any test that asks whether a transformation preserved something needs a stored definition of what that something was beforehand. A pair is the cheapest definition available, and it is the one that keeps working after everyone who wrote the test has forgotten what the file looked like.

Generating your inputs settles where they came from. It also creates an obligation in the other direction, because a system capable of rendering a persuasive certificate is, by the same machinery, capable of rendering a persuasive forgery. Small print does not address that. Only the artwork itself can.

Which is why the guard rails in the visual template library are built into the documents rather than printed beside them. Certificates and badges carry a sample marking the Studio will not let you edit or delete, and the issuing body named on them does not exist. Whole categories — legal, financial, healthcare, fitness, non-profit — carry a disclaimer under the same lock. The barcodes are decorative: they resolve to nothing and sit inside a prefix range reserved for restricted circulation. Healthcare stays on the operational side of the line entirely, with no patient intake or medical-history layouts anywhere in the set. Every name, figure and date in the artwork is invented.

A defect found during the audit that produced those rules shows how this category of protection tends to fail. In the badge family, the sample marking and the subtitle were a single node, so any row that supplied its own issuer wording quietly replaced its own marking with that wording. Every preview looked correct. The two are separate nodes now and the marking is locked. The same pass caught three places where a generic-looking safe-area dimension was in fact one named platform’s published specification — dimensions as a class are simply facts and stayed, but those three were a template rather than a fact, and they were removed.

The relevant lesson is not really about templates. It is that at any real scale, spot-checking is not a control. A sweep across the new scenes generated for that release found fifty truncated strings — captions cut off mid-word, like a web address ending in an ellipsis — and they are now zero. Nobody had noticed, because a preview only shows you the rows somebody chose to open, and whether a string truncates at all depends on how long that particular row’s words happen to be. Assertions inside the generator catch that class of defect on every row. Eyes catch it on the rows they happen to visit.

Very little of this depends on using our library, and the transferable version is short enough to adopt this week.

Commit the generator, not only the artifact. A checked-in script that produces your fixtures makes them reproducible, reviewable in a diff, and adjustable when you need a variant. The files themselves become build output rather than mysterious binary blobs in your history.

Seed anything random. A generator that produces a different file each run has given you variance you did not ask for and cannot subtract from your results.

State the property, and state it in a number. Put it in the filename or in a sheet beside the file, but do not rely on anybody remembering that this one is "the tricky one".

Keep a pristine reference beside every degraded input, and encode the reference better than the input if encoding is involved at all.

Vary one thing at a time, on purpose, and write down which thing. If two fixtures differ in two ways, a failure on one of them has told you almost nothing.

And recognise the questions no file can answer. Whether your scraper copes with content appended behind a sentinel, whether your crawler gets past a consent banner, whether focus stays trapped in a modal, whether a print stylesheet is applied at all — none of that is a property of a document you can download. It is behaviour that only exists while a page is running. The library keeps twelve live targets for precisely this, each page deliberately doing one thing and nothing else, and the full roster is set out in Novus Examples: free sample and dummy files for every format. The single-behaviour rule is the whole point: any real site is doing a dozen things simultaneously and may be rebuilt overnight, so a failure against one leaves you guessing, whereas a failure against a page with one job has already told you what broke.

The coverage page is where the library declines to imply it covers everything. 221 extensions appear there as available, each because a real, documented file exists behind it; a short list appears as still to come; and around fifteen appear as unsupported with a stated reason attached rather than being quietly left off. Availability is defined by whether you can download the file, not by whether the format has been heard of.

And a generated fixture is not production data. It will not reproduce the specific weirdness of your own customers’ files — the scanner firmware that writes a malformed timestamp, the legacy exporter with the off-by-one in its row count, the phone that rotates a photo through metadata rather than pixels. Generated fixtures give you controlled coverage of the properties that formats have. They are a complement to a sample of real inputs, not a replacement for one, and anyone telling you otherwise is selling something.

Novus Examples is also live and still growing, which is the same thing every honest product page in this portfolio says. The catalog keeps moving, the surfaces around it keep expanding, and the coverage page is where the current state is written down rather than in an article that was true on a Tuesday.

If you want the practical entry point rather than the argument: open the catalog, browse by the job you are testing rather than by format, and download the fixture whose sheet describes the property you care about. Read a spec sheet and use what it tells you covers turning that sheet into a test you can report on, and Test with clean and noisy pairs covers the paired-fixture pattern. Novus Examples covers everything in one place, and the current map of the app is Tool maps.

And if you want a worked example of why any of this matters, the companion article to this one — The four ways a file conversion goes wrong — and which ones a converter can catch — walks through evaluating a file converter, where the difference between a known input and a scrounged one is the difference between a conclusion and a shrug.