Novus Examples
Test with clean and noisy pairs
Paired fixtures — clean and noisy, colour and greyscale, source and scanned — differ in exactly one property, which is what makes a before/after test mean something.
A before/after test is only meaningful if exactly one thing differs between the two files. Take two arbitrary photographs and compare how a tool handles them and you have learned nothing, because everything differed.
Paired fixtures solve this by construction. Both halves come from the same deterministic script and the same seed, with one property varied — noise, colour, or scan simulation. Everything else is identical, so the difference in output is attributable.
Contents
Two ways to finish
Enhancement & denoise
Clean ↔ noisy at a known sigma.
Colour & scan
Colour ↔ greyscale, source ↔ scanned.
- 1
1. Understand what makes a pair a pair
Both files are produced by the same generator from the same seed. The clean half is the reference; the noisy half is the reference plus a precisely specified perturbation. There is no second photograph, no re-encode, and no incidental difference.
This is why generated fixtures beat collected ones for this job. Two real photographs of the same subject differ in a dozen ways you did not choose and cannot enumerate.
- Same generator, same seed, one property varied.
- The clean half is a true reference, not an approximation.
- Nothing incidental differs to confound the comparison.
- 2
2. Use clean ↔ noisy for denoise and enhancement
The classic use is evaluating a denoiser or an enhancement model. Run the noisy half through the tool and compare the output against the clean half, which is the ground truth the noisy file was derived from.
Because the spec sheet states the noise sigma, you can characterise where a tool starts failing rather than just noting that it did. "Recovers well below sigma 0.05, degrades above 0.1" is a finding; "sometimes it looks bad" is not.
- 3
3. Use colour ↔ greyscale for colourization
Colourization is unusual in that the correct answer genuinely exists: the greyscale half was derived from the colour half, so the colour original is ground truth rather than one plausible option among many.
That makes evaluation far less hand-wavy than it usually is. You are not asking whether the output looks nice, you are asking how close it got to a known target.
- 4
4. Use source ↔ scanned for OCR and document pipelines
Scan simulation produces the artefacts that break document pipelines — skew, compression, contrast loss — from a source whose text you already know exactly. Run OCR on the scanned half and diff the result against the source text.
That diff is a real accuracy number rather than an impression, and because the degradation is specified you can say which artefact caused which error.
- 5
5. Report failures with the property, not the file
When you file a bug from a paired test, lead with the property. "Fails at noise sigma 0.12 with this seed, passes at 0.05" is reproducible by anyone; "this image looked wrong" is not.
Since the files are generated by deterministic scripts, whoever picks up the report can regenerate exactly the same fixture. That is a meaningful advantage over attaching a screenshot.
- 6
6. Fold pairs into regression tests
Pairs are good regression fixtures precisely because they are stable. The same seed produces the same file forever, so a test that passed last month and fails today has detected a change in your code rather than a change in your test data.
Pick a small set that spans the difficulty range — one easy, one marginal, one that should fail — and keep them. Three well-chosen pairs beat thirty arbitrary files.
One variable, or it is not a test
Paired fixtures come from one generator and one seed with a single property varied, so the difference in output is attributable. Use clean ↔ noisy for denoise, colour ↔ greyscale for colourization, source ↔ scanned for OCR — and report failures by the property from the spec sheet rather than by attaching the file.
Frequently asked questions
Quick answers to common questions about this topic.
What pair types exist?
Clean ↔ noisy, colour ↔ greyscale, and source ↔ scanned are the main three. Each varies exactly one property while holding everything else identical.
Why not just use two real files?
Because two real files differ in ways you did not choose and cannot enumerate, so any difference in output is unattributable. Paired fixtures share a generator and a seed.
How do I report a failure from a pair?
By the property, using the spec sheet: "fails at noise sigma 0.12 with this seed, passes at 0.05". Anyone can regenerate the exact fixture from that, which a screenshot does not allow.
Are pairs good regression fixtures?
Yes. The same seed produces the same file indefinitely, so a newly failing test means your code changed rather than your data. A small spread across the difficulty range beats a large arbitrary set.