Novus Odds
Read the calibration statistics: Wilson intervals, z-tests, and chi-square
The Statistics pages aggregate calibration across runs and apply real significance testing — 95% Wilson intervals, a proportion z-test, and a chi-square goodness-of-fit test.
Calibration asks a simple question: when a price implies 20%, does the thing happen about 20% of the time? Answering it properly is harder than it looks, because the honest answer is always a range and the range is wide at exactly the probabilities people care about most.
The Statistics pages aggregate calibration across runs and apply actual significance testing rather than leaving you to eyeball a chart: 95% Wilson confidence intervals, a proportion z-test, and a chi-square goodness-of-fit test.
Contents
Two ways to finish
Per band
Wilson intervals on each odds band.
Overall
Chi-square across the whole distribution.
- 1
1. Read the odds-band aggregates first
Calibration is reported per odds band rather than as one number, because a book can be beautifully calibrated on favorites and badly calibrated on longshots while the overall average looks fine. Aggregating hides exactly the structure you are looking for.
Scan the bands for a pattern rather than a single outlier. Systematic drift in one direction across adjacent bands is a finding; one band out of line is usually a sample-size artifact.
- Per band, not overall — averaging hides the interesting structure.
- Look for patterns across adjacent bands, not single outliers.
- Longshot bands have the fewest events and the widest intervals.
- 2
2. Understand why Wilson rather than normal
The familiar normal approximation for a proportion interval breaks down at extreme probabilities and small samples — it produces intervals that are too narrow and, at the extremes, intervals that run below 0 or above 1, which is nonsense on its face.
That failure lands precisely where betting calibration is most interesting: longshot bands with implied probabilities of a few percent, and heavy favorites in the nineties. The Wilson interval stays inside [0, 1] and behaves sensibly at small counts, which is why it is the right default here.
- 3
3. Read an interval as a claim about ignorance
A 95% Wilson interval says: given what was observed, this is the range of true rates consistent with the data. A wide interval is not a failure of the tool, it is an accurate statement that the sample does not pin the answer down.
The practical rule follows directly. If the implied probability sits inside the interval, you have not shown miscalibration. You may suspect it; you have not shown it, and the honest move is to run more events rather than to argue.
- 4
4. Use the proportion z-test for a specific question
The z-test answers a narrower question than the interval: is this observed rate significantly different from this specific implied probability? Use it when you have a particular claim in mind rather than as a general trawl.
Testing many bands and reporting whichever came out significant is a well-known way to manufacture findings from noise. If you are going to test everything, expect roughly one band in twenty to look significant at the 95% level purely by chance.
- 5
5. Use chi-square for the whole distribution
The chi-square goodness-of-fit test asks whether the observed pattern across all bands is consistent with the implied probabilities taken together. It is the right test when the question is about the pricing process as a whole rather than about one band.
A significant chi-square with no single dramatic band is a meaningful result: it says the miscalibration is distributed, which is often more interesting than one obviously broken price band would be.
- 6
6. Change the seed before you believe yourself
The last discipline is the cheapest and the most often skipped. Rerun with a different seed. A calibration finding that survives several seeds is telling you about the generating process; one that does not was telling you about the seed.
Combine that with the significance tests and you have a result worth writing down. Neither is sufficient alone — a significant result on one seed is still a result about one sample.
Wide intervals are the honest answer
Read calibration per band, because averaging hides the structure. Wilson intervals rather than the normal approximation, because the approximation fails exactly where longshot and heavy-favorite bands live. If the implied probability sits inside the interval you have not shown miscalibration — run more events. Then change the seed before believing any of it.
Frequently asked questions
Quick answers to common questions about this topic.
Why Wilson intervals instead of the normal approximation?
The normal approximation is too narrow at small samples and can produce intervals running below 0 or above 1 at extreme probabilities — which is exactly where longshot and heavy-favorite bands sit. Wilson stays inside [0, 1] and behaves sensibly at small counts.
What does a wide interval mean?
That the sample does not pin the answer down. It is an accurate statement of ignorance, not a fault in the tool. If the implied probability falls inside the interval, you have not demonstrated miscalibration.
When should I use chi-square rather than a z-test?
Use the z-test for a specific claim about one band. Use chi-square when the question is whether the pattern across all bands is consistent with the implied probabilities taken together.
Is one significant result enough?
No. Rerun with different seeds. A finding that survives several seeds is about the generating process; one that does not was about the seed. Testing many bands and reporting the significant one manufactures findings from noise.