What a key detection confidence score should mean — and how we re-fitted ours
Most key detection tools give you a key. Fewer tell you how sure they are. And a confidence number on its own does not mean much — what matters is whether it is calibrated: whether, of all the tracks the tool marks 90, roughly nine in ten actually turn out to be right.
This post is about that promise: how we check it, what happened when we found ours had drifted, and how we chose where to draw the line between "trust this" and "check it by ear."
What "calibrated" means
A key detector does not see the answer; it sees evidence and scores it. That raw score is not a probability. A raw score of 40 could mean 80% likely to be right, or 50% — it depends entirely on how the engine was built.
Calibration is the step that turns the raw score into a number that means something. You run the engine on a large set of tracks whose key is already labelled, group the results by raw score, and look at how often each group was actually right. That table becomes a curve, and the curve is what the user sees.
The test of a calibrated score is simple: take every track the engine showed as 80–89, check them, and the share that agree with the label should land somewhere in the 80s. If it lands in the 60s, the number is overselling. If it lands in the 90s, it is underselling.
Ours had drifted — in the cautious direction
Our first calibration curve was fitted in August on 348 tracks. Since then the engine changed, most of all through a small decision layer that settles close calls between the top two candidate keys. That layer mainly fixed low-confidence tracks, so the old curve was now describing an engine that no longer existed.
On 21 September we re-measured on 8,004 tracks from two working DJs' libraries, my own and a DJ friend's, each labelled by Mixed In Key. The engine was underselling itself in every band:
| Shown confidence | Actual agreement |
|---|---|
| 40–44 | 48.9% |
| 50–59 | 64.0% |
| 80–89 | 91.6% |
An underconfident score sounds harmless — better safe than sorry. It is not. Every needless "check this by ear" teaches you to ignore the warning, and a warning you have learned to ignore is worse than no warning at all, because it is still on the screen when it matters.
How we re-fitted it
The method is deliberately plain:
- Split the raw score into 20 equal bands and measure the actual agreement in each.
- Force the curve to be monotonic with the pool-adjacent-violators algorithm, so a higher raw score can never show a lower confidence. Without this, noise in small bands produces nonsense like "73 is more trustworthy than 76."
- Cap the curve at 98. The engine never tells you it is certain.
The average calibration error, meaning how far the shown number sits from the measured agreement, went from 5.4 points to 1.3.
That 1.3 is measured on the same tracks the curve was fitted on, which flatters it. The honest test is to fit the curve on one library and measure the error on the other. Done in both directions, the error went from 5.1 to 1.5 and from 6.2 to 2.5. Not as tidy, but that is the figure that tells you how the curve behaves on a library it has never seen — which is yours.
Where to draw the line
A calibrated number still needs a decision on top: below which value does the app warn you? That is a trade-off, not a fact. A higher threshold flags more tracks, and the unflagged ones get more reliable. Measured on all 8,004 tracks with the new curve:
| Warn below | Share flagged | Agreement on unflagged tracks |
|---|---|---|
| 70 | 20.1% | 91.9% |
| 75 | 23.5% | 92.4% |
| 80 | 30.4% | 94.1% |
| 85 | 38.7% | 95.9% |
We chose 80 for "check this by ear" and 60 for "key uncertain." The more cautious option, 85, would have flagged almost four tracks in ten. At that rate the warning stops carrying information; it just becomes part of the screen.
Here is what the chosen thresholds look like on my friend's 6,281-track library, with the curve fitted only on my own library, so the numbers come from tracks the curve never saw:
| Band | Share of tracks | Agreement |
|---|---|---|
| 80 and above — no warning | 69.3% | 94.3% |
| 60–79 — "check this by ear" | 22.1% | 69.7% |
| Below 60 — "key uncertain" | 8.6% | 50.5% |
On the standard annotated benchmark we use for our headline accuracy, the 80 threshold covers 75.4% of tracks, and those agree with the annotation 88.2% of the time.
What we tried that did not help
The obvious next step was to replace the hand-built score with a model that learns confidence from nine signals, including how well the track's own sections agree with each other. It made calibration worse, not better: the error went from 1.5 to 2.6 points. We wrote that experiment up in the tech house measurement. A simpler score that keeps its promise beats a cleverer one that does not.
The limits of these numbers
- "Agreement" is agreement with Mixed In Key's labels, not ground truth. The labels are not perfect: across 635 tracks that appear in both libraries, Mixed In Key disagrees with itself on 45 (7.1%).
- Two libraries, both mostly house. Calibration is a property of the engine and the material. On music far from what we measured, the numbers could drift, which is why we re-fit every time the engine changes.
- A confidence score is about one result, not about a track. A score of 70 does not say the track is in a "bad" key. It says the evidence was mixed, and the app tells you why: whether the track's sections disagree with each other, or two keys are simply very close.
What to ask of any confidence number
If a tool shows you a confidence score or a warning flag, three questions are worth asking:
- Is it calibrated, and against what? A score that has never been checked against labelled tracks is a ranking, not a probability.
- Is it re-checked when the engine changes? Ours drifted after one engine change. Any engine that improves will drift the same way.
- Does the warning rate make sense? A tool that never flags anything is not more accurate; it is just not telling you.
We have not found published calibration figures for most key detection tools. That does not mean they do not exist; it means we could not find them to compare against. Ours are above, and the two libraries behind them are published on Kaggle with the engine's prediction and confidence for every track, so you can check the curve yourself.
To see where your own library lands, drop your tracks into Harmoniq. Every result comes with its confidence, and every flagged track with its reason. The analysis runs in your browser; nothing is uploaded.