Calibration, not detection power
We nearly shipped the naive version. Here's the benchmark that caught it before real customer data would have.
Most drift detectors are judged on one number: how often they catch a real anomaly. That number matters, but it hides the more important question — when you ask a detector for a 5% false-alarm rate, does it actually give you 5%, or does it drift as soon as the data stops looking perfectly Gaussian? A detector that silently over-alarms trains your team to ignore it. One that silently under-alarms is worse.
Our own first implementation failed this test. A chi-squared distance from a fitted Gaussian is a completely standard way to build a drift score. Asked for a 5% false -alarm rate on clean, unseen data, here's what it actually delivered:
| Data shape | Target | Kirelta chi² (naive) | Kirelta conformal (shipped) |
|---|---|---|---|
| Gaussian | 0.050 | 0.049 | 0.053 |
| Correlated features | 0.050 | 0.049 | 0.053 |
| Heavy-tailed (t₃) | 0.050 | 0.192 | 0.051 |
| Skewed (log-normal) | 0.050 | 0.294 | 0.053 |
On real Gaussian data, the naive path looks fine. On heavy-tailed or skewed data — which most real production features actually are — it flags up to 6× too often. Every number in that "shipped" column is a held-out conformal calibration step doing its actual job: turning a distributional assumption into an empirical guarantee. That's the whole reason the conformal path exists, not a nice-to-have layered on top.
How that compares to standard alternatives
Same protocol, same four data shapes, against real scikit-learn detectors — no per-method tuning, everyone asked for the same 5% rate:
| Method | Calibration error | Mean detection (6σ spike) |
|---|---|---|
| EllipticEnvelope | 0.0010 | 0.991 |
| Kirelta (conformal) | 0.0023 | 0.991 |
| LOF (novelty) | 0.0024 | 0.977 |
| IsolationForest | 0.0019 | 0.612 |
| OneClassSVM | 0.0282 | 0.983 |
Read that IsolationForest row carefully: it's reasonably well-calibrated (0.0019 error) and still misses almost 40% of large anomalies — calibration and detection power are genuinely separate properties, and a method can have one without the other. OneClassSVM has the opposite problem: strong detection, but it over-alarms on clean data by up to 8 points. Kirelta's conformal path and EllipticEnvelope are essentially tied for the best combination of both, here.
Calibration also has to survive the data changing shape
A fixed calibration is only correct for the regime it was set on. Real production data doesn't stay in one regime — traffic goes quiet, then a feature's variance spikes, then it settles into something seasonal. Held out across seven distinct synthetic regimes, with Adaptive Conformal Inference (ACI) turned off and then on:
| Regime | Without ACI | With ACI |
|---|---|---|
| Quiet | 0.0625 | 0.0504 |
| Slow drift | 0.0465 | 0.0500 |
| Sudden shift | 0.0478 | 0.0501 |
| Variance burst | 0.1177 | 0.0495 |
| Seasonal | 0.0529 | 0.0502 |
| Heavy tail | 0.0634 | 0.0506 |
| Autocorrelated | 0.0449 | 0.0499 |
Target is 0.050 in every row. Without adaptation, a sudden spike in variance alone pushes the realised rate to 0.1177 — more than double the target, exactly the kind of drift that should trigger a review, silently retraining the alarm to ignore itself instead. With ACI on, every regime lands within a few thousandths of target. This is on by default; turning it off is what it took to produce the left column.
What this benchmark doesn't show: one feature count (p=4), one sample size, one anomaly type (a uniform 6σ shift), default hyperparameters for every competitor. It's real evidence about calibration under a fixed protocol — not a general ranking of detectors, and we're not claiming it is. The script that produced every number above ships in the repository; the comparison is designed to be re-run, not just cited.