Conformal prediction typically promises distribution-free coverage, but that guarantee can break down when the data-generating process shifts. A new arXiv paper examines what happens when the shift is announced in advance—a known boundary before the first affected outcome. The authors show that such an announced break reveals the partition in the data stream, and a reset at that boundary can be used to support one of two distinct audit procedures.
The two audits correspond to different validity concepts. The per-time audit reads what the authors call conformal reliability: the expected distance of the threshold's coverage from the nominal level at each time step. The other audit, frequency calibration, concerns the long-run proportion of times the prediction set contains the true outcome. The paper argues that an announced break separates these two properties, meaning a single reset cannot simultaneously satisfy both in the same way.
This distinction matters for practitioners who use conformal methods in online settings where change points are known—such as model updates or policy changes. The choice of audit determines how the reset should be performed and what guarantee is actually being maintained. The abstract is brief, but the core contribution is a formal separation between two notions of validity that are often conflated in applied work.