In machine learning, claims that two training methods perform equally well often rest on a non-significant difference between just two or three runs per method. That is a weak foundation: a failure to detect a difference is not the same as evidence that no meaningful difference exists.
A new arXiv paper develops a protocol for certifying practical equivalence in per-run performance. The protocol comes with a power theory, allowing researchers to reason about the probability of detecting a real difference when one is present. It also addresses run-count design, so experimenters can decide how many runs are needed to support an equivalence claim.
The work targets a common but flawed practice in stochastic machine-learning experiments. Instead of treating a null result as a verdict of equality, the proposed approach gives researchers a way to state, with statistical backing, that two methods are practically equivalent within a defined bound.