Standard model evaluation typically certifies that a model meets a preset key performance indicator (KPI) level. But in many real-world settings, the target level is not known in advance; instead, users want to compare candidate models by examining the full trade-off between performance and reliability at test time. Doing this requires reliable estimates of test-time KPI distributions, which becomes tricky when the same dataset is used both to pre-select a subset of candidates and to estimate their KPI distributions. That reuse can introduce post-selection bias.

Farzaneh and Simeone propose post-selection distributional model evaluation (PS-DME), a framework for statistically valid distributional assessment after arbitrary data-dependent model pre-selection. The method builds on e-values and controls the post-selection false coverage rate (FCR) for distributional KPI estimates. The authors also establish explicit conditions under which PS-DME is provably more sample-efficient than a baseline approach based on sample splitting.

The framework is tested on synthetic data, text-to-SQL decoding with large language models, and telecom network performance evaluation. In these experiments, PS-DME enables reliable comparison of candidate configurations across a range of reliability levels, supporting statistically sound exploration of performance–reliability trade-offs rather than focusing on a single pre-specified KPI threshold.