Adaptive vision-language model (VLM) ensembles are designed to cut computational costs by letting easy images exit early while harder images are escalated to more expensive processing stages. To guarantee reliability, these systems often use a single conformal threshold that makes the final predictor marginally valid—meaning overall error rates stay within a bound across all inputs.
However, a new preprint from arXiv (2610.23864) argues that this marginal coverage can be misleading. The authors show that even when the ensemble as a whole meets its coverage target, individual processing routes—such as the path taken by images that are escalated—can suffer severe failures that are hidden by the aggregate statistic.
The work introduces SAGE-CP, a method that explicitly accounts for route-conditional coverage. Rather than relying on a single threshold, SAGE-CP aims to expose and control failures along each decision path. The authors suggest that practitioners should evaluate adaptive ensembles not just on overall accuracy but on how well they perform for every subgroup of inputs that follow a particular route through the system.