Few-shot recognition (FSR) models must adapt to new tasks with very little labeled data. Many methods lean on pretrained vision-language models and sometimes retrieve external, out-of-distribution (OOD) examples to supplement the scarce support set. Yet the authors of a new arXiv paper point out that model selection and hyperparameter tuning are often ignored or, worse, done using the test set—an unrealistic practice.

The paper argues that retrieved data are a natural validation source, but using them directly creates a paradox: adapting a vision-language model to the in-distribution (ID) few-shot data consistently degrades its performance on the retrieved OOD samples. So conventional accuracy-based validation on retrieved data would always prefer no adaptation, even when adaptation helps generalization.

To break this deadlock, the authors formulate validation as a multi-objective optimization problem: maximize performance gains on ID data while preserving performance on retrieved data. Their method, Validation-Enabled Stage-wise Tuning (VEST), selects an operating point on the Pareto front of these two objectives. This enables hyperparameter tuning without test data and, in experiments across five benchmarks, improves both ID and OOD recognition over existing approaches.