Vision Transformers (ViTs) have become popular for image classification, but they are notoriously data-hungry: they typically achieve strong results only when trained on very large datasets. A new arXiv preprint proposes a way to address this limitation by combining a multi-exit residual architecture with dynamic self-ensembling, aimed specifically at small-data image classification tasks.
The abstract describes a "dynamic self-ensemble on multi-exit residual vision transformer" but provides few implementation details. The core idea appears to be that adding multiple exits with residual connections can help the model learn more effectively from limited data, while a dynamic self-ensembling mechanism stabilises training and improves final predictions.
As the abstract is truncated, exact performance figures and comparisons are not available in the source. The work is positioned as a response to ViT's data-hungry nature, but the full paper would be needed to assess how well the method works in practice.