Self-supervised joint-embedding methods learn by aligning representations of augmented views while preventing collapse, typically by regularizing the distribution of representations. Most approaches compute this regularization only within the current minibatch, which can be a poor approximation when batches are small. The authors propose out-of-batch regularization, which maintains a dataset-level state in CPU memory and constructs global per-instance targets during training. This allows the regularizer to approximate the full-dataset gradient more faithfully than minibatch estimation.
Experiments on natural image and biomedical datasets paired the method with contrastive (SimCLR) and non-contrastive (SIGReg) objectives. In both cases, out-of-batch regularization consistently improved on in-batch baselines across image classification, minority-class performance under controlled imbalance, instance retrieval, and continued self-supervised pretraining. The largest improvements appeared at small batch sizes, indicating that the approach decouples regularizer geometry from batch-size choices.
The study is a single arXiv preprint and has not yet been peer-reviewed. Its findings are promising but would benefit from independent replication and broader evaluation across architectures and data domains. The core idea—that dataset-level regularization can be maintained outside the minibatch—offers a practical direction for making self-supervised learning more robust in memory-constrained settings.