Pretrained vision encoders for MRI often struggle to align with downstream tasks, unlike language models where pretraining and task objectives are naturally compatible. This paper explores continual pretraining as a way to close that gap, using a prostate segmentation task as the intermediate step.

The authors took a pretrained ViT encoder and continually pretrained it with Low-Rank Adaptation (LoRA), then used Centered Kernel Alignment (CKA) to measure layer-wise representational shifts. They found that this approach improved performance on 3 of 6 downstream tasks and matched baseline performance on the others.

Feature analysis revealed that the continually pretrained encoder's representations became more similar to those of a fully fine-tuned encoder than the original pretrained encoder's representations were. This similarity correlated with the observed performance gains, suggesting that continual pretraining can help align encoder features with task-specific objectives in medical imaging.