Vision-language models like CLIP retain strong zero-shot recognition, but real-world deployment often requires them to keep learning from a stream of tasks. In that setting, past data is unavailable and the task identity is not given, which makes standard fine-tuning prone to catastrophic forgetting. A new arXiv paper introduces COMET, a method designed for this task-agnostic continual learning scenario.

COMET's core idea is to decouple three components that are usually tangled in continual learning: knowledge distillation, routing of inputs to appropriate sub-models, and capacity control. By separating these, the model can allocate resources dynamically and preserve previously learned knowledge while adapting to new tasks, without needing explicit task labels.

The abstract is brief, so implementation details are not fully specified in the source. Still, the contribution appears to be a principled framework for making large vision-language models more practical in open-world settings where data arrives sequentially and tasks are not pre-defined.