Continual learning systems must update their parameters as new tasks arrive, but doing so risks forgetting earlier tasks. This paper studies that trade-off in a high-dimensional linear regression setting, where the learner observes a sequence of tasks and updates its parameters sequentially. The authors derive a closed-form expression for the optimal regularization strengths, covering both explicit penalties and implicit regularization.

The result is notable because it moves beyond heuristic tuning: for this model class, the best balance between fitting the current task and retaining past information can be computed exactly. The analysis also clarifies how the optimal strengths depend on problem dimensions, task similarity, and the order in which tasks are seen.

Because the setting is linear, the findings are not directly applicable to deep neural networks. However, they provide a rigorous reference point for understanding regularization in continual learning, and could inform the design of algorithms in more complex regimes.