Video object trackers keep a memory bank of past frames to maintain temporal consistency across complex scenes. The Segment Anything Model 3 (SAM 3) tracker, inherited from SAM 2, manages this bank with a simple first-in-first-out rule. Recent methods have proposed hand-crafted memory update rules to address specific failure modes such as distractors or occlusions, but those gains transfer only partially to the stronger SAM 3 backbone.

The authors instead treat memory control as a sequential decision-making problem. They train a lightweight controller with reinforcement learning to decide which frames enter the memory bank, with the goal of maximizing tracking performance. The resulting system, SAM3RL, sets a new state of the art on 10 of 12 tracking and video object segmentation benchmarks, outperforming all hand-crafted rules without adding runtime overhead.

The same approach applied to SAM 2, called SAM2RL, outperforms hand-crafted rules on 9 of the 12 benchmarks, suggesting the method generalizes across model generations. This summary is based on a single arXiv preprint; no independent verification or contrasting sources were included.