Visuomotor policies typically assume the target object stays still during execution, but real environments break that assumption: parts drift on conveyors, fruit sways in the wind. The new method, Static In, Dynamic Out (SIDO), addresses this gap by enabling a policy trained only on static demonstrations to handle unseen object motion at test time.

SIDO's core idea is to split moving-object manipulation into two sub-problems: predicting where the object will be, and reaching that predicted pose. During training, the object is displaced to a counterfactual future position and the demonstrated action chunk is morphed to preserve the hand-object relative pose, producing a goal-conditioned policy. At deployment, a separate object pose predictor supplies the future position.

The authors evaluated SIDO across three simulated tasks (Mug, Square, Stack) under five object motion patterns and two real-world tasks (Gantry, Peachtree). They report that SIDO improves moving-object success over baselines while preserving static-object performance, suggesting that counterfactual augmentation is a practical route to robust manipulation without needing dynamic demonstrations.