Most robotic manipulation systems learn action representations from the structure of the action stream itself, treating raw control trajectories as noisy sequences to compress or model. A new arXiv paper argues this misses something central: manipulation is about changing the world, and the same action segment can have different meanings depending on the scene context.
The proposed method, EDAR (Environment-Dependent Action Representation), grounds action tokens in both executable control structure and expected visual consequences. By coupling motor commands with their environment-conditioned effects, the learned action space captures interaction semantics rather than merely command-level patterns.
In experiments on simulated and real-robot manipulation benchmarks, EDAR improved downstream policy learning, with the largest gains in long-horizon manipulation. The authors highlight the importance of grounding action representations in both executable control and environment-conditioned visual change. Since this is a single source, no conflicting findings are noted.