Robot policies often rely on memory that evolves with each new observation. This design implicitly assumes that every update corresponds to the same amount of physical time, so a memory state is tied to how many steps have passed rather than how much time has actually elapsed.
The arXiv paper points out that this assumption can fail in the real world. Identical observation histories may occur over very different durations, and a robot that has watched a scene for two seconds may need to act differently than one that has watched the same scene for ten seconds. Ignoring physical time could leave the robot with a memory state that does not reflect the true situation.
The authors suggest treating physical time as a missing dimension in robot policy memory dynamics. By incorporating elapsed time explicitly, memory can better represent the temporal context of observations, potentially improving decision-making in tasks where timing matters.