Most spatial augmented reality systems assume the projector pose, target region, or motion path is fixed in advance. In a new arXiv preprint, Deng and colleagues propose LICOA, which instead treats projected light as an embodied action: given a language instruction and current visual observations, the system must decide how to move, where to aim, and at what scale to project.

The hardware uses a differential-drive chassis for room-scale repositioning and a two-degree-of-freedom steerable mirror plus projection scale for fast local beam control. A shared interface aligns synchronized external and onboard RGB cameras with robot control state, feeding pretrained action models. Because instruction-conditioned optical action lacked dedicated training data and a common evaluation protocol, the authors built a matched simulation embodiment and a corpus of 150 tasks with 7,500 rendered episodes.

Models trained only on that corpus were tested in previously unseen real-world environments. Across 480 closed-loop rollouts on 12 tasks, the best adapted action model achieved 57.50% success and 40.64% success weighted by path length, versus 8.33% and 5.15% for the same architecture trained from scratch. An additional 150 real-world ablation rollouts indicated that both camera views and coordinated control of the chassis and mirror are needed.