Editing agents that work on 2D infinite canvases face a fundamental problem: pixels do not tell the whole story. Targets can be offscreen or hidden behind other objects, and the visual layout may not match the underlying application state. A new arXiv paper, CanvasObs, tackles this by proposing a structured spatial observation approach.
Instead of relying solely on pixel data, CanvasObs is designed to give agents a more complete view of the canvas state. The paper notes that visual containment and touching arrows can disagree with recorded frame membership, suggesting that raw images are an unreliable guide for editing actions. The proposed method aims to reconcile these discrepancies.
By structuring spatial observation, the approach could make editing agents more robust in environments where the visible scene is only part of the state. The paper is available on arXiv under the identifier 2610.27176.