AI agents increasingly browse the web on behalf of users, but they often consume pages as screenshots or raw text. That means their underlying language models process whatever third-party content appears on the page, with no way to distinguish legitimate functionality from misleading or malicious elements.
The paper argues that browsers already have a semantic model that defines what an element is and what it can do—a button behaves differently from a link or an input field. The authors propose using these built-in semantics as a constraint on agent actions, rather than relying on visual or textual appearance alone.
This shifts the focus from what a page says to what a page is. By grounding agent decisions in the browser's own semantic interpretation, it may be possible to prevent agents from being tricked by content that looks like one thing but behaves like another. The abstract is truncated, but the core idea is to use the browser's rules as a security boundary.