GUI agents need to connect natural-language instructions to specific elements in a screenshot. The arXiv preprint Seg2Click (arXiv:2610.16234) takes on this visual-grounding problem, arguing that the common approach of predicting a single coordinate is too lossy. With single-point supervision, a spatially extended target — a button, a list item, a dialog box — is reduced to one point, discarding the very shape that defines it.
Seg2Click instead learns pixel-level action masks. Rather than outputting a point, the model marks the pixels that belong to the target, preserving its full extent. The paper frames this as a way to keep the grounding signal aligned with the actual interactive region in crowded screenshots, where nearby elements can easily be confused.
The announcement is brief, so details on architecture and evaluation are not available in the source. What is clear is the direction: move GUI grounding from point prediction toward dense, pixel-level target identification.