Visual GUI Localization

How to find an element in a screenshot. Approach with answer options: Set-of-Mark — pixel segmentation and numbered labels; structured indexing — DOM/accessibility tree (browser-use: interactive elements are numbered, the model chooses the ID). This turns the open task of “find and predict coordinates” into a closed “select from labeled”. Direct coordinate prediction — SeeClick, Claude Computer Use: the model “looks” and names the coordinates; requires bidirectional scaling to the training resolution (selection by aspect ratio). Rule of choice: where structure is available — indexing; native applications, Canvas, games — visual markup or coordinates.

Related: Computer Use, [Dense Embedding]