Visual GUI Localization

How to find an element in a screenshot.

Approach with answer options:

• Set-of-Mark — pixel segmentation and numbered labels;
• structured indexing — DOM/accessibility tree (browser-use: interactive elements are numbered, the model chooses the ID).

This transforms the open task of “find and predict coordinates” into a closed “select from labeled” task.

Direct coordinate prediction — SeeClick, Claude Computer Use: the model “looks” and names the coordinates; requires two-way scaling to the training resolution (selection by aspect ratio).

Rule of choice: where structure is available — indexing; native applications, Canvas, games — visual markup or coordinates.

Related: Computer Use, Dense Embedding