Computer Use

Computer Use – an agent manipulates a graphical user interface as a human would. Perception-reasoning-action loop: screenshot → multimodal model outputs reasoning and action → execution layer performs (mouse, keyboard, scrolling) → new screenshot. Three dimensions of design: action space (GUI tools + bash + file editing, per Anthropic), visual localization (where the target element is), model architecture. Typical task – 10-20 cycles of 2-5 seconds each: accuracy on OSWorld approached human level, but efficiency (more steps, growing time per step) – an unsolved bottleneck.

Related: [Visual GUI Localization], [Seven Basic Tools], [VLA Models]