Computer Use

Computer Use – agent manipulates a graphical user interface as a human would.

Perceive-deliberate-act loop: screenshot → multimodal model outputs reasoning and action → execution layer performs (mouse, keyboard, scrolling) → new screenshot.

Three dimensions of design:

action space – GUI tools + bash + file editing (per Anthropic);
visual localization – where the target element is;
model architecture.

Typical task – 10–20 loops of 2–5 seconds each: accuracy on OSWorld approached human level, but efficiency (more steps, growing time per step) – an unsolved bottleneck.