Skip to content
anythingai.space
Tool Use & Computer Agentsintermediate13 min · 9 steps

Computer-Use Agents

Let a model drive a GUI through screenshots and clicks: the loop, the sandbox, what each step costs, and how it fails.

no schemalook + clickAgentClaims APIdoesn't existLegacy web portalscreen only
signaldatarequesttool call
1 / 9
ProblemStep 1 of 9

The system you need has no API, only a screen

Tool calling needs a predefined API with a schema. Plenty of real work lives in legacy portals, thick-client apps and sites that block scrapers, where all you have is a screen.

A computer-use agent takes the human's place: it reads screenshots and issues mouse and keyboard input. It works with any app that has a visual interface.

Lesson 4 of 7 in Tool Use & Computer Agents