Computer use
Partly builtSome of this works today; the page says which parts.
Agents can see and operate the sandbox's desktop: take screenshots, click, drag, type, press keys, scroll, open web pages and apps, and read the screen's accessibility tree. Use it through the API from your own code, or let a harness inside the sandbox use it through MCP.
Status: works on real VMs, checked by HareBox's real-VM tests. On the labelled simulator, actions return placeholders.
harebox new --name cu
harebox computer current --action screenshot
harebox computer current --action click --x 960 --y 540
| Action | Fields | Does |
|---|---|---|
screenshot |
Returns the 1920×1080 screen as a base64 PNG in screenshot_png |
|
click, double_click, right_click |
x, y |
Clicks at a point |
move |
x, y |
Moves the pointer |
drag |
x, y, to_x, to_y |
Presses, moves and releases |
type |
text |
Types into whatever has focus |
key |
keys |
Presses keys, such as ["ctrl+l"] or ["Return"] |
scroll |
x, y, delta |
Scrolls at a point; positive delta scrolls down |
open_url |
url |
Opens an http or https page in Chrome |
open_app |
text |
Starts a desktop application, such as xfce4-terminal |
accessibility_tree |
Returns every element's role, name and screen bounds |
The first action starts the desktop if it isn't running. Coordinates are in pixels from the top left of the 1920×1080 screen.
Every harness in the base image is configured with HareBox's computer MCP server (harebox-computer mcp), so Claude Code, Codex and OpenCode can use the desktop with no setup. Ask for it in plain words:
harebox prompt current "Open https://example.com in Chrome, take a screenshot, and tell me what the page says" --harness claude --budget 1
The tools are screenshot, click, double_click, move, drag, type, key, scroll, open_url and accessibility_tree. To use them from another agent in the sandbox, run harebox-computer mcp as a stdio MCP server.
Each action runs as the sandbox's user account, and arguments reach programs as argument lists, never through a shell. Typed text is passed on standard input. Pages an agent opens can contain instructions meant to mislead it, so give agents that browse a budget and a firewall that allows only the sites they need.