Evals and benchmarks for agents

Why it pays

• Identical starting points Every task starts from the same template, so results compare fairly.
• Isolation per task A task that breaks its machine can't affect the next one.
• Cost per run, exactly Group usage by label to see what each model, harness or suite cost.

Step by step

1. Save the eval environment as a template

   Install the suite and its fixtures once.

   CLI

   harebox new --name env-base
   harebox exec current -- ./install-eval-suite.sh
   harebox template save current --name eval-env

2. Run a fleet from it

   Hold a number of workers at a desired size, from the template.

   CLI

   FLEET=$(harebox fleet new --name evals --desired 50 --template eval-env | jq -r .id)
   harebox fleet info "$FLEET"

   Python

   fleet = hb.create_fleet(body={"name": "evals", "desired": 50, "template": "eval-env",
                                 "labels": {"suite": "swe-lite"}})

   TypeScript

   const fleet = await hb.createFleet({ body: { name: "evals", desired: 50, template: "eval-env",
     labels: { suite: "swe-lite" } } });

3. Send each task to a worker

   Prompt the whole fleet, or each sandbox with its own task.

   CLI

   harebox fleet prompt "$FLEET" "Run task 17 and write results.json" --harness codex

   Python

   hb.prompt_fleet(fleet["id"], body={"prompt": "Run task 17 and write results.json", "harness": "codex"})

   TypeScript

   await hb.promptFleet(fleet.id, { body: { prompt: "Run task 17 and write results.json", harness: "codex" } });

4. Read the cost per suite, then scale to zero

   Usage grouped by label shows cost per model or harness.

   CLI

   harebox usage --all --group-by label
   harebox fleet set "$FLEET" --desired 0

Tips

• Fork a single prepared sandbox with --count when tasks need a warmed process, not only a disk.
• Set a budget on the suite's label before a large run.
• Starts beyond your plan can wait in a free queue instead of paying the extra-start fee.