Evals and RL
Works locallyWorks in the local build today.
Many short, identical runs. Use Spot, and let starts wait in a queue instead of failing.
harebox new --name env-base
harebox template save current --name eval-env
FLEET=$(harebox fleet new --name evals --desired 2 --tier spot --template eval-env | jq -r .id)
harebox fleet info "$FLEET"
harebox fleet set "$FLEET" --desired 0
harebox fleet rm "$FLEET"
Pin template@version, and record it with your results. Spot and Shared vary in speed, because they're shared. If timing itself is what you measure, use xlarge (whole threads) or record cpu_steal_percent with each result.