# Benchmark method

## Principles

1. **Published whatever the result.** A slow or costly run is published like any other.
2. **Repeatable by anyone.** Code, configuration and raw data are public.
3. **Dated and complete.** Every result names its date, site, tier and host class. Results from different dates are marked as indicative.
4. **No special treatment.** Runs use ordinary accounts and the normal product. Sandboxes are on the tier named, on ordinary hosts.

## What is measured

| Benchmark | Measures | Reported as |
|---|---|---|
| [hpc-sandbox-benchmarks](https://github.com/AnicetNgrt/hpc-sandbox-benchmarks) | Node build and test loop, 4 vCPU / 8 GB | Runs/s and dollars per million runs |
| Start | `POST /sandboxes` to first command | p50, p95, p99 per site and kind of host, by template size |
| Wake | Wake to first command, same host and other host | p50, p95, p99 |
| Fork | Fork to first command on the fork | p50, p95, p99 |
| Single-core | Fixed single-thread workload | Ops per second, per tier |
| Noisy neighbour | Your latency while neighbours are busy | p99 inflation, per tier |
| Disk | fio random and sequential; `fsync` latency | IOPS and latency |
| Network | Egress, inter-VM on mesh | Throughput and latency |
| Desktop | Added latency and frame rate at 1080p60 | Milliseconds, frames per second |
| Idle reclaim | Share of `ready` time that sleep removed | Percent, fleet-wide, from real use |
| Reliability | External uptime probe | Monthly per tier |

## Rules

- At least 30 runs on at least 5 different hosts per site per tier, with confidence intervals.
- Tier and host class are reported with each result.
- Results run continuously, on a schedule that is also public.
- A change in method restarts the series, and is noted.

## The first result

The first published figure will be the hpc-sandbox-benchmarks Node loop on the `default` size, 4 vCPU and 8 GB on Shared at $0.03564/h, reported as runs per second and dollars per million runs, with the host's CPU model. Until it runs on rented hosts, every speed figure HareBox gives is a target, not a measurement.