Benchmark method
Partly builtSome of this works today; the page says which parts.
- Published whatever the result. A slow or costly run is published like any other.
- Repeatable by anyone. Code, configuration and raw data are public.
- Dated and complete. Every result names its date, site, tier and host class. Results from different dates are marked as indicative.
- No special treatment. Runs use ordinary accounts and the normal product. Sandboxes are on the tier named, on ordinary hosts.
| Benchmark | Measures | Reported as |
|---|---|---|
| hpc-sandbox-benchmarks | Node build and test loop, 4 vCPU / 8 GB | Runs/s and dollars per million runs |
| Start | POST /sandboxes to first command |
p50, p95, p99 per site and kind of host, by template size |
| Wake | Wake to first command, same host and other host | p50, p95, p99 |
| Fork | Fork to first command on the fork | p50, p95, p99 |
| Single-core | Fixed single-thread workload | Ops per second, per tier |
| Noisy neighbour | Your latency while neighbours are busy | p99 inflation, per tier |
| Disk | fio random and sequential; fsync latency |
IOPS and latency |
| Network | Egress, inter-VM on mesh | Throughput and latency |
| Desktop | Added latency and frame rate at 1080p60 | Milliseconds, frames per second |
| Idle reclaim | Share of ready time that sleep removed |
Percent, fleet-wide, from real use |
| Reliability | External uptime probe | Monthly per tier |
- At least 30 runs on at least 5 different hosts per site per tier, with confidence intervals.
- Tier and host class are reported with each result.
- Results run continuously, on a schedule that is also public.
- A change in method restarts the series, and is noted.
The first published figure will be the hpc-sandbox-benchmarks Node loop on the default size, 4 vCPU and 8 GB on Shared at $0.03564/h, reported as runs per second and dollars per million runs, with the host's CPU model. Until it runs on rented hosts, every speed figure HareBox gives is a target, not a measurement.