Operate
Benchmark
This benchmark has been run. On 2026-09-22, one script pushed one deterministic
remote-write workload into Prometheus 3.1.0, into VictoriaMetrics single-node v1.152.0 and
into the Orbit Stack engine on the same machine, sequentially, and measured disk, ingest
throughput, resident memory and query latency on all three. Every engine figure in
CLAIMS.md comes from those runs, the reports are recorded under
services/rust/orbit-stack/bench/runs/, and the script that produced them is
services/rust/orbit-stack/bench/run.sh. On 2026-09-23 a second harness,
services/rust/orbit-stack/bench/soak.sh, ran the engine against VictoriaMetrics the way
VictoriaMetrics/prometheus-benchmark runs a database, with churn and an alerting-rules
suite; its first run is below, under its own heading, with the loss it found. A third,
services/rust/orbit-stack/bench/traces.sh, compares the engine's traces with Grafana Tempo's;
its second round, as an independent critic re-ran it, is below with its losses, and the
earlier trace figures are withdrawn. This page is the method as it was executed, the
results, and what they do and do not say.
Every figure on this page outside the 2026-10-03 section was measured on the Go engine, which
now lives at the git tag go-oracle. The engine that ships is the Rust one in services/rust/orbit-stack, and the
scripts measure it. This repository no longer builds Go, so the commands below run the same
workloads against the Rust engine, not the figures' engine. The Rust engine's own
runs are recorded under bench/runs/ too (their names carry rust); this page prints only
the ones the 2026-10-03 section names.
2026-10-03: the Rust stack, before and after a performance round
One builder per product, each measured against its direct competitor, then reviewed: report
services/rust/orbit-stack/bench/runs/2026-10-03-perf-round.md. The machine was shared by all
six at once (load 6 to 32), so the builders' competitor columns stay in that report. This
page prints only the rows a reviewer re-ran afterwards, back to back, at load 4 to 7.
| Figure, re-checked | Before | After | Competitor in the same run |
|---|---|---|---|
| Trace search for an attribute one trace has, 1.13 M spans, p50 | 447 ms | 8.0 ms | none |
| Trace search for an attribute no trace has, 1.13 M spans, p50 | 427 ms | 6.1 ms | none |
PromQL rate(...[5m]) with a 3.5 MB answer, 20k series, p50 | 19.1 ms | 11.2 ms | none |
PromQL sum by (instance) (rate(...[5m])), 20k series, p50 | 7.9 ms | 5.3 ms | none |
| orbit-collector, traces over gRPC at ~44k spans/s, CPU cores | 0.36 to 0.39 | 0.25 to 0.26 | otelcol-contrib 0.161.0: 0.49 to 0.59 |
| orbit-collector, traces over HTTP, CPU cores | 0.33 to 0.35 | 0.23 | otelcol-contrib: 0.55 to 0.69 |
| orbit-collector, RSS | 9 MiB | 10 MiB | otelcol-contrib: 219 to 221 MiB |
| orbit-agent, 500k series, steady RSS | 257 MB | 90 to 93 MB | none |
Not won in the re-check: orbit-agent's CPU per sample (a win in one round, a tie in the other), and its WAL, which still writes 48.8 B per sample where Prometheus' agent mode and Alloy write 5.3. The trace rows were not re-run at 3.4 M spans, where the losses in "Traces against Grafana Tempo" were measured, so those loss rows stand.
Results
Prometheus's own defaults: 2 h blocks on both engines that have them. 20,000 active series, 4 h of virtual time at a 15 s scrape, 19.2 M samples per engine, pushed as fast as each engine accepted them.
| Figure | Prometheus 3.1.0 | VictoriaMetrics v1.152.0 | Orbit Stack | vs Prometheus | vs VictoriaMetrics |
|---|---|---|---|---|---|
| Disk per sample, sealed | 5.38 B | 2.24 B | 1.87 B | 2.9x | 1.2x |
| Disk per sample, whole data directory | 11.45 B | 2.24 B | 1.87 B | 6.1x | 1.2x |
| Ingest accepted, samples/s | 467k | 567k | 555k | 1.19x | 0.98x (parity) |
| RSS with the head full (20k series x 4 h) | 257 MiB | 263 MiB | 105 MiB | 2.4x less | 2.5x less |
| Live Go heap at that moment | 165 MiB | 181 MiB | 46 MiB | 3.6x less | 4.0x less |
| RSS after the head is sealed to a block | n/a | n/a | 29 MiB | n/a | n/a |
query_range p50 / p90 over 60 runs, answers only | 97 / 108 ms | 16 / 20 ms | 24 / 33 ms | 4.0x / 3.3x faster | 1.5x / 1.7x slower |
query_range max (first query loads the frames it matches) | 152 ms | 33 ms | 102 ms | 1.5x faster | slower |
| Write errors | 0 | 0 | 0 | n/a | n/a |
VictoriaMetrics caches rollup results by default; the row above is the cached number and the
same query with nocache=1 answers in 16.1 / 19.9 ms, so the cache is not what wins it. Neither
Prometheus nor Orbit Stack caches a query response at all. The engine's query_range row was
corrected on 2026-09-23: the run timed the read limiter's 429 refusals together with the
answers, and timed on answers only it is the 24 / 33 ms above (CLAIMS.md, from the
re-check in bench/runs/2026-09-23-continuity-critic/).
At 200,000 series (2 h of virtual time, 96 M samples per engine, 2 h blocks, so Prometheus
had not cut a block yet): ingest 582k against VictoriaMetrics' 520k and Prometheus' 429k,
2.20 B per sample against 2.33 B and 13.54 B. query_range p50 is 68 ms against Prometheus'
506 ms and VictoriaMetrics' 54 ms: 1.3x behind its cached answer and 1.4x ahead of its
uncached one (93 ms). The first query after the seal is 259 ms against VictoriaMetrics' 248 ms. RSS
is 511 MiB against Prometheus' 856 MiB and VictoriaMetrics' 312 MiB: the engine beats
Prometheus at this scale and is 64% above VictoriaMetrics, which is the largest remaining
gap. The head drops to 77 MiB the moment it is sealed. Report:
bench/runs/2026-09-22-200k-2h-2hblocks-bursty.md.
The tables above are the script's default entropy, counter noise 0.3 (the bursty profile, see
Workload). The steady profile, noise 0.05, at 20,000 series over 4 h
(bench/runs/2026-09-22-20k-4h-2hblocks-steady.md): 5.215 B against VictoriaMetrics' 1.983 B
and the engine's 1.724 B on sealed blocks, ingest 363,658 / 485,841 / 513,798 samples/s, RSS
214 / 235 / 141 MiB, query p50 103.5 / 15.1 / 25.4 ms. Steadier counters compress better for
every engine, and the ordering does not change.
A second pair of runs put 15-minute blocks on Prometheus and the engine over 2 h of virtual
time, 9.6 M samples (bench/runs/2026-09-22-20k-2h-15mblocks-steady.md and
...-15mblocks-bursty.md). Sealed blocks: 3.58x and 3.28x against Prometheus, 1.18x and 1.22x
against VictoriaMetrics. Ingest against VictoriaMetrics is 0.98x on the steady run,
a loss, and 1.06x on the bursty one. Query p50, the three engines in order: 66.0 / 10.0 /
12.0 ms steady and 65.5 / 10.0 / 13.0 ms bursty. Prometheus is the smallest of the three in
RAM there, 87 and 91 MiB against the engine's 122 and 121: fifteen-minute blocks keep its head
short, while VictoriaMetrics sizes its caches from the machine's memory rather than from the
data, so a small dataset does not make it small (176 and 213 MiB). Short blocks cost
Prometheus index overhead per block, which is why the 2 h figures above are the ones
CLAIMS.md prints: they are the ones that favour Prometheus.
Read the two disk rows this way. Sealed blocks are the steady state: what disk costs per sample after the write-ahead log has been checkpointed and the head has been persisted. That is the 2.9x against Prometheus and the 1.2x against VictoriaMetrics. The whole-directory row is a snapshot during ingest: Prometheus keeps the last two to three hours in a write-ahead log at roughly 20 bytes per sample, VictoriaMetrics had been flushed and merged, and the engine's WAL is empty after a seal. That ratio against Prometheus is real while data is arriving and shrinks as retention grows, so it is printed only here and only with "during ingest" next to it.
Where the engine still loses. Three rows of these runs, and they are printed wherever their wins are; the soak below adds a fourth, the data directory while the data is in the head.
VictoriaMetrics answers the 20,000-series query 1.5x faster at p50 and 1.7x at p90 (16 ms against 24, 20 against 33). At 200,000 it is 1.3x faster than the engine with its response cache and 1.4x slower without it, because the work there is dominated by the series the query touches and the column-wise evaluator reduces each one in a single pass. What is left at 20,000 is fixed cost per query rather than per-sample work.
RSS at 200,000 series is 64% above VictoriaMetrics (511 MiB against 312). The head keeps one open chunk per series in the Go heap and spills the closed ones; VictoriaMetrics keeps almost nothing in memory and reads from its parts.
Ingest at 20,000 series is parity, not a win (555k against 567k, 0.98x). At 200,000 the engine is ahead, 582k against 520k.
Engine rounds: what changed between the reports
Round 1 was the read path, round 2 the write path and the memory it costs. Prometheus and
VictoriaMetrics were re-measured alongside the engine in each session, so their columns also
show what this shared machine's noise is worth. The reports named *-before-engine-round1.md
under bench/runs/ are the engine as it stood before any of this, measured in the same
session as the round-1 ones.
Only two states of the engine have reports under bench/runs/, so only those two are given
numbers: *-before-engine-round1.md, the engine as it stood before any of this work, and the
two bursty runs the tables above come from. The steady before-and-after pair is at 20,000
series over 4 h and at 200,000 over 2 h; the shipped column is the bursty profile, so read
each row against its own engine columns and not across the pair. The shipped 20,000-series
p50 is the corrected, answers-only figure.
| before round 1 (steady) | Prom | VM | shipped (bursty) | Prom | VM | |
|---|---|---|---|---|---|---|
query_range p50, 20k | 104.2 ms | 110.0 | 16.6 | 24.3 ms | 97.1 | 15.9 |
query_range max, 20k | 577.5 ms | 133.7 | 48.1 | 102.2 ms | 152.1 | 32.6 |
query_range p50, 200k | 1281.5 ms | 747.6 | 60.6 | 67.7 ms | 505.9 | 53.5 |
query_range max, 200k | 4316.7 ms | 1544.6 | 219.2 | 258.5 ms | 608.2 | 248.0 |
| Ingest, 20k | 436131/s | 379425 | 453422 | 554670/s | 467338 | 566680 |
| Ingest, 200k | 376004/s | 338722 | 387317 | 581530/s | 429263 | 520218 |
| RSS, 200k | 1017 MiB | 750 | 325 | 511 MiB | 856 | 312 |
| Live heap, 200k | 825 MiB | 406 | 210 | 288 MiB | 592 | 218 |
| Disk per sample, sealed, 20k | 1.676 B | 5.232 | 2.062 | 1.871 B | 5.381 | 2.243 |
The work between those two states came in rounds, and what each round did is below. The
intermediate measurements were taken with the same script but their results.md files were
not kept, so they carry no numbers here.
Round 5, the head chunk spill. A series' closed chunks were written to a memory-mapped
file under the data directory instead of staying in the Go heap, and the head kept only the
open chunk per series. Against the before-round-1 report at 200,000 series that is RSS 1017
to 511 MiB and live heap 825 to 288 MiB, which is what put the engine below Prometheus at
that scale.
That spill grew 8 MiB at a time and was deleted when the head was sealed, and run.sh reads
the data directory after the seal, so there the directory was its blocks plus a short
write-ahead log and the whole-directory disk row stayed a win. The soak below reads the
directory while the data is still in the head, and there it is a loss.
The spill has been redesigned twice since those reports. First its segments were read with
pread and never mapped, each series writing into 2 KiB extents of its own; that is the
build the soak below measured. Now the chunks of every series are packed into shared
segments with about 11 bytes of header each, the WAL writes each series' labels once per
file, and blocks are compacted in the background (orbit-stack.md has the design). No
report under bench/runs/ measures the current build yet, so every engine figure on this
page is one of the two earlier ones, and each says which.
Round 4, the evaluator. rate, increase, delta and irate over a range selector,
alone or under one sum/avg/min/max/count, are computed series-major: one pass per
series fills every step, the window ends are cursors rather than binary searches, the group of
a series is resolved once instead of at every step, and no per-step vector is built. Anything
else falls back to the general evaluator. The equivalence tests caught two float-associativity
bugs before they shipped (a prefix sum over counter resets and per-worker partial
aggregates), and both are now summed in the general evaluator's order, so the answers
are the same bits, not merely the same to fifteen digits. internal/promql/columnwise_test.go (in
the Go engine, tag go-oracle)
runs every expression named in the PromQL suite plus randomised series with resets, gaps and
NaN through both evaluators and compares sample for sample. The two were also compared over a
real 200,000-series directory by diffing the responses of both builds, which matched byte for
byte; that was a one-off check with no artifact kept, so it carries no figure here.
Round 3, the block format. A sealed block's columns are no longer one zstd stream: they
are independent frames of 4,096 series, the index records where each frame is and where each
series sits inside its own frame, and a query decompresses, in parallel, only the frames its
matchers land in, into a process-wide cache bounded by ORBIT_STACK_BLOCK_CACHE_BYTES. The
width trades disk for speed and was measured: a zstd frame starts its context from
nothing, so narrow frames cost compression. The width was chosen by sweeping it at 20,000
series and reading disk per sample against the first query after a seal: narrow frames cost
noticeably more disk for a little more speed, and 4,096 was the widest setting that still cut
the cold read. That sweep used the same script but its results.md files were not kept, so no
figure from it is printed here. The format is versioned: this build writes v3, which
checksums its index and every frame, and still opens a v1 or v2 block, reading it unverified
and counting it in orbit_stack_unverified_blocks; a v1 opens as the single frame it is.
There are tests for each and for a version it cannot read.
Round 2, the write path. The remote-write decoder carves label sets and samples out of
arenas it keeps between requests and interns every name and value within a request; the head
interns again across requests, so one instance string is shared by that instance's ninety
series; a sample of a series that already exists costs no allocation at all; the WAL's record
and compression buffers are pooled; and the head's chunks grow by an eighth instead of by
append, which was leaving a third of the head in slack. BenchmarkIngestRequest in
internal/tsdb measures what one 5,000-sample request allocates, and is how the change was
judged; it is a Go benchmark, go test -bench, not a run of bench/run.sh, so its figures
are not printed here. Across a whole run the effect shows up in the line every report prints under
its table, the total objects the process allocated. That bought the ingest rows and a fifth of
the live heap; it did not buy the RSS row, because the RSS row was never churn.
What the 200k RAM row is. Since the spill, the Go heap holds one open chunk per live series plus that series' labels and its place in the inverted index; every chunk the series has closed was in the memory-mapped file instead, and is in the spill segments now. What the shipped 200,000-series report prints is 511 MiB resident with 288 MiB of live heap while the head is full, and 77 MiB resident once the head is sealed. The difference between those two is what a full head costs. It scales with live series rather than with the samples they hold, which is why the block duration no longer moves it.
Replay is bounded now. A pod that restarts after its write-ahead log has grown for hours
used to replay all of it into the head and hold it: that is an OOMKill at a 1 GiB limit, and
it happened on k3d. The replay decodes record by record into reusable buffers, and when what
it replayed spans more than one block duration the head is sealed before the first request is
served, with the span and the seal in the log. TestReplayWiderThanABlockSealsOnOpen replays
three hours against a thirty-minute window and fails if the head still holds it.
The prometheus-benchmark soak
services/rust/orbit-stack/bench/soak.sh runs the load of VictoriaMetrics/prometheus-benchmark
at a pinned commit: node_exporter 1.4.0 behind a 1 s cache, multiplied into as many targets
as the scale needs, scraped every 10 s by vmagent, which relabels a slice of the targets
every few minutes (series churn) and remote-writes to the engine under test, while vmalert
evaluates the chart's 40 alerting rules against it every 10 s. The chart's components run
as checksummed release binaries next to the engines rather than in k3d, and each engine
runs in its own systemd scope with a memory cap and a CPU set; the run's summary.md lists
every deviation from the chart. It measures, per scale, sealed bytes per sample, the whole
data directory per accepted sample while the data is live, RSS (steady median and peak),
sustained ingest, CPU per sample, a maximum-ingest blast, and the latency of the rules and
of the continuity query.
- Pair mode (
soak.sh pair <scale> <reps>, and the pair cells ofofficial) runs Orbit Stack and VictoriaMetrics at the same time behind one vmagent, so both receive identical bytes, each with the same memory cap and a CPU set that shares no physical core with the other's; the two sets are swapped every rep. - The Prometheus pass follows each pair cell (
PAIR_PROM=1, the default): Prometheus alone with the same churn schedule, targets, blast seed and cap rules. node_exporter's values are live, so its input matches the pair's in shape, not byte for byte. recheckreplayed the continuity query the way the first benchmark ran it (the same blast, a flush, then the query back to back with each binary's default read limiter) against the loop-1 binary and the current one, alternating their order per rep, and kept every response's status, so a timing of a429could be told from a timing of an answer. That is how the 20,000-series query row above was found to include refusals and corrected. It built the loop-1 Go engine, so it left the script with Go on 2026-10-19.- Profiles:
official(10 minutes of warm-up and 2 h measured per cell),quick(5 minutes and 30 minutes, direction only) andsmoke(proves the harness, never a result).VALUES=diversegives every target its own values, because cloned targets compress unrealistically well;officialruns a diverse pair beside each chart pair and refuses to start on a machine thatsoak.sh quietdoes not call quiet.
The one run of it in the tree so far is bench/runs/2026-09-23-pb-quick-pair/: the quick
profile, three reps of the pair at each scale, no Prometheus pass, on a busy machine (the
report prints the load of every rep). It measured the engine as of dd0e579 with a
limiter patch the report names, which predates the WAL, spill and compaction of this
release. Read it for direction only. Cells are VictoriaMetrics / Orbit Stack, medians of
three reps, with Orbit Stack divided by VictoriaMetrics; below 1 is better for the engine.
| Measure | 20,000 series | 200,000 series | 1,000,000 series |
|---|---|---|---|
| Sealed bytes per sample | 2.742 / 0.547 (0.20x) | 2.472 / 0.463 (0.19x) | 2.741 / 0.364 (0.13x) |
| Whole data directory per accepted sample, live | 2.648 / 27.112 (10.24x) | 2.465 / 23.966 (9.72x) | 2.739 / 23.965 (8.75x) |
| RSS steady median, MiB | 95 / 42 (0.44x) | 266 / 173 (0.65x) | 880 / 678 (0.77x) |
| RSS peak, MiB | 124 / 50 (0.41x) | 406 / 207 (0.51x) | 1,334 / 793 (0.59x) |
| Engine CPU per sample, µs | 12.57 / 7.38 (0.59x) | 6.73 / 4.62 (0.69x) | 6.60 / 4.84 (0.73x) |
| Max ingest, samples/s | 607,145 / 529,343 (0.87x) | 546,969 / 558,503 (1.02x) | 515,626 / 557,130 (1.08x) |
| Rules p50, ms | 1.29 / 0.79 (0.61x) | 5.86 / 3.10 (0.53x) | 23.44 / 15.42 (0.66x) |
| Rules p99, ms | 8.99 / 5.41 (0.60x) | 71.78 / 38.11 (0.53x) | 398.55 / 188.99 (0.47x) |
Both engines kept up with the offered load in every rep, so sustained ingest is a tie. Max
ingest is a tie too: VictoriaMetrics was ahead in all three reps at 20,000 series, and the
ranges overlap at the other two scales. The 20,000-series rules p99 ranges overlap as
well. The sealed-bytes row is inflated for both engines by the cloned targets, which is
what VALUES=diverse is for.
The loss: the data directory while the data sits in the head, 8.75x to 10.24x
VictoriaMetrics' bytes per sample. Two things made it: the WAL logged every remote-write
request with its full label sets, and the spill wrote each series' short chunks into a
2 KiB extent of its own. This release rewrote both: series written once per WAL
file, sample columns of deltas and XORs, chunks packed into shared segments. No report
under bench/runs/ measures the result yet.
Traces against Grafana Tempo
Round 2, re-run by an independent critic. Report:
services/rust/orbit-stack/bench/runs/2026-09-24-traces-vs-tempo.md, section "Round 2, re-run by
an independent critic", with the critic's summaries in
bench/runs/2026-09-24-traces-vs-tempo/round2-critic/. The critic re-ran round 2's build
blind with bench/traces.sh's method: both engines up at once, reads interleaved repetition
by repetition, Tempo -target=all with its span-metrics and service-graphs processors
writing to Prometheus v3.1.0 and its limits raised, and Tempo read the way Grafana reads it
(a trace by id in protobuf, a panel's queries at once). Workloads: 50,000 shop-shaped traces
plus one of 10,000 spans (3 rounds), 300,000 traces = 3.4 M spans (1 round), and 20,000
traces all in the head (2 rounds); p50 of 20 repetitions, medians over rounds. The machine
was shared: load averages 14 to 35 on 12 CPUs. The fixes of round 3 (a memory bound, the
service-graph pairing) were not benchmarked: these are round 2's figures.
Withdrawn, and never printed here: round 1's trace-by-id ratio, which compared Tempo's
JSON path; round 1's claim that no trace row was a loss; round 2's own first ratio over 200
ids, which read 20 ids; round 2's own first ingest ratio at 50,000 traces, which the re-run
did not reproduce; and every services-summary and service-map ratio at 3.4 M spans, because
both engines' answers were incomplete there: Tempo's span metrics held a fraction of the
spans, and Orbit Stack's service graph dropped pairings at a fixed ceiling that round 3 made a
setting (ORBIT_STACK_TRACE_GRAPH_PENDING).
Wins and losses are printed together. Every row of the first table is read with the second table beside it.
| Figure | Grafana Tempo 3.0.0 | Orbit Stack | Ratio |
|---|---|---|---|
| Trace by id, one trace of 10,000 spans, p50 (50,000 traces) | 363 ms (protobuf) | 38 ms | 9.5x faster |
| Trace by id, 10-span traces, p50 over 200 ids (50,000 traces) | 40.6 ms | 0.90 ms | 45x faster |
| Search for common values (service; error; service and duration; attribute), Tempo's default mode, 50,000 traces | 19.9 / 23.1 / 22.8 / 35.3 ms | 1.13 / 1.02 / 1.11 / 1.18 ms | 18x to 30x faster |
| Same searches at 3.4 M spans | 19.0 / 47.6 / 38.0 / 73.6 ms | 1.56 / 1.59 / 1.46 / 1.94 ms | 12x to 38x faster |
| Ingest accepted, spans/s (50,000 traces; 3.4 M spans) | 197,481; 55,186 | 269,185; 257,594 | 1.4x; 4.7x |
| Container memory 45 s after ingest (50,000 traces; 3.4 M spans) | 680 MiB; 2,102 MiB | 356 MiB; 263 MiB | 1.9x; 8.0x less |
| Services summary (rate, errors, p50/p95/p99, pods per service), p50, 50,000 traces | 4.04 ms (PromQL over its generator's series); 6.73 ms (TraceQL metrics) | 2.92 ms | 1.4x; 2.3x faster |
| Service map, p50, 50,000 traces | 4.32 ms (PromQL over its generator's series) | 3.20 ms | 1.4x faster |
| Components to run for traces with a service map | 2 (Tempo and Prometheus) | 1 | n/a |
| Loss | Grafana Tempo 3.0.0 | Orbit Stack | Ratio |
|---|---|---|---|
| Search for an attribute value one trace has, 3.4 M spans | 11.2 ms | 677 ms | 60x slower |
| Search for an attribute value no trace has, 3.4 M spans | 9.0 ms | 623 ms | 69x slower |
| Search for an absent status on one service, 3.4 M spans | 105 ms | 544 ms | 5.2x slower |
Services summary, the first read right after /-/flush, everything in the head | 4.0 ms | 24.5 / 30.5 ms (2 rounds) | 6x to 8x slower |
Service map, the first read right after /-/flush, everything in the head | 4.0 ms | 16.3 / 67.6 ms (2 rounds) | 4x to 17x slower |
Search wins hold for common values only. A search win holds for values that are common in recent data: the search reads summaries newest first and stops once it has enough. A value that is rare or absent makes it read every summary in the window, and at 3.4 M spans that is 5x to 69x slower than Tempo, as the loss table shows. At 50,000 traces the same rare-value search is 2.6x faster.
On 2026-10-03 the rare-value search started skipping every split trace and summary chunk its attribute blooms rule out: at 1.13 M spans the rare-attribute search went from 447 to 8.0 ms (the 2026-10-03 section). It was not re-run at 3.4 M spans, so the loss rows above stand until it is.
Newest first. Tempo's default mode returns the matches it finds first; asked for the
newest (most_recent=true), which is what Orbit Stack's search always returns, Tempo is 30x
to 689x slower. That comparison belongs beside the default-mode rows above, never instead of
them.
The first read after a flush. The services summary and the service map read right after
/-/flush are slower than Tempo's median: the flush also seals the metrics head, so they are
the first PromQL over a freshly sealed block (the last two rows of the loss table).
Disk is not claimed: when it was measured, Tempo's data was its write-ahead log and live-store, not compacted blocks. These tables come from one shared machine under the load it had; a re-run on a quiet machine replaces them instead of being averaged into them.
Environment
The run.sh reports above: one machine, the three engines in turn:
- AMD Ryzen 5 5500, 12 threads, 31 GiB RAM, WSL2 (Linux 6.6), data directories on the distribution's ext4 root filesystem (an earlier pair of runs on the 9p-mounted Windows drive gave the same disk figures and slower I/O on both sides).
- Prometheus 3.1.0 from
prom/prometheus:v3.1.0in Docker,--web.enable-remote-write-receiver,--storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h, no scrape targets, so every stored sample was one the generator sent. - VictoriaMetrics single-node v1.152.0 from
victoriametrics/victoria-metrics:v1.152.0in Docker,-storageDataPathon a mounted volume,-retentionPeriod=30d, everything else at its defaults, which includes its response cache and its default memory allowance. Its disk is read afterPOST /internal/force_flushandPOST /internal/force_mergehave finished, which the script waits for by pollingvm_active_mergesandvm_pending_rowsto zero. - Orbit Stack built from this repository, run as a plain binary,
ORBIT_STACK_BLOCK_DURATION=2h,ORBIT_STACK_COMPRESSION=better(zstd), authentication disabled for the run.
Both Docker engines are docker rm -f'd before the next run starts, so nothing of one run
is alive during the next.
Disk is du -sb on each data directory after the run settled: Prometheus after its head
compaction cut the 2 h block (the script waits for it), VictoriaMetrics after its flush and
merge, Orbit Stack after POST /-/flush sealed the head. Samples in blocks come from each
block's own metadata for Prometheus and Orbit Stack; VictoriaMetrics has no such metadata, so
its data/ directory (small parts, big parts and indexdb) is divided by the samples sent,
and the report records how many index bytes were still in memory at that moment. RSS is
VmRSS of the engine process and docker stats for the two containers, read right after
ingest (for Prometheus, after the wait for its head compaction, since that wait is what puts
its block on disk, so its figure is the one taken latest). Query latency is 60 sequential
query_range calls of
sum by (instance) (rate(node_cpu_seconds_total{mode="user"}[5m])) over the whole window at
a 60 s step, timed by curl.
This machine is shared with other work. Every report records the one-minute load average it saw at each engine's ingest and at its query loop, on 12 cores. The two runs the tables above come from were taken on a quiet machine: the loads those reports print run from 1.82 to 11.38 across their twelve phase readings. Earlier reports in the same directory were taken while the box was busier and print higher ones, which is why their absolute figures are lower; the three engines were measured minutes apart under comparable load, and a run whose three phases saw very different load is not comparable between engines and says so in its own report.
Workload
orbit-bench (services/rust/orbit-tools) generates the data. It is synthetic, deterministic
(one seed produces the same bytes for every engine, and on amd64 the same bytes as its Go
original, cmd/orbit-bench at tag go-oracle), run at the script's default entropy of
-noise 0.3, and shaped like what a Kubernetes node
exports: 94 series per instance, cycling through node_exporter, cAdvisor and
kube-state-metrics metric names with their real value shapes.
| Shape | Examples | Share |
|---|---|---|
| Integer counters with a noisy rate | network and disk bytes, restarts, histogram buckets | 28% |
| Two-decimal counters | node_cpu_seconds_total per cpu and mode | 17% |
| Full-precision float counters | container_cpu_usage_seconds_total | 6% |
| Integer gauges on a random walk | memory, filesystem, load average, RSS, goroutines | 15% |
| Booleans that rarely flip | kube_pod_status_phase | 6% |
| Constant or zero | limits, sizes, replica counts, error and drop counters, up | 28% |
Every series of one instance carries the same scrape timestamp, with a per-scrape jitter of
±20 ms and a per-instance phase inside the interval, the way a scraper behaves. The
-noise flag sets the relative spread of each counter increment: 0.3 (bursty) and 0.05
(steady) are the two profiles reported. It is the workload's entropy knob, and it is the
main caveat of this page: no codec beats the entropy of a random increment, so the ratio
between the engines depends more on how noisy your metrics are than on any of them. A
fleet with many idle nodes and flat counters sits above these figures; a fleet of hot
services with full-precision float counters sits below them.
Reproduce it
# From the repository root. Needs cargo, Docker and ~4 GiB of free disk.# The Results tables came from the Go engine (tag go-oracle), which this repository# no longer builds; these two commands run the same two workloads on the Rust engine.# The entropy knob is left at the script's default, counter noise 0.3. The script# now defaults to a newer Prometheus, so the image is pinned to the one measured.PROM_IMAGE=prom/prometheus:v3.1.0 SERIES=20000 SPAN=4h BLOCK=2h services/rust/orbit-stack/bench/run.shPROM_IMAGE=prom/prometheus:v3.1.0 SERIES=200000 SPAN=2h BLOCK=2h QUERIES=10 services/rust/orbit-stack/bench/run.shThe trace comparison with Grafana Tempo (it needs cargo, Docker, python3 and an empty
ext4 output directory, under $TMPDIR or /tmp by default):
services/rust/orbit-stack/bench/traces.sh # 50,000 traces, 3 roundsTRACES=300000 ROUNDS=1 services/rust/orbit-stack/bench/traces.sh # 3.4 M spans, 1 roundTRACES=20000 ROUNDS=2 services/rust/orbit-stack/bench/traces.sh # all in the head, 2 roundsThe prometheus-benchmark soak:
# From services/rust/orbit-stack. Needs cargo, python3, systemd and an ext4 BENCH_ROOT# (default /tmp/orbit-bench); `plan` prints the free memory each scale needs.bench/soak.sh fetchbench/soak.sh planPROFILE=quick bench/soak.sh pair 200k 3plan prints every cell of the official sequence with its memory cap, the memory it needs
free and the wall clock, and official runs it. Each run.sh run prints its table and leaves the raw material under
services/rust/orbit-stack/bench/results/<timestamp>-s<series>-n<noise>/: the three data
directories, the generator's JSON for each engine, the engine's /api/v1/status/tsdb,
VictoriaMetrics' /metrics, the query timings and results.md. Change one variable at a
time. SERIES and SPAN scale the run; BLOCK must be the same on both sides that have
blocks or the comparison stops meaning anything.
Scope rules this page protects
- The disk ratio is per sample on sealed blocks. Never read it onto RAM.
- The ingest figure is throughput accepted by one engine on one machine, not a ceiling, and VictoriaMetrics accepts faster than this engine does.
- "1 binary" is the engine; the platform is four containers. VictoriaMetrics single-node is one binary too, so that line is a comparison with a Prometheus stack and with nothing else.
- A figure that is not in a report committed under
bench/runs/does not go on a page: not a Go benchmark, not a builder's own run. - A
quicksoak run gives direction, not a verdict, and a run on a busy machine says so in its own report; official numbers come from theofficialprofile on a quiet machine.
Where to go next
- The engine itself, its endpoints and its knobs:
orbit-stack.md. - The numbers as the site prints them, with their scope:
CLAIMS.md.