Skip to content

Operate

Benchmark

This benchmark has been run. On 2026-09-22, one script pushed one deterministic remote-write workload into Prometheus 3.1.0, into VictoriaMetrics single-node v1.152.0 and into the Orbit Stack engine on the same machine, sequentially, and measured disk, ingest throughput, resident memory and query latency on all three. Every engine figure in CLAIMS.md comes from those runs, the reports are recorded under services/rust/orbit-stack/bench/runs/, and the script that produced them is services/rust/orbit-stack/bench/run.sh. On 2026-09-23 a second harness, services/rust/orbit-stack/bench/soak.sh, ran the engine against VictoriaMetrics the way VictoriaMetrics/prometheus-benchmark runs a database, with churn and an alerting-rules suite; its first run is below, under its own heading, with the loss it found. A third, services/rust/orbit-stack/bench/traces.sh, compares the engine's traces with Grafana Tempo's; its second round, as an independent critic re-ran it, is below with its losses, and the earlier trace figures are withdrawn. This page is the method as it was executed, the results, and what they do and do not say.

Every figure on this page outside the 2026-10-03 section was measured on the Go engine, which now lives at the git tag go-oracle. The engine that ships is the Rust one in services/rust/orbit-stack, and the scripts measure it. This repository no longer builds Go, so the commands below run the same workloads against the Rust engine, not the figures' engine. The Rust engine's own runs are recorded under bench/runs/ too (their names carry rust); this page prints only the ones the 2026-10-03 section names.

2026-10-03: the Rust stack, before and after a performance round

One builder per product, each measured against its direct competitor, then reviewed: report services/rust/orbit-stack/bench/runs/2026-10-03-perf-round.md. The machine was shared by all six at once (load 6 to 32), so the builders' competitor columns stay in that report. This page prints only the rows a reviewer re-ran afterwards, back to back, at load 4 to 7.

Figure, re-checkedBeforeAfterCompetitor in the same run
Trace search for an attribute one trace has, 1.13 M spans, p50447 ms8.0 msnone
Trace search for an attribute no trace has, 1.13 M spans, p50427 ms6.1 msnone
PromQL rate(...[5m]) with a 3.5 MB answer, 20k series, p5019.1 ms11.2 msnone
PromQL sum by (instance) (rate(...[5m])), 20k series, p507.9 ms5.3 msnone
orbit-collector, traces over gRPC at ~44k spans/s, CPU cores0.36 to 0.390.25 to 0.26otelcol-contrib 0.161.0: 0.49 to 0.59
orbit-collector, traces over HTTP, CPU cores0.33 to 0.350.23otelcol-contrib: 0.55 to 0.69
orbit-collector, RSS9 MiB10 MiBotelcol-contrib: 219 to 221 MiB
orbit-agent, 500k series, steady RSS257 MB90 to 93 MBnone

Not won in the re-check: orbit-agent's CPU per sample (a win in one round, a tie in the other), and its WAL, which still writes 48.8 B per sample where Prometheus' agent mode and Alloy write 5.3. The trace rows were not re-run at 3.4 M spans, where the losses in "Traces against Grafana Tempo" were measured, so those loss rows stand.

Results

Prometheus's own defaults: 2 h blocks on both engines that have them. 20,000 active series, 4 h of virtual time at a 15 s scrape, 19.2 M samples per engine, pushed as fast as each engine accepted them.

FigurePrometheus 3.1.0VictoriaMetrics v1.152.0Orbit Stackvs Prometheusvs VictoriaMetrics
Disk per sample, sealed5.38 B2.24 B1.87 B2.9x1.2x
Disk per sample, whole data directory11.45 B2.24 B1.87 B6.1x1.2x
Ingest accepted, samples/s467k567k555k1.19x0.98x (parity)
RSS with the head full (20k series x 4 h)257 MiB263 MiB105 MiB2.4x less2.5x less
Live Go heap at that moment165 MiB181 MiB46 MiB3.6x less4.0x less
RSS after the head is sealed to a blockn/an/a29 MiBn/an/a
query_range p50 / p90 over 60 runs, answers only97 / 108 ms16 / 20 ms24 / 33 ms4.0x / 3.3x faster1.5x / 1.7x slower
query_range max (first query loads the frames it matches)152 ms33 ms102 ms1.5x fasterslower
Write errors000n/an/a

VictoriaMetrics caches rollup results by default; the row above is the cached number and the same query with nocache=1 answers in 16.1 / 19.9 ms, so the cache is not what wins it. Neither Prometheus nor Orbit Stack caches a query response at all. The engine's query_range row was corrected on 2026-09-23: the run timed the read limiter's 429 refusals together with the answers, and timed on answers only it is the 24 / 33 ms above (CLAIMS.md, from the re-check in bench/runs/2026-09-23-continuity-critic/).

At 200,000 series (2 h of virtual time, 96 M samples per engine, 2 h blocks, so Prometheus had not cut a block yet): ingest 582k against VictoriaMetrics' 520k and Prometheus' 429k, 2.20 B per sample against 2.33 B and 13.54 B. query_range p50 is 68 ms against Prometheus' 506 ms and VictoriaMetrics' 54 ms: 1.3x behind its cached answer and 1.4x ahead of its uncached one (93 ms). The first query after the seal is 259 ms against VictoriaMetrics' 248 ms. RSS is 511 MiB against Prometheus' 856 MiB and VictoriaMetrics' 312 MiB: the engine beats Prometheus at this scale and is 64% above VictoriaMetrics, which is the largest remaining gap. The head drops to 77 MiB the moment it is sealed. Report: bench/runs/2026-09-22-200k-2h-2hblocks-bursty.md.

The tables above are the script's default entropy, counter noise 0.3 (the bursty profile, see Workload). The steady profile, noise 0.05, at 20,000 series over 4 h (bench/runs/2026-09-22-20k-4h-2hblocks-steady.md): 5.215 B against VictoriaMetrics' 1.983 B and the engine's 1.724 B on sealed blocks, ingest 363,658 / 485,841 / 513,798 samples/s, RSS 214 / 235 / 141 MiB, query p50 103.5 / 15.1 / 25.4 ms. Steadier counters compress better for every engine, and the ordering does not change.

A second pair of runs put 15-minute blocks on Prometheus and the engine over 2 h of virtual time, 9.6 M samples (bench/runs/2026-09-22-20k-2h-15mblocks-steady.md and ...-15mblocks-bursty.md). Sealed blocks: 3.58x and 3.28x against Prometheus, 1.18x and 1.22x against VictoriaMetrics. Ingest against VictoriaMetrics is 0.98x on the steady run, a loss, and 1.06x on the bursty one. Query p50, the three engines in order: 66.0 / 10.0 / 12.0 ms steady and 65.5 / 10.0 / 13.0 ms bursty. Prometheus is the smallest of the three in RAM there, 87 and 91 MiB against the engine's 122 and 121: fifteen-minute blocks keep its head short, while VictoriaMetrics sizes its caches from the machine's memory rather than from the data, so a small dataset does not make it small (176 and 213 MiB). Short blocks cost Prometheus index overhead per block, which is why the 2 h figures above are the ones CLAIMS.md prints: they are the ones that favour Prometheus.

Read the two disk rows this way. Sealed blocks are the steady state: what disk costs per sample after the write-ahead log has been checkpointed and the head has been persisted. That is the 2.9x against Prometheus and the 1.2x against VictoriaMetrics. The whole-directory row is a snapshot during ingest: Prometheus keeps the last two to three hours in a write-ahead log at roughly 20 bytes per sample, VictoriaMetrics had been flushed and merged, and the engine's WAL is empty after a seal. That ratio against Prometheus is real while data is arriving and shrinks as retention grows, so it is printed only here and only with "during ingest" next to it.

Where the engine still loses. Three rows of these runs, and they are printed wherever their wins are; the soak below adds a fourth, the data directory while the data is in the head.

VictoriaMetrics answers the 20,000-series query 1.5x faster at p50 and 1.7x at p90 (16 ms against 24, 20 against 33). At 200,000 it is 1.3x faster than the engine with its response cache and 1.4x slower without it, because the work there is dominated by the series the query touches and the column-wise evaluator reduces each one in a single pass. What is left at 20,000 is fixed cost per query rather than per-sample work.

RSS at 200,000 series is 64% above VictoriaMetrics (511 MiB against 312). The head keeps one open chunk per series in the Go heap and spills the closed ones; VictoriaMetrics keeps almost nothing in memory and reads from its parts.

Ingest at 20,000 series is parity, not a win (555k against 567k, 0.98x). At 200,000 the engine is ahead, 582k against 520k.

Engine rounds: what changed between the reports

Round 1 was the read path, round 2 the write path and the memory it costs. Prometheus and VictoriaMetrics were re-measured alongside the engine in each session, so their columns also show what this shared machine's noise is worth. The reports named *-before-engine-round1.md under bench/runs/ are the engine as it stood before any of this, measured in the same session as the round-1 ones.

Only two states of the engine have reports under bench/runs/, so only those two are given numbers: *-before-engine-round1.md, the engine as it stood before any of this work, and the two bursty runs the tables above come from. The steady before-and-after pair is at 20,000 series over 4 h and at 200,000 over 2 h; the shipped column is the bursty profile, so read each row against its own engine columns and not across the pair. The shipped 20,000-series p50 is the corrected, answers-only figure.

before round 1 (steady)PromVMshipped (bursty)PromVM
query_range p50, 20k104.2 ms110.016.624.3 ms97.115.9
query_range max, 20k577.5 ms133.748.1102.2 ms152.132.6
query_range p50, 200k1281.5 ms747.660.667.7 ms505.953.5
query_range max, 200k4316.7 ms1544.6219.2258.5 ms608.2248.0
Ingest, 20k436131/s379425453422554670/s467338566680
Ingest, 200k376004/s338722387317581530/s429263520218
RSS, 200k1017 MiB750325511 MiB856312
Live heap, 200k825 MiB406210288 MiB592218
Disk per sample, sealed, 20k1.676 B5.2322.0621.871 B5.3812.243

The work between those two states came in rounds, and what each round did is below. The intermediate measurements were taken with the same script but their results.md files were not kept, so they carry no numbers here.

Round 5, the head chunk spill. A series' closed chunks were written to a memory-mapped file under the data directory instead of staying in the Go heap, and the head kept only the open chunk per series. Against the before-round-1 report at 200,000 series that is RSS 1017 to 511 MiB and live heap 825 to 288 MiB, which is what put the engine below Prometheus at that scale. That spill grew 8 MiB at a time and was deleted when the head was sealed, and run.sh reads the data directory after the seal, so there the directory was its blocks plus a short write-ahead log and the whole-directory disk row stayed a win. The soak below reads the directory while the data is still in the head, and there it is a loss. The spill has been redesigned twice since those reports. First its segments were read with pread and never mapped, each series writing into 2 KiB extents of its own; that is the build the soak below measured. Now the chunks of every series are packed into shared segments with about 11 bytes of header each, the WAL writes each series' labels once per file, and blocks are compacted in the background (orbit-stack.md has the design). No report under bench/runs/ measures the current build yet, so every engine figure on this page is one of the two earlier ones, and each says which.

Round 4, the evaluator. rate, increase, delta and irate over a range selector, alone or under one sum/avg/min/max/count, are computed series-major: one pass per series fills every step, the window ends are cursors rather than binary searches, the group of a series is resolved once instead of at every step, and no per-step vector is built. Anything else falls back to the general evaluator. The equivalence tests caught two float-associativity bugs before they shipped (a prefix sum over counter resets and per-worker partial aggregates), and both are now summed in the general evaluator's order, so the answers are the same bits, not merely the same to fifteen digits. internal/promql/columnwise_test.go (in the Go engine, tag go-oracle) runs every expression named in the PromQL suite plus randomised series with resets, gaps and NaN through both evaluators and compares sample for sample. The two were also compared over a real 200,000-series directory by diffing the responses of both builds, which matched byte for byte; that was a one-off check with no artifact kept, so it carries no figure here.

Round 3, the block format. A sealed block's columns are no longer one zstd stream: they are independent frames of 4,096 series, the index records where each frame is and where each series sits inside its own frame, and a query decompresses, in parallel, only the frames its matchers land in, into a process-wide cache bounded by ORBIT_STACK_BLOCK_CACHE_BYTES. The width trades disk for speed and was measured: a zstd frame starts its context from nothing, so narrow frames cost compression. The width was chosen by sweeping it at 20,000 series and reading disk per sample against the first query after a seal: narrow frames cost noticeably more disk for a little more speed, and 4,096 was the widest setting that still cut the cold read. That sweep used the same script but its results.md files were not kept, so no figure from it is printed here. The format is versioned: this build writes v3, which checksums its index and every frame, and still opens a v1 or v2 block, reading it unverified and counting it in orbit_stack_unverified_blocks; a v1 opens as the single frame it is. There are tests for each and for a version it cannot read.

Round 2, the write path. The remote-write decoder carves label sets and samples out of arenas it keeps between requests and interns every name and value within a request; the head interns again across requests, so one instance string is shared by that instance's ninety series; a sample of a series that already exists costs no allocation at all; the WAL's record and compression buffers are pooled; and the head's chunks grow by an eighth instead of by append, which was leaving a third of the head in slack. BenchmarkIngestRequest in internal/tsdb measures what one 5,000-sample request allocates, and is how the change was judged; it is a Go benchmark, go test -bench, not a run of bench/run.sh, so its figures are not printed here. Across a whole run the effect shows up in the line every report prints under its table, the total objects the process allocated. That bought the ingest rows and a fifth of the live heap; it did not buy the RSS row, because the RSS row was never churn.

What the 200k RAM row is. Since the spill, the Go heap holds one open chunk per live series plus that series' labels and its place in the inverted index; every chunk the series has closed was in the memory-mapped file instead, and is in the spill segments now. What the shipped 200,000-series report prints is 511 MiB resident with 288 MiB of live heap while the head is full, and 77 MiB resident once the head is sealed. The difference between those two is what a full head costs. It scales with live series rather than with the samples they hold, which is why the block duration no longer moves it.

Replay is bounded now. A pod that restarts after its write-ahead log has grown for hours used to replay all of it into the head and hold it: that is an OOMKill at a 1 GiB limit, and it happened on k3d. The replay decodes record by record into reusable buffers, and when what it replayed spans more than one block duration the head is sealed before the first request is served, with the span and the seal in the log. TestReplayWiderThanABlockSealsOnOpen replays three hours against a thirty-minute window and fails if the head still holds it.

The prometheus-benchmark soak

services/rust/orbit-stack/bench/soak.sh runs the load of VictoriaMetrics/prometheus-benchmark at a pinned commit: node_exporter 1.4.0 behind a 1 s cache, multiplied into as many targets as the scale needs, scraped every 10 s by vmagent, which relabels a slice of the targets every few minutes (series churn) and remote-writes to the engine under test, while vmalert evaluates the chart's 40 alerting rules against it every 10 s. The chart's components run as checksummed release binaries next to the engines rather than in k3d, and each engine runs in its own systemd scope with a memory cap and a CPU set; the run's summary.md lists every deviation from the chart. It measures, per scale, sealed bytes per sample, the whole data directory per accepted sample while the data is live, RSS (steady median and peak), sustained ingest, CPU per sample, a maximum-ingest blast, and the latency of the rules and of the continuity query.

  • Pair mode (soak.sh pair <scale> <reps>, and the pair cells of official) runs Orbit Stack and VictoriaMetrics at the same time behind one vmagent, so both receive identical bytes, each with the same memory cap and a CPU set that shares no physical core with the other's; the two sets are swapped every rep.
  • The Prometheus pass follows each pair cell (PAIR_PROM=1, the default): Prometheus alone with the same churn schedule, targets, blast seed and cap rules. node_exporter's values are live, so its input matches the pair's in shape, not byte for byte.
  • recheck replayed the continuity query the way the first benchmark ran it (the same blast, a flush, then the query back to back with each binary's default read limiter) against the loop-1 binary and the current one, alternating their order per rep, and kept every response's status, so a timing of a 429 could be told from a timing of an answer. That is how the 20,000-series query row above was found to include refusals and corrected. It built the loop-1 Go engine, so it left the script with Go on 2026-10-19.
  • Profiles: official (10 minutes of warm-up and 2 h measured per cell), quick (5 minutes and 30 minutes, direction only) and smoke (proves the harness, never a result). VALUES=diverse gives every target its own values, because cloned targets compress unrealistically well; official runs a diverse pair beside each chart pair and refuses to start on a machine that soak.sh quiet does not call quiet.

The one run of it in the tree so far is bench/runs/2026-09-23-pb-quick-pair/: the quick profile, three reps of the pair at each scale, no Prometheus pass, on a busy machine (the report prints the load of every rep). It measured the engine as of dd0e579 with a limiter patch the report names, which predates the WAL, spill and compaction of this release. Read it for direction only. Cells are VictoriaMetrics / Orbit Stack, medians of three reps, with Orbit Stack divided by VictoriaMetrics; below 1 is better for the engine.

Measure20,000 series200,000 series1,000,000 series
Sealed bytes per sample2.742 / 0.547 (0.20x)2.472 / 0.463 (0.19x)2.741 / 0.364 (0.13x)
Whole data directory per accepted sample, live2.648 / 27.112 (10.24x)2.465 / 23.966 (9.72x)2.739 / 23.965 (8.75x)
RSS steady median, MiB95 / 42 (0.44x)266 / 173 (0.65x)880 / 678 (0.77x)
RSS peak, MiB124 / 50 (0.41x)406 / 207 (0.51x)1,334 / 793 (0.59x)
Engine CPU per sample, µs12.57 / 7.38 (0.59x)6.73 / 4.62 (0.69x)6.60 / 4.84 (0.73x)
Max ingest, samples/s607,145 / 529,343 (0.87x)546,969 / 558,503 (1.02x)515,626 / 557,130 (1.08x)
Rules p50, ms1.29 / 0.79 (0.61x)5.86 / 3.10 (0.53x)23.44 / 15.42 (0.66x)
Rules p99, ms8.99 / 5.41 (0.60x)71.78 / 38.11 (0.53x)398.55 / 188.99 (0.47x)

Both engines kept up with the offered load in every rep, so sustained ingest is a tie. Max ingest is a tie too: VictoriaMetrics was ahead in all three reps at 20,000 series, and the ranges overlap at the other two scales. The 20,000-series rules p99 ranges overlap as well. The sealed-bytes row is inflated for both engines by the cloned targets, which is what VALUES=diverse is for.

The loss: the data directory while the data sits in the head, 8.75x to 10.24x VictoriaMetrics' bytes per sample. Two things made it: the WAL logged every remote-write request with its full label sets, and the spill wrote each series' short chunks into a 2 KiB extent of its own. This release rewrote both: series written once per WAL file, sample columns of deltas and XORs, chunks packed into shared segments. No report under bench/runs/ measures the result yet.

Traces against Grafana Tempo

Round 2, re-run by an independent critic. Report: services/rust/orbit-stack/bench/runs/2026-09-24-traces-vs-tempo.md, section "Round 2, re-run by an independent critic", with the critic's summaries in bench/runs/2026-09-24-traces-vs-tempo/round2-critic/. The critic re-ran round 2's build blind with bench/traces.sh's method: both engines up at once, reads interleaved repetition by repetition, Tempo -target=all with its span-metrics and service-graphs processors writing to Prometheus v3.1.0 and its limits raised, and Tempo read the way Grafana reads it (a trace by id in protobuf, a panel's queries at once). Workloads: 50,000 shop-shaped traces plus one of 10,000 spans (3 rounds), 300,000 traces = 3.4 M spans (1 round), and 20,000 traces all in the head (2 rounds); p50 of 20 repetitions, medians over rounds. The machine was shared: load averages 14 to 35 on 12 CPUs. The fixes of round 3 (a memory bound, the service-graph pairing) were not benchmarked: these are round 2's figures.

Withdrawn, and never printed here: round 1's trace-by-id ratio, which compared Tempo's JSON path; round 1's claim that no trace row was a loss; round 2's own first ratio over 200 ids, which read 20 ids; round 2's own first ingest ratio at 50,000 traces, which the re-run did not reproduce; and every services-summary and service-map ratio at 3.4 M spans, because both engines' answers were incomplete there: Tempo's span metrics held a fraction of the spans, and Orbit Stack's service graph dropped pairings at a fixed ceiling that round 3 made a setting (ORBIT_STACK_TRACE_GRAPH_PENDING).

Wins and losses are printed together. Every row of the first table is read with the second table beside it.

FigureGrafana Tempo 3.0.0Orbit StackRatio
Trace by id, one trace of 10,000 spans, p50 (50,000 traces)363 ms (protobuf)38 ms9.5x faster
Trace by id, 10-span traces, p50 over 200 ids (50,000 traces)40.6 ms0.90 ms45x faster
Search for common values (service; error; service and duration; attribute), Tempo's default mode, 50,000 traces19.9 / 23.1 / 22.8 / 35.3 ms1.13 / 1.02 / 1.11 / 1.18 ms18x to 30x faster
Same searches at 3.4 M spans19.0 / 47.6 / 38.0 / 73.6 ms1.56 / 1.59 / 1.46 / 1.94 ms12x to 38x faster
Ingest accepted, spans/s (50,000 traces; 3.4 M spans)197,481; 55,186269,185; 257,5941.4x; 4.7x
Container memory 45 s after ingest (50,000 traces; 3.4 M spans)680 MiB; 2,102 MiB356 MiB; 263 MiB1.9x; 8.0x less
Services summary (rate, errors, p50/p95/p99, pods per service), p50, 50,000 traces4.04 ms (PromQL over its generator's series); 6.73 ms (TraceQL metrics)2.92 ms1.4x; 2.3x faster
Service map, p50, 50,000 traces4.32 ms (PromQL over its generator's series)3.20 ms1.4x faster
Components to run for traces with a service map2 (Tempo and Prometheus)1n/a
LossGrafana Tempo 3.0.0Orbit StackRatio
Search for an attribute value one trace has, 3.4 M spans11.2 ms677 ms60x slower
Search for an attribute value no trace has, 3.4 M spans9.0 ms623 ms69x slower
Search for an absent status on one service, 3.4 M spans105 ms544 ms5.2x slower
Services summary, the first read right after /-/flush, everything in the head4.0 ms24.5 / 30.5 ms (2 rounds)6x to 8x slower
Service map, the first read right after /-/flush, everything in the head4.0 ms16.3 / 67.6 ms (2 rounds)4x to 17x slower

Search wins hold for common values only. A search win holds for values that are common in recent data: the search reads summaries newest first and stops once it has enough. A value that is rare or absent makes it read every summary in the window, and at 3.4 M spans that is 5x to 69x slower than Tempo, as the loss table shows. At 50,000 traces the same rare-value search is 2.6x faster.

On 2026-10-03 the rare-value search started skipping every split trace and summary chunk its attribute blooms rule out: at 1.13 M spans the rare-attribute search went from 447 to 8.0 ms (the 2026-10-03 section). It was not re-run at 3.4 M spans, so the loss rows above stand until it is.

Newest first. Tempo's default mode returns the matches it finds first; asked for the newest (most_recent=true), which is what Orbit Stack's search always returns, Tempo is 30x to 689x slower. That comparison belongs beside the default-mode rows above, never instead of them.

The first read after a flush. The services summary and the service map read right after /-/flush are slower than Tempo's median: the flush also seals the metrics head, so they are the first PromQL over a freshly sealed block (the last two rows of the loss table).

Disk is not claimed: when it was measured, Tempo's data was its write-ahead log and live-store, not compacted blocks. These tables come from one shared machine under the load it had; a re-run on a quiet machine replaces them instead of being averaged into them.

Environment

The run.sh reports above: one machine, the three engines in turn:

  • AMD Ryzen 5 5500, 12 threads, 31 GiB RAM, WSL2 (Linux 6.6), data directories on the distribution's ext4 root filesystem (an earlier pair of runs on the 9p-mounted Windows drive gave the same disk figures and slower I/O on both sides).
  • Prometheus 3.1.0 from prom/prometheus:v3.1.0 in Docker, --web.enable-remote-write-receiver, --storage.tsdb.min-block-duration=2h --storage.tsdb.max-block-duration=2h, no scrape targets, so every stored sample was one the generator sent.
  • VictoriaMetrics single-node v1.152.0 from victoriametrics/victoria-metrics:v1.152.0 in Docker, -storageDataPath on a mounted volume, -retentionPeriod=30d, everything else at its defaults, which includes its response cache and its default memory allowance. Its disk is read after POST /internal/force_flush and POST /internal/force_merge have finished, which the script waits for by polling vm_active_merges and vm_pending_rows to zero.
  • Orbit Stack built from this repository, run as a plain binary, ORBIT_STACK_BLOCK_DURATION=2h, ORBIT_STACK_COMPRESSION=better (zstd), authentication disabled for the run.

Both Docker engines are docker rm -f'd before the next run starts, so nothing of one run is alive during the next.

Disk is du -sb on each data directory after the run settled: Prometheus after its head compaction cut the 2 h block (the script waits for it), VictoriaMetrics after its flush and merge, Orbit Stack after POST /-/flush sealed the head. Samples in blocks come from each block's own metadata for Prometheus and Orbit Stack; VictoriaMetrics has no such metadata, so its data/ directory (small parts, big parts and indexdb) is divided by the samples sent, and the report records how many index bytes were still in memory at that moment. RSS is VmRSS of the engine process and docker stats for the two containers, read right after ingest (for Prometheus, after the wait for its head compaction, since that wait is what puts its block on disk, so its figure is the one taken latest). Query latency is 60 sequential query_range calls of sum by (instance) (rate(node_cpu_seconds_total{mode="user"}[5m])) over the whole window at a 60 s step, timed by curl.

This machine is shared with other work. Every report records the one-minute load average it saw at each engine's ingest and at its query loop, on 12 cores. The two runs the tables above come from were taken on a quiet machine: the loads those reports print run from 1.82 to 11.38 across their twelve phase readings. Earlier reports in the same directory were taken while the box was busier and print higher ones, which is why their absolute figures are lower; the three engines were measured minutes apart under comparable load, and a run whose three phases saw very different load is not comparable between engines and says so in its own report.

Workload

orbit-bench (services/rust/orbit-tools) generates the data. It is synthetic, deterministic (one seed produces the same bytes for every engine, and on amd64 the same bytes as its Go original, cmd/orbit-bench at tag go-oracle), run at the script's default entropy of -noise 0.3, and shaped like what a Kubernetes node exports: 94 series per instance, cycling through node_exporter, cAdvisor and kube-state-metrics metric names with their real value shapes.

ShapeExamplesShare
Integer counters with a noisy ratenetwork and disk bytes, restarts, histogram buckets28%
Two-decimal countersnode_cpu_seconds_total per cpu and mode17%
Full-precision float counterscontainer_cpu_usage_seconds_total6%
Integer gauges on a random walkmemory, filesystem, load average, RSS, goroutines15%
Booleans that rarely flipkube_pod_status_phase6%
Constant or zerolimits, sizes, replica counts, error and drop counters, up28%

Every series of one instance carries the same scrape timestamp, with a per-scrape jitter of ±20 ms and a per-instance phase inside the interval, the way a scraper behaves. The -noise flag sets the relative spread of each counter increment: 0.3 (bursty) and 0.05 (steady) are the two profiles reported. It is the workload's entropy knob, and it is the main caveat of this page: no codec beats the entropy of a random increment, so the ratio between the engines depends more on how noisy your metrics are than on any of them. A fleet with many idle nodes and flat counters sits above these figures; a fleet of hot services with full-precision float counters sits below them.

Reproduce it

bash
# From the repository root. Needs cargo, Docker and ~4 GiB of free disk.# The Results tables came from the Go engine (tag go-oracle), which this repository# no longer builds; these two commands run the same two workloads on the Rust engine.# The entropy knob is left at the script's default, counter noise 0.3. The script# now defaults to a newer Prometheus, so the image is pinned to the one measured.PROM_IMAGE=prom/prometheus:v3.1.0 SERIES=20000  SPAN=4h BLOCK=2h services/rust/orbit-stack/bench/run.shPROM_IMAGE=prom/prometheus:v3.1.0 SERIES=200000 SPAN=2h BLOCK=2h QUERIES=10 services/rust/orbit-stack/bench/run.sh

The trace comparison with Grafana Tempo (it needs cargo, Docker, python3 and an empty ext4 output directory, under $TMPDIR or /tmp by default):

bash
services/rust/orbit-stack/bench/traces.sh                # 50,000 traces, 3 roundsTRACES=300000 ROUNDS=1 services/rust/orbit-stack/bench/traces.sh   # 3.4 M spans, 1 roundTRACES=20000 ROUNDS=2 services/rust/orbit-stack/bench/traces.sh    # all in the head, 2 rounds

The prometheus-benchmark soak:

bash
# From services/rust/orbit-stack. Needs cargo, python3, systemd and an ext4 BENCH_ROOT# (default /tmp/orbit-bench); `plan` prints the free memory each scale needs.bench/soak.sh fetchbench/soak.sh planPROFILE=quick bench/soak.sh pair 200k 3

plan prints every cell of the official sequence with its memory cap, the memory it needs free and the wall clock, and official runs it. Each run.sh run prints its table and leaves the raw material under services/rust/orbit-stack/bench/results/<timestamp>-s<series>-n<noise>/: the three data directories, the generator's JSON for each engine, the engine's /api/v1/status/tsdb, VictoriaMetrics' /metrics, the query timings and results.md. Change one variable at a time. SERIES and SPAN scale the run; BLOCK must be the same on both sides that have blocks or the comparison stops meaning anything.

Scope rules this page protects

  • The disk ratio is per sample on sealed blocks. Never read it onto RAM.
  • The ingest figure is throughput accepted by one engine on one machine, not a ceiling, and VictoriaMetrics accepts faster than this engine does.
  • "1 binary" is the engine; the platform is four containers. VictoriaMetrics single-node is one binary too, so that line is a comparison with a Prometheus stack and with nothing else.
  • A figure that is not in a report committed under bench/runs/ does not go on a page: not a Go benchmark, not a builder's own run.
  • A quick soak run gives direction, not a verdict, and a run on a busy machine says so in its own report; official numbers come from the official profile on a quiet machine.

Where to go next

  • The engine itself, its endpoints and its knobs: orbit-stack.md.
  • The numbers as the site prints them, with their scope: CLAIMS.md.