Reference
Orbit Stack engine
orbit-stack is the fourth container: one binary that takes Prometheus remote-write,
stores samples in a write-ahead log and an in-memory head, seals the head on a schedule into
immutable columnar blocks compressed with zstd, merges small blocks into larger ones in the
background, and answers PromQL. Since Addendum v0.5 it also takes OpenTelemetry traces over
OTLP/HTTP, keeps them in segments of its own and derives span metrics from them ("Traces"
below). orbit-api reads it behind GET /api/v1/query_range, the dashboards and the APM
routes; the Deck's ad-hoc panel reaches it the same way; your clusters push into it. It is
specified in CONTRACT.md Addendum v0.3 and measured in benchmark.md.
What it stores, and how
- A write request lands in the WAL first, then in the head. The WAL is fsynced on a one-second timer, so a power cut loses at most that interval, like Prometheus. A write the log cannot take (a full disk, a device error) refuses the request that hit it, so the sender retries it, while records already accepted stay pending for the next flush; the log is reopened and cut back to its last whole record, so a truncated record never ends the replay early. The failure is not sticky: once there is room again the next write goes through, with no restart.
- The WAL writes a series' labels once per file. The first record that carries a series defines it and gives it a ref; every later record names it by that ref alone, the way Prometheus' series records work. The samples follow column by column: the refs as deltas, the sample count per series, each timestamp as a delta from the sample before it, and each value XOR'd with the value that series logged last, so a value that did not move costs almost nothing. The record is compressed with zstd at its fastest level with a 512 KiB window. Refs belong to one file and one writer: a writer's first record in a file starts an epoch, and a replay forgets every ref it knows when it meets one, so a process that appends to a log after a restart needs nothing from the process before it, and a rotated file replays on its own. A request refused because its flush failed takes the series it defined back out of the table, so no later record can name a series whose definition never reached the file. Labels are compared exactly, and two label sets that share a hash are two series. Records written by an older build (snappy, full label sets in every record) still replay, including a file that mixes both formats, so an existing log upgrades without a migration step.
- On Linux, every fsync of the WAL is followed by
POSIX_FADV_DONTNEEDon the whole pages it synced, so only the page still being appended to stays in the page cache; a rotated file is dropped whole, and so is a file once it has been replayed. Nothing but a replay reads the log, and its pages were most of what a busy engine's cgroup was charged for. - The head keeps one open Gorilla-encoded chunk per series (delta-of-delta timestamps,
XOR values) in a fixed 131-byte buffer that never grows; with its bookkeeping the open
chunk is one 160-byte allocation, and the series record that points to it is 64 bytes.
The chunk is closed once fewer than 20 bytes of the buffer are left, the most one sample
can need: that is about 12 samples of a noisy full-precision float, and up to
ORBIT_STACK_HEAD_CHUNK_SAMPLES(120) for a constant, whichever bound comes first. The timestamp encoding carries one code Prometheus' does not: five bits that mean "the same delta as the one before last", which is what a scraper's schedule shift followed by its correction costs, where the plain scheme spends two wider codes on it. The format lives only in the head (the spill is rebuilt from the WAL at every start and blocks use their own codec), so the extra code has no compatibility cost. - A closed chunk goes to the spill: shared append-only segment files under
chunks_head/in the data directory, into which chunks are packed one after another in the order they close, like Prometheus' head chunks. Each chunk record carries a crc32c, the body's length, the sample count and a back-link to the series' previous chunk, then the body: about 11 bytes of header per chunk. A back-link only ever points backwards, so a chain of chunks cannot loop, and a series keeps nothing in memory but the ref of its newest chunk. So the heap holds one open chunk per series and none of the window's closed ones, and a full head does not grow withORBIT_STACK_BLOCK_DURATION. A segment takes up to 128 MiB of chunks and a new one starts when it is full; the file grows as chunks are written, so the data directory holds exactly the bytes written and never the cap. Segments are written withpwriteand read withpread, onepreadper chunk, never mapped: a mapped page that a read faults in is charged to the process's resident set, so a seal reading the spill through a mapping raised RSS by the size of the window. Every read verifies a chunk's crc32c before it trusts the header or decodes the body: a chunk that fails it fails the query, with the series named, rather than dropping that series from a200. Unlike the WAL's, the spill's pages are not dropped from the page cache: the seal reads every one of them in label order, which from disk would be one random read per chunk. - The spill is not a durability record. Every sample reaches the write-ahead log before it
reaches a chunk, and a seal deletes the rotated log only after its block has landed, so
the log always holds at least what the spill holds. On start the spill directory is
therefore deleted outright and the replay writes it again: a segment a crash tore needs
no repair, and no second index on disk can disagree with the log. A rotated log the
engine cannot read is left where it is and named in the error Open fails with; a log that
ends mid-record is replayed up to that byte, logged at WARN and counted in
orbit_stack_wal_truncations_total. The seal builds the block from the spilled chunks plus the open ones and gives its segments back as soon as the block has landed, so a flushed window leaves nothing on disk but the block; a query that was on its way to that head still reads it: the files go when that query does. A seal whose block will not write keeps its head, so the window stays queryable, and the next tick writes the block again. - Every
ORBIT_STACK_BLOCK_DURATION(default 2 h) the head is sealed: each series is re-encoded column by column (timestamps as delta-of-delta varints; values as integer deltas when every value is an integer or a decimal with up to four places, XOR otherwise), the columns are concatenated and the block is zstd-compressed. The block lands in a temporary directory and is renamed into place, so a crash leaves a whole block or none, and the rotated WAL is replayed on the next start if the seal never finished. Before the rename, the block'smeta.jsonnames the rotated logs it was sealed from; a log that a block on disk names is deleted at start instead of being replayed, so a crash between a block landing and its log's removal no longer seals the same window twice. Two blocks with the same window, series and sample counts whose index and chunk files are byte-identical are one block: Open keeps the one created first. - Adjacent blocks are compacted in the background, the way Prometheus levels its blocks. A
block's index repeats every series' labels, so at a short block duration the indexes are
most of the disk and most of a long query's work. Compaction builds blocks of three times
the block duration, then three times that, up to a tenth of the retention (31 days with
no retention); with the defaults, 2 h blocks and 15 days, that is 6 h and 18 h. A group of
blocks is merged only once its span is closed (the oldest sample in the head is past its
end, so no block that starts inside it is still to come), at most 16 blocks in one merge
and one merge at a time. The merged block is written into a temporary directory and read
back in full before anything uses it: the index checksum, every frame's checksum, every
series against its index entry in strict time order, and a hash of the whole written
stream. Only then is it renamed into place and swapped in for its parents in one step;
the parents' directories go once no query holds them. The merged block names its parents
in its
meta.json, so a crash between the swap and their deletion is finished by the next Open. Blocks that overlap are merged one sample per timestamp, and the block that sorts first wins, as it does in a query. A merge that fails keeps its blocks as they were, is counted inorbit_stack_compaction_failures_total, and keeps that group out of every plan until the next restart. - A v3 block checksums its index and its frames. The index carries a crc32c of its own
bytes in a four-byte trailer, verified when the block is opened, and the frame table a
crc32c of every frame, verified before the frame is decompressed. A changed byte in either
is then an error that names the block or the frame, where before it served a wrong label
value or a wrong series span as fact with a
200. The checksums cost five bytes per frame of 4,096 series and four per block.meta.jsonis not covered by them; it is held to the index instead, because queries skip a block by itsminTandmaxTand retention deletes it bymaxT. The index records every series' first and last timestamp and its sample count, so on open ameta.jsonwhose span, series or samples disagree with it (or that is missing or unreadable) is corrected from the index, written again, and named in a WARN line. A v1 or v2 block still opens, is read unverified and is counted inorbit_stack_unverified_blocks, which Open also says once per block in the log; itsmeta.jsonis held to an index that no checksum covers. Opening a block reads its index once to check it and keeps each series' labels as symbol references until a query selects the series, so a directory of large blocks does not have to fit in memory to start. - A block that fails verification at open (a bad checksum, a format version this build
does not know) is skipped, the rest is served, and the engine reports the skip. Its window
comes from its directory name, else from its
meta.json, else it is unknown and covers all time. Aquery,query_range,seriesor windowed label response whose window reaches it carries Prometheus'warningsfield naming the block and the reason, so a missing window never answers as a clean200./readyzstays200with"degraded": trueand the skipped blocks inreason, andorbit_stack_skipped_blockscounts them. The directory stays for the operator; retention deletes it once its window is past the retention. A block whose window is unknown warns on every query and never reaches retention, so it needs to be removed by hand. - Blocks sealed by a build older than the delta-of-delta fix can carry shifted timestamps: that encoder wrote a delta-of-delta of exactly +64, +256 or +2048 ms in a width the reader extends as negative, so every later timestamp of that chunk moved, and the seal wrote the moved ones into the block. Nothing in the block records the original value, so such a block cannot be repaired; it leaves with retention.
- A sample the block just sealed already holds is refused, not accepted and then never
returned. The seal carries each series' last timestamp into the head that replaces it, by
label hash, and consults it once when a series is created there; a sample at or before it
counts as
out_of_order, the way Prometheus refuses a duplicate. The map is freed one window later. - Retention (
ORBIT_STACK_RETENTION, default 15 d) deletes whole blocks whose newest sample is older than the window. A merged block is deleted by its newest sample, so data can outlive the retention by up to a tenth of it, which is Prometheus' rule too. A block leaves the index at once and its directory goes when the last query that could name it is done. - A query reads the matching series from every overlapping block plus the head, merging samples in time order. Matchers are resolved through a per-block inverted index, so a query never walks the series it does not want; the blocks' indexes and frames are loaded in parallel, and the series a query does want are decoded on every core, straight into the samples the querier returns. The steps of a range query are evaluated on every core as well: each selector's window is loaded once before the first step and nothing writes to it afterwards, so the steps are independent.
- A block's columns are stored as independent zstd frames of 4,096 series each, and a query
decompresses only the frames its matchers land in. Series are written in label order, so a
selector on a metric name touches a run of frames rather than the block. Decoded frames sit
in a process-wide cache bounded by
ORBIT_STACK_BLOCK_CACHE_BYTES(a quarter of the memory limit, or 256 MiB when there is none), least recently used dropped first, and a block's frames are released once nothing has touched it for five minutes. The block format is versioned: this build writes v3 and still reads a v1 or v2 block, opening the v1 as the single frame it is.
A stale marker (the NaN bit pattern 0x7ff0000000000002 that Prometheus writes when a
target disappears) is stored byte for byte and honoured on the way out: a range selector
drops it from its window, and an instant selector that finds one as the newest sample at or
before the timestamp returns nothing rather than reaching further back. This keeps
rate() from returning NaN across every pod restart.
Out-of-order samples (a timestamp at or before the series' last one), samples without a
metric name, samples that would create a series past ORBIT_STACK_MAX_SERIES or
ORBIT_STACK_MAX_INDEX_BYTES (over_limit), and samples of a new series with more labels
than ORBIT_STACK_MAX_LABELS_PER_SERIES, a label name longer than
ORBIT_STACK_MAX_LABEL_NAME_LEN or a value longer than ORBIT_STACK_MAX_LABEL_VALUE_LEN
bytes (label_limit) are dropped, counted in orbit_stack_samples_total{outcome} and
reported in an X-Orbit-Stack-Dropped response header
(out_of_order=0 invalid=0 over_limit=0 label_limit=3). The request still succeeds: a retry
would meet the same drop. The engine checks labels once, when it creates a series: lowering a
limit keeps the series the head already holds, and the head keeps no string of a series it
refused. orbit-agent applies the same three limits to every series it scrapes
(ORBIT_AGENT_LABEL_LIMIT, default 64, counting the target's labels;
ORBIT_AGENT_LABEL_NAME_LENGTH_LIMIT, 256; ORBIT_AGENT_LABEL_VALUE_LENGTH_LIMIT,
4096; 0 for none), drops a series past one before it sends it, and counts its samples in
orbit_agent_samples_dropped_total{reason="label_limit"} on every scrape.
Endpoints
| Route | What |
|---|---|
POST /api/v1/write | Remote-write 1.0 (protobuf, snappy or gzip). Answers 204 with Accept-Encoding: snappy, gzip: orbit-agent sends gzip once an answer says so, 40 to 70% fewer bytes than snappy, and snappy to an engine that does not. A gzip body inflates to ORBIT_STACK_MAX_BODY_BYTES at most (413 past it). Remote-write 2.0 is refused with 415 until implemented. The metric metadata a sender attaches is kept in memory, the latest per metric family, for /api/v1/metadata. |
GET|POST /api/v1/query_range | ?query=&start=&end=&step= → Prometheus matrix. start/end as RFC 3339 or Unix seconds, step as a duration or seconds, at most 11,000 points per series. |
GET|POST /api/v1/query | ?query=&time= → Prometheus vector, scalar, or a matrix for a range selector. Both routes keep in memory (64 MiB) the answer to a window that ended 15 minutes or more ago, until a write carries a sample older than 15 minutes or a block is sealed, merged or dropped; nocache=1 computes the answer again. query_range also keeps, under the same rules and another 32 MiB, the settled steps of each read by expression and step: a later window on the same grid, slid by a step or still open at its end, computes only the steps it lacks, as VictoriaMetrics' rollup cache does. What a seal or a late write left behind goes back within 10 seconds, with no read needed. The Deck ends ranges of 7 days or more on a multiple of their step so its viewers share them. |
GET|POST /api/v1/series | ?match[]=&start=&end=&limit= → the label sets with a sample in the window. |
GET|POST /api/v1/labels, GET /api/v1/label/{name}/values | ?match[]=&start=&end=&limit= → label names and values across blocks and head. |
GET /api/v1/metadata | ?metric=&limit= → the metadata remote-write senders attached. |
GET /api/v1/rules | No groups: the engine evaluates no rules. Grafana asks on every dashboard load. |
GET|POST /api/v1/query_exemplars | Always empty: exemplars are not stored. The query and the window are still validated. |
GET|POST /api/v1/format_query | ?query= → the parser's canonical form, every binary operation in parentheses. |
GET /api/v1/status/buildinfo | Prometheus' shape, with this engine's version and commit. |
GET /api/v1/status/tsdb | Series, samples and bytes in the head and in blocks, series created and removed, the ten metrics with the most head series (seriesCountByMetricName), the ten labels with the most head values (labelValueCountByLabelName), drops, seals, skipped blocks and compactions. |
POST /-/flush | Seal the head now; also write the span metrics so far and cut the trace head into a segment. |
POST /v1/traces | OTLP/HTTP traces, protobuf or JSON, gzip, zstd or neither, answered in OTLP's shapes. See "Traces". |
GET /api/v1/traces/{traceId}, GET /api/v1/traces | One trace by id (with start and end, epoch ms, only the segments within an hour of them are read, then every segment if they hold none of it); a search by service, operation, status, duration and attributes. The contract's shapes and error envelope, not Prometheus'. |
GET /api/v1/apm/services, GET /api/v1/apm/services/{service}/operations, GET /api/v1/apm/servicemap | Rate, errors and latency per service and operation, and the service map, computed from the span metrics. |
GET /api/v1/apm/heatmap | Span durations over time for the latency heatmap: every span from the span metrics, or the stored traces a search would match under an attribute filter. |
GET /healthz, GET /readyz, GET /-/healthy, GET /-/ready | Probes, exempt from authentication. |
GET /api/v1/version | Build stamp, like every other backend service. |
GET /metrics | On ORBIT_STACK_METRICS_ADDR, its own listener, never routed by an ingress. gzip when the scraper sends Accept-Encoding: gzip. |
The telemetry routes answer in Prometheus's shapes ({"status":"success","data":...} and
{"status":"error","errorType":"bad_data","error":"..."}), so Grafana, promtool and any
agent treat the engine as a Prometheus. A response may carry Prometheus' warnings list: a
window that reaches a block skipped at open, or results truncated due to limit when a
limit cut the answer. Without match[], the label routes answer from the index of the
head and of the blocks that overlap start/end, as Prometheus does. With match[], a
window narrows them to the matching series with a sample in it. The probes, the version route, the 404 and the 429 answer in the CONTRACT.md shapes
like every other service.
Reads are metered per client IP: every read route above, but not writes, /-/flush,
status/tsdb, status/buildinfo, the version route or the probes. ORBIT_STACK_QUERY_RPM
(3000 a minute) is the sustained rate and ORBIT_STACK_QUERY_BURST (300) what one address
may send at once; past them the engine answers 429 with a Retry-After header. Everyone
behind one Grafana or one orbit-api shares its address, so size the two for the busiest
dashboard times its viewers; setting either to 0 turns the limiter off. The address is
the connection's peer unless it is in ORBIT_STACK_TRUSTED_PROXIES.
PromQL
Prometheus 3's language for float samples: selectors with =, !=, =~ and !~
(metric names may lead with :), range selectors, offset and subqueries; arithmetic,
comparison (with or without bool), atan2 and the set operators and, or and
unless, with on/ignoring and group_left/group_right; the aggregations sum,
avg, min, max, count, group, stddev, stdvar, topk, bottomk, quantile
and count_values with by and without; and the functions rate, irate, increase,
delta, idelta, resets, changes, deriv, predict_linear, every *_over_time
(sum, avg, min, max, count, last, stddev, stdvar, present, absent, quantile), the
per-element math and trigonometric functions, round, clamp, clamp_min, clamp_max,
timestamp, time, vector, scalar, absent, sort, sort_desc, label_replace,
label_join, histogram_quantile over classic buckets and the date functions. Keywords
are read in any case. A regex matcher is anchored and compiled as ^(?s:re)$, exactly as
Prometheus compiles it, so . also matches a newline. A regex may compile to at most 4 MiB
(a 500-value alternation fits, \w{180} does not), and a query holds at most 32 of them,
its matchers and label_replace calls together; a labels or series read counts them across all
its match[]. Past either, the query is refused as invalid. rate and increase extrapolate
exactly as Prometheus 3.x does, with counter-reset detection.
Answers are held to results recorded from Prometheus 3.14.0
(services/rust/orbit-stack/testdata/oracle_prometheus.json, test data the Rust
engine's tests read) and to the query set of
prometheus/compliance. In the compliance run of 2026-09-23 against Prometheus v3.14.0
every query of that set passed and none was reported unsupported; that run's output is not
committed under bench/runs/, so its count is not printed here.
Outside the language as implemented: the @ modifier, native histograms, experimental
functions (limitk, double_exponential_smoothing, mad_over_time and the like) and a
string as a query result. Each is refused with an error that names it. The engine never
answers an unsupported expression with a wrong number.
Configuration
| Var | Default | Meaning |
|---|---|---|
ORBIT_STACK_ADDR | :8082 | API listener. |
ORBIT_STACK_METRICS_ADDR | :9090 | Metrics listener; never the API port, never routed. |
ORBIT_STACK_LOG_LEVEL | info | debug info warn error. |
ORBIT_STACK_LOG_PROBES | false | Log probe and /metrics requests at info; otherwise they are logged at debug only. |
ORBIT_STACK_DATA_DIR | /data | Blocks, the WAL and the spill. A named volume in compose, a PersistentVolumeClaim in Kubernetes. |
ORBIT_STACK_RETENTION | 15d | Blocks whose newest sample is older than this are deleted. Days or a Go duration. Also caps the largest compaction level at a tenth of it. |
ORBIT_STACK_BLOCK_DURATION | 2h | Head window sealed into one block, and the smallest compaction level. Minimum 1m. Shorter windows mean less RAM per series and more blocks for compaction to merge. |
ORBIT_STACK_COMPRESSION | better | zstd level for sealed and merged blocks: fast, default, better, best. |
ORBIT_STACK_BLOCK_CACHE_BYTES | quarter of the cgroup's memory.max, else 256 MiB | Decompressed block frames and indexes the queries may hold. A plain byte count or a binary unit (512MiB). |
ORBIT_STACK_HEAD_CHUNK_SAMPLES | 120 | Samples after which a series' open chunk is closed and written to the spill. The chunk's fixed buffer closes it first for anything but a near-constant series, so only a lower value changes much: lower trades more spill records for a smaller read per chunk. Minimum 2. |
ORBIT_STACK_MAX_SERIES | 0 | Ceiling on live series; 0 means none. Past it, new series are dropped and counted. |
ORBIT_STACK_MAX_LABELS_PER_SERIES | 64 | Labels a new series may carry, __name__ among them; 0 means no limit. Past it, the series is dropped and counted as label_limit. |
ORBIT_STACK_MAX_LABEL_NAME_LEN | 256 | Bytes of one label name of a new series, checked the same way. |
ORBIT_STACK_MAX_LABEL_VALUE_LEN | 4096 | Bytes of one label value of a new series, the metric name included, checked the same way. |
ORBIT_STACK_MAX_INDEX_BYTES | three eighths of the cgroup's memory.max, at least 64 MiB; none without a cgroup limit | Bytes the head's index may take (2GiB or bytes; 0 for none). The engine charges 128 bytes per series, 16 per label of a series, 160 per distinct name=value pair, 512 per distinct label name and 80 per distinct label string plus its length, costs measured with the allocator's overhead. Until a write ends, each label of the series it creates counts as a new pair and a new name, so a large write near the ceiling drops some new series early. Past it, new series are dropped and counted as over_limit until a seal removes the series that stopped. It bounds the head's memory even with ORBIT_STACK_MAX_SERIES at 0: a series whose 63 labels each carry a value no other series has costs about 11 KiB, a Kubernetes-shaped one a few hundred bytes. orbit_stack_head_index_bytes shows the figure now. |
ORBIT_STACK_QUERY_TIMEOUT | 30s | Per-query budget. |
ORBIT_STACK_QUERY_MAX_SAMPLES | 50000000 | Points one query may hold (its answer's points, a range selector's samples in an instant query, or one series' samples in the window) before it is refused. A 30-day rate reads every sample once and keeps 121 points per series, so the window's length alone never refuses it; the timeout bounds the reading. The APM summaries skip the ceiling: they sum increase() one series at a time, and the timeout bounds them. |
ORBIT_STACK_QUERY_TURNS | 0 | PromQL reads that run at once; the others wait for a turn. 0 is half the cores the engine sees, at least 2. A read that reaches 2 hours back or less (its window, its longest range and the 5-minute lookback) takes one of four light turns per core instead, so a 1-hour page and alert reads never wait behind 30-day reads. |
ORBIT_STACK_QUERY_MAX_BYTES | a third of the cgroup's memory | Memory all PromQL reads may hold together (1GiB, 512MiB or bytes; 0 for none, the default without a cgroup limit). Each read counts its matched series before it gathers them, then the rows and points it keeps, until its answer is written. A read that passes the ceiling fails with 422 and gives its share back. Sixteen reads of a million series each took the engine past a 3 GiB limit; under a 1 GiB ceiling they peaked at 1.5 GB and two of them ran to the end. orbit_stack_query_held_bytes shows what the reads running now hold. |
ORBIT_STACK_QUERY_RPM | 3000 | Sustained reads per minute per client IP; 0 turns the read limiter off. |
ORBIT_STACK_QUERY_BURST | 300 | Reads one client IP may send at once; 0 turns the read limiter off. |
ORBIT_STACK_MAX_BODY_BYTES | 67108864 | Remote-write request ceiling (64 MiB). All remote writes in flight together hold at most a quarter of the cgroup's memory, from 64 MiB to 1 GiB (1 GiB without a limit), as sent and decompressed; a write past that answers 503 with Retry-After: 1, and the sender retries it. A snappy body that claims to decompress past 64/3 of its own size is refused with 400 before anything is allocated, and one whose series, labels and samples would take more than four times its decompressed size (an empty one is two bytes) with 400 as it is read. |
ORBIT_STACK_MEMORY_LIMIT | unset | Read only by the Go engine (tag go-oracle), which held its garbage-collected heap to it. The Rust engine has no collector to budget and ignores it; see Memory. |
ORBIT_STACK_AUTH_MODE | static | static or disabled (local demos only; logs a WARN on every start). |
ORBIT_STACK_AUTH_TOKEN | empty | Required when mode is static, at least 32 bytes. Agents and orbit-api present it. It reads, writes and changes settings. |
ORBIT_STACK_INGEST_TOKEN | empty | Optional second bearer, at least 32 bytes and different from ORBIT_STACK_AUTH_TOKEN. It may only POST to /api/v1/write, /v1/traces and /v1/logs; any other route answers 401. Give this one to applications and Collectors outside the release. Helm sets it from secrets.orbitStackIngestToken, and then the chart's trace collector and demo shop send with it instead of the main token. |
ORBIT_STACK_TRUSTED_PROXIES | empty | CIDRs whose X-Forwarded-For the per-IP read limiter believes. |
ORBIT_STACK_SHUTDOWN_GRACE | 30s | Drain budget. The WAL is synced; nothing is sealed under a deadline. |
ORBIT_STACK_TRACE_RETENTION | 7d | Trace segments whose newest span is older than this are deleted whole. Days or a Go duration. |
ORBIT_STACK_TRACE_HEAD_BYTES | 64MiB | Spans held in memory before the head is cut into a segment; the trace read cache gets half of it, and the OTLP request bytes held in flight (at least 32 MiB) the same budget. At least 1MiB. |
ORBIT_STACK_TRACE_MAX_BYTES | 5MiB | One trace in the head, Tempo's max_bytes_per_trace default; spans past it are refused one by one (OTLP partial success). From 64 KiB to half of ORBIT_STACK_TRACE_HEAD_BYTES. |
ORBIT_STACK_TRACE_GRAPH_PENDING | 100000 | Spans waiting for the other side of a service-graph edge, 1000 to 10000000. Past it a span goes unpaired: the edge undercounts, traces_service_graph_dropped_spans_total counts it, and the service map answers partial: true. |
orbit-api reaches the engine with ORBIT_STACK_URL and ORBIT_STACK_TOKEN.
Memory
The head holds every series of the current block window, but only each live series' open chunk is in the heap: the chunks it has closed are in the spill segments on disk. In the heap, a live series costs its 64-byte record, its open chunk (one 160-byte allocation that never grows), a few bytes of symbol references for its labels, its share of the inverted index, and its entry in the WAL's series table. Neither the block duration nor the scrape interval changes that; they change only how much goes to the spill segments and how often a chunk is written. A series that stops sending keeps its labels and index entries in the head until the seal of the window it stopped in: the seal writes its samples to the block and removes it. A series silent for less than 15 minutes at that seal stays one window longer. With pods coming and going, the head holds the live series plus the ones that stopped since the last seal.
The one measurement of the whole process at 200,000 series in a report under bench/runs/
is in benchmark.md: 511 MiB resident with a full head of 200,000
series, and 77 MiB once that head is sealed. That report measured the Go engine (tag
go-oracle) before the head, the spill and the WAL took their current layout; the prometheus-benchmark soak on the same
page measured a later build that still predates them. Until a report under bench/runs/
measures this build, size the container against the 511 MiB as a ceiling, and pair it with
ORBIT_STACK_MAX_SERIES: a ceiling drops and counts what it cannot hold, which is a better
outcome than the pod being OOMKilled with the head it already has. The chart's values do
both.
Sealing reads the head's spilled chunks and its open ones, streams the columns through
zstd series by series and hands the seal's peak back to the OS (malloc_trim), so the
process does not sit at that peak until the next window. At startup the process pins
glibc's mmap threshold at 64 KiB: every buffer from that size up gets a mapping of its
own, returned to the OS the moment it is freed, instead of fragmenting the allocator's
arenas past what a trim can give back. There is no garbage collector to tune: GOGC,
GOMEMLIMIT and ORBIT_STACK_MEMORY_LIMIT were the Go engine's (tag go-oracle), and
the Rust engine ignores them. Three things bound its memory: the head above,
ORBIT_STACK_BLOCK_CACHE_BYTES for the block pages queries hold, and
ORBIT_STACK_MAX_SERIES.
Long reads also leave what lets the next one skip work, each under a fixed budget and kept until its block goes: Gorilla columns' seek points (64 MiB), and integer columns' run ends and timestamp indexes (64 MiB). A 30-day rate then unpacks only the runs at its windows' edges.
A restart replays the write-ahead log into the head before serving. The replay decodes record by record into reusable buffers, and if what it replayed spans more than one block duration (a pod that had been down while the log grew), the head is sealed before the first request is served and the span and the seal are logged. Neither the replay nor a restart loop can hold hours of samples in memory.
Traces
apm.md covers what to send and what the engine derives; this section covers
how the engine keeps the spans.
- The log. A request's spans are compressed with zstd and appended to the trace
write-ahead log, each record framed with its length and a crc32c and stamped with its
arrival time, and the request is answered once the record is in the log file: a process
killed after answering loses nothing it acknowledged. The log is fsynced every second, so
a power cut loses at most that second. A torn tail after a crash is replayed up to the
tear, logged and counted in
orbit_stack_trace_wal_truncations_total. - The head. Accepted spans are grouped by trace in memory, up to
ORBIT_STACK_TRACE_HEAD_BYTES, which counts spans, summaries, span-id sets and every distinct resource. A trace pastORBIT_STACK_TRACE_MAX_BYTEShas its further spans refused one by one (OTLP's partial success), and a span the trace already holds, an exporter's retry, is kept once. While a segment is being written, a head at twice its bound answers503withRetry-After, which senders retry. - Segments. The head is cut when it reaches its bound or is 15 minutes old, into an immutable segment of format version 2: whole traces as zstd frames in start order; summary chunks of 512 traces, each with a zone map (start and duration bounds, service and operation masks, error flags) and a 512-bit attribute filter per trace; a sorted id table read in place and a bloom filter; every section checksummed. A segment keeps only its directories, symbols and bloom filter in memory. It names the logs it absorbed, so a log found beside it at start is deleted rather than replayed twice.
- Format version 2 only. A segment this build cannot read, a version 1 segment written
by the first trace release included, is refused at start by name: the log line
trace segment skippednames the file and the reason, and it is counted inorbit_stack_trace_skipped_segments. It is neither served nor deleted, since it may be the only copy of its traces, and retention does not reach it: delete it by hand once they no longer matter. The write-ahead log of either version replays into the head as usual. - Retention. A segment is deleted whole once its last span arrived longer ago than
ORBIT_STACK_TRACE_RETENTION: arrival time, not the spans' own clocks, so a span stamped years in the future cannot keep a segment forever. - Reads. A trace by id checks the head, then each segment's bloom filter and id table, merges the parts of a trace that arrived on both sides of a cut, and streams the answer rather than building it whole; a span sent twice comes back once, and spans come sorted by start. A search reads the newest summary chunks first across every segment, skips each chunk whose zone map rules the query out, and stops once older chunks cannot change the answer; a trace split by a cut is judged whole. A search for an attribute value also skips each chunk whose attribute filter (a bloom filter of the distinct attributes its spans carry, written with the segment) rules the value out, so a rare or absent request id reads few or no chunks. Frames, summaries and resources go through a read cache bounded by half the head's figure. Another 16 MiB, apart so that a scan cannot flush them, keep, per segment and filter, what a scan of the whole segment found: its matches' number, its share of the newest and sampled ones the read kept (twice its limit over the whole window, so a week of a broad filter holds a few hundred KB), and up to 256 values of each custom facet key. So the same filter over a week reads a segment once, in any window that holds it whole, while that answer stays in those 16 MiB; a window much narrower or a limit much higher reads again the segments it needs, once. A list goes past the chunks that hold none of its newest. A count its budget cuts short has read whole segments spread over the window, which leaves the next read less to do.
- Span metrics. Every accepted span also feeds the generator that writes
traces_spanmetrics_*,traces_service_graph_*andtraces_target_infointo the metric head every 15 seconds; the APM routes are PromQL over those series. Their names, labels and rules are onapm.md.
Everything lives under ORBIT_STACK_DATA_DIR/traces, beside the metric blocks, so the same
volume and the same backups cover both. GET /api/v1/status/tsdb reports the trace store
under traces.
Pointing an agent at it
Prometheus, in agent mode or not:
remote_write:
- url: http://orbit-stack:8082/api/v1/write
authorization:
credentials: <the ORBIT_STACK_AUTH_TOKEN>vmagent: -remoteWrite.url=http://orbit-stack:8082/api/v1/write -remoteWrite.bearerToken=<token>.
The OpenTelemetry collector's prometheusremotewrite exporter works the same way. Inside the
cluster the address is the orbit-stack Service on port 8082; from a laptop against the compose
stack it is http://127.0.0.1:8082, published on loopback only.
What to watch
orbit_stack_head_series, orbit_stack_head_series_created_total,
orbit_stack_head_series_removed_total, orbit_stack_head_bytes (the open chunks, in the heap),
orbit_stack_head_spill_bytes (the closed ones, in the spill segments),
orbit_stack_wal_bytes, orbit_stack_wal_truncations_total,
orbit_stack_seal_failures_total, orbit_stack_spill_errors_total,
orbit_stack_unverified_blocks, orbit_stack_skipped_blocks, orbit_stack_blocks,
orbit_stack_block_bytes (disk), orbit_stack_compactions_total,
orbit_stack_compaction_failures_total, orbit_stack_query_cache_resets_total (late writes
and block changes that dropped the kept PromQL answers), orbit_stack_samples_total{outcome} (accepted,
out_of_order, invalid, over_limit, label_limit), orbit_stack_head_index_bytes, orbit_stack_query_duration_seconds and
orbit_stack_seal_duration_seconds, all on the metrics listener. A rising out_of_order
count means two senders write the same series, or one sender retries what it already sent.
A rising over_limit count means ORBIT_STACK_MAX_SERIES or ORBIT_STACK_MAX_INDEX_BYTES
is dropping new series, as configured. A rising label_limit count means a sender's new series
carry more or longer labels than the limits allow; GET /api/v1/status/tsdb reports
droppedLabelLimit and indexBytes too.
The rate of orbit_stack_head_series_created_total against orbit_stack_head_series is the
churn; GET /api/v1/status/tsdb names the ten metrics with the most head series, where a
cardinality bomb shows first, and the ten labels with the most values, the label that carries it.
Ingestion control's Metrics tab shows both, through orbit-api (see ingestion.md).
For traces: orbit_stack_spans_total{outcome} (accepted, rejected),
orbit_stack_trace_head_bytes, orbit_stack_trace_head_spans,
orbit_stack_trace_segments, orbit_stack_trace_segment_bytes,
orbit_stack_trace_wal_bytes, orbit_stack_trace_cache_bytes,
orbit_stack_trace_skipped_segments, orbit_stack_trace_cuts_total,
orbit_stack_trace_cut_failures_total, orbit_stack_trace_wal_truncations_total and
orbit_stack_trace_generator_dropped_total. Alert on cut failures and skipped segments
the way you alert on seal failures and skipped blocks.
A seal that keeps failing and a block skipped at open are both reported in /readyz's
body ({"status":"ready","degraded":true,"reason":"..."}) and not in its status, which
stays 200. The window is still queryable and still backed by its log, or answers with a
warning; answering 503 would take the engine out of its Service and stop ingest, which
loses more than a window that is late to disk. Alert on orbit_stack_seal_failures_total
(it counts a write-ahead log that would not rotate as well as a block that could not
be written), on orbit_stack_skipped_blocks above zero and on the degraded field, not on
readiness.
Ceilings in this release
Single node, no replication, no object-store tiering, no downsampling, remote-write 1.0 only,
no rule evaluation and no exemplars, and the PromQL above. For traces: OTLP/HTTP only, no
gRPC, no sampling, span links and the instrumentation scope not stored, and the generator
capped in label sets. Each is a ponytail: comment in the source naming the upgrade path.
Three of them have a measured price, all against VictoriaMetrics rather than Prometheus: a
sum by (...) (rate(...)) over 20,000 series answers in 24 ms against its 16, a full head
of 200,000 series is 511 MiB resident against its 312, and in the prometheus-benchmark soak
the data directory held 24 to 27 bytes per sample while the data sat in the head, against
its 2.5 to 2.7. All three are in benchmark.md with the numbers beside
the rows they lose to. The third was measured on the WAL and spill this release replaced;
no report under bench/runs/ measures the new ones yet.
Traces against Grafana Tempo. On the same traces, the engine reads a trace by id faster,
accepts spans faster and holds less memory, and its services summary and service map are
faster at 50,000 traces. Search is faster only for values common in recent data: a rare or
absent value makes it read every summary in the window, and at 3.4 M spans that is slower
than Tempo. The services summary and the service map read right after a flush are slower
than Tempo's. The figures, the loss table and the withdrawn runs are in
benchmark.md → "Traces against Grafana Tempo", the only page that prints
them.
The evaluator computes rate, increase, delta and irate over a range selector, alone or
under one sum, avg, min, max or count, series-major: one pass per series fills every
step of the range query, and the group a series belongs to is resolved once rather than at
every step. Anything else falls back to the general evaluator, and the two are held to the
same answer sample for sample by the tests.
orbit_stack_head_bytes is the open chunks only, the part of the head that is in the heap;
orbit_stack_head_spill_bytes is what its closed chunks hold in the spill segments.