Skip to content

Reference

Orbit Stack engine

orbit-stack is the fourth container: one binary that takes Prometheus remote-write, stores samples in a write-ahead log and an in-memory head, seals the head on a schedule into immutable columnar blocks compressed with zstd, merges small blocks into larger ones in the background, and answers PromQL. Since Addendum v0.5 it also takes OpenTelemetry traces over OTLP/HTTP, keeps them in segments of its own and derives span metrics from them ("Traces" below). orbit-api reads it behind GET /api/v1/query_range, the dashboards and the APM routes; the Deck's ad-hoc panel reaches it the same way; your clusters push into it. It is specified in CONTRACT.md Addendum v0.3 and measured in benchmark.md.

What it stores, and how

  • A write request lands in the WAL first, then in the head. The WAL is fsynced on a one-second timer, so a power cut loses at most that interval, like Prometheus. A write the log cannot take (a full disk, a device error) refuses the request that hit it, so the sender retries it, while records already accepted stay pending for the next flush; the log is reopened and cut back to its last whole record, so a truncated record never ends the replay early. The failure is not sticky: once there is room again the next write goes through, with no restart.
  • The WAL writes a series' labels once per file. The first record that carries a series defines it and gives it a ref; every later record names it by that ref alone, the way Prometheus' series records work. The samples follow column by column: the refs as deltas, the sample count per series, each timestamp as a delta from the sample before it, and each value XOR'd with the value that series logged last, so a value that did not move costs almost nothing. The record is compressed with zstd at its fastest level with a 512 KiB window. Refs belong to one file and one writer: a writer's first record in a file starts an epoch, and a replay forgets every ref it knows when it meets one, so a process that appends to a log after a restart needs nothing from the process before it, and a rotated file replays on its own. A request refused because its flush failed takes the series it defined back out of the table, so no later record can name a series whose definition never reached the file. Labels are compared exactly, and two label sets that share a hash are two series. Records written by an older build (snappy, full label sets in every record) still replay, including a file that mixes both formats, so an existing log upgrades without a migration step.
  • On Linux, every fsync of the WAL is followed by POSIX_FADV_DONTNEED on the whole pages it synced, so only the page still being appended to stays in the page cache; a rotated file is dropped whole, and so is a file once it has been replayed. Nothing but a replay reads the log, and its pages were most of what a busy engine's cgroup was charged for.
  • The head keeps one open Gorilla-encoded chunk per series (delta-of-delta timestamps, XOR values) in a fixed 131-byte buffer that never grows; with its bookkeeping the open chunk is one 160-byte allocation, and the series record that points to it is 64 bytes. The chunk is closed once fewer than 20 bytes of the buffer are left, the most one sample can need: that is about 12 samples of a noisy full-precision float, and up to ORBIT_STACK_HEAD_CHUNK_SAMPLES (120) for a constant, whichever bound comes first. The timestamp encoding carries one code Prometheus' does not: five bits that mean "the same delta as the one before last", which is what a scraper's schedule shift followed by its correction costs, where the plain scheme spends two wider codes on it. The format lives only in the head (the spill is rebuilt from the WAL at every start and blocks use their own codec), so the extra code has no compatibility cost.
  • A closed chunk goes to the spill: shared append-only segment files under chunks_head/ in the data directory, into which chunks are packed one after another in the order they close, like Prometheus' head chunks. Each chunk record carries a crc32c, the body's length, the sample count and a back-link to the series' previous chunk, then the body: about 11 bytes of header per chunk. A back-link only ever points backwards, so a chain of chunks cannot loop, and a series keeps nothing in memory but the ref of its newest chunk. So the heap holds one open chunk per series and none of the window's closed ones, and a full head does not grow with ORBIT_STACK_BLOCK_DURATION. A segment takes up to 128 MiB of chunks and a new one starts when it is full; the file grows as chunks are written, so the data directory holds exactly the bytes written and never the cap. Segments are written with pwrite and read with pread, one pread per chunk, never mapped: a mapped page that a read faults in is charged to the process's resident set, so a seal reading the spill through a mapping raised RSS by the size of the window. Every read verifies a chunk's crc32c before it trusts the header or decodes the body: a chunk that fails it fails the query, with the series named, rather than dropping that series from a 200. Unlike the WAL's, the spill's pages are not dropped from the page cache: the seal reads every one of them in label order, which from disk would be one random read per chunk.
  • The spill is not a durability record. Every sample reaches the write-ahead log before it reaches a chunk, and a seal deletes the rotated log only after its block has landed, so the log always holds at least what the spill holds. On start the spill directory is therefore deleted outright and the replay writes it again: a segment a crash tore needs no repair, and no second index on disk can disagree with the log. A rotated log the engine cannot read is left where it is and named in the error Open fails with; a log that ends mid-record is replayed up to that byte, logged at WARN and counted in orbit_stack_wal_truncations_total. The seal builds the block from the spilled chunks plus the open ones and gives its segments back as soon as the block has landed, so a flushed window leaves nothing on disk but the block; a query that was on its way to that head still reads it: the files go when that query does. A seal whose block will not write keeps its head, so the window stays queryable, and the next tick writes the block again.
  • Every ORBIT_STACK_BLOCK_DURATION (default 2 h) the head is sealed: each series is re-encoded column by column (timestamps as delta-of-delta varints; values as integer deltas when every value is an integer or a decimal with up to four places, XOR otherwise), the columns are concatenated and the block is zstd-compressed. The block lands in a temporary directory and is renamed into place, so a crash leaves a whole block or none, and the rotated WAL is replayed on the next start if the seal never finished. Before the rename, the block's meta.json names the rotated logs it was sealed from; a log that a block on disk names is deleted at start instead of being replayed, so a crash between a block landing and its log's removal no longer seals the same window twice. Two blocks with the same window, series and sample counts whose index and chunk files are byte-identical are one block: Open keeps the one created first.
  • Adjacent blocks are compacted in the background, the way Prometheus levels its blocks. A block's index repeats every series' labels, so at a short block duration the indexes are most of the disk and most of a long query's work. Compaction builds blocks of three times the block duration, then three times that, up to a tenth of the retention (31 days with no retention); with the defaults, 2 h blocks and 15 days, that is 6 h and 18 h. A group of blocks is merged only once its span is closed (the oldest sample in the head is past its end, so no block that starts inside it is still to come), at most 16 blocks in one merge and one merge at a time. The merged block is written into a temporary directory and read back in full before anything uses it: the index checksum, every frame's checksum, every series against its index entry in strict time order, and a hash of the whole written stream. Only then is it renamed into place and swapped in for its parents in one step; the parents' directories go once no query holds them. The merged block names its parents in its meta.json, so a crash between the swap and their deletion is finished by the next Open. Blocks that overlap are merged one sample per timestamp, and the block that sorts first wins, as it does in a query. A merge that fails keeps its blocks as they were, is counted in orbit_stack_compaction_failures_total, and keeps that group out of every plan until the next restart.
  • A v3 block checksums its index and its frames. The index carries a crc32c of its own bytes in a four-byte trailer, verified when the block is opened, and the frame table a crc32c of every frame, verified before the frame is decompressed. A changed byte in either is then an error that names the block or the frame, where before it served a wrong label value or a wrong series span as fact with a 200. The checksums cost five bytes per frame of 4,096 series and four per block. meta.json is not covered by them; it is held to the index instead, because queries skip a block by its minT and maxT and retention deletes it by maxT. The index records every series' first and last timestamp and its sample count, so on open a meta.json whose span, series or samples disagree with it (or that is missing or unreadable) is corrected from the index, written again, and named in a WARN line. A v1 or v2 block still opens, is read unverified and is counted in orbit_stack_unverified_blocks, which Open also says once per block in the log; its meta.json is held to an index that no checksum covers. Opening a block reads its index once to check it and keeps each series' labels as symbol references until a query selects the series, so a directory of large blocks does not have to fit in memory to start.
  • A block that fails verification at open (a bad checksum, a format version this build does not know) is skipped, the rest is served, and the engine reports the skip. Its window comes from its directory name, else from its meta.json, else it is unknown and covers all time. A query, query_range, series or windowed label response whose window reaches it carries Prometheus' warnings field naming the block and the reason, so a missing window never answers as a clean 200. /readyz stays 200 with "degraded": true and the skipped blocks in reason, and orbit_stack_skipped_blocks counts them. The directory stays for the operator; retention deletes it once its window is past the retention. A block whose window is unknown warns on every query and never reaches retention, so it needs to be removed by hand.
  • Blocks sealed by a build older than the delta-of-delta fix can carry shifted timestamps: that encoder wrote a delta-of-delta of exactly +64, +256 or +2048 ms in a width the reader extends as negative, so every later timestamp of that chunk moved, and the seal wrote the moved ones into the block. Nothing in the block records the original value, so such a block cannot be repaired; it leaves with retention.
  • A sample the block just sealed already holds is refused, not accepted and then never returned. The seal carries each series' last timestamp into the head that replaces it, by label hash, and consults it once when a series is created there; a sample at or before it counts as out_of_order, the way Prometheus refuses a duplicate. The map is freed one window later.
  • Retention (ORBIT_STACK_RETENTION, default 15 d) deletes whole blocks whose newest sample is older than the window. A merged block is deleted by its newest sample, so data can outlive the retention by up to a tenth of it, which is Prometheus' rule too. A block leaves the index at once and its directory goes when the last query that could name it is done.
  • A query reads the matching series from every overlapping block plus the head, merging samples in time order. Matchers are resolved through a per-block inverted index, so a query never walks the series it does not want; the blocks' indexes and frames are loaded in parallel, and the series a query does want are decoded on every core, straight into the samples the querier returns. The steps of a range query are evaluated on every core as well: each selector's window is loaded once before the first step and nothing writes to it afterwards, so the steps are independent.
  • A block's columns are stored as independent zstd frames of 4,096 series each, and a query decompresses only the frames its matchers land in. Series are written in label order, so a selector on a metric name touches a run of frames rather than the block. Decoded frames sit in a process-wide cache bounded by ORBIT_STACK_BLOCK_CACHE_BYTES (a quarter of the memory limit, or 256 MiB when there is none), least recently used dropped first, and a block's frames are released once nothing has touched it for five minutes. The block format is versioned: this build writes v3 and still reads a v1 or v2 block, opening the v1 as the single frame it is.

A stale marker (the NaN bit pattern 0x7ff0000000000002 that Prometheus writes when a target disappears) is stored byte for byte and honoured on the way out: a range selector drops it from its window, and an instant selector that finds one as the newest sample at or before the timestamp returns nothing rather than reaching further back. This keeps rate() from returning NaN across every pod restart.

Out-of-order samples (a timestamp at or before the series' last one), samples without a metric name, samples that would create a series past ORBIT_STACK_MAX_SERIES or ORBIT_STACK_MAX_INDEX_BYTES (over_limit), and samples of a new series with more labels than ORBIT_STACK_MAX_LABELS_PER_SERIES, a label name longer than ORBIT_STACK_MAX_LABEL_NAME_LEN or a value longer than ORBIT_STACK_MAX_LABEL_VALUE_LEN bytes (label_limit) are dropped, counted in orbit_stack_samples_total{outcome} and reported in an X-Orbit-Stack-Dropped response header (out_of_order=0 invalid=0 over_limit=0 label_limit=3). The request still succeeds: a retry would meet the same drop. The engine checks labels once, when it creates a series: lowering a limit keeps the series the head already holds, and the head keeps no string of a series it refused. orbit-agent applies the same three limits to every series it scrapes (ORBIT_AGENT_LABEL_LIMIT, default 64, counting the target's labels; ORBIT_AGENT_LABEL_NAME_LENGTH_LIMIT, 256; ORBIT_AGENT_LABEL_VALUE_LENGTH_LIMIT, 4096; 0 for none), drops a series past one before it sends it, and counts its samples in orbit_agent_samples_dropped_total{reason="label_limit"} on every scrape.

Endpoints

RouteWhat
POST /api/v1/writeRemote-write 1.0 (protobuf, snappy or gzip). Answers 204 with Accept-Encoding: snappy, gzip: orbit-agent sends gzip once an answer says so, 40 to 70% fewer bytes than snappy, and snappy to an engine that does not. A gzip body inflates to ORBIT_STACK_MAX_BODY_BYTES at most (413 past it). Remote-write 2.0 is refused with 415 until implemented. The metric metadata a sender attaches is kept in memory, the latest per metric family, for /api/v1/metadata.
GET|POST /api/v1/query_range?query=&start=&end=&step= → Prometheus matrix. start/end as RFC 3339 or Unix seconds, step as a duration or seconds, at most 11,000 points per series.
GET|POST /api/v1/query?query=&time= → Prometheus vector, scalar, or a matrix for a range selector. Both routes keep in memory (64 MiB) the answer to a window that ended 15 minutes or more ago, until a write carries a sample older than 15 minutes or a block is sealed, merged or dropped; nocache=1 computes the answer again. query_range also keeps, under the same rules and another 32 MiB, the settled steps of each read by expression and step: a later window on the same grid, slid by a step or still open at its end, computes only the steps it lacks, as VictoriaMetrics' rollup cache does. What a seal or a late write left behind goes back within 10 seconds, with no read needed. The Deck ends ranges of 7 days or more on a multiple of their step so its viewers share them.
GET|POST /api/v1/series?match[]=&start=&end=&limit= → the label sets with a sample in the window.
GET|POST /api/v1/labels, GET /api/v1/label/{name}/values?match[]=&start=&end=&limit= → label names and values across blocks and head.
GET /api/v1/metadata?metric=&limit= → the metadata remote-write senders attached.
GET /api/v1/rulesNo groups: the engine evaluates no rules. Grafana asks on every dashboard load.
GET|POST /api/v1/query_exemplarsAlways empty: exemplars are not stored. The query and the window are still validated.
GET|POST /api/v1/format_query?query= → the parser's canonical form, every binary operation in parentheses.
GET /api/v1/status/buildinfoPrometheus' shape, with this engine's version and commit.
GET /api/v1/status/tsdbSeries, samples and bytes in the head and in blocks, series created and removed, the ten metrics with the most head series (seriesCountByMetricName), the ten labels with the most head values (labelValueCountByLabelName), drops, seals, skipped blocks and compactions.
POST /-/flushSeal the head now; also write the span metrics so far and cut the trace head into a segment.
POST /v1/tracesOTLP/HTTP traces, protobuf or JSON, gzip, zstd or neither, answered in OTLP's shapes. See "Traces".
GET /api/v1/traces/{traceId}, GET /api/v1/tracesOne trace by id (with start and end, epoch ms, only the segments within an hour of them are read, then every segment if they hold none of it); a search by service, operation, status, duration and attributes. The contract's shapes and error envelope, not Prometheus'.
GET /api/v1/apm/services, GET /api/v1/apm/services/{service}/operations, GET /api/v1/apm/servicemapRate, errors and latency per service and operation, and the service map, computed from the span metrics.
GET /api/v1/apm/heatmapSpan durations over time for the latency heatmap: every span from the span metrics, or the stored traces a search would match under an attribute filter.
GET /healthz, GET /readyz, GET /-/healthy, GET /-/readyProbes, exempt from authentication.
GET /api/v1/versionBuild stamp, like every other backend service.
GET /metricsOn ORBIT_STACK_METRICS_ADDR, its own listener, never routed by an ingress. gzip when the scraper sends Accept-Encoding: gzip.

The telemetry routes answer in Prometheus's shapes ({"status":"success","data":...} and {"status":"error","errorType":"bad_data","error":"..."}), so Grafana, promtool and any agent treat the engine as a Prometheus. A response may carry Prometheus' warnings list: a window that reaches a block skipped at open, or results truncated due to limit when a limit cut the answer. Without match[], the label routes answer from the index of the head and of the blocks that overlap start/end, as Prometheus does. With match[], a window narrows them to the matching series with a sample in it. The probes, the version route, the 404 and the 429 answer in the CONTRACT.md shapes like every other service.

Reads are metered per client IP: every read route above, but not writes, /-/flush, status/tsdb, status/buildinfo, the version route or the probes. ORBIT_STACK_QUERY_RPM (3000 a minute) is the sustained rate and ORBIT_STACK_QUERY_BURST (300) what one address may send at once; past them the engine answers 429 with a Retry-After header. Everyone behind one Grafana or one orbit-api shares its address, so size the two for the busiest dashboard times its viewers; setting either to 0 turns the limiter off. The address is the connection's peer unless it is in ORBIT_STACK_TRUSTED_PROXIES.

PromQL

Prometheus 3's language for float samples: selectors with =, !=, =~ and !~ (metric names may lead with :), range selectors, offset and subqueries; arithmetic, comparison (with or without bool), atan2 and the set operators and, or and unless, with on/ignoring and group_left/group_right; the aggregations sum, avg, min, max, count, group, stddev, stdvar, topk, bottomk, quantile and count_values with by and without; and the functions rate, irate, increase, delta, idelta, resets, changes, deriv, predict_linear, every *_over_time (sum, avg, min, max, count, last, stddev, stdvar, present, absent, quantile), the per-element math and trigonometric functions, round, clamp, clamp_min, clamp_max, timestamp, time, vector, scalar, absent, sort, sort_desc, label_replace, label_join, histogram_quantile over classic buckets and the date functions. Keywords are read in any case. A regex matcher is anchored and compiled as ^(?s:re)$, exactly as Prometheus compiles it, so . also matches a newline. A regex may compile to at most 4 MiB (a 500-value alternation fits, \w{180} does not), and a query holds at most 32 of them, its matchers and label_replace calls together; a labels or series read counts them across all its match[]. Past either, the query is refused as invalid. rate and increase extrapolate exactly as Prometheus 3.x does, with counter-reset detection.

Answers are held to results recorded from Prometheus 3.14.0 (services/rust/orbit-stack/testdata/oracle_prometheus.json, test data the Rust engine's tests read) and to the query set of prometheus/compliance. In the compliance run of 2026-09-23 against Prometheus v3.14.0 every query of that set passed and none was reported unsupported; that run's output is not committed under bench/runs/, so its count is not printed here.

Outside the language as implemented: the @ modifier, native histograms, experimental functions (limitk, double_exponential_smoothing, mad_over_time and the like) and a string as a query result. Each is refused with an error that names it. The engine never answers an unsupported expression with a wrong number.

Configuration

VarDefaultMeaning
ORBIT_STACK_ADDR:8082API listener.
ORBIT_STACK_METRICS_ADDR:9090Metrics listener; never the API port, never routed.
ORBIT_STACK_LOG_LEVELinfodebug info warn error.
ORBIT_STACK_LOG_PROBESfalseLog probe and /metrics requests at info; otherwise they are logged at debug only.
ORBIT_STACK_DATA_DIR/dataBlocks, the WAL and the spill. A named volume in compose, a PersistentVolumeClaim in Kubernetes.
ORBIT_STACK_RETENTION15dBlocks whose newest sample is older than this are deleted. Days or a Go duration. Also caps the largest compaction level at a tenth of it.
ORBIT_STACK_BLOCK_DURATION2hHead window sealed into one block, and the smallest compaction level. Minimum 1m. Shorter windows mean less RAM per series and more blocks for compaction to merge.
ORBIT_STACK_COMPRESSIONbetterzstd level for sealed and merged blocks: fast, default, better, best.
ORBIT_STACK_BLOCK_CACHE_BYTESquarter of the cgroup's memory.max, else 256 MiBDecompressed block frames and indexes the queries may hold. A plain byte count or a binary unit (512MiB).
ORBIT_STACK_HEAD_CHUNK_SAMPLES120Samples after which a series' open chunk is closed and written to the spill. The chunk's fixed buffer closes it first for anything but a near-constant series, so only a lower value changes much: lower trades more spill records for a smaller read per chunk. Minimum 2.
ORBIT_STACK_MAX_SERIES0Ceiling on live series; 0 means none. Past it, new series are dropped and counted.
ORBIT_STACK_MAX_LABELS_PER_SERIES64Labels a new series may carry, __name__ among them; 0 means no limit. Past it, the series is dropped and counted as label_limit.
ORBIT_STACK_MAX_LABEL_NAME_LEN256Bytes of one label name of a new series, checked the same way.
ORBIT_STACK_MAX_LABEL_VALUE_LEN4096Bytes of one label value of a new series, the metric name included, checked the same way.
ORBIT_STACK_MAX_INDEX_BYTESthree eighths of the cgroup's memory.max, at least 64 MiB; none without a cgroup limitBytes the head's index may take (2GiB or bytes; 0 for none). The engine charges 128 bytes per series, 16 per label of a series, 160 per distinct name=value pair, 512 per distinct label name and 80 per distinct label string plus its length, costs measured with the allocator's overhead. Until a write ends, each label of the series it creates counts as a new pair and a new name, so a large write near the ceiling drops some new series early. Past it, new series are dropped and counted as over_limit until a seal removes the series that stopped. It bounds the head's memory even with ORBIT_STACK_MAX_SERIES at 0: a series whose 63 labels each carry a value no other series has costs about 11 KiB, a Kubernetes-shaped one a few hundred bytes. orbit_stack_head_index_bytes shows the figure now.
ORBIT_STACK_QUERY_TIMEOUT30sPer-query budget.
ORBIT_STACK_QUERY_MAX_SAMPLES50000000Points one query may hold (its answer's points, a range selector's samples in an instant query, or one series' samples in the window) before it is refused. A 30-day rate reads every sample once and keeps 121 points per series, so the window's length alone never refuses it; the timeout bounds the reading. The APM summaries skip the ceiling: they sum increase() one series at a time, and the timeout bounds them.
ORBIT_STACK_QUERY_TURNS0PromQL reads that run at once; the others wait for a turn. 0 is half the cores the engine sees, at least 2. A read that reaches 2 hours back or less (its window, its longest range and the 5-minute lookback) takes one of four light turns per core instead, so a 1-hour page and alert reads never wait behind 30-day reads.
ORBIT_STACK_QUERY_MAX_BYTESa third of the cgroup's memoryMemory all PromQL reads may hold together (1GiB, 512MiB or bytes; 0 for none, the default without a cgroup limit). Each read counts its matched series before it gathers them, then the rows and points it keeps, until its answer is written. A read that passes the ceiling fails with 422 and gives its share back. Sixteen reads of a million series each took the engine past a 3 GiB limit; under a 1 GiB ceiling they peaked at 1.5 GB and two of them ran to the end. orbit_stack_query_held_bytes shows what the reads running now hold.
ORBIT_STACK_QUERY_RPM3000Sustained reads per minute per client IP; 0 turns the read limiter off.
ORBIT_STACK_QUERY_BURST300Reads one client IP may send at once; 0 turns the read limiter off.
ORBIT_STACK_MAX_BODY_BYTES67108864Remote-write request ceiling (64 MiB). All remote writes in flight together hold at most a quarter of the cgroup's memory, from 64 MiB to 1 GiB (1 GiB without a limit), as sent and decompressed; a write past that answers 503 with Retry-After: 1, and the sender retries it. A snappy body that claims to decompress past 64/3 of its own size is refused with 400 before anything is allocated, and one whose series, labels and samples would take more than four times its decompressed size (an empty one is two bytes) with 400 as it is read.
ORBIT_STACK_MEMORY_LIMITunsetRead only by the Go engine (tag go-oracle), which held its garbage-collected heap to it. The Rust engine has no collector to budget and ignores it; see Memory.
ORBIT_STACK_AUTH_MODEstaticstatic or disabled (local demos only; logs a WARN on every start).
ORBIT_STACK_AUTH_TOKENemptyRequired when mode is static, at least 32 bytes. Agents and orbit-api present it. It reads, writes and changes settings.
ORBIT_STACK_INGEST_TOKENemptyOptional second bearer, at least 32 bytes and different from ORBIT_STACK_AUTH_TOKEN. It may only POST to /api/v1/write, /v1/traces and /v1/logs; any other route answers 401. Give this one to applications and Collectors outside the release. Helm sets it from secrets.orbitStackIngestToken, and then the chart's trace collector and demo shop send with it instead of the main token.
ORBIT_STACK_TRUSTED_PROXIESemptyCIDRs whose X-Forwarded-For the per-IP read limiter believes.
ORBIT_STACK_SHUTDOWN_GRACE30sDrain budget. The WAL is synced; nothing is sealed under a deadline.
ORBIT_STACK_TRACE_RETENTION7dTrace segments whose newest span is older than this are deleted whole. Days or a Go duration.
ORBIT_STACK_TRACE_HEAD_BYTES64MiBSpans held in memory before the head is cut into a segment; the trace read cache gets half of it, and the OTLP request bytes held in flight (at least 32 MiB) the same budget. At least 1MiB.
ORBIT_STACK_TRACE_MAX_BYTES5MiBOne trace in the head, Tempo's max_bytes_per_trace default; spans past it are refused one by one (OTLP partial success). From 64 KiB to half of ORBIT_STACK_TRACE_HEAD_BYTES.
ORBIT_STACK_TRACE_GRAPH_PENDING100000Spans waiting for the other side of a service-graph edge, 1000 to 10000000. Past it a span goes unpaired: the edge undercounts, traces_service_graph_dropped_spans_total counts it, and the service map answers partial: true.

orbit-api reaches the engine with ORBIT_STACK_URL and ORBIT_STACK_TOKEN.

Memory

The head holds every series of the current block window, but only each live series' open chunk is in the heap: the chunks it has closed are in the spill segments on disk. In the heap, a live series costs its 64-byte record, its open chunk (one 160-byte allocation that never grows), a few bytes of symbol references for its labels, its share of the inverted index, and its entry in the WAL's series table. Neither the block duration nor the scrape interval changes that; they change only how much goes to the spill segments and how often a chunk is written. A series that stops sending keeps its labels and index entries in the head until the seal of the window it stopped in: the seal writes its samples to the block and removes it. A series silent for less than 15 minutes at that seal stays one window longer. With pods coming and going, the head holds the live series plus the ones that stopped since the last seal.

The one measurement of the whole process at 200,000 series in a report under bench/runs/ is in benchmark.md: 511 MiB resident with a full head of 200,000 series, and 77 MiB once that head is sealed. That report measured the Go engine (tag go-oracle) before the head, the spill and the WAL took their current layout; the prometheus-benchmark soak on the same page measured a later build that still predates them. Until a report under bench/runs/ measures this build, size the container against the 511 MiB as a ceiling, and pair it with ORBIT_STACK_MAX_SERIES: a ceiling drops and counts what it cannot hold, which is a better outcome than the pod being OOMKilled with the head it already has. The chart's values do both.

Sealing reads the head's spilled chunks and its open ones, streams the columns through zstd series by series and hands the seal's peak back to the OS (malloc_trim), so the process does not sit at that peak until the next window. At startup the process pins glibc's mmap threshold at 64 KiB: every buffer from that size up gets a mapping of its own, returned to the OS the moment it is freed, instead of fragmenting the allocator's arenas past what a trim can give back. There is no garbage collector to tune: GOGC, GOMEMLIMIT and ORBIT_STACK_MEMORY_LIMIT were the Go engine's (tag go-oracle), and the Rust engine ignores them. Three things bound its memory: the head above, ORBIT_STACK_BLOCK_CACHE_BYTES for the block pages queries hold, and ORBIT_STACK_MAX_SERIES.

Long reads also leave what lets the next one skip work, each under a fixed budget and kept until its block goes: Gorilla columns' seek points (64 MiB), and integer columns' run ends and timestamp indexes (64 MiB). A 30-day rate then unpacks only the runs at its windows' edges.

A restart replays the write-ahead log into the head before serving. The replay decodes record by record into reusable buffers, and if what it replayed spans more than one block duration (a pod that had been down while the log grew), the head is sealed before the first request is served and the span and the seal are logged. Neither the replay nor a restart loop can hold hours of samples in memory.

Traces

apm.md covers what to send and what the engine derives; this section covers how the engine keeps the spans.

  • The log. A request's spans are compressed with zstd and appended to the trace write-ahead log, each record framed with its length and a crc32c and stamped with its arrival time, and the request is answered once the record is in the log file: a process killed after answering loses nothing it acknowledged. The log is fsynced every second, so a power cut loses at most that second. A torn tail after a crash is replayed up to the tear, logged and counted in orbit_stack_trace_wal_truncations_total.
  • The head. Accepted spans are grouped by trace in memory, up to ORBIT_STACK_TRACE_HEAD_BYTES, which counts spans, summaries, span-id sets and every distinct resource. A trace past ORBIT_STACK_TRACE_MAX_BYTES has its further spans refused one by one (OTLP's partial success), and a span the trace already holds, an exporter's retry, is kept once. While a segment is being written, a head at twice its bound answers 503 with Retry-After, which senders retry.
  • Segments. The head is cut when it reaches its bound or is 15 minutes old, into an immutable segment of format version 2: whole traces as zstd frames in start order; summary chunks of 512 traces, each with a zone map (start and duration bounds, service and operation masks, error flags) and a 512-bit attribute filter per trace; a sorted id table read in place and a bloom filter; every section checksummed. A segment keeps only its directories, symbols and bloom filter in memory. It names the logs it absorbed, so a log found beside it at start is deleted rather than replayed twice.
  • Format version 2 only. A segment this build cannot read, a version 1 segment written by the first trace release included, is refused at start by name: the log line trace segment skipped names the file and the reason, and it is counted in orbit_stack_trace_skipped_segments. It is neither served nor deleted, since it may be the only copy of its traces, and retention does not reach it: delete it by hand once they no longer matter. The write-ahead log of either version replays into the head as usual.
  • Retention. A segment is deleted whole once its last span arrived longer ago than ORBIT_STACK_TRACE_RETENTION: arrival time, not the spans' own clocks, so a span stamped years in the future cannot keep a segment forever.
  • Reads. A trace by id checks the head, then each segment's bloom filter and id table, merges the parts of a trace that arrived on both sides of a cut, and streams the answer rather than building it whole; a span sent twice comes back once, and spans come sorted by start. A search reads the newest summary chunks first across every segment, skips each chunk whose zone map rules the query out, and stops once older chunks cannot change the answer; a trace split by a cut is judged whole. A search for an attribute value also skips each chunk whose attribute filter (a bloom filter of the distinct attributes its spans carry, written with the segment) rules the value out, so a rare or absent request id reads few or no chunks. Frames, summaries and resources go through a read cache bounded by half the head's figure. Another 16 MiB, apart so that a scan cannot flush them, keep, per segment and filter, what a scan of the whole segment found: its matches' number, its share of the newest and sampled ones the read kept (twice its limit over the whole window, so a week of a broad filter holds a few hundred KB), and up to 256 values of each custom facet key. So the same filter over a week reads a segment once, in any window that holds it whole, while that answer stays in those 16 MiB; a window much narrower or a limit much higher reads again the segments it needs, once. A list goes past the chunks that hold none of its newest. A count its budget cuts short has read whole segments spread over the window, which leaves the next read less to do.
  • Span metrics. Every accepted span also feeds the generator that writes traces_spanmetrics_*, traces_service_graph_* and traces_target_info into the metric head every 15 seconds; the APM routes are PromQL over those series. Their names, labels and rules are on apm.md.

Everything lives under ORBIT_STACK_DATA_DIR/traces, beside the metric blocks, so the same volume and the same backups cover both. GET /api/v1/status/tsdb reports the trace store under traces.

Pointing an agent at it

Prometheus, in agent mode or not:

yaml
remote_write:
  - url: http://orbit-stack:8082/api/v1/write
    authorization:
      credentials: <the ORBIT_STACK_AUTH_TOKEN>

vmagent: -remoteWrite.url=http://orbit-stack:8082/api/v1/write -remoteWrite.bearerToken=<token>. The OpenTelemetry collector's prometheusremotewrite exporter works the same way. Inside the cluster the address is the orbit-stack Service on port 8082; from a laptop against the compose stack it is http://127.0.0.1:8082, published on loopback only.

What to watch

orbit_stack_head_series, orbit_stack_head_series_created_total, orbit_stack_head_series_removed_total, orbit_stack_head_bytes (the open chunks, in the heap), orbit_stack_head_spill_bytes (the closed ones, in the spill segments), orbit_stack_wal_bytes, orbit_stack_wal_truncations_total, orbit_stack_seal_failures_total, orbit_stack_spill_errors_total, orbit_stack_unverified_blocks, orbit_stack_skipped_blocks, orbit_stack_blocks, orbit_stack_block_bytes (disk), orbit_stack_compactions_total, orbit_stack_compaction_failures_total, orbit_stack_query_cache_resets_total (late writes and block changes that dropped the kept PromQL answers), orbit_stack_samples_total{outcome} (accepted, out_of_order, invalid, over_limit, label_limit), orbit_stack_head_index_bytes, orbit_stack_query_duration_seconds and orbit_stack_seal_duration_seconds, all on the metrics listener. A rising out_of_order count means two senders write the same series, or one sender retries what it already sent. A rising over_limit count means ORBIT_STACK_MAX_SERIES or ORBIT_STACK_MAX_INDEX_BYTES is dropping new series, as configured. A rising label_limit count means a sender's new series carry more or longer labels than the limits allow; GET /api/v1/status/tsdb reports droppedLabelLimit and indexBytes too. The rate of orbit_stack_head_series_created_total against orbit_stack_head_series is the churn; GET /api/v1/status/tsdb names the ten metrics with the most head series, where a cardinality bomb shows first, and the ten labels with the most values, the label that carries it. Ingestion control's Metrics tab shows both, through orbit-api (see ingestion.md).

For traces: orbit_stack_spans_total{outcome} (accepted, rejected), orbit_stack_trace_head_bytes, orbit_stack_trace_head_spans, orbit_stack_trace_segments, orbit_stack_trace_segment_bytes, orbit_stack_trace_wal_bytes, orbit_stack_trace_cache_bytes, orbit_stack_trace_skipped_segments, orbit_stack_trace_cuts_total, orbit_stack_trace_cut_failures_total, orbit_stack_trace_wal_truncations_total and orbit_stack_trace_generator_dropped_total. Alert on cut failures and skipped segments the way you alert on seal failures and skipped blocks.

A seal that keeps failing and a block skipped at open are both reported in /readyz's body ({"status":"ready","degraded":true,"reason":"..."}) and not in its status, which stays 200. The window is still queryable and still backed by its log, or answers with a warning; answering 503 would take the engine out of its Service and stop ingest, which loses more than a window that is late to disk. Alert on orbit_stack_seal_failures_total (it counts a write-ahead log that would not rotate as well as a block that could not be written), on orbit_stack_skipped_blocks above zero and on the degraded field, not on readiness.

Ceilings in this release

Single node, no replication, no object-store tiering, no downsampling, remote-write 1.0 only, no rule evaluation and no exemplars, and the PromQL above. For traces: OTLP/HTTP only, no gRPC, no sampling, span links and the instrumentation scope not stored, and the generator capped in label sets. Each is a ponytail: comment in the source naming the upgrade path.

Three of them have a measured price, all against VictoriaMetrics rather than Prometheus: a sum by (...) (rate(...)) over 20,000 series answers in 24 ms against its 16, a full head of 200,000 series is 511 MiB resident against its 312, and in the prometheus-benchmark soak the data directory held 24 to 27 bytes per sample while the data sat in the head, against its 2.5 to 2.7. All three are in benchmark.md with the numbers beside the rows they lose to. The third was measured on the WAL and spill this release replaced; no report under bench/runs/ measures the new ones yet.

Traces against Grafana Tempo. On the same traces, the engine reads a trace by id faster, accepts spans faster and holds less memory, and its services summary and service map are faster at 50,000 traces. Search is faster only for values common in recent data: a rare or absent value makes it read every summary in the window, and at 3.4 M spans that is slower than Tempo. The services summary and the service map read right after a flush are slower than Tempo's. The figures, the loss table and the withdrawn runs are in benchmark.md → "Traces against Grafana Tempo", the only page that prints them.

The evaluator computes rate, increase, delta and irate over a range selector, alone or under one sum, avg, min, max or count, series-major: one pass per series fills every step of the range query, and the group a series belongs to is resolved once rather than at every step. Anything else falls back to the general evaluator, and the two are held to the same answer sample for sample by the tests.

orbit_stack_head_bytes is the open chunks only, the part of the head that is in the heap; orbit_stack_head_spill_bytes is what its closed chunks hold in the spill segments.