Skip to content

Features

APM and traces

orbit-stack takes OpenTelemetry traces over OTLP/HTTP, keeps the spans, and derives from them the span metrics and service graph series Grafana Tempo's metrics-generator writes, under the same names, into its own time-series store. orbit-api serves the reads to the Deck, which draws a services table, a page per service, a service map, a trace search and a waterfall per trace. CONTRACT.md Addendum v0.5 is the specification; the storage engine behind it is in orbit-stack.md.

Send traces

WhatValue
EndpointPOST /v1/traces on the engine's API port, 8082: http://<release>-orbit-stack.<namespace>.svc.cluster.local:8082/v1/traces in Kubernetes, http://orbit-stack:8082/v1/traces inside the compose network, http://127.0.0.1:8082/v1/traces from the compose host
ProtocolOTLP/HTTP only. There is no gRPC listener: an exporter set to grpc, or pointed at 4317, reaches nothing.
BodyContent-Type: application/x-protobuf or application/json (hex ids; base64 ids and enum names are accepted too)
CompressionContent-Encoding: gzip or zstd, or none. orbit-collector exports zstd and falls back to gzip for 5 minutes when an engine answers 415
CredentialAuthorization: Bearer <ORBIT_STACK_INGEST_TOKEN> from outside the release (it can send and nothing else), or ORBIT_STACK_AUTH_TOKEN; none when ORBIT_STACK_AUTH_MODE=disabled
Size16 MiB once gzip or zstd is undone, or ORBIT_STACK_MAX_BODY_BYTES if that is smaller; one trace at most ORBIT_STACK_TRACE_MAX_BYTES (5 MiB)

The answer comes back in the encoding of the request, as OTLP/HTTP specifies:

StatusMeaningThe sender should
200, empty bodyEvery span acceptedCarry on
200 with partialSuccessSome spans refused one by one (no valid trace or span id, or a trace grown past ORBIT_STACK_TRACE_MAX_BYTES), counted in rejectedSpans with the first reason in errorMessageNot retry them
400 with a google.rpc.StatusThe body is not an export requestNot retry
401Missing or wrong bearerFix the credential
413, 415Too large (past 16 MiB once inflated, or more spans, resources or JSON objects than the in-flight budget can hold decoded); wrong Content-Type or Content-EncodingFix the request
503 + Retry-AfterThe OTLP bytes held by requests in flight would pass their budget (ORBIT_STACK_TRACE_HEAD_BYTES, at least 32 MiB), the head is at twice its bound while a segment is written, or the write-ahead log failedRetry after the wait

A span is written to the write-ahead log before the answer goes out, and the log is fsynced every second: a crash of the process loses nothing it acknowledged, a power cut at most that second. A span the trace already holds (an exporter's retry) is accepted and kept once. A refused body is read to its end before the answer, so the sender reads the answer rather than a reset connection. Span links and the instrumentation scope are not stored.

From a shell

One span, as OTLP/JSON, against the compose stack or a port-forward (kubectl -n sreorbit port-forward svc/sreorbit-orbit-stack 8082:8082). The two timestamps are nanoseconds from your own clock (GNU date; a trace outside the window you read is not shown):

bash
NOW=$(date +%s%N)cat > span.json <<EOF{"resourceSpans":[{"resource":{"attributes":[  {"key":"service.name","value":{"stringValue":"checkout"}},  {"key":"k8s.namespace.name","value":{"stringValue":"shop"}},  {"key":"k8s.deployment.name","value":{"stringValue":"checkout"}},  {"key":"k8s.pod.name","value":{"stringValue":"checkout-7d9c5-abcde"}}]}, "scopeSpans":[{"spans":[{  "traceId":"5b8efff798038103d269b633813fc60c","spanId":"eee19b7ec3c1b174",  "name":"POST /checkout","kind":2,  "startTimeUnixNano":"$NOW","endTimeUnixNano":"$((NOW+120000000))",  "attributes":[{"key":"http.response.status_code","value":{"intValue":"200"}}],  "status":{"code":1}}]}]}]}EOFcurl -fsS -X POST localhost:8082/v1/traces -H "Authorization: Bearer $ORBIT_STACK_AUTH_TOKEN" \  -H 'Content-Type: application/json' --data-binary @span.jsoncurl -fsS -H "Authorization: Bearer $ORBIT_STACK_AUTH_TOKEN" localhost:8082/api/v1/traces/5b8efff798038103d269b633813fc60c

The first call answers {}; the second returns the trace with its span, its resource and "kind":"server". On compose ORBIT_STACK_AUTH_TOKEN is the value in .env; under make up the engine runs with authentication disabled and ignores the header.

From an OpenTelemetry Collector

Point the Collector's OTLP/HTTP exporter at the engine and let the Collector keep receiving whatever your applications send it, gRPC included:

yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
processors:
  k8sattributes: {}
  batch: {}
exporters:
  otlphttp/orbit:
    endpoint: http://sreorbit-orbit-stack.sreorbit.svc.cluster.local:8082
    headers:
      Authorization: "Bearer ${env:ORBIT_STACK_AUTH_TOKEN}"
    compression: gzip
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [k8sattributes, batch]
      exporters: [otlphttp/orbit]

The exporter appends /v1/traces to endpoint itself; use traces_endpoint instead to give the full URL. encoding defaults to proto; json works too. k8sattributes adds the Kubernetes resource attributes of the next section (its defaults already extract the namespace, pod, Deployment and node names) and needs the RBAC its README describes; it ships in the contrib and Kubernetes distributions of the Collector. Recent Collector releases spell these two components otlp_http and k8s_attributes and still accept the names above as deprecated aliases. The Collector reads the bearer from its own environment: give its pod a Secret holding the token.

From an SDK, with no Collector

Every OpenTelemetry SDK reads the standard variables. In a pod, with the bearer in a Secret of the application's namespace (kubectl -n APP_NAMESPACE create secret generic orbit-stack-otlp --from-literal=token="$ORBIT_STACK_INGEST_TOKEN", where APP_NAMESPACE is the namespace the application runs in). Hand applications the ingest token, never ORBIT_STACK_AUTH_TOKEN: with the ingest token a pod in that namespace can send spans and cannot read anyone's metrics, logs or traces.

yaml
env:
  - name: POD_NAME
    valueFrom: { fieldRef: { fieldPath: metadata.name } }
  - name: POD_NAMESPACE
    valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
  - name: NODE_NAME
    valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
  - name: OTLP_BEARER
    valueFrom: { secretKeyRef: { name: orbit-stack-otlp, key: token } }
  - name: OTEL_SERVICE_NAME
    value: checkout
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: k8s.namespace.name=$(POD_NAMESPACE),k8s.pod.name=$(POD_NAME),k8s.node.name=$(NODE_NAME),k8s.deployment.name=checkout,service.version=5.0.2,deployment.environment.name=production
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://sreorbit-orbit-stack.sreorbit.svc.cluster.local:8082
  - name: OTEL_EXPORTER_OTLP_PROTOCOL
    value: http/protobuf
  - name: OTEL_EXPORTER_OTLP_COMPRESSION
    value: gzip
  - name: OTEL_EXPORTER_OTLP_HEADERS
    value: Authorization=Bearer%20$(OTLP_BEARER)

Kubernetes expands $(NAME) from the variables above it. %20 is the space, percent-encoded as the header variable's format requires. OTEL_EXPORTER_OTLP_ENDPOINT is the base URL, and the SDK adds /v1/traces; the per-signal OTEL_EXPORTER_OTLP_TRACES_ENDPOINT takes the full URL instead.

The engine stores the k8s.* attributes an SDK sends as they come: it does not check them against the sending pod, as orbit-collector does. A pod holding the ingest token can name another pod, namespace or workload in its spans. When the namespace on a span must be the one it came from, send through the chart's collector (apm.collector, engine orbit) and leave the applications' namespaces out of the engine's allowFromNamespaces.

Let the spans through the NetworkPolicy

With networkPolicy.enabled (the chart's default) the engine admits orbit-api, the chart's own agent and the demo shop on 8082, and nothing else. On a CNI that enforces policy, a Collector or an application in another namespace needs that namespace named:

bash
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \  --set 'components.orbit-stack.networkPolicy.allowFromNamespaces={observability}'

observability is the namespace your Collector runs in; list several separated by commas. The plain manifests need the same rule added to sreorbit-orbit-stack in 80-networkpolicies.yaml.

Resource attributes that tie a service to Kubernetes

The engine names a service by service.name (a span without it is unknown_service, the SDKs' own default) and remembers, per service, the Kubernetes attributes of the resources that sent it spans:

Resource attributeWhere it shows
k8s.namespace.name, k8s.deployment.nameThe service's "Runs on" panel and the Workload column, linked to the Deck's workload page
k8s.pod.nameThe pods of the service, and the span drawer's Infrastructure tab: the pod's CPU, memory and restarts, and its logs within a minute of the span
k8s.node.nameThe node in the span drawer
service.version, deployment.environment.name (or deployment.environment)traces_target_info, below

Without them a service still gets its rates, latencies, map and traces, and the Deck says the spans carry no Kubernetes resource attributes. Peers that never send spans (a database, a queue, a third-party API) are named from the client span's attributes: peer.service (or service.peer.name), server.address (or net.peer.name), network.peer.address (or net.sock.peer.addr), db.system (or db.system.name), db.namespace (or db.name), messaging.system and messaging.destination.name (or messaging.destination).

Through the chart's collector, an app that sends its own spans gets its own pod's k8s.* attributes, whatever it sent, and keeps its service.name when its pod's spec declares that name. Only pods of the release's node namespace and of apm.collector.orbit.trustedNamespaces may name other pods. The rules and how to declare a name are in Security.

Versions without service.version

A service whose spans carry no service.version still has versions: orbit-api takes them from its Deployment's ReplicaSets, named in each pod's name (<deployment>-<pod-template hash>-<suffix>). A ReplicaSet's version is the image tag of the app's container, followed by its pod-template hash (1.31.6 · 6f7b5956c6) when another kept ReplicaSet of the same Deployment has the same tag (a kubectl rollout restart, say), so two ReplicaSets never share one. The app's container is the one named like the Deployment, else the one two signals agree on: its image changed between the Deployment's ReplicaSets, and it declares a port. A signal no container shows has no say. When they disagree or leave several (a rollout of a sidecar alone, sidecars and no change yet), or the image has no tag or is tagged latest, the version is the hash alone. The service page shows, as Version followed by "(inferred)", the version of the ReplicaSet behind the pods that sent the window's newest spans, or the current ReplicaSet's when they name no pod: a stuck rollout shows what still serves, and a past window what ran then. Image shows the app's container. The faulty release card heads its version columns "Version (inferred)", every sentence of the finding names the version with "(inferred)", and the faulty release detector judges these versions as it judges an SDK's. A service with service.version anywhere in the last 90 minutes keeps only those. Pods of a StatefulSet, DaemonSet or CronJob, and services outside Kubernetes, have no version to infer: for a scope with only such services, GET /api/v1/nocturne/insights lists errors_after_rollout under noVersion, never under checked. A release the detector could not judge (too few requests, no version before it) leaves error and latency spikes to the spike detectors, and a latency spike names a rollout within 5 minutes of its step. Once a service has a faulty release finding, a spike from the hour's first version change on yields to it.

The rollout markers on a service's charts and its operations' charts, and Last rollout, come from the same ReplicaSets' creation times, so they outlive the API server's events (kept 1 hour by default). While the Deployment is back on an older ReplicaSet, a newer one whose pods never sent a span (an image that never pulled, then undone) gets no marker; the next rollout forward brings it back, since only a rollback reads the spans. A rollback to a ReplicaSet older than another is marked "Rollback" when the last newer one that ran sent its last span, a time a replaced pod does not move; with no span to read, when the Deployment's Progressing condition last named it. Only the last such return is known. Last rollout shows the UTC time today and how long ago on another day.

What the engine derives

Every accepted span also feeds a generator that writes ordinary series into the engine's own store every 15 seconds, under the names and labels of Tempo's metrics-generator, so a dashboard or an alert written for Tempo's output reads this engine's:

SeriesLabels
traces_spanmetrics_calls_totalservice, span_name, span_kind, status_code
traces_spanmetrics_latency_bucket, _sum, _countThe same, plus le, in seconds
traces_spanmetrics_http_4xx_totalThe same as the calls, for server spans only: those whose http.response.status_code (or http.status_code) is 4xx, which the semantic conventions leave unset; this engine's own, from 2026-11-06
traces_service_graph_request_totalclient, server, connection_type
traces_service_graph_request_failed_totalThe same
traces_service_graph_request_server_seconds_bucket, _sum, _countThe same, plus le
traces_service_graph_request_client_total, traces_service_graph_request_nested_total, traces_service_graph_request_nested_seconds_totalclient, server; this engine's own, Tempo writes no such series
traces_target_infojob (the service), k8s_namespace_name, k8s_deployment_name, k8s_pod_name, k8s_node_name, service_version, deployment_environment; 1 while a pod sends spans
  • Label values are Tempo's: span_kind="SPAN_KIND_SERVER", status_code="STATUS_CODE_ERROR".
  • Latency buckets: 1, 1.5, 2, 3, 5 and 7.5 in each decade from 1 ms to 30 s, 28 bounds, six rows a decade for the latency heatmap. They include every bound written before 2026-10-28 (1, 2, 5, 10, 20, 50, 100, 150, 200, 300, 500 and 750 ms, then 1, 2, 5, 10, 20 and 30 s) under the same le, so those series carry on unchanged through an upgrade or a rollback. A label set writes 31 latency series instead of 21.
  • Counters are cumulative since the engine started, so a restart is a counter reset that rate() handles; a series gets a zero one interval before its first value, so its first calls count in increase().
  • An edge is a client span and the server span it parented, matched by span id alone, so the two pair even when they arrived under different traces. A client span with db.system, db.namespace or db.name is an edge to that database at once (connection_type="database"). Producer and consumer spans with messaging.system are two edges through the queue, producer to destination to consumer (connection_type="messaging_system"; Tempo pairs the two services directly). A server span with no parent is an edge from user, and a client span nobody answered within 10 s is an edge to its peer.service or server.address (both connection_type="virtual_node"). Without a server span, the edge's latency is the client's.
  • traces_service_graph_request_client_total counts the calls a service's client spans made on an edge, and the nested series those of them inside a request, with the edge's seconds: the client span has a parent, or the engine stitched it under one within 2.5 minutes. A client span with no parent is background work. A call under a parentless internal span, such as an SDK worker's job, has a parent and counts as nested, as does orbit-probe's copy of a call an SDK traced with no parent. An edge's background calls are the first less the second, so none that an engine before these series counted reads as background. They stop being written when the rollups below let go of the edge, about 32 minutes after its last span.
  • The engine names each operation once at intake. A server span with http.route and http.request.method (or http.method) takes the name {method} {route}. In any name, a word or path segment that is a number, an email, a UUID or hex digest, or a token of 16 or more characters with 3 or more digits becomes {id}, and a URL's query string goes: GET /users/12345?tab=2 is GET /users/{id}. When the name changes, the span keeps the one it arrived with in orbit.raw_name, so you can still search for it. Ingestion rules see the new name under name; match attr.orbit.raw_name for the old one.
  • A service holds at most 1000 operations and, as a client, 1000 edges. A new one past that, or past the generator's 20000 label sets, counts as span_name="(other)" or server="(other)", and orbit_stack_trace_generator_overflow_total counts its spans. The ones it already holds keep counting.
  • The generator holds at most 20000 label sets, plus 2000 for (other); past that it drops new ones and counts them in orbit_stack_trace_generator_dropped_total. It holds at most ORBIT_STACK_TRACE_GRAPH_PENDING spans (default 100000) waiting for the other side of an edge; past that a span goes unpaired, the edge undercounts, and traces_service_graph_dropped_spans_total counts it. A series that saw no span for 15 minutes stops being written.
  • The services summary and the service map also read rollups the generator keeps per service and per client and server pair, outside the label sets above, for its last 64 collections (16 minutes at 15 s). A window that starts inside them lists every service that sent spans, with the spans the collections inside the window counted, without extrapolating to the window's edges as increase() does. They hold at most 100000 services and 20000 edges, plus one server="(other)" edge per client past that. A longer window, or one that starts before the engine did while older series exist, reads the series.

Stitched traces

orbit-probe gives each TCP hop its own trace. A client span the probe records while the same process serves a request takes that request's trace and names the hop's own trace in orbit.hop.trace_id. The engine then keeps the hop's spans under the request's trace:

  • Hop spans that arrive after the client span move to the request's trace at intake. Those already in the live head merge into it there. A chain of hops ends at the outermost request, at most 16 deep, so one trace runs from the HTTP request through its Redis and SQL calls to the next service and its own database calls.
  • An alias lives 15 minutes, and the engine holds at most 262144 of them. orbit_stack_trace_aliases counts the live ones and orbit_stack_trace_alias_events_total{event} counts what happened to them: recorded, refused (a trace named as its own hop, a loop, or a hop already stitched into another trace), expired, evicted, rewritten spans and merged head parts.
  • A hop part the head wrote to a segment before its client span arrived stays under its own id. GET /api/v1/traces/{id} follows orbit.hop.trace_id to it and returns its spans under the request's trace id, at most 16 hops deep and ORBIT_STACK_TRACE_MAX_BYTES of followed spans.
  • The write-ahead log holds the spans as stitched, and a restart replays the aliases with them.

Calls with no parent

When orbit-probe loses the context of a call, a Redis EXPIRE or a SELECT reaches the engine as a client span with no parent and its own trace. This happens when the probe misses the hand-off from nginx to a php-fpm worker. The engine puts such a call inside the request that made it when only one request can be the caller:

  • Same process. The call's process (same pod and process.pid) served requests, and when the call started exactly one trace was in flight on its thread.id, or in the whole process. A request here is a server or consumer span, or a span with a parent that hands work to a process, such as orbit-probe's FastCGI span. Requests chained by parent (nginx, its FastCGI span, the php-fpm hand-off) count as one; the call goes under the innermost.
  • Same pod. The call's process shows no request around it (it never served one, or the probe missed this hand-off), another process of the same executable in the pod served requests, and exactly one chain was in flight in the pod. The engine decides once a request of the pod ends 2 s past the call's start plus the pod's longest request and the longest a request of the pod arrived late (each at most 60 s), so every request around the call has arrived.
  • Ambiguity stops it. A process seen with two traces in flight at once on one thread stops matching by thread; seen with two at once at all, it stops matching by process. A pod needs 8 requests before its calls are stitched. The engine follows a process from its first parentless call, and that process needs 8 requests after it.

A stitched call takes the request's span as its parent and orbit.link = stitched, and its trace moves into the request's trace, so the root-traces list stops showing it on its own. A call that named its downstream hop (orbit.hop.trace_id) also leaves its trace as an alias of the request's trace, so the hop's later spans land there too. The trace page draws its elbow dashed, like the probe's own inferred links. Calls from queue workers, crons and heartbeats have no request around them and stay their own traces.

eBPF and code together. When a pod also sends spans from an SDK, an eBPF call waits up to 10 s of the pod's SDK span time for the SDK client span of the same call: one that wraps it (exactly, else within 1 ms) and whose name holds the call's first word (GET in redis redis-global-5 get). The engine drops the eBPF copy and keeps the SDK span in its trace only when exactly one such span wraps the call and that span wraps no other eBPF call of the pod: an SDK that samples leaves requests out, and their calls can fall inside the spans of a request it kept. An SDK span that may have wrapped a call the engine dropped for room, or one sent before the pod's first SDK span, merges nothing. Every call that does not merge goes on to the rules above. SDK requests stand in for the pod's requests only once the SDK has wrapped 64 eBPF calls of the words it uses and missed at most 1% of them; until then, and in a pod whose SDK samples, the rules use orbit-probe's requests alone. Matching uses the pod, not the pid, since an SDK in a container sees its own pid namespace. An eBPF span that already has a parent, such as one the probe linked into the SDK's trace through traceparent, is not merged yet.

The engine matches only in its live head: both traces must still be there. It follows at most 4096 processes with parentless calls and 4096 pods for 10 minutes after their last span, 64 requests each, and 65536 waiting calls; past that, the pod holding the oldest waiting call drops all of its own. A call waiting for its SDK twin takes its request while the pod still lists it. orbit_stack_trace_alias_events_total{event} counts stitched calls and twins merged into their SDK span.

Root traces

A root trace has a span with no parent, or a span whose parent is not in the trace: an orbit-probe server span names its caller's client span even when the probe never saw it. The root span is the earliest parentless span, else the earliest of those orphans. Almost every trace is a root trace; its root span's kind tells a request (server, consumer) from background work (client, producer, internal), such as the Redis heartbeat a queue worker sends with no request around it.

GET /api/v1/traces?root=true keeps root traces and rootKind=server,consumer keeps those whose root span has one of the listed kinds; either one adds rootKind to every hit. The engine records the kind in each trace's summary row, in each chunk's zone map and in the segment's aggregates, so this search reads no spans. Segments written before this release carry no root kind: there root=true falls back to a parentless span, and rootKind matches nothing.

Up to 10 key (attribute names) on GET /api/v1/traces add custom: {key: {values: [{value, count}], capped}}: each key's 10 most frequent values over every trace the search matches in the window, a value counted once per trace that carries it (a span's own attribute first, then its resource's). The Deck's custom facets read it under any filter beyond a status. That search reads the spans of every match, so it runs until the request's timeout, and partial says it stopped there. capped says the search met more than 4,096 distinct values of that key: its counts are then lower bounds.

A request rate and error ratio per service, for a panel or an alert:

promql
sum by (service) (rate(traces_spanmetrics_calls_total{span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))
sum by (service) (rate(traces_spanmetrics_calls_total{span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER", status_code="STATUS_CODE_ERROR"}[5m]))
  / sum by (service) (rate(traces_spanmetrics_calls_total{span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))

The built-in dashboard orbit-apm-red draws the same series (dashboards.md).

Reading traces in the Deck

All under /deck/apm, with the apm.read permission, over the window picked in the top bar:

  • Services (/deck/apm): every service with its request rate, error rate, p50, p95 and p99, operation count and workload. A service is degraded at 1% errors and failing at 5%; one with no requests in the window shows as having no traffic. A service's requests are its server and consumer spans, or all its spans when it has neither.

  • A service (/deck/apm/services/<name>): requests by outcome, error rate, latency and its heatmap over time (of the whole service or of the operation picked above it), the operations table, what calls it and what it calls, recent traces and recent errors, and the Kubernetes objects it runs on. Workers and crons that report the same service.name count under it, so the page keeps them apart from its requests. Time by dependency is each callee's share of the time inside the service's requests, from the nested series. Calls leads with the calls made inside a request and says how many ran in the background; its errors and p95 cover every call. Recent traces, recent errors and error types read the traces a request started; when the window has none, they read every root, and the lists say so. Pods sending spans counts every pod whose spans carry the name, which can be more than the replicas of the workload named above it. A background service reads all its calls instead. In a window that reaches back before the engine wrote these series, that chart and table read every call made before then, and none of them as background. Time spent in the operations table is each operation's share of the time the listed operations took (their span durations added up), and the table sorts by it. Slower than p95, here and on an entry operation's page, opens the traces whose entry span took longer than the window's p95.

  • An operation (/deck/apm/services/<name>/resource?op=<operation>, a row of the operations table): its requests by outcome, error rate, p50, p95 and p99 (with the p95 of a day earlier and the service's rollouts on the charts), its latency heatmap, its error types, recent traces and errors, and a span summary. The summary reads 20 traces of the operation spread over the window and lists where its requests came from (the nearest request above each, or none when it starts its trace) and what they ran, by service and operation: spans per request, the share of requests that ran it, average duration, and its own time inside the request (children taken out) as a share of that request's own time, averaged over the requests, so a stream one request left open counts as that one request. A row opens that operation's own page, so you can walk up or down the calls. It is a sample: a rare branch can be missing from it.

  • Service map (/deck/apm/map): services, databases, queues and external calls as nodes, calls as edges with rate, error rate and p95; an edge turns red at 5% errors. When spans went unpaired in the window, the map's answer says partial: true with unpairedSpans, because its edges undercount. Past 300 nodes the Deck draws the 300 busiest and says how many it left out; Service map on a service's page draws that service with its neighbours first. An edge's width is all its calls; its label and a node's panel lead with those made inside a request and name the rest as background.

  • Root traces (/deck/apm/traces, the default list): the traces a request started, whose root span is a server or consumer span, searched with the query bar below. Under the bar, traces per step as stacked columns, passing in grey and failed in red, with the failed share as a line on its own right axis; a click on a column narrows the window to it, and a click on a line also filters on the value it draws. The controls read as a sentence, Show [measure] by [key] top [n] every [step], for the error rate or a duration percentile, overall or by service, root operation, pod, node, version, route or error type; by a key, as lines over time or as a top list of the whole window, ranked by the measure. The duration histogram, the scatter of duration over time, the latency heatmap (below) and the comparison with passing traces open from Charts on the line above the results, which also holds the findings; the browser remembers whether you left them open. A trace whose root is a client, producer or internal span had no request behind it: a queue worker polling Redis, a cron's query, a heartbeat. These background calls are hidden by default, because on a busy cluster they outnumber the requests many times over. A link above the list says how many it hid and shows them in one click (?background=1). Each row whose root is not a server span names its kind. An engine older than Addendum v0.27 does not report root kinds: the page then lists every trace, with no kind and no such link, and says so.

  • Saved views beside the Search button on both lists: the team's views of this page, as on Logs (logs.md → "Saved views").

  • All spans (/deck/apm/traces?scope=spans): stored spans one by one, newest first, 100 to a page, with the same filters plus span kind and minus maximum duration. Older spans moves the page to just before the oldest span shown. Each row opens its trace on that span.

  • Query bar: type filters as key:value separated by spaces; all of them must hold. Each one turns into a chip under the bar: select it to edit it, its minus to exclude it, its x to remove it. service:, operation:, status:error or ok, duration:>100ms and duration:<2s (units ns, us, ms, s, m, h; two chips make a range) and, on All spans, kind: are named keys; any other key is a span or resource attribute, as in http.route:/api/cart or user.tier:"gold plus". A minus excludes an attribute (-http.response.status_code:200) or a service (-service:frontend): traces where any span carries it drop out. Up to 8 attributes to meet and 8 to exclude. Datadog's @duration:>100ms works as duration:, env:prod reads the resource's deployment.environment.name and resource_name: the span name; operation_name: has no OpenTelemetry field and is refused. Wildcards, comparisons on attributes (http.response.status_code:>=500) and free text are refused with the reason: the engine indexes exact key and value pairs. In quotes a value is taken as written, * included (http.accept:"*/*"). The list counts its matches for 0.4 s at most; past that it shows the newest ones exactly and says "more than N". Switching between Root traces and All spans keeps the terms the other view takes and drops the rest (kind:, a maximum duration). Tab completes keys and values: services and operations from the window, attributes from its newest 200 spans. The query lives in the URL (?q=), so back, forward and a pasted link replay it; links that still carry ?service=, ?attr=k=v and the other old parameters open as chips. Each facet value adds its chip in one click, and its minus adds the exclusion.

  • A trace (/deck/apm/traces/<id>): three views of one trace, each drawing only what is on screen. Waterfall shows the spans as a tree, with the critical path and a minimap to zoom, which marks every failed span in red. In the Flame graph, spans that ran at the same time stack, so none hides behind another; scroll zooms around the pointer, Shift+scroll or a sideways swipe moves in time, and a drag pans both ways. Operations is the trace by service and operation, with calls, average, slowest, self time and time on the critical path. A row with 10 or more calls under one parent says so, which is how an N+1 looks, and a click on a row finds its spans in the waterfall. The view lives in the URL (?view=flame).

    Compare with in Operations takes another trace's id, or A typical trace, the median clean trace of the same root operation in a uniform sample of up to 200 from the hour around this one. The critical path crosses a trace's roots, orphans included, from the last to finish back; with the time no root covers, shown as its own row, it adds up to the trace's duration, so the differences per operation say where a slow trace spent what the other did not. Operations pair by exact service and name, so a name with an id in it reads as one gone and one new. The comparison lives in the URL (?compare=<id>).

    Find takes a word, or the explorer's terms: service:cart, operation:"GET /cart", status:error, kind:client, duration:>100ms, any attribute as key:value, and a minus to exclude on any of them. It reads the spans themselves, so it also takes what the explorer's engine cannot: http.status_code:>=500, OR, NOT, parentheses and Datadog's group form, service:(cart OR payment). A term it cannot answer (status:foo, a wildcard, a bracket range) says so. Self time colours the bars by the time each span spent on its own. Root errors walks the failed spans under which nothing else failed. Fold to the hops between services, beside expand and collapse, leaves one row per hop in both graphs; a folded bar says how many spans it holds. Keys, with either graph focused: arrows walk (in the flame graph, up the parent, down the first child, left and right the siblings, so spans that ran at once are one key apart), Enter opens, Z zooms to the span, 0 shows the whole trace, E goes to the next root error, / finds, and in the flame graph + and - zoom.

    The drawer for a span shows its attributes grouped by namespace, its events, resource and infrastructure. Each attribute copies its value, finds the spans that carry it in this trace, and searches the explorer with or without it. Walking spans with the drawer open reads a span's logs and infrastructure once the cursor rests for 250 ms. A dashed elbow before a span means orbit-probe inferred its parent (orbit.link is single or route); hover the link icon next to it to read how. The drawer shows how the span was linked and the process that ran it: executable, process.pid and thread.id.

  • Who called this? When a trace's root span is a client, producer or internal span, the trace page asks GET /api/v1/spans for the requests the same process (same k8s.pod.name and process.pid) was serving when that span started, and for the other spans of that process within 2 s either side, each linked to its trace. When the process was serving no request, the page says so and names the executable: the call is background work, such as a scheduled job, a timer or a queue worker. A request here is a server span, or a FastCGI span, which orbit-probe marks internal when nginx and php-fpm share a pod. The panel looks back 60 s for a request still in flight. A trace that starts at a request makes none of these reads.

  • Related: a row of links on a span's drawer, a service, a workload, a pod, a pod's terminal (each in a new tab, so the shell stays open), a node and an alert's drawer, to the other signals of the same object and window: its logs (a span's by trace_id, a pod's by namespace and pod, a service's by service, a workload's by namespace and its pods' names or its service's name, whichever finds more (a file's lines name no pod; a service's page tries its workload's pods the same way), a node's by node; an alert on a node opens the node's) with their count and errors, its traces with request and error counts (a pod's link filters on k8s.pod.name and has no count, which would be its whole service's), its service (for a pod or an alert, the service whose spans name that pod or workload in k8s.pod.name or k8s.deployment.name on traces_target_info; the row says so, with no traces link, when no span does; a worker or cron with no server or consumer span keeps the link, with no count), its workload, pod and node pages, its Kubernetes events with warnings, and the alerts raised on it. A span's window is 15 minutes either side of it; an alert's is 15 minutes either side of when it fired; every other page keeps the top bar's. The links show at once; one server call fills the counts. A span's row adds that span's lines with their count, when it wrote any, and its pod's or service's lines in the same window, and says so when no line carries the trace id. A role without logs.read or apm.read gets no link to that signal. When traces_target_info cannot be read, or the role lacks telemetry.query, a pod gets no service link and keeps its traces link; a workload gets no traces link.

Releases of a Deployment

The Releases tab of a Deployment's page (?tab=releases) answers "did this revision make anything worse, and where?" with no version label. It needs apm.read, a live cluster and the apm switch; the rollback and workload_changes switches leave it alone, so a read-only install keeps it.

  • A row per kept revision, newest first, from the Deployment's ReplicaSets.
  • One timeline of rollouts. Every revision number a kept ReplicaSet held is a rollout: its own and those in deployment.kubernetes.io/revision-history, which the controller writes when it reuses a ReplicaSet (a rollback, or a template applied again). A ReplicaSet's first rollout is its creation. A later one is when its pods came back, to the pod's own second, or, when they never left because the revision before got stuck (a crash loop the rolling update would not let replace them), when the pods of that revision were all gone, to the last scrape (kube_pod_created and the last time each pod was read). Such a row is tagged re-applied. When the engine no longer holds those pods, the row has no numbers.
  • The windows. Before is the run of the revision this one replaced, after is this revision's own run, 30 minutes each at most, and after never reaches past now. A rollout merged into a later revision number still ends the run before it, so an undo applied later does not change an older row. The oldest kept revision's run before starts with the first pod of the ReplicaSet the controller pruned (kube_pod_created). The span numbers skip each side's first minute, while the revision before still answers, so an undo read minutes later does not inherit the slow pods' tail. Reads reach back 15 days.
  • The numbers, for each service whose spans name the workload (traces_target_info, by k8s.deployment.name or its pods' names): requests and error share of the entry spans (server and consumer, as on the service page), the 4xx share of the server spans (traces_spanmetrics_http_4xx_total; a dash where the engine was not counting them at the window's start), p95, restarts of the Deployment's pods (kube-state-metrics) and new error lines: messages with 5 lines or more after the rollout and none in the 30 minutes before it, whatever ran in those 30. Error lines are read only for a role with logs.read, with stored logs on.
  • Routes, under the row: entry operations that appeared (5 calls or more, none before), went (5 or more due in the window after and under a tenth of them came, since the old pods answer while the rollout runs) or doubled their 4xx share past the bar below.
  • The verdict takes errors_after_rollout's bars. A share got worse with 100 requests or more on both sides, 5 failures or more after, at 1% or more, twice the share before and 3 standard deviations over it; latency got worse at twice the p95 before and 100 ms more, unless the run it replaced was failing (5% or more of errors and 4xx, which fail fast) and this one left half that or less; restarts, when the window after has more. A route's 4xx counts only with 100 calls after, a share at least 1 point higher and the service's own 4xx share not falling; otherwise it stays in the list as evidence. Worse names what got worse. No worse needs 100 requests on both sides of a service; with fewer the row says so, with no request on either side it says No entry spans, and with no window to read No data.
  • Copy puts the row on the clipboard as plain text for a ticket: the revision, when it rolled out, its image, the verdict, every number and route, and both windows.

Nothing in the tab opens by itself, and it raises no finding.

The latency heatmap

The Heatmap view of Root traces (?chart=heatmap) and the latency panel of a service's page draw span durations over the window: a column per time step, a row per latency bucket on a log axis, each cell coloured by how many spans landed in it, on a log scale whose legend prints the range. It counts every span, whatever the ingestion rules stored, because it reads the span metrics (ingestion.md):

  • With a service in the query, that service's server and consumer spans, narrowed by operation: and status:. On a service's page, the whole service or the operation picked above the chart (?op=); a background service's client and producer calls instead.
  • With no service, the requests entering the system: the service graph's calls from user.
  • A duration in the query is outlined on the heatmap, not filtered out of it, so the rest of the distribution stays in view.
  • An attribute (or operation: or status: with no service) is something the span metrics do not label. The heatmap then counts the stored traces the list would match, in one pass over the window, and says so under the chart; a pass that runs out of time says its counts are a lower bound.

Hover over a cell to read its time, its duration range and its count. Click a cell, or drag a rectangle across cells, and the list narrows to those minutes and that duration range: the time range in the top bar (a single cell's is widened to two minutes around it), and duration:> and duration:< in the query. On the keyboard the plot is one tab stop: the arrows move between cells, Shift with the arrows selects a rectangle, Enter opens it and Escape drops it; a screen reader hears each cell. The list holds stored traces only: when the ingestion rules dropped spans in the engine's last 5 minutes, hour or day (the one closest to the window), the line under the chart gives the share they kept.

A window gets about 120 columns: 30 s steps for an hour, 15 minutes for a day, 2 hours for a week, never finer than the 15 s at which the span metrics are written. The route is GET /api/v1/apm/heatmap (api.md).

Wherever a stack trace shows, in a span's exception event (exception.stacktrace) or in a log line (the message, its exception.stacktrace, error.stack or stack_trace attribute, or any string field of a JSON message, Monolog's trace array included), the Deck turns each frame into a link to its file and line in the repository, like Datadog's Source Code Integration. It reads PHP (getTraceAsString, PHP's own in /path.php:12 messages, Monolog's path:line entries and the OpenTelemetry PHP SDK's at App.Services.PaymentService.charge(PaymentService.php:87)), Java, Kotlin and Scala, Python, Node.js, Go, .NET and Ruby.

  • Which commit. The span's or the log's own vcs.repository.url.full and vcs.ref.head.revision (OpenTelemetry), or Datadog's git.repository_url and git.commit.sha, on the resource or the record. For a service that sends neither, an admin sets the repository and branch under Integrations, Source code.
  • Which file. The frame's path minus the directory the image copied the repository into: the one set for the service (Path in the container), else a common one (/var/www/html, /app, /usr/src/app, /src, /build, /go/src/app; the full list is ROOTS in apps/web/src/lib/stack.ts), a Go module path or GOPATH with the repository's own name, or a GitHub Actions or Azure Pipelines checkout. A JVM frame names a class, so its file is the package path under src/main/java (src/main/kotlin, src/main/scala). The OpenTelemetry PHP SDK writes the class and the file's base name only, so the Deck builds the path from Laravel's namespaces: App\ under app/ (in a trace with Laravel's frames; Symfony and Slim keep App\ in src/), Database\Seeders\ and Database\Factories\ under database/, Tests\ under tests/. Any other namespace (Modules\, a package missing from the vendor list) gets a link only from the repository's file list, with a GitHub token. {main} sits at public/index.php or artisan; a closure outside a class or in Laravel's RouteFileRegistrar at routes/web.php or routes/api.php; a closure of routes/console.php (Artisan::command, Schedule::call) carries Laravel's console Kernel or ClosureCommand and links there. A trait's method names the class that uses it and the trait's file: Laravel's own traits (AuthorizesRequests, Notifiable and others; the list is PHP_VENDOR_TRAIT) count as vendor code, and any other gets a link only from the repository's file list, with a GitHub token. Directory in the repository goes in front of the result, for a monorepo or a Maven module.
  • Vendor code. Frames in vendor/, node_modules/, site-packages/, gems, the Go module cache and standard library, the JDK and common JVM frameworks, System. or Microsoft. in .NET, PHP namespaces of common packages (Illuminate, Symfony, GuzzleHttp, Doctrine, Monolog and others; the list is PHP_VENDOR_NS in apps/web/src/lib/stack.ts), PHP's own functions and classes, compiled Blade views and eval()'d code stay as dimmed text with no link.
  • The code inline. With a GitHub token saved under Integrations, each linked frame gets a button that shows the 11 lines around it, which orbit-api reads from api.github.com at that commit. orbit-api seals the token with ORBIT_SEAL_KEY, never answers it, and sends it to api.github.com only, so a frame on another host gets a link and no code. orbit-api reads only a repository linked to a service under Integrations, Source code, at the ref that service sets (any ref when it sets none) or at a full commit id, the one telemetry carries for the build. A repository that telemetry names and no service links gets links and no code: link it to see its code. Use a fine-grained token with read-only access to the Contents of the repositories you link: any role with apm.read or logs.read reads their code through it.
  • The repository's file list. With the token, orbit-api also reads the repository's file list at that commit, and each frame links to the file that shares the longest tail of path segments with it: a Maven module, a monorepo directory or an unusual WORKDIR needs no setting, and a frame whose file is not in the repository (a third-party JVM package, say) gets no link. When two files tie, the one under the service's Directory in the repository wins; with no directory set, or none of them under it, the frame gets no link.

GitHub, GitLab, Bitbucket and Azure DevOps (https) URLs link; only GitHub shows code inline. The parser reads at most 256 KiB of text and skips lines over 4 KiB, so hostile text costs linear time. A TypeScript or bundled JavaScript frame links to the compiled file (dist/...): the Deck does not read source maps.

The routes: GET /api/v1/source (apm.read or logs.read) answers the services and whether a token is set; PUT /api/v1/source (settings.manage, audited) saves them; GET /api/v1/source/snippet?repo=&ref=&path=&line= (apm.read or logs.read) answers {"path", "start", "lines"}; POST /api/v1/source/resolve (apm.read or logs.read) takes {"repo", "ref", "dir", "files"} (at most 200 files) and answers {"paths"}, the repository path of each file or null (also when more than 256 repository files share its name). Both answer 403, before any call to GitHub, for a repository or ref no service links.

The same reads over the API are the apm.read routes in api.md. A span time is epoch microseconds (startUs, durationUs); every other time on those routes is epoch milliseconds.

Service SLOs

A service page's header carries Set SLO for a role with settings.manage, on a service that took requests in the window (a worker or a load generator has none to keep). An SLO is one availability target on one service: the share of its entry requests (server and consumer spans, the ones its pages count) that end without an error over the last 7 days, which the engine's 15 days of metrics hold. The header then reads "SLO 99.5% over 7 d: 99.93%, 86% of the error budget left", and the SLO 7d column of Services shows the same share with the budget under it; a service without an SLO leaves both empty. Both are cut down, never rounded up: 98.996% of a 99% target reads 98.99%, and a budget just spent reads -0.4%, not 0%. While the data holds less than the 7 days (a new service, a young engine), the header says how much: "SLO 99.5% over 7 d, 3 h of data: ...", its start read to the hour. The budget is 100% minus the target, and what is left is 1 minus the failed share over the budget, below 0 once the week spent it.

Saving writes two rules into the PromQL alert rules (mcp.md), the Google SRE workbook's multiwindow burn-rate alerts, where a burn rate of 1 spends the budget in exactly the window:

RuleFires when the failed share over the budget passesSeverity
slo-<id>-fast14.4 over the last hour and over the last 5 minutescritical
slo-<id>-slow6 over the last 6 hours and over the last 30 minutescritical

Both page, as Grafana SLO's fast burn does: over 7 days, the fast rule fires once an hour spent 8.6% of the week's budget, the slow one once 6 hours spent 21%. Each carries slo: <id> and is named SLO <service>: 14.4x 1h/5m (or 6x 6h/30m). Its alert's value is the burn over the long window, and it carries the service's namespace, which the APM pages name and the save reads, so a notification channel kept to that namespace hears it; a channel says the burn in its own language. On Alerts, its row opens the service and the service's logs. Saving the SLO again rewrites both rules (and reads the namespace again) and keeps off one that someone switched off; removing it deletes both.

On Observability, Error budget burn draws each SLO's burn over the last hour, one line per service, and says so when the cluster has no SLO. Its latency and request rate panels and the Overview's Requests read the span metrics when no app exports http_requests_total.

The routes: GET /api/v1/alerting/slos (apm.read) answers every SLO with namespace, window, requests, errors, sli and budget (null with no request in the window), since (the hour its data starts in the window, Unix ms), and error when the engine of its cluster did not answer. PUT /api/v1/alerting/slos (settings.manage) takes {cluster, service, target}, with target from 0.5 to 0.99999, and creates or replaces the service's SLO. DELETE /api/v1/alerting/slos/{id} removes it with its two rules. A console holds 50 SLOs, and their rules count toward the 200 alert rules. Each save and removal is one audit event, slo.create, slo.update or slo.delete, that names the rules.

Storage and retention

VariableDefaultWhat it does
ORBIT_STACK_TRACE_RETENTION7dA trace segment is deleted whole once its last span arrived longer ago than this. Days or a Go duration.
ORBIT_STACK_TRACE_HEAD_BYTES64MiBSpans held in memory before they are written out as a segment. The read cache of decompressed traces gets half this budget and the OTLP request bytes in flight the same budget, so traces add about two and a half times this figure to the engine's memory. At least 1MiB.
ORBIT_STACK_TRACE_MAX_BYTES5MiBOne trace in the head, Tempo's max_bytes_per_trace default: spans past it are refused one by one. From 64 KiB to half of ORBIT_STACK_TRACE_HEAD_BYTES.
ORBIT_STACK_TRACE_GRAPH_PENDING100000Spans waiting for the other side of a service-graph edge, 1000 to 10000000; past it they go unpaired and the service map says so.

Spans go to a write-ahead log first, then to the in-memory head; the head becomes an immutable segment when it reaches ORBIT_STACK_TRACE_HEAD_BYTES or is 15 minutes old, whichever comes first. A segment, format version 2, holds whole traces as zstd frames in start order, summaries in chunks with a zone map each, a sorted id table and a bloom filter, every part checksummed. A segment the engine cannot read, including a version 1 segment from the first trace release, is skipped at start by name (the log line trace segment skipped names the file), counted and left on disk for you to delete; the write-ahead log replays whatever its version. Everything lives under ORBIT_STACK_DATA_DIR/traces, on the engine's volume, beside the metric blocks. Beside each segment the engine keeps agg-<seq>, its aggregates: per service, entry span and status, how many traces it holds and the few a search keeps of them, so a search with a total or a spread sample reads a segment its window covers whole without decoding it (about 5% of the segment bytes on disk, a few KiB of memory per segment). The file is rebuilt from the segment when it is missing or does not verify, and the Go engine ignores it, so a rollback keeps working; one left behind by the Go engine's retention is deleted at the next start. POST /-/flush writes the span metrics and the head out immediately. Size the volume from your own traffic: GET /api/v1/status/tsdb reports segmentBytes and segmentSpans under traces, which give your bytes per span; multiply by your span rate and the retention.

Metrics to watch, on the engine's metrics listener: orbit_stack_spans_total{outcome} (accepted, rejected), orbit_stack_trace_head_bytes, orbit_stack_trace_head_spans, orbit_stack_trace_segments, orbit_stack_trace_segment_bytes, orbit_stack_trace_wal_bytes, orbit_stack_trace_cache_bytes, orbit_stack_trace_skipped_segments, orbit_stack_trace_cuts_total, orbit_stack_trace_cut_failures_total, orbit_stack_trace_wal_truncations_total and orbit_stack_trace_generator_dropped_total. GET /api/v1/status/tsdb carries the same figures under traces.

The demo shop

orbit-shop, shipped in the orbit-stack image as /orbit-shop, is a small shop made of real HTTP services that call each other with W3C traceparent and export OTLP to the engine, so the APM pages have traces that went through the real pipeline:

  • web-frontend calls api-gateway, which calls auth on every request and then catalog (which queries postgres), cart (which talks to redis) or checkout.
  • checkout calls payment, which calls stripe-api (an external peer that sends no spans); once the charge succeeds, checkout publishes to the Kafka topic orders, which fulfillment-worker consumes before writing to postgres.
  • payment fails a small share of charges (a gateway timeout or a declined card), cart answers the odd 500 on a Redis pool timeout, and for one minute in every ten a third of catalog's queries take a slow plan: there is always something to find.
  • A load generator drives web-frontend like shoppers, mostly browsing, some carts, fewer checkouts, at --rps requests a second (default 5); each request is one trace.
Install pathHow it runsSwitch
ComposeThe orbit-shop service, orbit-shop --all: every service in one process on 127.0.0.1:18110 to 18117, two made-up pods eachOn by default. ORBIT_SHOP_RPS in .env sets the rate; docker compose --project-name sreorbit --env-file .env -f deploy/compose/docker-compose.yml stop orbit-shop stops it.
HelmOne Deployment and one Service per service, plus a load generator Deployment, each pod naming itself through the downward APIdemo.shop.enabled (default false), demo.shop.rps (default 5), demo.shop.resources
Plain manifestsNot shippednone

On Helm:

bash
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \  --set demo.shop.enabled=true --set demo.shop.rps=5

The chart refuses demo.shop.enabled without components.orbit-stack.enabled. The shop pods use the engine's bearer from the release Secret, and a NetworkPolicy lets them reach each other on 8080 and the engine on 8082, nothing else. They are load you did not ask for, which is why the chart leaves them off.

Zero-touch APM with the Helm chart

Two blocks of the chart's values.yaml, both off by default, send traces with no change to the apps. apm.collector is an OpenTelemetry Collector at <release>-otel-collector on 4317 and 4318 that forwards to this engine: by default orbit-collector, the Rust collector (apm.collector.engine: orbit); upstream runs otelcol-k8s behind the same Service, ports and pipeline. orbit-collector holds at most a fifth of its memory limit (64 MiB at least) in request bytes at once, as received and decompressed; a request past that gets 503, or gRPC UNAVAILABLE, and the exporter retries it. A request whose resources, scopes and spans would take more than four times its size once split apart (an empty one is two bytes) gets 400, or gRPC INTERNAL, as one that does not parse.

orbit-collector tells who sent a span by the address of the connection. A pod in the release's node namespace (orbit-probe, OBI, orbit-node) or in apm.collector.orbit.trustedNamespaces reports for other pods, so the collector finds the pod from the span's own k8s.pod.uid, k8s.pod.name and k8s.namespace.name, as upstream's k8sattributes does; OBI on the host network (contextPropagation) counts by its node's address. Any other sender gets the pod, namespace, workload and node of the pod at its address, whatever its spans say: the collector drops every k8s.* attribute it sent and writes its own, and replaces its service.name unless the pod's spec declares that name (Security). A sender outside the cluster keeps none of them, and neither does one on the host network, unless a hostNetwork pod of a trusted namespace runs on its node: apm.autoInstrument.contextPropagation (OBI) and telemetry.nodeExporter.engine: upstream each put one on every node they cover. From then on any hostNetwork pod on that node connects from the node's address and keeps the attributes its payload names. Keep application namespaces at Pod Security baseline or stricter, which refuses hostNetwork, so only the release's node namespace runs pods on the host network. Add the namespace of a Collector gateway of your own to trustedNamespaces when it forwards spans for other pods. Keep the collector's pod out of a service mesh: a sidecar in front of it connects from its own address, so every sender would lose its pod. apm.collector.engine: upstream keeps otelcol's own association, which believes the pod a payload names.

Span attributes named orbit.* and a resource's telemetry.sdk.name of orbit-probe belong to orbit-probe: the engine stitches traces by them. The collector drops them from any sender it does not trust, so an app cannot move another service's trace into its own with orbit.hop.trace_id, set orbit.sql.signature itself, or pass for the probe.

apm.autoInstrument runs an eBPF DaemonSet in the release's node namespace that turns the calls of every process in the chosen namespaces into spans and sends them to that collector. It has two engines, and renders exactly one:

apm.autoInstrument.engineDaemonSetWhat it turns into spans
orbit (the default)<release>-orbit-probe: orbit-probe, eBPF instrumentation in Rust (services/rust/orbit-probe)HTTP/1.1, HTTP/2 and gRPC, FastCGI, PostgreSQL, MySQL and Redis, in plaintext or inside TLS made by OpenSSL
obi<release>-obi: the OpenTelemetry eBPF Instrumentation (OBI) v1.0.0The same protocols, plus the others OBI knows

Both need Linux 5.8 or later with BTF (/sys/kernel/btf/vmlinux) on every node; AL2023 and Bottlerocket kernels carry it. Both send traces and nothing else. namespaces, excludeNamespaces, excludeExecutables, privileged, healthPort, logLevel and the scheduling keys apply to either engine. contextPropagation, extraCapabilities, goMemLimit, mapsScaleFactor and resources are OBI's; with engine: orbit the chart refuses the first two. Switching is one value: Helm removes one DaemonSet and creates the other.

bash
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \  --set apm.autoInstrument.engine=obi

Three defaults keep the DaemonSet cheap and the service map readable:

ValueDefaultEffect
apm.autoInstrument.excludeExecutablesshells, su and sudo, process wrappers, busybox, coreutilsGlobs on the resolved executable path that neither engine instruments, in any namespace. */busybox covers every applet. No language runtime or server is in the list.
apm.autoInstrument.goMemLimit96MiBOBI only: GOMEMLIMIT for its Go heap. The eBPF maps are kernel memory outside it. An empty string keeps the Go default.
apm.collector.dropRedisCommandsBLPOP, BRPOP, BLMPOP, BLMOVE, BRPOPLPUSH, BZPOPMIN, BZPOPMAX, BZMPOP, XREAD, XREADGROUPRedis spans the collector drops, by command: the blocking pops a queue worker or a cron job loops on. XREAD and XREADGROUP go only with BLOCK, when the span carries its query text. [] keeps every span.
apm.collector.replicaCount1Collector pods behind the Service, either engine. A pod keeps nothing between requests, so with two or more the chart spreads them across nodes (the kubernetes.io/hostname rule of topologySpread, in orbit-stack's zone) and adds a PodDisruptionBudget of one: losing a node leaves no gap in traces or logs. values-prod.yaml sets 2.

Both engines turn peer addresses into names from the Kubernetes informers first, Services and pods, then by reverse DNS. A database or cache in the cluster shows under its Service name, a managed one under its DNS name when its address has a PTR record, and only an address with neither stays a bare IP.

orbit-probe (engine: orbit)

  • In the kernel: fentry and fexit programs where the kernel's BTF allows them, kprobes otherwise, on tcp_sendmsg, tcp_recvmsg, inet_csk_accept, tcp_connect and tcp_close. On kprobes a receive is lost while more threads are blocked in one than the kernel's kretprobe pool holds (one a CPU without kernel preemption); the pod's log says when it attached that way. The bytes reach the agent through one ring buffer per node, apm.autoInstrument.orbit.ringbufBytes (4 MiB by default). A process outside the instrumented containers' cgroups sends nothing: that filter runs in the kernel.
  • Spans: one server span per request a process answers and one client span per request it makes, named the way OBI names them (GET /api/users/{id}, SELECT orders, /pkg.Svc/Method) and carrying the semantic-convention attributes this engine reads. Each span carries thread.id; each process's resource carries process.pid and process.executable.name. A SQL span's db.query.text is the statement with every literal as ? and its comments dropped. It is empty when the keyword the span is named from sits inside a string, a comment or a name, and it ends at a string holding a backslash or at a block comment with another /* inside, since PostgreSQL and MySQL read what follows those differently.
  • Connected traces: a call is the child of the inbound request its thread is serving when the call starts. With a thread per request (php-fpm, Java) the link is exact. On an event loop (nginx, Node) it is inferred, and orbit.link says how: single when one request was in flight, route when several were and the call's method and path matched the oldest of them. A call made while no request was being served (a queue worker, a timer) has no parent and starts its own trace. Both ends of a connection derive the same ids from the request's first TCP byte, so a client span and the server span on another node pair up without a header; when they land in different traces, orbit.hop.trace_id on the client span tells the store to merge them.
  • FastCGI: nginx to php-fpm (fastcgi_pass), recognised by its bytes on any port, named and attributed like HTTP with network.protocol.name fastcgi. When both ends sit in one pod, both spans are INTERNAL, so the pod's requests are not counted twice.
  • TLS: a process that speaks TLS through OpenSSL's libssl (curl, PHP, Python, Ruby, nginx, libpq, MySQL clients) has its bytes read before encryption and after decryption, at SSL_write and SSL_read, so its calls become the same spans as plaintext ones. The probe hooks a container's libssl files the first time one of its processes sends a TLS record, found in the maps of its processes and in the directories distributions keep the library in, and a hook serves every process on the node that maps that file; orbit_probe_tls_libraries counts the files. A connection that was open before the hook landed is read from its next call on. Both ends pair up without a header when both use OpenSSL and both saw the connection's first request: they count the same plaintext bytes. Not read: TLS made by Go, Java, Node or rustls, and OpenSSL driven through BIO_f_ssl or a memory BIO. Those connections produce no spans unless the process has an SDK. The hooks are uprobe-multi links on Linux 6.6 and later; before it they are perf events, which a host with kernel.perf_event_paranoid at 4 (Ubuntu's default) refuses to the probe's capabilities, and the pod's log says so. apm.autoInstrument.orbit.tls: false turns it off.
  • Service names: the resource.opentelemetry.io/service.name annotation, then OTEL_SERVICE_NAME in the process's environment, the app.kubernetes.io/name label, the owner's name and the pod's.
  • Privilege: uid 0 with BPF, PERFMON, SYS_PTRACE and DAC_READ_SEARCH added and every other capability dropped (SYS_RESOURCE too with apm.autoInstrument.lowKernel, for kernels older than Linux 5.11), a read-only root, seccomp RuntimeDefault, and /sys/kernel/btf and /sys/fs/cgroup mounted read-only. Its ClusterRole reads pods, Services, EndpointSlices and namespaces and nothing else. Its NetworkPolicy lets in only the chart's agent, on its metrics port, and lets out the collector's 4318, DNS and the API server (networkPolicy.apiServerCIDRs, without which it cannot watch pods where a CNI enforces policy). deploy/check_hardening.py holds the chart to exactly that.
  • Memory: the chart requests 48Mi and 20m of CPU, with limits of 1 CPU and 256Mi; the memory limit also holds the ring buffer the kernel charges to the pod. The agent's state is bounded (connections, pods, DNS names): past a bound it drops and counts, it never grows.
  • Its own health: /healthz answers once the programs are attached and the node's pod watch has synced; /metrics on the same port (healthPort, 9112, named metrics) counts events, lost events, spans by protocol, dropped spans and export time (orbit_probe_*). The chart's agent scrapes it as job="sreorbit", component="orbit-probe", and the Components table of /deck/observability shows its lost events over 24 hours (see operations.md).
  • SDK apps: a W3C traceparent on HTTP/1, HTTP/2, gRPC or FastCGI (HTTP_TRACEPARENT) names the trace and the parent, and orbit.link is traceparent. A process seen exporting OTLP traces to this release's collector gets no eBPF spans from then on (apm.autoInstrument.orbit.skipSdkProcesses, on by default; counted as orbit_probe_spans_dropped_total{reason="sdk_process"}). A process that exports elsewhere keeps its eBPF spans, and the shared trace id joins both.

OBI (engine: obi)

Memory, measured on k3d with three nodes and a PHP sample: the working set of one OBI pod was 255 to 264 MiB, of which 200 MiB are eBPF maps the kernel charges to the pod and 51 to 53 MiB the Go heap. With the defaults above the Go heap drops to 25 to 26 MiB and the pod to 228 to 232 MiB. apm.autoInstrument.mapsScaleFactor: -1 halves the maps (104 MiB), which puts the pod at 130 MiB; it also halves the requests OBI can track in flight at once on a node, so keep 0 on busy nodes. The chart requests 256Mi with a 768Mi limit. On nodes whose memory is already fully requested, request CPU only:

yaml
apm:
  autoInstrument:
    engine: obi
    resources:
      requests: {cpu: 50m, memory: null}
      limits: {memory: null}

The scheduler then places OBI on every node. With no memory request it ranks first for the kubelet's eviction and the kernel's OOM killer, ahead of any app pod that stays within its own request: under memory pressure you lose spans and your applications keep running.

Ceilings

  • OTLP/HTTP, traces only. No gRPC listener, and no OTLP metrics or logs: metrics arrive by Prometheus remote-write (orbit-stack.md).
  • No sampling. The engine keeps every span it accepts; sample in your SDK or Collector.
  • Single node. No replication and no object storage for traces, like the metric blocks.
  • Not stored: span links and the instrumentation scope.
  • Measured against Grafana Tempo: ahead on reading a trace by id, on ingest and on memory, and on the services summary and the service map at 50,000 traces. Search is ahead only for values common in recent data; for a rare or absent value at 3.4 M spans it is slower than Tempo, and the summary and the map read right after a flush are slower too. benchmark.md prints the figures with the loss table.

Where to go next

  • The storage engine, its other endpoints and its configuration: orbit-stack.md.
  • The built-in RED dashboard and how to build your own: dashboards.md.
  • Every trace and APM route, with its parameters: api.md.