Features
APM and traces
orbit-stack takes OpenTelemetry traces over OTLP/HTTP, keeps the spans, and derives from
them the span metrics and service graph series Grafana Tempo's metrics-generator writes,
under the same names, into its own time-series store. orbit-api serves the reads to the
Deck, which draws a services table, a page per service, a service map, a trace search and
a waterfall per trace. CONTRACT.md Addendum v0.5 is the specification; the storage
engine behind it is in orbit-stack.md.
Send traces
| What | Value |
|---|---|
| Endpoint | POST /v1/traces on the engine's API port, 8082: http://<release>-orbit-stack.<namespace>.svc.cluster.local:8082/v1/traces in Kubernetes, http://orbit-stack:8082/v1/traces inside the compose network, http://127.0.0.1:8082/v1/traces from the compose host |
| Protocol | OTLP/HTTP only. There is no gRPC listener: an exporter set to grpc, or pointed at 4317, reaches nothing. |
| Body | Content-Type: application/x-protobuf or application/json (hex ids; base64 ids and enum names are accepted too) |
| Compression | Content-Encoding: gzip or zstd, or none. orbit-collector exports zstd and falls back to gzip for 5 minutes when an engine answers 415 |
| Credential | Authorization: Bearer <ORBIT_STACK_INGEST_TOKEN> from outside the release (it can send and nothing else), or ORBIT_STACK_AUTH_TOKEN; none when ORBIT_STACK_AUTH_MODE=disabled |
| Size | 16 MiB once gzip or zstd is undone, or ORBIT_STACK_MAX_BODY_BYTES if that is smaller; one trace at most ORBIT_STACK_TRACE_MAX_BYTES (5 MiB) |
The answer comes back in the encoding of the request, as OTLP/HTTP specifies:
| Status | Meaning | The sender should |
|---|---|---|
200, empty body | Every span accepted | Carry on |
200 with partialSuccess | Some spans refused one by one (no valid trace or span id, or a trace grown past ORBIT_STACK_TRACE_MAX_BYTES), counted in rejectedSpans with the first reason in errorMessage | Not retry them |
400 with a google.rpc.Status | The body is not an export request | Not retry |
401 | Missing or wrong bearer | Fix the credential |
413, 415 | Too large (past 16 MiB once inflated, or more spans, resources or JSON objects than the in-flight budget can hold decoded); wrong Content-Type or Content-Encoding | Fix the request |
503 + Retry-After | The OTLP bytes held by requests in flight would pass their budget (ORBIT_STACK_TRACE_HEAD_BYTES, at least 32 MiB), the head is at twice its bound while a segment is written, or the write-ahead log failed | Retry after the wait |
A span is written to the write-ahead log before the answer goes out, and the log is fsynced every second: a crash of the process loses nothing it acknowledged, a power cut at most that second. A span the trace already holds (an exporter's retry) is accepted and kept once. A refused body is read to its end before the answer, so the sender reads the answer rather than a reset connection. Span links and the instrumentation scope are not stored.
From a shell
One span, as OTLP/JSON, against the compose stack or a port-forward
(kubectl -n sreorbit port-forward svc/sreorbit-orbit-stack 8082:8082). The two
timestamps are nanoseconds from your own clock (GNU date; a trace outside the window you
read is not shown):
NOW=$(date +%s%N)cat > span.json <<EOF{"resourceSpans":[{"resource":{"attributes":[ {"key":"service.name","value":{"stringValue":"checkout"}}, {"key":"k8s.namespace.name","value":{"stringValue":"shop"}}, {"key":"k8s.deployment.name","value":{"stringValue":"checkout"}}, {"key":"k8s.pod.name","value":{"stringValue":"checkout-7d9c5-abcde"}}]}, "scopeSpans":[{"spans":[{ "traceId":"5b8efff798038103d269b633813fc60c","spanId":"eee19b7ec3c1b174", "name":"POST /checkout","kind":2, "startTimeUnixNano":"$NOW","endTimeUnixNano":"$((NOW+120000000))", "attributes":[{"key":"http.response.status_code","value":{"intValue":"200"}}], "status":{"code":1}}]}]}]}EOFcurl -fsS -X POST localhost:8082/v1/traces -H "Authorization: Bearer $ORBIT_STACK_AUTH_TOKEN" \ -H 'Content-Type: application/json' --data-binary @span.jsoncurl -fsS -H "Authorization: Bearer $ORBIT_STACK_AUTH_TOKEN" localhost:8082/api/v1/traces/5b8efff798038103d269b633813fc60cThe first call answers {}; the second returns the trace with its span, its resource and
"kind":"server". On compose ORBIT_STACK_AUTH_TOKEN is the value in .env; under
make up the engine runs with authentication disabled and ignores the header.
From an OpenTelemetry Collector
Point the Collector's OTLP/HTTP exporter at the engine and let the Collector keep receiving whatever your applications send it, gRPC included:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
k8sattributes: {}
batch: {}
exporters:
otlphttp/orbit:
endpoint: http://sreorbit-orbit-stack.sreorbit.svc.cluster.local:8082
headers:
Authorization: "Bearer ${env:ORBIT_STACK_AUTH_TOKEN}"
compression: gzip
service:
pipelines:
traces:
receivers: [otlp]
processors: [k8sattributes, batch]
exporters: [otlphttp/orbit]The exporter appends /v1/traces to endpoint itself; use traces_endpoint instead to
give the full URL. encoding defaults to proto; json works too. k8sattributes adds
the Kubernetes resource attributes of the next section (its defaults already extract the
namespace, pod, Deployment and node names) and needs the RBAC its README describes; it
ships in the contrib and Kubernetes distributions of the Collector. Recent Collector
releases spell these two components otlp_http and k8s_attributes and still accept the
names above as deprecated aliases. The Collector reads the bearer from its own environment:
give its pod a Secret holding the token.
From an SDK, with no Collector
Every OpenTelemetry SDK reads the standard variables. In a pod, with the bearer in a
Secret of the application's namespace
(kubectl -n APP_NAMESPACE create secret generic orbit-stack-otlp --from-literal=token="$ORBIT_STACK_INGEST_TOKEN",
where APP_NAMESPACE is the namespace the application runs in). Hand applications the ingest
token, never ORBIT_STACK_AUTH_TOKEN: with the ingest token a pod in that namespace can send
spans and cannot read anyone's metrics, logs or traces.
env:
- name: POD_NAME
valueFrom: { fieldRef: { fieldPath: metadata.name } }
- name: POD_NAMESPACE
valueFrom: { fieldRef: { fieldPath: metadata.namespace } }
- name: NODE_NAME
valueFrom: { fieldRef: { fieldPath: spec.nodeName } }
- name: OTLP_BEARER
valueFrom: { secretKeyRef: { name: orbit-stack-otlp, key: token } }
- name: OTEL_SERVICE_NAME
value: checkout
- name: OTEL_RESOURCE_ATTRIBUTES
value: k8s.namespace.name=$(POD_NAMESPACE),k8s.pod.name=$(POD_NAME),k8s.node.name=$(NODE_NAME),k8s.deployment.name=checkout,service.version=5.0.2,deployment.environment.name=production
- name: OTEL_EXPORTER_OTLP_ENDPOINT
value: http://sreorbit-orbit-stack.sreorbit.svc.cluster.local:8082
- name: OTEL_EXPORTER_OTLP_PROTOCOL
value: http/protobuf
- name: OTEL_EXPORTER_OTLP_COMPRESSION
value: gzip
- name: OTEL_EXPORTER_OTLP_HEADERS
value: Authorization=Bearer%20$(OTLP_BEARER)Kubernetes expands $(NAME) from the variables above it. %20 is the space, percent-encoded
as the header variable's format requires. OTEL_EXPORTER_OTLP_ENDPOINT is the base URL, and
the SDK adds /v1/traces; the per-signal OTEL_EXPORTER_OTLP_TRACES_ENDPOINT takes the
full URL instead.
The engine stores the k8s.* attributes an SDK sends as they come: it does not check them
against the sending pod, as orbit-collector does. A pod holding the ingest token can name
another pod, namespace or workload in its spans. When the namespace on a span must be the one
it came from, send through the chart's collector (apm.collector, engine orbit) and leave
the applications' namespaces out of the engine's allowFromNamespaces.
Let the spans through the NetworkPolicy
With networkPolicy.enabled (the chart's default) the engine admits orbit-api, the
chart's own agent and the demo shop on 8082, and nothing else. On a CNI that enforces
policy, a Collector or an application in another namespace needs that namespace named:
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \ --set 'components.orbit-stack.networkPolicy.allowFromNamespaces={observability}'observability is the namespace your Collector runs in; list several separated by commas.
The plain manifests need the same rule added to sreorbit-orbit-stack in
80-networkpolicies.yaml.
Resource attributes that tie a service to Kubernetes
The engine names a service by service.name (a span without it is unknown_service, the
SDKs' own default) and remembers, per service, the Kubernetes attributes of the resources
that sent it spans:
| Resource attribute | Where it shows |
|---|---|
k8s.namespace.name, k8s.deployment.name | The service's "Runs on" panel and the Workload column, linked to the Deck's workload page |
k8s.pod.name | The pods of the service, and the span drawer's Infrastructure tab: the pod's CPU, memory and restarts, and its logs within a minute of the span |
k8s.node.name | The node in the span drawer |
service.version, deployment.environment.name (or deployment.environment) | traces_target_info, below |
Without them a service still gets its rates, latencies, map and traces, and the Deck says
the spans carry no Kubernetes resource attributes. Peers that never send spans (a
database, a queue, a third-party API) are named from the client span's attributes:
peer.service (or service.peer.name), server.address (or net.peer.name),
network.peer.address (or net.sock.peer.addr), db.system (or db.system.name),
db.namespace (or db.name), messaging.system and messaging.destination.name (or
messaging.destination).
Through the chart's collector, an app that sends its own spans gets its own pod's k8s.*
attributes, whatever it sent, and keeps its service.name when its pod's spec declares that
name. Only pods of the release's node namespace and of apm.collector.orbit.trustedNamespaces
may name other pods. The rules and how to declare a name are in
Security.
Versions without service.version
A service whose spans carry no service.version still has versions: orbit-api takes them from
its Deployment's ReplicaSets, named in each pod's name (<deployment>-<pod-template hash>-<suffix>).
A ReplicaSet's version is the image tag of the app's container, followed by its pod-template hash
(1.31.6 · 6f7b5956c6) when another kept ReplicaSet of the same Deployment has the same tag (a
kubectl rollout restart, say), so two ReplicaSets never share one. The app's container is the
one named like the Deployment, else the one two signals agree on: its image changed between the
Deployment's ReplicaSets, and it declares a port. A signal no container shows has no say. When
they disagree or leave several (a rollout of a sidecar alone, sidecars and no change yet), or the
image has no tag or is tagged latest, the version is the hash alone. The service page shows, as
Version followed by "(inferred)", the version of the ReplicaSet behind the pods that sent the
window's newest spans, or the current ReplicaSet's when they name no pod: a stuck rollout shows
what still serves, and a past window what ran then. Image shows the app's container. The
faulty release card heads its version columns "Version (inferred)", every sentence of the finding
names the version with "(inferred)", and the faulty release detector judges these versions as it
judges an SDK's. A service with service.version anywhere in the last 90 minutes keeps only
those. Pods of a StatefulSet, DaemonSet or CronJob, and services outside Kubernetes, have no
version to infer: for a scope with only such services, GET /api/v1/nocturne/insights lists
errors_after_rollout under noVersion, never under checked. A release the detector could not
judge (too few requests, no version before it) leaves error and latency spikes to the spike
detectors, and a latency spike names a rollout within 5 minutes of its step. Once a service has a
faulty release finding, a spike from the hour's first version change on yields to it.
The rollout markers on a service's charts and its operations' charts, and Last rollout, come from the same ReplicaSets' creation times, so they outlive the API server's events (kept 1 hour by default). While the Deployment is back on an older ReplicaSet, a newer one whose pods never sent a span (an image that never pulled, then undone) gets no marker; the next rollout forward brings it back, since only a rollback reads the spans. A rollback to a ReplicaSet older than another is marked "Rollback" when the last newer one that ran sent its last span, a time a replaced pod does not move; with no span to read, when the Deployment's Progressing condition last named it. Only the last such return is known. Last rollout shows the UTC time today and how long ago on another day.
What the engine derives
Every accepted span also feeds a generator that writes ordinary series into the engine's own store every 15 seconds, under the names and labels of Tempo's metrics-generator, so a dashboard or an alert written for Tempo's output reads this engine's:
| Series | Labels |
|---|---|
traces_spanmetrics_calls_total | service, span_name, span_kind, status_code |
traces_spanmetrics_latency_bucket, _sum, _count | The same, plus le, in seconds |
traces_spanmetrics_http_4xx_total | The same as the calls, for server spans only: those whose http.response.status_code (or http.status_code) is 4xx, which the semantic conventions leave unset; this engine's own, from 2026-11-06 |
traces_service_graph_request_total | client, server, connection_type |
traces_service_graph_request_failed_total | The same |
traces_service_graph_request_server_seconds_bucket, _sum, _count | The same, plus le |
traces_service_graph_request_client_total, traces_service_graph_request_nested_total, traces_service_graph_request_nested_seconds_total | client, server; this engine's own, Tempo writes no such series |
traces_target_info | job (the service), k8s_namespace_name, k8s_deployment_name, k8s_pod_name, k8s_node_name, service_version, deployment_environment; 1 while a pod sends spans |
- Label values are Tempo's:
span_kind="SPAN_KIND_SERVER",status_code="STATUS_CODE_ERROR". - Latency buckets: 1, 1.5, 2, 3, 5 and 7.5 in each decade from 1 ms to 30 s, 28 bounds, six
rows a decade for the latency heatmap. They include every bound written before 2026-10-28
(1, 2, 5, 10, 20, 50, 100, 150, 200, 300, 500 and 750 ms, then 1, 2, 5, 10, 20 and 30 s)
under the same
le, so those series carry on unchanged through an upgrade or a rollback. A label set writes 31 latency series instead of 21. - Counters are cumulative since the engine started, so a restart is a counter reset that
rate()handles; a series gets a zero one interval before its first value, so its first calls count inincrease(). - An edge is a client span and the server span it parented, matched by span id alone, so
the two pair even when they arrived under different traces. A client span with
db.system,db.namespaceordb.nameis an edge to that database at once (connection_type="database"). Producer and consumer spans withmessaging.systemare two edges through the queue, producer to destination to consumer (connection_type="messaging_system"; Tempo pairs the two services directly). A server span with no parent is an edge fromuser, and a client span nobody answered within 10 s is an edge to itspeer.serviceorserver.address(bothconnection_type="virtual_node"). Without a server span, the edge's latency is the client's. traces_service_graph_request_client_totalcounts the calls a service's client spans made on an edge, and thenestedseries those of them inside a request, with the edge's seconds: the client span has a parent, or the engine stitched it under one within 2.5 minutes. A client span with no parent is background work. A call under a parentless internal span, such as an SDK worker's job, has a parent and counts asnested, as does orbit-probe's copy of a call an SDK traced with no parent. An edge's background calls are the first less the second, so none that an engine before these series counted reads as background. They stop being written when the rollups below let go of the edge, about 32 minutes after its last span.- The engine names each operation once at intake. A server span with
http.routeandhttp.request.method(orhttp.method) takes the name{method} {route}. In any name, a word or path segment that is a number, an email, a UUID or hex digest, or a token of 16 or more characters with 3 or more digits becomes{id}, and a URL's query string goes:GET /users/12345?tab=2isGET /users/{id}. When the name changes, the span keeps the one it arrived with inorbit.raw_name, so you can still search for it. Ingestion rules see the new name undername; matchattr.orbit.raw_namefor the old one. - A service holds at most 1000 operations and, as a client, 1000 edges. A new one past that,
or past the generator's 20000 label sets, counts as
span_name="(other)"orserver="(other)", andorbit_stack_trace_generator_overflow_totalcounts its spans. The ones it already holds keep counting. - The generator holds at most 20000 label sets, plus 2000 for
(other); past that it drops new ones and counts them inorbit_stack_trace_generator_dropped_total. It holds at mostORBIT_STACK_TRACE_GRAPH_PENDINGspans (default 100000) waiting for the other side of an edge; past that a span goes unpaired, the edge undercounts, andtraces_service_graph_dropped_spans_totalcounts it. A series that saw no span for 15 minutes stops being written. - The services summary and the service map also read rollups the generator keeps per service
and per client and server pair, outside the label sets above, for its last 64 collections
(16 minutes at 15 s). A window that starts inside them lists every service that sent spans,
with the spans the collections inside the window counted, without extrapolating to the
window's edges as
increase()does. They hold at most 100000 services and 20000 edges, plus oneserver="(other)"edge per client past that. A longer window, or one that starts before the engine did while older series exist, reads the series.
Stitched traces
orbit-probe gives each TCP hop its own trace. A client span the probe records while the same
process serves a request takes that request's trace and names the hop's own trace in
orbit.hop.trace_id. The engine then keeps the hop's spans under the request's trace:
- Hop spans that arrive after the client span move to the request's trace at intake. Those already in the live head merge into it there. A chain of hops ends at the outermost request, at most 16 deep, so one trace runs from the HTTP request through its Redis and SQL calls to the next service and its own database calls.
- An alias lives 15 minutes, and the engine holds at most 262144 of them.
orbit_stack_trace_aliasescounts the live ones andorbit_stack_trace_alias_events_total{event}counts what happened to them:recorded,refused(a trace named as its own hop, a loop, or a hop already stitched into another trace),expired,evicted,rewrittenspans andmergedhead parts. - A hop part the head wrote to a segment before its client span arrived stays under its own
id.
GET /api/v1/traces/{id}followsorbit.hop.trace_idto it and returns its spans under the request's trace id, at most 16 hops deep andORBIT_STACK_TRACE_MAX_BYTESof followed spans. - The write-ahead log holds the spans as stitched, and a restart replays the aliases with them.
Calls with no parent
When orbit-probe loses the context of a call, a Redis EXPIRE or a SELECT reaches the engine
as a client span with no parent and its own trace. This happens when the probe misses the
hand-off from nginx to a php-fpm worker. The engine puts such a call inside the request that made
it when only one request can be the caller:
- Same process. The call's process (same pod and
process.pid) served requests, and when the call started exactly one trace was in flight on itsthread.id, or in the whole process. A request here is a server or consumer span, or a span with a parent that hands work to a process, such as orbit-probe's FastCGI span. Requests chained by parent (nginx, its FastCGI span, the php-fpm hand-off) count as one; the call goes under the innermost. - Same pod. The call's process shows no request around it (it never served one, or the probe missed this hand-off), another process of the same executable in the pod served requests, and exactly one chain was in flight in the pod. The engine decides once a request of the pod ends 2 s past the call's start plus the pod's longest request and the longest a request of the pod arrived late (each at most 60 s), so every request around the call has arrived.
- Ambiguity stops it. A process seen with two traces in flight at once on one thread stops matching by thread; seen with two at once at all, it stops matching by process. A pod needs 8 requests before its calls are stitched. The engine follows a process from its first parentless call, and that process needs 8 requests after it.
A stitched call takes the request's span as its parent and orbit.link = stitched, and its
trace moves into the request's trace, so the root-traces list stops showing it on its own. A call
that named its downstream hop (orbit.hop.trace_id) also leaves its trace as an alias of the
request's trace, so the hop's later spans land there too. The trace page draws its elbow dashed, like the probe's own inferred links. Calls from
queue workers, crons and heartbeats have no request around them and stay their own traces.
eBPF and code together. When a pod also sends spans from an SDK, an eBPF call waits up to
10 s of the pod's SDK span time for the SDK client span of the same call: one that wraps it
(exactly, else within 1 ms) and whose name holds the call's first word (GET in
redis redis-global-5 get). The engine drops the eBPF copy and keeps the SDK span in its trace
only when exactly one such span wraps the call and that span wraps no other eBPF call of the pod:
an SDK that samples leaves requests out, and their calls can fall inside the spans of a request it
kept. An SDK span that may have wrapped a call the engine dropped for room, or one sent before the
pod's first SDK span, merges nothing. Every call that does not merge goes on to the rules above.
SDK requests stand in for the pod's requests only once the SDK has wrapped 64 eBPF calls of the
words it uses and missed at most 1% of them; until then, and in a pod whose SDK samples, the
rules use orbit-probe's requests alone. Matching uses the pod, not the pid, since an SDK in a container sees its own
pid namespace. An eBPF span that already has a parent, such as one the probe linked into the SDK's
trace through traceparent, is not merged yet.
The engine matches only in its live head: both traces must still be there. It follows at most
4096 processes with parentless calls and 4096 pods for 10 minutes after their last span, 64
requests each, and 65536 waiting calls; past that, the pod holding the oldest waiting call drops
all of its own. A call waiting for its SDK twin takes its request while the pod still lists it.
orbit_stack_trace_alias_events_total{event} counts stitched calls and twins merged into
their SDK span.
Root traces
A root trace has a span with no parent, or a span whose parent is not in the trace: an
orbit-probe server span names its caller's client span even when the probe never saw it. The
root span is the earliest parentless span, else the earliest of those orphans. Almost every
trace is a root trace; its root span's kind tells a request (server, consumer) from
background work (client, producer, internal), such as the Redis heartbeat a queue
worker sends with no request around it.
GET /api/v1/traces?root=true keeps root traces and rootKind=server,consumer keeps those
whose root span has one of the listed kinds; either one adds rootKind to every hit. The
engine records the kind in each trace's summary row, in each chunk's zone map and in the
segment's aggregates, so this search reads no spans. Segments written before this release
carry no root kind: there root=true falls back to a parentless span, and rootKind
matches nothing.
Up to 10 key (attribute names) on GET /api/v1/traces add custom: {key: {values: [{value, count}], capped}}: each key's 10 most frequent values over every trace the search matches in the
window, a value counted once per trace that carries it (a span's own attribute first, then its
resource's). The Deck's custom facets read it under any filter beyond a status. That search reads
the spans of every match, so it runs until the request's timeout, and partial says it stopped
there. capped says the search met more than 4,096 distinct values of that key: its counts are
then lower bounds.
A request rate and error ratio per service, for a panel or an alert:
sum by (service) (rate(traces_spanmetrics_calls_total{span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))
sum by (service) (rate(traces_spanmetrics_calls_total{span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER", status_code="STATUS_CODE_ERROR"}[5m]))
/ sum by (service) (rate(traces_spanmetrics_calls_total{span_kind=~"SPAN_KIND_SERVER|SPAN_KIND_CONSUMER"}[5m]))The built-in dashboard orbit-apm-red draws the same series (dashboards.md).
Reading traces in the Deck
All under /deck/apm, with the apm.read permission, over the window picked in the top
bar:
Services (
/deck/apm): every service with its request rate, error rate, p50, p95 and p99, operation count and workload. A service is degraded at 1% errors and failing at 5%; one with no requests in the window shows as having no traffic. A service's requests are its server and consumer spans, or all its spans when it has neither.A service (
/deck/apm/services/<name>): requests by outcome, error rate, latency and its heatmap over time (of the whole service or of the operation picked above it), the operations table, what calls it and what it calls, recent traces and recent errors, and the Kubernetes objects it runs on. Workers and crons that report the sameservice.namecount under it, so the page keeps them apart from its requests. Time by dependency is each callee's share of the time inside the service's requests, from thenestedseries. Calls leads with the calls made inside a request and says how many ran in the background; its errors and p95 cover every call. Recent traces, recent errors and error types read the traces a request started; when the window has none, they read every root, and the lists say so. Pods sending spans counts every pod whose spans carry the name, which can be more than the replicas of the workload named above it. A background service reads all its calls instead. In a window that reaches back before the engine wrote these series, that chart and table read every call made before then, and none of them as background. Time spent in the operations table is each operation's share of the time the listed operations took (their span durations added up), and the table sorts by it. Slower than p95, here and on an entry operation's page, opens the traces whose entry span took longer than the window's p95.An operation (
/deck/apm/services/<name>/resource?op=<operation>, a row of the operations table): its requests by outcome, error rate, p50, p95 and p99 (with the p95 of a day earlier and the service's rollouts on the charts), its latency heatmap, its error types, recent traces and errors, and a span summary. The summary reads 20 traces of the operation spread over the window and lists where its requests came from (the nearest request above each, or none when it starts its trace) and what they ran, by service and operation: spans per request, the share of requests that ran it, average duration, and its own time inside the request (children taken out) as a share of that request's own time, averaged over the requests, so a stream one request left open counts as that one request. A row opens that operation's own page, so you can walk up or down the calls. It is a sample: a rare branch can be missing from it.Service map (
/deck/apm/map): services, databases, queues and external calls as nodes, calls as edges with rate, error rate and p95; an edge turns red at 5% errors. When spans went unpaired in the window, the map's answer sayspartial: truewithunpairedSpans, because its edges undercount. Past 300 nodes the Deck draws the 300 busiest and says how many it left out; Service map on a service's page draws that service with its neighbours first. An edge's width is all its calls; its label and a node's panel lead with those made inside a request and name the rest as background.Root traces (
/deck/apm/traces, the default list): the traces a request started, whose root span is a server or consumer span, searched with the query bar below. Under the bar, traces per step as stacked columns, passing in grey and failed in red, with the failed share as a line on its own right axis; a click on a column narrows the window to it, and a click on a line also filters on the value it draws. The controls read as a sentence, Show [measure] by [key] top [n] every [step], for the error rate or a duration percentile, overall or by service, root operation, pod, node, version, route or error type; by a key, as lines over time or as a top list of the whole window, ranked by the measure. The duration histogram, the scatter of duration over time, the latency heatmap (below) and the comparison with passing traces open from Charts on the line above the results, which also holds the findings; the browser remembers whether you left them open. A trace whose root is a client, producer or internal span had no request behind it: a queue worker polling Redis, a cron's query, a heartbeat. These background calls are hidden by default, because on a busy cluster they outnumber the requests many times over. A link above the list says how many it hid and shows them in one click (?background=1). Each row whose root is not a server span names its kind. An engine older than Addendum v0.27 does not report root kinds: the page then lists every trace, with no kind and no such link, and says so.Saved views beside the Search button on both lists: the team's views of this page, as on Logs (
logs.md→ "Saved views").All spans (
/deck/apm/traces?scope=spans): stored spans one by one, newest first, 100 to a page, with the same filters plus span kind and minus maximum duration. Older spans moves the page to just before the oldest span shown. Each row opens its trace on that span.Query bar: type filters as
key:valueseparated by spaces; all of them must hold. Each one turns into a chip under the bar: select it to edit it, its minus to exclude it, its x to remove it.service:,operation:,status:errororok,duration:>100msandduration:<2s(units ns, us, ms, s, m, h; two chips make a range) and, on All spans,kind:are named keys; any other key is a span or resource attribute, as inhttp.route:/api/cartoruser.tier:"gold plus". A minus excludes an attribute (-http.response.status_code:200) or a service (-service:frontend): traces where any span carries it drop out. Up to 8 attributes to meet and 8 to exclude. Datadog's@duration:>100msworks asduration:,env:prodreads the resource'sdeployment.environment.nameandresource_name:the span name;operation_name:has no OpenTelemetry field and is refused. Wildcards, comparisons on attributes (http.response.status_code:>=500) and free text are refused with the reason: the engine indexes exact key and value pairs. In quotes a value is taken as written,*included (http.accept:"*/*"). The list counts its matches for 0.4 s at most; past that it shows the newest ones exactly and says "more than N". Switching between Root traces and All spans keeps the terms the other view takes and drops the rest (kind:, a maximum duration). Tab completes keys and values: services and operations from the window, attributes from its newest 200 spans. The query lives in the URL (?q=), so back, forward and a pasted link replay it; links that still carry?service=,?attr=k=vand the other old parameters open as chips. Each facet value adds its chip in one click, and its minus adds the exclusion.A trace (
/deck/apm/traces/<id>): three views of one trace, each drawing only what is on screen. Waterfall shows the spans as a tree, with the critical path and a minimap to zoom, which marks every failed span in red. In the Flame graph, spans that ran at the same time stack, so none hides behind another; scroll zooms around the pointer, Shift+scroll or a sideways swipe moves in time, and a drag pans both ways. Operations is the trace by service and operation, with calls, average, slowest, self time and time on the critical path. A row with 10 or more calls under one parent says so, which is how an N+1 looks, and a click on a row finds its spans in the waterfall. The view lives in the URL (?view=flame).Compare with in Operations takes another trace's id, or A typical trace, the median clean trace of the same root operation in a uniform sample of up to 200 from the hour around this one. The critical path crosses a trace's roots, orphans included, from the last to finish back; with the time no root covers, shown as its own row, it adds up to the trace's duration, so the differences per operation say where a slow trace spent what the other did not. Operations pair by exact service and name, so a name with an id in it reads as one gone and one new. The comparison lives in the URL (
?compare=<id>).Find takes a word, or the explorer's terms:
service:cart,operation:"GET /cart",status:error,kind:client,duration:>100ms, any attribute askey:value, and a minus to exclude on any of them. It reads the spans themselves, so it also takes what the explorer's engine cannot:http.status_code:>=500,OR,NOT, parentheses and Datadog's group form,service:(cart OR payment). A term it cannot answer (status:foo, a wildcard, a bracket range) says so. Self time colours the bars by the time each span spent on its own. Root errors walks the failed spans under which nothing else failed. Fold to the hops between services, beside expand and collapse, leaves one row per hop in both graphs; a folded bar says how many spans it holds. Keys, with either graph focused: arrows walk (in the flame graph, up the parent, down the first child, left and right the siblings, so spans that ran at once are one key apart), Enter opens, Z zooms to the span, 0 shows the whole trace, E goes to the next root error, / finds, and in the flame graph + and - zoom.The drawer for a span shows its attributes grouped by namespace, its events, resource and infrastructure. Each attribute copies its value, finds the spans that carry it in this trace, and searches the explorer with or without it. Walking spans with the drawer open reads a span's logs and infrastructure once the cursor rests for 250 ms. A dashed elbow before a span means orbit-probe inferred its parent (
orbit.linkissingleorroute); hover the link icon next to it to read how. The drawer shows how the span was linked and the process that ran it: executable,process.pidandthread.id.Who called this? When a trace's root span is a client, producer or internal span, the trace page asks
GET /api/v1/spansfor the requests the same process (samek8s.pod.nameandprocess.pid) was serving when that span started, and for the other spans of that process within 2 s either side, each linked to its trace. When the process was serving no request, the page says so and names the executable: the call is background work, such as a scheduled job, a timer or a queue worker. A request here is a server span, or a FastCGI span, which orbit-probe marks internal when nginx and php-fpm share a pod. The panel looks back 60 s for a request still in flight. A trace that starts at a request makes none of these reads.Related: a row of links on a span's drawer, a service, a workload, a pod, a pod's terminal (each in a new tab, so the shell stays open), a node and an alert's drawer, to the other signals of the same object and window: its logs (a span's by
trace_id, a pod's bynamespaceandpod, a service's byservice, a workload's bynamespaceand its pods' names or its service's name, whichever finds more (a file's lines name no pod; a service's page tries its workload's pods the same way), a node's bynode; an alert on a node opens the node's) with their count and errors, its traces with request and error counts (a pod's link filters onk8s.pod.nameand has no count, which would be its whole service's), its service (for a pod or an alert, the service whose spans name that pod or workload ink8s.pod.nameork8s.deployment.nameontraces_target_info; the row says so, with no traces link, when no span does; a worker or cron with no server or consumer span keeps the link, with no count), its workload, pod and node pages, its Kubernetes events with warnings, and the alerts raised on it. A span's window is 15 minutes either side of it; an alert's is 15 minutes either side of when it fired; every other page keeps the top bar's. The links show at once; one server call fills the counts. A span's row adds that span's lines with their count, when it wrote any, and its pod's or service's lines in the same window, and says so when no line carries the trace id. A role withoutlogs.readorapm.readgets no link to that signal. Whentraces_target_infocannot be read, or the role lackstelemetry.query, a pod gets no service link and keeps its traces link; a workload gets no traces link.
Releases of a Deployment
The Releases tab of a Deployment's page (?tab=releases) answers "did this revision make
anything worse, and where?" with no version label. It needs apm.read, a live cluster and the
apm switch; the rollback and workload_changes switches leave it alone, so a read-only
install keeps it.
- A row per kept revision, newest first, from the Deployment's ReplicaSets.
- One timeline of rollouts. Every revision number a kept ReplicaSet held is a rollout: its own
and those in
deployment.kubernetes.io/revision-history, which the controller writes when it reuses a ReplicaSet (a rollback, or a template applied again). A ReplicaSet's first rollout is its creation. A later one is when its pods came back, to the pod's own second, or, when they never left because the revision before got stuck (a crash loop the rolling update would not let replace them), when the pods of that revision were all gone, to the last scrape (kube_pod_createdand the last time each pod was read). Such a row is tagged re-applied. When the engine no longer holds those pods, the row has no numbers. - The windows. Before is the run of the revision this one replaced, after is this revision's
own run, 30 minutes each at most, and after never reaches past now. A rollout merged into a later
revision number still ends the run before it, so an undo applied later does not change an older
row. The oldest kept revision's run before starts with the first pod of the ReplicaSet the controller
pruned (
kube_pod_created). The span numbers skip each side's first minute, while the revision before still answers, so an undo read minutes later does not inherit the slow pods' tail. Reads reach back 15 days. - The numbers, for each service whose spans name the workload (
traces_target_info, byk8s.deployment.nameor its pods' names): requests and error share of the entry spans (server and consumer, as on the service page), the 4xx share of the server spans (traces_spanmetrics_http_4xx_total; a dash where the engine was not counting them at the window's start), p95, restarts of the Deployment's pods (kube-state-metrics) and new error lines: messages with 5 lines or more after the rollout and none in the 30 minutes before it, whatever ran in those 30. Error lines are read only for a role withlogs.read, with stored logs on. - Routes, under the row: entry operations that appeared (5 calls or more, none before), went (5 or more due in the window after and under a tenth of them came, since the old pods answer while the rollout runs) or doubled their 4xx share past the bar below.
- The verdict takes
errors_after_rollout's bars. A share got worse with 100 requests or more on both sides, 5 failures or more after, at 1% or more, twice the share before and 3 standard deviations over it; latency got worse at twice the p95 before and 100 ms more, unless the run it replaced was failing (5% or more of errors and 4xx, which fail fast) and this one left half that or less; restarts, when the window after has more. A route's 4xx counts only with 100 calls after, a share at least 1 point higher and the service's own 4xx share not falling; otherwise it stays in the list as evidence. Worse names what got worse. No worse needs 100 requests on both sides of a service; with fewer the row says so, with no request on either side it says No entry spans, and with no window to read No data. - Copy puts the row on the clipboard as plain text for a ticket: the revision, when it rolled out, its image, the verdict, every number and route, and both windows.
Nothing in the tab opens by itself, and it raises no finding.
The latency heatmap
The Heatmap view of Root traces (?chart=heatmap) and the latency panel of a
service's page draw span durations over the window: a column per time step, a row per latency
bucket on a log axis, each cell coloured by how many spans landed in it, on a log scale whose
legend prints the range. It counts every span, whatever the ingestion rules stored, because it
reads the span metrics (ingestion.md):
- With a service in the query, that service's server and consumer spans, narrowed by
operation:andstatus:. On a service's page, the whole service or the operation picked above the chart (?op=); a background service's client and producer calls instead. - With no service, the requests entering the system: the service graph's calls from
user. - A duration in the query is outlined on the heatmap, not filtered out of it, so the rest of the distribution stays in view.
- An attribute (or
operation:orstatus:with no service) is something the span metrics do not label. The heatmap then counts the stored traces the list would match, in one pass over the window, and says so under the chart; a pass that runs out of time says its counts are a lower bound.
Hover over a cell to read its time, its duration range and its count. Click a cell, or drag a
rectangle across cells, and the list narrows to those minutes and that duration range: the
time range in the top bar (a single cell's is widened to two minutes around it), and
duration:> and duration:< in the query. On the keyboard the plot is one tab stop: the
arrows move between cells, Shift with the arrows selects a rectangle, Enter opens it and Escape
drops it; a screen reader hears each cell. The list holds stored traces only: when the
ingestion rules dropped spans in the engine's last 5 minutes, hour or day (the one closest to
the window), the line under the chart gives the share they kept.
A window gets about 120 columns: 30 s steps for an hour, 15 minutes for a day, 2 hours for a
week, never finer than the 15 s at which the span metrics are written. The route is
GET /api/v1/apm/heatmap (api.md).
Stack traces link to the code
Wherever a stack trace shows, in a span's exception event (exception.stacktrace) or in a
log line (the message, its exception.stacktrace, error.stack or stack_trace attribute,
or any string field of a JSON message, Monolog's trace array included), the Deck turns each
frame into a link to its file and line in the repository, like Datadog's Source Code
Integration. It reads PHP (getTraceAsString, PHP's own in /path.php:12 messages,
Monolog's path:line entries and the OpenTelemetry PHP SDK's
at App.Services.PaymentService.charge(PaymentService.php:87)), Java, Kotlin and Scala, Python,
Node.js, Go, .NET and Ruby.
- Which commit. The span's or the log's own
vcs.repository.url.fullandvcs.ref.head.revision(OpenTelemetry), or Datadog'sgit.repository_urlandgit.commit.sha, on the resource or the record. For a service that sends neither, an admin sets the repository and branch under Integrations, Source code. - Which file. The frame's path minus the directory the image copied the repository into:
the one set for the service (Path in the container), else a common one (
/var/www/html,/app,/usr/src/app,/src,/build,/go/src/app; the full list isROOTSinapps/web/src/lib/stack.ts), a Go module path or GOPATH with the repository's own name, or a GitHub Actions or Azure Pipelines checkout. A JVM frame names a class, so its file is the package path undersrc/main/java(src/main/kotlin,src/main/scala). The OpenTelemetry PHP SDK writes the class and the file's base name only, so the Deck builds the path from Laravel's namespaces:App\underapp/(in a trace with Laravel's frames; Symfony and Slim keepApp\insrc/),Database\Seeders\andDatabase\Factories\underdatabase/,Tests\undertests/. Any other namespace (Modules\, a package missing from the vendor list) gets a link only from the repository's file list, with a GitHub token.{main}sits atpublic/index.phporartisan; a closure outside a class or in Laravel'sRouteFileRegistraratroutes/web.phporroutes/api.php; a closure ofroutes/console.php(Artisan::command,Schedule::call) carries Laravel's consoleKernelorClosureCommandand links there. A trait's method names the class that uses it and the trait's file: Laravel's own traits (AuthorizesRequests,Notifiableand others; the list isPHP_VENDOR_TRAIT) count as vendor code, and any other gets a link only from the repository's file list, with a GitHub token. Directory in the repository goes in front of the result, for a monorepo or a Maven module. - Vendor code. Frames in
vendor/,node_modules/,site-packages/, gems, the Go module cache and standard library, the JDK and common JVM frameworks,System.orMicrosoft.in .NET, PHP namespaces of common packages (Illuminate,Symfony,GuzzleHttp,Doctrine,Monologand others; the list isPHP_VENDOR_NSinapps/web/src/lib/stack.ts), PHP's own functions and classes, compiled Blade views andeval()'d code stay as dimmed text with no link. - The code inline. With a GitHub token saved under Integrations, each linked frame gets a
button that shows the 11 lines around it, which orbit-api reads from
api.github.comat that commit. orbit-api seals the token withORBIT_SEAL_KEY, never answers it, and sends it toapi.github.comonly, so a frame on another host gets a link and no code. orbit-api reads only a repository linked to a service under Integrations, Source code, at the ref that service sets (any ref when it sets none) or at a full commit id, the one telemetry carries for the build. A repository that telemetry names and no service links gets links and no code: link it to see its code. Use a fine-grained token with read-only access to the Contents of the repositories you link: any role withapm.readorlogs.readreads their code through it. - The repository's file list. With the token, orbit-api also reads the repository's file list at that commit, and each frame links to the file that shares the longest tail of path segments with it: a Maven module, a monorepo directory or an unusual WORKDIR needs no setting, and a frame whose file is not in the repository (a third-party JVM package, say) gets no link. When two files tie, the one under the service's Directory in the repository wins; with no directory set, or none of them under it, the frame gets no link.
GitHub, GitLab, Bitbucket and Azure DevOps (https) URLs link; only GitHub shows code inline. The parser reads at most
256 KiB of text and skips lines over 4 KiB, so hostile text costs linear time. A TypeScript or
bundled JavaScript frame links to the compiled file (dist/...): the Deck does not read source
maps.
The routes: GET /api/v1/source (apm.read or logs.read) answers the services and whether
a token is set; PUT /api/v1/source (settings.manage, audited) saves them; GET /api/v1/source/snippet?repo=&ref=&path=&line= (apm.read or logs.read) answers
{"path", "start", "lines"}; POST /api/v1/source/resolve (apm.read or logs.read) takes
{"repo", "ref", "dir", "files"} (at most 200 files) and answers {"paths"}, the repository
path of each file or null (also when more than 256 repository files share its name). Both
answer 403, before any call to GitHub, for a repository or ref no service links.
The same reads over the API are the apm.read routes in api.md. A span
time is epoch microseconds (startUs, durationUs); every other time on those routes is
epoch milliseconds.
Service SLOs
A service page's header carries Set SLO for a role with settings.manage, on a service that
took requests in the window (a worker or a load generator has none to keep). An SLO is one
availability target on one service: the share of its entry requests (server and consumer
spans, the ones its pages count) that end without an error over the last 7 days, which the
engine's 15 days of metrics hold. The header then reads "SLO 99.5% over 7 d: 99.93%, 86% of
the error budget left", and the SLO 7d column of Services shows the same share with
the budget under it; a service without an SLO leaves both empty. Both are cut down, never
rounded up: 98.996% of a 99% target reads 98.99%, and a budget just spent reads -0.4%, not 0%.
While the data holds less than the 7 days (a new service, a young engine), the header says how
much: "SLO 99.5% over 7 d, 3 h of data: ...", its start read to the hour. The budget is 100%
minus the target, and what is left is 1 minus the failed share over the budget, below 0 once
the week spent it.
Saving writes two rules into the PromQL alert rules (mcp.md), the
Google SRE workbook's multiwindow burn-rate alerts, where a burn rate of 1 spends the budget
in exactly the window:
| Rule | Fires when the failed share over the budget passes | Severity |
|---|---|---|
slo-<id>-fast | 14.4 over the last hour and over the last 5 minutes | critical |
slo-<id>-slow | 6 over the last 6 hours and over the last 30 minutes | critical |
Both page, as Grafana SLO's fast burn does: over 7 days, the fast rule fires once an hour spent
8.6% of the week's budget, the slow one once 6 hours spent 21%. Each carries slo: <id> and is
named SLO <service>: 14.4x 1h/5m (or 6x 6h/30m). Its alert's value is the burn over the
long window, and it carries the service's namespace, which the APM pages name and the save
reads, so a notification channel kept to that namespace hears it; a channel says the burn in its
own language. On Alerts, its row opens the service and the service's logs. Saving the SLO
again rewrites both rules (and reads the namespace again) and keeps off one that someone
switched off; removing it deletes both.
On Observability, Error budget burn draws each SLO's burn over the last hour, one line
per service, and says so when the cluster has no SLO. Its latency and request rate panels and
the Overview's Requests read the span metrics when no app exports http_requests_total.
The routes: GET /api/v1/alerting/slos (apm.read) answers every SLO with namespace,
window, requests, errors, sli and budget (null with no request in the window),
since (the hour its data starts in the window, Unix ms), and error when the engine of its
cluster did not answer. PUT /api/v1/alerting/slos (settings.manage) takes
{cluster, service, target}, with target from 0.5 to 0.99999, and creates or replaces the
service's SLO. DELETE /api/v1/alerting/slos/{id} removes it with its two rules. A console
holds 50 SLOs, and their rules count toward the 200 alert rules. Each save and removal is one
audit event, slo.create, slo.update or slo.delete, that names the rules.
Storage and retention
| Variable | Default | What it does |
|---|---|---|
ORBIT_STACK_TRACE_RETENTION | 7d | A trace segment is deleted whole once its last span arrived longer ago than this. Days or a Go duration. |
ORBIT_STACK_TRACE_HEAD_BYTES | 64MiB | Spans held in memory before they are written out as a segment. The read cache of decompressed traces gets half this budget and the OTLP request bytes in flight the same budget, so traces add about two and a half times this figure to the engine's memory. At least 1MiB. |
ORBIT_STACK_TRACE_MAX_BYTES | 5MiB | One trace in the head, Tempo's max_bytes_per_trace default: spans past it are refused one by one. From 64 KiB to half of ORBIT_STACK_TRACE_HEAD_BYTES. |
ORBIT_STACK_TRACE_GRAPH_PENDING | 100000 | Spans waiting for the other side of a service-graph edge, 1000 to 10000000; past it they go unpaired and the service map says so. |
Spans go to a write-ahead log first, then to the in-memory head; the head becomes an
immutable segment when it reaches ORBIT_STACK_TRACE_HEAD_BYTES or is 15 minutes old,
whichever comes first. A segment, format version 2, holds whole traces as zstd frames in
start order, summaries in chunks with a zone map each, a sorted id table and a bloom filter,
every part checksummed. A segment the engine cannot read, including a version 1 segment
from the first trace release, is skipped at start by name (the log line
trace segment skipped names the file), counted and left on disk for you to delete; the
write-ahead log replays whatever its version. Everything lives under
ORBIT_STACK_DATA_DIR/traces, on the engine's volume, beside the metric blocks.
Beside each segment the engine keeps agg-<seq>, its aggregates: per service, entry span
and status, how many traces it holds and the few a search keeps of them, so a search with a
total or a spread sample reads a segment its window covers whole without decoding it
(about 5% of the segment bytes on disk, a few KiB of memory per segment). The file is
rebuilt from the segment when it is missing or does not verify, and the Go engine ignores
it, so a rollback keeps working; one left behind by the Go engine's retention is deleted at
the next start.
POST /-/flush writes the span metrics and the head out immediately. Size the volume from
your own traffic: GET /api/v1/status/tsdb reports segmentBytes and segmentSpans
under traces, which give your bytes per span; multiply by your span rate and the
retention.
Metrics to watch, on the engine's metrics listener: orbit_stack_spans_total{outcome}
(accepted, rejected), orbit_stack_trace_head_bytes, orbit_stack_trace_head_spans,
orbit_stack_trace_segments, orbit_stack_trace_segment_bytes,
orbit_stack_trace_wal_bytes, orbit_stack_trace_cache_bytes,
orbit_stack_trace_skipped_segments, orbit_stack_trace_cuts_total,
orbit_stack_trace_cut_failures_total, orbit_stack_trace_wal_truncations_total and
orbit_stack_trace_generator_dropped_total. GET /api/v1/status/tsdb carries the same
figures under traces.
The demo shop
orbit-shop, shipped in the orbit-stack image as /orbit-shop, is a small shop made of
real HTTP services that call each other with W3C traceparent and export OTLP to the
engine, so the APM pages have traces that went through the real pipeline:
web-frontendcallsapi-gateway, which callsauthon every request and thencatalog(which queriespostgres),cart(which talks toredis) orcheckout.checkoutcallspayment, which callsstripe-api(an external peer that sends no spans); once the charge succeeds,checkoutpublishes to the Kafka topicorders, whichfulfillment-workerconsumes before writing topostgres.paymentfails a small share of charges (a gateway timeout or a declined card),cartanswers the odd500on a Redis pool timeout, and for one minute in every ten a third ofcatalog's queries take a slow plan: there is always something to find.- A load generator drives
web-frontendlike shoppers, mostly browsing, some carts, fewer checkouts, at--rpsrequests a second (default 5); each request is one trace.
| Install path | How it runs | Switch |
|---|---|---|
| Compose | The orbit-shop service, orbit-shop --all: every service in one process on 127.0.0.1:18110 to 18117, two made-up pods each | On by default. ORBIT_SHOP_RPS in .env sets the rate; docker compose --project-name sreorbit --env-file .env -f deploy/compose/docker-compose.yml stop orbit-shop stops it. |
| Helm | One Deployment and one Service per service, plus a load generator Deployment, each pod naming itself through the downward API | demo.shop.enabled (default false), demo.shop.rps (default 5), demo.shop.resources |
| Plain manifests | Not shipped | none |
On Helm:
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \ --set demo.shop.enabled=true --set demo.shop.rps=5The chart refuses demo.shop.enabled without components.orbit-stack.enabled. The shop
pods use the engine's bearer from the release Secret, and a NetworkPolicy lets them reach
each other on 8080 and the engine on 8082, nothing else. They are load you did not ask
for, which is why the chart leaves them off.
Zero-touch APM with the Helm chart
Two blocks of the chart's values.yaml, both off by default, send traces with no change to
the apps. apm.collector is an OpenTelemetry Collector at <release>-otel-collector on
4317 and 4318 that forwards to this engine: by default orbit-collector, the Rust
collector (apm.collector.engine: orbit); upstream runs otelcol-k8s behind the same Service,
ports and pipeline. orbit-collector holds at most a fifth of its memory limit (64 MiB at least)
in request bytes at once, as received and decompressed; a request past that gets 503, or gRPC
UNAVAILABLE, and the exporter retries it. A request whose resources, scopes and spans would
take more than four times its size once split apart (an empty one is two bytes) gets 400, or
gRPC INTERNAL, as one that does not parse.
orbit-collector tells who sent a span by the address of the connection. A pod in the
release's node namespace (orbit-probe, OBI, orbit-node) or in
apm.collector.orbit.trustedNamespaces reports for other pods, so the collector finds the pod
from the span's own k8s.pod.uid, k8s.pod.name and k8s.namespace.name, as upstream's
k8sattributes does; OBI on the host network (contextPropagation) counts by its node's
address. Any other sender gets the pod, namespace, workload and node of the pod at its address,
whatever its spans say: the collector drops every k8s.* attribute it sent and writes its
own, and replaces its service.name unless the pod's spec declares that name
(Security). A sender outside the
cluster keeps none of them, and neither does one on the host network, unless a hostNetwork
pod of a trusted namespace runs on its node: apm.autoInstrument.contextPropagation (OBI)
and telemetry.nodeExporter.engine: upstream each put one on every node they cover. From
then on any hostNetwork pod on that node connects from the node's address and keeps the
attributes its payload names. Keep application namespaces at Pod Security baseline or
stricter, which refuses hostNetwork, so only the release's node namespace runs pods on the
host network. Add the namespace of a Collector gateway of
your own to trustedNamespaces when it forwards spans for other pods. Keep the collector's pod
out of a service mesh: a sidecar in front of it connects from its own address, so every sender
would lose its pod. apm.collector.engine: upstream keeps otelcol's own association, which
believes the pod a payload names.
Span attributes named orbit.* and a resource's telemetry.sdk.name of orbit-probe belong to
orbit-probe: the engine stitches traces by them. The collector drops them from any sender it
does not trust, so an app cannot move another service's trace into its own with
orbit.hop.trace_id, set orbit.sql.signature itself, or pass for the probe.
apm.autoInstrument runs an eBPF DaemonSet in the release's node
namespace that turns the calls of every process in the chosen namespaces into spans and sends
them to that collector. It has two engines, and renders exactly one:
apm.autoInstrument.engine | DaemonSet | What it turns into spans |
|---|---|---|
orbit (the default) | <release>-orbit-probe: orbit-probe, eBPF instrumentation in Rust (services/rust/orbit-probe) | HTTP/1.1, HTTP/2 and gRPC, FastCGI, PostgreSQL, MySQL and Redis, in plaintext or inside TLS made by OpenSSL |
obi | <release>-obi: the OpenTelemetry eBPF Instrumentation (OBI) v1.0.0 | The same protocols, plus the others OBI knows |
Both need Linux 5.8 or later with BTF (/sys/kernel/btf/vmlinux) on every node; AL2023 and
Bottlerocket kernels carry it. Both send traces and nothing else. namespaces,
excludeNamespaces, excludeExecutables, privileged, healthPort, logLevel and the
scheduling keys apply to either engine. contextPropagation, extraCapabilities,
goMemLimit, mapsScaleFactor and resources are OBI's; with engine: orbit the chart
refuses the first two. Switching is one value: Helm removes one DaemonSet and creates the
other.
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \ --set apm.autoInstrument.engine=obiThree defaults keep the DaemonSet cheap and the service map readable:
| Value | Default | Effect |
|---|---|---|
apm.autoInstrument.excludeExecutables | shells, su and sudo, process wrappers, busybox, coreutils | Globs on the resolved executable path that neither engine instruments, in any namespace. */busybox covers every applet. No language runtime or server is in the list. |
apm.autoInstrument.goMemLimit | 96MiB | OBI only: GOMEMLIMIT for its Go heap. The eBPF maps are kernel memory outside it. An empty string keeps the Go default. |
apm.collector.dropRedisCommands | BLPOP, BRPOP, BLMPOP, BLMOVE, BRPOPLPUSH, BZPOPMIN, BZPOPMAX, BZMPOP, XREAD, XREADGROUP | Redis spans the collector drops, by command: the blocking pops a queue worker or a cron job loops on. XREAD and XREADGROUP go only with BLOCK, when the span carries its query text. [] keeps every span. |
apm.collector.replicaCount | 1 | Collector pods behind the Service, either engine. A pod keeps nothing between requests, so with two or more the chart spreads them across nodes (the kubernetes.io/hostname rule of topologySpread, in orbit-stack's zone) and adds a PodDisruptionBudget of one: losing a node leaves no gap in traces or logs. values-prod.yaml sets 2. |
Both engines turn peer addresses into names from the Kubernetes informers first, Services and pods, then by reverse DNS. A database or cache in the cluster shows under its Service name, a managed one under its DNS name when its address has a PTR record, and only an address with neither stays a bare IP.
orbit-probe (engine: orbit)
- In the kernel: fentry and fexit programs where the kernel's BTF allows them, kprobes
otherwise, on
tcp_sendmsg,tcp_recvmsg,inet_csk_accept,tcp_connectandtcp_close. On kprobes a receive is lost while more threads are blocked in one than the kernel's kretprobe pool holds (one a CPU without kernel preemption); the pod's log says when it attached that way. The bytes reach the agent through one ring buffer per node,apm.autoInstrument.orbit.ringbufBytes(4 MiB by default). A process outside the instrumented containers' cgroups sends nothing: that filter runs in the kernel. - Spans: one server span per request a process answers and one client span per request it
makes, named the way OBI names them (
GET /api/users/{id},SELECT orders,/pkg.Svc/Method) and carrying the semantic-convention attributes this engine reads. Each span carriesthread.id; each process's resource carriesprocess.pidandprocess.executable.name. A SQL span'sdb.query.textis the statement with every literal as?and its comments dropped. It is empty when the keyword the span is named from sits inside a string, a comment or a name, and it ends at a string holding a backslash or at a block comment with another/*inside, since PostgreSQL and MySQL read what follows those differently. - Connected traces: a call is the child of the inbound request its thread is serving when
the call starts. With a thread per request (php-fpm, Java) the link is exact. On an event
loop (nginx, Node) it is inferred, and
orbit.linksays how:singlewhen one request was in flight,routewhen several were and the call's method and path matched the oldest of them. A call made while no request was being served (a queue worker, a timer) has no parent and starts its own trace. Both ends of a connection derive the same ids from the request's first TCP byte, so a client span and the server span on another node pair up without a header; when they land in different traces,orbit.hop.trace_idon the client span tells the store to merge them. - FastCGI: nginx to php-fpm (
fastcgi_pass), recognised by its bytes on any port, named and attributed like HTTP withnetwork.protocol.namefastcgi. When both ends sit in one pod, both spans areINTERNAL, so the pod's requests are not counted twice. - TLS: a process that speaks TLS through OpenSSL's
libssl(curl, PHP, Python, Ruby, nginx, libpq, MySQL clients) has its bytes read before encryption and after decryption, atSSL_writeandSSL_read, so its calls become the same spans as plaintext ones. The probe hooks a container'slibsslfiles the first time one of its processes sends a TLS record, found in the maps of its processes and in the directories distributions keep the library in, and a hook serves every process on the node that maps that file;orbit_probe_tls_librariescounts the files. A connection that was open before the hook landed is read from its next call on. Both ends pair up without a header when both use OpenSSL and both saw the connection's first request: they count the same plaintext bytes. Not read: TLS made by Go, Java, Node or rustls, and OpenSSL driven throughBIO_f_sslor a memory BIO. Those connections produce no spans unless the process has an SDK. The hooks are uprobe-multi links on Linux 6.6 and later; before it they are perf events, which a host withkernel.perf_event_paranoidat 4 (Ubuntu's default) refuses to the probe's capabilities, and the pod's log says so.apm.autoInstrument.orbit.tls: falseturns it off. - Service names: the
resource.opentelemetry.io/service.nameannotation, thenOTEL_SERVICE_NAMEin the process's environment, theapp.kubernetes.io/namelabel, the owner's name and the pod's. - Privilege: uid 0 with
BPF,PERFMON,SYS_PTRACEandDAC_READ_SEARCHadded and every other capability dropped (SYS_RESOURCEtoo withapm.autoInstrument.lowKernel, for kernels older than Linux 5.11), a read-only root, seccompRuntimeDefault, and/sys/kernel/btfand/sys/fs/cgroupmounted read-only. Its ClusterRole reads pods, Services, EndpointSlices and namespaces and nothing else. Its NetworkPolicy lets in only the chart's agent, on its metrics port, and lets out the collector's4318, DNS and the API server (networkPolicy.apiServerCIDRs, without which it cannot watch pods where a CNI enforces policy).deploy/check_hardening.pyholds the chart to exactly that. - Memory: the chart requests
48Miand20mof CPU, with limits of1CPU and256Mi; the memory limit also holds the ring buffer the kernel charges to the pod. The agent's state is bounded (connections, pods, DNS names): past a bound it drops and counts, it never grows. - Its own health:
/healthzanswers once the programs are attached and the node's pod watch has synced;/metricson the same port (healthPort,9112, namedmetrics) counts events, lost events, spans by protocol, dropped spans and export time (orbit_probe_*). The chart's agent scrapes it asjob="sreorbit",component="orbit-probe", and the Components table of/deck/observabilityshows its lost events over 24 hours (seeoperations.md). - SDK apps: a W3C
traceparenton HTTP/1, HTTP/2, gRPC or FastCGI (HTTP_TRACEPARENT) names the trace and the parent, andorbit.linkistraceparent. A process seen exporting OTLP traces to this release's collector gets no eBPF spans from then on (apm.autoInstrument.orbit.skipSdkProcesses, on by default; counted asorbit_probe_spans_dropped_total{reason="sdk_process"}). A process that exports elsewhere keeps its eBPF spans, and the shared trace id joins both.
OBI (engine: obi)
Memory, measured on k3d with three nodes and a PHP sample: the working set of one OBI pod
was 255 to 264 MiB, of which 200 MiB are eBPF maps the kernel charges to the pod and 51 to
53 MiB the Go heap. With the defaults above the Go heap drops to 25 to 26 MiB and the pod to
228 to 232 MiB. apm.autoInstrument.mapsScaleFactor: -1 halves the maps (104 MiB), which
puts the pod at 130 MiB; it also halves the requests OBI can track in flight at once on a
node, so keep 0 on busy nodes. The chart requests 256Mi with a 768Mi limit. On nodes
whose memory is already fully requested, request CPU only:
apm:
autoInstrument:
engine: obi
resources:
requests: {cpu: 50m, memory: null}
limits: {memory: null}The scheduler then places OBI on every node. With no memory request it ranks first for the kubelet's eviction and the kernel's OOM killer, ahead of any app pod that stays within its own request: under memory pressure you lose spans and your applications keep running.
Ceilings
- OTLP/HTTP, traces only. No gRPC listener, and no OTLP metrics or logs: metrics arrive
by Prometheus remote-write (
orbit-stack.md). - No sampling. The engine keeps every span it accepts; sample in your SDK or Collector.
- Single node. No replication and no object storage for traces, like the metric blocks.
- Not stored: span links and the instrumentation scope.
- Measured against Grafana Tempo: ahead on reading a trace by id, on ingest and on
memory, and on the services summary and the service map at 50,000 traces. Search is ahead
only for values common in recent data; for a rare or absent value at 3.4 M spans it is
slower than Tempo, and the summary and the map read right after a flush are slower too.
benchmark.mdprints the figures with the loss table.
Where to go next
- The storage engine, its other endpoints and its configuration:
orbit-stack.md. - The built-in RED dashboard and how to build your own:
dashboards.md. - Every trace and APM route, with its parameters:
api.md.