APM and distributed tracing
Traces, a service map and span metrics, kept in your own cluster
SRE Orbit APM is self-hosted distributed tracing for teams that run Kubernetes. It takes OpenTelemetry traces over OTLP/HTTP into the engine that already stores your metrics, draws the service map and takes you from a slow span to the pod that served it.
APM/Traces/POST /checkout
Duration517 msStatusOK12 spans · 5 services
Service and operation
checkout-webPOST /checkout
checkout-webHTTP POST checkout-api
checkout-apiPOST /v1/orders
checkout-apicart.load
cart-sessionGET /cart/{id}
checkout-apiorders.create
checkout-apipricing.apply
checkout-apiinventory.reserve
checkout-apiHTTP POST payments-api
payments-apiPOST /v1/charges
payments-apiledger.append
payments-ledgerINSERT ledger_entries
Span attributes · checkout-api
- service.name
- checkout-api
- service.version
- 2.14.3
- k8s.namespace.name
- checkout
- k8s.deployment.name
- checkout-api
- http.request.method
- POST
- http.route
- /v1/orders
Send traces the way you already do
Point any OpenTelemetry SDK's OTLP/HTTP exporter, or a Collector's otlphttp exporter, at the engine with its bearer token. Your applications can keep sending gRPC to a Collector; the Collector forwards over HTTP.
Limits. OTLP/HTTP only, as protobuf or JSON, gzip allowed. There is no gRPC listener.
receivers: otlp: protocols: grpc: { endpoint: 0.0.0.0:4317 } http: { endpoint: 0.0.0.0:4318 }exporters: otlphttp/orbit: endpoint: http://sreorbit-orbit-stack.sreorbit.svc.cluster.local:8082 headers: Authorization: "Bearer ${env:ORBIT_STACK_AUTH_TOKEN}" compression: gzipservice: pipelines: traces: receivers: [otlp] exporters: [otlphttp/orbit]Every service, what it calls and where it runs
Services, databases, queues and external calls become nodes, and each call an edge labelled with its request rate; open an edge for its error rate and p95. Each service names its namespace, Deployment and pods, read from the resource attributes your SDKs already set.
Limits. A service without k8s.* resource attributes still gets its rates, map and traces, and the console says which attributes are missing.
APM/Service map
- checkout-webHealthy8/8 pods
- checkout-apiDegradedp99 517 ms · +33.6%
- cart-sessionHealthy6/6 pods
- payments-apiDegraded9/10 pods
- payments-ledgerHealthy3/3 pods
Find the slow request and read the whole trip
Search by service, operation, status, duration or any span attribute. The waterfall marks the critical path, and each span opens with its attributes, events, resource and the pod it ran on, with that pod's logs around the span.
Limits. The engine keeps every span it accepts, for the retention you set (ORBIT_STACK_TRACE_RETENTION); sample in your SDK or Collector if you want fewer. Span links are not stored.
APM/Traces
Matching traces · newest first
Span metrics your Tempo queries already read
Every span feeds series under the names and labels Grafana Tempo's metrics-generator writes, in the same engine as your metrics. Queries and dashboards written for Tempo's output read them unchanged, and a built-in dashboard draws them per service.
Limits. Label values and latency buckets follow Tempo's: span_kind="SPAN_KIND_SERVER", status_code="STATUS_CODE_ERROR".
# p99 of checkout-api, from its server spanshistogram_quantile(0.99, sum by (le) ( rate(traces_spanmetrics_latency_bucket{ service="checkout-api", span_kind="SPAN_KIND_SERVER"}[5m])))| Series | First | Mean | Max | Last |
|---|---|---|---|---|
| p99 | 387 ms | 464 ms | 595 ms | 517 ms |
One permission decides who reads traces
Every APM view needs apm.read, which the built-in viewer, editor and admin roles hold. A custom role can leave it out, and every refused read becomes an audit event.
Limits. Traces live on the engine's volume in your account, beside the metrics.
| Role | apm.read |
|---|---|
| viewer | Granted |
| editor | Granted |
| admin | Granted |
| finopscustom | Not granted |
deniedaccess.deniedapm.read refused for the finops role
See SRE Orbit on your own clusters
Tell us what you run. A person replies within one business day to book a demo of the console, and because SRE Orbit is early, the engineers who build it install it with you.
- Installed with you, next to the tools you run today.
- A copy of your remote-write, so you compare disk, memory and query latency on your own data.
- Direct work with the engineers who build it.