Skip to content

APM and distributed tracing

Traces, a service map and span metrics, kept in your own cluster

SRE Orbit APM is self-hosted distributed tracing for teams that run Kubernetes. It takes OpenTelemetry traces over OTLP/HTTP into the engine that already stores your metrics, draws the service map and takes you from a slow span to the pod that served it.

A checkout request traced across 5 services in the Orbit Deck trace waterfall.

APM/Traces/POST /checkout

Duration517 msStatusOK12 spans · 5 services

Service and operation

checkout-webPOST /checkout

checkout-webHTTP POST checkout-api

checkout-apiPOST /v1/orders

checkout-apicart.load

cart-sessionGET /cart/{id}

checkout-apiorders.create

checkout-apipricing.apply

checkout-apiinventory.reserve

checkout-apiHTTP POST payments-api

payments-apiPOST /v1/charges

payments-apiledger.append

payments-ledgerINSERT ledger_entries

Span attributes · checkout-api

service.name
checkout-api
service.version
2.14.3
k8s.namespace.name
checkout
k8s.deployment.name
checkout-api
http.request.method
POST
http.route
/v1/orders

Send traces the way you already do

Point any OpenTelemetry SDK's OTLP/HTTP exporter, or a Collector's otlphttp exporter, at the engine with its bearer token. Your applications can keep sending gRPC to a Collector; the Collector forwards over HTTP.

Limits. OTLP/HTTP only, as protobuf or JSON, gzip allowed. There is no gRPC listener.

otel-collector.yaml
receivers:  otlp:    protocols:      grpc: { endpoint: 0.0.0.0:4317 }      http: { endpoint: 0.0.0.0:4318 }exporters:  otlphttp/orbit:    endpoint: http://sreorbit-orbit-stack.sreorbit.svc.cluster.local:8082    headers:      Authorization: "Bearer ${env:ORBIT_STACK_AUTH_TOKEN}"    compression: gzipservice:  pipelines:    traces:      receivers: [otlp]      exporters: [otlphttp/orbit]

Every service, what it calls and where it runs

Services, databases, queues and external calls become nodes, and each call an edge labelled with its request rate; open an edge for its error rate and p95. Each service names its namespace, Deployment and pods, read from the resource attributes your SDKs already set.

Limits. A service without k8s.* resource attributes still gets its rates, map and traces, and the console says which attributes are missing.

Service map of the checkout services in prod-use1: checkout-api's p99 is up since deploy 7f3ac1, the other services are healthy.

APM/Service map

namespace checkout
  • checkout-webHealthy8/8 pods
  • checkout-apiDegradedp99 517 ms · +33.6%
  • cart-sessionHealthy6/6 pods
  • payments-apiDegraded9/10 pods
  • payments-ledgerHealthy3/3 pods

Search by service, operation, status, duration or any span attribute. The waterfall marks the critical path, and each span opens with its attributes, events, resource and the pod it ran on, with that pod's logs around the span.

Limits. The engine keeps every span it accepts, for the retention you set (ORBIT_STACK_TRACE_RETENTION); sample in your SDK or Collector if you want fewer. Span links are not stored.

Trace search

APM/Traces

servicecheckout-apistatusanyduration≥ 500 msattributek8s.namespace.name=checkout

Matching traces · newest first

POST /checkoutcheckout-web · 12 spans · 5 services517 ms

Span metrics your Tempo queries already read

Every span feeds series under the names and labels Grafana Tempo's metrics-generator writes, in the same engine as your metrics. Queries and dashboards written for Tempo's output read them unchanged, and a built-in dashboard draws them per service.

Limits. Label values and latency buckets follow Tempo's: span_kind="SPAN_KIND_SERVER", status_code="STATUS_CODE_ERROR".

PromQL on the engine
# p99 of checkout-api, from its server spanshistogram_quantile(0.99, sum by (le) (  rate(traces_spanmetrics_latency_bucket{    service="checkout-api",    span_kind="SPAN_KIND_SERVER"}[5m])))
p99 of checkout-api, from the span metrics
350 ms400 ms450 ms500 ms550 ms600 ms
deploy 7f3ac1 02:08 · +33.6%
p99 of checkout-api, from the span metrics, Sep 20 16:00 UTC to Sep 21 16:00 UTC
SeriesFirstMeanMaxLast
p99387 ms464 ms595 ms517 ms

One permission decides who reads traces

Every APM view needs apm.read, which the built-in viewer, editor and admin roles hold. A custom role can leave it out, and every refused read becomes an audit event.

Limits. Traces live on the engine's volume in your account, beside the metrics.

Roleapm.read
viewerGranted
editorGranted
adminGranted
finopscustomNot granted

deniedaccess.deniedapm.read refused for the finops role

See SRE Orbit on your own clusters

Tell us what you run. A person replies within one business day to book a demo of the console, and because SRE Orbit is early, the engineers who build it install it with you.

  • Installed with you, next to the tools you run today.
  • A copy of your remote-write, so you compare disk, memory and query latency on your own data.
  • Direct work with the engineers who build it.

Open the live demo

A human answers within one business day. We use your details only to answer you.