Skip to content

Operate

Operations

What to watch once the stack is running, what saturation looks like from the outside, what the logs look like, and what each probe failure means. For install-time verification ("did this come up healthy at all"), see the "Verify" section of your install path instead; this page covers steady-state operation.

What to watch

Both backend services expose Prometheus metrics on their own listener (ORBIT_METRICS_ADDR / NOCTURNE_METRICS_ADDR, default :9090; see configuration.md). CONTRACT.md guarantees three metric shapes exist; it does not pin the exact metric name string, which is conventionally prefixed per service (orbit_... / nocturne_...). Confirm the literal name for your build with curl localhost:9090/metrics (port-forwarded or, on compose, published directly). Build alerts and dashboards on these three:

Metric shapeTypeWhat it tells you
*_provider_requests_total{tier,outcome}counterRequest volume per model tier (reasoning/fast) and outcome. A rising outcome="error" rate against a flat total points at the provider, since load has not changed.
*_stream_duration_secondshistogram, buckets out to 300sHow long a chat SSE stream runs. nocturne's NOCTURNE_TIMEOUT default is 120s: if p99 here is climbing toward that ceiling, callers are about to start timing out.
in-flight gaugegaugeConcurrent requests/streams right now. This is your saturation signal for capacity planning and HPA tuning: a gauge sitting near your replica count's practical concurrency ceiling means it's time to scale nocturne rather than look for a bug.

Build a specific alert on two behaviors, because a naive success-rate dashboard can miss them:

  • These metrics count an event: error inside a 200 HTTP response as a failure. SSE responses start with a 200 before the model has said anything; the outcome that matters arrives inside the stream. A dashboard that only looks at HTTP status codes will read a broken model integration as 100% healthy.
  • /readyz returning 503 is designed behavior; on its own it is not an incident. orbit-api checks its source and nocturne; nocturne checks provider reachability (cached, not per-request). A 503 means "don't route here yet," and Kubernetes acts on it correctly without your intervention. Alert on sustained 503 (minutes, not one scrape interval) and let the first occurrence pass.

Container-level resource pressure (CPU throttling, memory approaching the limit) is outside these two services' own /metrics; watch it the ordinary way: kubectl top pods -n sreorbit, or docker stats on compose. Every container in the chart has a CPU limit set well above what it uses (1 for web, 4 for orbit-stack), so a runaway loop cannot take a node's CPU from its neighbours; lower one only after you measure it, because throttling a request-serving process shows up as p99 latency. Every container also has a memory limit, because OOM is the only honest response to a leak: a web pod cycling on OOM is a real signal.

Three more signals came with Addendum v0.5, and none of them is a metric:

  • The audit trail. A burst of auth.login_failed from one address, or of access.denied from one user, is someone guessing or probing. Filter for it at /deck/admin/audit (outcome:denied, action:auth.login_failed), or alert on the audit lines in orbit-api's log, which carry the same fields. See audit.md.
  • The trace store. On the engine's metrics listener: orbit_stack_spans_total{outcome} (a rising rejected means senders are sending spans without ids or oversized traces), orbit_stack_trace_head_bytes against ORBIT_STACK_TRACE_HEAD_BYTES, orbit_stack_trace_cut_failures_total (a segment that would not write; the head keeps its spans and retries), orbit_stack_trace_skipped_segments above zero, and orbit_stack_trace_generator_dropped_total (span-metric label sets past the generator's ceiling). See apm.md.
  • Derived alerts. The Alerts page is orbit-api evaluating six rules on the cluster it reads; each rule, and what to check when it fires, is in alerts.md.

The Components table

/deck/observability ends its telemetry engine section with one row per controller this install runs (every Deployment, StatefulSet and DaemonSet labelled app.kubernetes.io/part-of: sreorbit, the demo shop left out), from GET /api/v1/clusters/{id}/components (telemetry.query and estate.read):

ColumnWhere it comes from
Component, VersionThe first container's image: its repository's last part, and its tag (or its digest's first 12 hex digits). When the ready pods run another version than the template asks for, both, as "running → asked": an upgrade stuck on its old pods.
ReadyReady against desired replicas; a DaemonSet's ready pods against every node of the cluster, so a node it does not cover shows. Amber, with the workloads page's status word, when it is short, when a rollout is not done or the version asked for is not running, and at zero replicas unless a feature switch parked it ("parked").
Restarts 24 hincrease(kube_pod_container_status_restarts_total[24h]) over the component's pods, by name. Empty without kube-state-metrics.
Memory against requestThe workloads page's usage against the pods' requests.
Issues 24 hWhat the component lost, counted on its own pods (by the series' address or node): orbit-collector's dropped spans and log records, orbit-probe's lost kernel events, orbit-agent's dropped samples; export or remote-write retries beside it. A 4xx or 5xx from the engine drops a batch with no retry, so loss is the count that matters. Empty when the engine holds no such series for its pods.
Diskorbit-stack only: what the engine stores (metric blocks, write-ahead logs and spill, trace and log segments, its own count) against its claim's PersistentVolume and, when that grew over the last day, how many days are left at that pace.

The agent scrapes orbit-collector's /metrics (the collector Service's metrics port, its health port 13133) and orbit-probe's (the probe pod's metrics port, 9112) as job="sreorbit", with component set to otel-collector and orbit-probe. The table opens nothing and raises nothing: it is where to look before reaching for kubectl.

Every claim of the cluster, not only SRE Orbit's, is watched the same way a node's disk is: the agent reads the kubelet's /metrics once a minute (the kubelet refreshes volume stats every minute or two) and keeps two series a claim of it, kubelet_volume_stats_used_bytes and _capacity_bytes. When a claim mounted by a running pod climbs steadily toward full, Nocturne forecasts when (finding volume_full, from the filesystem_full detector: the last half hour reaches two hours ahead, the last six hours a day, and the peak of each two hours over the last day four days, as kube-prometheus' KubePersistentVolumeFillingUp does, so a database's sawtooth still forecasts; the soonest forecast is kept, critical under 30 minutes), and the workloads list, the workload, its namespace and its node show it as one collapsed line. A claim at 97% or more with no forecast ahead (volume_filled, critical) appears on the Nocturne page alone: what is left, and since when, to the 5 minutes of its daily read, or that it has been there for over 24 hours. A volume that is only ReadOnlyMany takes no writes and is left out. Orbit SQL's persistent_volumes.used_bytes holds the same reading. A hostPath volume (local-path, k3s' default class) has no size of its own: the kubelet measures it as its node's whole disk, so it gets no forecast and no used_bytes, and the node's own disk rows cover it.

The data directory: back it up

orbit-api keeps users, roles, sessions, saved dashboards, alert acknowledgements, contact-form leads, the audit trail and the change timeline under ORBIT_DATA_DIR (identity.md → "Where the state lives"). Nothing else in the product keeps state that the engine does not. Losing it means every account, every saved dashboard and the audit history: back it up the way you back up a database.

A copy taken while orbit-api runs is consistent: every document is replaced by a rename after an fsync, so a snapshot holds the old version or the new one, never half; and every audit line is fsynced when written, while a reader skips a last line that has no newline yet. There is nothing to quiesce.

Kubernetes, with components.orbit-api.persistence.enabled. Snapshot the claim <release>-orbit-api-data with your CSI driver's snapshots, or with the tool you already use for volumes (Velero, your cloud's backup service). With the snapshot controller installed, kubectl get volumesnapshotclass names the class to put in place of YOUR_SNAPSHOT_CLASS:

yaml
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: orbit-api-data-manual
  namespace: sreorbit
spec:
  volumeSnapshotClassName: YOUR_SNAPSHOT_CLASS
  source:
    persistentVolumeClaimName: sreorbit-orbit-api-data

The orbit-api image has no shell and no tar, so kubectl cp and kubectl exec cannot copy the directory out: snapshot the volume instead.

Kubernetes without persistence. The directory is an emptyDir and there is nothing to back up: every user, session and audit line goes with the pod. Turn persistence on before anyone depends on the console (values-prod.yaml and values-eks.yaml do).

Compose. The named volume is sreorbit_orbit-api-data. Back it up:

bash
docker run --rm -v sreorbit_orbit-api-data:/data:ro -v "$PWD":/backup busybox \  tar czf /backup/orbit-api-data.tgz -C /data .

Restore it with orbit-api stopped (docker compose --project-name sreorbit --env-file .env -f deploy/compose/docker-compose.yml stop orbit-api), then start it again:

bash
docker run --rm -v sreorbit_orbit-api-data:/data -v "$PWD":/backup busybox \  sh -c 'rm -rf /data/* && tar xzf /backup/orbit-api-data.tgz -C /data'

A restored directory brings its sessions back. When ORBIT_SESSION_SECRET is not set, it also brings session.key, and anyone holding the backup could sign session tokens for the sessions in it: treat backups like the credentials Secret, or set ORBIT_SESSION_SECRET so the key is never on the volume. Changing that secret, or deleting session.key, signs everyone out, which is the fast way to end every session at once.

Audit retention

ORBIT_AUDIT_RETENTION (default 90d) is how long a day of the trail is kept; whole day files are deleted at startup and at the first event of each new UTC day. Set it to what your review or compliance window asks for (365d for a yearly review), and size the volume from your own trail: du -sh the audit directory after a representative week and scale by the retention. The same volume holds everything else in the directory, which is small next to the trail. Anything that has to outlive the retention, or survive an administrator, belongs in your log pipeline: every event is also an audit line in orbit-api's own log.

Rolling orbit-api

orbit-api runs one replica on every install path (a single-writer store). With persistence on, the chart uses Recreate: the old pod stops before the new one mounts the claim, so a rollout is a short gap in the console's API, and sessions survive it. The plain manifests ship an emptyDir and a rolling update, and there a new pod starts with an empty directory; mount a claim before relying on it (install/kubectl.md).

What the logs look like

slog JSON to stdout on every component, level controlled by ORBIT_LOG_LEVEL / NOCTURNE_LOG_LEVEL (debug/info/warn/error). CONTRACT.md guarantees one line per request carrying method, path, status, and duration. The exact key spelling is an implementation detail: confirm it against your build (docker compose logs orbit-api or kubectl logs). This page doesn't pin it, because CONTRACT.md doesn't either.

Know these three log lines before you meet them; each looks alarming and is the system behaving correctly:

  • A WARN naming the risk, on every startup, when ORBIT_AUTH_MODE or NOCTURNE_AUTH_MODE is disabled. Expected and correct on install/local-k3d.md and install/docker-compose.md's default dev loop. It should never appear in a production log stream; if it does, ORBIT_DEMO_DATA/auth mode is misconfigured for that environment.
  • shutdown incomplete, logged on a graceful-shutdown overrun, with exit code 0. This means an in-flight request (usually a nocturne SSE stream) outlived ORBIT_SHUTDOWN_GRACE / NOCTURNE_SHUTDOWN_GRACE. It is not reported as a startup or crash failure. Don't page on it in isolation; if it's frequent, your grace period is shorter than your real p99 stream duration, so raise it (see configuration.md).
  • One line per request, always, even for 401s and 429s. A spike in 401 lines from a single source is someone probing credentials (see security.md).

Probe failures: quick triage

ProbeFailing meansFirst thing to check
Liveness (/healthz on every component)The process itself is wedged.Restart is already happening (that's what liveness is for). Check logs from immediately before the restart for a panic/OOM.
Readiness (/readyz on the backend services; /healthz on web)A dependency isn't answering, or (on web) the server itself isn't accepting connections.For orbit-api: is nocturne up? For nocturne: is the configured provider reachable? Neither check is per-request: give the cache a moment before assuming it's stuck.
Startup probeSlow cold start, not yet a failure: its generous failureThreshold is there to absorb this.Only worth investigating if it eventually fails outright; check image pull status and ORBIT_KUBECONFIG mount if ORBIT_DEMO_DATA=false.
No probe at all: orbit-api exits at startA configuration it refuses, named in its last log line.Addendum v0.5 adds five: ORBIT_DATA_DIR not writable, ORBIT_ADMIN_USER without ORBIT_ADMIN_PASSWORD_VERIFIER (or the reverse), a verifier it cannot parse, an ORBIT_SESSION_SECRET under 32 bytes, and a session.key of the wrong size.

Three specific, already-diagnosed probe failure modes have their own entries, with exact symptoms, in troubleshooting.md. Check there before assuming you've found a new one:

  1. web's pod restart-looping because something pointed its probe at /healthz. Pre-change images only; current images serve /healthz themselves (see troubleshooting.md).
  2. An SSE stream that arrives all at once at the end instead of token-by-token.
  3. A Helm rollback that doesn't land on the bytes you tested.

Where to go next

  • The exact metric/route/limit shapes referenced above: api.md.
  • Named failure modes with symptom → cause → fix: troubleshooting.md.
  • Upgrade and rollback procedures per install path: install/helm.md, install/kubectl.md, install/docker-compose.md.