Operate
Operations
What to watch once the stack is running, what saturation looks like from the outside, what the logs look like, and what each probe failure means. For install-time verification ("did this come up healthy at all"), see the "Verify" section of your install path instead; this page covers steady-state operation.
What to watch
Both backend services expose Prometheus metrics on their own listener
(ORBIT_METRICS_ADDR / NOCTURNE_METRICS_ADDR, default :9090; see
configuration.md). CONTRACT.md guarantees three metric shapes exist; it does
not pin the exact metric name string, which is conventionally prefixed per service
(orbit_... / nocturne_...). Confirm the literal name for your build with
curl localhost:9090/metrics (port-forwarded or, on compose, published directly). Build
alerts and dashboards on these three:
| Metric shape | Type | What it tells you |
|---|---|---|
*_provider_requests_total{tier,outcome} | counter | Request volume per model tier (reasoning/fast) and outcome. A rising outcome="error" rate against a flat total points at the provider, since load has not changed. |
*_stream_duration_seconds | histogram, buckets out to 300s | How long a chat SSE stream runs. nocturne's NOCTURNE_TIMEOUT default is 120s: if p99 here is climbing toward that ceiling, callers are about to start timing out. |
| in-flight gauge | gauge | Concurrent requests/streams right now. This is your saturation signal for capacity planning and HPA tuning: a gauge sitting near your replica count's practical concurrency ceiling means it's time to scale nocturne rather than look for a bug. |
Build a specific alert on two behaviors, because a naive success-rate dashboard can miss them:
- These metrics count an
event: errorinside a200HTTP response as a failure. SSE responses start with a200before the model has said anything; the outcome that matters arrives inside the stream. A dashboard that only looks at HTTP status codes will read a broken model integration as 100% healthy. /readyzreturning503is designed behavior; on its own it is not an incident.orbit-apichecks its source andnocturne;nocturnechecks provider reachability (cached, not per-request). A503means "don't route here yet," and Kubernetes acts on it correctly without your intervention. Alert on sustained503(minutes, not one scrape interval) and let the first occurrence pass.
Container-level resource pressure (CPU throttling, memory approaching the limit) is
outside these two services' own /metrics; watch it the ordinary way:
kubectl top pods -n sreorbit, or docker stats on compose. Every container in the
chart has a CPU limit set well above what it uses (1 for web, 4 for orbit-stack),
so a runaway loop cannot take a node's CPU from its neighbours; lower one only after you
measure it, because throttling a request-serving process shows up as p99 latency. Every
container also has a memory limit, because OOM is the only honest response to a leak: a
web pod cycling on OOM is a real signal.
Three more signals came with Addendum v0.5, and none of them is a metric:
- The audit trail. A burst of
auth.login_failedfrom one address, or ofaccess.deniedfrom one user, is someone guessing or probing. Filter for it at/deck/admin/audit(outcome:denied,action:auth.login_failed), or alert on theauditlines inorbit-api's log, which carry the same fields. Seeaudit.md. - The trace store. On the engine's metrics listener:
orbit_stack_spans_total{outcome}(a risingrejectedmeans senders are sending spans without ids or oversized traces),orbit_stack_trace_head_bytesagainstORBIT_STACK_TRACE_HEAD_BYTES,orbit_stack_trace_cut_failures_total(a segment that would not write; the head keeps its spans and retries),orbit_stack_trace_skipped_segmentsabove zero, andorbit_stack_trace_generator_dropped_total(span-metric label sets past the generator's ceiling). Seeapm.md. - Derived alerts. The Alerts page is
orbit-apievaluating six rules on the cluster it reads; each rule, and what to check when it fires, is inalerts.md.
The Components table
/deck/observability ends its telemetry engine section with one row per controller this
install runs (every Deployment, StatefulSet and DaemonSet labelled
app.kubernetes.io/part-of: sreorbit, the demo shop left out), from
GET /api/v1/clusters/{id}/components (telemetry.query and estate.read):
| Column | Where it comes from |
|---|---|
| Component, Version | The first container's image: its repository's last part, and its tag (or its digest's first 12 hex digits). When the ready pods run another version than the template asks for, both, as "running → asked": an upgrade stuck on its old pods. |
| Ready | Ready against desired replicas; a DaemonSet's ready pods against every node of the cluster, so a node it does not cover shows. Amber, with the workloads page's status word, when it is short, when a rollout is not done or the version asked for is not running, and at zero replicas unless a feature switch parked it ("parked"). |
| Restarts 24 h | increase(kube_pod_container_status_restarts_total[24h]) over the component's pods, by name. Empty without kube-state-metrics. |
| Memory against request | The workloads page's usage against the pods' requests. |
| Issues 24 h | What the component lost, counted on its own pods (by the series' address or node): orbit-collector's dropped spans and log records, orbit-probe's lost kernel events, orbit-agent's dropped samples; export or remote-write retries beside it. A 4xx or 5xx from the engine drops a batch with no retry, so loss is the count that matters. Empty when the engine holds no such series for its pods. |
| Disk | orbit-stack only: what the engine stores (metric blocks, write-ahead logs and spill, trace and log segments, its own count) against its claim's PersistentVolume and, when that grew over the last day, how many days are left at that pace. |
The agent scrapes orbit-collector's /metrics (the collector Service's metrics port, its
health port 13133) and orbit-probe's (the probe pod's metrics port, 9112) as
job="sreorbit", with component set to otel-collector and orbit-probe. The table opens
nothing and raises nothing: it is where to look before reaching for kubectl.
Every claim of the cluster, not only SRE Orbit's, is watched the same way a node's disk is:
the agent reads the kubelet's /metrics once a minute (the kubelet refreshes volume stats every
minute or two) and keeps two series a claim of it, kubelet_volume_stats_used_bytes and
_capacity_bytes. When a claim mounted by a running pod climbs steadily toward full, Nocturne
forecasts when (finding volume_full, from the filesystem_full detector: the last half hour
reaches two hours ahead, the last six hours a day, and the peak of each two hours over the last day
four days, as kube-prometheus' KubePersistentVolumeFillingUp does, so a database's sawtooth still
forecasts; the soonest forecast is kept, critical under 30 minutes), and the workloads list, the
workload, its namespace and its node show it as one collapsed line. A claim at 97% or more with no
forecast ahead (volume_filled, critical) appears on the Nocturne page alone: what is left, and since
when, to the 5 minutes of its daily read, or that it has been there for over 24 hours. A volume that is
only ReadOnlyMany takes no writes and is left out. Orbit SQL's persistent_volumes.used_bytes holds the same reading. A
hostPath volume (local-path, k3s' default class) has no size of its own: the kubelet measures it as
its node's whole disk, so it gets no forecast and no used_bytes, and the node's own disk rows
cover it.
The data directory: back it up
orbit-api keeps users, roles, sessions, saved dashboards, alert acknowledgements,
contact-form leads, the audit trail and the change timeline under ORBIT_DATA_DIR
(identity.md → "Where the state lives"). Nothing else in the product
keeps state that the engine does not. Losing it means every account, every saved
dashboard and the audit history: back it up the way you back up a database.
A copy taken while orbit-api runs is consistent: every document is replaced by a
rename after an fsync, so a snapshot holds the old version or the new one, never half;
and every audit line is fsynced when written, while a reader skips a last line that has
no newline yet. There is nothing to quiesce.
Kubernetes, with components.orbit-api.persistence.enabled. Snapshot the claim
<release>-orbit-api-data with your CSI driver's snapshots, or with the tool you already
use for volumes (Velero, your cloud's backup service). With the snapshot controller
installed, kubectl get volumesnapshotclass names the class to put in place of
YOUR_SNAPSHOT_CLASS:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: orbit-api-data-manual
namespace: sreorbit
spec:
volumeSnapshotClassName: YOUR_SNAPSHOT_CLASS
source:
persistentVolumeClaimName: sreorbit-orbit-api-dataThe orbit-api image has no shell and no tar, so kubectl cp and kubectl exec
cannot copy the directory out: snapshot the volume instead.
Kubernetes without persistence. The directory is an emptyDir and there is nothing
to back up: every user, session and audit line goes with the pod. Turn persistence on
before anyone depends on the console (values-prod.yaml and values-eks.yaml do).
Compose. The named volume is sreorbit_orbit-api-data. Back it up:
docker run --rm -v sreorbit_orbit-api-data:/data:ro -v "$PWD":/backup busybox \ tar czf /backup/orbit-api-data.tgz -C /data .Restore it with orbit-api stopped (docker compose --project-name sreorbit --env-file .env -f deploy/compose/docker-compose.yml stop orbit-api), then start it again:
docker run --rm -v sreorbit_orbit-api-data:/data -v "$PWD":/backup busybox \ sh -c 'rm -rf /data/* && tar xzf /backup/orbit-api-data.tgz -C /data'A restored directory brings its sessions back. When ORBIT_SESSION_SECRET is not set,
it also brings session.key, and anyone holding the backup could sign session tokens for
the sessions in it: treat backups like the credentials Secret, or set
ORBIT_SESSION_SECRET so the key is never on the volume. Changing that secret, or
deleting session.key, signs everyone out, which is the fast way to end every session at
once.
Audit retention
ORBIT_AUDIT_RETENTION (default 90d) is how long a day of the trail is kept; whole day
files are deleted at startup and at the first event of each new UTC day. Set it to what
your review or compliance window asks for (365d for a yearly review), and size the
volume from your own trail: du -sh the audit directory after a representative week and
scale by the retention. The same volume holds everything else in the directory, which is
small next to the trail. Anything that has to outlive the retention, or survive an
administrator, belongs in your log pipeline: every event is also an audit line in
orbit-api's own log.
Rolling orbit-api
orbit-api runs one replica on every install path (a single-writer store). With
persistence on, the chart uses Recreate: the old pod stops before the new one mounts the
claim, so a rollout is a short gap in the console's API, and sessions survive it. The
plain manifests ship an emptyDir and a rolling update, and there a new pod starts with an
empty directory; mount a claim before relying on it
(install/kubectl.md).
What the logs look like
slog JSON to stdout on every component, level controlled by ORBIT_LOG_LEVEL /
NOCTURNE_LOG_LEVEL (debug/info/warn/error). CONTRACT.md guarantees one
line per request carrying method, path, status, and duration. The exact key
spelling is an implementation detail: confirm it against your build
(docker compose logs orbit-api or kubectl logs). This page doesn't pin it, because
CONTRACT.md doesn't either.
Know these three log lines before you meet them; each looks alarming and is the system behaving correctly:
- A
WARNnaming the risk, on every startup, whenORBIT_AUTH_MODEorNOCTURNE_AUTH_MODEisdisabled. Expected and correct oninstall/local-k3d.mdandinstall/docker-compose.md's default dev loop. It should never appear in a production log stream; if it does,ORBIT_DEMO_DATA/auth mode is misconfigured for that environment. shutdown incomplete, logged on a graceful-shutdown overrun, with exit code0. This means an in-flight request (usually anocturneSSE stream) outlivedORBIT_SHUTDOWN_GRACE/NOCTURNE_SHUTDOWN_GRACE. It is not reported as a startup or crash failure. Don't page on it in isolation; if it's frequent, your grace period is shorter than your real p99 stream duration, so raise it (seeconfiguration.md).- One line per request, always, even for
401s and429s. A spike in401lines from a single source is someone probing credentials (seesecurity.md).
Probe failures: quick triage
| Probe | Failing means | First thing to check |
|---|---|---|
Liveness (/healthz on every component) | The process itself is wedged. | Restart is already happening (that's what liveness is for). Check logs from immediately before the restart for a panic/OOM. |
Readiness (/readyz on the backend services; /healthz on web) | A dependency isn't answering, or (on web) the server itself isn't accepting connections. | For orbit-api: is nocturne up? For nocturne: is the configured provider reachable? Neither check is per-request: give the cache a moment before assuming it's stuck. |
| Startup probe | Slow cold start, not yet a failure: its generous failureThreshold is there to absorb this. | Only worth investigating if it eventually fails outright; check image pull status and ORBIT_KUBECONFIG mount if ORBIT_DEMO_DATA=false. |
No probe at all: orbit-api exits at start | A configuration it refuses, named in its last log line. | Addendum v0.5 adds five: ORBIT_DATA_DIR not writable, ORBIT_ADMIN_USER without ORBIT_ADMIN_PASSWORD_VERIFIER (or the reverse), a verifier it cannot parse, an ORBIT_SESSION_SECRET under 32 bytes, and a session.key of the wrong size. |
Three specific, already-diagnosed probe failure modes have their own entries, with
exact symptoms, in troubleshooting.md. Check there before
assuming you've found a new one:
web's pod restart-looping because something pointed its probe at/healthz. Pre-change images only; current images serve/healthzthemselves (seetroubleshooting.md).- An SSE stream that arrives all at once at the end instead of token-by-token.
- A Helm rollback that doesn't land on the bytes you tested.
Where to go next
- The exact metric/route/limit shapes referenced above:
api.md. - Named failure modes with symptom → cause → fix:
troubleshooting.md. - Upgrade and rollback procedures per install path:
install/helm.md,install/kubectl.md,install/docker-compose.md.