Skip to content

Operate

Troubleshooting

Real failure modes, indexed by symptom. The first three are already-discovered, specific issues you will hit if you skip this page; the rest are the class of problem this contract's own design implies. If your symptom isn't here, check operations.md's probe-failure table first, then security.md for anything auth/network-shaped.

web's pod CrashLoopBackOffs on its startup probe in Kubernetes

Scope: pre-change images only. Current images serve /healthz themselves, from web's own auth proxy, answered before auth and before locale routing (see security.md). The chart and the plain kubectl manifests already probe /healthz for web like every other component. If that's the image you're running, this entry isn't your bug. It still applies to any image built before that change.

Symptom. kubectl get pods -n sreorbit shows sreorbit-web cycling through CrashLoopBackOff. kubectl describe pod shows the startup or liveness probe failing against /healthz. This does not show up on docker compose: the compose healthcheck for web passes.

Cause. CONTRACT.md scopes /healthz and /readyz to the two backend services only; web has no such route. Worse, it isn't a clean 404-and-move-on: apps/web's next-intl middleware rewrites any unmatched path into the locale-aware routing tree rather than issuing a redirect, so a probe against /healthz resolves to a real 404 response. Kubernetes' httpGet probe only treats status 200 to 399 as success, so that 404 fails the probe outright. (Had the middleware redirected instead of rewritten, the 3xx status would have accidentally fallen inside the success range and masked the problem; this is why the behavior is "rewrite, not redirect.") Compose's own healthcheck doesn't hit this because it's a hand-rolled Node one-liner that explicitly accepts any status under 500.

Fix. On a pre-change image, probe / for web, not /healthz: that's what those images serve, and it's what an old Helm chart or kubectl manifest already does by default; if you hand-rolled an ingress or a probe config, match that. On a current image, do the opposite: the chart and the plain kubectl manifests already point web's startup, liveness, and readiness probes at /healthz, answered by the proxy itself, so don't "fix" it back to /.

Every request to web answers 401, or the browser keeps prompting for credentials

Symptom. Every page, the Orbit Deck, and /api/* return 401 with WWW-Authenticate: Basic realm="SRE Orbit", charset="UTF-8"; the browser keeps re-prompting for a username and password, or accepts none you try.

Cause. WEB_AUTH_MODE=basic is on and either the credential you're entering doesn't match WEB_AUTH_USER/WEB_AUTH_VERIFIER, or you're also running ingress-level basic auth: ingress-nginx strips the Authorization header once its own check succeeds. To avoid this, the Helm render refuses outright when ingress.basicAuth.enabled and web auth are both on.

Fix. Confirm the credential against the password you hashed with hash-password.mjs; regenerate WEB_AUTH_VERIFIER if in doubt (see configuration.md) and restart web's pods to pick it up. If you intentionally want ingress-level basic auth, turn components.web.auth.enabled off instead of running both.

web pods never go Ready, and every path returns 503

Symptom. kubectl get pods shows web never Ready; every path, including /healthz, returns 503 with body code not_ready. A log line starting web auth: is printed once per pod.

Cause. WEB_AUTH_MODE=basic is set but WEB_AUTH_USER or WEB_AUTH_VERIFIER is missing or malformed. web fails closed rather than serve traffic with broken auth: the pod is never marked Ready, by design, so a rollout with a bad verifier stalls instead of replacing working pods.

Fix. Read the web auth: log line for the specific problem, fix the Secret keys (see configuration.md), then roll the deployment: kubectl -n <ns> rollout restart deploy/<release>-web.

web pod stuck in CreateContainerConfigError: couldn't find key WEB_AUTH_USER

Symptom. kubectl get pods shows web in CreateContainerConfigError; kubectl describe pod names the error: couldn't find key WEB_AUTH_USER in Secret ... (or the same for WEB_AUTH_VERIFIER).

Cause. Web auth is on (components.web.auth.enabled), and the chart wires WEB_AUTH_USER/WEB_AUTH_VERIFIER from the release Secret via secretKeyRef with no optional, on purpose: a Secret missing either key fails immediately and visibly instead of starting web with auth half-configured.

Fix. Add the missing key(s) to the Secret. No restart needed after that: the pod already exists in a failed state and starts by itself once the keys exist.

429 rate_limited while signing in to web

Symptom. Signing in to web returns 429 with body code rate_limited, either with a Retry-After: 1 header or a larger Retry-After that grows with how many wrong guesses this client already made.

Cause. Two layers, and the header says which one tripped. Retry-After: 1 is the global layer: more than two password derivations are already in flight (usually many wrong guesses, or automated probing, arriving concurrently), and it bounds the CPU a flood can burn no matter the source. A larger Retry-After is the per-client layer: after 5 wrong passwords from the same client, each further wrong one doubles the wait (1, 2, 4 s, up to 60 s) until 15 minutes pass without a miss, or the correct password clears it outright. The client is the last X-Forwarded-For hop, which a pod that reaches web directly can forge and rotate, so spreading a flood across fake clients still falls back on the global cap above.

Fix. Wait however many seconds Retry-After gives and retry. If this is constant rather than occasional, something is hammering web with credentials: chase where that traffic originates rather than raising either limit. Retries that never succeed mean the credential itself is wrong: confirm it against the password you hashed with hash-password.mjs.

Chat responses arrive all at once instead of token-by-token

Symptom. The Nocturne AI chat UI (or a direct curl against /api/v1/nocturne/chat) sits with no output, then either dumps the entire reply at once or cuts off around the 60-second mark, instead of streaming tokens as they're generated.

Cause. An ingress or reverse proxy between the caller and nocturne is buffering the response. A buffered stream defeats SSE by definition, and a default proxy_read_timeout around 60s will also sever a longer generation mid-response. That is why the Helm chart auto-renders SSE-safe annotations for ingress.className: nginx (proxy-buffering: "off", proxy-request-buffering: "off", proxy-read-timeout: "300", proxy-send-timeout: "300", and a 1m body cap matching the contract) and why deploy/kubectl/60-ingress.yaml ships the same annotations by hand.

Fix. If you're on nginx via the Helm chart, this is already handled; don't override ingress.annotations with a proxy-buffering value of "on". If you're using a different ingress class or a hand-rolled proxy in front of any of these three components, apply that technology's own equivalent:

Ingress / proxyWhat to set
nginxproxy-buffering: "off", proxy-request-buffering: "off", proxy-read-timeout: "300", proxy-send-timeout: "300"
TraefikUnbuffered by default. For a hard guarantee, set flushInterval: "1ms" on a ServersTransport/TraefikService.
HAProxyhaproxy.org/timeout-server: "300s", haproxy.org/response-buffer-size: "0"
AWS ALBalb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,deregistration_delay.timeout_seconds=300
GCEA BackendConfig with timeoutSec: 300; annotations alone cannot do this on GCE.
ContourAn HTTPProxy with timeoutPolicy.response: 300s

Also confirm the application timeout isn't the real ceiling: NOCTURNE_TIMEOUT (default 120s) is the upstream provider timeout, independent of any proxy setting.

A Helm rollback doesn't land on the bytes you tested

Symptom. helm -n sreorbit rollback sreorbit "succeeds," but the behavior you see afterward isn't what you remember testing before the bad release, as if the rollback had not restored the old version.

Cause. Without an image digest, a component's image reference in the chart is a tag (ghcr.io/sreorbit/orbit-api:0.1.0), and a tag is a mutable pointer. If anyone pushed a new image to that same tag between your original install and the rollback, helm rollback re-applies the recorded release values, which still say :0.1.0, and Kubernetes pulls whatever :0.1.0 currently resolves to, not what it resolved to at install time. The rollback is byte-identical to the release record, not necessarily to the bytes that ran.

Fix. Never install production on a bare tag. values-prod.yaml sets requireDigest: true so the chart refuses to render at all if any enabled component's image.digest is empty. Resolve every digest in CI first (deploy/helm/sreorbit/digests.example.yaml has the exact crane digest loop) and pass it with -f digests.yaml or --set-string components.<name>.image.digest=sha256:.... With digests pinned, a rollback restores the exact same immutable content address every time. See install/helm.md.

orbit-api or nocturne won't start at all: it exits, with no crash loop

Symptom. The container/pod exits immediately on boot; it never reaches even the startup probe. A log line names a token and a byte count.

Cause. ORBIT_AUTH_MODE / NOCTURNE_AUTH_MODE defaults to static, which requires ORBIT_AUTH_TOKEN / NOCTURNE_AUTH_TOKEN to be at least 32 bytes. An empty value (the chart's own default; it never ships one) or a short hand-typed one gets refused at startup, on purpose: misconfiguration fails closed, before serving a single request, not after.

Fix. Generate a real token and supply it as a Secret: see configuration.md → "Generating a static token" and your install path's "Secrets" section. This is not a bug to route around; a service that started anyway would be the defect.

orbit-api refuses five more things the same way, each named in its last log line: an ORBIT_DATA_DIR it cannot write, ORBIT_ADMIN_USER without ORBIT_ADMIN_PASSWORD_VERIFIER or the reverse, a verifier it cannot parse, an ORBIT_SESSION_SECRET under 32 bytes, and a session.key of the wrong size (delete the file to mint a new one; every session ends). See identity.md.

orbit-api is running but never becomes Ready

Symptom. GET /readyz (or kubectl get pods) shows orbit-api perpetually not Ready; logs show no crash, only a steady 503 on /readyz.

Cause. ORBIT_DEMO_DATA=false (the production overlay's default) with ORBIT_KUBECONFIG unset or pointing at a path that isn't mounted. orbit-api reports "I can't reach my data source" rather than serving nothing and claiming to be fine. This is the addendum's readiness requirement working as intended, not a hung process.

Fix. Mount a real kubeconfig (via extraVolumes/extraVolumeMounts in Helm, or a Volume/VolumeMount in a hand-rolled manifest) and point ORBIT_KUBECONFIG at its mount path, or set ORBIT_DEMO_DATA=true if you intend demo data in that environment.

A pod sits in ImagePullBackOff on the k3d loop only

Symptom. One or more of web/orbit-api/nocturne never leaves ImagePullBackOff on sreorbit-dev, even though the same chart installs cleanly elsewhere.

Cause. values-dev.yaml sets image.pullPolicy: Never on purpose, so a missing k3d image import fails loudly and immediately instead of silently reaching out to ghcr.io and timing out. If a component's image was never built and imported (most commonly: its build failed, or SKIP_BUILD was set on a cluster that never had it, when you ran the loop), this is the expected symptom, not a networking bug.

Fix. ./deploy/k3d/up.sh (or make k3d-up) already prints note: <name> was not built, so the pods running <image> will not start. at the end of its run when this happens; read that line. Build and import the missing image (re-run without SKIP_BUILD), then make k3d-deploy.

A NetworkPolicy exists, applies cleanly, and blocks nothing

Symptom. A pod in an unrelated namespace can still reach orbit-api or nocturne (or scrape :9090) even though the matching NetworkPolicy object is present and kubectl get networkpolicy shows it applied without error.

Cause. NetworkPolicy is only enforced by a CNI that implements it. Flannel alone does not; k3s adds kube-router's policy controller, so install/local-k3d.md's cluster does enforce. EKS enforces only with the VPC CNI's network policy agent turned on, and most EKS clusters run without it. The object applying cleanly proves the YAML is valid; it proves nothing about enforcement.

Fix. It is a property of the CNI, not this chart: turn enforcement on (on EKS, enableNetworkPolicy on the VPC CNI add-on, or a Calico/Cilium add-on) and prove it with a pod from another namespace before relying on this control for anything. Full detail: security.md.

401 unauthorized with a token you're sure is correct

Symptom. Every request gets 401 / {"error":{"code":"unauthorized",...}} despite the Authorization header visibly containing the token you generated.

Cause, almost always. A trailing newline or stray whitespace character got embedded in the secret value, most commonly from piping openssl rand -hex 32 straight into a command that doesn't strip the newline, or from a copy-paste that picked up a line ending. The stored token and the token you think you set differ by one invisible byte, and a constant-time comparison doesn't partially match.

Fix. Regenerate and set the value with printf or echo -n, never a bare echo, and double-check any --from-literal=/.env value has no trailing newline (xxd or cat -A on the file will show it if present). See configuration.md.

401 with the right bearer: "sign in: this needs a session"

Symptom. A call to orbit-api with a bearer you know is right answers 401, and the message reads sign in: this needs a session, and there is none or it ended, not valid credentials are required.

Cause. The bearer passed; the session did not. Since Addendum v0.5 every route but the probes, the sign-in routes and contact needs a signed-in user as well, sent as X-Orbit-Session (by web, from the orbit_session cookie). The session is missing, expired (24 hours after sign-in, or 8 hours with no request), revoked (logout, a password change, a disabled account, "End all sessions"), or signed with a key that has since changed.

Fix. Sign in and send the token: identity.md → "Signing in from a shell". In a browser, sign in again at /login. Use GET /api/v1/auth/state to check a bearer alone.

403 forbidden: "this needs the ... permission"

Symptom. A page of the Deck says you have no access, or a call answers 403 naming a permission and your role.

Cause. The session is fine and its role lacks that permission. Each refusal is also an access.denied event in the audit trail. A common one: a custom role with dashboards.read but not telemetry.query opens dashboards and has every panel refused.

Fix. An admin gives the user a role that holds it, or adds it to their custom role, at /deck/admin/roles; the change applies on the user's next request. See identity.md → "Roles and permissions".

Nobody can sign in as an admin

Symptom. The only admin's password is lost, or the setup code in the log no longer works, or ORBIT_ADMIN_USER changed and nothing happened.

Cause. The setup code lives only in the running process and is printed again, new, at every start while no user exists; once one exists there is no code. The admin pair in the Secret creates an account only while no user exists, and is ignored afterwards.

Fix. Another user with users.manage resets the password at /deck/admin/users. If there is none, empty the accounts, keep everything else, and let the admin pair create a new admin: stop orbit-api, delete users/ and sessions/ from its data directory (roles, dashboards and the audit trail stay), set ORBIT_ADMIN_USER and ORBIT_ADMIN_PASSWORD_VERIFIER, and start it. On compose, put the pair in .env, then:

bash
docker compose --project-name sreorbit --env-file .env -f deploy/compose/docker-compose.yml stop orbit-apidocker run --rm -v sreorbit_orbit-api-data:/data busybox rm -rf /data/users /data/sessions

and start the stack again the way you started it, make up or make up-prod.

In Kubernetes the orbit-api image has no shell: scale it to 0, mount its claim in a temporary pod that has one (the claim is ReadWriteOnce, so only once orbit-api is gone), delete the same two directories, then scale it back to 1 with the pair in the Secret. Every user signs in again afterwards; recreate the other accounts.

APM pages are empty, or say the engine is not ready

Symptom. /deck/apm shows no service, or says the telemetry engine is not ready.

Cause, in order of likelihood. Nothing sends spans in the window you are looking at; the sender is refused (401 without the engine's bearer, 415 for a wrong Content-Type) or speaks gRPC, which the engine does not serve; a NetworkPolicy drops the sender's connection to 8082, which shows as a timeout at the sender, not as an error here; or orbit-api has no ORBIT_STACK_URL, which answers 503 not_ready on every APM route.

Fix. Send one span by hand (apm.md → "From a shell"), then read it back by id. If that works, check the sender's exporter protocol and bearer, and add its namespace to components.orbit-stack.networkPolicy.allowFromNamespaces. orbit_stack_spans_total counts what arrived.

Every audit event has the same address

Symptom. The ip of every event in the audit trail is the same private address.

Cause. That address is web's, or the load balancer's in front of the ingress. web passes the client on only with WEB_TRUST_PROXY=true (compose sets false), and the ingress controller can only pass on the client it sees.

Fix. audit.md → "How the client IP is determined": turn WEB_TRUST_PROXY on behind an ingress, and have the edge preserve the client address. Those edge settings are controller-wide, so they are the cluster owner's call.

Where to go next