Operate
Troubleshooting
Real failure modes, indexed by symptom. The first three are already-discovered,
specific issues you will hit if you skip this page; the rest are the class of
problem this contract's own design implies. If your symptom isn't here, check
operations.md's probe-failure table first, then security.md for anything
auth/network-shaped.
web's pod CrashLoopBackOffs on its startup probe in Kubernetes
Scope: pre-change images only. Current images serve /healthz themselves, from
web's own auth proxy, answered before auth and before locale routing (see
security.md). The chart and the plain kubectl manifests already probe
/healthz for web like every other component. If that's the image you're running,
this entry isn't your bug. It still applies to any image built before that change.
Symptom. kubectl get pods -n sreorbit shows sreorbit-web cycling through
CrashLoopBackOff. kubectl describe pod shows the startup or liveness probe
failing against /healthz. This does not show up on docker compose: the
compose healthcheck for web passes.
Cause. CONTRACT.md scopes /healthz and /readyz to the two backend services only;
web has no such route. Worse, it isn't a clean 404-and-move-on: apps/web's
next-intl middleware rewrites any unmatched path into the locale-aware routing
tree rather than issuing a redirect, so a probe against /healthz resolves to a real
404 response. Kubernetes' httpGet probe only treats status 200 to 399 as
success, so that 404 fails the probe outright. (Had the middleware redirected
instead of rewritten, the 3xx status would have accidentally fallen inside the
success range and masked the problem; this is why the behavior is
"rewrite, not redirect.") Compose's own healthcheck doesn't hit this because it's a
hand-rolled Node one-liner that explicitly accepts any status under 500.
Fix. On a pre-change image, probe / for web, not /healthz: that's what
those images serve, and it's what an old Helm chart or kubectl manifest
already does by default; if you hand-rolled an ingress or a probe config, match that.
On a current image, do the opposite: the chart and the plain kubectl manifests
already point web's startup, liveness, and readiness probes at /healthz, answered
by the proxy itself, so don't "fix" it back to /.
Every request to web answers 401, or the browser keeps prompting for credentials
Symptom. Every page, the Orbit Deck, and /api/* return 401 with
WWW-Authenticate: Basic realm="SRE Orbit", charset="UTF-8"; the browser keeps re-prompting for a
username and password, or accepts none you try.
Cause. WEB_AUTH_MODE=basic is on and either the credential you're entering
doesn't match WEB_AUTH_USER/WEB_AUTH_VERIFIER, or you're also running
ingress-level basic auth: ingress-nginx strips the Authorization header once its
own check succeeds. To avoid this, the Helm render refuses outright when
ingress.basicAuth.enabled and web auth are both on.
Fix. Confirm the credential against the password you hashed with
hash-password.mjs; regenerate WEB_AUTH_VERIFIER if in doubt (see
configuration.md) and restart web's pods to pick it up. If you intentionally want
ingress-level basic auth, turn components.web.auth.enabled off instead of running
both.
web pods never go Ready, and every path returns 503
Symptom. kubectl get pods shows web never Ready; every path, including
/healthz, returns 503 with body code not_ready. A log line starting
web auth: is printed once per pod.
Cause. WEB_AUTH_MODE=basic is set but WEB_AUTH_USER or WEB_AUTH_VERIFIER is
missing or malformed. web fails closed rather than serve traffic with broken auth:
the pod is never marked Ready, by design, so a rollout with a bad verifier stalls
instead of replacing working pods.
Fix. Read the web auth: log line for the specific problem, fix the Secret keys
(see configuration.md), then roll the deployment:
kubectl -n <ns> rollout restart deploy/<release>-web.
web pod stuck in CreateContainerConfigError: couldn't find key WEB_AUTH_USER
Symptom. kubectl get pods shows web in CreateContainerConfigError;
kubectl describe pod names the error:
couldn't find key WEB_AUTH_USER in Secret ... (or the same for WEB_AUTH_VERIFIER).
Cause. Web auth is on (components.web.auth.enabled), and the chart wires
WEB_AUTH_USER/WEB_AUTH_VERIFIER from the release Secret via secretKeyRef with
no optional, on purpose: a Secret missing either key fails immediately and
visibly instead of starting web with auth half-configured.
Fix. Add the missing key(s) to the Secret. No restart needed after that: the pod already exists in a failed state and starts by itself once the keys exist.
429 rate_limited while signing in to web
Symptom. Signing in to web returns 429 with body code rate_limited,
either with a Retry-After: 1 header or a larger Retry-After that grows
with how many wrong guesses this client already made.
Cause. Two layers, and the header says which one tripped. Retry-After: 1 is the global layer: more than two password derivations are already in
flight (usually many wrong guesses, or automated probing, arriving
concurrently), and it bounds the CPU a flood can burn no matter the source.
A larger Retry-After is the per-client layer: after 5 wrong passwords from
the same client, each further wrong one doubles the wait (1, 2, 4 s, up to
60 s) until 15 minutes pass without a miss, or the correct password clears
it outright. The client is the last X-Forwarded-For hop, which a pod that
reaches web directly can forge and rotate, so spreading a flood across
fake clients still falls back on the global cap above.
Fix. Wait however many seconds Retry-After gives and retry. If this is
constant rather than occasional, something is hammering web with
credentials: chase where that traffic originates rather than raising
either limit. Retries that never succeed mean the credential itself is
wrong: confirm it against the password you hashed with hash-password.mjs.
Chat responses arrive all at once instead of token-by-token
Symptom. The Nocturne AI chat UI (or a direct curl against /api/v1/nocturne/chat)
sits with no output, then either dumps the entire reply at once or cuts off around
the 60-second mark, instead of streaming tokens as they're generated.
Cause. An ingress or reverse proxy between the caller and nocturne is buffering
the response. A buffered stream defeats SSE by definition,
and a default proxy_read_timeout around 60s will also sever a longer generation
mid-response. That is why the Helm chart auto-renders
SSE-safe annotations for ingress.className: nginx
(proxy-buffering: "off", proxy-request-buffering: "off",
proxy-read-timeout: "300", proxy-send-timeout: "300", and a 1m body cap
matching the contract) and why deploy/kubectl/60-ingress.yaml ships the same
annotations by hand.
Fix. If you're on nginx via the Helm chart, this is already handled; don't
override ingress.annotations with a proxy-buffering value of "on". If you're
using a different ingress class or a hand-rolled proxy in front of any of these three
components, apply that technology's own equivalent:
| Ingress / proxy | What to set |
|---|---|
| nginx | proxy-buffering: "off", proxy-request-buffering: "off", proxy-read-timeout: "300", proxy-send-timeout: "300" |
| Traefik | Unbuffered by default. For a hard guarantee, set flushInterval: "1ms" on a ServersTransport/TraefikService. |
| HAProxy | haproxy.org/timeout-server: "300s", haproxy.org/response-buffer-size: "0" |
| AWS ALB | alb.ingress.kubernetes.io/target-group-attributes: stickiness.enabled=true,deregistration_delay.timeout_seconds=300 |
| GCE | A BackendConfig with timeoutSec: 300; annotations alone cannot do this on GCE. |
| Contour | An HTTPProxy with timeoutPolicy.response: 300s |
Also confirm the application timeout isn't the real ceiling: NOCTURNE_TIMEOUT
(default 120s) is the upstream provider timeout, independent of any proxy setting.
A Helm rollback doesn't land on the bytes you tested
Symptom. helm -n sreorbit rollback sreorbit "succeeds," but the behavior you
see afterward isn't what you remember testing before the bad release, as if the
rollback had not restored the old version.
Cause. Without an image digest, a component's image reference in the chart is
a tag (ghcr.io/sreorbit/orbit-api:0.1.0), and a tag is a mutable pointer. If anyone
pushed a new image to that same tag between your original install and the rollback,
helm rollback re-applies the recorded release values, which still say
:0.1.0, and Kubernetes pulls whatever :0.1.0 currently resolves to, not what it
resolved to at install time. The rollback is byte-identical to the release record,
not necessarily to the bytes that ran.
Fix. Never install production on a bare tag. values-prod.yaml sets
requireDigest: true so the chart refuses to render at all if any
enabled component's image.digest is empty. Resolve every digest in CI first
(deploy/helm/sreorbit/digests.example.yaml has the exact crane digest loop) and
pass it with -f digests.yaml or --set-string components.<name>.image.digest=sha256:.... With digests pinned, a rollback restores
the exact same immutable content address every time. See install/helm.md.
orbit-api or nocturne won't start at all: it exits, with no crash loop
Symptom. The container/pod exits immediately on boot; it never reaches even the startup probe. A log line names a token and a byte count.
Cause. ORBIT_AUTH_MODE / NOCTURNE_AUTH_MODE defaults to static, which
requires ORBIT_AUTH_TOKEN / NOCTURNE_AUTH_TOKEN to be at least 32 bytes. An
empty value (the chart's own default; it never ships one) or a short hand-typed one
gets refused at startup, on purpose: misconfiguration fails closed, before serving
a single request, not after.
Fix. Generate a real token and supply it as a Secret: see configuration.md →
"Generating a static token" and your install path's "Secrets" section. This is not a
bug to route around; a service that started anyway would be the defect.
orbit-api refuses five more things the same way, each named in its last log line: an
ORBIT_DATA_DIR it cannot write, ORBIT_ADMIN_USER without
ORBIT_ADMIN_PASSWORD_VERIFIER or the reverse, a verifier it cannot parse, an
ORBIT_SESSION_SECRET under 32 bytes, and a session.key of the wrong size (delete the
file to mint a new one; every session ends). See identity.md.
orbit-api is running but never becomes Ready
Symptom. GET /readyz (or kubectl get pods) shows orbit-api perpetually not
Ready; logs show no crash, only a steady 503 on /readyz.
Cause. ORBIT_DEMO_DATA=false (the production overlay's default) with
ORBIT_KUBECONFIG unset or pointing at a path that isn't mounted.
orbit-api reports "I can't reach my data source" rather than serving
nothing and claiming to be fine. This is the addendum's readiness requirement
working as intended, not a hung process.
Fix. Mount a real kubeconfig (via extraVolumes/extraVolumeMounts in Helm, or
a Volume/VolumeMount in a hand-rolled manifest) and point ORBIT_KUBECONFIG at
its mount path, or set ORBIT_DEMO_DATA=true if you intend demo data in
that environment.
A pod sits in ImagePullBackOff on the k3d loop only
Symptom. One or more of web/orbit-api/nocturne never leaves
ImagePullBackOff on sreorbit-dev, even though the same chart installs cleanly
elsewhere.
Cause. values-dev.yaml sets image.pullPolicy: Never on purpose, so a missing
k3d image import fails loudly and immediately instead of silently reaching out to
ghcr.io and timing out. If a component's image was never built and imported (most
commonly: its build failed, or SKIP_BUILD was set on a cluster that never had it, when you ran the
loop), this is the expected symptom, not a networking bug.
Fix. ./deploy/k3d/up.sh (or make k3d-up) already prints
note: <name> was not built, so the pods running <image> will not start. at the end of its run
when this happens; read that line. Build and import the missing image (re-run
without SKIP_BUILD), then make k3d-deploy.
A NetworkPolicy exists, applies cleanly, and blocks nothing
Symptom. A pod in an unrelated namespace can still reach orbit-api or
nocturne (or scrape :9090) even though the matching NetworkPolicy object is
present and kubectl get networkpolicy shows it applied without error.
Cause. NetworkPolicy is only enforced by a CNI that implements it. Flannel alone
does not; k3s adds kube-router's policy controller, so install/local-k3d.md's cluster
does enforce. EKS enforces only with the VPC CNI's network policy agent turned on, and
most EKS clusters run without it. The object applying cleanly proves the YAML is
valid; it proves nothing about enforcement.
Fix. It is a property of the CNI, not this chart: turn enforcement on (on EKS,
enableNetworkPolicy on the VPC CNI add-on, or a Calico/Cilium add-on) and prove it
with a pod from another namespace before relying on this control for anything. Full
detail: security.md.
401 unauthorized with a token you're sure is correct
Symptom. Every request gets 401 / {"error":{"code":"unauthorized",...}}
despite the Authorization header visibly containing the token you generated.
Cause, almost always. A trailing newline or stray whitespace character got
embedded in the secret value, most commonly from piping openssl rand -hex 32
straight into a command that doesn't strip the newline, or from a copy-paste that
picked up a line ending. The stored token and the token you think you set differ
by one invisible byte, and a constant-time comparison doesn't partially match.
Fix. Regenerate and set the value with printf or echo -n, never a bare
echo, and double-check any --from-literal=/.env value has no trailing newline
(xxd or cat -A on the file will show it if present). See configuration.md.
401 with the right bearer: "sign in: this needs a session"
Symptom. A call to orbit-api with a bearer you know is right answers 401, and the
message reads sign in: this needs a session, and there is none or it ended, not
valid credentials are required.
Cause. The bearer passed; the session did not. Since Addendum v0.5 every route but the
probes, the sign-in routes and contact needs a signed-in user as well, sent as
X-Orbit-Session (by web, from the orbit_session cookie). The session is missing,
expired (24 hours after sign-in, or 8 hours with no request), revoked (logout, a password change, a disabled account,
"End all sessions"), or signed with a key that has since changed.
Fix. Sign in and send the token: identity.md → "Signing in from a shell". In a
browser, sign in again at /login. Use GET /api/v1/auth/state to check a bearer alone.
403 forbidden: "this needs the ... permission"
Symptom. A page of the Deck says you have no access, or a call answers 403 naming a
permission and your role.
Cause. The session is fine and its role lacks that permission. Each refusal is also an
access.denied event in the audit trail. A common one: a custom role with
dashboards.read but not telemetry.query opens dashboards and has every panel refused.
Fix. An admin gives the user a role that holds it, or adds it to their custom role, at
/deck/admin/roles; the change applies on the user's next request. See identity.md →
"Roles and permissions".
Nobody can sign in as an admin
Symptom. The only admin's password is lost, or the setup code in the log no longer
works, or ORBIT_ADMIN_USER changed and nothing happened.
Cause. The setup code lives only in the running process and is printed again, new, at every start while no user exists; once one exists there is no code. The admin pair in the Secret creates an account only while no user exists, and is ignored afterwards.
Fix. Another user with users.manage resets the password at /deck/admin/users. If
there is none, empty the accounts, keep everything else, and let the admin pair create a
new admin: stop orbit-api, delete users/ and sessions/ from its data directory (roles,
dashboards and the audit trail stay), set ORBIT_ADMIN_USER and
ORBIT_ADMIN_PASSWORD_VERIFIER, and start it. On compose, put the pair in .env, then:
docker compose --project-name sreorbit --env-file .env -f deploy/compose/docker-compose.yml stop orbit-apidocker run --rm -v sreorbit_orbit-api-data:/data busybox rm -rf /data/users /data/sessionsand start the stack again the way you started it, make up or make up-prod.
In Kubernetes the orbit-api image has no shell: scale it to 0, mount its claim in a
temporary pod that has one (the claim is ReadWriteOnce, so only once orbit-api is gone),
delete the same two directories, then scale it back to 1 with the pair in the Secret.
Every user signs in again afterwards; recreate the other accounts.
APM pages are empty, or say the engine is not ready
Symptom. /deck/apm shows no service, or says the telemetry engine is not ready.
Cause, in order of likelihood. Nothing sends spans in the window you are looking at;
the sender is refused (401 without the engine's bearer, 415 for a wrong
Content-Type) or speaks gRPC, which the engine does not serve; a NetworkPolicy drops the sender's connection to 8082, which
shows as a timeout at the sender, not as an error here; or orbit-api has no
ORBIT_STACK_URL, which answers 503 not_ready on every APM route.
Fix. Send one span by hand (apm.md → "From a shell"), then read it back by id. If
that works, check the sender's exporter protocol and bearer, and add its namespace to
components.orbit-stack.networkPolicy.allowFromNamespaces. orbit_stack_spans_total
counts what arrived.
Every audit event has the same address
Symptom. The ip of every event in the audit trail is the same private address.
Cause. That address is web's, or the load balancer's in front of the ingress. web
passes the client on only with WEB_TRUST_PROXY=true (compose sets false), and the
ingress controller can only pass on the client it sees.
Fix. audit.md → "How the client IP is determined": turn WEB_TRUST_PROXY on behind an
ingress, and have the edge preserve the client address. Those edge settings are
controller-wide, so they are the cluster owner's call.
Where to go next
- The probe/metric behavior these symptoms sit on top of:
operations.md. - The exact contract these fixes restore:
configuration.md,api.md. - The threat model behind why auth and NetworkPolicy fail closed:
security.md.