Skip to content

Install

Helm

The production path, for any real cluster. This page is validated against the current chart: deploy/helm/sreorbit implements the full Addendum v0.2 design (auth, metrics isolation, single-ingress routing, digest-pinning), and every command below was run against this repository's chart while writing this page.

Prerequisites

RequirementVersionWhy
Kubernetes1.30+preStopSleepSeconds (on web by default, so a node drain does not cut a Nocturne answer; on every component in values-prod.yaml) needs PodLifecycleSleepAction, stable since 1.30.
Helm3.x (validated against 3.16.4)values.schema.json validation and --dry-run=server behavior assume Helm 3.
kubectlmatching your cluster's minor versionVerification and rollout-watching commands below.
An ingress controllernginx assumed by values-prod.yaml (ingress.className: nginx); the chart auto-renders SSE-safe annotations for that classSet ingress.className to match the controller you run; see troubleshooting.md if streaming looks buffered.
docker buildx (or crane)ships with modern DockerResolving image digests before a production install; see below.
Images published to a registry you controln/avalues-prod.yaml refuses to render without digests, and a digest can only be resolved once an image exists in a registry. This page assumes ghcr.io/sreorbit/* (or your fork's registry) already has the tag you're installing.

Confirm what you have:

bash
kubectl version --clienthelm version

Before your first production install

Four things, in this order, straight from values-prod.yaml's own header:

  1. Point ingress.host at a real DNS name you control.
  2. Create the TLS secret (or let cert-manager create it via the cert-manager.io/cluster-issuer annotation) and set its name.
  3. Create the credentials Secret out of band (below) and set secrets.existingSecret.
  4. Resolve every component's image to a digest (below).

And two that Addendum v0.5 adds: name a StorageClass for orbit-api's volume (components.orbit-api.persistence.storageClassName, a CHANGEME in values-prod.yaml), and decide how the first admin comes to exist ("The first admin" below).

Every value, one place

The complete environment-variable table (defaults and meanings) is in ../configuration.md; each of those maps to components.<name>.env.<VAR> or components.<name>.secretEnv.<VAR> in values.yaml. Ports: web 3000, orbit-api 8080+9090, nocturne 8081+9090, orbit-stack 8082+9090, fixed and not configurable via values. values.schema.json rejects an unknown key or a typo'd path at install time (additionalProperties: false throughout) rather than letting it silently no-op through a rollout.

Generate your tokens

bash
ORBIT_AUTH_TOKEN=$(openssl rand -hex 32)NOCTURNE_AUTH_TOKEN=$(openssl rand -hex 32)ORBIT_STACK_AUTH_TOKEN=$(openssl rand -hex 32)

Create the credentials Secret

The chart never ships a value: "Secrets are referenced, never embedded in a chart default." Create the Secret yourself, with these four keys (an empty NOCTURNE_API_KEY is valid: it's what NOCTURNE_PROVIDER=mock needs; the optional keys come later on this page):

bash
kubectl create namespace sreorbit --dry-run=client -o yaml | kubectl apply -f - kubectl -n sreorbit create secret generic sreorbit-secrets \  --from-literal=ORBIT_AUTH_TOKEN="$ORBIT_AUTH_TOKEN" \  --from-literal=NOCTURNE_AUTH_TOKEN="$NOCTURNE_AUTH_TOKEN" \  --from-literal=ORBIT_STACK_AUTH_TOKEN="$ORBIT_STACK_AUTH_TOKEN" \  --from-literal=NOCTURNE_API_KEY=""

(The chart renders the namespace itself, with the Pod Security Admission labels, but Helm needs the namespace to exist before it renders anything and only adopts an object that carries its ownership metadata. Create it once, tagged as owned by the release, before the Secret goes in:

bash
kubectl create namespace sreorbitkubectl label namespace sreorbit app.kubernetes.io/managed-by=Helmkubectl annotate namespace sreorbit \  meta.helm.sh/release-name=sreorbit meta.helm.sh/release-namespace=sreorbit

helm upgrade --install then applies the PSA labels from the template on every install and upgrade. A namespace created plainly, or by --create-namespace, makes Helm refuse with "exists and cannot be imported into the current release".)

For a real provider instead of mock, put a real value in NOCTURNE_API_KEY and set components.nocturne.env.NOCTURNE_PROVIDER and components.nocturne.networkPolicy.allowInternetEgress: true at install time (see "Install" below). Both switches are required together; neither implies the other.

Prefer External Secrets, Sealed Secrets, or SOPS over a hand-run kubectl create secret for anything you'll operate long-term. secrets.existingSecret (used below) works the same with any of them, since the chart only references the Secret by name.

Web's own login

web authenticates every request it serves with HTTP Basic: pages, the Deck, every /api/* call including the SSE streams, and static files. Only GET /healthz is exempt: the proxy answers that one itself, 200 {"status":"ok"}, before auth or locale routing run. Without it, any pod that can reach <release>-web:3000 directly (routine wherever the CNI enforces no NetworkPolicy) could read logs, ConfigMaps, env literals, events, inventory and PromQL with no credential; ingress-level basic auth alone never covered that path. See security.md for the full threat model.

components.web.auth.enabled turns it on; the default is true whether ingress.enabled is on or off, since a pod can reach web directly without going through the ingress. Set it to false yourself to opt out.

Generate a password and hash it; the chart and web only ever see the hash:

bash
openssl rand -base64 24 > web-password && chmod 600 web-passwordWEB_AUTH_VERIFIER="$(node apps/web/scripts/hash-password.mjs < web-password)"

hash-password.mjs reads the password from stdin only, never a command-line argument; refuses an interactive terminal; strips one trailing newline; and refuses anything shorter than 15 characters. It prints a PBKDF2 verifier, never the password itself: pbkdf2-sha256$<iterations>$<base64 salt>$<base64 hash>, PBKDF2-HMAC-SHA256 with a 16-byte random salt and a 32-byte hash, at the 600,000 iterations the script writes (the verifier format accepts anything from 210,000 up).

Add WEB_AUTH_USER and WEB_AUTH_VERIFIER to the Secret alongside the four keys above:

bash
kubectl -n sreorbit create secret generic sreorbit-secrets \  --from-literal=ORBIT_AUTH_TOKEN="$ORBIT_AUTH_TOKEN" \  --from-literal=NOCTURNE_AUTH_TOKEN="$NOCTURNE_AUTH_TOKEN" \  --from-literal=ORBIT_STACK_AUTH_TOKEN="$ORBIT_STACK_AUTH_TOKEN" \  --from-literal=NOCTURNE_API_KEY="" \  --from-literal=WEB_AUTH_USER="admin" \  --from-literal=WEB_AUTH_VERIFIER="$WEB_AUTH_VERIFIER"

Both keys reach the container by secretKeyRef with no optional: true, so a missing one is a CreateContainerConfigError, not a silent bypass, and the pod starts by itself once both keys exist. With the chart-created Secret instead of secrets.existingSecret, set secrets.webAuthUser and secrets.webAuthVerifier and the chart writes the keys for you (it only renders them when set, so keys you added later by hand survive the next helm upgrade); values.schema.json rejects a secrets.webAuthVerifier that isn't in the verifier format above. Both are ignored, though, when the release doesn't create the Secret itself (secrets.existingSecret set, or secrets.create=false); NOTES.txt warns about it then.

ingress.basicAuth.enabled and components.web.auth.enabled can't both be on; "Amazon EKS" below explains why the chart refuses to render that combination.

Turning it on for an install that's already running: add the keys before you flip the switch, so the rollout that turns auth on never lands on a pod missing them:

bash
VERIFIER="$(node apps/web/scripts/hash-password.mjs < /path/to/web-password)"kubectl -n sreorbit patch secret sreorbit-secrets --type merge --dry-run=server \  -p "{\"stringData\":{\"WEB_AUTH_USER\":\"admin\",\"WEB_AUTH_VERIFIER\":\"$VERIFIER\"}}"kubectl -n sreorbit patch secret sreorbit-secrets --type merge \  -p "{\"stringData\":{\"WEB_AUTH_USER\":\"admin\",\"WEB_AUTH_VERIFIER\":\"$VERIFIER\"}}"unset VERIFIERhelm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \  --set components.web.auth.enabled=true --set ingress.basicAuth.enabled=false

web's startup, liveness and readiness probes all moved to GET /healthz (off /, which now needs the credential). A plain curl there, with no -u, still tells the probe path apart from everything else.

web's Basic auth guards the whole site. The next section covers who a person is inside the console and what they may do there.

The first admin, and orbit-api's volume

Every page of the Deck needs a signed-in user, and orbit-api creates the first one at its first start (../identity.md → "The first admin"). In production, give it the admin in the Secret, before the install:

bash
openssl rand -base64 24 > admin-password && chmod 600 admin-passwordVERIFIER="$(node apps/web/scripts/hash-password.mjs < admin-password)"kubectl -n sreorbit patch secret sreorbit-secrets --type merge \  -p "{\"stringData\":{\"ORBIT_ADMIN_USER\":\"admin\",\"ORBIT_ADMIN_PASSWORD_VERIFIER\":\"$VERIFIER\",\"ORBIT_SESSION_SECRET\":\"$(openssl rand -base64 48 | tr -d '\n')\"}}"unset VERIFIER

All three keys are optional (secretKeyRef with optional: true), and read once, when the container starts:

  • ORBIT_ADMIN_USER and ORBIT_ADMIN_PASSWORD_VERIFIER create that admin only while no user exists, and are ignored afterwards. Leave both out and orbit-api logs a one-time setup code instead: kubectl -n sreorbit logs deploy/sreorbit-orbit-api | grep setup_code, then open https://YOUR_REAL_HOSTNAME/login.
  • ORBIT_SESSION_SECRET signs the session tokens; without it a key is generated into the data directory. Setting it keeps the key off the volume and out of its backups.

With the chart-created Secret instead of secrets.existingSecret, the same three are secrets.adminUser, secrets.adminPasswordVerifier and secrets.sessionSecret, rendered only when set.

Users, roles, sessions, saved dashboards and the audit trail live under ORBIT_DATA_DIR (/var/lib/sreorbit). components.orbit-api.persistence puts it on a ReadWriteOnce claim, <release>-orbit-api-data, kept on uninstall like the engine's; off (the default in values.yaml), it is an emptyDir sized by emptyDirSizeLimit (256Mi) and everything in it goes with the pod. values-prod.yaml turns it on at 1Gi with a storageClassName you must name, and values-eks.yaml creates an encrypted gp3 class for it, as it does for the engine. The storage keys are the engine's: see "The engine's volume" below. Because the store has a single writer, the chart runs orbit-api as one replica whatever replicaCount says, with a Recreate rollout when persistence is on and no PodDisruptionBudget, and refuses to render with components.orbit-api.autoscaling.enabled. Back the claim up: ../operations.md.

Resolve image digests

values-prod.yaml sets requireDigest: true: the chart refuses to render if any enabled component's image.digest is empty, and names the exact fix in its own error message. Resolve one digest per component, and one per orbit-* engine the default values run:

bash
for c in web orbit-api nocturne orbit-stack; do  echo "$c: $(docker buildx imagetools inspect ghcr.io/sreorbit/$c:0.1.0 | awk '/^Digest:/{print $2}')"doneREPO=ghcr.io/sreorbitfor e in orbit-agent orbit-node orbit-state orbit-collector; do  echo "$e: $(docker buildx imagetools inspect $REPO/$e:0.1.0 | awk '/^Digest:/{print $2}')"done

(crane digest ghcr.io/sreorbit/$c:0.1.0 is the equivalent one-liner if you have crane installed; either tool resolves the same digest, since the digest belongs to the image, not to the tool reading it.)

Put the eight results in a file (an engine switched to its upstream image stops reading its orbit digest, and apm.collector's is read only when the collector is enabled):

bash
cat > digests.yaml <<'EOF'components:  web:    image:      digest: sha256:PASTE_THE_WEB_DIGEST_HERE  orbit-api:    image:      digest: sha256:PASTE_THE_ORBIT_API_DIGEST_HERE  nocturne:    image:      digest: sha256:PASTE_THE_NOCTURNE_DIGEST_HERE  orbit-stack:    image:      digest: sha256:PASTE_THE_ORBIT_STACK_DIGEST_HEREtelemetry:  agent:    orbit:      image:        digest: sha256:PASTE_THE_ORBIT_AGENT_DIGEST_HERE  nodeExporter:    orbit:      image:        digest: sha256:PASTE_THE_ORBIT_NODE_DIGEST_HERE  kubeStateMetrics:    orbit:      image:        digest: sha256:PASTE_THE_ORBIT_STATE_DIGEST_HEREapm:  collector:    orbit:      image:        digest: sha256:PASTE_THE_ORBIT_COLLECTOR_DIGEST_HEREEOF

(deploy/helm/sreorbit/digests.example.yaml is the same file, checked in as syntax reference only: its digests are sha256:0000...0000 and resolve to nothing, so don't install with it verbatim.)

Preview before you install

bash
helm lint deploy/helm/sreorbit -f deploy/helm/sreorbit/values-prod.yaml -f digests.yamlhelm template sreorbit deploy/helm/sreorbit -n sreorbit \  -f deploy/helm/sreorbit/values-prod.yaml -f digests.yaml \  --set ingress.host=YOUR_REAL_HOSTNAME \  --set secrets.existingSecret=sreorbit-secrets \  --set secrets.create=false \  | less

Read it, and confirm: exactly one Ingress rule (web only; a second path: block routing to orbit-api means your values overlay is wrong), every container has both requests and limits, and every image reference ends in @sha256:..., not :0.1.0.

Install

bash
helm upgrade --install sreorbit deploy/helm/sreorbit \  --namespace sreorbit \  -f deploy/helm/sreorbit/values-prod.yaml -f digests.yaml \  --set ingress.host=YOUR_REAL_HOSTNAME \  --set ingress.tls.secretName=YOUR_TLS_SECRET_NAME \  --set secrets.existingSecret=sreorbit-secrets \  --set secrets.create=false \  --timeout 5m --wait

No --create-namespace: namespace.create: true (the default) renders the Namespace itself, carrying the Pod Security Admission labels, and adopts the one you tagged above. --create-namespace would make an untagged one first and Helm would then refuse to adopt it. If the namespace is managed elsewhere, set namespace.create=false and label it pod-security.kubernetes.io/enforce: restricted yourself.

What a healthy install looks like

bash
kubectl -n sreorbit rollout status deploy/sreorbit-web --timeout=180skubectl -n sreorbit rollout status deploy/sreorbit-orbit-api --timeout=180skubectl -n sreorbit rollout status deploy/sreorbit-nocturne --timeout=180skubectl -n sreorbit get pods,svc,ingress -l app.kubernetes.io/instance=sreorbit

Every pod Running and Ready; the Ingress shows your hostname with exactly one address; helm install's own NOTES.txt output (printed automatically, or replay with helm -n sreorbit get notes sreorbit) lists every enabled component with its resolved image reference and replica count, and prints a WARNING if any component has no resource limits, which should not happen.

Verify

bash
kubectl -n sreorbit port-forward svc/sreorbit-orbit-api 8080:8080 &curl -fsS localhost:8080/healthzcurl -fsS localhost:8080/readyzcurl -fsS -H "Authorization: Bearer $ORBIT_AUTH_TOKEN" localhost:8080/api/v1/auth/statekill %1   # stop the port-forward

The last call proves the bearer is right (401 otherwise) and says whether the first admin exists: "setupRequired":false after the Secret's admin was created, true while orbit-api waits for the setup code. Every other route also needs a session; ../identity.md → "Signing in from a shell" shows how to get one.

Then in a browser: https://YOUR_REAL_HOSTNAME/. The marketing site renders once you clear the browser's own Basic-auth prompt with the WEB_AUTH_USER/password pair from above, and /deck sends you to /login, where the admin signs in. If ORBIT_DEMO_DATA is false (the values-prod.yaml default) and you haven't mounted a kubeconfig yet, orbit-api reports NotReady, which is the correct signal and not a bug; see troubleshooting.md.

Telemetry

telemetry in values.yaml is on by default and needs nothing already in the cluster: no Prometheus, kube-state-metrics, node-exporter, ServiceMonitor CRD or monitoring namespace. Four independent switches: telemetry.enabled, telemetry.agent.enabled, telemetry.kubeStateMetrics.enabled, telemetry.nodeExporter.enabled.

ComponentDefault (orbit)Upstream, still selectableWhat it does
Scrape agentorbit-agentquay.io/prometheus/prometheus:v3.15.1 (telemetry.agent.engine: prometheus)Scrapes the cluster, remote-writes to orbit-stack. Nothing else reads it.
kube-state-metricsorbit-stateregistry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 (telemetry.kubeStateMetrics.engine: upstream)Object counts and states: deployments, pods, node capacity.
node-exporterorbit-nodequay.io/prometheus/node-exporter:v1.12.1 (telemetry.nodeExporter.engine: upstream)Host CPU, memory, disk, network; one DaemonSet pod per node.

The default engines are the Rust ones in services/rust/: product images under image.registry, pinned by digest in production like every component (telemetry.<part>.orbit.image). They keep the upstream ones' Services, jobs, labels, RBAC and NetworkPolicies, so switching an engine is one rolling update and no panel changes. The upstream images are multi-arch (amd64 and arm64, so Graviton nodes work) and pinned by digest as well as tag, each with its own telemetry.<part>.image.registry for a private mirror (an ECR pull-through cache keeps the digest; a mirror that re-pushes the image doesn't).

The agent scrapes and remote-writes only, with no query API of its own, as a single-replica Recreate Deployment (a surge pod would double-write the same series under different timestamps). Its WAL lives on an emptyDir sized by telemetry.agent.walSizeLimit (default 1Gi); if orbit-stack stays unreachable, orbit-agent keeps 2h and 90% of that limit at most, oldest first, and Prometheus drops samples older than 4h, rather than growing the WAL without bound. It remote-writes to http://<release>-orbit-stack.<namespace>.svc.cluster.local:8082/api/v1/write, presenting ORBIT_STACK_AUTH_TOKEN as a mounted file when orbit-stack's auth is static, and stamps every series with cluster=<telemetry.clusterName>; an empty value falls back to components.orbit-api.env.ORBIT_CLUSTER_ID, then the release name.

orbit-agent sends only what the product reads. It drops the families in its DEFAULT_DROP (services/rust/orbit-agent/src/config.rs) and cAdvisor's cgroups that are not a container: a pod's, its sandbox's, a system slice's. Each pod's network counters and the node's root cgroup stay. A pod then costs about 65 series instead of 170. telemetry.agent.orbit.keepAllSeries: true sends everything.

orbit-state and orbit-node don't even render the dropped families nothing reads: kube_replicaset_* (orbit-state then doesn't watch ReplicaSets), kube_*_annotations, kube_pod_tolerations and node_ipvs_*. They come back with keepAllSeries, with telemetry.agent.engine: prometheus or with telemetry.agent.enabled: false, where something else may read them: the chart then sets ORBIT_STATE_UNREAD_FAMILIES and ORBIT_NODE_UNREAD_FAMILIES to true.

Scrape jobs, named exactly as orbit-api and the Deck query them (rename one and a panel goes empty):

JobTarget
kubeletEach node's InternalIP:10250, agent's own ServiceAccount token; engine: prometheus only, since nothing queries it
cadvisorSame host, /metrics/cadvisor
kube-state-metricsEndpointSlice discovery in the release namespace
node-exporterPod discovery in <release>-node
sreorbitThe release's own :9090 ports (orbit-api, nocturne, orbit-stack)

kube-state-metrics and node-exporter are dropped from the sreorbit job so nothing is scraped twice. Fargate nodes are skipped by kubelet and cadvisor: a Fargate kubelet answers only through nodes/proxy, which this chart never grants.

TLS on the kubelet/cadvisor jobs: telemetry.agent.kubeletInsecureSkipVerify defaults to false: the agent checks the kubelet's serving certificate against the cluster CA, same as stock EKS nodes (AL2023, Bottlerocket) and k3s already do. Set it true only for node images that present a self-signed kubelet certificate (kubeadm's defaults, some custom AMIs); the symptom otherwise is the kubelet and cadvisor targets down with an x509 error.

RBAC is scoped tight: a ClusterRole gets nodes/metrics (get) and nodes (list/watch) only; namespaced Roles add list/watch on pods, services and endpointslices in the release namespace, and on pods in <release>-node. The chart never grants nodes/proxy, which would reach every kubelet endpoint, exec included, through the API server. kube-state-metrics' own ClusterRole is the standard set minus Secrets (the secrets collector is off), with no tokenreviews or subjectaccessreviews. telemetry.kubeStateMetrics.metricLabelsAllowlist exports the node capacity-type labels the Deck's spot panel reads.

node-exporter runs as a DaemonSet in its own chart-created namespace, <release>-node (Pod Security enforce=privileged), because it mounts host paths. orbit-node, the default engine, stays on the pod network and in its own PID namespace, and mounts six host paths read-only under /host/root: /proc/1/net (the node's network, through its init), /proc/1/mountinfo, /sys/class/net and the /sys/devices it links to, /run/udev/data and /etc/os-release, plus /var/log/pods with telemetry.logs and the directory of each telemetry.logs.files source (logs.md). Its own /proc and /sys give the kernel-wide counters. node-exporter (engine: upstream) needs hostPID, hostNetwork and the node's / at /host/root instead. orbit-node listens on its pod IP, upstream on the node's own IP, both on telemetry.nodeExporter.port (default 9111, never 9100: the chart refuses that port so upstream can't collide with a node-exporter the cluster already runs; two releases on one cluster need two ports). Its filesystem collector excludes the usual virtual filesystems plus the container runtimes' own mounts (run/containerd, var/lib/containers/storage: containerd on EKS and Bottlerocket) via telemetry.nodeExporter.mountPointsExclude. It runs runAsNonRoot (65534), readOnlyRootFilesystem, every capability dropped, seccomp RuntimeDefault. It needs no SELinux options, even on Bottlerocket, since the default container_t label already covers what the container reads. It tolerates every taint by default, and a node affinity keeps it off Fargate, where DaemonSets don't schedule.

telemetry.nodeExporter.engine: orbit, the default, runs orbit-node (the Rust exporter, services/rust/orbit-node) in the same DaemonSet, with the same port, mounts, excludes and labels: node-exporter's families, label values and units, minus the Go runtime's go_*, promhttp_* and node_exporter_build_info, at about a third of the memory and CPU. Its image is telemetry.nodeExporter.orbit.image, digest-pinned like the product images. upstream runs node-exporter itself.

When networkPolicy.enabled, the agent may reach orbit-stack on 8082, kube-state-metrics on 8080, every component's 9090, and the node CIDRs (telemetry.agent.networkPolicy.nodeCIDRs, default the private ranges) on 10250 and the node-exporter port, and nothing more. Enforcement depends on the CNI: k3s (and so k3d) enforces it through its bundled kube-router controller; flannel alone does not; Calico and Cilium do; EKS only once the VPC CNI's network policy agent is turned on. Where nothing enforces them, the objects apply and block nothing, and the install still works. One Cilium caveat: an ipBlock never matches a node address there unless Cilium's policy-cidr-match-mode is nodes, so set it or the kubelet, cAdvisor and node-exporter scrapes are dropped. networkPolicy.apiServerCIDRs is empty by default; on an enforcing CNI that leaves the agent unable to discover scrape targets and kube-state-metrics unable to list anything, and NOTES.txt warns about it after install.

Confirm the engine is receiving samples:

bash
kubectl -n sreorbit port-forward svc/sreorbit-orbit-stack 8082:8082 &curl -fsS localhost:8082/api/v1/query --data-urlencode 'query=count by (job) ({__name__=~".+"})' \  -H "Authorization: Bearer $ORBIT_STACK_AUTH_TOKEN"

(the Authorization header only when orbit-stack's auth is static). Every job above should show a non-zero count.

Upgrade

Same command as install (helm upgrade --install is idempotent), with fresh digests for whatever you published:

bash
helm upgrade --install sreorbit deploy/helm/sreorbit \  --namespace sreorbit \  -f deploy/helm/sreorbit/values-prod.yaml -f digests.yaml \  --set ingress.host=YOUR_REAL_HOSTNAME \  --set ingress.tls.secretName=YOUR_TLS_SECRET_NAME \  --set secrets.existingSecret=sreorbit-secrets \  --set secrets.create=false \  --timeout 5m --wait

maxUnavailable: 0, maxSurge: 1 on every multi-replica Deployment means capacity never dips during the rollout: a new pod comes up and passes readiness before an old one is removed. The two components with a volume, orbit-api (with persistence on) and orbit-stack, run one replica with Recreate: the old pod stops before the new one mounts the claim, so their rollout is a short gap, and nothing on the volume is lost.

Roll back

bash
helm -n sreorbit history sreorbithelm -n sreorbit rollback sreorbit

This is byte-identical to what you tested because you installed with digests: helm rollback re-applies the recorded release, and a digest can't drift to different bytes the way a tag can. If a rollback does not match what you tested, see troubleshooting.md; on a bare-tag install, suspect the tag first.

Uninstall cleanly

bash
helm -n sreorbit uninstall sreorbit

The credentials Secret carries helm.sh/resource-policy: keep when the chart creates it (not when you used secrets.existingSecret, which Helm never owned). It survives uninstall on purpose, so a reinstall doesn't regenerate credentials you still rely on elsewhere. Remove it yourself when you are done:

bash
kubectl -n sreorbit delete secret sreorbit-secrets   # only if the chart created itkubectl delete namespace sreorbit                     # removes everything else

Traces and the demo shop

The engine takes OpenTelemetry traces on the same port and bearer as remote-write, at http://<release>-orbit-stack.<namespace>.svc.cluster.local:8082/v1/traces; with networkPolicy.enabled, add your Collector's or your applications' namespace to components.orbit-stack.networkPolicy.allowFromNamespaces first. ../apm.md has the exporter configuration.

To see APM with traffic in it before your own services send any, turn on the demo shop: eight small services and a load generator from the orbit-stack image, one Deployment each, sending traces to the engine with its bearer:

bash
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \  --set demo.shop.enabled=true

demo.shop.rps sets the load (default 5 requests a second) and demo.shop.resources the pods' requests and limits; --set demo.shop.enabled=false removes it all again. It needs components.orbit-stack.enabled, and is off by default because it is load you did not ask for.

The engine's volume

components.orbit-stack.persistence (on by default, 20Gi) renders a PersistentVolumeClaim named <release>-orbit-stack-data with helm.sh/resource-policy: keep, so helm uninstall leaves the telemetry behind for you to delete deliberately. An empty storageClassName takes the cluster's default class, except on EKS (a server version carrying -eks-) or with requireStorageClass: true, where the render refuses rather than leave a claim waiting for a default that may not exist: name an existing storageClassName, or set storageClass.create: true to have the chart provision one (see "Amazon EKS" below for the EBS specifics). A claim that already exists keeps the class it was bound with, since that field is immutable. The engine runs one replica with a Recreate strategy because the claim is ReadWriteOnce. The claim also holds it to one availability zone, so the agent, the OTLP collector and orbit-api prefer that zone (a preferred podAffinity of weight 100 on its pods): their traffic to orbit-stack stays off the cross-zone bill, and a full zone still schedules them elsewhere. A component's own affinity replaces that preference (for orbit-api, the global one too). Size it for ORBIT_STACK_RETENTION at your bytes per sample (../benchmark.md measured 1.7 B on a scrape-shaped workload) plus one block duration of write-ahead log. Point any agent of your own at http://<release>-orbit-stack.<namespace>.svc.cluster.local:8082/api/v1/write with the bearer above, as the chart's own agent (see "Telemetry" above) does; ../orbit-stack.md has the full configuration.

Amazon EKS

deploy/helm/sreorbit/values-eks.yaml is the EKS overlay: the whole product on any EKS cluster, on top of the chart's own defaults (values.yaml). The cluster only needs to supply an ingress controller and the Amazon EBS CSI driver add-on (with its IAM role) for orbit-stack's volume; nothing here assumes a public API endpoint, since the chart talks to the cluster from inside it.

bash
helm upgrade --install sreorbit deploy/helm/sreorbit -n sreorbit \  -f deploy/helm/sreorbit/values-eks.yaml \  --set-string secrets.orbitAuthToken="$(openssl rand -hex 32)" \  --set-string secrets.nocturneAuthToken="$(openssl rand -hex 32)" \  --set-string secrets.orbitStackAuthToken="$(openssl rand -hex 32)"

Tokens passed that way rotate on the next such run; for production, manage a Secret yourself and set secrets.existingSecret instead, same as above. Without an admin in the Secret, this install's orbit-api logs a one-time setup code for the first sign-in ("The first admin, and orbit-api's volume" above).

values-eks.yaml sets components.web.auth.enabled to true, so this overlay turns on web's own Basic auth, and pairs it with ingress.basicAuth.enabled: false. The quickstart above doesn't set WEB_AUTH_USER or WEB_AUTH_VERIFIER; add both to the release Secret (sreorbit-secrets, or whatever secrets.existingSecret names; see "Web's own login" above) before web can start, or the pod sits in CreateContainerConfigError. values-eks.yaml also caps requests at the edge with nginx.ingress.kubernetes.io/limit-rps: "20" and nginx.ingress.kubernetes.io/limit-connections: "20"; raise both when many operators share one NAT address.

ingress.className is a CHANGEME (kubectl get ingressclass; many clusters have no default class). The overlay also sets ingress.controller (nginx/traefik/haproxy/alb/none; empty derives it from className), which picks the SSE-safe annotations and gates basic auth, so a class named e.g. nginx-internal still gets nginx's annotations. TLS is optional. ingress.basicAuth.{enabled,secretName,realm} puts a password on the whole console via ingress-nginx annotations, reading an htpasswd file from a Secret keyed auth; the render fails if basic auth is on and the controller isn't nginx.

Turning ingress.basicAuth.enabled on alongside components.web.auth.enabled is the one combination the chart refuses to render at all: ingress-nginx strips the Authorization header once its own basic auth has accepted it, so web would never see a credential to check, and would reject every request on its own instead. Pick one layer, not both.

orbit-stack's persistence, and orbit-api's, assume no default StorageClass exists: storageClass.create: true provisions <release>-orbit-stack and <release>-orbit-api themselves (ebs.csi.aws.com, gp3, encrypted, WaitForFirstConsumer, allowVolumeExpansion, Retain; every field on it is immutable once created). Under EKS Auto Mode, EBS is provisioned through its own driver: set storageClass.provisioner to ebs.csi.eks.amazonaws.com there instead. requireStorageClass: true fails the render if neither storageClassName nor storageClass.create is set. A PVC stuck Pending means the CSI driver or its IAM role is missing, and NOTES.txt says so.

networkPolicy.apiServerCIDRs needs the same care: on EKS, kubernetes.default is the first address of the cluster's service CIDR, and its endpoints are the control plane's ENIs on 443. Name both: the VPC CNI's network policy agent judges the Service IP itself, before kube-proxy's rewrite, while Calico and Cilium judge the endpoints instead:

bash
kubectl get svc kubernetes -n defaultkubectl get endpointslices -n default -l kubernetes.io/service-name=kubernetes

Nocturne stays on the mock provider; the write path is off (rbac.write.enabled: false, ORBIT_WRITE_ENABLED: "false"); ORBIT_DEMO_DATA: "false" with orbit-api's rbac.enabled: true gives a read-only live console of the real cluster instead. image.registry and telemetry.*.image.registry are both overridable for a private ECR mirror.

Where to go next