Install
Helm
The production path, for any real cluster. This page is validated against the
current chart: deploy/helm/sreorbit implements the full Addendum v0.2 design
(auth, metrics isolation, single-ingress routing, digest-pinning), and every command
below was run against this repository's chart while writing this page.
Prerequisites
| Requirement | Version | Why |
|---|---|---|
| Kubernetes | 1.30+ | preStopSleepSeconds (on web by default, so a node drain does not cut a Nocturne answer; on every component in values-prod.yaml) needs PodLifecycleSleepAction, stable since 1.30. |
| Helm | 3.x (validated against 3.16.4) | values.schema.json validation and --dry-run=server behavior assume Helm 3. |
kubectl | matching your cluster's minor version | Verification and rollout-watching commands below. |
| An ingress controller | nginx assumed by values-prod.yaml (ingress.className: nginx); the chart auto-renders SSE-safe annotations for that class | Set ingress.className to match the controller you run; see troubleshooting.md if streaming looks buffered. |
docker buildx (or crane) | ships with modern Docker | Resolving image digests before a production install; see below. |
| Images published to a registry you control | n/a | values-prod.yaml refuses to render without digests, and a digest can only be resolved once an image exists in a registry. This page assumes ghcr.io/sreorbit/* (or your fork's registry) already has the tag you're installing. |
Confirm what you have:
kubectl version --clienthelm versionBefore your first production install
Four things, in this order, straight from values-prod.yaml's own header:
- Point
ingress.hostat a real DNS name you control. - Create the TLS secret (or let cert-manager create it via the
cert-manager.io/cluster-issuerannotation) and set its name. - Create the credentials Secret out of band (below) and set
secrets.existingSecret. - Resolve every component's image to a digest (below).
And two that Addendum v0.5 adds: name a StorageClass for orbit-api's volume
(components.orbit-api.persistence.storageClassName, a CHANGEME in values-prod.yaml),
and decide how the first admin comes to exist ("The first admin" below).
Every value, one place
The complete environment-variable table (defaults and meanings) is in
../configuration.md; each of those maps to
components.<name>.env.<VAR> or components.<name>.secretEnv.<VAR> in
values.yaml. Ports: web 3000, orbit-api 8080+9090, nocturne
8081+9090, orbit-stack 8082+9090, fixed and not configurable via values. values.schema.json rejects an
unknown key or a typo'd path at install time (additionalProperties: false
throughout) rather than letting it silently no-op through a rollout.
Generate your tokens
ORBIT_AUTH_TOKEN=$(openssl rand -hex 32)NOCTURNE_AUTH_TOKEN=$(openssl rand -hex 32)ORBIT_STACK_AUTH_TOKEN=$(openssl rand -hex 32)Create the credentials Secret
The chart never ships a value: "Secrets are referenced, never embedded in a chart
default." Create the Secret yourself, with these four keys (an empty
NOCTURNE_API_KEY is valid: it's what NOCTURNE_PROVIDER=mock needs; the optional keys
come later on this page):
kubectl create namespace sreorbit --dry-run=client -o yaml | kubectl apply -f - kubectl -n sreorbit create secret generic sreorbit-secrets \ --from-literal=ORBIT_AUTH_TOKEN="$ORBIT_AUTH_TOKEN" \ --from-literal=NOCTURNE_AUTH_TOKEN="$NOCTURNE_AUTH_TOKEN" \ --from-literal=ORBIT_STACK_AUTH_TOKEN="$ORBIT_STACK_AUTH_TOKEN" \ --from-literal=NOCTURNE_API_KEY=""(The chart renders the namespace itself, with the Pod Security Admission labels, but Helm needs the namespace to exist before it renders anything and only adopts an object that carries its ownership metadata. Create it once, tagged as owned by the release, before the Secret goes in:
kubectl create namespace sreorbitkubectl label namespace sreorbit app.kubernetes.io/managed-by=Helmkubectl annotate namespace sreorbit \ meta.helm.sh/release-name=sreorbit meta.helm.sh/release-namespace=sreorbithelm upgrade --install then applies the PSA labels from the template on every install
and upgrade. A namespace created plainly, or by --create-namespace, makes Helm refuse
with "exists and cannot be imported into the current release".)
For a real provider instead of mock, put a real value in NOCTURNE_API_KEY and
set components.nocturne.env.NOCTURNE_PROVIDER and
components.nocturne.networkPolicy.allowInternetEgress: true at install time (see
"Install" below). Both switches are required together; neither implies the other.
Prefer External Secrets, Sealed Secrets, or SOPS over a hand-run kubectl create secret for anything you'll operate long-term. secrets.existingSecret (used
below) works the same with any of them, since the chart only references the
Secret by name.
Web's own login
web authenticates every request it serves with HTTP Basic: pages, the
Deck, every /api/* call including the SSE streams, and static files. Only
GET /healthz is exempt: the proxy answers that one itself,
200 {"status":"ok"}, before auth or locale routing run. Without it,
any pod that can reach <release>-web:3000 directly (routine wherever the
CNI enforces no NetworkPolicy) could read logs, ConfigMaps, env literals,
events, inventory and PromQL with no credential; ingress-level basic
auth alone never covered that path. See security.md for the full threat
model.
components.web.auth.enabled turns it on; the default is true whether
ingress.enabled is on or off, since a pod can reach web directly without
going through the ingress. Set it to false yourself to opt out.
Generate a password and hash it; the chart and web only ever see the
hash:
openssl rand -base64 24 > web-password && chmod 600 web-passwordWEB_AUTH_VERIFIER="$(node apps/web/scripts/hash-password.mjs < web-password)"hash-password.mjs reads the password from stdin only, never a
command-line argument; refuses an interactive terminal; strips one trailing
newline; and refuses anything shorter than 15 characters. It prints a PBKDF2
verifier, never the password itself:
pbkdf2-sha256$<iterations>$<base64 salt>$<base64 hash>, PBKDF2-HMAC-SHA256
with a 16-byte random salt and a 32-byte hash, at the 600,000 iterations the
script writes (the verifier format accepts anything from 210,000 up).
Add WEB_AUTH_USER and WEB_AUTH_VERIFIER to the Secret alongside the four
keys above:
kubectl -n sreorbit create secret generic sreorbit-secrets \ --from-literal=ORBIT_AUTH_TOKEN="$ORBIT_AUTH_TOKEN" \ --from-literal=NOCTURNE_AUTH_TOKEN="$NOCTURNE_AUTH_TOKEN" \ --from-literal=ORBIT_STACK_AUTH_TOKEN="$ORBIT_STACK_AUTH_TOKEN" \ --from-literal=NOCTURNE_API_KEY="" \ --from-literal=WEB_AUTH_USER="admin" \ --from-literal=WEB_AUTH_VERIFIER="$WEB_AUTH_VERIFIER"Both keys reach the container by secretKeyRef with no optional: true, so
a missing one is a CreateContainerConfigError, not a silent bypass, and the
pod starts by itself once both keys exist. With the chart-created Secret
instead of secrets.existingSecret, set secrets.webAuthUser and
secrets.webAuthVerifier and the chart writes the keys for you (it only
renders them when set, so keys you added later by hand survive the next
helm upgrade); values.schema.json rejects a secrets.webAuthVerifier
that isn't in the verifier format above. Both are ignored, though, when the
release doesn't create the Secret itself (secrets.existingSecret set, or
secrets.create=false); NOTES.txt warns about it then.
ingress.basicAuth.enabled and components.web.auth.enabled can't both be
on; "Amazon EKS" below explains why the chart refuses to render that
combination.
Turning it on for an install that's already running: add the keys before you flip the switch, so the rollout that turns auth on never lands on a pod missing them:
VERIFIER="$(node apps/web/scripts/hash-password.mjs < /path/to/web-password)"kubectl -n sreorbit patch secret sreorbit-secrets --type merge --dry-run=server \ -p "{\"stringData\":{\"WEB_AUTH_USER\":\"admin\",\"WEB_AUTH_VERIFIER\":\"$VERIFIER\"}}"kubectl -n sreorbit patch secret sreorbit-secrets --type merge \ -p "{\"stringData\":{\"WEB_AUTH_USER\":\"admin\",\"WEB_AUTH_VERIFIER\":\"$VERIFIER\"}}"unset VERIFIERhelm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \ --set components.web.auth.enabled=true --set ingress.basicAuth.enabled=falseweb's startup, liveness and readiness probes all moved to GET /healthz
(off /, which now needs the credential). A plain curl there, with no -u,
still tells the probe path apart from everything else.
web's Basic auth guards the whole site. The next section covers who a person is
inside the console and what they may do there.
The first admin, and orbit-api's volume
Every page of the Deck needs a signed-in user, and orbit-api creates the first one
at its first start (../identity.md → "The first admin"). In
production, give it the admin in the Secret, before the install:
openssl rand -base64 24 > admin-password && chmod 600 admin-passwordVERIFIER="$(node apps/web/scripts/hash-password.mjs < admin-password)"kubectl -n sreorbit patch secret sreorbit-secrets --type merge \ -p "{\"stringData\":{\"ORBIT_ADMIN_USER\":\"admin\",\"ORBIT_ADMIN_PASSWORD_VERIFIER\":\"$VERIFIER\",\"ORBIT_SESSION_SECRET\":\"$(openssl rand -base64 48 | tr -d '\n')\"}}"unset VERIFIERAll three keys are optional (secretKeyRef with optional: true), and read once, when
the container starts:
ORBIT_ADMIN_USERandORBIT_ADMIN_PASSWORD_VERIFIERcreate that admin only while no user exists, and are ignored afterwards. Leave both out andorbit-apilogs a one-time setup code instead:kubectl -n sreorbit logs deploy/sreorbit-orbit-api | grep setup_code, then openhttps://YOUR_REAL_HOSTNAME/login.ORBIT_SESSION_SECRETsigns the session tokens; without it a key is generated into the data directory. Setting it keeps the key off the volume and out of its backups.
With the chart-created Secret instead of secrets.existingSecret, the same three are
secrets.adminUser, secrets.adminPasswordVerifier and secrets.sessionSecret, rendered
only when set.
Users, roles, sessions, saved dashboards and the audit trail live under
ORBIT_DATA_DIR (/var/lib/sreorbit). components.orbit-api.persistence puts it on a
ReadWriteOnce claim, <release>-orbit-api-data, kept on uninstall like the engine's;
off (the default in values.yaml), it is an emptyDir sized by emptyDirSizeLimit
(256Mi) and everything in it goes with the pod. values-prod.yaml turns it on at
1Gi with a storageClassName you must name, and values-eks.yaml creates an encrypted
gp3 class for it, as it does for the engine. The storage keys are the engine's: see "The
engine's volume" below. Because the store has a single writer, the chart runs orbit-api
as one replica whatever replicaCount says, with a Recreate rollout when persistence is
on and no PodDisruptionBudget, and refuses to render with
components.orbit-api.autoscaling.enabled. Back the claim up: ../operations.md.
Resolve image digests
values-prod.yaml sets requireDigest: true: the chart refuses to render if
any enabled component's image.digest is empty, and names the exact fix in its own
error message. Resolve one digest per component, and one per orbit-* engine the
default values run:
for c in web orbit-api nocturne orbit-stack; do echo "$c: $(docker buildx imagetools inspect ghcr.io/sreorbit/$c:0.1.0 | awk '/^Digest:/{print $2}')"doneREPO=ghcr.io/sreorbitfor e in orbit-agent orbit-node orbit-state orbit-collector; do echo "$e: $(docker buildx imagetools inspect $REPO/$e:0.1.0 | awk '/^Digest:/{print $2}')"done(crane digest ghcr.io/sreorbit/$c:0.1.0 is the equivalent one-liner if you have
crane installed; either tool resolves the same digest, since the digest belongs to
the image, not to the tool reading it.)
Put the eight results in a file (an engine switched to its upstream image stops reading
its orbit digest, and apm.collector's is read only when the collector is enabled):
cat > digests.yaml <<'EOF'components: web: image: digest: sha256:PASTE_THE_WEB_DIGEST_HERE orbit-api: image: digest: sha256:PASTE_THE_ORBIT_API_DIGEST_HERE nocturne: image: digest: sha256:PASTE_THE_NOCTURNE_DIGEST_HERE orbit-stack: image: digest: sha256:PASTE_THE_ORBIT_STACK_DIGEST_HEREtelemetry: agent: orbit: image: digest: sha256:PASTE_THE_ORBIT_AGENT_DIGEST_HERE nodeExporter: orbit: image: digest: sha256:PASTE_THE_ORBIT_NODE_DIGEST_HERE kubeStateMetrics: orbit: image: digest: sha256:PASTE_THE_ORBIT_STATE_DIGEST_HEREapm: collector: orbit: image: digest: sha256:PASTE_THE_ORBIT_COLLECTOR_DIGEST_HEREEOF(deploy/helm/sreorbit/digests.example.yaml is the same file, checked in as syntax
reference only: its digests are sha256:0000...0000 and resolve to nothing, so
don't install with it verbatim.)
Preview before you install
helm lint deploy/helm/sreorbit -f deploy/helm/sreorbit/values-prod.yaml -f digests.yamlhelm template sreorbit deploy/helm/sreorbit -n sreorbit \ -f deploy/helm/sreorbit/values-prod.yaml -f digests.yaml \ --set ingress.host=YOUR_REAL_HOSTNAME \ --set secrets.existingSecret=sreorbit-secrets \ --set secrets.create=false \ | lessRead it, and confirm: exactly one Ingress rule (web only; a second path: block
routing to orbit-api means your values overlay is wrong), every container has both requests and limits, and every image
reference ends in @sha256:..., not :0.1.0.
Install
helm upgrade --install sreorbit deploy/helm/sreorbit \ --namespace sreorbit \ -f deploy/helm/sreorbit/values-prod.yaml -f digests.yaml \ --set ingress.host=YOUR_REAL_HOSTNAME \ --set ingress.tls.secretName=YOUR_TLS_SECRET_NAME \ --set secrets.existingSecret=sreorbit-secrets \ --set secrets.create=false \ --timeout 5m --waitNo --create-namespace: namespace.create: true (the default) renders the
Namespace itself, carrying the Pod Security Admission labels, and adopts the one you
tagged above. --create-namespace would make an untagged one first and Helm would then
refuse to adopt it. If the namespace is managed elsewhere, set namespace.create=false
and label it pod-security.kubernetes.io/enforce: restricted yourself.
What a healthy install looks like
kubectl -n sreorbit rollout status deploy/sreorbit-web --timeout=180skubectl -n sreorbit rollout status deploy/sreorbit-orbit-api --timeout=180skubectl -n sreorbit rollout status deploy/sreorbit-nocturne --timeout=180skubectl -n sreorbit get pods,svc,ingress -l app.kubernetes.io/instance=sreorbitEvery pod Running and Ready; the Ingress shows your hostname with exactly one
address; helm install's own NOTES.txt output (printed automatically, or replay
with helm -n sreorbit get notes sreorbit) lists every enabled component with its
resolved image reference and replica count, and prints a WARNING if any component
has no resource limits, which should not happen.
Verify
kubectl -n sreorbit port-forward svc/sreorbit-orbit-api 8080:8080 &curl -fsS localhost:8080/healthzcurl -fsS localhost:8080/readyzcurl -fsS -H "Authorization: Bearer $ORBIT_AUTH_TOKEN" localhost:8080/api/v1/auth/statekill %1 # stop the port-forwardThe last call proves the bearer is right (401 otherwise) and says whether the first admin
exists: "setupRequired":false after the Secret's admin was created, true while
orbit-api waits for the setup code. Every other route also needs a session;
../identity.md → "Signing in from a shell" shows how to get one.
Then in a browser: https://YOUR_REAL_HOSTNAME/. The marketing site renders once you
clear the browser's own Basic-auth prompt with the WEB_AUTH_USER/password pair from
above, and /deck sends you to /login, where the admin signs in. If ORBIT_DEMO_DATA is
false (the values-prod.yaml default) and you haven't mounted a kubeconfig yet,
orbit-api reports NotReady, which is the correct signal and not a bug; see
troubleshooting.md.
Telemetry
telemetry in values.yaml is on by default and needs nothing already in the
cluster: no Prometheus, kube-state-metrics, node-exporter, ServiceMonitor CRD
or monitoring namespace. Four independent switches: telemetry.enabled,
telemetry.agent.enabled, telemetry.kubeStateMetrics.enabled,
telemetry.nodeExporter.enabled.
| Component | Default (orbit) | Upstream, still selectable | What it does |
|---|---|---|---|
| Scrape agent | orbit-agent | quay.io/prometheus/prometheus:v3.15.1 (telemetry.agent.engine: prometheus) | Scrapes the cluster, remote-writes to orbit-stack. Nothing else reads it. |
| kube-state-metrics | orbit-state | registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.20.0 (telemetry.kubeStateMetrics.engine: upstream) | Object counts and states: deployments, pods, node capacity. |
| node-exporter | orbit-node | quay.io/prometheus/node-exporter:v1.12.1 (telemetry.nodeExporter.engine: upstream) | Host CPU, memory, disk, network; one DaemonSet pod per node. |
The default engines are the Rust ones in services/rust/: product images under
image.registry, pinned by digest in production like every component
(telemetry.<part>.orbit.image). They keep the upstream ones' Services, jobs, labels,
RBAC and NetworkPolicies, so switching an engine is one rolling update and no panel
changes. The upstream images are multi-arch (amd64 and arm64, so Graviton nodes work)
and pinned by digest as well as tag, each with its own telemetry.<part>.image.registry
for a private mirror (an ECR pull-through cache keeps the digest; a mirror
that re-pushes the image doesn't).
The agent scrapes and remote-writes only, with no query API
of its own, as a single-replica Recreate Deployment (a surge pod would
double-write the same series under different timestamps). Its WAL lives on an
emptyDir sized by telemetry.agent.walSizeLimit (default 1Gi); if
orbit-stack stays unreachable, orbit-agent keeps 2h and 90% of that limit at most,
oldest first, and Prometheus drops samples older than
4h, rather than growing the WAL without bound. It remote-writes to
http://<release>-orbit-stack.<namespace>.svc.cluster.local:8082/api/v1/write,
presenting ORBIT_STACK_AUTH_TOKEN as a mounted file when orbit-stack's auth
is static, and stamps every series with cluster=<telemetry.clusterName>;
an empty value falls back to components.orbit-api.env.ORBIT_CLUSTER_ID, then the
release name.
orbit-agent sends only what the product reads. It drops the families in its
DEFAULT_DROP (services/rust/orbit-agent/src/config.rs) and cAdvisor's cgroups that are
not a container: a pod's, its sandbox's, a system slice's. Each pod's network counters and
the node's root cgroup stay. A pod then costs about 65 series instead of 170.
telemetry.agent.orbit.keepAllSeries: true sends everything.
orbit-state and orbit-node don't even render the dropped families nothing reads:
kube_replicaset_* (orbit-state then doesn't watch ReplicaSets), kube_*_annotations,
kube_pod_tolerations and node_ipvs_*. They come back with keepAllSeries, with
telemetry.agent.engine: prometheus or with telemetry.agent.enabled: false, where something
else may read them: the chart then sets ORBIT_STATE_UNREAD_FAMILIES and
ORBIT_NODE_UNREAD_FAMILIES to true.
Scrape jobs, named exactly as orbit-api and the Deck query them (rename one and a panel goes empty):
| Job | Target |
|---|---|
kubelet | Each node's InternalIP:10250, agent's own ServiceAccount token; engine: prometheus only, since nothing queries it |
cadvisor | Same host, /metrics/cadvisor |
kube-state-metrics | EndpointSlice discovery in the release namespace |
node-exporter | Pod discovery in <release>-node |
sreorbit | The release's own :9090 ports (orbit-api, nocturne, orbit-stack) |
kube-state-metrics and node-exporter are dropped from the sreorbit job so
nothing is scraped twice. Fargate nodes are skipped by kubelet and
cadvisor: a Fargate kubelet answers only through nodes/proxy, which this
chart never grants.
TLS on the kubelet/cadvisor jobs: telemetry.agent.kubeletInsecureSkipVerify
defaults to false: the agent checks the kubelet's serving certificate
against the cluster CA, same as stock EKS nodes (AL2023, Bottlerocket) and k3s
already do. Set it true only for node images that present a self-signed
kubelet certificate (kubeadm's defaults, some custom AMIs); the symptom
otherwise is the kubelet and cadvisor targets down with an x509 error.
RBAC is scoped tight: a ClusterRole gets nodes/metrics (get) and nodes
(list/watch) only; namespaced Roles add list/watch on pods, services
and endpointslices in the release namespace, and on pods in <release>-node.
The chart never grants nodes/proxy, which would reach every kubelet endpoint,
exec included, through the API server. kube-state-metrics' own ClusterRole is the standard
set minus Secrets (the secrets collector is off), with no tokenreviews or
subjectaccessreviews. telemetry.kubeStateMetrics.metricLabelsAllowlist
exports the node capacity-type labels the Deck's spot panel reads.
node-exporter runs as a DaemonSet in its own chart-created namespace,
<release>-node (Pod Security enforce=privileged), because it mounts host
paths. orbit-node, the default engine, stays on the pod network and in its own
PID namespace, and mounts six host paths read-only under /host/root:
/proc/1/net (the node's network, through its init), /proc/1/mountinfo,
/sys/class/net and the /sys/devices it links to, /run/udev/data and
/etc/os-release, plus /var/log/pods with telemetry.logs and the directory of each
telemetry.logs.files source (logs.md). Its own /proc
and /sys give the kernel-wide counters. node-exporter (engine: upstream)
needs hostPID, hostNetwork and the node's / at /host/root instead.
orbit-node listens on its pod IP, upstream on the node's own IP, both on
telemetry.nodeExporter.port (default 9111, never 9100: the chart
refuses that port so upstream can't collide with a node-exporter the cluster
already runs; two releases on one cluster need two ports). Its filesystem collector excludes the usual virtual
filesystems plus the container runtimes' own mounts (run/containerd,
var/lib/containers/storage: containerd on EKS and Bottlerocket) via
telemetry.nodeExporter.mountPointsExclude. It runs runAsNonRoot (65534),
readOnlyRootFilesystem, every capability dropped, seccomp RuntimeDefault.
It needs no SELinux options, even on Bottlerocket, since the default
container_t label already covers what the container reads. It tolerates
every taint by default, and a node affinity keeps it off Fargate, where
DaemonSets don't schedule.
telemetry.nodeExporter.engine: orbit, the default, runs orbit-node (the Rust exporter,
services/rust/orbit-node) in the same DaemonSet, with the same port, mounts,
excludes and labels: node-exporter's families, label values and units, minus
the Go runtime's go_*, promhttp_* and node_exporter_build_info, at about
a third of the memory and CPU. Its image is telemetry.nodeExporter.orbit.image,
digest-pinned like the product images. upstream runs node-exporter itself.
When networkPolicy.enabled, the agent may reach orbit-stack on 8082,
kube-state-metrics on 8080, every component's 9090, and the node CIDRs
(telemetry.agent.networkPolicy.nodeCIDRs, default the private ranges) on
10250 and the node-exporter port, and nothing more. Enforcement depends
on the CNI: k3s (and so k3d) enforces it through its bundled
kube-router controller; flannel alone does not; Calico and Cilium do; EKS
only once the VPC CNI's network policy agent is turned on. Where nothing
enforces them, the objects apply and block nothing, and the install still works.
One Cilium caveat: an ipBlock never matches a node address there unless
Cilium's policy-cidr-match-mode is nodes, so set it or the kubelet,
cAdvisor and node-exporter scrapes are dropped.
networkPolicy.apiServerCIDRs is empty by default; on an enforcing CNI that
leaves the agent unable to discover scrape targets and kube-state-metrics
unable to list anything, and NOTES.txt warns about it after install.
Confirm the engine is receiving samples:
kubectl -n sreorbit port-forward svc/sreorbit-orbit-stack 8082:8082 &curl -fsS localhost:8082/api/v1/query --data-urlencode 'query=count by (job) ({__name__=~".+"})' \ -H "Authorization: Bearer $ORBIT_STACK_AUTH_TOKEN"(the Authorization header only when orbit-stack's auth is static). Every
job above should show a non-zero count.
Upgrade
Same command as install (helm upgrade --install is idempotent), with fresh
digests for whatever you published:
helm upgrade --install sreorbit deploy/helm/sreorbit \ --namespace sreorbit \ -f deploy/helm/sreorbit/values-prod.yaml -f digests.yaml \ --set ingress.host=YOUR_REAL_HOSTNAME \ --set ingress.tls.secretName=YOUR_TLS_SECRET_NAME \ --set secrets.existingSecret=sreorbit-secrets \ --set secrets.create=false \ --timeout 5m --waitmaxUnavailable: 0, maxSurge: 1 on every multi-replica Deployment means capacity never
dips during the rollout: a new pod comes up and passes readiness before an old one is
removed. The two components with a volume, orbit-api (with persistence on) and
orbit-stack, run one replica with Recreate: the old pod stops before the new one mounts
the claim, so their rollout is a short gap, and nothing on the volume is lost.
Roll back
helm -n sreorbit history sreorbithelm -n sreorbit rollback sreorbitThis is byte-identical to what you tested because you installed with digests:
helm rollback re-applies the recorded release, and a digest can't drift to
different bytes the way a tag can. If a rollback does not match what you tested,
see troubleshooting.md; on a bare-tag install, suspect the tag first.
Uninstall cleanly
helm -n sreorbit uninstall sreorbitThe credentials Secret carries helm.sh/resource-policy: keep when the chart
creates it (not when you used secrets.existingSecret, which Helm never owned).
It survives uninstall on purpose, so a reinstall doesn't regenerate credentials you
still rely on elsewhere. Remove it yourself when you are done:
kubectl -n sreorbit delete secret sreorbit-secrets # only if the chart created itkubectl delete namespace sreorbit # removes everything elseTraces and the demo shop
The engine takes OpenTelemetry traces on the same port and bearer as remote-write, at
http://<release>-orbit-stack.<namespace>.svc.cluster.local:8082/v1/traces; with
networkPolicy.enabled, add your Collector's or your applications' namespace to
components.orbit-stack.networkPolicy.allowFromNamespaces first.
../apm.md has the exporter configuration.
To see APM with traffic in it before your own services send any, turn on the demo shop:
eight small services and a load generator from the orbit-stack image, one Deployment
each, sending traces to the engine with its bearer:
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \ --set demo.shop.enabled=truedemo.shop.rps sets the load (default 5 requests a second) and demo.shop.resources
the pods' requests and limits; --set demo.shop.enabled=false removes it all again. It
needs components.orbit-stack.enabled, and is off by default because it is load you did
not ask for.
The engine's volume
components.orbit-stack.persistence (on by default, 20Gi) renders a
PersistentVolumeClaim named <release>-orbit-stack-data with
helm.sh/resource-policy: keep, so helm uninstall leaves the telemetry behind for you
to delete deliberately. An empty storageClassName takes the cluster's default class,
except on EKS (a server version carrying -eks-) or with requireStorageClass: true,
where the render refuses rather than leave a claim waiting for a default that may not
exist: name an existing storageClassName, or set storageClass.create: true to have the
chart provision one (see "Amazon EKS" below for the EBS specifics). A claim that already
exists keeps the class it was bound with, since that field is immutable. The
engine runs one replica with a Recreate strategy because the claim is ReadWriteOnce.
The claim also holds it to one availability zone, so the agent, the OTLP collector and
orbit-api prefer that zone (a preferred podAffinity of weight 100 on its pods): their traffic
to orbit-stack stays off the cross-zone bill, and a full zone still schedules them elsewhere.
A component's own affinity replaces that preference (for orbit-api, the global one too).
Size it for ORBIT_STACK_RETENTION at your bytes per sample
(../benchmark.md measured 1.7 B on a scrape-shaped workload) plus one
block duration of write-ahead log. Point any agent of your own at
http://<release>-orbit-stack.<namespace>.svc.cluster.local:8082/api/v1/write with the
bearer above, as the chart's own agent (see "Telemetry" above) does;
../orbit-stack.md has the full configuration.
Amazon EKS
deploy/helm/sreorbit/values-eks.yaml is the EKS overlay: the whole product on any EKS
cluster, on top of the chart's own defaults (values.yaml). The cluster only needs to supply an
ingress controller and the Amazon EBS CSI driver add-on (with its IAM role) for
orbit-stack's volume; nothing here assumes a public API endpoint, since the chart talks
to the cluster from inside it.
helm upgrade --install sreorbit deploy/helm/sreorbit -n sreorbit \ -f deploy/helm/sreorbit/values-eks.yaml \ --set-string secrets.orbitAuthToken="$(openssl rand -hex 32)" \ --set-string secrets.nocturneAuthToken="$(openssl rand -hex 32)" \ --set-string secrets.orbitStackAuthToken="$(openssl rand -hex 32)"Tokens passed that way rotate on the next such run; for production, manage a Secret
yourself and set secrets.existingSecret instead, same as above. Without an admin in the
Secret, this install's orbit-api logs a one-time setup code for the first sign-in ("The
first admin, and orbit-api's volume" above).
values-eks.yaml sets components.web.auth.enabled to true, so this overlay
turns on web's own Basic auth, and pairs it with ingress.basicAuth.enabled: false.
The quickstart above doesn't set WEB_AUTH_USER or WEB_AUTH_VERIFIER; add both to
the release Secret (sreorbit-secrets, or whatever secrets.existingSecret names;
see "Web's own login" above) before web can start, or the pod sits in
CreateContainerConfigError. values-eks.yaml also caps requests at the edge
with nginx.ingress.kubernetes.io/limit-rps: "20" and
nginx.ingress.kubernetes.io/limit-connections: "20"; raise both when many
operators share one NAT address.
ingress.className is a CHANGEME (kubectl get ingressclass; many clusters have no
default class). The overlay also sets ingress.controller
(nginx/traefik/haproxy/alb/none; empty derives it from className), which picks
the SSE-safe annotations and gates basic auth, so a class named e.g. nginx-internal
still gets nginx's annotations. TLS is optional. ingress.basicAuth.{enabled,secretName,realm}
puts a password on the whole console via ingress-nginx annotations, reading an htpasswd
file from a Secret keyed auth; the render fails if basic auth is on and the controller
isn't nginx.
Turning ingress.basicAuth.enabled on alongside components.web.auth.enabled is the
one combination the chart refuses to render at all: ingress-nginx strips the
Authorization header once its own basic auth has accepted it, so web would never
see a credential to check, and would reject every request on its own instead. Pick
one layer, not both.
orbit-stack's persistence, and orbit-api's, assume no default StorageClass exists:
storageClass.create: true provisions <release>-orbit-stack and <release>-orbit-api
themselves (ebs.csi.aws.com, gp3, encrypted,
WaitForFirstConsumer, allowVolumeExpansion, Retain; every field on it is immutable
once created). Under EKS Auto Mode, EBS is provisioned through its own driver: set
storageClass.provisioner to ebs.csi.eks.amazonaws.com there instead.
requireStorageClass: true fails the render if neither storageClassName nor
storageClass.create is set. A PVC stuck Pending means the CSI driver or its IAM role
is missing, and NOTES.txt says so.
networkPolicy.apiServerCIDRs needs the same care: on EKS, kubernetes.default is the first
address of the cluster's service CIDR, and its endpoints are the control plane's ENIs on 443.
Name both: the VPC CNI's network policy agent judges the Service IP itself, before
kube-proxy's rewrite, while Calico and Cilium judge the endpoints instead:
kubectl get svc kubernetes -n defaultkubectl get endpointslices -n default -l kubernetes.io/service-name=kubernetesNocturne stays on the mock provider; the write path is off (rbac.write.enabled: false,
ORBIT_WRITE_ENABLED: "false"); ORBIT_DEMO_DATA: "false" with orbit-api's
rbac.enabled: true gives a read-only live console of the real cluster instead.
image.registry and telemetry.*.image.registry are both overridable for a private ECR
mirror.
Where to go next
- Every variable referenced above:
../configuration.md. - What the NetworkPolicy objects this install created do and don't enforce on your
cluster's CNI:
../security.md. - No Helm allowed:
kubectl.md.