Skip to content

Operate

Alert rules

There is no Alertmanager in this build. With ORBIT_DEMO_DATA=false, orbit-api derives alerts from the same Kubernetes objects the rest of the Deck reads, with six rules, each one a thing an SRE would otherwise write a PrometheusRule for. This page is the runbook for each of them: what makes it fire and what to check first. The Deck's runbook links land on the section of the rule, and its address is stable: /docs/alerts#nodenotready, and so on for the other five. The demo estate serves its own fixed set of alerts instead, the ones CLAIMS.md tells the story of.

How a derived alert behaves

  • Evaluated on every read. Each GET /api/v1/clusters/{id}/alerts runs the six rules against the live source's cache of the cluster; there is no evaluation loop and no history. An alert exists while its condition holds and is gone from the next read after it stops.
  • Always firing, never routed. A derived rule has no for duration to be pending through, so every alert is state: "firing", and it carries no team: nothing in-cluster says who owns a workload, and inventing an owner pages the wrong people.
  • Stable identity. The id is the cluster, the rule, the namespace and the object the alert names, so the same rule on the same object is the same alert across reads and restarts, and an acknowledgement stays attached to it. For the two pod rules that object is the pod's workload: two failing pods of one workload come back as two rows with one id, and share one acknowledgement.
  • Severity. critical for NodeNotReady and PodCrashLooping, warning for PodPending and DeploymentReplicasUnavailable, info for KubeletVersionSkew and ContainerWithoutLimits. The list comes back sorted by severity, then by how long each alert has been open.

Acknowledge and silence

Anyone whose role holds workloads.write can acknowledge an alert, or silence it for 15 minutes, 1 hour, 4 hours or 24 hours, one row at a time or a whole selection at once from the Alerts page, with an optional note of up to 500 characters. The mark is orbit-api's own state, kept per cluster under ORBIT_DATA_DIR/alertstate, and laid over every read of the alerts, Nocturne's included: an alert then shows ack as acked or silenced, with ackedBy, ackedAt, silencedUntil and ackNote.

RouteBodyEffect
POST /api/v1/clusters/{id}/alerts/{alertId}/ack{note?}acked
DELETE /api/v1/clusters/{id}/alerts/{alertId}/acknoneback to open when it is acked
POST /api/v1/clusters/{id}/alerts/{alertId}/silence{minutes: 15, 60, 240 or 1440, note?}silenced until then
DELETE /api/v1/clusters/{id}/alerts/{alertId}/silencenoneback to open when it is silenced

A silence that runs out needs nobody to clear it. The DELETEs reopen an alert whoever set its state, and answer 200 without changing or recording anything when the alert is not in that state, so a bulk clear over a mixed selection is safe. An unknown alert is 404, a duration off the list or a longer note 400. Every change is one audit event, alert.ack, alert.unack, alert.silence or alert.unsilence, with the cluster, the state before and after, and the note (audit.md). Through web these four routes need a session and same-origin proof and no operator token, since they change Orbit's own state and never the cluster. An acknowledgement outlives an alert that stopped firing, and applies again if the same alert fires later.

NodeNotReady

Critical. A node whose Ready condition is not True, or that is cordoned (spec.unschedulable). The summary says which: Node ip-100-64-42-19 is NotReady, or ... is SchedulingDisabled. "Since" is the last transition of the node's Ready condition.

What to check, with NODE the node the alert names:

bash
kubectl describe node NODEkubectl get node NODE -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'kubectl get pods -A --field-selector spec.nodeName=NODE -o wide
  • NotReady with MemoryPressure, DiskPressure or PIDPressure true: the node is starved, and its pods are being evicted. Find what grew on it.
  • NotReady with every other condition Unknown: the kubelet stopped reporting. The instance is down, out of the network, or its kubelet died: look at the cloud console and the kubelet's own logs, and at whether a spot interruption drained it.
  • SchedulingDisabled: someone or something cordoned it, usually a drain or an upgrade in progress. kubectl uncordon NODE once that work is done; if no one is draining it, find who cordoned it before undoing it.

PodCrashLooping

Critical. A pod that has not finished, with a container in CrashLoopBackOff or a container restart that ended within the last 15 minutes. One alert per pod, named after its workload: Pod checkout-api-6f4b7c-2xk8p restarted 4 times; last restart inside 15m. "Since" is the last restart.

What to check, with NAMESPACE and POD from the alert:

bash
kubectl -n NAMESPACE logs POD --all-containers --previous --tail=100kubectl -n NAMESPACE describe pod PODkubectl -n NAMESPACE get events --field-selector involvedObject.name=POD --sort-by=.lastTimestamp
  • Last State: Terminated with Reason: OOMKilled: the container outgrew its memory limit. Raise the limit or find the leak; the workload's page shows memory against it.
  • An Error exit with a stack trace in the previous logs: the process crashes on start or under load. A rollout that started with the restarts is the first suspect: compare the image the pod runs with the previous one.
  • A liveness probe that fails in the events: the process is alive and too slow to answer. Check what the probe calls and its timeouts before raising them.

The Deck's logs tab reads the previous container with one click, which is the one an operator opens this for.

PodPending

Warning. A pod still Pending more than 5 minutes after it was created. The summary carries the scheduler's own message: Pod batch-7c9d-x2 has been Pending for more than 5m: 0/3 nodes are available: 3 Insufficient cpu. A pod that was scheduled and is still pending (pulling an image, waiting on a volume, running init containers) reads not scheduled there instead; its node and container states tell the real story.

What to check, with NAMESPACE and POD from the alert:

bash
kubectl -n NAMESPACE describe pod PODkubectl -n NAMESPACE get pvckubectl top nodes
  • Insufficient cpu or Insufficient memory: the requests do not fit on any node. Rightsize the requests or add capacity; the FinOps page shows which requests sit idle.
  • didn't match Pod's node affinity/selector or had untolerated taint: the pod may only run on nodes that are not there. Check its nodeSelector, affinity and tolerations against the node groups.
  • unbound immediate PersistentVolumeClaims, or a claim stuck Pending: no volume can be provisioned. Check the StorageClass and the CSI driver.
  • Scheduled, and waiting on ErrImagePull or ImagePullBackOff: the image name, tag or pull secret is wrong, or the registry cannot be reached.

DeploymentReplicasUnavailable

Warning. A Deployment with fewer ready replicas than it asks for: 2 of 3 replicas available. It fires on any gap, including the seconds of an ordinary rollout or scale-up, because a derived rule has no for duration; one that stays is the one to look at. "Since" counts from the Deployment's creation, not from when the replicas went missing. Only Deployments are checked; a StatefulSet or DaemonSet short of replicas shows in its crash-looping or pending pods instead.

What to check, with NAME the workload the alert names:

bash
kubectl -n NAMESPACE rollout status deploy/NAME --timeout=10skubectl -n NAMESPACE describe deploy NAMEkubectl -n NAMESPACE get pods -o wide
  • A rollout that does not progress: the new pods fail their readiness probe or crash. A PodCrashLooping or PodPending on the same workload names the pod; the workload's Events tab shows the rollout step by step.
  • exceeded quota in the conditions: a ResourceQuota of the namespace refuses the new pods.
  • Everything running and not ready: the readiness probe fails. Read what it checks, and whether a dependency it calls is down.

KubeletVersionSkew

Info. A node whose kubelet is more than one minor version behind the control plane: Node ip-100-64-7-12 runs kubelet v1.29.8 against control plane v1.31.4. Kubernetes itself supports a wider skew, so the alert reminds you that an upgrade left a node group behind; on its own it does not mean an outage.

What to check:

bash
kubectl versionkubectl get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersion

Upgrade the node group, or replace its nodes, before the next control-plane upgrade widens the gap past what Kubernetes supports.

ContainerWithoutLimits

Info. A workload whose pods have a container with no CPU limit, no memory limit, or neither: No memory limit on worker; No CPU limit on sidecar. One alert per workload, not per pod. The signal is memory when any container lacks a memory limit, CPU otherwise.

What to check, for a Deployment, with NAMESPACE and NAME from the alert:

bash
kubectl -n NAMESPACE get deploy NAME -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.resources.limits}{"\n"}{end}'
  • No memory limit: one container can take the node's memory and have its neighbours evicted. Set a limit above its peak working set, which the workload's page charts.
  • No CPU limit: often deliberate, because a CPU limit throttles a request-serving process at its peaks. This product's own web runs without one for that reason. Leave it off when the requests are right and the node has headroom.

The Scale and resources tab of the workload changes requests and limits as a dry run first, when the deployment allows writes (Addendum v0.4).

Where to go next

  • What else to watch, and what the probes mean: operations.md.
  • The alert shape, the ack routes and their errors in the reference: api.md.
  • Who may acknowledge and silence, and how to grant it: identity.md.