Operate
Alert rules
There is no Alertmanager in this build. With ORBIT_DEMO_DATA=false, orbit-api derives
alerts from the same Kubernetes objects the rest of the Deck reads, with six rules, each one
a thing an SRE would otherwise write a PrometheusRule for. This page is the runbook for
each of them: what makes it fire and what to check first. The Deck's runbook links land on
the section of the rule, and its address is stable: /docs/alerts#nodenotready, and so on
for the other five. The demo estate serves its own fixed set of alerts instead, the ones
CLAIMS.md tells the story of.
How a derived alert behaves
- Evaluated on every read. Each
GET /api/v1/clusters/{id}/alertsruns the six rules against the live source's cache of the cluster; there is no evaluation loop and no history. An alert exists while its condition holds and is gone from the next read after it stops. - Always firing, never routed. A derived rule has no
forduration to be pending through, so every alert isstate: "firing", and it carries noteam: nothing in-cluster says who owns a workload, and inventing an owner pages the wrong people. - Stable identity. The id is the cluster, the rule, the namespace and the object the alert names, so the same rule on the same object is the same alert across reads and restarts, and an acknowledgement stays attached to it. For the two pod rules that object is the pod's workload: two failing pods of one workload come back as two rows with one id, and share one acknowledgement.
- Severity.
criticalforNodeNotReadyandPodCrashLooping,warningforPodPendingandDeploymentReplicasUnavailable,infoforKubeletVersionSkewandContainerWithoutLimits. The list comes back sorted by severity, then by how long each alert has been open.
Acknowledge and silence
Anyone whose role holds workloads.write can acknowledge an alert, or silence it for 15
minutes, 1 hour, 4 hours or 24 hours, one row at a time or a whole selection at once from
the Alerts page, with an optional note of up to 500 characters. The mark is orbit-api's
own state, kept per cluster under ORBIT_DATA_DIR/alertstate, and laid over every read of
the alerts, Nocturne's included: an alert then shows ack as acked or silenced, with
ackedBy, ackedAt, silencedUntil and ackNote.
| Route | Body | Effect |
|---|---|---|
POST /api/v1/clusters/{id}/alerts/{alertId}/ack | {note?} | acked |
DELETE /api/v1/clusters/{id}/alerts/{alertId}/ack | none | back to open when it is acked |
POST /api/v1/clusters/{id}/alerts/{alertId}/silence | {minutes: 15, 60, 240 or 1440, note?} | silenced until then |
DELETE /api/v1/clusters/{id}/alerts/{alertId}/silence | none | back to open when it is silenced |
A silence that runs out needs nobody to clear it. The DELETEs reopen an alert whoever set
its state, and answer 200 without changing or recording anything when the alert is not
in that state, so a bulk clear over a mixed selection is safe. An unknown alert is 404, a
duration off the list or a longer note 400. Every change is one audit event, alert.ack,
alert.unack, alert.silence or alert.unsilence, with the cluster, the state before and
after, and the note (audit.md). Through web these four routes need a
session and same-origin proof and no operator token, since they change Orbit's own state
and never the cluster. An acknowledgement outlives an alert that stopped firing, and
applies again if the same alert fires later.
NodeNotReady
Critical. A node whose Ready condition is not True, or that is cordoned
(spec.unschedulable). The summary says which: Node ip-100-64-42-19 is NotReady, or
... is SchedulingDisabled. "Since" is the last transition of the node's Ready
condition.
What to check, with NODE the node the alert names:
kubectl describe node NODEkubectl get node NODE -o jsonpath='{range .status.conditions[*]}{.type}={.status} {.reason}{"\n"}{end}'kubectl get pods -A --field-selector spec.nodeName=NODE -o wideNotReadywithMemoryPressure,DiskPressureorPIDPressuretrue: the node is starved, and its pods are being evicted. Find what grew on it.NotReadywith every other conditionUnknown: the kubelet stopped reporting. The instance is down, out of the network, or its kubelet died: look at the cloud console and the kubelet's own logs, and at whether a spot interruption drained it.SchedulingDisabled: someone or something cordoned it, usually a drain or an upgrade in progress.kubectl uncordon NODEonce that work is done; if no one is draining it, find who cordoned it before undoing it.
PodCrashLooping
Critical. A pod that has not finished, with a container in CrashLoopBackOff or a
container restart that ended within the last 15 minutes. One alert per pod, named after its workload:
Pod checkout-api-6f4b7c-2xk8p restarted 4 times; last restart inside 15m. "Since" is the
last restart.
What to check, with NAMESPACE and POD from the alert:
kubectl -n NAMESPACE logs POD --all-containers --previous --tail=100kubectl -n NAMESPACE describe pod PODkubectl -n NAMESPACE get events --field-selector involvedObject.name=POD --sort-by=.lastTimestampLast State: TerminatedwithReason: OOMKilled: the container outgrew its memory limit. Raise the limit or find the leak; the workload's page shows memory against it.- An
Errorexit with a stack trace in the previous logs: the process crashes on start or under load. A rollout that started with the restarts is the first suspect: compare the image the pod runs with the previous one. - A liveness probe that fails in the events: the process is alive and too slow to answer. Check what the probe calls and its timeouts before raising them.
The Deck's logs tab reads the previous container with one click, which is the one an operator opens this for.
PodPending
Warning. A pod still Pending more than 5 minutes after it was created. The summary
carries the scheduler's own message:
Pod batch-7c9d-x2 has been Pending for more than 5m: 0/3 nodes are available: 3 Insufficient cpu.
A pod that was scheduled and is still pending (pulling an image, waiting on a volume, running
init containers) reads not scheduled there instead; its node and container states tell the
real story.
What to check, with NAMESPACE and POD from the alert:
kubectl -n NAMESPACE describe pod PODkubectl -n NAMESPACE get pvckubectl top nodesInsufficient cpuorInsufficient memory: the requests do not fit on any node. Rightsize the requests or add capacity; the FinOps page shows which requests sit idle.didn't match Pod's node affinity/selectororhad untolerated taint: the pod may only run on nodes that are not there. Check itsnodeSelector, affinity and tolerations against the node groups.unbound immediate PersistentVolumeClaims, or a claim stuckPending: no volume can be provisioned. Check the StorageClass and the CSI driver.- Scheduled, and waiting on
ErrImagePullorImagePullBackOff: the image name, tag or pull secret is wrong, or the registry cannot be reached.
DeploymentReplicasUnavailable
Warning. A Deployment with fewer ready replicas than it asks for:
2 of 3 replicas available. It fires on any gap, including the seconds of an ordinary
rollout or scale-up, because a derived rule has no for duration; one that stays is the
one to look at. "Since" counts from the Deployment's creation, not from when the replicas
went missing. Only Deployments are checked; a StatefulSet or DaemonSet short of replicas
shows in its crash-looping or pending pods instead.
What to check, with NAME the workload the alert names:
kubectl -n NAMESPACE rollout status deploy/NAME --timeout=10skubectl -n NAMESPACE describe deploy NAMEkubectl -n NAMESPACE get pods -o wide- A rollout that does not progress: the new pods fail their readiness probe or crash. A
PodCrashLoopingorPodPendingon the same workload names the pod; the workload's Events tab shows the rollout step by step. exceeded quotain the conditions: aResourceQuotaof the namespace refuses the new pods.- Everything running and not ready: the readiness probe fails. Read what it checks, and whether a dependency it calls is down.
KubeletVersionSkew
Info. A node whose kubelet is more than one minor version behind the control plane:
Node ip-100-64-7-12 runs kubelet v1.29.8 against control plane v1.31.4. Kubernetes itself
supports a wider skew, so the alert reminds you that an upgrade left a node group behind;
on its own it does not mean an outage.
What to check:
kubectl versionkubectl get nodes -o custom-columns=NAME:.metadata.name,KUBELET:.status.nodeInfo.kubeletVersionUpgrade the node group, or replace its nodes, before the next control-plane upgrade widens the gap past what Kubernetes supports.
ContainerWithoutLimits
Info. A workload whose pods have a container with no CPU limit, no memory limit, or
neither: No memory limit on worker; No CPU limit on sidecar. One alert per workload, not
per pod. The signal is memory when any container lacks a memory limit, CPU otherwise.
What to check, for a Deployment, with NAMESPACE and NAME from the alert:
kubectl -n NAMESPACE get deploy NAME -o jsonpath='{range .spec.template.spec.containers[*]}{.name}{"\t"}{.resources.limits}{"\n"}{end}'- No memory limit: one container can take the node's memory and have its neighbours evicted. Set a limit above its peak working set, which the workload's page charts.
- No CPU limit: often deliberate, because a CPU limit throttles a request-serving process
at its peaks. This product's own
webruns without one for that reason. Leave it off when the requests are right and the node has headroom.
The Scale and resources tab of the workload changes requests and limits as a dry run first, when the deployment allows writes (Addendum v0.4).
Where to go next
- What else to watch, and what the probes mean:
operations.md. - The alert shape, the ack routes and their errors in the reference:
api.md. - Who may acknowledge and silence, and how to grant it:
identity.md.