Features
Audit trail
Every sign-in, every change to users, roles, dashboards, alerts and workloads, and every
refusal, as one event with who, what, when, from where and with what result. orbit-api
writes it (CONTRACT.md Addendum v0.5), the Deck reads it at /deck/admin/audit, and it
exports to CSV. Reading it needs the audit.read permission, which only the built-in
admin role holds (see identity.md).
What is recorded
| Action | When |
|---|---|
auth.login | A sign-in succeeded. |
auth.login_failed | A sign-in was refused: wrong password, unknown username, disabled account, or a wait still running. |
auth.logout | A session was ended by its owner. |
auth.setup | The first admin was created with the setup code, or a wrong code was tried. |
auth.password_change | Someone changed their own password, or typed a wrong current one. |
user.create | An admin created a user; also written once per account orbit-api creates at startup (the env admin, the demo users), with no actor. |
user.update | A user's name or email changed, or several fields at once. |
user.role_change, user.disable, user.enable | Exactly that one field changed. |
user.password_reset, user.mfa_reset | A new password, or the second factor turned off, whatever else changed with it (the password wins when both change). With outcome failure, someone tried it on their own account without the right current password or code. |
user.delete, user.revoke_sessions | An account was removed, or all its sessions ended. |
role.create, role.update, role.delete | A custom role changed. |
dashboard.create, dashboard.update, dashboard.delete | A saved dashboard changed. |
alert.ack, alert.unack, alert.silence, alert.unsilence | An alert's acknowledgement or silence changed. |
workload.scale, workload.resources, workload.rollback, workload.restart | Every write attempt, the dry runs, the applies and the refusals. A rollback adds fromRevision, toRevision, images, changes (env by name only, never a value) and, on an apply, typedConfirmation. A restart's diff is the template's restartedAt before and after, and its apply adds typedConfirmation. |
workload.rollback.result | How the rollout of an applied rollback ended: outcome (completed, failed or timeout), durationMs, updated, ready, desired. |
access.denied | A signed-in user asked for a route their role does not grant. |
A user change is named by what changed, so a form that resends every field still
records a role change as user.role_change, and a save that changed nothing records
nothing. The Deck's settings page is read-only in this build and records nothing.
One event, as GET /api/v1/audit returns it:
{
"id": "1790227603035-7bf2888b",
"ts": 1790227603035,
"actor": { "id": "25c24c6f2e7315a6", "username": "viewer", "role": "viewer" },
"action": "access.denied",
"target": { "type": "route", "id": "GET /api/v1/users", "name": "GET /api/v1/users" },
"outcome": "denied",
"ip": "203.0.113.24",
"userAgent": "Mozilla/5.0 (X11; Linux x86_64)",
"requestId": "78162c82575d135d8642787e",
"details": { "permission": "users.manage" }
}tsis epoch milliseconds and never goes backwards in the trail, even when the clock does.idis that time plus 8 random hex digits.actorisnullwhen nobody was signed in: a failed sign-in, the accounts created at startup.roleis the actor's role at that moment.target.typeisuser,role,route,dashboard,alertorworkload.outcomeissuccess,denied(refused by a rule: a permission, a wait, a disabled account) orfailure(attempted and failed: a wrong password, an API server error).detailsholds thebeforeandafterof a change (for a dashboard, a summary: title, version, panel and variable counts, tags), the permission a refusal named, the dry run and diff of a workload write. It never holds a password, a token or a verifier; a password reset records only that it happened.
Three rules keep the trail honest without letting it be flooded or turned into a leak:
- A failed sign-in names its target only when the username exists. Anything else is kept
as
usernameFingerprint, the first 12 hex digits of its SHA-256, so a password typed into the username field never lands in the trail while repeated attempts still correlate. - A refusal inside a sign-in wait is recorded once per wait, with
retryAfterSeconds. access.deniedis answered every time, and recorded at most once a second per user (bursts of 30), so a script looping on a forbidden route cannot fill the disk.
Every event is written to disk and fsynced before the request that caused it is answered,
and is also one audit line in orbit-api's own log (id, action, outcome, actor,
target, client_ip, request_id), so a log pipeline receives the trail too.
Reading it in the Deck
/deck/admin/audit shows the window picked in the top bar (the last hour by default): a
histogram of events over time, stacked by outcome, where a click on a bar zooms into it;
facets by outcome, action and actor, where each facet counts the events matching every
other filter; the events, newest first, 100 at a time; and a drawer per event with its
changes as a before and after table and the raw JSON. The search box takes plain words,
which must each appear in some value of the event, and five fields:
| Term | Matches |
|---|---|
actor:grace | The actor's username |
action:user.* | The action |
outcome:denied | success, denied or failure |
ip:10.0.* | The client address |
target:checkout | Part of the target's type, id or name |
The first four match exactly, or as a prefix when the value ends in *. Terms combine
with AND, matching ignores case, and "quoted words" stay one term. The page's address
keeps the filters, so a link opens the same view.
Reading it over the API
GET /api/v1/audit takes from and to (epoch milliseconds, both inclusive; the last
24 hours by default), actor, action and outcome (exact), q (the search language
above), limit (1 to 1000, default 100) and cursor. With the SESSION of a user who
holds audit.read (identity.md → "Signing in from a shell"):
curl -fsS -G -H "Authorization: Bearer $ORBIT_AUTH_TOKEN" -H "X-Orbit-Session: $SESSION" \ --data-urlencode 'q=outcome:denied' --data-urlencode 'limit=50' localhost:8080/api/v1/auditThe answer is {events, nextCursor, total, facets: {action, actor, outcome}, interval, histogram: [{t, n, denied, failure}]}. total, the facets and the histogram cover the
whole window on every page; interval is the histogram's bucket width in milliseconds,
chosen so the window fits in 60 bars. For the next page, send the same filters with
cursor set to nextCursor, which is empty on the last page. A filter value longer than
the route accepts (q 200 bytes, actor and action 64, outcome 16) is 400.
Exporting CSV
The Deck's "Export CSV" downloads the current filters. Over the API, format=csv streams
every event matching the same filters, newest first, at most 100,000 rows, ignoring
limit and cursor:
curl -fsS -G -H "Authorization: Bearer $ORBIT_AUTH_TOKEN" -H "X-Orbit-Session: $SESSION" \ --data-urlencode 'format=csv' -o orbit-audit.csv localhost:8080/api/v1/auditColumns: time (RFC 3339, UTC), actor, role, action, outcome, target_type,
target_id, target_name, ip, user_agent, request_id, details (JSON). A cell that
starts with =, +, -, @, a tab or a carriage return gets a leading ', so a user
agent or a username cannot run as a spreadsheet formula. Both answers are sent with
Cache-Control: no-store.
Retention and storage
The trail is append-only JSONL under ORBIT_DATA_DIR/audit, one file per UTC day
(2026-09-24.jsonl). ORBIT_AUDIT_RETENTION decides how long a day is kept: days such as
90d (the default) or 365d, or a Go duration of 24h or more; anything shorter refuses
to start. Whole day files older than that are deleted when orbit-api starts and when the
first event of a new day is written. Size the volume from your own trail: run
du -sh on the audit directory after a representative week and scale by the retention.
A query reads the day files its window overlaps, line by line, so a wide window over a
long retention costs a scan of those files; the upgrade path the source names is an index
in SQLite or Postgres. Backing the directory up is covered in
operations.md.
How the client IP is determined
The ip of an event, and the address every per-client limit in orbit-api keys on, is
decided in two places.
web decides what it forwards. Every browser request reaches orbit-api through
web, so the socket peer orbit-api sees is always web's own address. web passes an
X-Forwarded-For header on only when WEB_TRUST_PROXY=true, which says an edge proxy in
front of web wrote it; otherwise it sends none, and a header the browser set never
crosses.
orbit-api decides what it believes. It reads X-Forwarded-For only on a request
that carries its own service bearer, which only web holds, or that comes from a peer in
ORBIT_TRUSTED_PROXIES. It walks the header from the right, skips every address in
ORBIT_TRUSTED_PROXIES, and records the first other one. Without a vouched header, or
with a malformed one, it records the socket peer.
| Install path | WEB_TRUST_PROXY | What ip shows |
|---|---|---|
| Helm | Derived: true when ingress.enabled, false otherwise | The client as the ingress saw it |
| Plain manifests | true (60-ingress.yaml is the edge) | The client as the ingress saw it |
| Compose | false (web is the edge) | web's container address, for every visitor |
| k3d loop | Derived by the chart, true (Traefik) | The client as Traefik saw it |
ORBIT_TRUSTED_PROXIES is empty by default on every install path, and should stay so:
it is for proxies between web and orbit-api (a mesh sidecar, an internal gateway),
never web itself, which the bearer already vouches for. With ORBIT_AUTH_MODE=disabled
there is no bearer, so only a peer in ORBIT_TRUSTED_PROXIES is believed. And
WEB_TRUST_PROXY=true with nothing in front of web lets a browser choose the address it
is recorded under.
Behind a cloud load balancer or a CDN
The audit trail records whatever address the ingress controller sees.
ingress-nginx and Traefik replace a client's X-Forwarded-For with the address of the
connection by default, and an AWS ALB appends the address it saw, so the last entry is
always the controller's view of the caller. When the controller sits behind a load
balancer or a CDN that hides the client, that view is the load balancer, and every event
carries the load balancer's address until the edge passes the real client on. With
ingress-nginx, that is its ConfigMap (the chart's controller.config):
| Key | Set it to | Why |
|---|---|---|
use-forwarded-headers | "true" | Believe the X-Forwarded-For of the L7 load balancer or CDN in front, instead of replacing it. Default "false". |
proxy-real-ip-cidr | The load balancer's or CDN's own ranges (on AWS, the VPC CIDR for an ALB) | Whom to believe it from. The default, "0.0.0.0/0", believes anyone, which lets every client forge its address: always narrow it. |
compute-full-forwarded-for | Leave "false" | "true" appends the load balancer's address to the chain, and orbit-api, which records the rightmost address it does not trust, would record the load balancer again. |
enable-real-ip with forwarded-for-header | "true", and the CDN's client header (Cloudflare's is CF-Connecting-IP) | For a CDN that names the client in a header of its own; pair it with proxy-real-ip-cidr set to the CDN's published ranges. |
use-proxy-protocol | "true", with proxy protocol turned on at the load balancer | For an L4 load balancer (an AWS NLB), which cannot add a header. The alternative there is externalTrafficPolicy: Local on the controller's Service. |
These settings are controller-wide: they change how every Ingress in the cluster sees its
clients, and a wrong proxy-real-ip-cidr lets anyone choose their address. Changing them is
the cluster owner's decision, and the chart sets none of them. To check the
result, sign in from a known address and look at the ip of your own auth.login event.
The same edge hides the browser's scheme. By default ingress-nginx overwrites
X-Forwarded-Proto with the scheme of its own connection, so an ingress with no TLS behind a CDN
tells web the browser is on http when it is not. web then sets its cookies without Secure or
__Host- and prints http:// in the MCP snippet and the SSO redirect URIs, and orbit-api refuses
a plain http redirect URI outside localhost. Set the chart's publicUrl to the https address people open
(configuration.md), with no change to the controller. That corrects what web tells the browser
and encrypts nothing: passwords, one-time codes and session tokens cross the leg from the CDN to
the origin in clear until the origin serves TLS and the CDN verifies it (on Cloudflare, the
encryption mode Full (strict)).
The same address keys the per-client limits. When orbit-api has no trusted client address
(compose, or an edge that passes none), every visitor arrives from web's address, and
sign-in falls back to its per-username waits plus one bucket shared by all visitors, so one
visitor's failures cannot lock everyone else out (identity.md → "What a
refusal means").
The change timeline
The namespace page and the APM service page each have a Changes tab: what changed in the namespace (or in the
service's workload) in the page's time window, newest first, and through which tool or Deck user. Kubernetes keeps
events for about an hour (--event-ttl, 1h by default) and an alert's history ends when it resolves, so orbit-api
records these itself, from what it already watches, in a loop of its own at the alerting interval
(ORBIT_ALERTING_INTERVAL, 30 s). A slow API server holds that loop and not the alert notifications: its reads of
field managers share a budget of 10 seconds per tick, and past it a row goes without "who".
| Kind | Recorded when | Who |
|---|---|---|
| Rollout | A Deployment's newest ReplicaSet revision changes: a new ReplicaSet, at its creation, or an older one brought back by a rollback, at the Deployment controller's event that scaled it up from 0. The row names the revision, the ReplicaSet, and each container's image, requests and limits before and after. | The field manager whose write of spec.template is the newest in the 5 seconds before that time, from the Deployment's managedFields. An entry keeps only a tool's last write and the fields it still owns, so a later write can hide an earlier one (a rollback that only removed a field owns nothing), and then nobody is named. Typical names: helm, argocd-controller, kubectl-set, kubectl-rollout, kubectl-client-side-apply. |
| Scale | A Deployment's spec.replicas changes: one row per step, each at the time of the Deployment controller's ScalingReplicaSet event. While a rollout is in progress its own steps read like a scale, so the row takes the time of the HPA's SuccessfulRescale event when there is one, and else the tick's, with ~. A row the HPA made carries the message of that event. | The manager that wrote spec.replicas: kubectl, helm, or HPA when the controller manager wrote it and the HPA's event explains it. |
| Warning | Warning events of one namespace, reason, workload and message's first word (a Liveness failure is not a Readiness one) are one row, however many events and objects it covers. The row goes to disk when it opens, a minute later, then every 10 minutes while it grows: that is how far the tab can lag behind the cluster. A new revision of the workload or an hour closes it and the next event opens a new row, so a chronic failure reads hour by hour and what a rollout's pods do is never folded into what came before. The row keeps the latest message, says how many times, and names the first object with how many more. | Nobody. |
| Alert | A critical or warning alert in the namespace starts firing or resolves. Info alerts are left out. | Nobody. |
| Deck action | Read from this trail when the tab opens: scale, resources, suspend, run now, restart, rollback, a terminal opened, an alert acknowledged or silenced. Dry runs are left out. | The Deck user. |
Nothing guesses a person. A tool's change names the tool, a change with no field manager near it names nobody, and a
change orbit-api wrote reads "the Deck": the Deck's own row beside it names the user, for a role with audit.read.
The person behind a CI run or a kubectl session is in the cluster's own audit log, which orbit-api does not read.
Without audit.read the tab leaves out the Deck's rows and their chip, and the postmortem says they are not visible to
the role. The first tick on a cluster takes stock of its Deployments and alerts without writing (what stood before
recording began is no change) and copies the Warning events Kubernetes still holds; the tab says since when it records.
A change made while orbit-api was down gets its row at the next tick. A shutdown writes what the open rows hold and
saves them, so the next start goes on counting them and the counts match the cluster's. A crash loses at most the last
10 minutes of a row's growth, and an event that grew while orbit-api was down counts once.
Times. A row's time is the cluster's: a ReplicaSet's creation, the controller's event, a field manager's write. When
the event is gone and no write dates the change, as after a long outage or on a busy Deployment (Kubernetes folds
events past 10 alike in 10 minutes), the row takes the time a tick saw it, a rollback stays before the revision that
followed it, and the time reads ~ on screen and in the Markdown.
Bounds. One JSONL file per UTC day under ORBIT_DATA_DIR/changes/<cluster>/, kept 14 days: whole day files past
that are deleted with the first write of a day. A day takes at most 5,000 rows, at most 4,500 of them Warning rows and
at most 500 Warning rows per namespace, however many lines each took, so one namespace's storm leaves room for the others
and for rollouts, scales and alerts. A row's later lines stop once the day's file reaches 5 MiB, and the row keeps its last
count. What a full day leaves out is counted per namespace and logged once; the tab and the Markdown say how many. A full
day is at most about 12 MB, since an event message is cut at 1 KiB, so a cluster's worst case is about 160 MB: size
persistence.size with it. One document per cluster under changestate/ holds what the last tick saw. A read answers
at most 500 rows, newest first, and the oldest Warnings are the first left out, so a storm cannot hide the rollouts
before it. The chips count before that cut, and the tab and the Markdown say when it cut.
Copy as postmortem copies the rows the kind chips leave, oldest first, as Markdown: a title with the scope, the window in UTC, a note on anything missing, an empty Impact section, the timeline as a table (time, kind, object, what changed with a row's last event and whether it began before the window, who), every Deck action of the window as a list, and an empty Cause section. Text from the cluster is escaped, so pasted into a tracker it stays one cell, with no link or image. If the browser refuses the clipboard, the button says so. Nothing opens by itself, no incident is created, and no model writes any of it.
Not recorded: rollouts and scales of StatefulSets and DaemonSets (their revisions live in ControllerRevisions, which
orbit-api does not watch), which the tab says; ConfigMap and Secret edits; the events of cluster-scoped objects such as
nodes, and alerts with no namespace.
What the trail is not
- Not tamper-evident. The files are plain JSONL on a volume: anyone who can write to
it can edit history. For evidence that survives an administrator, ship the
auditlog lines to an append-only store you do not administer from the same account. - Not a record of reads. A successful read is in the request log, not the trail; a
refused one is
access.denied. - Not complete across a lost volume. With
components.orbit-api.persistence.enabledoff the directory is anemptyDir, and the trail goes with the pod.
Where to go next
- The roles and permissions every
access.deniednames:identity.md. - The route, its parameters and its errors in the reference:
api.md. - Backups, retention sizing and what to alert on:
operations.md.