Skip to content

Features

Audit trail

Every sign-in, every change to users, roles, dashboards, alerts and workloads, and every refusal, as one event with who, what, when, from where and with what result. orbit-api writes it (CONTRACT.md Addendum v0.5), the Deck reads it at /deck/admin/audit, and it exports to CSV. Reading it needs the audit.read permission, which only the built-in admin role holds (see identity.md).

What is recorded

ActionWhen
auth.loginA sign-in succeeded.
auth.login_failedA sign-in was refused: wrong password, unknown username, disabled account, or a wait still running.
auth.logoutA session was ended by its owner.
auth.setupThe first admin was created with the setup code, or a wrong code was tried.
auth.password_changeSomeone changed their own password, or typed a wrong current one.
user.createAn admin created a user; also written once per account orbit-api creates at startup (the env admin, the demo users), with no actor.
user.updateA user's name or email changed, or several fields at once.
user.role_change, user.disable, user.enableExactly that one field changed.
user.password_reset, user.mfa_resetA new password, or the second factor turned off, whatever else changed with it (the password wins when both change). With outcome failure, someone tried it on their own account without the right current password or code.
user.delete, user.revoke_sessionsAn account was removed, or all its sessions ended.
role.create, role.update, role.deleteA custom role changed.
dashboard.create, dashboard.update, dashboard.deleteA saved dashboard changed.
alert.ack, alert.unack, alert.silence, alert.unsilenceAn alert's acknowledgement or silence changed.
workload.scale, workload.resources, workload.rollback, workload.restartEvery write attempt, the dry runs, the applies and the refusals. A rollback adds fromRevision, toRevision, images, changes (env by name only, never a value) and, on an apply, typedConfirmation. A restart's diff is the template's restartedAt before and after, and its apply adds typedConfirmation.
workload.rollback.resultHow the rollout of an applied rollback ended: outcome (completed, failed or timeout), durationMs, updated, ready, desired.
access.deniedA signed-in user asked for a route their role does not grant.

A user change is named by what changed, so a form that resends every field still records a role change as user.role_change, and a save that changed nothing records nothing. The Deck's settings page is read-only in this build and records nothing.

One event, as GET /api/v1/audit returns it:

json
{
  "id": "1790227603035-7bf2888b",
  "ts": 1790227603035,
  "actor": { "id": "25c24c6f2e7315a6", "username": "viewer", "role": "viewer" },
  "action": "access.denied",
  "target": { "type": "route", "id": "GET /api/v1/users", "name": "GET /api/v1/users" },
  "outcome": "denied",
  "ip": "203.0.113.24",
  "userAgent": "Mozilla/5.0 (X11; Linux x86_64)",
  "requestId": "78162c82575d135d8642787e",
  "details": { "permission": "users.manage" }
}
  • ts is epoch milliseconds and never goes backwards in the trail, even when the clock does. id is that time plus 8 random hex digits.
  • actor is null when nobody was signed in: a failed sign-in, the accounts created at startup. role is the actor's role at that moment.
  • target.type is user, role, route, dashboard, alert or workload.
  • outcome is success, denied (refused by a rule: a permission, a wait, a disabled account) or failure (attempted and failed: a wrong password, an API server error).
  • details holds the before and after of a change (for a dashboard, a summary: title, version, panel and variable counts, tags), the permission a refusal named, the dry run and diff of a workload write. It never holds a password, a token or a verifier; a password reset records only that it happened.

Three rules keep the trail honest without letting it be flooded or turned into a leak:

  • A failed sign-in names its target only when the username exists. Anything else is kept as usernameFingerprint, the first 12 hex digits of its SHA-256, so a password typed into the username field never lands in the trail while repeated attempts still correlate.
  • A refusal inside a sign-in wait is recorded once per wait, with retryAfterSeconds.
  • access.denied is answered every time, and recorded at most once a second per user (bursts of 30), so a script looping on a forbidden route cannot fill the disk.

Every event is written to disk and fsynced before the request that caused it is answered, and is also one audit line in orbit-api's own log (id, action, outcome, actor, target, client_ip, request_id), so a log pipeline receives the trail too.

Reading it in the Deck

/deck/admin/audit shows the window picked in the top bar (the last hour by default): a histogram of events over time, stacked by outcome, where a click on a bar zooms into it; facets by outcome, action and actor, where each facet counts the events matching every other filter; the events, newest first, 100 at a time; and a drawer per event with its changes as a before and after table and the raw JSON. The search box takes plain words, which must each appear in some value of the event, and five fields:

TermMatches
actor:graceThe actor's username
action:user.*The action
outcome:deniedsuccess, denied or failure
ip:10.0.*The client address
target:checkoutPart of the target's type, id or name

The first four match exactly, or as a prefix when the value ends in *. Terms combine with AND, matching ignores case, and "quoted words" stay one term. The page's address keeps the filters, so a link opens the same view.

Reading it over the API

GET /api/v1/audit takes from and to (epoch milliseconds, both inclusive; the last 24 hours by default), actor, action and outcome (exact), q (the search language above), limit (1 to 1000, default 100) and cursor. With the SESSION of a user who holds audit.read (identity.md → "Signing in from a shell"):

bash
curl -fsS -G -H "Authorization: Bearer $ORBIT_AUTH_TOKEN" -H "X-Orbit-Session: $SESSION" \  --data-urlencode 'q=outcome:denied' --data-urlencode 'limit=50' localhost:8080/api/v1/audit

The answer is {events, nextCursor, total, facets: {action, actor, outcome}, interval, histogram: [{t, n, denied, failure}]}. total, the facets and the histogram cover the whole window on every page; interval is the histogram's bucket width in milliseconds, chosen so the window fits in 60 bars. For the next page, send the same filters with cursor set to nextCursor, which is empty on the last page. A filter value longer than the route accepts (q 200 bytes, actor and action 64, outcome 16) is 400.

Exporting CSV

The Deck's "Export CSV" downloads the current filters. Over the API, format=csv streams every event matching the same filters, newest first, at most 100,000 rows, ignoring limit and cursor:

bash
curl -fsS -G -H "Authorization: Bearer $ORBIT_AUTH_TOKEN" -H "X-Orbit-Session: $SESSION" \  --data-urlencode 'format=csv' -o orbit-audit.csv localhost:8080/api/v1/audit

Columns: time (RFC 3339, UTC), actor, role, action, outcome, target_type, target_id, target_name, ip, user_agent, request_id, details (JSON). A cell that starts with =, +, -, @, a tab or a carriage return gets a leading ', so a user agent or a username cannot run as a spreadsheet formula. Both answers are sent with Cache-Control: no-store.

Retention and storage

The trail is append-only JSONL under ORBIT_DATA_DIR/audit, one file per UTC day (2026-09-24.jsonl). ORBIT_AUDIT_RETENTION decides how long a day is kept: days such as 90d (the default) or 365d, or a Go duration of 24h or more; anything shorter refuses to start. Whole day files older than that are deleted when orbit-api starts and when the first event of a new day is written. Size the volume from your own trail: run du -sh on the audit directory after a representative week and scale by the retention.

A query reads the day files its window overlaps, line by line, so a wide window over a long retention costs a scan of those files; the upgrade path the source names is an index in SQLite or Postgres. Backing the directory up is covered in operations.md.

How the client IP is determined

The ip of an event, and the address every per-client limit in orbit-api keys on, is decided in two places.

web decides what it forwards. Every browser request reaches orbit-api through web, so the socket peer orbit-api sees is always web's own address. web passes an X-Forwarded-For header on only when WEB_TRUST_PROXY=true, which says an edge proxy in front of web wrote it; otherwise it sends none, and a header the browser set never crosses.

orbit-api decides what it believes. It reads X-Forwarded-For only on a request that carries its own service bearer, which only web holds, or that comes from a peer in ORBIT_TRUSTED_PROXIES. It walks the header from the right, skips every address in ORBIT_TRUSTED_PROXIES, and records the first other one. Without a vouched header, or with a malformed one, it records the socket peer.

Install pathWEB_TRUST_PROXYWhat ip shows
HelmDerived: true when ingress.enabled, false otherwiseThe client as the ingress saw it
Plain manifeststrue (60-ingress.yaml is the edge)The client as the ingress saw it
Composefalse (web is the edge)web's container address, for every visitor
k3d loopDerived by the chart, true (Traefik)The client as Traefik saw it

ORBIT_TRUSTED_PROXIES is empty by default on every install path, and should stay so: it is for proxies between web and orbit-api (a mesh sidecar, an internal gateway), never web itself, which the bearer already vouches for. With ORBIT_AUTH_MODE=disabled there is no bearer, so only a peer in ORBIT_TRUSTED_PROXIES is believed. And WEB_TRUST_PROXY=true with nothing in front of web lets a browser choose the address it is recorded under.

Behind a cloud load balancer or a CDN

The audit trail records whatever address the ingress controller sees. ingress-nginx and Traefik replace a client's X-Forwarded-For with the address of the connection by default, and an AWS ALB appends the address it saw, so the last entry is always the controller's view of the caller. When the controller sits behind a load balancer or a CDN that hides the client, that view is the load balancer, and every event carries the load balancer's address until the edge passes the real client on. With ingress-nginx, that is its ConfigMap (the chart's controller.config):

KeySet it toWhy
use-forwarded-headers"true"Believe the X-Forwarded-For of the L7 load balancer or CDN in front, instead of replacing it. Default "false".
proxy-real-ip-cidrThe load balancer's or CDN's own ranges (on AWS, the VPC CIDR for an ALB)Whom to believe it from. The default, "0.0.0.0/0", believes anyone, which lets every client forge its address: always narrow it.
compute-full-forwarded-forLeave "false""true" appends the load balancer's address to the chain, and orbit-api, which records the rightmost address it does not trust, would record the load balancer again.
enable-real-ip with forwarded-for-header"true", and the CDN's client header (Cloudflare's is CF-Connecting-IP)For a CDN that names the client in a header of its own; pair it with proxy-real-ip-cidr set to the CDN's published ranges.
use-proxy-protocol"true", with proxy protocol turned on at the load balancerFor an L4 load balancer (an AWS NLB), which cannot add a header. The alternative there is externalTrafficPolicy: Local on the controller's Service.

These settings are controller-wide: they change how every Ingress in the cluster sees its clients, and a wrong proxy-real-ip-cidr lets anyone choose their address. Changing them is the cluster owner's decision, and the chart sets none of them. To check the result, sign in from a known address and look at the ip of your own auth.login event.

The same edge hides the browser's scheme. By default ingress-nginx overwrites X-Forwarded-Proto with the scheme of its own connection, so an ingress with no TLS behind a CDN tells web the browser is on http when it is not. web then sets its cookies without Secure or __Host- and prints http:// in the MCP snippet and the SSO redirect URIs, and orbit-api refuses a plain http redirect URI outside localhost. Set the chart's publicUrl to the https address people open (configuration.md), with no change to the controller. That corrects what web tells the browser and encrypts nothing: passwords, one-time codes and session tokens cross the leg from the CDN to the origin in clear until the origin serves TLS and the CDN verifies it (on Cloudflare, the encryption mode Full (strict)).

The same address keys the per-client limits. When orbit-api has no trusted client address (compose, or an edge that passes none), every visitor arrives from web's address, and sign-in falls back to its per-username waits plus one bucket shared by all visitors, so one visitor's failures cannot lock everyone else out (identity.md → "What a refusal means").

The change timeline

The namespace page and the APM service page each have a Changes tab: what changed in the namespace (or in the service's workload) in the page's time window, newest first, and through which tool or Deck user. Kubernetes keeps events for about an hour (--event-ttl, 1h by default) and an alert's history ends when it resolves, so orbit-api records these itself, from what it already watches, in a loop of its own at the alerting interval (ORBIT_ALERTING_INTERVAL, 30 s). A slow API server holds that loop and not the alert notifications: its reads of field managers share a budget of 10 seconds per tick, and past it a row goes without "who".

KindRecorded whenWho
RolloutA Deployment's newest ReplicaSet revision changes: a new ReplicaSet, at its creation, or an older one brought back by a rollback, at the Deployment controller's event that scaled it up from 0. The row names the revision, the ReplicaSet, and each container's image, requests and limits before and after.The field manager whose write of spec.template is the newest in the 5 seconds before that time, from the Deployment's managedFields. An entry keeps only a tool's last write and the fields it still owns, so a later write can hide an earlier one (a rollback that only removed a field owns nothing), and then nobody is named. Typical names: helm, argocd-controller, kubectl-set, kubectl-rollout, kubectl-client-side-apply.
ScaleA Deployment's spec.replicas changes: one row per step, each at the time of the Deployment controller's ScalingReplicaSet event. While a rollout is in progress its own steps read like a scale, so the row takes the time of the HPA's SuccessfulRescale event when there is one, and else the tick's, with ~. A row the HPA made carries the message of that event.The manager that wrote spec.replicas: kubectl, helm, or HPA when the controller manager wrote it and the HPA's event explains it.
WarningWarning events of one namespace, reason, workload and message's first word (a Liveness failure is not a Readiness one) are one row, however many events and objects it covers. The row goes to disk when it opens, a minute later, then every 10 minutes while it grows: that is how far the tab can lag behind the cluster. A new revision of the workload or an hour closes it and the next event opens a new row, so a chronic failure reads hour by hour and what a rollout's pods do is never folded into what came before. The row keeps the latest message, says how many times, and names the first object with how many more.Nobody.
AlertA critical or warning alert in the namespace starts firing or resolves. Info alerts are left out.Nobody.
Deck actionRead from this trail when the tab opens: scale, resources, suspend, run now, restart, rollback, a terminal opened, an alert acknowledged or silenced. Dry runs are left out.The Deck user.

Nothing guesses a person. A tool's change names the tool, a change with no field manager near it names nobody, and a change orbit-api wrote reads "the Deck": the Deck's own row beside it names the user, for a role with audit.read. The person behind a CI run or a kubectl session is in the cluster's own audit log, which orbit-api does not read. Without audit.read the tab leaves out the Deck's rows and their chip, and the postmortem says they are not visible to the role. The first tick on a cluster takes stock of its Deployments and alerts without writing (what stood before recording began is no change) and copies the Warning events Kubernetes still holds; the tab says since when it records. A change made while orbit-api was down gets its row at the next tick. A shutdown writes what the open rows hold and saves them, so the next start goes on counting them and the counts match the cluster's. A crash loses at most the last 10 minutes of a row's growth, and an event that grew while orbit-api was down counts once.

Times. A row's time is the cluster's: a ReplicaSet's creation, the controller's event, a field manager's write. When the event is gone and no write dates the change, as after a long outage or on a busy Deployment (Kubernetes folds events past 10 alike in 10 minutes), the row takes the time a tick saw it, a rollback stays before the revision that followed it, and the time reads ~ on screen and in the Markdown.

Bounds. One JSONL file per UTC day under ORBIT_DATA_DIR/changes/<cluster>/, kept 14 days: whole day files past that are deleted with the first write of a day. A day takes at most 5,000 rows, at most 4,500 of them Warning rows and at most 500 Warning rows per namespace, however many lines each took, so one namespace's storm leaves room for the others and for rollouts, scales and alerts. A row's later lines stop once the day's file reaches 5 MiB, and the row keeps its last count. What a full day leaves out is counted per namespace and logged once; the tab and the Markdown say how many. A full day is at most about 12 MB, since an event message is cut at 1 KiB, so a cluster's worst case is about 160 MB: size persistence.size with it. One document per cluster under changestate/ holds what the last tick saw. A read answers at most 500 rows, newest first, and the oldest Warnings are the first left out, so a storm cannot hide the rollouts before it. The chips count before that cut, and the tab and the Markdown say when it cut.

Copy as postmortem copies the rows the kind chips leave, oldest first, as Markdown: a title with the scope, the window in UTC, a note on anything missing, an empty Impact section, the timeline as a table (time, kind, object, what changed with a row's last event and whether it began before the window, who), every Deck action of the window as a list, and an empty Cause section. Text from the cluster is escaped, so pasted into a tracker it stays one cell, with no link or image. If the browser refuses the clipboard, the button says so. Nothing opens by itself, no incident is created, and no model writes any of it.

Not recorded: rollouts and scales of StatefulSets and DaemonSets (their revisions live in ControllerRevisions, which orbit-api does not watch), which the tab says; ConfigMap and Secret edits; the events of cluster-scoped objects such as nodes, and alerts with no namespace.

What the trail is not

  • Not tamper-evident. The files are plain JSONL on a volume: anyone who can write to it can edit history. For evidence that survives an administrator, ship the audit log lines to an append-only store you do not administer from the same account.
  • Not a record of reads. A successful read is in the request log, not the trail; a refused one is access.denied.
  • Not complete across a lost volume. With components.orbit-api.persistence.enabled off the directory is an emptyDir, and the trail goes with the pod.

Where to go next

  • The roles and permissions every access.denied names: identity.md.
  • The route, its parameters and its errors in the reference: api.md.
  • Backups, retention sizing and what to alert on: operations.md.