Skip to content

Features

Ingestion control

What the engine stores of the spans and log lines that reach it, and so what the trace and log explorers can find. Span metrics, the service graph and the log volume counters count everything that arrives whatever the rules say, so the RED charts, the service map and the alerts built on them never change; the rules decide only what is stored, searchable and retained. The default rules keep everything. The page is /deck/admin/ingestion, and changing the rules needs the settings.manage permission, which only the built-in admin role holds (see identity.md). The routes and shapes are CONTRACT.md Addendum v0.21.

What the rules decide

A trace is kept or dropped whole, the same on every span: the decision hashes the trace id, so a trace kept at 10% is also kept at 25%, and a lower rate keeps a subset of what a higher one keeps. For each trace, in this order:

  1. With "Keep every trace with an error" on, a trace with any failed span is kept.
  2. With "Keep slow traces" on, a trace whose root span (or, until the root arrives, its longest span) lasts at least the threshold is kept.
  3. The exclusion filters run top to bottom against the trace's root span (its earliest span, until the root arrives): the first enabled filter that matches decides, and the whole trace is kept at that filter's rate.
  4. Everything else is kept at its service's rate, or at the default rate when the service has none of its own. The service is the root span's (the first span's, until the root arrives).

A log line is decided on its own: the first enabled log filter that matches keeps it at its rate, and a line no filter matches is kept at the default log rate. A line that carries a trace id is sampled on that id, so it follows its trace.

The engine applies a saved change at once, with no restart, and keeps the rules on its volume.

The rules document

json
{
  "version": 7,
  "updatedAt": "2026-09-28T03:10:00Z",
  "updatedBy": "ana@example.com",
  "traces": {
    "defaultRate": 0.25,
    "keepErrors": true,
    "keepSlowerThanMs": 2000,
    "services": [{ "service": "payments-api", "rate": 1.0 }],
    "exclusions": [
      {
        "id": "health-checks",
        "name": "Health checks",
        "enabled": true,
        "service": "*",
        "conditions": [{ "key": "attr.url.path", "op": "regex", "value": "(^|/)(health(z|check|-check)?|ready(z|ness)?|live(z|ness)?|ping)/?$" }],
        "keepRate": 0.0
      }
    ]
  },
  "logs": {
    "defaultRate": 1.0,
    "exclusions": [
      {
        "id": "debug-trace",
        "name": "Drop DEBUG and TRACE",
        "enabled": true,
        "service": "*",
        "conditions": [{ "key": "severity", "op": "lte", "value": "DEBUG" }],
        "keepRate": 0.0
      }
    ]
  }
}
  • version is the engine's: it starts at 1 and grows by one per accepted save. updatedBy is the email of the signed-in user who saved, set by orbit-api.
  • A rate is a number from 0 to 1 (the page shows percentages). keepSlowerThanMs is 0 for off, else 1 to 3,600,000.
  • At most 500 service rates, each service named once, and at most 50 filters per signal. A filter's id is lower-case letters, digits and hyphens, unique per signal; its name is up to 80 characters; service is one service's exact name or *; it has 1 to 10 conditions, and all of them must match.

A condition is a key, an operator and a value. A key an item does not carry never matches, and that includes neq.

SignalKeys
Tracesname, kind, status, durationMs, attr.<span attribute>, resource.<resource attribute>
Logsseverity, body, service, namespace, pod, container, node, attr.<attribute>

The operators are eq, neq, contains, regex, in (the value is a comma-separated list), gte and lte. gte and lte compare numbers, and levels for severity, in the order TRACE, DEBUG, INFO, WARN, ERROR, FATAL. A line's level is its severity number, else its severity text read as the log explorer reads it (notice as INFO, err as ERROR, critical as FATAL). A regex runs on the engine's regex engine: no look-around, no back-references, at most 1,024 characters.

Using the page

The headline strip reads the last 5 minutes, hour or day: spans and log lines stored per second against what arrived, the storage a month of that takes, and the share saved against keeping everything.

The Traces tab sets the default rate, the two keep rules, a rate per service beside what each service sends and stores (an empty box means the default), and the trace filters. The Logs tab sets the default log rate and the log filters, lists what each service sends and stores, and shows the retention, which is the engine's to set and read-only here. Each tab offers its common filters in one click:

FilterWhat it matches
Health checksattr.url.path or attr.http.route matching (^|/)(health(z|check|-check)?|ready(z|ness)?|live(z|ness)?|ping)/?$, as two filters
Kubernetes probesattr.user_agent.original containing kube-probe
Prometheus scrapesattr.url.path equal to /metrics
Static assetsserver spans whose attr.url.path ends in a script, style sheet, image or font extension
Redis blocking readsattr.db.system equal to redis and a span name such as BLPOP, BRPOP or XREAD
Drop DEBUG and TRACEseverity at most DEBUG
Keep 1% of INFOseverity equal to INFO, kept at 1%
Health-check linesa body matching GET or HEAD on the usual health paths
kube-probe linesa body containing kube-probe

"Add filter" opens the builder: a name, the service it applies to, the share of matches it keeps, and its conditions. A filter is switched on and off in place, and its order changes by dragging it or with its arrows; the first match wins, so order matters. A saved filter shows what it dropped in the stats window.

Preview impact asks the engine to replay the last hour of stored data under the draft, per service and per filter. It can only show what that data would lose: an item dropped before it was stored is not there to count, so a higher rate shows no gain.

Review and save shows the draft as a list of changes. The save sends If-Match with the version the page read; if someone saved in the meantime the engine refuses it with 412, and the page loads their version and says the draft was not saved. An accepted save is recorded in the audit trail as ingestion.rules.update with a one-line summary of the change, and the page links to those events.

The Metrics tab

The rules cover spans and log lines; the Metrics tab shows where the metric series are, read only, from the engine's head now: its active series, the series it created in the last hour (increase(orbit_stack_head_series_created_total[1h]), the churn), the ten metrics with the most series with their share of the total, and the ten labels with the most values. A metric that jumps up the first list, or a label like pod or id at the top of the second, is where a cardinality bomb starts. It reads GET /api/v1/status/tsdb and GET /api/v1/query through orbit-api (telemetry.query); without that permission, or on demo data, there is no tab. Rules that drop metric series are not part of this page yet.

Over the API

With the SESSION of a user who holds settings.manage (identity.md → "Signing in from a shell"):

RouteAnswer
GET /api/v1/ingestion/rulesThe rules
PUT /api/v1/ingestion/rulesThe stored rules; 412 when If-Match names an old version, 400 with the field when a value is refused
POST /api/v1/ingestion/preview{window, traces: {spans, keptSpans, byService, byRule}, logs: {records, keptRecords, byRule}} for a draft
GET /api/v1/ingestion/stats?window=5m|1h|24hWhat arrived and what was stored, per service and per filter, the storage, and the monthly estimate
bash
curl -fsS -H "Authorization: Bearer $ORBIT_AUTH_TOKEN" -H "X-Orbit-Session: $SESSION" \  localhost:8080/api/v1/ingestion/rules > rules.json# edit rules.json, then save it against the version readcurl -fsS -X PUT -H "Authorization: Bearer $ORBIT_AUTH_TOKEN" -H "X-Orbit-Session: $SESSION" \  -H "Content-Type: application/json" -H "If-Match: $(jq .version rules.json)" \  --data @rules.json localhost:8080/api/v1/ingestion/rules

Demo data

With NEXT_PUBLIC_DEMO_MODE left on, the page shows the demo estate's rules (version 7) and its traffic: every service of orbit-prod-use1, its probes, scrapes and static files, and its log lines. The preview is worked out in the page from that traffic, the same way the engine would replay it, and a save stays in the browser tab: nothing is sent and nothing is applied.