Features
Log explorer
The engine stores the container log lines the node agent ships, under the ingestion rules
(ingestion.md), and the Deck searches them at /deck/logs. Reading them
needs the logs.read permission, which the built-in viewer, editor and admin roles all
hold (see identity.md). The routes and shapes are CONTRACT.md Addendum
v0.21.
How lines arrive
Collection is off until the chart turns it on with telemetry.logs.enabled (default false);
telemetry.logs.excludeNamespaces leaves whole namespaces out. With it on:
orbit-node, the DaemonSet, tails/var/log/pods/<ns>_<pod>_<uid>/<container>/*.login the CRI format and joins partial lines. It keeps its read positions in an emptyDir, so a container restart neither repeats nor skips a line. A new pod starts each container's log at the first line written in the last 120 s (the chart'stelemetry.logs.lookbackSeconds;0starts at the end) and reports ready only once it has placed every file. A rollout stops a node's pod, which reads every file to its end as it stops, then starts the new one: what the node wrote in between arrives once and the rest of those 120 s arrives twice, with its original timestamps. The same holds for a pod deleted or evicted outside a rollout. A node writing 100 lines a second sends at most 12,000 lines again per new pod, so anything that counts lines (error counts, volume) counts those twice. Still skipped: what a node wrote more than 120 s before the new pod's first pass, when it stayed that long without a pod; the oldest lines of a container that wrote more than 16 MiB in those 120 s; a rotation already compressed when the pod starts, which takes two rotations of one container in 120 s; and a log whose lines carry no CRI timestamp.- It reads the level from JSON (
level,severity,lvl,log.level,level_name, as a word or as pino's, OTLP's or Monolog's number), from logfmt (level=), from klog prefixes and from a leadingINFO,WARN,ERROR,DEBUG,TRACEorFATALtoken. It reads trace and span ids from JSON (trace_id,traceId,trace.id,span_id,spanId) or from a W3Ctraceparent. - It ships OTLP/HTTP batches to
orbit-collector, which adds the same Kubernetes attributes it adds to spans and exports to the engine. When the collector is down the agent drops its oldest lines first and counts the drops; it never blocks the node.
orbit-node also reads log files that applications write on the node; see "Application log
files" below.
A line's service is the resource's service.name, else the Deployment, StatefulSet or
DaemonSet that owns the pod, else the container's name.
A line that arrives with no level gets the one its text names, which the engine reads from its
first KiB: a klog prefix, an HTTP access line's status (5xx ERROR, 4xx WARN, else INFO), level=,
lvl= or a JSON "level" (pino's numbers too), a bracketed level ([WARN]), a level and a colon
at the start of the line (.NET's fail:) or in upper case (ERROR: after Postgres's
UTC [pid]) or after failed, Valkey's and Redis's marker after the time (* INFO, # WARN,
. and - DEBUG), or an upper-case level word (Kafka's ] INFO [). A line that goes on the one before
it from the same container, indented or Postgres's STATEMENT: and DETAIL:, takes that line's
level. The histogram, the facets, severity: terms, the patterns and the analytics count that
level, and the list draws its dot as a ring. The ingestion rules do not: a rule on the level acts on
the level the line itself carries, so a line with none is kept as before. Lines stored before the
engine began reading levels get theirs read from their text when you query them, the same way, so
the bars, facets and patterns of a window that spans that moment use one level source. A line whose
text names none shows as "No level".
The engine refuses a record larger than 1 MiB, and every record of a resource whose attributes encode to more than 64 KiB, and counts them in the OTLP partial success; it stores the rest of the request.
Application log files
Some applications log to a file rather than to stdout. The case this is built for is Laravel:
Monolog writes storage/logs/laravel.log, or laravel-YYYY-MM-DD.log with the daily channel,
and puts each error's stack trace on the lines after it. Inside a PHP pod that is
/var/www/storage/logs.
The file reaches the node when the pod mounts that directory from a hostPath. This pod keeps
/var/www/storage/logs in /var/log/application/checkout-api on its node:
spec:
containers:
- name: app
volumeMounts:
- name: app-logs
mountPath: /var/www/storage/logs
volumes:
- name: app-logs
hostPath:
path: /var/log/application/checkout-api
type: DirectoryOrCreateThe chart points orbit-node at the node side of that mount, one directory per service:
telemetry:
logs:
enabled: true
files:
- path: /var/log/application/{service}/laravel*.loglaravel*.log takes both laravel.log and the daily laravel-2026-10-28.log.
- The path is a glob on the node:
*and?match within one directory.{service}matches like*and becomes the line'sservice.name, socheckout-apiabove is the service.{namespace}setsk8s.namespace.namethe same way. The keysserviceandnamespaceset fixed names instead; a source uses the key or the placeholder, not both. Lines with neither carry no service. - What is mounted. The directory before the first wildcard (
/var/log/applicationabove) is mounted read-only intoorbit-node, and nothing else. It may not be/or a system directory (/etc,/proc,/sys,/dev,/run,/home,/root,/usr,/var/lib/kubelet, the container runtimes' directories,/var/log/pods), sit under one, or hold one:/var/log/*.logis refused because/var/logholds/var/log/pods. The chart refuses such a path when it renders, andorbit-noderefuses it again when it starts. Give the application logs a directory of their own. - Entries. A line that matches
multilineStartopens an entry. The default matches Laravel's and Monolog's header,[2026-10-28 15:00:00] production.ERROR:or[2026-10-28T15:00:00.123456+00:00] app.INFO:. Every other line ([stacktrace], a#0 /var/www/...frame, a blank line,[previous exception]) joins the entry above it, up to 256 KiB. The last entry goes out once a second passes with no new line. A line before the first header is a line of its own.multilineStart: "^"makes every line an entry. - Level, time and trace. The header gives the level: DEBUG; INFO and NOTICE as INFO;
WARNING as WARN; ERROR; CRITICAL, ALERT and EMERGENCY as FATAL. It gives the time too, and a
time without an offset, Laravel's default, is read as UTC: an app whose
app.timezoneis not UTC should write the offset. A line without a header takes the time it was read. Trace and span ids come from a"trace_id"and"span_id"in the line, or from atraceparent. - Attributes.
service.name,k8s.namespace.namewhen known,k8s.node.name,log.file.path(the path on the node) andlog.source(file). Query them asservice:checkout-apior@log.file.path:/var/log/application/checkout-api/*. - Reading works as for container logs, except that a new pod starts at the end of every
file: the positions in the same emptyDir, a file that appears later (tomorrow's daily file)
from its start, a rename (logrotate's default) or a copytruncate read through, a restart from
where it stopped, a rotation compressed to
.gzwhileorbit-nodewas down read from where it stood.excludeis a glob on the whole path whose files are never read; its*crosses directories (exclude: "*.gz"). Symlinks are not followed, and neither a file that is not a regular one (a FIFO an app swaps in is opened without waiting and skipped) nor a name with a control character in it is read. The lines every file leaves unfinished hold at most 16 MiB together; past that, a file's unfinished line and entry go out as they stand.orbit-nodereads as uid 65534 with group 0, so a file only its owner can read is skipped, andorbit-node's own log says so once. - Ingestion control applies to these lines as to any other: an exclusion on
service:checkout-apiwithseverity:<=INFOdrops them here too.
What it does not do:
- Duplicates. An app that logs to the file and to stdout or stderr, such as Laravel's
stackchannel withstderrin it, arrives twice: once from the container log and once from the file. Keep one of the two. - Which pod wrote a line. Replicas on one node that share the hostPath write one file, which is read once. Its lines carry the service, the namespace and the node, never the pod or the container, so a pod's "Stored logs" leaves them out and "Lines around" shows the file's lines.
- Logs that never reach the node. A pod whose
/var/www/storage/logsis an emptyDir, or no volume at all, keeps the file inside the pod, out oforbit-node's reach. Mount the directory from a hostPath as above, or send the log to stderr (LOG_CHANNEL=stderr, orstderradded to thestackchannel) so the container log carries it.orbit-nodenever mounts/var/lib/kubelet/podsor the runtime's directories to look inside pods: they hold every pod's secrets.
The query language
Terms are separated by spaces, and a line must satisfy every term. An empty query matches everything.
| Term | Matches |
|---|---|
service:checkout-api | A key holding exactly that value, ignoring case |
service:cart,checkout | A comma between values takes any of them (quoted, "a,b" is one value) |
-namespace:kube-system | A leading minus turns any term around |
pod:checkout-web-* | An asterisk matches any run of characters |
severity:>=WARN | Levels in the order TRACE, DEBUG, INFO, WARN, ERROR, FATAL; >= and <= compare |
@log.iostream:stderr | An attribute of the line |
@route:"GET /api/cart" | A key with no dot also reads a JSON message's own key, when the line carries no such attribute |
@status:503 | The line's HTTP status: an attribute, its JSON message's status or status_code, or an access line's code |
shape:"http request" | A line whose pattern is exactly this one (a pattern's plus and minus write it) |
timeout | A word anywhere in the message, ignoring case |
"connection refused" | A phrase anywhere in the message, ignoring case |
/timeout after \d+ms/ | A regular expression on the message |
The keys are service, namespace, pod, container, node, severity, trace_id,
span_id and shape. A key a line does not carry never matches, so trace_id:<id> finds only lines that
carry an id, and -trace_id:<id> keeps every line without one. A regular expression runs on
the engine's regex engine, which has no look-around and no back-references; (?i) at its start
ignores case, and it may be at most 1,024 characters long. The page checks the grammar before
it sends a query and says what is wrong where you typed it; a query the engine still refuses
comes back as 400 with the engine's own reason.
The engine skips the blocks a term cannot match. A word goes by its letters, and an id of letters and digits by a filter of the ids in each block, so an id no line holds answers without reading the messages. A search for a long id skips almost every block without it; one under 10 characters often has nothing to look up and reads the messages. The top values of an attribute come from each block's own counts, unless the key is not among the block's first 16, it has more than 256 values there, more than one per 4 records or about a thousand counts by stream and severity, a bucket boundary crosses the block or the query judges the lines themselves. Segments written before these indexes answer the same, by reading more.
As you type, the bar suggests keys and the values the window holds. Enter runs the query as you typed it. Tab takes the first suggestion; the arrow keys pick another one for Enter or Tab.
Reading the page
- The window is the top bar's time range, like every other Deck page. The query and the window
are kept in the address (
?q=and?range=), so a view is a link that opens as it was. - The histogram stacks the lines of each bucket by level, errors on top. Drag across it to zoom into a stretch, or click a column to zoom into its bucket; on the keyboard, the arrow keys move between columns and Enter zooms. The top bar's zoom out undoes it.
- Under the histogram, a line counts the window's ERROR and FATAL lines; a click shows only them. When they stepped up, to 3 times their earlier rate or more and held there (their average from that bucket on stays at it, over 20 lines at least, and the newest buckets still run at twice the rate), it says since when, the earliest bucket that holds, and a click opens the error lines from that moment on. The earlier rate is the interquartile mean of the earlier buckets, so a burst early in the window does not hide a later step, and a spike that falls back is no step. A burst that ended (5 times the rate at its peak, 20 lines at least) reads "a burst of N errors at HH:MM, back to the earlier rate since HH:MM", and a click opens its lines. Two engine reads beside the page's own add the 3 services that wrote most of them and their most frequent shape, each a click to its own lines.
- The facets, as Datadog's: service, level (the worst first), namespace, workload (deployment,
statefulset or daemonset), pod, container, node, image tag, version and HTTP status code, the
last four only when a line of the window carries them. The status comes from an attribute, a JSON
message's status key or an access line's code, with 4xx and 5xx in colour; the engine reads every
message for it, so it fills in a moment after the first screen. Each counts under the query without its
own terms, so every value stays listed while one is checked. A value's click checks it and joins
the values already checked (
service:cart,checkout, any of them), or unchecks it; "Only" beside it keeps that value alone, the minus excludes it, and "All" beside a filtered facet's name takes its terms out of the query. A facet with more than 10 values has a box that filters them, among its 50 with the most lines. The first five facets start open, and so does any facet the query filters on; the rest open with a click and say how many values they hold. - The lines come newest first, 200 at a time, and "Load more" continues from where the last
page stopped. Only the rows in view are in the page, so thousands of lines scroll as fast as
a few dozen. Each row carries its level's colour down its left edge, and an ERROR or FATAL row
has a red tint. A row opens in place to its fields, its whole message (pretty-printed when it is
JSON, each number as the body wrote it) and its attributes; when those would fall below the
list, the line moves to its top. The open line gives its time in full, to the millisecond, with
how long ago it was written, and says so when the engine read its level from its text. Each field and each plain attribute has a plus that adds
key:value(or@name:value) to the query and a minus that excludes it, and beside it a chart button counts the lines by it over time (see the analytics below). A message also lists its own fields, with no parsing rule to set up: a JSON object's leaves as dotted names (also behind a prefix such as a timestamp and a level), or the pairs of a logfmt line (two or more). A plus on a JSON string, such asorder_id, adds the phrase"\"<id>\"", the id as a whole JSON string, so a request id inside the message filters like an attribute and the engine skips every block without it. A string a JSON encoder may escape (accents, slashes, quotes,<) adds a regex that finds it however the body holds it, such asn\u00e3ofornão. A plus on a number or a boolean adds a regex for exactly what the body wrote under that key (/"status"\s*:\s*502\s*[,}\]]/, and12.0stays12.0), and one on a logfmt pair a regex for the whole pair (took=1does not findtook=12); a regex reads every body in the window. Arrays are shown without the plus. A terminal's colour codes never show: they are taken out of the list, the open line and the lines around, and stay in "Copy as JSON"; a plus on a coloured pair still finds its line, since the engine reads the body as stored. "Trace of this log" sits at the top right of the open line, where the trace page has "Logs for this trace", for a reader who holdsapm.readand a line with a trace id; it opens the trace on the line's span. A line with no trace id says so, and offers its pod's traces in the 15 minutes either side (a file's line, having no pod, its service's). A line read from a file says it has no pod. "Lines around" sits under it, and the lines it reads show right after the message. Below the fields: "Lines of this trace", "Copy as JSON" and "Copy link" (the five minutes up to the line under the query, the line open and the list ending at it, with the page's columns and wrap; a link whose line has left the window says so, and a window you then move to shows its own newest lines), then the Related panel: the line's pod, node, workload and service, with their logs, traces, events and alerts in the 15 minutes either side. - "Lines around" shows up to 25 lines before and 25 after the open one from the same pod and container (the same file on the same node, or the same service, when the line has no pod), oldest first, scrolled to the open line, which is marked. It reads the 15 minutes before and the minute after the line, and when more than 1,000 lines fill that minute, only the second after it, so the lines right after it show.
- "Patterns" beside "Lines" groups the query's lines by shape: numbers, ids, addresses and the
values of
key=valueturn into*, and a JSON message counts by itsmsgormessagewith the keys that say why it failed beside it when it failed (a 4xx or 5xx status, or an error class): its HTTP status, method, route (else its path, a segment holding a digit as*) and error class, so a 503 is a row apart, shown ashttp request status=503 route=GET /api/v1/archive/rehydrations, while the requests that did not fail share the bare message, whatever their route. FATAL, ERROR and WARN rows come first. An access line and astatus=pair keep their HTTP status too. The newest 1,000 lines answer first; one engine read then counts every line in the window, a row per pattern, level and service as Datadog splits them, for the 50 rows with the most lines, with each one's trend over the window (a read that runs out of time says so, and an engine without that read leaves the sample, where a row names its worst level and the service that wrote most of it). A row shows its count, share and trend (in its level's colour from WARN up), its level and its service, its*marked, and opens to its newest 20 lines in the window, each with its time and level, read through the same terms as its plus, with the values each*held in them, most frequent first. Its plus shows only its lines: its level and service andshape:"<pattern>", which the engine checks against each line's own pattern, keys and all, after skipping the blocks without the pattern's longest word. Its minus leaves the pattern out with-shape:"<pattern>"and stays on the patterns, so the noise comes off one pattern at a time andok *never takesbroken pipewith it, nor a bare message its failed requests. The page's auto-refresh waits while the patterns are on screen, and the refresh button counts them again. - "Time series", "Top list", "Table" and "Pie" beside "Patterns" count or measure every line the
query matches in the window. "Show" picks the measure: the count of lines, the unique count of a
field, the average, minimum, maximum, sum, p50, p90, p95 or p99 of a number the newest lines
carry (an attribute, or a JSON message's own key such as
duration_ms), or the error ratio (ERROR and FATAL lines over all lines). A measure ranks the values highest first, reads in its unit (ms or s, KB or MB, a percent) and stays in the address (?m=p95:@duration_ms) and in saved views; the table adds its column and a row for all lines, and up to two more measures as columns, read with the first so every row has a value. A share shows only beside a count or a sum. The split is by one field: the service (the default), the level, namespace, pod, container, node, message pattern or any attribute of the newest lines, its top 5, 8, 10, 25 or 50 values (a pie 5 or 8, one colour each) and the rest as one. It is one engine read,GET /api/v1/logs/aggregate, the one a log dashboard panel makes. A time series stacks the values per bucket (drag across it to zoom), a top list and a pie give each value's share, and the table adds each value's trend over the window. A plus or a value adds its term to the query, and a minus excludes it. The view, the field and the top live in the address (?tab=,?by=,?top=) and in saved views. A rehydrated index answers no aggregate, so it shows lines and patterns only. - Above the lines, "Wrap" shows up to three lines of each message, and "Columns" picks what stands
between the level and the message, up to six: the fields and any attribute of the newest lines
(or the column button beside an attribute of an open line). Columns narrow together to leave the
message its room. Both live in the address (
?wrap=1,?cols=) and in saved views. - An empty query bar offers your last five searches in this browser, and
/anywhere on the page comes back to the bar. - "Saved views" under the query bar holds your team's views of this page: see "Saved views" below.
- The list is one stop on the keyboard: the arrow keys (or
jandk), Page Up, Page Down, Home and End move, Enter opens and closes a line, Escape closes it. With a line open, the arrow keys open the previous or next line in its place. - "Live" reads what arrived every 2 s, and the histogram and the facets every 16 s; a hidden tab reads nothing. It holds still while you read: scrolled away from the newest line, a line open or text selected, new lines wait and are counted, and "Resume", or coming back to the newest line, brings them in. Lines from different nodes arrive out of order, so each read goes back 15 s behind the newest line. A line further behind than that shows after a reload, as does one over 5 minutes late. One read returns 500 lines at most: when more arrive, the list says so and shows the newest of each read, the rest of those 2 s show after a reload, and a narrower query shows every line. While live, the header gives the lines a second in the histogram's last whole bucket.
Five other pages lead here. The trace view's "Logs for this trace" opens trace_id:<id> over
the trace and 15 minutes either side; when no line in the window carries the id the page says so,
and one click keeps the window and swaps the id for the service. A service's "Top error messages" groups the
newest 500 error lines of its window by shape (by its workload's pods when no line names the
service), and a row opens that line, with the query and window kept, and its trace beside it. A dashboard's log list opens its line: the five minutes up
to it under the panel's query, the line open (line= in the address names it). A workload's logs tab has "Stored logs" beside the live
tail, for namespace:<ns> pod:<workload>-*, or for the pod picked there, with what the search box holds; a live
line that writes a trace id links that trace. A pod page has "Stored
logs" for namespace:<ns> pod:<name>.
Custom facets
Any attribute of a log line or of a span can become a facet for the whole team. Open a line (or a
span in a trace) and use the facet button beside the attribute: "Add user_id as a facet". From that
moment the engine counts the attribute's values as the lines and spans arrive, and a "Group chart"
link takes you to the explorer with the chart grouped by it. Adding or removing a facet takes the
dashboards.write permission plus the signal's read permission (logs.read or apm.read), and the
audit trail records who did it (facets.add, facets.remove).
- The rail lists the custom facets under "Custom facets", up to 10 per explorer, each with its top
values and counts. A value adds
@key:valueto the query (key:valueon the traces page), the minus excludes it, the chart button groups the chart by the facet, and the cross removes it for everyone.(other)and a value cut at 128 bytes show their counts but filter nothing. - "Manage" beside the title lists every facet of that explorer (50 at most) with who added it and when. There you pick up to 10 for your own rail, group the chart by any of them, or remove one for the team. Your pick lives in a cookie of this browser, as Datadog keeps hidden facets per user; without one the rail shows the first 10 added.
- The counts cover the whole window. With a query of levels (logs) or a status (traces) alone, the rail reads the facet's series. With any other term, one engine read counts every facet of your rail under the whole query: every line the query matches, or every matching trace once per trace that carries the value. That read also covers data from before the facet existed. When one read meets more than 4,096 distinct values of an attribute (a user id, say), the rail says its counts are lower bounds.
- Grouping draws the values with the most lines or spans in the window, 10 at most, per step of the
chart; on the traces page it can show their mean or p95 latency instead. p95 reads each value's
latency heatmap at every query, one read per value, and draws the p95 of its duration rows per
column, as
histogram_quantilereads them on the APM pages: every trace of the value, and a duration filter does not narrow it. With a query of levels or a status alone it reads the facet's series. With any other term the logs page asks the engine for the 10 values with the most matching lines in the window and their lines per step, in one read. The traces page searches once per value: each gets its exact total, spread over time as a uniform sample of up to 500 traces falls. It draws the sampled traces' 10 most frequent values first, then the 10 of the rail's engine read when that read lands and names other values. - Under such a term the rail shows the loaded lines' (or sampled traces') counts at once and replaces them with the engine's when its read lands. On the logs page that read starts after the chart's, so the two do not compete for the engine's cores.
- The series behind a query of levels or a status start when the facet is added. Nothing from
before is counted there, and the panel says since when. A search for
@key:valuestill reads every stored line, older ones included. - Each facet keeps its first 500 values apart; later values count under
(other)until the engine restarts. A value longer than 128 bytes is cut there. - The engine keeps the list in
<data dir>/facets.jsonand writes the counts as the seriesorbit_facet_logs{facet,value,severity},orbit_facet_spans{facet,value,status_code}andorbit_facet_span_seconds{facet,value,status_code}. Each sample holds what arrived since the one before, sosum_over_timegives the exact count of any window.
Saved views
A saved view is a name, an optional description and the page's state: the query, the time range, "Lines" or "Patterns", and the columns and wrap (on the Traces page: the query, the range, root traces or all spans, and background roots). A rolling range such as the last 7 days stays rolling: the view opens on the 7 days that end when you click it. The cluster stays yours.
- Every view is the team's. Anyone who reads the page sees it, with who created it and who
edited it last. Its owner, or anyone who holds
dashboards.write, renames, updates or deletes it; the rest of the team saves a copy instead. - "Saved views (N)" opens the list, starred first. One click on a name opens the filtered list. The star puts a view next to the query bar for you alone.
- The pin makes a view your default for that page: opening Logs (or Traces) with no query of its own opens that view, on any browser you sign in to. "Open without a view" opens the page as it comes. A page you have no default on does not wait for the list.
- With a view on screen, the bar names it and says "modified" once you change the query, the range or the grouping. "Update view" saves the change into it; "Save as new" keeps both.
- Views that this browser kept before the shared list show "Import N views saved in this browser" once; the import posts them as your views and clears the browser's copy.
- If orbit-api does not answer the list within 2 seconds, the bar says the views did not load; reload the page to try again.
Views live in orbit-api's data directory (views/<id>.json), so they survive a restart, and
every save and delete is in the audit trail (view.save, view.delete, access.denied) with
the view's name and page, never its query. On demo data they live in memory, and a demo
visitor's views stay with that visitor's session for a day at most.
| Route | Who | Answer |
|---|---|---|
GET /api/v1/views?page=logs|traces | logs.read or apm.read, per page | {views: [{id, page, name, description, query, owner, editor, createdAt, updatedAt, favorite, default, writable}]}, starred first |
POST /api/v1/views | the page's read permission | {page, name, description, query}; 201 with the view |
PUT /api/v1/views/{id} | its owner or dashboards.write | same body, same page; 200 |
DELETE /api/v1/views/{id} | its owner or dashboards.write | 204 |
PUT /api/v1/views/{id}/prefs | the page's read permission | {favorite, default}, either or both, yours alone; 200 |
name is 1 to 120 characters, description at most 500, query a URL query string of at most
8 KiB with no control characters. One account keeps at most 500 views.
Over the API
With the SESSION of a user who holds logs.read (identity.md → "Signing
in from a shell"):
| Route | Answer |
|---|---|
GET /api/v1/logs/query?start&end&q&limit&cursor | {records, next, truncated}, newest first; limit 1 to 1000, default 200 |
GET /api/v1/logs/histogram?start&end&q&buckets | {step, buckets: [{t, counts}]}; buckets 1 to 240, default 60 |
GET /api/v1/logs/facets?start&end&q&fields&limit | {facets: {service: [{value, count}]}}; limit 1 to 100, default 20. Up to 10 key (attribute names) add custom: {key: {values: [{value, count}], capped}}: each key's top values under the whole query, the line's own attribute first, then its resource's |
GET /api/v1/logs/aggregate?start&end&q&by&top&buckets&timeout counts what the dashboards'
log panels draw (dashboards.md → "Log panels"): the lines q matches in the
window, in total and in buckets equal steps (0 to 240, 0 for the window as one), split by by
(service, namespace, pod, container, node, severity, message or @name) into its
top values (1 to 50, default 10). The answer is {start, end, step, buckets, total, totals, groups: [{value, count, counts}], other, missing, distinct, capped, partial, ingest}; split by
message, each group adds worst (its worst level), service (the service that wrote most of its
lines) and services (how many wrote it). other
counts the lines with another value, missing those without the field, each {count, counts};
counts and totals come only with buckets. partial says the deadline or the scan budget cut
the read (3 s and 4 GiB for an hour or less, 6 s and 8 GiB up to a day, 10 s and 16 GiB beyond;
timeout, 100 to 10000 ms, shortens it). capped says a scan thread met more than 1,024 values
of the split. ingest is {seen, kept, since, levels, service}: the lines that arrived in the
window and those stored, by the engine's counters, scoped to the query's levels, else to its one
service, else every line, from since on, the first minute counted, which is never before the
engine started. A line the receiver refused for a malformed field is in neither count: ingestion
control never saw it, and orbit_stack_log_records_rejected_total counts it. A service past a
minute's 500 has no count of its own that minute, so service is left out and the counts are of
every line; a service name keeps its first 128 bytes in these counters and in /api/v1/logs/stats.
start and end are epoch milliseconds. For the next page send the same query with cursor
set to next, which is null on the last page. A record carries ts (RFC 3339 with
nanoseconds), tsNs, severity and severityNumber, severityInferred (true when the engine
read the level from the body), body, service, namespace, pod, container, node,
traceId, spanId and attrs, and leaves out the fields it does not have.
curl -fsS -G -H "Authorization: Bearer $ORBIT_AUTH_TOKEN" -H "X-Orbit-Session: $SESSION" \ --data-urlencode 'q=severity:>=WARN -namespace:kube-system' \ --data-urlencode "start=$(( ($(date +%s) - 3600) * 1000 ))" \ --data-urlencode "end=$(( $(date +%s) * 1000 ))" \ localhost:8080/api/v1/logs/queryDemo data
With NEXT_PUBLIC_DEMO_MODE left on, the page reads the demo estate's own lines: every
workload of orbit-prod-use1, in the formats those programs write (zap JSON, Spring Boot,
PostgreSQL, Elasticsearch, logfmt), stored under the demo's saved ingestion rules, with the
14:26 incident in the payments path and promo-engine crash-looping. They are made up, the
same on every read, and the live tail keeps running past the demo's fixed clock.