Skip to content

Features

S3 archive

With archive.enabled, every metrics block orbit-stack seals and every trace and log segment it writes goes to an S3 bucket you own, as it lands. Local retention still deletes data on its own schedule (15 days of metrics and 7 of traces and logs by default); the bucket keeps it for as long as its lifecycle rules say. A rehydration copies one time window back, for example a day from two months ago, and you query it in the Deck for 1 to 180 days. Then it is deleted again.

How the data gets to S3

orbit-stack hard-links each sealed block and segment into <data>/archive/outbox/ the moment it writes one. The link costs no copy and no work on the ingest path, and the file stays there after retention or compaction deletes the original, until it is shipped. Every 15 seconds orbit-api reads the outbox, puts each file in the bucket (multipart above 64 MiB) and removes it from the outbox, many blocks and segments at once under a 128 MiB gate, a block's files in order. A failed upload stays queued and is retried; nothing is acknowledged before S3 answers 200.

The objects are the engine's own files, already compressed, under one key per file:

text
<prefix>/<signal>/dt=YYYYMMDD/hour=HH/<oldest ms>-<newest ms>-<name>[/<block file>]

<signal> is metrics, traces or logs. The day and hour are those of the newest data in the file. A metrics block is six objects, meta.json last, and a rehydration reads a block only once its meta.json is there.

Partitions

Partition attributes keep one value's data apart in the bucket, so a rehydration filtered on that value copies only its part. Set up to two labels per signal:

text
helm upgrade sreorbit deploy/helm/sreorbit --reuse-values \
  --set 'archive.partitions.metrics={namespace}' --set 'archive.partitions.logs={service}'
  • Metrics. orbit-stack splits each sealed block into one block per value of the labels, a series without the label going to the empty value. It copies the series records and their timestamp columns as they are, with nothing decoded or compressed again, when orbit-api next lists the outbox (one block at a time; until then the staged block is hard links), and packs the parts back to back into one object, <block>.pack. Beside it, <block>.pack.parts maps each part's label values to the byte range of its files. A rehydration reads the map, then fetches only the parts its filter selects with ranged GETs, and orbit-stack writes each piece straight into the part's block files.
  • Logs. A segment stays one object, its streams in label order, so one service's blocks sit together. Beside it, <segment>.parts maps each value of the labels (service, namespace, pod, container, node) to the blocks that hold its streams. A filtered rehydration reads the map, then copies the segment's sections and those blocks with ranged GETs and leaves the rest of the file a hole. A read narrowed to the value reads only blocks with its streams, the ones copied.
  • Traces and spans are not partitioned: a trace crosses services, and the segment keeps traces in start order for the newest-first search. A filtered rehydration copies their files whole.

A filter prunes when it names a partition label: {namespace="payments"}, =~, != and !~ for metrics, and an unquoted service:api term for logs. Any other filter copies the window whole. orbit-stack still applies the full filter to every read, so a pruned rehydration answers the same as the live store does for that filter.

Lookup attribute. trace_id is the high-cardinality attribute a log filter can look up. Each log segment's map carries the trace-id bloom of every block, built when orbit-stack cuts the segment, with or without partitions. A rehydration or an Archive Search with a trace_id:<32 hex digits> term copies only the blocks whose bloom can hold that id, so finding one trace's lines from a month ago reads a few blocks per segment instead of the day.

Pick labels with tens or hundreds of values: each value adds a part to the pack and an entry to its map, two objects per seal whatever the count. Until orbit-api ships it, the pack is a copy on orbit-stack's volume, about the block's size. A change to the labels applies to what orbit-stack seals from then on; older objects keep their layout (earlier builds wrote one six-object block per value under .../hour=HH/<label>=<value>/), and a rehydration reads both.

Setting it up on EKS

  1. Create the bucket: private, Block Public Access on, default encryption on, and a lifecycle rule that aborts incomplete multipart uploads after a day. Add your own transition or expiration.
  2. Give orbit-api's role the bucket. EKS binds one role per ServiceAccount, so the policy goes on the role orbit-api already has (or a new one):
text
CLUSTER=my-eks NAMESPACE=sreorbit RELEASE=sreorbit ARCHIVE_BUCKET=my-archive \
  deploy/aws/setup-pod-identity.sh

The policy allows s3:ListBucket on the bucket and s3:PutObject, s3:GetObject and s3:AbortMultipartUpload on its objects, nothing else. 3. Open orbit-api's egress to S3 and name the bucket, in the chart or in the Deck:

text
helm upgrade sreorbit deploy/helm/sreorbit --reuse-values \
  --set archive.enabled=true --set archive.bucket=my-archive

With archive.enabled=true and no archive.bucket, nothing is archived until an admin adds an archive in Admin > Archive (below).

Archives in the Deck

Admin > Archive (settings.manage) shows Set up the archive until an archive exists, then the Archives list. Each archive has a name, a bucket, prefix and region, a storage class, an encryption (the bucket's default, SSE-S3, or SSE-KMS with a key ARN, id or alias), a scan limit per rehydration in GB, the signals it takes (metrics, traces and spans, logs) and an inclusion filter. Add archive, Edit, the arrows, Turn off and Delete change the list; the Partitions panel sets the partition labels every archive's files split by.

orbit-api sends each queued file to the first archive in the list, top to bottom, that is on, takes the file's signal and whose filter matches the file. The filter is label matchers over the partition labels, as {namespace=~"prod-.*", app!="debug"}; a label the data lacks reads as empty, and an empty filter takes everything. A metrics block goes partition by partition: a block whose partitions match different archives becomes one smaller pack per archive, each with its own map. A log segment is cut by its partition values before it ships: one smaller segment per value, with its own map, under <label>=<value>/ in the bucket, so {service="checkout"} takes checkout's streams out of a segment that holds fifty services. Saving adds each label a filter names to the partitions of the signals its archive takes; a filter can name service, namespace, pod, container or node for logs, and two labels per signal at most. Traces go whole, segment by segment: an archive with a filter takes metrics and logs only. Two archives cannot share a bucket and prefix, or sit one inside the other. A file no archive takes leaves the queue unarchived and counts as skipped on its signal's row. Put an archive with no filter last to catch the rest.

Logs that an ingestion rule drops or samples out still reach the archive, as Datadog archives every ingested log, indexed or not. orbit-stack writes them to a store no page reads, under <data>/logs/dropped/, and deletes each of its segments once the archive has queued it. A rehydration brings them back with the rest, so the window shows what arrived, not only what the rules kept. Metrics and spans the rules drop are not archived.

Save and test puts a 25-byte object, <prefix>/.orbit-write-test, with the class and encryption you picked, lists <prefix>/, then deletes the object (a role without s3:DeleteObject leaves it; the next test overwrites it). orbit-api tests each new archive and each whose bucket, prefix, region, class or encryption changed. If S3 refuses the write or the list, nothing changes: the form and the panel show S3's answer, for example archive cold: ... s3 PUT 403 AccessDenied, until a save goes through. A save that goes through:

  • takes the place of the chart's archive.* destination from then on, restarts included (orbit-api keeps the list in its data directory, archive/archives.json). An archive the Deck saved before archives were a list (archive/config.json) becomes the first one, named default;
  • keeps every place an archive left, when you move it to another bucket, prefix or region or delete it, as a read-only source named <archive>-<YYYYMMDDHHMM> (UTC). Nothing new goes there; a rehydration still reads it. The Deck asks before a move or a delete;
  • on the first save, with no archive before, turns metrics, traces and logs on.

The next shipping cycle, within 15 seconds, follows the new list. The shipping counters (uploaded, waiting, last upload, newest in S3, last failure) are kept per signal and per archive, saved with each cycle that moves a file or fails, so a restart of orbit-api keeps them. An archive whose last upload failed shows Failing and S3's answer under the list.

ValueDefaultWhat it does
archive.bucketemptyThe bucket. Empty: an admin sets it in Admin > Archive.
archive.signals[metrics, traces, logs]What goes to the bucket until Admin > Archive changes it. traces carries their spans; a signal left out stays local only.
archive.prefixsreorbitThe first part of every key.
archive.regionemptyThe bucket's region; empty is the pod's AWS_REGION, else us-east-1.
archive.storageClassSTANDARDSTANDARD, STANDARD_IA, ONEZONE_IA, GLACIER_IR or INTELLIGENT_TIERING.
archive.encryptionemptyEmpty keeps the bucket's default; AES256 forces SSE-S3; aws:kms forces SSE-KMS with archive.kmsKeyId.
archive.kmsKeyIdemptyThe KMS key's ARN, with aws:kms. The role then needs kms:GenerateDataKey and kms:Decrypt on it.
archive.maxScanGB100The most a rehydration may read from the bucket.
archive.retentionDays[1, 3, 7, 15, 30, 45, 60, 90, 180]The retention periods a rehydration may pick.
archive.maxLogs0The default log limit of a rehydration; 0 is none, and a rehydration may set its own.
archive.partitions.metrics[]Up to two metric labels each block is split by.
archive.partitions.logs[]Up to two of service, namespace, pod, container, node each log segment is mapped by.
archive.restrictions{}Restriction queries on rehydrated data, by role id (metrics, logs, traces).

On IRSA, put eks.amazonaws.com/role-arn in components.orbit-api.serviceAccount.annotations instead of step 2's association.

Rehydrating

When a Deck page's range starts before what orbit-stack keeps (ORBIT_STACK_RETENTION for metrics, ORBIT_STACK_TRACE_RETENTION for traces, ORBIT_STACK_LOGS_RETENTION for logs; the chart copies them to the web as ORBIT_RETENTION_METRICS, ORBIT_RETENTION_TRACES and ORBIT_RETENTION_LOGS), a note above the page says from when it has data. For an admin, the note links to Admin > Rehydrations with the missing window and the signal filled in.

In the Deck, Admin > Archive has one row per signal (metrics, logs, traces, spans): its archive on or off (or failing, with the last error), the archives that take it in order and the labels it is split by, what goes (everything it stores; spans ride in the trace segments), the last upload, the age of the newest data in the bucket, what was uploaded and what waits, and Turn off or Turn on (audited as archive.signals.update; it holds through restarts, and what is queued when you turn a signal off still ships). The page reads all this again every 15 s. Under it sit the partitions and the read-only sources. Each row's Bring back opens Admin > Rehydrations with that signal picked, where you create one:

  • the archive to read, when there is more than one: an archive or a read-only source by name, or every one of them (the default), each place read once;
  • a window (fromMs, toMs) and the signals to bring back (metrics, traces, logs, or all three);
  • an optional filter per signal: label matchers for metrics ({namespace="payments"}), a log query (service:api severity:>=WARN), a trace service and operation;
  • a name (a lowercase letter, then lowercase letters, digits and -) and a retention from archive.retentionDays (15 days by default);
  • optionally, a log limit: the job keeps the oldest log lines up to it, whole segments at a time, and reports how many it kept;
  • optionally, alert channels from Alerts > Notifications to tell when the job ends: Slack, Microsoft Teams or Google Chat get its name, status, window, scan size, file count and a link.

Estimate lists the objects the window touches and adds up their size before anything runs, next to the room orbit-stack's volume has for rehydrations: its free space less a reserve of a tenth of the volume, 1 GiB at least, which stays for the live store. A window past archive.maxScanGB is refused with 413, and one past the room with 507; narrow it or delete a rehydration. orbit-stack checks the room again on every piece it writes, so two jobs started together cannot fill the volume either.

A job goes pending, running (with files and bytes copied so far), then active, or failed with the reason. A failed job leaves no file on orbit-stack's volume. Every job, however many run, shares one copy gate of 64 MiB: a file takes its size, 16 MiB at most, so four big files or up to 64 small ones (a partition's) copy at once, and orbit-api holds at most 128 MiB of rehydration data at once.

Delete on a running job cancels it: the copies in flight finish their piece, and the job opens with the files it copied whole, active, its note saying how many of how many. On an active job, Delete stops its reads and keeps its data for a day: Restore brings it back as it was until then, and after that the data goes. Ended jobs stay listed for a year, through restarts of orbit-api, and a name can be used again once its job has ended. The audit trail has an event when a job is created, deleted, restored and when it ends, with its status, scan size, files and log lines, and one when shipping starts failing (archive.shipping.failed, with the error) and when it recovers.

Reading rehydrated data in the Deck

The Logs explorer, the APM traces explorer (with the trace view) and Explore each have an Index picker next to the page's actions: Live, or one of the active rehydrations that holds that signal. A rehydration picked shows From the archive: beside the picker, and on the trace view, linked to Admin > Rehydrations. With a rehydration picked, the page reads that job's stores over the job's window, with every filter, facet, histogram and chart it has on live data; the live tail is off. Admin > Rehydrations links each active job to the three pages. Download CSV saves what a page lists, live or rehydrated: the loaded log lines (5,000 at most), the listed traces, or one Explore query's points from its inspector.

An active rehydration answers the engine's own read API under /api/v1/archive/rehydrations/<name>/q/: PromQL (api/v1/query, api/v1/query_range, api/v1/series, labels), api/v1/logs with its histogram and facets, api/v1/traces, a trace by id and the APM routes. Every read is narrowed to the job's window and filter, so a filtered rehydration never shows what the filter left out. Reads need the signal's own permission (telemetry.query, logs.read, apm.read); creating, deleting and the archive's status need settings.manage. Every create and delete is in the audit trail.

A filter on a partition label narrows what the job copies, and the estimate shows the scan size next to the window's (windowBytes). Any other filter applies when you read: the job copies the window's files whole, and you narrow the window to read less.

Admin > Rehydrations > Archive Search runs one read over the form's window and filters, straight from the bucket, and leaves no rehydration behind. orbit-api copies the files the read needs (the partitions its filters name, when they do) into a scratch store on orbit-stack, answers the read through your role's restriction query and deletes the store. The scan limit and the volume's room apply as to a rehydration. Use it for a lookup: one trace id, one user, one namespace's errors a month ago.

Restriction queries

archive.restrictions gives a role a restriction query on rehydrated data: label matchers for metrics, log query terms and the one service whose traces it reads, keyed by the role's id.

text
archive:
  restrictions:
    viewer:
      metrics: '{namespace="payments"}'
      logs: "namespace:payments"
      traces: checkout

A member of viewer reads every rehydration with these ANDed into the job's filter, in the Deck's pages and in the API alike: PromQL selectors gain the matchers, log reads gain the terms. Trace searches and span reads keep the service, and a trace opens with that service's spans only. The service list and the service map would name every service, so the role gets 403 there, and a job filtered to another service answers 403 too. Roles without an entry read the job's filter alone.

Limits

  • A rehydration's window is at most 31 days, and eight rehydrations are open at a time. Together they cache what one live store caches.
  • A log limit counts whole segments, so a job keeps up to the limit, not exactly the limit.
  • A rehydration lists at most 200,000 objects per signal.
  • Glacier Flexible Retrieval and Deep Archive need a restore first; move those objects back to a class from the table before you rehydrate them.
  • orbit-stack keeps the outbox on its own volume until orbit-api ships it: while the bucket is unreachable, the outbox grows by what the engine seals. GET /api/v1/archive shows the backlog, and orbit-api's /metrics exports orbit_archive_backlog_bytes, orbit_archive_last_shipped_timestamp_seconds and orbit_archive_last_error_timestamp_seconds to alert on.
  • The files are orbit-stack's own block and segment formats. Read them back with orbit-stack.

API

RoutePermissionWhat it does
GET /api/v1/archivesettings.manageThe configuration (config, null before any archive exists: archives in order, retired read-only sources, retentionDays, maxLogs; configError when the chart's values are invalid; setIn, deck or chart; refused, the last refused save), the shipping counters, byArchive (each archive's counters by name), and signals: per signal, on, its partitions, its own shipping (with newestMs and skippedFiles) and archives, the names of the archives that take it, in order.
PUT /api/v1/archive/configsettings.manage{archives: [{name, bucket, prefix, region, storageClass, encryption, kmsKeyId, maxScanGB, filter, signals, enabled}], partitions: {metrics, logs}}: the whole list, in order, over the chart's values, after a test write to each new or changed archive. 32 archives at most, one name each. 400 with S3's answer when a bucket refuses it, or when a value is not one S3 takes; 502 when orbit-stack refuses the partitions. Audited as archive.config.update, refusals included.
PUT /api/v1/archive/signalssettings.manage{"metrics": true, "traces": false, "logs": true}: what orbit-stack queues from now on, kept through restarts; a signal left out keeps its state. Audited.
POST /api/v1/archive/estimatesettings.manage{fromMs, toMs, signals, metrics, logs, archive} (archive: one archive or read-only source by name; none reads them all): the objects and bytes the window reads after the partitions the filters leave out (scanBytes, and windowBytes before), per archive (byArchive), whether any archive's share is over its limit (overLimit; maxScanBytes is their limits together), and orbit-stack's room (freeBytes, overFree).
GET /api/v1/archive/rehydrationssettings.manage or logs.readEvery rehydration with its status, progress, expiry, log limit and lines kept.
POST /api/v1/archive/rehydrationssettings.manageStarts one: {name, fromMs, toMs, signals, metrics, logs, service, operation, retentionDays, maxLogs, notify, archive}. 202, 413 past the scan limit, or 507 past the volume's room, both with the estimate.
DELETE /api/v1/archive/rehydrations/<name>settings.manageCancels a running job, which opens with the files it copied whole; on an active one, stops its reads and keeps the data a day (purgeMs).
POST /api/v1/archive/rehydrations/<name>/restoresettings.manageUndoes a delete within that day.
POST /api/v1/archive/searchsettings.manageArchive Search: {fromMs, toMs, signals, metrics, logs, service, operation, read, params}, read an engine read path (/api/v1/logs, /api/v1/query, ...). Answers the read's status with {scanBytes, windowBytes, files, secsToCopy, secs, answer}; 413 past the scan limit or the room.
GET /api/v1/archive/rehydrations/<name>/q/<read>the signal'sA read of the rehydrated data, narrowed to its window and filter. A path segment that is an escaped . or .. (%2e%2e) is refused with 400 on every route.