Features
S3 archive
With archive.enabled, every metrics block orbit-stack seals and every trace and log segment it
writes goes to an S3 bucket you own, as it lands. Local retention still deletes data on its own
schedule (15 days of metrics and 7 of traces and logs by default); the bucket keeps it for as long
as its lifecycle rules say. A rehydration copies one time window back, for example a day from two
months ago, and you query it in the Deck for 1 to 180 days. Then it is deleted again.
How the data gets to S3
orbit-stack hard-links each sealed block and segment into <data>/archive/outbox/ the moment it
writes one. The link costs no copy and no work on the ingest path, and the file stays there after
retention or compaction deletes the original, until it is shipped. Every 15 seconds orbit-api reads
the outbox, puts each file in the bucket (multipart above 64 MiB) and removes it from the outbox,
many blocks and segments at once under a 128 MiB gate, a block's files in order. A failed upload
stays queued and is retried; nothing is acknowledged before S3 answers 200.
The objects are the engine's own files, already compressed, under one key per file:
<prefix>/<signal>/dt=YYYYMMDD/hour=HH/<oldest ms>-<newest ms>-<name>[/<block file>]<signal> is metrics, traces or logs. The day and hour are those of the newest data in the
file. A metrics block is six objects, meta.json last, and a rehydration reads a block only once
its meta.json is there.
Partitions
Partition attributes keep one value's data apart in the bucket, so a rehydration filtered on that value copies only its part. Set up to two labels per signal:
helm upgrade sreorbit deploy/helm/sreorbit --reuse-values \
--set 'archive.partitions.metrics={namespace}' --set 'archive.partitions.logs={service}'- Metrics. orbit-stack splits each sealed block into one block per value of the labels, a
series without the label going to the empty value. It copies the series records and their
timestamp columns as they are, with nothing decoded or compressed again, when orbit-api next
lists the outbox (one block at a time; until then the staged block is hard links), and packs the parts back to back into one object,
<block>.pack. Beside it,<block>.pack.partsmaps each part's label values to the byte range of its files. A rehydration reads the map, then fetches only the parts its filter selects with ranged GETs, and orbit-stack writes each piece straight into the part's block files. - Logs. A segment stays one object, its streams in label order, so one service's blocks sit
together. Beside it,
<segment>.partsmaps each value of the labels (service, namespace, pod, container, node) to the blocks that hold its streams. A filtered rehydration reads the map, then copies the segment's sections and those blocks with ranged GETs and leaves the rest of the file a hole. A read narrowed to the value reads only blocks with its streams, the ones copied. - Traces and spans are not partitioned: a trace crosses services, and the segment keeps traces in start order for the newest-first search. A filtered rehydration copies their files whole.
A filter prunes when it names a partition label: {namespace="payments"}, =~, != and !~
for metrics, and an unquoted service:api term for logs. Any other filter copies the window
whole. orbit-stack still applies the full filter to every read, so a pruned rehydration answers
the same as the live store does for that filter.
Lookup attribute. trace_id is the high-cardinality attribute a log filter can look up. Each
log segment's map carries the trace-id bloom of every block, built when orbit-stack cuts the
segment, with or without partitions. A rehydration or an Archive Search with a
trace_id:<32 hex digits> term copies only the blocks whose bloom can hold that id, so finding one
trace's lines from a month ago reads a few blocks per segment instead of the day.
Pick labels with tens or hundreds of values: each value adds a part to the pack and an entry to
its map, two objects per seal whatever the count. Until orbit-api ships it, the pack is a copy on
orbit-stack's volume, about the block's size. A change to the labels applies to what orbit-stack
seals from then on; older objects keep their layout (earlier builds wrote one six-object block
per value under .../hour=HH/<label>=<value>/), and a rehydration reads both.
Setting it up on EKS
- Create the bucket: private, Block Public Access on, default encryption on, and a lifecycle rule that aborts incomplete multipart uploads after a day. Add your own transition or expiration.
- Give orbit-api's role the bucket. EKS binds one role per ServiceAccount, so the policy goes on the role orbit-api already has (or a new one):
CLUSTER=my-eks NAMESPACE=sreorbit RELEASE=sreorbit ARCHIVE_BUCKET=my-archive \
deploy/aws/setup-pod-identity.shThe policy allows s3:ListBucket on the bucket and s3:PutObject, s3:GetObject and
s3:AbortMultipartUpload on its objects, nothing else.
3. Open orbit-api's egress to S3 and name the bucket, in the chart or in the Deck:
helm upgrade sreorbit deploy/helm/sreorbit --reuse-values \
--set archive.enabled=true --set archive.bucket=my-archiveWith archive.enabled=true and no archive.bucket, nothing is archived until an admin adds an
archive in Admin > Archive (below).
Archives in the Deck
Admin > Archive (settings.manage) shows Set up the archive until an archive exists, then
the Archives list. Each archive has a name, a bucket, prefix and region, a storage class, an
encryption (the bucket's default, SSE-S3, or SSE-KMS with a key ARN, id or alias), a scan limit per
rehydration in GB, the signals it takes (metrics, traces and spans, logs) and an inclusion filter.
Add archive, Edit, the arrows, Turn off and Delete change the list; the
Partitions panel sets the partition labels every archive's files split by.
orbit-api sends each queued file to the first archive in the list, top to bottom, that is on, takes
the file's signal and whose filter matches the file. The filter is label matchers over the
partition labels, as {namespace=~"prod-.*", app!="debug"}; a label the data lacks reads as empty,
and an empty filter takes everything. A metrics block goes partition by partition: a block whose
partitions match different archives becomes one smaller pack per archive, each with its own map. A
log segment is cut by its partition values before it ships: one smaller segment per value, with its
own map, under <label>=<value>/ in the bucket, so {service="checkout"} takes checkout's
streams out of a segment that holds fifty services. Saving adds each label a filter names to the
partitions of the signals its archive takes; a filter can name service, namespace, pod,
container or node for logs, and two labels per signal at most. Traces go whole, segment by
segment: an archive with a filter takes metrics and logs only. Two archives cannot share a bucket
and prefix, or sit one inside the other. A file no archive takes leaves the queue unarchived and
counts as skipped on its signal's row. Put an archive with no filter last to catch the rest.
Logs that an ingestion rule drops or samples out still reach the archive, as Datadog archives every
ingested log, indexed or not. orbit-stack writes them to a store no page reads, under
<data>/logs/dropped/, and deletes each of its segments once the archive has queued it. A
rehydration brings them back with the rest, so the window shows what arrived, not only what the
rules kept. Metrics and spans the rules drop are not archived.
Save and test puts a 25-byte object, <prefix>/.orbit-write-test, with the class and
encryption you picked, lists <prefix>/, then deletes the object (a role without
s3:DeleteObject leaves it; the next test overwrites it). orbit-api tests each new archive and each
whose bucket, prefix, region, class or encryption changed. If S3 refuses the write or the list,
nothing changes: the form and the panel show S3's answer, for example
archive cold: ... s3 PUT 403 AccessDenied, until a save goes through. A save that goes through:
- takes the place of the chart's
archive.*destination from then on, restarts included (orbit-api keeps the list in its data directory,archive/archives.json). An archive the Deck saved before archives were a list (archive/config.json) becomes the first one, nameddefault; - keeps every place an archive left, when you move it to another bucket, prefix or region or delete
it, as a read-only source named
<archive>-<YYYYMMDDHHMM>(UTC). Nothing new goes there; a rehydration still reads it. The Deck asks before a move or a delete; - on the first save, with no archive before, turns metrics, traces and logs on.
The next shipping cycle, within 15 seconds, follows the new list. The shipping counters (uploaded, waiting, last upload, newest in S3, last failure) are kept per signal and per archive, saved with each cycle that moves a file or fails, so a restart of orbit-api keeps them. An archive whose last upload failed shows Failing and S3's answer under the list.
| Value | Default | What it does |
|---|---|---|
archive.bucket | empty | The bucket. Empty: an admin sets it in Admin > Archive. |
archive.signals | [metrics, traces, logs] | What goes to the bucket until Admin > Archive changes it. traces carries their spans; a signal left out stays local only. |
archive.prefix | sreorbit | The first part of every key. |
archive.region | empty | The bucket's region; empty is the pod's AWS_REGION, else us-east-1. |
archive.storageClass | STANDARD | STANDARD, STANDARD_IA, ONEZONE_IA, GLACIER_IR or INTELLIGENT_TIERING. |
archive.encryption | empty | Empty keeps the bucket's default; AES256 forces SSE-S3; aws:kms forces SSE-KMS with archive.kmsKeyId. |
archive.kmsKeyId | empty | The KMS key's ARN, with aws:kms. The role then needs kms:GenerateDataKey and kms:Decrypt on it. |
archive.maxScanGB | 100 | The most a rehydration may read from the bucket. |
archive.retentionDays | [1, 3, 7, 15, 30, 45, 60, 90, 180] | The retention periods a rehydration may pick. |
archive.maxLogs | 0 | The default log limit of a rehydration; 0 is none, and a rehydration may set its own. |
archive.partitions.metrics | [] | Up to two metric labels each block is split by. |
archive.partitions.logs | [] | Up to two of service, namespace, pod, container, node each log segment is mapped by. |
archive.restrictions | {} | Restriction queries on rehydrated data, by role id (metrics, logs, traces). |
On IRSA, put eks.amazonaws.com/role-arn in components.orbit-api.serviceAccount.annotations
instead of step 2's association.
Rehydrating
When a Deck page's range starts before what orbit-stack keeps (ORBIT_STACK_RETENTION for
metrics, ORBIT_STACK_TRACE_RETENTION for traces, ORBIT_STACK_LOGS_RETENTION for logs; the chart
copies them to the web as ORBIT_RETENTION_METRICS, ORBIT_RETENTION_TRACES and
ORBIT_RETENTION_LOGS), a note above the page says from when it has data. For an admin, the note
links to Admin > Rehydrations with the missing window and the signal filled in.
In the Deck, Admin > Archive has one row per signal (metrics, logs, traces, spans): its
archive on or off (or failing, with the last error), the archives that take it in order and the
labels it is split by, what goes (everything it stores; spans ride in the trace segments),
the last upload, the age of the newest data in the bucket, what was uploaded and what waits, and
Turn off or Turn on (audited as archive.signals.update; it holds through restarts, and what
is queued when you turn a signal off still ships). The page reads all this again every 15 s. Under
it sit the partitions and the read-only sources. Each row's Bring back opens
Admin > Rehydrations with that signal picked, where you create one:
- the archive to read, when there is more than one: an archive or a read-only source by name, or every one of them (the default), each place read once;
- a window (
fromMs,toMs) and the signals to bring back (metrics, traces, logs, or all three); - an optional filter per signal: label matchers for metrics (
{namespace="payments"}), a log query (service:api severity:>=WARN), a trace service and operation; - a name (a lowercase letter, then lowercase letters, digits and
-) and a retention fromarchive.retentionDays(15 days by default); - optionally, a log limit: the job keeps the oldest log lines up to it, whole segments at a time, and reports how many it kept;
- optionally, alert channels from Alerts > Notifications to tell when the job ends: Slack, Microsoft Teams or Google Chat get its name, status, window, scan size, file count and a link.
Estimate lists the objects the window touches and adds up their size before anything runs,
next to the room orbit-stack's volume has for rehydrations: its free space less a reserve of a
tenth of the volume, 1 GiB at least, which stays for the live store. A window past
archive.maxScanGB is refused with 413, and one past the room with 507; narrow it or delete a
rehydration. orbit-stack checks the room again on every piece it writes, so two jobs started
together cannot fill the volume either.
A job goes pending, running (with files and bytes copied so far), then active, or failed
with the reason. A failed job leaves no file on orbit-stack's volume. Every job,
however many run, shares one copy gate of 64 MiB: a file takes its size, 16 MiB at most, so four
big files or up to 64 small ones (a partition's) copy at once, and orbit-api holds at most 128 MiB
of rehydration data at once.
Delete on a running job cancels it: the copies in flight finish their piece, and the job opens
with the files it copied whole, active, its note saying how many of how many. On an active job,
Delete stops its reads and keeps its data for a day: Restore brings it back as it was until then, and after that the
data goes. Ended jobs stay listed for a year, through restarts of orbit-api, and a name can be
used again once its job has ended. The audit trail has an event when a job is created, deleted,
restored and when it ends, with its status, scan size, files and log lines, and one when shipping
starts failing (archive.shipping.failed, with the error) and when it recovers.
Reading rehydrated data in the Deck
The Logs explorer, the APM traces explorer (with the trace view) and Explore each have an
Index picker next to the page's actions: Live, or one of the active rehydrations that holds
that signal. A rehydration picked shows From the archive:
An active rehydration answers the engine's own read API under
/api/v1/archive/rehydrations/<name>/q/: PromQL (api/v1/query, api/v1/query_range,
api/v1/series, labels), api/v1/logs with its histogram and facets, api/v1/traces, a trace by
id and the APM routes. Every read is narrowed to the job's window and filter, so a filtered
rehydration never shows what the filter left out. Reads need the signal's own permission
(telemetry.query, logs.read, apm.read); creating, deleting and the archive's status need
settings.manage. Every create and delete is in the audit trail.
A filter on a partition label narrows what the job copies, and the estimate shows the scan size
next to the window's (windowBytes). Any other filter applies when you read: the job copies the
window's files whole, and you narrow the window to read less.
Archive Search
Admin > Rehydrations > Archive Search runs one read over the form's window and filters, straight from the bucket, and leaves no rehydration behind. orbit-api copies the files the read needs (the partitions its filters name, when they do) into a scratch store on orbit-stack, answers the read through your role's restriction query and deletes the store. The scan limit and the volume's room apply as to a rehydration. Use it for a lookup: one trace id, one user, one namespace's errors a month ago.
Restriction queries
archive.restrictions gives a role a restriction query on rehydrated data: label matchers for
metrics, log query terms and the one service whose traces it reads, keyed by the role's id.
archive:
restrictions:
viewer:
metrics: '{namespace="payments"}'
logs: "namespace:payments"
traces: checkoutA member of viewer reads every rehydration with these ANDed into the job's filter, in the
Deck's pages and in the API alike: PromQL selectors gain the matchers, log reads gain the terms.
Trace searches and span reads keep the service, and a trace opens with that service's spans only.
The service list and the service map would name every service, so the role gets 403 there, and
a job filtered to another service answers 403 too. Roles without an entry read the job's filter
alone.
Limits
- A rehydration's window is at most 31 days, and eight rehydrations are open at a time. Together they cache what one live store caches.
- A log limit counts whole segments, so a job keeps up to the limit, not exactly the limit.
- A rehydration lists at most 200,000 objects per signal.
- Glacier Flexible Retrieval and Deep Archive need a restore first; move those objects back to a class from the table before you rehydrate them.
- orbit-stack keeps the outbox on its own volume until orbit-api ships it: while the bucket is
unreachable, the outbox grows by what the engine seals.
GET /api/v1/archiveshows the backlog, and orbit-api's/metricsexportsorbit_archive_backlog_bytes,orbit_archive_last_shipped_timestamp_secondsandorbit_archive_last_error_timestamp_secondsto alert on. - The files are orbit-stack's own block and segment formats. Read them back with orbit-stack.
API
| Route | Permission | What it does |
|---|---|---|
GET /api/v1/archive | settings.manage | The configuration (config, null before any archive exists: archives in order, retired read-only sources, retentionDays, maxLogs; configError when the chart's values are invalid; setIn, deck or chart; refused, the last refused save), the shipping counters, byArchive (each archive's counters by name), and signals: per signal, on, its partitions, its own shipping (with newestMs and skippedFiles) and archives, the names of the archives that take it, in order. |
PUT /api/v1/archive/config | settings.manage | {archives: [{name, bucket, prefix, region, storageClass, encryption, kmsKeyId, maxScanGB, filter, signals, enabled}], partitions: {metrics, logs}}: the whole list, in order, over the chart's values, after a test write to each new or changed archive. 32 archives at most, one name each. 400 with S3's answer when a bucket refuses it, or when a value is not one S3 takes; 502 when orbit-stack refuses the partitions. Audited as archive.config.update, refusals included. |
PUT /api/v1/archive/signals | settings.manage | {"metrics": true, "traces": false, "logs": true}: what orbit-stack queues from now on, kept through restarts; a signal left out keeps its state. Audited. |
POST /api/v1/archive/estimate | settings.manage | {fromMs, toMs, signals, metrics, logs, archive} (archive: one archive or read-only source by name; none reads them all): the objects and bytes the window reads after the partitions the filters leave out (scanBytes, and windowBytes before), per archive (byArchive), whether any archive's share is over its limit (overLimit; maxScanBytes is their limits together), and orbit-stack's room (freeBytes, overFree). |
GET /api/v1/archive/rehydrations | settings.manage or logs.read | Every rehydration with its status, progress, expiry, log limit and lines kept. |
POST /api/v1/archive/rehydrations | settings.manage | Starts one: {name, fromMs, toMs, signals, metrics, logs, service, operation, retentionDays, maxLogs, notify, archive}. 202, 413 past the scan limit, or 507 past the volume's room, both with the estimate. |
DELETE /api/v1/archive/rehydrations/<name> | settings.manage | Cancels a running job, which opens with the files it copied whole; on an active one, stops its reads and keeps the data a day (purgeMs). |
POST /api/v1/archive/rehydrations/<name>/restore | settings.manage | Undoes a delete within that day. |
POST /api/v1/archive/search | settings.manage | Archive Search: {fromMs, toMs, signals, metrics, logs, service, operation, read, params}, read an engine read path (/api/v1/logs, /api/v1/query, ...). Answers the read's status with {scanBytes, windowBytes, files, secsToCopy, secs, answer}; 413 past the scan limit or the room. |
GET /api/v1/archive/rehydrations/<name>/q/<read> | the signal's | A read of the rehydrated data, narrowed to its window and filter. A path segment that is an escaped . or .. (%2e%2e) is refused with 400 on every route. |