Features
CloudWatch metrics
With cloudwatch.enabled, orbit-agent reads AWS CloudWatch metrics, RDS by default, and writes them
to orbit-stack beside everything else it collects. Dashboards, alerts and Orbit SQL then query RDS
CPU, connections or IOPS with PromQL, like any other series.
What is collected and how it is named
Each entry of cloudwatch.metrics is Namespace:Metric:Statistic. The default is eight RDS metrics:
| Metric | Statistic | Series |
|---|---|---|
CPUUtilization | Average | aws_rds_cpuutilization_average |
DatabaseConnections | Average | aws_rds_database_connections_average |
FreeableMemory | Average | aws_rds_freeable_memory_average |
FreeStorageSpace | Average | aws_rds_free_storage_space_average |
ReadIOPS | Sum | aws_rds_read_iops_sum |
WriteIOPS | Sum | aws_rds_write_iops_sum |
DMLLatency | Average | aws_rds_dmllatency_average (Aurora) |
BlockedTransactions | Sum | aws_rds_blocked_transactions_sum (Aurora) |
The names follow prometheus/cloudwatch_exporter: aws_<namespace>_<metric>_<statistic> in snake case
(an underscore only where a lowercase letter or digit meets an uppercase one), with one label per
CloudWatch dimension in the same snake case (DBInstanceIdentifier becomes dbinstance_identifier)
and job="aws_rds". Points keep CloudWatch's own 1-minute timestamps.
So a Grafana dashboard written for cloudwatch_exporter works unchanged:
max(aws_rds_cpuutilization_average{job="aws_rds"}) by (dbinstance_identifier)A Grafana v2 dashboard (dashboard.grafana.app/v2) imports through Dashboards → Import, and its
CloudWatch panels (namespace, metric name, statistic) are translated to the same PromQL.
Latency
CloudWatch itself publishes RDS metrics about 1 to 2 minutes after the minute they describe. On top of
that, orbit-agent reads every cloudwatch.interval (3 minutes by default) and asks for the whole
interval plus three minutes each time, so a point shows in the Deck at most one interval after
CloudWatch has it and no minute is skipped, however long the interval. Points that arrive late in
CloudWatch are picked up on the next read. A read that fails (throttling, credentials) leaves its
minutes to the next one that goes through, which reads back to three minutes before the last read
that did, two hours at most; a restart of orbit-agent starts over from the last six minutes.
Cost
GetMetricData has no free tier: AWS bills $0.01 per 1,000 metrics requested, whatever the
number of points each one returns. Asking for the last minutes at 1-minute resolution in one read
therefore costs the same as asking for one point. The interval sets the bill; the resolution leaves it unchanged:
monthly cost = series × (30 × 24 × 60 / interval_minutes) × 0.01 / 1000For 10 instances × 8 metrics (80 series):
| Interval | Reads a day | Per month |
|---|---|---|
| 1m | 1,440 | $34.56 |
| 2m | 720 | $17.28 |
| 3m (default) | 480 | $11.52 |
| 5m | 288 | $6.91 |
| 10m | 144 | $3.46 |
| 30m | 48 | $1.15 |
A longer interval only delays the points: every read still brings each minute since the last one, so 10m or 30m cuts the bill by the same factor and keeps 1-minute resolution.
Only series CloudWatch lists as recently active (ListMetrics with RecentlyActive) are requested: a
deleted instance stops costing within hours. Discovery calls (ListMetrics) are billed as ordinary
API requests, a negligible amount at this rate.
Configuration
| Value | Env on orbit-agent | Default |
|---|---|---|
cloudwatch.enabled | ORBIT_AGENT_CLOUDWATCH_ENABLED | false |
cloudwatch.interval | ORBIT_AGENT_CLOUDWATCH_INTERVAL | 3m (1m minimum) |
cloudwatch.region | ORBIT_AGENT_CLOUDWATCH_REGION | the pod's AWS_REGION, else us-east-1 |
cloudwatch.metrics | ORBIT_AGENT_CLOUDWATCH_METRICS | the eight RDS metrics above |
cloudwatch.serviceAccount.annotations | {}; IRSA's role ARN goes here |
It needs telemetry.agent with engine orbit. With networkPolicy.enabled, it opens orbit-agent's
egress to TCP 443 (CloudWatch and STS; private ranges and instance metadata stay closed) and to the
Pod Identity agent at 169.254.170.23:80.
IAM: the same read-only role
orbit-agent uses the role orbit-api uses for the bill (docs/finops-aws.md), with two more read-only actions:
| Action | Why |
|---|---|
cloudwatch:GetMetricData | the points, up to 500 series per call |
cloudwatch:ListMetrics | which series exist and are recently active |
The policy is deploy/aws/finops-readonly-policy.json. It names no account and no resource.
EKS Pod Identity (recommended). Run the script again: it rewrites the inline policy and adds the
association for <release>-orbit-agent, leaving what already exists.
CLUSTER=my-eks NAMESPACE=sreorbit RELEASE=sreorbit deploy/aws/setup-pod-identity.shhelm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values --set cloudwatch.enabled=trueIRSA. Give the role the trust in deploy/aws/trust-irsa.json for
system:serviceaccount:<namespace>:<release>-orbit-agent, then:
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \ --set cloudwatch.enabled=true \ --set cloudwatch.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::<account>:role/sreorbit-finops-readonlyVerification
orbit-agent reports on its own /metrics, which the chart already scrapes:
| Series | What |
|---|---|
orbit_agent_cloudwatch_requests_total{api,result} | calls to ListMetrics and GetMetricData, ok or error |
orbit_agent_cloudwatch_metrics_requested_total | the billed unit |
orbit_agent_cloudwatch_series | series being read |
orbit_agent_cloudwatch_last_success_timestamp_seconds | the last read that worked |
What the last 30 days cost, from those numbers:
increase(orbit_agent_cloudwatch_metrics_requested_total[30d]) * 0.01 / 1000AWS Cost Explorer shows the same bill under the CloudWatch usage type CW:GMD-Metrics. When nothing
else in the account calls GetMetricData, the two numbers match.
A read that stopped working shows as result="error" and an old last-success timestamp. Denied calls
mean the role or the association is missing: run the script again.
Removal
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values --set cloudwatch.enabled=falseaws eks delete-pod-identity-association --cluster-name my-eks --association-id <id>Find the id with aws eks list-pod-identity-associations --cluster-name my-eks --namespace sreorbit.
Keep the role if orbit-api still reads the bill; remove the SreOrbitCloudWatchReadOnly statement from
the policy if you want it gone too.