Skip to content

Features

CloudWatch metrics

With cloudwatch.enabled, orbit-agent reads AWS CloudWatch metrics, RDS by default, and writes them to orbit-stack beside everything else it collects. Dashboards, alerts and Orbit SQL then query RDS CPU, connections or IOPS with PromQL, like any other series.

What is collected and how it is named

Each entry of cloudwatch.metrics is Namespace:Metric:Statistic. The default is eight RDS metrics:

MetricStatisticSeries
CPUUtilizationAverageaws_rds_cpuutilization_average
DatabaseConnectionsAverageaws_rds_database_connections_average
FreeableMemoryAverageaws_rds_freeable_memory_average
FreeStorageSpaceAverageaws_rds_free_storage_space_average
ReadIOPSSumaws_rds_read_iops_sum
WriteIOPSSumaws_rds_write_iops_sum
DMLLatencyAverageaws_rds_dmllatency_average (Aurora)
BlockedTransactionsSumaws_rds_blocked_transactions_sum (Aurora)

The names follow prometheus/cloudwatch_exporter: aws_<namespace>_<metric>_<statistic> in snake case (an underscore only where a lowercase letter or digit meets an uppercase one), with one label per CloudWatch dimension in the same snake case (DBInstanceIdentifier becomes dbinstance_identifier) and job="aws_rds". Points keep CloudWatch's own 1-minute timestamps.

So a Grafana dashboard written for cloudwatch_exporter works unchanged:

promql
max(aws_rds_cpuutilization_average{job="aws_rds"}) by (dbinstance_identifier)

A Grafana v2 dashboard (dashboard.grafana.app/v2) imports through Dashboards → Import, and its CloudWatch panels (namespace, metric name, statistic) are translated to the same PromQL.

Latency

CloudWatch itself publishes RDS metrics about 1 to 2 minutes after the minute they describe. On top of that, orbit-agent reads every cloudwatch.interval (3 minutes by default) and asks for the whole interval plus three minutes each time, so a point shows in the Deck at most one interval after CloudWatch has it and no minute is skipped, however long the interval. Points that arrive late in CloudWatch are picked up on the next read. A read that fails (throttling, credentials) leaves its minutes to the next one that goes through, which reads back to three minutes before the last read that did, two hours at most; a restart of orbit-agent starts over from the last six minutes.

Cost

GetMetricData has no free tier: AWS bills $0.01 per 1,000 metrics requested, whatever the number of points each one returns. Asking for the last minutes at 1-minute resolution in one read therefore costs the same as asking for one point. The interval sets the bill; the resolution leaves it unchanged:

text
monthly cost = series × (30 × 24 × 60 / interval_minutes) × 0.01 / 1000

For 10 instances × 8 metrics (80 series):

IntervalReads a dayPer month
1m1,440$34.56
2m720$17.28
3m (default)480$11.52
5m288$6.91
10m144$3.46
30m48$1.15

A longer interval only delays the points: every read still brings each minute since the last one, so 10m or 30m cuts the bill by the same factor and keeps 1-minute resolution.

Only series CloudWatch lists as recently active (ListMetrics with RecentlyActive) are requested: a deleted instance stops costing within hours. Discovery calls (ListMetrics) are billed as ordinary API requests, a negligible amount at this rate.

Configuration

ValueEnv on orbit-agentDefault
cloudwatch.enabledORBIT_AGENT_CLOUDWATCH_ENABLEDfalse
cloudwatch.intervalORBIT_AGENT_CLOUDWATCH_INTERVAL3m (1m minimum)
cloudwatch.regionORBIT_AGENT_CLOUDWATCH_REGIONthe pod's AWS_REGION, else us-east-1
cloudwatch.metricsORBIT_AGENT_CLOUDWATCH_METRICSthe eight RDS metrics above
cloudwatch.serviceAccount.annotations{}; IRSA's role ARN goes here

It needs telemetry.agent with engine orbit. With networkPolicy.enabled, it opens orbit-agent's egress to TCP 443 (CloudWatch and STS; private ranges and instance metadata stay closed) and to the Pod Identity agent at 169.254.170.23:80.

IAM: the same read-only role

orbit-agent uses the role orbit-api uses for the bill (docs/finops-aws.md), with two more read-only actions:

ActionWhy
cloudwatch:GetMetricDatathe points, up to 500 series per call
cloudwatch:ListMetricswhich series exist and are recently active

The policy is deploy/aws/finops-readonly-policy.json. It names no account and no resource.

EKS Pod Identity (recommended). Run the script again: it rewrites the inline policy and adds the association for <release>-orbit-agent, leaving what already exists.

bash
CLUSTER=my-eks NAMESPACE=sreorbit RELEASE=sreorbit deploy/aws/setup-pod-identity.shhelm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values --set cloudwatch.enabled=true

IRSA. Give the role the trust in deploy/aws/trust-irsa.json for system:serviceaccount:<namespace>:<release>-orbit-agent, then:

bash
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values \  --set cloudwatch.enabled=true \  --set cloudwatch.serviceAccount.annotations."eks\.amazonaws\.com/role-arn"=arn:aws:iam::<account>:role/sreorbit-finops-readonly

Verification

orbit-agent reports on its own /metrics, which the chart already scrapes:

SeriesWhat
orbit_agent_cloudwatch_requests_total{api,result}calls to ListMetrics and GetMetricData, ok or error
orbit_agent_cloudwatch_metrics_requested_totalthe billed unit
orbit_agent_cloudwatch_seriesseries being read
orbit_agent_cloudwatch_last_success_timestamp_secondsthe last read that worked

What the last 30 days cost, from those numbers:

promql
increase(orbit_agent_cloudwatch_metrics_requested_total[30d]) * 0.01 / 1000

AWS Cost Explorer shows the same bill under the CloudWatch usage type CW:GMD-Metrics. When nothing else in the account calls GetMetricData, the two numbers match.

A read that stopped working shows as result="error" and an old last-success timestamp. Denied calls mean the role or the association is missing: run the script again.

Removal

bash
helm upgrade sreorbit deploy/helm/sreorbit -n sreorbit --reuse-values --set cloudwatch.enabled=falseaws eks delete-pod-identity-association --cluster-name my-eks --association-id <id>

Find the id with aws eks list-pod-identity-associations --cluster-name my-eks --namespace sreorbit. Keep the role if orbit-api still reads the bill; remove the SreOrbitCloudWatchReadOnly statement from the policy if you want it gone too.