Skip to content

Features

Alert notifications

orbit-api can post the Deck's alerts to a Slack channel, a Microsoft Teams channel or a Google Chat space through an incoming webhook. An admin sets each destination up under Alerts โ†’ Notifications: pick the platform, paste the webhook URL, choose which alerts it takes, send a test, save. There is no Alertmanager to run and nothing to install in the chat tool beyond the webhook itself.

What gets sent, and when

Every alerting.interval (30 seconds by default) orbit-api reads each cluster's alerts, with the acknowledgements and silences set in the Deck laid over them, exactly as the Alerts page shows them. For each enabled channel whose rules take an alert:

MessageSent when
FiringThe alert has held for the channel's wait (0, 2, 5 or 10 minutes; 2 by default) and this channel has not been told yet.
RepeatIt is still firing, still open, and the channel's repeat (1 hour, 4 hours or 24 hours; 4 hours by default, or never) has passed since the last message.
ResolvedIt is gone and this channel was told it was firing, when the channel sends resolved messages (on by default).
  • Acknowledged or silenced alerts page nobody. No first message and no repeat while an alert is acked or silenced. When a silence runs out and the alert still holds, the firing message goes out on the next read. An alert that clears after a channel was told still sends its resolved message.
  • Grouped. Everything one read has for one channel goes out as one message per kind, so an incident that fires twenty alerts posts once, not twenty times. Messages leave at most one a second.
  • No repeats after a restart. What each channel was told is kept in ORBIT_DATA_DIR, so restarting orbit-api neither re-pages nor forgets a resolved message it still owes.
  • A failed delivery is retried. The next read sends it again, after a wait that doubles from the interval up to 5 minutes for that channel. Nothing is marked as sent until the webhook took it.
  • Live clusters only. With ORBIT_DEMO_DATA=true nothing is sent on its own; a test message still goes out, so the setup can be tried.

Creating the webhook

The channel a message lands in is the one the webhook was created for: choosing the channel happens in the chat tool, when the webhook is made. One webhook per channel; add the same alerts to several channels with one Orbit channel each.

Slack. In the Slack workspace, add the Incoming Webhooks app (or create a Slack app with incoming webhooks turned on), pick the channel, and copy the URL. It starts with https://hooks.slack.com/services/.

Microsoft Teams. Office 365 connectors are retired; Teams takes webhooks through Workflows. In the channel, open Workflows and pick the template Post to a channel when a webhook request is received, choose the team and channel, and copy the URL it gives. It is on logic.azure.com or on environment.api.powerplatform.com.

Google Chat. In the space, open Apps & integrations โ†’ Webhooks, add a webhook with a name and an optional avatar, and copy the URL. It starts with https://chat.googleapis.com/v1/spaces/.

Choosing which alerts a channel takes

SettingDefaultMeaning
Severitiescritical, warningWhich of critical, warning and info it takes.
Clustersevery clusterOnly these clusters, when any are listed.
Namespacesevery namespaceOnly these namespaces, when any are listed.
Wait2 minutesHow long an alert must hold before the first message.
Repeat4 hoursHow often a still-firing, open alert is posted again; never is an option.
Resolved messagesonWhether a cleared alert is posted.
Security noticesoffAlso posts a notice when any account's password changes or an admin resets it, a second factor goes on or off, an email changes, an API token is created, or someone types an account's password with a code it already used.

A security notice names the accounts, who acted and the client address, in the channel's language, and never a password, a code, a token or an email address. Each send lands in the channel's deliveries as kind security. Point one channel your admins read at it: SRE Orbit sends no email, so this is how an account's holder hears that their credentials changed.

Making the message yours

  • Emoji per severity: one each for critical, warning, info and resolved (๐Ÿ”ด ๐ŸŸ  ๐Ÿ”ต โœ… by default).
  • Title: a template with these variables: {emoji}, {severity}, {name}, {summary}, {cluster}, {namespace}, {workload}, {since}, {link}. The default is {emoji} [{severity}] {name} ยท {cluster}. An optional footer line takes the same variables.
  • Mention (Slack only): @here or @channel on critical messages.
  • Language: the fixed words of a message (resolved, the link's label, how many more) in Portuguese or English.
  • Link: a button to the alert in the Deck, built from publicUrl (or ORBIT_PUBLIC_URL). Left empty, the chart derives it from the ingress host; with no ingress, messages carry no link.

The Deck shows a live preview of the message for the chosen platform while it is edited.

Security

  • Webhook URLs are secrets. Whoever holds one can post to the channel. orbit-api stores them sealed with ORBIT_SEAL_KEY (AES-256-GCM), the key the database connections use, and never returns them: the Deck shows only the host and the last characters. Without ORBIT_SEAL_KEY a channel cannot be saved.
  • Only the platforms' own hosts. A URL must be HTTPS on hooks.slack.com, chat.googleapis.com, or a Teams host: Workflows on logic.azure.com or environment.api.powerplatform.com, or a legacy connector on webhook.office.com (Microsoft retired those; they still validate). alerting.allowedHosts (ORBIT_ALERTING_ALLOWED_HOSTS) adds hosts, for an egress proxy or a sovereign cloud. Redirects are not followed.
  • Who can change them. Listing, saving, deleting, testing and previewing channels need settings.manage. Through web they need a session and same-origin proof and no operator token, since they change Orbit's own state, never the cluster.
  • Audited. Saving, deleting and testing a channel are audit events, alerting.channel.update, alerting.channel.delete and alerting.channel.test, without the URL.
  • Egress. With networkPolicy.enabled and alerting.notifications.enabled (on by default), orbit-api may open TCP 443 to public addresses, never to private ranges or the cloud metadata address. Turned off on a cluster that enforces NetworkPolicy, channels still save but every delivery fails on the network.

Configuration

ValueVariableDefault
alerting.notifications.enabled(egress only)true
alerting.intervalORBIT_ALERTING_INTERVAL30s, at least 10s
alerting.allowedHostsORBIT_ALERTING_ALLOWED_HOSTSnone
publicUrlORBIT_PUBLIC_URLderived from the ingress

Troubleshooting

  • Delivery history. Each channel keeps its last 50 deliveries (time, kind, how many alerts, the webhook's HTTP status and a short error, never the URL), and its card shows the last one.
  • The metric. orbit_api_alert_notifications_total{kind,result} counts messages by kind (firing, renotify, resolved, test) and result (ok, error):
promql
sum by (kind) (increase(orbit_api_alert_notifications_total{result="error"}[1h]))
  • 403 or 404 from the webhook: the webhook was deleted or its workflow turned off; create a new one and paste it into the channel.
  • 429: the platform is rate limiting; orbit-api backs off and retries on its own.
  • Network errors on every delivery: check alerting.notifications.enabled and any egress firewall or proxy between the cluster and the platform.