Features
Alert notifications
orbit-api can post the Deck's alerts to a Slack channel, a Microsoft Teams channel or a Google Chat space through an incoming webhook. An admin sets each destination up under Alerts โ Notifications: pick the platform, paste the webhook URL, choose which alerts it takes, send a test, save. There is no Alertmanager to run and nothing to install in the chat tool beyond the webhook itself.
What gets sent, and when
Every alerting.interval (30 seconds by default) orbit-api reads each cluster's alerts, with the
acknowledgements and silences set in the Deck laid over them, exactly as the Alerts page shows them.
For each enabled channel whose rules take an alert:
| Message | Sent when |
|---|---|
| Firing | The alert has held for the channel's wait (0, 2, 5 or 10 minutes; 2 by default) and this channel has not been told yet. |
| Repeat | It is still firing, still open, and the channel's repeat (1 hour, 4 hours or 24 hours; 4 hours by default, or never) has passed since the last message. |
| Resolved | It is gone and this channel was told it was firing, when the channel sends resolved messages (on by default). |
- Acknowledged or silenced alerts page nobody. No first message and no repeat while an alert is acked or silenced. When a silence runs out and the alert still holds, the firing message goes out on the next read. An alert that clears after a channel was told still sends its resolved message.
- Grouped. Everything one read has for one channel goes out as one message per kind, so an incident that fires twenty alerts posts once, not twenty times. Messages leave at most one a second.
- No repeats after a restart. What each channel was told is kept in
ORBIT_DATA_DIR, so restarting orbit-api neither re-pages nor forgets a resolved message it still owes. - A failed delivery is retried. The next read sends it again, after a wait that doubles from the interval up to 5 minutes for that channel. Nothing is marked as sent until the webhook took it.
- Live clusters only. With
ORBIT_DEMO_DATA=truenothing is sent on its own; a test message still goes out, so the setup can be tried.
Creating the webhook
The channel a message lands in is the one the webhook was created for: choosing the channel happens in the chat tool, when the webhook is made. One webhook per channel; add the same alerts to several channels with one Orbit channel each.
Slack. In the Slack workspace, add the Incoming Webhooks app (or create a Slack app with
incoming webhooks turned on), pick the channel, and copy the URL. It starts with
https://hooks.slack.com/services/.
Microsoft Teams. Office 365 connectors are retired; Teams takes webhooks through Workflows. In
the channel, open Workflows and pick the template Post to a channel when a webhook request is
received, choose the team and channel, and copy the URL it gives. It is on logic.azure.com or on
environment.api.powerplatform.com.
Google Chat. In the space, open Apps & integrations โ Webhooks, add a webhook with a name and
an optional avatar, and copy the URL. It starts with https://chat.googleapis.com/v1/spaces/.
Choosing which alerts a channel takes
| Setting | Default | Meaning |
|---|---|---|
| Severities | critical, warning | Which of critical, warning and info it takes. |
| Clusters | every cluster | Only these clusters, when any are listed. |
| Namespaces | every namespace | Only these namespaces, when any are listed. |
| Wait | 2 minutes | How long an alert must hold before the first message. |
| Repeat | 4 hours | How often a still-firing, open alert is posted again; never is an option. |
| Resolved messages | on | Whether a cleared alert is posted. |
| Security notices | off | Also posts a notice when any account's password changes or an admin resets it, a second factor goes on or off, an email changes, an API token is created, or someone types an account's password with a code it already used. |
A security notice names the accounts, who acted and the client address, in the channel's
language, and never a password, a code, a token or an email address. Each send lands in the
channel's deliveries as kind security. Point one channel your admins read at it: SRE Orbit
sends no email, so this is how an account's holder hears that their credentials changed.
Making the message yours
- Emoji per severity: one each for critical, warning, info and resolved (๐ด ๐ ๐ต โ by default).
- Title: a template with these variables:
{emoji},{severity},{name},{summary},{cluster},{namespace},{workload},{since},{link}. The default is{emoji} [{severity}] {name} ยท {cluster}. An optional footer line takes the same variables. - Mention (Slack only):
@hereor@channelon critical messages. - Language: the fixed words of a message (resolved, the link's label, how many more) in Portuguese or English.
- Link: a button to the alert in the Deck, built from
publicUrl(orORBIT_PUBLIC_URL). Left empty, the chart derives it from the ingress host; with no ingress, messages carry no link.
The Deck shows a live preview of the message for the chosen platform while it is edited.
Security
- Webhook URLs are secrets. Whoever holds one can post to the channel. orbit-api stores them
sealed with
ORBIT_SEAL_KEY(AES-256-GCM), the key the database connections use, and never returns them: the Deck shows only the host and the last characters. WithoutORBIT_SEAL_KEYa channel cannot be saved. - Only the platforms' own hosts. A URL must be HTTPS on
hooks.slack.com,chat.googleapis.com, or a Teams host: Workflows onlogic.azure.comorenvironment.api.powerplatform.com, or a legacy connector onwebhook.office.com(Microsoft retired those; they still validate).alerting.allowedHosts(ORBIT_ALERTING_ALLOWED_HOSTS) adds hosts, for an egress proxy or a sovereign cloud. Redirects are not followed. - Who can change them. Listing, saving, deleting, testing and previewing channels need
settings.manage. Throughwebthey need a session and same-origin proof and no operator token, since they change Orbit's own state, never the cluster. - Audited. Saving, deleting and testing a channel are audit events,
alerting.channel.update,alerting.channel.deleteandalerting.channel.test, without the URL. - Egress. With
networkPolicy.enabledandalerting.notifications.enabled(on by default), orbit-api may open TCP 443 to public addresses, never to private ranges or the cloud metadata address. Turned off on a cluster that enforces NetworkPolicy, channels still save but every delivery fails on the network.
Configuration
| Value | Variable | Default |
|---|---|---|
alerting.notifications.enabled | (egress only) | true |
alerting.interval | ORBIT_ALERTING_INTERVAL | 30s, at least 10s |
alerting.allowedHosts | ORBIT_ALERTING_ALLOWED_HOSTS | none |
publicUrl | ORBIT_PUBLIC_URL | derived from the ingress |
Troubleshooting
- Delivery history. Each channel keeps its last 50 deliveries (time, kind, how many alerts, the webhook's HTTP status and a short error, never the URL), and its card shows the last one.
- The metric.
orbit_api_alert_notifications_total{kind,result}counts messages by kind (firing,renotify,resolved,test) and result (ok,error):
sum by (kind) (increase(orbit_api_alert_notifications_total{result="error"}[1h]))403or404from the webhook: the webhook was deleted or its workflow turned off; create a new one and paste it into the channel.429: the platform is rate limiting; orbit-api backs off and retries on its own.- Network errors on every delivery: check
alerting.notifications.enabledand any egress firewall or proxy between the cluster and the platform.