Before you begin
- A Forge project with a hosted environment that declares
observability = forge.Observability {}(deploy a Go API or a static site). - The organization admin role (the
obs:writegrant) to create receivers. The control plane accepts this only from a person. An agent’s credential is refused (see Troubleshooting). - A Slack incoming-webhook URL, a PagerDuty Events v2 routing key, or both.
1
See which alerts the environment already has
Forge generates default alert rules. You do not need to write any rules to start routing:A Go API with a database records 20 defaults. Their severities are what you will route on:A static site has no services, so it gets only the four uptime and certificate defaults.With no routes declared, the shape records
terminal
[{"receiver":"reliant-inapp","continue":true},{"receiver":"slack"}]. That means the in-app inbox, then a receiver named slack, and never a pager.2
Create the receivers
Create receivers in the console, under Observability → Alerts → Receivers, or with The secret is write-only and is never shown again. A Receivers belong to the organization, so every project’s routes can name them.
obs receivers create from a terminal where you are signed in as yourself. Pass each secret from an environment variable so it stays out of your shell history:terminal
webhook type also exists (--setting url=https://…, with an optional bearer --secret). List what the organization has:terminal
3
Route alerts by severity
Add routes to the environment’s Declaring routes replaces the default pair, so keep the
forge.Observability. An alert goes to the first route it matches. continue = True keeps matching, so one alert can reach several receivers:deploy/kcl/prod/main.k
reliant-inapp route if you still want the in-app inbox. The last route must have an empty match, so that every alert has somewhere to go. To route the email list, add a route for it, for example match = {severity = "warning"} with continue = True.4
Check the routes before you ship
Rendering validates the routes:A route that names a URL instead of a receiver fails here, before anything ships. Then build and deploy a release as usual (promote a release). The routes take effect when that release is promoted.
terminal
5
Send a test alert
Take the receiver’s id from Check that it arrived in the Slack channel, PagerDuty service or inbox. The console’s Receivers tab also flags a route that names a receiver you have not created.
obs receivers list and send a test alert through it:terminal
6
Remove a receiver you no longer need
terminal
What this does not do
- It does not page anyone by default. Until you declare a route to a PagerDuty receiver, alerts go only to the in-app inbox and to a receiver named
slack, if you created one. - An alert does not fix anything. To have an agent investigate, see alerts to GitHub issues. An agent files an investigated issue, a person labels it ready, an agent opens a pull request, and a person merges it.
- Routes are per environment, but receivers are per organization. Two projects that both route to
team-slackshare one channel. forge.AlertReceiverentries in KCL configure only the local Alertmanager thatforge env upruns. A hosted environment ignores them.
Troubleshooting
this alerting operation is not available to machine credentials
this alerting operation is not available to machine credentials
obs receivers create, test and delete run only for a person who holds obs:write. A Reliant agent’s credential, a CI token and other machine credentials are refused with HTTP 403. The CLI’s hint to “sign in again” does not help in this case. Create, test and delete receivers in the console, or ask an organization admin to do it. obs receivers list works with any credential that can read the organization’s observability data.AlertRoute.receiver '…' must be a receiver NAME (lowercase letters, digits, '-' and '_'), never a URL or a key
AlertRoute.receiver '…' must be a receiver NAME (lowercase letters, digits, '-' and '_'), never a URL or a key
A route names a receiver. Put the webhook URL or routing key in the receiver (
--secret), and use the receiver’s name in the route.Next steps
Debug frontend errors with session timelines
Follow a
NewIssue alert to what the user did before the error.Find the release that caused an error
Trace an alerted issue to its release and suspect commits.
Dashboards and alerts
Every default alert, how to tune thresholds, and how to write your own rules.
Production alerts to GitHub issues
Let on-call investigate alerts and file issues.