Skip to main content
By the end of this guide, critical alerts from your production environment page whoever is on call through PagerDuty. Every other alert goes to a Slack channel, and you can add an email list. Two pieces make this work, and they live in different places on purpose. A route in your environment’s KCL matches alerts by label and names a receiver. A receiver is an organization-wide destination: a Slack webhook, a PagerDuty routing key or an email list. The receiver’s secret is stored in the control plane, never in your repository. Routes ship inside the release, so promoting a release applies its routes (alerts as code).

Before you begin

  • A Forge project with a hosted environment that declares observability = forge.Observability {} (deploy a Go API or a static site).
  • The organization admin role (the obs:write grant) to create receivers. The control plane accepts this only from a person. An agent’s credential is refused (see Troubleshooting).
  • A Slack incoming-webhook URL, a PagerDuty Events v2 routing key, or both.
1

See which alerts the environment already has

Forge generates default alert rules. You do not need to write any rules to start routing:
terminal
A Go API with a database records 20 defaults. Their severities are what you will route on:
A static site has no services, so it gets only the four uptime and certificate defaults.With no routes declared, the shape records [{"receiver":"reliant-inapp","continue":true},{"receiver":"slack"}]. That means the in-app inbox, then a receiver named slack, and never a pager.
2

Create the receivers

Create receivers in the console, under Observability → Alerts → Receivers, or with obs receivers create from a terminal where you are signed in as yourself. Pass each secret from an environment variable so it stays out of your shell history:
terminal
The secret is write-only and is never shown again. A webhook type also exists (--setting url=https://…, with an optional bearer --secret). List what the organization has:
terminal
Receivers belong to the organization, so every project’s routes can name them.
3

Route alerts by severity

Add routes to the environment’s forge.Observability. An alert goes to the first route it matches. continue = True keeps matching, so one alert can reach several receivers:
deploy/kcl/prod/main.k
Declaring routes replaces the default pair, so keep the reliant-inapp route if you still want the in-app inbox. The last route must have an empty match, so that every alert has somewhere to go. To route the email list, add a route for it, for example match = {severity = "warning"} with continue = True.
4

Check the routes before you ship

Rendering validates the routes:
terminal
A route that names a URL instead of a receiver fails here, before anything ships. Then build and deploy a release as usual (promote a release). The routes take effect when that release is promoted.
5

Send a test alert

Take the receiver’s id from obs receivers list and send a test alert through it:
terminal
Check that it arrived in the Slack channel, PagerDuty service or inbox. The console’s Receivers tab also flags a route that names a receiver you have not created.
6

Remove a receiver you no longer need

terminal
Remove any route that names it too, in the next release.

What this does not do

  • It does not page anyone by default. Until you declare a route to a PagerDuty receiver, alerts go only to the in-app inbox and to a receiver named slack, if you created one.
  • An alert does not fix anything. To have an agent investigate, see alerts to GitHub issues. An agent files an investigated issue, a person labels it ready, an agent opens a pull request, and a person merges it.
  • Routes are per environment, but receivers are per organization. Two projects that both route to team-slack share one channel.
  • forge.AlertReceiver entries in KCL configure only the local Alertmanager that forge env up runs. A hosted environment ignores them.

Troubleshooting

obs receivers create, test and delete run only for a person who holds obs:write. A Reliant agent’s credential, a CI token and other machine credentials are refused with HTTP 403. The CLI’s hint to “sign in again” does not help in this case. Create, test and delete receivers in the console, or ask an organization admin to do it. obs receivers list works with any credential that can read the organization’s observability data.
A route names a receiver. Put the webhook URL or routing key in the receiver (--secret), and use the receiver’s name in the route.

Next steps

Debug frontend errors with session timelines

Follow a NewIssue alert to what the user did before the error.

Find the release that caused an error

Trace an alerted issue to its release and suspect commits.

Dashboards and alerts

Every default alert, how to tune thresholds, and how to write your own rules.

Production alerts to GitHub issues

Let on-call investigate alerts and file issues.