> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reliantlabs.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Route production alerts to Slack, PagerDuty or email

> Send Forge's default alerts to Slack, PagerDuty or email: create receivers in your organization, then route by severity with forge.AlertRoute in KCL.

By the end of this guide, critical alerts from your production environment page whoever is on call through PagerDuty. Every other alert goes to a Slack channel, and you can add an email list.

Two pieces make this work, and they live in different places on purpose. A **route** in your environment's KCL matches alerts by label and names a receiver. A **receiver** is an organization-wide destination: a Slack webhook, a PagerDuty routing key or an email list. The receiver's secret is stored in the control plane, never in your repository. Routes ship inside the release, so promoting a release applies its routes ([alerts as code](https://docs.reliantlabs.io/hosting/observability/dashboards-and-alerts.md#alerts)).

## Before you begin

* A Forge project with a hosted environment that declares `observability = forge.Observability {}` ([deploy a Go API](https://docs.reliantlabs.io/guides/hosting/deploy-go-api-managed-postgres.md) or [a static site](https://docs.reliantlabs.io/guides/forge/deploy-static-site-cdn.md)).
* The organization admin role (the `obs:write` grant) to create receivers. The control plane accepts this only from a person. An agent's credential is refused (see [Troubleshooting](#troubleshooting)).
* A Slack incoming-webhook URL, a PagerDuty Events v2 routing key, or both.

**1. See which alerts the environment already has**

Forge generates default alert rules. You do not need to write any rules to start routing:

```bash terminal theme={null}
reliant forge env shape prod
reliant forge env shape prod --json | jq -r '.shape.alerts[] | [.alert, .labels.severity] | @tsv'
```

A Go API with a database records 20 defaults. Their severities are what you will route on:

```text theme={null}
alerts:    20 (0 yours, 20 generated defaults), 2 route(s) (rules only; receivers' secrets live in the control plane)

CertificateExpiringSoon    critical
DeployFailed               critical
IssueRegressed             warning
NewIssue                   warning
ReleaseRegressed           warning
ServiceErrorRateHigh       critical
ServiceLatencyP95High      warning
UptimeCheckDown            critical
WorkloadCrashLooping       critical
WorkloadOOMKilled          critical
…
```

A static site has no services, so it gets only the four uptime and certificate defaults.

With no routes declared, the shape records `[{"receiver":"reliant-inapp","continue":true},{"receiver":"slack"}]`. That means the in-app inbox, then a receiver named `slack`, and never a pager.

**2. Create the receivers**

Create receivers in the console, under **Observability → Alerts → Receivers**, or with `obs receivers create` from a terminal where you are signed in as yourself. Pass each secret from an environment variable so it stays out of your shell history:

```bash terminal theme={null}
reliant forge obs receivers create pager --type pagerduty --secret "$PD_ROUTING_KEY" --setting severity=critical
reliant forge obs receivers create team-slack --type slack --secret "$SLACK_WEBHOOK_URL" --setting channel=#alerts
reliant forge obs receivers create ops-email --type email --setting to=ops@example.com,lead@example.com
```

The secret is write-only and is never shown again. A `webhook` type also exists (`--setting url=https://…`, with an optional bearer `--secret`). List what the organization has:

```bash terminal theme={null}
reliant forge obs receivers list
```

```text theme={null}
ID                                    NAME         TYPE   SETTINGS
287c8385-f816-4cd7-87d5-445744be79c6  owner-email  email  to=you@example.com
```

Receivers belong to the organization, so every project's routes can name them.

**3. Route alerts by severity**

Add routes to the environment's `forge.Observability`. An alert goes to the first route it matches. `continue = True` keeps matching, so one alert can reach several receivers:

```kcl deploy/kcl/prod/main.k theme={null}
observability = forge.Observability {
    alerts = forge.Alerts {
        routes = [
            forge.AlertRoute {match = {}, receiver = "reliant-inapp", continue = True}
            forge.AlertRoute {match = {severity = "critical"}, receiver = "pager", continue = True}
            forge.AlertRoute {match = {}, receiver = "team-slack"}
        ]
    }
}
```

Declaring routes replaces the default pair, so keep the `reliant-inapp` route if you still want the in-app inbox. The last route must have an empty `match`, so that every alert has somewhere to go. To route the email list, add a route for it, for example `match = {severity = "warning"}` with `continue = True`.

**4. Check the routes before you ship**

Rendering validates the routes:

```bash terminal theme={null}
reliant forge env render prod
reliant forge env shape prod
reliant forge env shape prod --json | jq -c '.shape.alert_routes'
```

```text theme={null}
alerts:    4 (0 yours, 4 generated defaults), 3 route(s) (rules only; receivers' secrets live in the control plane)
[{"receiver":"reliant-inapp","continue":true},{"match":{"severity":"critical"},"receiver":"pager","continue":true},{"receiver":"team-slack"}]
```

A route that names a URL instead of a receiver fails here, before anything ships. Then build and deploy a release as usual ([promote a release](https://docs.reliantlabs.io/guides/hosting/promote-release-staging-to-production.md)). The routes take effect when that release is promoted.

**5. Send a test alert**

Take the receiver's id from `obs receivers list` and send a test alert through it:

```bash terminal theme={null}
reliant forge obs receivers test 287c8385-f816-4cd7-87d5-445744be79c6
```

Check that it arrived in the Slack channel, PagerDuty service or inbox. The console's Receivers tab also flags a route that names a receiver you have not created.

**6. Remove a receiver you no longer need**

```bash terminal theme={null}
reliant forge obs receivers delete <receiver-id>
```

Remove any route that names it too, in the next release.

## What this does not do

* It does not page anyone by default. Until you declare a route to a PagerDuty receiver, alerts go only to the in-app inbox and to a receiver named `slack`, if you created one.
* An alert does not fix anything. To have an agent investigate, see [alerts to GitHub issues](https://docs.reliantlabs.io/guides/workflows/production-alerts-to-github-issues.md). An agent files an investigated issue, a person labels it ready, an agent opens a pull request, and a person merges it.
* Routes are per environment, but receivers are per organization. Two projects that both route to `team-slack` share one channel.
* `forge.AlertReceiver` entries in KCL configure only the local Alertmanager that `forge env up` runs. A hosted environment ignores them.

## Troubleshooting

> [accordion] this alerting operation is not available to machine credentials
>
> `obs receivers create`, `test` and `delete` run only for a person who holds `obs:write`. A Reliant agent's credential, a CI token and other machine credentials are refused with HTTP 403. The CLI's hint to "sign in again" does not help in this case. Create, test and delete receivers in the console, or ask an organization admin to do it. `obs receivers list` works with any credential that can read the organization's observability data.

> [accordion] AlertRoute.receiver '…' must be a receiver NAME (lowercase letters, digits, '-' and '\_'), never a URL or a key
>
> A route names a receiver. Put the webhook URL or routing key in the receiver (`--secret`), and use the receiver's name in the route.

## Next steps

> [card] [Debug frontend errors with session timelines](https://docs.reliantlabs.io/guides/observability/debug-frontend-errors-session-timelines.md)
>
> Follow a `NewIssue` alert to what the user did before the error.

> [card] [Find the release that caused an error](https://docs.reliantlabs.io/guides/observability/find-release-that-caused-error.md)
>
> Trace an alerted issue to its release and suspect commits.

> [card] [Dashboards and alerts](https://docs.reliantlabs.io/hosting/observability/dashboards-and-alerts.md)
>
> Every default alert, how to tune thresholds, and how to write your own rules.

> [card] [Production alerts to GitHub issues](https://docs.reliantlabs.io/guides/workflows/production-alerts-to-github-issues.md)
>
> Let on-call investigate alerts and file issues.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.