> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reliantlabs.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Write alert rules as code and test them locally

> Declare an alert rule in KCL as a SQL query, check it against your local stack's telemetry, watch it fire on your machine, and ship it with the next release.

By the end of this guide you will have an alert rule of your own in the repository, evaluated against real telemetry on your machine, and recorded in the release that ships it. A rule is one SQL query over your logs, traces and metrics: every row it returns is a firing alert. The local stack evaluates your rules continuously and routes what fires with the same configuration renderer the platform uses, so a route that reaches a receiver on your laptop reaches it in production too.

The defaults Forge generates (error rate, p95 latency, crash loops, failed deploys and more) already cover the common cases: [Hosted dashboards, alerts and issues](https://docs.reliantlabs.io/hosting/observability/dashboards-and-alerts.md#alerts). Write your own for what only your app knows.

## Before you begin

* A Forge project running locally with `reliant forge env up dev`: [Build a Go and React app](https://docs.reliantlabs.io/guides/forge/build-go-react-app-connect-rpc.md).
* Forge v1.6.1 or later in the project (`reliant forge --version`).

The rule below watches the `shop` app's `GetProduct` RPC: five or more `not_found` answers in five minutes usually means a broken link somewhere.

**1. Try the query first**

Write the condition as a ClickHouse `SELECT` over `otel_logs`, `otel_traces` or the `otel_metrics_*` tables, and run it against the local store:

```bash terminal theme={null}
reliant forge obs query dev --sql "SELECT LogAttributes['procedure'] AS procedure, count() AS value FROM otel_logs WHERE Body = 'rpc failed' AND LogAttributes['code'] = 'not_found' AND Timestamp >= now() - INTERVAL 15 MINUTE GROUP BY procedure"
```

```text Output theme={null}
procedure                                       value
/services.catalog.v1.CatalogService/GetProduct  1
```

Columns other than `value` become the alert's labels, and `value` is its number. Put the threshold in the query with `HAVING`, so a healthy system returns no rows.

**2. Declare the rule once for every environment**

Rules are per environment, so keep them in a shared file and pass them to each env:

```kcl deploy/kcl/alerts.k theme={null}
import forge

rules = [
    forge.AlertRule {
        alert = "ProductLookupsFailing"
        expr = """
SELECT LogAttributes['procedure'] AS procedure, count() AS value
FROM otel_logs
WHERE Body = 'rpc failed' AND LogAttributes['code'] = 'not_found'
  AND Timestamp >= now() - INTERVAL 5 MINUTE
GROUP BY procedure HAVING value >= 5"""
        "for" = "2m"
        labels = {severity = "warning", team = "catalog"}
        annotations = {summary = "{{ $labels.procedure }} returned not_found {{ $value }} times in 5m"}
    }
]
```

```kcl deploy/kcl/prod/main.k theme={null}
import ..alerts as al

_bundle = forge.Bundle {
    observability = forge.Observability {alerts = forge.Alerts {rules = al.rules}}
    # ...
}
```

In `deploy/kcl/dev/main.k`, add the same import and `alerts = forge.Alerts {rules = al.rules}` inside the existing `forge.Observability` block. `for` is a KCL keyword, so it is quoted. `severity` is `critical`, `warning` or `info`.

**3. Evaluate it once**

`obs alerts check` runs every rule, yours and the defaults, against the local store and prints what returns rows now. It ignores `for` and notifies nobody. After a few lookups of products that do not exist:

```bash terminal theme={null}
reliant forge obs alerts check --rule ProductLookupsFailing
```

```text Output theme={null}
alert rules for env "dev", evaluated once against the local ClickStack (no notifications; `for` not applied)

ProductLookupsFailing              warning   user     for 2m     FIRING x1
    {procedure="/services.catalog.v1.CatalogService/GetProduct", team="catalog"}  value=7

1 rule(s), 1 instance(s) returning now, 0 rule(s) failed
```

Without `--rule` it lists every rule with `ok`, `FIRING` or the error its query hit. `--sql` prints each rule's query instead of running it.

**4. Watch it fire on your machine**

Run `reliant forge env up dev` again to load the new rule. The stack's `alerts` service evaluates the rules every interval with `for` applied, and sends what fires through a local Alertmanager to the in-app receiver, which writes `.forge/logs/dev/alerts.log`:

```bash terminal theme={null}
reliant forge obs logs dev --service alerts --since 15m
```

```text Output (trimmed) theme={null}
TIME          SEVERITY  SERVICE  BODY
16:30:30.402  info      alerts   ALERT firing   ProductLookupsFailing procedure=/services.catalog.v1.CatalogService/GetProduct severity=warning team=catalog — /services.catalog.v1.CatalogService/GetProduct returned not_found 6 times…
16:26:20.023  info      alerts   alert rules loaded
```

The annotation template filled in the procedure and the count. `grep 'ALERT firing' .forge/logs/dev/alerts.log` finds the same lines on disk.

**5. Check what the release records**

`env render` refuses a malformed rule before anything ships. A rule labelled `severity = "high"` fails with:

```text Output theme={null}
Check failed on the condition: AlertRule 'ProductLookupsFailing': label severity must be one of critical, warning, info
```

A valid rule is recorded in the environment's release shape, beside the defaults:

```bash terminal theme={null}
reliant forge env shape prod --json | jq -c '.shape.alerts[] | select(.source=="user") | {alert, for: .["for"], interval, labels}'
```

```text Output theme={null}
{"alert":"ProductLookupsFailing","for":"2m","interval":"1m","labels":{"severity":"warning","team":"catalog"}}
```

When you deploy that release to a hosted environment, the platform evaluates the rule there. Promoting a later release applies its rules; deploying an earlier release restores that release's rules.

## What this does not do

* **It does not decide who gets paged.** With no routes declared, alerts go to in-app notifications and then to a receiver named `slack`, never to a pager. Routing is a separate step: [Route alerts to Slack, PagerDuty or email](https://docs.reliantlabs.io/guides/observability/route-alerts-slack-pagerduty.md).
* **It does not fix anything.** An alert can start an investigation that files an issue, and a person decides what happens next: [Turn production alerts into GitHub issues](https://docs.reliantlabs.io/guides/workflows/production-alerts-to-github-issues.md).
* **Healthy traces are sampled in hosted environments.** Count logs, metrics or error spans; do not divide by a count of healthy spans.

## Troubleshooting

> [accordion] expr must be a single statement, with no ';'
>
> A rule is one read-only `SELECT` or `WITH … SELECT`. Remove the trailing semicolon.

> [accordion] The rule shows ok in obs alerts check but you expected FIRING
>
> The query returned no rows. Run it with `reliant forge obs query dev --sql "…"` without the `HAVING` clause to see the raw counts, and check the time window: a rule only sees data inside its own `INTERVAL`.

> [accordion] FIRING in obs alerts check, but nothing in alerts.log
>
> The check ignores `for`; the running evaluator waits for it, and evaluates once per `interval` (default `1m`). Also make sure you ran `reliant forge env up dev` after adding the rule.

## Next steps

> [card] [Route alerts](https://docs.reliantlabs.io/guides/observability/route-alerts-slack-pagerduty.md)
>
> Send critical alerts to a pager and the rest to Slack.

> [card] [Add a custom metric](https://docs.reliantlabs.io/guides/observability/add-custom-metric.md)
>
> Record a number of your own, then alert on it.

> [card] [Dashboards and alerts](https://docs.reliantlabs.io/hosting/observability/dashboards-and-alerts.md#alerts)
>
> Every default rule and how to tune or disable it.

> [card] [Querying with forge obs](https://docs.reliantlabs.io/hosting/observability/forge-obs.md)
>
> The obs verbs, flags and exit codes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.