Skip to main content
By the end of this guide you will have an alert rule of your own in the repository, evaluated against real telemetry on your machine, and recorded in the release that ships it. A rule is one SQL query over your logs, traces and metrics: every row it returns is a firing alert. The local stack evaluates your rules continuously and routes what fires with the same configuration renderer the platform uses, so a route that reaches a receiver on your laptop reaches it in production too. The defaults Forge generates (error rate, p95 latency, crash loops, failed deploys and more) already cover the common cases: Hosted dashboards, alerts and issues. Write your own for what only your app knows.

Before you begin

  • A Forge project running locally with reliant forge env up dev: Build a Go and React app.
  • Forge v1.6.1 or later in the project (reliant forge --version).
The rule below watches the shop app’s GetProduct RPC: five or more not_found answers in five minutes usually means a broken link somewhere.
1

Try the query first

Write the condition as a ClickHouse SELECT over otel_logs, otel_traces or the otel_metrics_* tables, and run it against the local store:
terminal
Output
Columns other than value become the alert’s labels, and value is its number. Put the threshold in the query with HAVING, so a healthy system returns no rows.
2

Declare the rule once for every environment

Rules are per environment, so keep them in a shared file and pass them to each env:
deploy/kcl/alerts.k
deploy/kcl/prod/main.k
In deploy/kcl/dev/main.k, add the same import and alerts = forge.Alerts {rules = al.rules} inside the existing forge.Observability block. for is a KCL keyword, so it is quoted. severity is critical, warning or info.
3

Evaluate it once

obs alerts check runs every rule, yours and the defaults, against the local store and prints what returns rows now. It ignores for and notifies nobody. After a few lookups of products that do not exist:
terminal
Output
Without --rule it lists every rule with ok, FIRING or the error its query hit. --sql prints each rule’s query instead of running it.
4

Watch it fire on your machine

Run reliant forge env up dev again to load the new rule. The stack’s alerts service evaluates the rules every interval with for applied, and sends what fires through a local Alertmanager to the in-app receiver, which writes .forge/logs/dev/alerts.log:
terminal
Output (trimmed)
The annotation template filled in the procedure and the count. grep 'ALERT firing' .forge/logs/dev/alerts.log finds the same lines on disk.
5

Check what the release records

env render refuses a malformed rule before anything ships. A rule labelled severity = "high" fails with:
Output
A valid rule is recorded in the environment’s release shape, beside the defaults:
terminal
Output
When you deploy that release to a hosted environment, the platform evaluates the rule there. Promoting a later release applies its rules; deploying an earlier release restores that release’s rules.

What this does not do

  • It does not decide who gets paged. With no routes declared, alerts go to in-app notifications and then to a receiver named slack, never to a pager. Routing is a separate step: Route alerts to Slack, PagerDuty or email.
  • It does not fix anything. An alert can start an investigation that files an issue, and a person decides what happens next: Turn production alerts into GitHub issues.
  • Healthy traces are sampled in hosted environments. Count logs, metrics or error spans; do not divide by a count of healthy spans.

Troubleshooting

A rule is one read-only SELECT or WITH … SELECT. Remove the trailing semicolon.
The query returned no rows. Run it with reliant forge obs query dev --sql "…" without the HAVING clause to see the raw counts, and check the time window: a rule only sees data inside its own INTERVAL.
The check ignores for; the running evaluator waits for it, and evaluates once per interval (default 1m). Also make sure you ran reliant forge env up dev after adding the rule.

Next steps

Route alerts

Send critical alerts to a pager and the rest to Slack.

Add a custom metric

Record a number of your own, then alert on it.

Dashboards and alerts

Every default rule and how to tune or disable it.

Querying with forge obs

The obs verbs, flags and exit codes.