Before you begin
- A Forge project running locally with
reliant forge env up dev: Build a Go and React app. - Forge v1.6.1 or later in the project (
reliant forge --version).
shop app’s GetProduct RPC: five or more not_found answers in five minutes usually means a broken link somewhere.
1
Try the query first
Write the condition as a ClickHouse Columns other than
SELECT over otel_logs, otel_traces or the otel_metrics_* tables, and run it against the local store:terminal
Output
value become the alert’s labels, and value is its number. Put the threshold in the query with HAVING, so a healthy system returns no rows.2
Declare the rule once for every environment
Rules are per environment, so keep them in a shared file and pass them to each env:In
deploy/kcl/alerts.k
deploy/kcl/prod/main.k
deploy/kcl/dev/main.k, add the same import and alerts = forge.Alerts {rules = al.rules} inside the existing forge.Observability block. for is a KCL keyword, so it is quoted. severity is critical, warning or info.3
Evaluate it once
obs alerts check runs every rule, yours and the defaults, against the local store and prints what returns rows now. It ignores for and notifies nobody. After a few lookups of products that do not exist:terminal
Output
--rule it lists every rule with ok, FIRING or the error its query hit. --sql prints each rule’s query instead of running it.4
Watch it fire on your machine
Run The annotation template filled in the procedure and the count.
reliant forge env up dev again to load the new rule. The stack’s alerts service evaluates the rules every interval with for applied, and sends what fires through a local Alertmanager to the in-app receiver, which writes .forge/logs/dev/alerts.log:terminal
Output (trimmed)
grep 'ALERT firing' .forge/logs/dev/alerts.log finds the same lines on disk.5
Check what the release records
env render refuses a malformed rule before anything ships. A rule labelled severity = "high" fails with:Output
terminal
Output
What this does not do
- It does not decide who gets paged. With no routes declared, alerts go to in-app notifications and then to a receiver named
slack, never to a pager. Routing is a separate step: Route alerts to Slack, PagerDuty or email. - It does not fix anything. An alert can start an investigation that files an issue, and a person decides what happens next: Turn production alerts into GitHub issues.
- Healthy traces are sampled in hosted environments. Count logs, metrics or error spans; do not divide by a count of healthy spans.
Troubleshooting
expr must be a single statement, with no ';'
expr must be a single statement, with no ';'
A rule is one read-only
SELECT or WITH … SELECT. Remove the trailing semicolon.The rule shows ok in obs alerts check but you expected FIRING
The rule shows ok in obs alerts check but you expected FIRING
The query returned no rows. Run it with
reliant forge obs query dev --sql "…" without the HAVING clause to see the raw counts, and check the time window: a rule only sees data inside its own INTERVAL.FIRING in obs alerts check, but nothing in alerts.log
FIRING in obs alerts check, but nothing in alerts.log
The check ignores
for; the running evaluator waits for it, and evaluates once per interval (default 1m). Also make sure you ran reliant forge env up dev after adding the rule.Next steps
Route alerts
Send critical alerts to a pager and the rest to Slack.
Add a custom metric
Record a number of your own, then alert on it.
Dashboards and alerts
Every default rule and how to tune or disable it.
Querying with forge obs
The obs verbs, flags and exit codes.