> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reliantlabs.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Turn production alerts into GitHub issues with an on-call agent

> Get one investigated GitHub issue per new production alert or error: turn on Autopilot on-call, which deduplicates, reads the code and writes up a likely cause.

By the end of this guide, each new production alert or error becomes one GitHub issue, investigated read-only and written up with a severity, evidence, a suspected cause and a proposed fix. A repeat of the same alert comments on that issue instead of opening another.

There is no alert route or token to set up: on Reliant hosting the on-call trigger is declared `webhook: {source: alerting}`, so turning it on connects it to your organization's alerts ([Webhooks](https://docs.reliantlabs.io/automations/webhooks.md#your-organizations-alerts)).

## Before you begin

* A Forge project deployed to a hosted environment, which gets the [default alert rules and error issues](https://docs.reliantlabs.io/hosting/observability/dashboards-and-alerts.md#alerts) ([Hosting overview](https://docs.reliantlabs.io/hosting/overview.md)).
* The Reliant GitHub App installed on the repository the issues should go to ([GitHub integration](https://docs.reliantlabs.io/integrations/github.md)).
* A [machine](https://docs.reliantlabs.io/machines/overview.md) that can hold a checkout of that repository.
* The `reliant` CLI v1.21.3 or later, [signed in](https://docs.reliantlabs.io/cli/sign-in.md), or the app.

**1. Put the repository's checkout in a Reliant project**

The investigator reads the checkout at `<project>/<repo name>`, or the project folder itself, whose `origin` is the alert's repository. Without one, the issue is filed but not investigated.

```bash terminal theme={null}
cd ~/work/acme-app
git clone https://github.com/acme/app.git app
reliant project create --path .
```

The project also scopes the alerts: those of the Forge project its `forge.yaml` names, and of environments deployed from repositories it holds.

**2. Check the workflow's behaviour before it runs for real**

On-call ships with scenarios that mock GitHub and the investigator, so you see what it files, comments on and skips without spending tokens:

```bash terminal theme={null}
reliant workflow scenario run --include-builtins | grep autopilot-oncall
```

```text theme={null}
  ✓ autopilot-oncall/never_sets_the_ready_label (20ms)
  ✓ autopilot-oncall/new_fingerprint_investigated (14ms)
  ✓ autopilot-oncall/new_issue_alert_is_keyed_by_its_issue (17ms)
  ✓ autopilot-oncall/over_the_investigation_cap_files_without_investigating (19ms)
  ✓ autopilot-oncall/resolved_alert_comments_only (11ms)
  ✓ autopilot-oncall/same_fingerprint_dedupes (9ms)
  ✓ autopilot-oncall/search_failure_is_not_treated_as_new (9ms)
  ✓ autopilot-oncall/service_maps_to_its_repository (12ms)
  …
```

All 22 pass. `same_fingerprint_dedupes` shows a re-sent alert getting a comment, and `search_failure_is_not_treated_as_new` that it files nothing when it cannot rule out a duplicate ([scenario format](https://docs.reliantlabs.io/workflows/testing.md)).

**3. Turn on-call on**

In the app, open **Workflows → Library → Autopilot · on-call** and choose **Set up**. From the terminal, inside the project folder:

```bash terminal theme={null}
reliant trigger activate --workflow builtin://autopilot-oncall \
  --machine <machine id or hostname> --name-prefix "acme-app" \
  --params-json '{"service_repos": {"web": "acme/app"}, "default_repo": "acme/app"}'
```

```text theme={null}
TRIGGER       OUTCOME  ACTIVATION                                                   NOTE
alertmanager  enabled  acme-app · alertmanager (07cb9eab-a972-4d2b-84e1-f032606d8c23)  -
```

`service_repos` maps an alert's service label to a repository; `default_repo` takes the rest, or leave it empty to ignore unmapped alerts. Other settings are in [Autopilot on-call settings](https://docs.reliantlabs.io/guides/autopilot.md#on-call).

**4. Confirm it is connected to your alerts**

Turning it on creates an alert receiver named after the activation. From the Forge project:

```bash terminal theme={null}
reliant forge obs receivers list --json
```

```json theme={null}
{
  "name": "autopilot-07cb9eab-a972-4d2b-84e1-f032606d8c23",
  "type": "RECEIVER_TYPE_AUTOPILOT",
  "settings": {
    "always_alertnames": "NewIssue,IssueRegressed,IssueSpike,ReleaseRegressed,ServiceAnomaly",
    "env_kinds": "persistent,self_managed",
    "projects": "guides-sandbox-oncall",
    "repos": "github.com/reliant-labs/guides-sandbox",
    "severities": "critical,error"
  }
}
```

It takes `critical` and `error` alerts, plus the error and release alerts at any severity, from every environment of this project that raises alerts (`persistent` ones Reliant hosts and `self_managed` ones such as an environment `forge obs init` registered), not previews. Your other receivers are unchanged.

**5. Make an alert fire**

Any real error will do. Here a deliberately broken function threw a `TypeError` on every page view; within minutes it was an issue with a firing `NewIssue` alert:

```bash terminal theme={null}
reliant forge obs status prod
```

```text theme={null}
ISSUE                                 STATUS  SERVICE                    EVENTS  LAST SEEN  TITLE
74c29185-252c-428c-9d32-894fd570b0f3  new     guides-sandbox-oncall-web  4       2m         TypeError: Cannot read properties of undefined (reading <st…

FIRING   NewIssue: New issue in guides-sandbox-oncall-web (prod): TypeError: Cannot read properties of undefined (reading ‹str›). Details: https://forge.reliantlabs.io/forge/p/guides-sandbox-oncall/observability/errors?env=prod&issue=74c29185-…
```

**6. Check the issue it filed**

Each notification is a firing of the activation. A new one starts a run under **Workflows → Runs**; a repeat within four hours starts nothing.

```bash terminal theme={null}
reliant trigger events <activation id>
```

For a new alert or error, the run files an issue in the mapped repository and labels it `agent:triage`:

* **Title:** `[error] <service>: <error title>` or `[alert] <alertname>: <summary>`, then the prefix plus the investigator's one-line summary.
* **Body:** severity, confidence, the key it deduplicates on (an error's issue id or an alert's fingerprint), then **Summary**, **Evidence**, **Suspected cause** and **Proposed fix**.

If the investigator ends without an answer, a stub says so and the run is marked failed.

**7. Decide what happens next**

The issue stays `agent:triage` until a person acts on it:

* **Have an agent fix it:** replace `agent:triage` with `agent:ready`. If the [issue worker](https://docs.reliantlabs.io/guides/autopilot.md#issue-worker) is on for that repository, it writes a failing test and the fix, and opens a pull request for you to review and merge.
* **Fix it yourself**, or **close it** as noise. A later alert with the same key opens a new issue only once this one is closed.

**8. Turn it off when you are done**

To stop on-call for this project, deactivate it in **Workflows → Automations**, or:

```bash terminal theme={null}
reliant trigger deactivate --workflow builtin://autopilot-oncall --project-id <project id>
```

Without `--project-id` it pauses every project's on-call. Pausing or deleting removes the alert receiver.

## What this does not do

* **It does not fix or merge anything.** On-call only files and comments; its investigator has no edit tools, no GitHub write tools and a shell with no credentials. Alert to fix is two linked steps with a person at each gate: a person labels the issue `agent:ready`, the issue worker opens a pull request, and a person merges it. Autopilot never adds `agent:ready` and never merges.
* **It does not read your logs or traces.** For an error it gets the service, title, culprit, first build and a console link, works from those and the code, and says what it could not reach.
* **It does not see every alert.** Other warnings, preview environments and [silenced](https://docs.reliantlabs.io/hosting/observability/dashboards-and-alerts.md#alerts) alerts never reach it. By default it investigates five new alerts per notification (`max_investigations`) and files the rest without a write-up.
* **It does not page anyone.** It runs beside your Slack, PagerDuty or email receivers.

## Troubleshooting

> [accordion] no project workflow is named "autopilot-oncall"
>
> A built-in workflow is named with its scheme on the command line: `--workflow builtin://autopilot-oncall`.

> [accordion] No checkout of acme/app was found under …, so … was filed but not investigated
>
> No clone of that repository at `<project>/<repo name>` or the project root, or its `origin` points elsewhere. Clone it there.

> [accordion] on-call: none of the 1 alert(s) has a usable fingerprint and a repository in service\_repos / default\_repo, so nothing was filed.
>
> No `service_repos` key matched and `default_repo` is empty. Run `reliant trigger activate` again with new `--params-json`.

> [accordion] No firings yet.
>
> `obs status` shows an alert firing but `reliant trigger events` lists nothing. Check the alert passes the filter (step 4) and that `obs receivers list` shows the `autopilot-<activation id>` receiver. If its `env_kinds` is `persistent` alone, it was created before self-managed environments were included: pause and resume the activation to recreate it with the current filter. Pausing and resuming the activation connects its receiver again. `NewIssue` fires once per issue, so test again with a new error.

## Next steps

> [card] [Autopilot: alerts and errors](https://docs.reliantlabs.io/guides/autopilot.md#alerts-and-errors)
>
> Deduplication and what the investigator sees.

> [card] [Go from a production error to a reviewed fix](https://docs.reliantlabs.io/guides/workflows/production-error-to-fix.md)
>
> From an on-call issue to a merged fix.

> [card] [Dashboards and alerts](https://docs.reliantlabs.io/hosting/observability/dashboards-and-alerts.md)
>
> Default alert rules and error issues.

> [card] [Webhooks](https://docs.reliantlabs.io/automations/webhooks.md)
>
> The `source: alerting` trigger, and self-hosted Alertmanager.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.