Skip to main content
In short
  • Your services report a span per RPC, your pages report browser errors and page loads, and the platform adds restarts and alerts. Errors are grouped into issues, each tied to the release it first appeared in.
  • A release is judged by comparing a window after its promotion with an equal window before it, per service: requests, error rate and p95 latency, plus the issues, alerts and restarts it opened.
  • There are three verdicts. PASS: we looked, and nothing got worse. FAIL: we looked, and it did. INCONCLUSIVE: not proven good, which is what missing or thin evidence produces. Missing telemetry is never green.
  • One rule computes the verdict for reliant forge obs check, the console and the control plane. On Reliant hosting the control plane judges every promotion once, stores the verdict, raises ReleaseRegressed on a FAIL, and an opt-in health gate can hold the next environment’s promotion on it.
  • Nothing rolls back. Recovering is deploying a fixed or earlier release.

The problem this solves

“Did the deploy work?” usually gets answered by a finished rollout, a health endpoint returning 200 and a glance at a dashboard. None of those says whether the new code is worse than the old. A rollout finishes on a release that throws on one RPC in ten, and a dashboard shows a spike without saying which release caused it. Stitched-together stacks close the gap with canary tooling that needs its own metric queries, per-service thresholds and a separate place to look. And a typical alert treats “no data” as “no problem”: a query that returns nothing fires nothing, so an app that stopped sending telemetry looks healthy. Reliant builds the verdict from data the app already sends, judges every release with the same declared policy, and treats an absence of evidence as an answer of its own.

How it works

Signals: what each release is measured on

What every boundary records covers the instrumentation. Release health reads three kinds of evidence:
  • Server spans. Each service’s requests, error rate and p50/p95/p99 latency come from its own OpenTelemetry server spans, one per RPC, named for the RPC and carrying its Connect status code.
  • Browser telemetry. A scaffolded frontend reports JavaScript errors, failed fetch and XHR calls, and page loads. A frontend is judged on errors per page view, the share of its requests that failed, and its p95 page load, beside any backend or on its own.
  • What the platform sees. Workload restarts in the after window, and alerts that opened in it and are still firing. These count even for an app that sends no traces.
Only errors that are the server’s fault count. A caller’s mistake, such as InvalidArgument, is not an issue, and in the run below an API answering every call unauthenticated measured a 0% error rate. Error levels explains how a status maps to a class.

Sampling is corrected for

On hosting, the collector tail-samples traces: it keeps every trace with an error and every slow one, and a sample of the healthy rest. Counting stored spans naively would inflate error rates roughly tenfold. So each kept span carries its sampling probability in its W3C tracestate, and every count, ratio and percentile is weighted by the inverse of it. The request counts a check reports are estimates of real requests, not counts of stored rows.

Issues tie errors to releases

An issue is one distinct bug. The platform fingerprints server errors by where they were raised, the RPC, the message with its variable parts masked and the kind of failure, never by line number, so an issue keeps its identity across releases; How errors are grouped has the details. Every issue records the version it was first seen in and its state per environment. That gives release health two findings that need no comparison at all:
  • new_issue: an issue first seen in this release, at or after the promotion, with at least new_issue_min_events events (3 by default) in the after window.
  • regressed_issue: a resolved issue that came back at or after the promotion, with at least regressed_issue_min_events events (the same, unless set).
The platform’s NewIssue and IssueRegressed alerts fire on the same issues, so release health judges each as its issue rather than counting the bug twice, unless you made that alert rule critical.

The windows

A promotion is the moment an environment was pointed at a release; the deploy ledger records it as promoted_at. Release health takes a window width w (15 minutes by default, 1 minute to 6 hours) and compares:
  • the before window, [P − w, P): the previous release’s traffic;
  • the after window, [P, P + w): this release’s.
Each window stops at the neighbouring promotion, so a release is never blamed for its predecessor or credited with its successor, and the after window stops at the present. When a promotion’s deploy gates record that a later attempt shipped it, the after window starts then instead.

Attribution: a finding has to be the release’s

Before a finding may fail a release, three questions are asked:
  1. Was the window the live release’s? If the window ended before the release went live, it measured the previous release. The report says window_not_live and the verdict is INCONCLUSIVE.
  2. Did the release serve the window? version_share is the judged release’s share of the after window’s traffic. Under 50%, the rollout had not finished there, rollout_incomplete is set, and a verdict that would have been PASS is INCONCLUSIVE.
  3. Is the finding the release’s? A service’s error rate or latency is not the release’s when another version served most of that service’s traffic. An issue is not the release’s when every one of its after-window events came from another version.
A finding that fails a question stays in the report as a warning, with the reason in findings[].unattributed. Alerts and restarts carry no version, so a low share never excuses them.

The rule: findings, then a verdict

Comparative findings, error_rate, latency and the browser ones, need a baseline. A service is compared only when it has min_requests requests (50 by default) in both windows. An error_rate finding also needs the rise to exceed max_error_rate_increase (one percentage point by default) and at least error_rate_min_errors errors (10 by default) after the promotion, so a handful of failures on a quiet service is not read as a broken release. Where per-RPC figures are available, the rise and the p95 ratio are computed per RPC and weighted by traffic, so a window that happens to hold a different mix of endpoints does not look like a regression. Each finding’s policy is fail when its code is in the environment’s fail_on list and warn otherwise. By default error_rate, new_issue, regressed_issue and alert_firing fail; latency and restarts warn. A critical alert that opened in the after window fails the release whatever fail_on says. Then the verdict, in this order: An INCONCLUSIVE document names its cause in reason_code, in that same order of precedence. The common ones: window_open (check again), no_request_telemetry (no request span and no browser telemetry in either window), thin_baseline (enough requests after the promotion, too few before it, typical of a first deploy) and low_traffic (too few after it). Once the window has closed, a check of the same promotion reads the same window, so the last three are final for that promotion. Release gates says what a pipeline does with each.

Where the verdict goes

reliant forge obs check <env> computes the verdict on request and exits with it: 0 PASS, 1 FAIL, 2 INCONCLUSIVE or could not look, and 8 when --wait ran out before the after window closed. The exit code always equals the document’s exit_code. Against a hosted environment the CLI asks the control plane, which reads the telemetry and runs the rule; against your local stack it reads the local store and runs the same rule itself. On Reliant hosting the control plane also judges every promotion on its own. Two minutes after the after window closes, so late telemetry is in, it judges the promotion with the environment’s declared release_health and no client overrides, and writes the verdict once: one row per promotion, first writer wins, and the table rejects updates. A restart or a second evaluator reads the stored row instead of judging again. reliant forge env releases list <env> shows it in the HEALTH column. A stored FAIL raises a ReleaseRegressed alert naming the release, the previous release, the commit range and the count of failing findings. It fires for an hour, then resolves; PASS and INCONCLUSIVE raise nothing. It never counts against the next release, so a fix promoted right after a bad deploy is not failed by its predecessor’s alert.

The health gate

The gate is how a verdict holds a promotion. With delivery_policy.health_gate.after = ["staging"] in prod’s KCL, the control plane refuses to promote release R to prod until staging’s stored verdict for R is pass. The check runs inside the promotion’s transaction, so no client can skip it; only a recorded --override-health-gate "<why>" gets past. Because a verdict row never changes, a PASS the gate admitted on cannot turn into a FAIL, and a verdict not yet written reads as pending, never as admission. The health gate (opt-in) has the refusal codes and the audit mode.

A real run, three releases

These are real results from a throwaway Go API on Reliant hosting, with release_health { window = "5m", min_requests = 20 } declared in its prod KCL and a loop calling one RPC twice a second. v9.3.0, the first deploy. The traffic loop started only when env deploy returned, seconds before the 5-minute after window closed, and no request landed in either window. No evidence, so the verdict is not a pass:
obs check prod --wait --json (v9.3.0, trimmed)
Every threshold in the document says "source": "release_shape": it came from the environment’s declaration, recorded in the release. v9.4.0, a no-op change, deployed while the traffic loop ran. Both windows had traffic, the new version served 89% of the after window, and nothing got worse:
obs check prod --wait --json (v9.4.0, trimmed)
v9.5.0, a deliberate regression. GetProduct was opened to signed-out callers and made to return an error, and a second loop started calling it. The error rate rose by 31 points with 194 errors, so error_rate fails. The same failure opened new issues first seen in v9.5.0, so new_issue fails too. The p95 ratio passed its limit as well, but latency only warns by default:
obs check prod --json (v9.5.0, trimmed)
The one failure was grouped as two issues, because the request log and the audit record each reported it. window_versions is the attribution check at work: every event of the issue came from v9.5.0, so the issue is the release’s. suggested holds the next commands to run, so you or an agent can go from the verdict straight to the issue. The control plane judged each promotion on its own once its window closed and stored the verdict. env releases list shows it in the HEALTH column:
env releases list prod (WHO column removed)
The fail raised ReleaseRegressed less than three minutes after v9.5.0’s window closed, read here from obs status prod --json:
ReleaseRegressed (trimmed)
Nothing rolled back. v9.5.0 stayed live until the environment was deleted.

Design decisions and trade-offs

One rule, three callers. The rule is a pure function with no I/O and no clock: callers fetch the facts and it decides. obs check, the control plane’s API and the ReleaseRegressed evaluator all call it, so the CLI, the console and the gate cannot disagree. The cost is version skew: a control plane can predate a rule a newer CLI knows, so the CLI re-applies the rules it knows to the control plane’s answer and never prints PASS where the newer rule would hold the release. Compare against the previous release, not a fixed SLO. A relative comparison needs no per-service tuning and catches “this release made it worse” on day one. It misses a service that was already bad, and it cannot judge without a baseline, which is why a first deploy or a quiet environment comes back thin_baseline or low_traffic. A quiet environment lowers min_requests, widens window or sets require_baseline = "none" in its release_health. Three verdicts, and no data is never green. A two-state check has to pick a side when evidence is missing, and most tools pick “no alert”. Reliant makes “not proven good” its own outcome and exit code. The cost is friction: a pipeline on a quiet environment stops at exit 2 until someone accepts that, with --allow-inconclusive on one call or a lower threshold in KCL. Judge once, store forever. A verdict written once can always explain the gate decision that read it, and a PASS cannot later become a FAIL under a promotion admitted on it. The cost: telemetry arriving after the two-minute settle is not in the stored verdict. obs check re-reads the window and can say something different; the gate and the alert go by the stored row. The server decides the gate, not the client. --threshold, --fail-on and --allow-inconclusive change what your command prints and exits with, never the stored verdict or what the gate admits. A CI script cannot loosen production’s bar by accident; loosening it on purpose is a KCL change that ships in a release. Attribution before blame. Without the version checks, a release still rolling out would be judged on its predecessor’s traffic, and an old bug’s events would fail whichever release was live. The cost: a release still rolling out in the window can be INCONCLUSIVE but never PASS. A verdict is evidence, not an actuator. By the time a window closes, the release may have run migrations and written data in its new shape, and rolling back automatically past that is how a regression becomes data loss. So a FAIL alerts and can refuse the next promotion, and reverts nothing.

Limits and what it does not do

  • No automatic rollback. A FAIL raises ReleaseRegressed and, through the opt-in health gate, holds a promotion to the next environment. The failing release stays live until you deploy something else: a fix, or an earlier release your rollback floors admit.
  • The health gate is opt-in and off by default. A project that declares no health_gate is never held. A self-hosted environment with no control plane records the declaration and refuses nothing.
  • No request telemetry, no request verdict. Request figures come only from your services’ own server spans and your pages’ browser telemetry; the platform’s ingress counts are not used, even though the INCONCLUSIVE text mentions the ingress. An app forge did not build, such as one adopted by its Dockerfile, gets no_request_telemetry until it exports OpenTelemetry traces to $OTEL_EXPORTER_OTLP_ENDPOINT. Restarts, alerts and issues are still judged.
  • A static site is judged by real visitors. With no server spans, a site’s only request evidence is the telemetry its pages send from real browsers. curl and uptime pings run no JavaScript, so a site with no visitors stays INCONCLUSIVE no_request_telemetry. The reason text’s advice to add an OpenTelemetry SDK does not apply to a static site; the remedy is page views.
  • Synthetic release probes are not generally available. The default fail_on list includes probe_failed, but the platform runs release probes only where an operator has switched them on. Do not count on probes to judge a quiet environment.
  • The first deploy has no baseline. Its before window holds no traffic, so with the default require_baseline it comes back thin_baseline (or no_request_telemetry if nothing was called), however healthy it is.
  • Short windows are easy to miss. The after window starts at the promotion, not when your traffic does, and env deploy returns only after the whole rollout, database and site included.
  • Locally, only part of the rule runs. A local environment has no promotions, so you name the start with --since, and regressed issues, alerts and restarts are reported as not evaluated.
  • The stored verdict is not re-judged. Late telemetry is not in it. A promotion the evaluator never reached within 72 hours has no stored verdict, which a health gate reads as not proven, and ReleaseRegressed fires only within the hour after the window closed.

See it for yourself

Every flag, exit code and reason code is in the help text of the forge your project pins (forge v1.5.0 or later for thin_baseline and the attribution fields):
terminal
After a hosted deploy, wait for the verdict and pull out the parts this page describes:
terminal
See which threshold came from where (release_shape, default or request):
terminal
Compare with what the control plane stored, which is what the gate reads:
terminal

Guides

Gate prod on release health

Hold a production promotion until staging proves the release good.

Send OpenTelemetry from any app

Turn no_request_telemetry into real request, error and latency data.

Find the release that caused an error

Go from an issue to the version and commits that introduced it.

Find your slowest RPCs

Read p95 latency per RPC, then the trace behind it.

Debug production with an AI agent

Let an agent read the verdict, the issue and the trace for you.

Reference