forge obs and the on-call workflow work with.
Dashboards
Forge generates dashboards from your project’s own shape every time you runforge generate: an overview of every service, one dashboard per service with a row for every RPC, a frontend dashboard (errors, sessions, page views, failed API calls, web vitals by route, errors by release), a workloads dashboard and a releases dashboard. They are cut into every release, so the console shows the set that belongs to the release an environment is running, and a rollback shows the older set.
The Dashboards page lists three kinds, and the difference decides what you can do with each:
A workspace dashboard is edited in place. Edit layout lets you drag panels, set their width in quarters of the grid and remove them; Add panel opens a builder where you choose a visualization, a signal, series with an aggregation (count, p50/p95/p99, average, max, sum, distinct), filters, group-by and units, with a live preview over the dashboard’s scope and a Spec tab holding the panel’s JSON. Saving is checked against the version you opened, so two people editing at once cannot silently overwrite each other. When a workspace dashboard is worth keeping, Promote to code gives you the forge dashboard file to commit, and from then on it ships with your releases as a repo dashboard.
Every dashboard has environment, service and time variables, compares against the previous period or the previous release, refreshes on an interval, can be shared as a link or shown full screen, and draws a marker for each release and a band wherever an alert was firing. Clicking into a chart drills down to the matching logs, traces or workload. Star a dashboard and it appears under Dashboards in the navigation.
Dashboards as code
Panels are forge’s dashboard spec, so a dashboard file in your repo and a panel built in the console are the same thing. Besides time series, numbers, tables and log searches, a panel can be a bar chart, a heatmap, a read-only ClickHousesql query, a product funnel, a list of frontend issues or of browser sessions, and a line or number panel can plot a formula over its series (A / B * 100).
To keep a dashboard in your repository, put its JSON in deploy/observability/dashboards/ and commit it; the next release carries it. The quickest ways to get that JSON:
- Promote to code on a workspace dashboard you built in the console.
- Copy a generated file to change it:
cp <project>-overview_gen.json <project>-overview.json, edit the copy, and forge stops generating the original while yours exists. Never edit a*_gen.jsondirectly; the nextforge generaterefuses the drift. Metrics you declare ininternal/metrics/metrics.goget a generated custom-metrics dashboard in every release without a file of your own.forge obs dashboards checkvalidates the files and lists what the next release would carry. Agents can load theobservability/dashboardsskill for the full spec.
Alerts
Alert rules are code. Forge generates a default set for every project, and you tune or disable each one per environment indeploy/kcl/<env>/main.k, so promoting a release applies its rules and rolling back restores the previous ones.
Each default has thresholds you can change, and your own rules are a ClickHouse
SELECT where every returned row is a firing alert:
forge env render refuses a malformed rule before anything ships, and forge obs alerts check evaluates your rules against the local stack. Agents can load the observability/alerts skill for the rule format and every default’s parameters.
A route sends alerts to a receiver, and a receiver is only a name in your code. What the name points at, a Slack channel, an email list, a webhook or a PagerDuty service, is set in Observability → Alerts → Receivers, because a webhook URL or a routing key does not belong in a repository. The built-in In-app receiver needs no setup. With no routes declared, alerts go to in-app notifications and then to a receiver named slack, and never to a pager until you route them there. The Receivers tab calls out a route that names a receiver you have not set up. Creating or changing receivers needs the organization admin role; their secrets are write-only.
The same page shows what is Firing, the alert History, the declared Rules and Silences, which mute matching alerts for a while without changing code.
Frontend errors, issues and source maps
The browser SDK reports an error with a stack from your minified bundle, which changes with every build. Forge builds every deployed frontend with source maps, moves the maps out of the files it serves, and files them privately with the release, so your users never download them. The platform resolves each new error stack through the maps of the build that produced it, so the same bug is recognised as one issue across releases, and tracks each issue per environment as new, unresolved or regressed. Theissues panel on the frontend dashboard lists errors grouped by stack, most frequent first, with the page they happened on, the release, how often and since when, and a sample trace.
Your frontend’s forge.Frontend declaration can tune this:
ignore still reaches the store but is never raised as an issue. sourcemaps = "public" serves the maps instead of filing them, and "off" drops them. A funnel names events your frontend records with track("signup_started") from @reliantlabs/forge-web-runtime; forge draws each declared funnel as a panel on the frontend dashboard.
Who and where
The console shows, for each browser session, error and trace, who it was and where they were:- User: the id your frontend passes to
setTelemetryUser(id), shown as given (pass an internal id, never an email). Backend spans carry the id of the caller your forge service authenticated, so a trace shows its user even when the browser never named one. Sessions, errors and traces can be filtered by user id. - Location: country, region and city, derived by the platform when your browser telemetry arrives, from the address it came from (a local DB-IP City Lite lookup, CC BY 4.0). The IP address itself is dropped: it is not stored and your app never sees it. A page cannot set its own location.
- Session links: every request your frontend makes carries its session id, which your forge services record on their side of the request. A session’s page lists its traces, and a trace opens its session, and its replay when it has one.
observe.IdentitySpanInterceptor and a @reliantlabs/forge-web-runtime that sends the session as baggage; until your project pins both, users and session links come from browser spans only. The console’s user and session filters also need a control plane that applies them: on one that does not, it says so instead of showing unfiltered results.
Which device, client and build
The console also shows which device and client a session, error or trace came from, and which builds served it, so a regression can be tied to a browser, an OS or a release:- Device and client: browser and version (or, for an Electron app, the Electron and Chromium versions), OS and version, and device type (desktop, mobile, tablet or bot). The platform derives them when your browser telemetry arrives, from the request’s own
User-Agentand client-hint headers, which page script cannot set; a page cannot claim a different device. - What your frontend reports: its build (
service.version), the viewport and screen size, the low-entropy client hints ofnavigator.userAgentData, and, in an Electron app whose preload exposeswindow.forgeDesktop = { appVersion, electronVersion, chromiumVersion }, the desktop app’s version (@reliantlabs/forge-web-runtime0.13.1 or later). - Versions: every service of a trace with its own
service.version(your services ship separately), and on a session the forge release its frontend build shipped in, with the build id beneath. A frontend deployed outside a release has no release to name, so its sessions show the build id.
signals = forge.ObservabilitySignals {browser = False} (or enabled = False) turns it off with the rest. If your privacy notice must say what you collect: an approximate location derived from the IP address, not the address, and the browser, OS, device type and app version derived from the User-Agent.
Session replay
Session replay records what a user’s browser displayed, so it is opt-in. It is switched on per frontend in the environment’s KCL withreplay = True in forge.BrowserTelemetry, and nothing records until the app is in the full analytics tier with consent granted and the visitor’s browser sends no Global Privacy Control or Do Not Track signal. When replay is on, all text and every input is masked and canvases are not recorded, and there is no setting that records unmasked. replay_sample_rate in forge.BrowserTelemetry records only a share of sessions. The sessions panel marks the sessions that have a recording. The build variables VITE_OTEL_REPLAY and NEXT_PUBLIC_OTEL_REPLAY and the environment-wide signals = forge.ObservabilitySignals {replay = True} still work but are deprecated. Session timelines and replay has the full set of conditions, what is masked, and how to find a session from an error.
How errors are grouped
An issue is one distinct bug. Server errors are grouped by a fingerprint that is stable across releases, so an issue keeps its identity when a line number moves. The fingerprint is built from where the error was raised (the function), the RPC it happened in, the message with its variable parts masked out, and the kind of failure it carries (the error’s Go type or its SQLSTATE). It never uses a line number, because lines move between releases and would split one issue into many. An error that carries an explicit group key uses that instead. The kind of failure matters. When a genericTransaction failed message covers UTF-8 corruption, serialization failures and timeouts, folding the SQLSTATE into the fingerprint keeps them as separate issues, each of which has a different fix. Only errors that count as the server’s fault are grouped (see error levels); a caller’s InvalidArgument is never an issue. Browser errors are grouped by their stack after source maps resolve it.
Each issue tracks its state per environment: new, unresolved, resolved or regressed. NewIssue, IssueRegressed and IssueSpike alerts fire on those transitions, once each. forge obs issues and forge obs issue <id> read the same issues from the terminal, locally and hosted; see Querying with forge obs.