Before you begin
- A Forge Go project running locally with
reliant forge env up dev: Build a Go and React app. - Forge v1.6.1 or later in the project (
reliant forge --version).
1
Declare the metric
Every metric lives in Choose
internal/metrics/metrics.go, which the scaffold created with one example. Add yours below it: a name, a UCUM unit, a description and the attribute keys it accepts.internal/metrics/metrics.go
Int64Counter for a count that only rises, Int64UpDownCounter for one that rises and falls, Int64Gauge for a latest value and a histogram for a distribution. Values closes an attribute to a fixed set; leave it out only for a key bounded some other way, such as a route. Keep the instrument unexported and expose a function, so callers cannot attach an attribute you did not plan for.2
Record it
Call the function where the event happens. The scaffolded The dev stack rebuilds the API when you save.
CreateProduct already counts creations; add the price after a successful create:internal/handlers/catalog/handlers_crud.go
3
Check the declaration
The declared set is a schema you can print without running a query:
terminal
Output (trimmed)
forge build writes the same schema into the release, and a hosted environment draws a Custom metrics dashboard from it.4
Query it
Create a few products, wait for the next export (every 60 seconds), then ask what the store holds:Two series, however many products you create. Break the distribution down by the attribute:Quantiles are estimated from the bucket boundaries you declared, so pick boundaries around the values you care about. Replace
terminal
Output
terminal
Output
dev with a hosted env name to read production.5
See the guard catch a stray attribute
Suppose someone adds the product’s name to the measurement: Outside development the registry never panics. It drops the undeclared key, replaces a value outside a declared set with
attribute.String("product_name", name). In development the registry panics at that call, so the mistake is found where it was made. The request fails with internal, and the API log says why:.forge/logs/dev/api.log (trimmed)
_other, and counts each correction on forge.metrics.rejected_attributes by metric and reason, so you can alert on it.What this does not do
- It does not stop a bad declaration. An attribute declared without
Valuesaccepts any value. Never declare an id, an email or a URL as an attribute; put it on a span or a log line instead. - It does not cover instruments it did not create. Observable (callback) instruments and metrics a third-party library creates for itself are listed in the schema but not guarded.
- It does not alert by itself. To be told when the metric crosses a line, write a rule over
otel_metrics_histogramorotel_metrics_sum: Write alert rules as code.
Troubleshooting
declared, but no data in the window
declared, but no data in the window
The schema has the metric but no sample has arrived. Metrics are exported every 60 seconds, so wait a minute after the first recording, and check that the code path ran (
grep CreateProduct .forge/logs/dev/api.log).attribute "…" has value "…", which is not one of its declared values
attribute "…" has value "…", which is not one of its declared values
The key is declared with a closed set and the value is not in it. Add the value to
Values, or map it to one of the declared values before you record.Next steps
Alert rules as code
Turn a metric or a log pattern into an alert and test it locally.
Find your slowest endpoints
The built-in RPC metrics and traces, per endpoint.
Signals
What a Forge app records at every boundary with no code.
Querying with forge obs
Every metrics verb and flag.