> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reliantlabs.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Add a custom metric without a cardinality explosion

> Declare a business metric with a unit and a closed set of attribute keys, record it from a handler, and query it with forge obs without one series per user.

By the end of this guide your API records a metric of its own, a distribution of product prices split by stock state, and you can query it next to the built-in RPC metrics. The usual way a custom metric goes wrong is an attribute such as a user id or a product name: each distinct value becomes its own time series, and the metrics store grows without bound. In a Forge app every metric is declared with the attribute keys it may carry, and a key you did not declare never reaches the store.

[Signals](https://docs.reliantlabs.io/hosting/observability/signals.md) describes what is recorded without any code. This guide covers what is not.

## Before you begin

* A Forge Go project running locally with `reliant forge env up dev`: [Build a Go and React app](https://docs.reliantlabs.io/guides/forge/build-go-react-app-connect-rpc.md).
* Forge v1.6.1 or later in the project (`reliant forge --version`).

**1. Declare the metric**

Every metric lives in `internal/metrics/metrics.go`, which the scaffold created with one example. Add yours below it: a name, a [UCUM](https://ucum.org/) unit, a description and the attribute keys it accepts.

```go internal/metrics/metrics.go theme={null}
// productPrice is the distribution of new products' prices. The price is the
// measurement, never an attribute; in_stock is a closed set of two values.
var productPrice = metrics.Default().Int64Histogram(metrics.Spec{
	Name:        "catalog.product_price",
	Unit:        "{cent}",
	Description: "Price of each product created, by stock state.",
	Attributes: []metrics.Attribute{
		{Key: "in_stock", Values: []string{"true", "false"}},
	},
	Since:   "v0.2.0",
	Buckets: []float64{100, 1_000, 10_000, 100_000},
})

// RecordProductPrice records the price of one product that was created.
func RecordProductPrice(ctx context.Context, priceCents int64, inStock bool) {
	productPrice.Record(ctx, priceCents, metric.WithAttributes(
		attribute.Bool("in_stock", inStock),
	))
}
```

Choose `Int64Counter` for a count that only rises, `Int64UpDownCounter` for one that rises and falls, `Int64Gauge` for a latest value and a histogram for a distribution. `Values` closes an attribute to a fixed set; leave it out only for a key bounded some other way, such as a route. Keep the instrument unexported and expose a function, so callers cannot attach an attribute you did not plan for.

**2. Record it**

Call the function where the event happens. The scaffolded `CreateProduct` already counts creations; add the price after a successful create:

```go internal/handlers/catalog/handlers_crud.go theme={null}
resp, err := crud.HandleCreate(s.crudCreateProductOp())(ctx, req)
metrics.RecordCreated(ctx, "Product", err)
if err == nil {
	metrics.RecordProductPrice(ctx, req.Msg.GetPriceCents(), req.Msg.GetInStock())
}
return resp, err
```

The dev stack rebuilds the API when you save.

**3. Check the declaration**

The declared set is a schema you can print without running a query:

```bash terminal theme={null}
reliant forge obs metrics declared
```

```text Output (trimmed) theme={null}
NAME                                     TYPE           UNIT          STABILITY     SINCE    ATTRIBUTES
catalog.product_price                    histogram      {cent}        experimental  v0.2.0   in_stock=true|false
forge.metrics.rejected_attributes        counter        {attribute}   experimental  v0.1.45  metric, reason=undeclared_key|undeclared_value
records_created                          counter        {record}      experimental  v0.1.0   entity, outcome=ok|failed
rpc.server.duration                      histogram      s             stable        v0.1.0   procedure
```

`forge build` writes the same schema into the release, and a hosted environment draws a Custom metrics dashboard from it.

**4. Query it**

Create a few products, wait for the next export (every 60 seconds), then ask what the store holds:

```bash terminal theme={null}
reliant forge obs metrics describe dev catalog.product_price --since 15m
```

```text Output theme={null}
catalog.product_price  histogram  cumulative  unit {cent}
Price of each product created, by stock state.
services: api
about 2 series, exported every 60s
declared: experimental

ATTRIBUTE  KIND    VALUES  NOTE
in_stock   metric  2       enumerated: true, false

break down by: in_stock
```

Two series, however many products you create. Break the distribution down by the attribute:

```bash terminal theme={null}
reliant forge obs metrics query dev catalog.product_price --agg p50 --group-by in_stock --kind table --since 15m
```

```text Output theme={null}
A  catalog.product_price  histogram p50  unit {cent}
table panel, step 60s
SERIES  GROUP                VALUE
A       map[in_stock:false]  32500
A       map[in_stock:true]   3250
```

Quantiles are estimated from the bucket boundaries you declared, so pick boundaries around the values you care about. Replace `dev` with a hosted env name to read production.

**5. See the guard catch a stray attribute**

Suppose someone adds the product's name to the measurement: `attribute.String("product_name", name)`. In development the registry panics at that call, so the mistake is found where it was made. The request fails with `internal`, and the API log says why:

```text .forge/logs/dev/api.log (trimmed) theme={null}
{"level":"ERROR","msg":"panic recovered","procedure":"/services.catalog.v1.CatalogService/CreateProduct","panic":"metric \"catalog.product_price\": attribute \"product_name\" is not declared; add it to the metric's Attributes or stop recording it","exception.type":"*metrics.AttributeError",…}
```

Outside development the registry never panics. It drops the undeclared key, replaces a value outside a declared set with `_other`, and counts each correction on `forge.metrics.rejected_attributes` by metric and reason, so you can alert on it.

## What this does not do

* **It does not stop a bad declaration.** An attribute declared without `Values` accepts any value. Never declare an id, an email or a URL as an attribute; put it on a span or a log line instead.
* **It does not cover instruments it did not create.** Observable (callback) instruments and metrics a third-party library creates for itself are listed in the schema but not guarded.
* **It does not alert by itself.** To be told when the metric crosses a line, write a rule over `otel_metrics_histogram` or `otel_metrics_sum`: [Write alert rules as code](https://docs.reliantlabs.io/guides/observability/custom-alert-rules-as-code.md).

## Troubleshooting

> [accordion] declared, but no data in the window
>
> The schema has the metric but no sample has arrived. Metrics are exported every 60 seconds, so wait a minute after the first recording, and check that the code path ran (`grep CreateProduct .forge/logs/dev/api.log`).

> [accordion] attribute "…" has value "…", which is not one of its declared values
>
> The key is declared with a closed set and the value is not in it. Add the value to `Values`, or map it to one of the declared values before you record.

## Next steps

> [card] [Alert rules as code](https://docs.reliantlabs.io/guides/observability/custom-alert-rules-as-code.md)
>
> Turn a metric or a log pattern into an alert and test it locally.

> [card] [Find your slowest endpoints](https://docs.reliantlabs.io/guides/observability/find-slowest-rpcs.md)
>
> The built-in RPC metrics and traces, per endpoint.

> [card] [Signals](https://docs.reliantlabs.io/hosting/observability/signals.md)
>
> What a Forge app records at every boundary with no code.

> [card] [Querying with forge obs](https://docs.reliantlabs.io/hosting/observability/forge-obs.md)
>
> Every metrics verb and flag.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.