> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reliantlabs.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Find your slowest API endpoints with per-RPC p95 latency

> Rank every RPC by p95 latency with forge obs endpoints, open its slowest traces, and see the N+1 query or CPU work behind it, locally or in production.

Your API is slow and you want to know which call, and why. Every Go service Forge builds traces each RPC with no instrumentation code, so `forge obs endpoints` can rank all of them by p95 latency, and the slowest traces show where the time went. This guide finds an N+1 query this way, on a local stack and in production.

The same verbs read `dev` (the stack `reliant forge env up dev` runs on your machine) and a hosted environment such as `prod`: the env's declaration picks the backend. [Querying with forge obs](https://docs.reliantlabs.io/hosting/observability/forge-obs.md) covers every verb.

## Before you begin

* A Forge API with some traffic: running locally with `reliant forge env up dev` ([Build a Go and React app](https://docs.reliantlabs.io/guides/forge/build-go-react-app-connect-rpc.md)), or deployed ([Deploy a Go API with managed Postgres](https://docs.reliantlabs.io/guides/hosting/deploy-go-api-managed-postgres.md)).
* Forge v1.6.1 or later in the project (`reliant forge --version`).
* For a hosted environment, `obs:read` on the organization. Under Reliant your sign-in already carries it.

The app used here has two RPCs that are slow on purpose: `GetCatalogReport` lists 200 products and then loads each one again by ID, and `FingerprintCatalog` hashes its input a million times.

**1. Rank endpoints by p95**

```bash terminal theme={null}
reliant forge obs endpoints prod --since 15m --sort p95
```

```text Output theme={null}
SERVICE  ENDPOINT                                               REQUESTS  ERRORS  RATE    P50    P95    P99
api      services.catalog.v1.CatalogService/FingerprintCatalog  152       0.0%    0.17/s  1.38s  1.50s  1.58s
api      services.catalog.v1.CatalogService/GetCatalogReport    134       0.0%    0.15/s  833ms  914ms  1.07s
```

Each row is one RPC with its request count, error rate and latency percentiles. `--sort` takes `requests`, `p95`, `errors` or `rate`, and `--top` sets how many rows (default 10). Use `dev` instead of `prod` for your local stack.

**2. List the slowest traces for one RPC**

`--route` matches the root span name, and `--slow` sorts slowest first:

```bash terminal theme={null}
reliant forge obs traces prod --route GetCatalogReport --slow --since 15m --limit 2
```

```text Output (trimmed) theme={null}
85 traces, 0 with errors; p50 833ms  p95 937ms  p99 1.07s

TRACE                             START     DURATION  SERVICE  ROOT                                                 SPANS
e67dddb2789975b2260105d63f556e48  11:22:29  1.07s     api      services.catalog.v1.CatalogService/GetCatalogReport  403 spans
64c604333fd35cefad79da69675db94d  11:20:41  937ms     api      services.catalog.v1.CatalogService/GetCatalogReport  403 spans
```

403 spans for one request is already a hint. `--min-duration 500ms` keeps only traces at least that long.

**3. Read one trace**

```bash terminal theme={null}
reliant forge obs trace e67dddb2789975b2260105d63f556e48 prod
```

```text Output (trimmed) theme={null}
trace e67dddb2789975b2260105d63f556e48  1.07s  403 spans
* api services.catalog.v1.CatalogService/GetCatalogReport  1.07s
 └─  api orm.ListProduct  4.6ms
    └─  api SELECT products  1.4ms
 └─  ×199 api orm.GetProduct  min 0.8ms p50 1.2ms p95 47ms max 86ms
*└─  api orm.GetProduct  87ms  db="SELECT \"products\".\"id\", \"products\".\"name\", …"
*   └─  api SELECT products  86ms
```

The tree folds the 199 repeated `orm.GetProduct` spans into one line with their spread: one query per product after the list, an N+1. The `*` marks the critical path.

A slow RPC with no child spans looks different. Here is `FingerprintCatalog` on the local stack:

```text Output theme={null}
trace aa4360c812ad4101cd1f01938043034a  113ms  1 spans
* api services.catalog.v1.CatalogService/FingerprintCatalog  113ms
  └─ ⋯ uninstrumented 113ms of 113ms (100%): no child spans; the time went to code that emitted no span
```

No database or outbound call took the time, so it went to your own code. A CPU profile shows which function: [Find CPU hotspots with continuous profiling](https://docs.reliantlabs.io/guides/observability/find-cpu-hotspots-continuous-profiling.md).

**4. See the query behind any verb**

`--explain` (alias `--print-sql`) prints the compiled SQL and its parameters instead of running it:

```bash terminal theme={null}
reliant forge obs trace e67dddb2789975b2260105d63f556e48 prod --explain
```

```text Output (trimmed) theme={null}
-- trace spans (compiled locally; the control plane runs it against the org's own database)
SELECT Timestamp, SpanId, ParentSpanId, SpanName, SpanKind, ServiceName, Duration, StatusCode, …
FROM default.otel_traces
WHERE Timestamp >= fromUnixTimestamp64Nano({p1:Int64}) AND Timestamp < fromUnixTimestamp64Nano({p2:Int64})
  AND TraceId = {p3:String}
```

`obs endpoints` and `obs traces` take `--explain` too. Copy a statement into `reliant forge obs query prod --sql "…"` when the structured verbs cannot express what you need, for example grouping by a span attribute.

**5. Fix it and verify**

Here the fix was to total the prices from the list instead of loading each product again. After the change, the same two reads on the local stack:

```bash terminal theme={null}
reliant forge obs endpoints dev --since 90s --sort p95
reliant forge obs trace e60e8480971b2308433113211e8c8dc6 dev
```

```text Output theme={null}
SERVICE  ENDPOINT                                               REQUESTS  ERRORS  RATE    P50    P95    P99
api      services.catalog.v1.CatalogService/FingerprintCatalog  715       0.0%    7.94/s  80ms   90ms   99ms
api      services.catalog.v1.CatalogService/GetCatalogReport    715       0.0%    7.94/s  0.8ms  1.1ms  3.7ms

trace e60e8480971b2308433113211e8c8dc6  0.8ms  3 spans
* api services.catalog.v1.CatalogService/GetCatalogReport  0.8ms
*└─  api orm.ListProduct  0.7ms
*   └─  api SELECT products  0.3ms
```

`GetCatalogReport`'s p95 fell from 29ms to 1.1ms locally, and each trace has 3 spans instead of 403. In production, deploy the fix as a new release and run `reliant forge obs check prod --wait --json`, which compares p95 per endpoint before and after the promotion: [How release verdicts work](https://docs.reliantlabs.io/deep-dives/how-release-verdicts-work.md).

> [!NOTE]
>
> Numbers differ between environments. Locally the API had the machine's CPU; the hosted `api` workload requests a quarter of a core, so the same hashing loop took 80ms on the laptop and 1.4s in production. Compare an RPC against itself in one environment, not across them.

## What this does not do

* `obs endpoints` ranks server spans that your services exported. An app Forge did not build has no rows until it sends OpenTelemetry traces: [Send OpenTelemetry from any app](https://docs.reliantlabs.io/guides/observability/send-opentelemetry-from-any-app.md).
* It reads; it does not alert. Latency alerts and dashboards are generated from the code: [Hosted dashboards, alerts and issues](https://docs.reliantlabs.io/hosting/observability/dashboards-and-alerts.md).
* A slow release is not rolled back for you. You recover by deploying a fixed or earlier release.

## Troubleshooting

> [accordion] exit 2 from obs endpoints
>
> Forge could not look: the local stack is down (start it with `reliant forge env up dev`), the credential lacks `obs:read`, or a read was rate limited. It does not mean "no slow endpoints". See [the exit codes](https://docs.reliantlabs.io/hosting/observability/forge-obs.md#the-exit-codes).

> [accordion] /bin/sh: 1: forge: not found
>
> `env up dev` has only `reliant forge` to call. The fix is in [Build a Go and React app](https://docs.reliantlabs.io/guides/forge/build-go-react-app-connect-rpc.md#troubleshooting).

## Next steps

> [card] [Find CPU hotspots](https://docs.reliantlabs.io/guides/observability/find-cpu-hotspots-continuous-profiling.md)
>
> When a trace says "uninstrumented", find the function with a CPU profile.

> [card] [Debug production with an AI agent](https://docs.reliantlabs.io/guides/observability/debug-production-with-ai-agent.md)
>
> Let an agent run these reads and follow a failing request to the line.

> [card] [Querying with forge obs](https://docs.reliantlabs.io/hosting/observability/forge-obs.md)
>
> Every verb, the JSON contract and the exit codes.

> [card] [How release verdicts work](https://docs.reliantlabs.io/deep-dives/how-release-verdicts-work.md)
>
> How `obs check` judges p95 per endpoint after a deploy.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.