> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reliantlabs.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Find CPU hotspots with continuous profiling

> Find the Go function burning your CPU: turn on continuous profiling, read the flamegraph, export pprof with forge obs profile, and narrow it to one trace.

A trace can tell you an RPC spent its time in your own code. A CPU profile tells you which function. Every Go service Forge builds profiles itself in-process and ships the samples beside its traces, so you get a flamegraph without attaching a profiler, and each sample is tagged with the request it ran for.

This guide finds a hashing loop that dominates an API's CPU, on a local stack and in production. [Continuous profiling](https://docs.reliantlabs.io/hosting/observability/profiling.md) is the reference: cadence, sample types, retention.

## Before you begin

* A Forge Go API, running locally with `reliant forge env up dev` or deployed: [Deploy a Go API with managed Postgres](https://docs.reliantlabs.io/guides/hosting/deploy-go-api-managed-postgres.md).
* Forge v1.6.1 or later in the project (`reliant forge --version`), and Go installed for `go tool pprof`.
* An RPC you suspect. [Find your slowest API endpoints](https://docs.reliantlabs.io/guides/observability/find-slowest-rpcs.md) gets you there: a slow trace whose time is `uninstrumented` points at CPU.

The app here has an RPC, `FingerprintCatalog`, that calls `stretchFingerprint`, which runs SHA-256 a million times.

**1. Turn profiling on**

Locally, profiling is on by default: `env up dev` runs a profiles collector beside the rest of the telemetry. A hosted environment collects profiles only when its `main.k` says so. The default cadence is a 5-second CPU window every 5 minutes; while you investigate, raise it on one workload through its `env` (at least `30s`):

```kcl deploy/kcl/prod/main.k theme={null}
_api = fw.Workload {
    name = "api"
    # … unchanged …
    env = {"FORGE_PROFILES_CPU_INTERVAL" = "30s"}
}

_bundle = forge.Bundle {
    # … unchanged …
    observability = forge.Observability {
        signals = forge.ObservabilitySignals {profiles = True}
    }
}
```

Deploy a release as usual. The change applies from that deploy.

**2. Send traffic**

The profiler samples what the process is doing, so the RPC must run during a capture window. A few minutes of steady calls is enough:

```bash terminal theme={null}
end=$(( $(date +%s) + 420 ))
while [ "$(date +%s)" -lt "$end" ]; do
  curl -s -o /dev/null -XPOST -H 'content-type: application/json' \
    -d '{"snapshot":"spring-2026"}' "$API_URL/services.catalog.v1.CatalogService/FingerprintCatalog"
done
```

**3. Read a hosted profile in Apps**

Open **Apps**, pick the project and environment, and choose **Profiles** under Observability. It draws a flamegraph of one service's CPU, memory or goroutines over the range you pick, with the heaviest functions beside it, and opening it from a trace narrows it to that trace.

Here, the production `api` held 12.3s of sampled CPU over 30 minutes in 176 stacks. `stretchFingerprint` was on the stack for 69.5% of it, and the SHA-256 block function alone (`sha256.blockAVX2`) was 54.9%.

**4. Export a local profile for pprof**

`forge obs profile` exports what the local stack collected:

```bash terminal theme={null}
reliant forge obs profile --list --since 15m
reliant forge obs profile --service api --type cpu --since 15m -o cpu.pb.gz
go tool pprof -top -nodecount=10 cpu.pb.gz
```

```text Output (trimmed) theme={null}
{"service":"api","rows":162}
wrote cpu.pb.gz: 116 stacks, total 4360000000 nanoseconds (pprof)

Duration: 900s, Total samples = 4360ms ( 0.48%)
      flat  flat%   sum%        cum   cum%
    1230ms 28.21% 28.21%     1230ms 28.21%  crypto/internal/fips140/sha256.blockSHANI
     510ms 11.70% 39.91%     2090ms 47.94%  crypto/internal/fips140/sha256.(*Digest).checkSum
     420ms  9.63% 49.54%      420ms  9.63%  runtime.memmove
     390ms  8.94% 58.49%     2710ms 62.16%  crypto/internal/fips140/sha256.(*Digest).Sum
     130ms  2.98% 66.28%     3070ms 70.41%  crypto/sha256.Sum256
      80ms  1.83% 72.48%     3150ms 72.25%  example.com/guides-sandbox-perf/internal/handlers/catalog.stretchFingerprint
```

`--format speedscope` writes a file for speedscope.app, and `--format folded` writes collapsed stacks for `flamegraph.pl`. `go tool pprof -http=: cpu.pb.gz` opens the flamegraph in a browser. `--type inuse_space` shows the live heap.

**5. Confirm the hot path**

`-top` ranks leaf functions; `-cum` ranks by time spent under each function, which names your code:

```bash terminal theme={null}
go tool pprof -top -cum cpu.pb.gz | grep -E 'catalog\.|sha256.Sum256'
```

```text Output theme={null}
     0     0%     0%      3.15s 72.25%  example.com/guides-sandbox-perf/internal/handlers/catalog.(*Service).FingerprintCatalog
 0.08s  1.83%  1.83%      3.15s 72.25%  example.com/guides-sandbox-perf/internal/handlers/catalog.stretchFingerprint
 0.13s  2.98%  4.82%      3.07s 70.41%  crypto/sha256.Sum256
     0     0% 57.11%      0.67s 15.37%  example.com/guides-sandbox-perf/internal/handlers/catalog.(*Service).GetCatalogReport
```

72% of the service's CPU is under `stretchFingerprint`, almost all of it in `sha256.Sum256`. That is the line to change: fewer rounds, a cached result, or a cheaper hash.

**6. Narrow to one request**

Each sample carries the trace and span it ran for, so you can ask where one slow request spent its CPU:

```bash terminal theme={null}
reliant forge obs profile --trace a9bead2891720624bcee86f0c0a01707 -o trace.pb.gz
go tool pprof -top -nodecount=4 trace.pb.gz
```

```text Output (trimmed) theme={null}
wrote trace.pb.gz: 5 stacks, total 90000000 nanoseconds (pprof)
      flat  flat%   sum%        cum   cum%
      30ms 33.33% 33.33%       60ms 66.67%  crypto/internal/fips140/sha256.(*Digest).checkSum
      30ms 33.33% 66.67%       30ms 33.33%  runtime.memmove
      20ms 22.22% 88.89%       20ms 22.22%  crypto/internal/fips140/sha256.blockSHANI
```

Raw samples are kept 24 hours, and samples are taken at 100 Hz, so a span under about 10ms usually has none.

> [!TIP]
>
> Most traces have no samples, because the CPU window is a few seconds out of every interval. To find traces that do, ask the local profile store: `reliant forge obs query dev --sql "SELECT TraceId, count() AS n FROM default.otel_profiles WHERE SampleType = 'cpu' AND TraceId != '' GROUP BY TraceId ORDER BY n DESC LIMIT 5"`.

**7. Put the cadence back**

Remove `FORGE_PROFILES_CPU_INTERVAL` from the workload's `env` and deploy. Profiles are billed as telemetry, and a faster cadence stores more: [Profile volume](https://docs.reliantlabs.io/hosting/observability/where-it-goes.md#profile-volume).

## What this does not do

* `forge obs profile` exports profiles from the local stack only. Its `--env` flag filters samples by environment name; it does not read a hosted environment, so `--env prod` finds nothing unless your local stack holds samples labelled `prod`. Read hosted profiles in Apps.
* Profiles are sparse by design: about a minute of CPU capture an hour at the default cadence. They find sustained hotspots and can miss a short incident or a single slow request.
* Hosted profiling is off until the environment opts in. Allocation, goroutine, mutex and block profiles are off until a workload turns them on.
* It profiles Go services Forge builds. A service in another language is not profiled by Forge's runtime.

## Next steps

> [card] [Find your slowest endpoints](https://docs.reliantlabs.io/guides/observability/find-slowest-rpcs.md)
>
> Rank RPCs by p95 and read the slowest traces.

> [card] [Debug production with an AI agent](https://docs.reliantlabs.io/guides/observability/debug-production-with-ai-agent.md)
>
> Let an agent read errors, logs and traces and propose the fix.

> [card] [Continuous profiling](https://docs.reliantlabs.io/hosting/observability/profiling.md)
>
> Cadence, sample types, retention and the `profiling` block.

> [card] [Where the telemetry goes](https://docs.reliantlabs.io/hosting/observability/where-it-goes.md)
>
> The local and hosted stores, retention and profile volume.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.