Skip to main content
A trace can tell you an RPC spent its time in your own code. A CPU profile tells you which function. Every Go service Forge builds profiles itself in-process and ships the samples beside its traces, so you get a flamegraph without attaching a profiler, and each sample is tagged with the request it ran for. This guide finds a hashing loop that dominates an API’s CPU, on a local stack and in production. Continuous profiling is the reference: cadence, sample types, retention.

Before you begin

  • A Forge Go API, running locally with reliant forge env up dev or deployed: Deploy a Go API with managed Postgres.
  • Forge v1.6.1 or later in the project (reliant forge --version), and Go installed for go tool pprof.
  • An RPC you suspect. Find your slowest API endpoints gets you there: a slow trace whose time is uninstrumented points at CPU.
The app here has an RPC, FingerprintCatalog, that calls stretchFingerprint, which runs SHA-256 a million times.
1

Turn profiling on

Locally, profiling is on by default: env up dev runs a profiles collector beside the rest of the telemetry. A hosted environment collects profiles only when its main.k says so. The default cadence is a 5-second CPU window every 5 minutes; while you investigate, raise it on one workload through its env (at least 30s):
deploy/kcl/prod/main.k
Deploy a release as usual. The change applies from that deploy.
2

Send traffic

The profiler samples what the process is doing, so the RPC must run during a capture window. A few minutes of steady calls is enough:
terminal
3

Read a hosted profile in Apps

Open Apps, pick the project and environment, and choose Profiles under Observability. It draws a flamegraph of one service’s CPU, memory or goroutines over the range you pick, with the heaviest functions beside it, and opening it from a trace narrows it to that trace.Here, the production api held 12.3s of sampled CPU over 30 minutes in 176 stacks. stretchFingerprint was on the stack for 69.5% of it, and the SHA-256 block function alone (sha256.blockAVX2) was 54.9%.
4

Export a local profile for pprof

forge obs profile exports what the local stack collected:
terminal
Output (trimmed)
--format speedscope writes a file for speedscope.app, and --format folded writes collapsed stacks for flamegraph.pl. go tool pprof -http=: cpu.pb.gz opens the flamegraph in a browser. --type inuse_space shows the live heap.
5

Confirm the hot path

-top ranks leaf functions; -cum ranks by time spent under each function, which names your code:
terminal
Output
72% of the service’s CPU is under stretchFingerprint, almost all of it in sha256.Sum256. That is the line to change: fewer rounds, a cached result, or a cheaper hash.
6

Narrow to one request

Each sample carries the trace and span it ran for, so you can ask where one slow request spent its CPU:
terminal
Output (trimmed)
Raw samples are kept 24 hours, and samples are taken at 100 Hz, so a span under about 10ms usually has none.
Most traces have no samples, because the CPU window is a few seconds out of every interval. To find traces that do, ask the local profile store: reliant forge obs query dev --sql "SELECT TraceId, count() AS n FROM default.otel_profiles WHERE SampleType = 'cpu' AND TraceId != '' GROUP BY TraceId ORDER BY n DESC LIMIT 5".
7

Put the cadence back

Remove FORGE_PROFILES_CPU_INTERVAL from the workload’s env and deploy. Profiles are billed as telemetry, and a faster cadence stores more: Profile volume.

What this does not do

  • forge obs profile exports profiles from the local stack only. Its --env flag filters samples by environment name; it does not read a hosted environment, so --env prod finds nothing unless your local stack holds samples labelled prod. Read hosted profiles in Apps.
  • Profiles are sparse by design: about a minute of CPU capture an hour at the default cadence. They find sustained hotspots and can miss a short incident or a single slow request.
  • Hosted profiling is off until the environment opts in. Allocation, goroutine, mutex and block profiles are off until a workload turns them on.
  • It profiles Go services Forge builds. A service in another language is not profiled by Forge’s runtime.

Next steps

Find your slowest endpoints

Rank RPCs by p95 and read the slowest traces.

Debug production with an AI agent

Let an agent read errors, logs and traces and propose the fix.

Continuous profiling

Cadence, sample types, retention and the profiling block.

Where the telemetry goes

The local and hosted stores, retention and profile volume.