> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reliantlabs.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Continuous profiling for Go services

> How forge samples CPU and heap profiles from your Go services, and how to read a flamegraph from the local stack or in the Reliant app.

A profile answers a question that logs, traces and metrics only point at: which functions are using the CPU, or holding the memory. Every Go service forge builds profiles itself, in-process, and pushes the result over OpenTelemetry beside its traces. You get a flamegraph without attaching a profiler.

## What is collected, and how often

The profiler is part of forge's runtime library, started when a service sets up its telemetry. It runs on its own goroutine, never blocks a request, and backs off if the collector is unreachable.

| Profile | Sample types | Default | Cadence |
| - | - | - | - |
| CPU | `cpu` (nanoseconds) | on | a 5-second window every 5 minutes, jittered |
| Live heap | `inuse_space`, `inuse_objects` | on | every 2 hours, plus one snapshot on the first cycle of a new process |
| Heap allocations | `alloc_space`, `alloc_objects` | off | with the heap snapshot |
| Goroutines | `goroutine` | off | every cycle |
| Mutex and block | `contentions`, `delay` | off | every cycle |

These are sparse on purpose, because every stored sample repeats its whole stack and profiles are the costliest signal per useful byte. A long-lived process has roughly a minute of CPU capture an hour. That is enough to find a sustained hotspot or a steady heap leak, and it can miss a short incident or a single slow request entirely. To look harder for a while, raise the cadence on one workload through its `env` (`FORGE_PROFILES_CPU_INTERVAL`, at least `30s`) and put it back afterwards; [Where the telemetry goes](https://docs.reliantlabs.io/hosting/observability/where-it-goes.md#profile-volume) lists the variables.

The allocation counters are off because Go reports them cumulatively since process start. Every snapshot repeats every allocation site the process ever had, so the cost grows with uptime. Turn them on for one workload when you are hunting allocations, and turn them off when you are done. Agents can load `observability/profiles` for the workload `profiling` block that does it.

## Is it on?

It depends on where the environment's telemetry goes. The environment's `observability` block decides, like every other signal, and `signals.profiles` is the switch.

| Environment | `signals.profiles` unset | To change it |
| - | - | - |
| Local (`forge env up dev`, the default local stack) | **On** | `profiles = False` turns it off |
| Reliant hosting (`forge.OnHosted`) | **Off** | `profiles = True` turns it on |
| A cluster you run, or an environment with a `destination` | **Off** | `profiles = True` sends them; the destination must accept OTLP profiles |

So the five-minute CPU and two-hour heap cadence is what a process does once profiling is on, and a hosted environment does not collect profiles until you opt in. To turn it on for an environment, declare the signal in its `main.k` and deploy:

```python theme={null}
observability = forge.Observability {
    signals = forge.ObservabilitySignals {profiles = True}
}
```

The change takes effect with that environment's next deploy. To switch profiling off for a single process without touching the environment, set `OTEL_PROFILES_EXPORTER=none` or `FORGE_PROFILES=none` in its `env`.

> [!NOTE]
>
> Profiles are billed as telemetry once they are collected, and a hosted environment that is over its observability cap stops storing them first. Retention is 24 hours for raw samples and 30 days for the per-minute aggregates behind range flamegraphs. [Observability billing](https://docs.reliantlabs.io/hosting/observability-billing.md) has the included amounts and the cap.

## Reading a profile

Local and hosted profiles are read differently. Locally, the CLI exports them from the stack on your machine. On hosting, you read them in the Reliant app.

### Locally, with `forge obs profile`

`forge env up` keeps profiles in the same local store as your other telemetry, but the HyperDX UI cannot draw flamegraphs. `forge obs profile` exports them in a form other tools can open. It needs the local stack running.

```bash theme={null}
forge obs profile --list                                       # which services sent profiles
forge obs profile --service api --since 30m -o api.pb.gz
go tool pprof -http=: api.pb.gz                                # flamegraph, top, source view
forge obs profile --service api --format speedscope -o api.speedscope.json
```

`--format` is `pprof` (the default), `speedscope` or `folded`. Open the speedscope file at speedscope.app, or pipe the folded stacks to `flamegraph.pl` for an SVG. The default `--type` is `cpu`; `inuse_space` and `inuse_objects` show the live heap, and the allocation, goroutine, mutex and block types appear only for a workload that turned them on. `--since` defaults to one hour, and `--from` and `--to` take RFC 3339 times. `--sql` prints the query instead of running it, so you can run it yourself.

CPU time is summed over the range. The live heap and goroutines are different: they are snapshots, so the export uses each instance's newest snapshot in the range, and looks back up to 24 hours when the range holds none. A one-hour window therefore still shows the heap even though snapshots are two hours apart.

To see where one slow request spent its CPU, narrow to a trace, which uses the raw samples from the last 24 hours:

```bash theme={null}
forge obs profile --trace 0af7651916cd43dd8448eb211c80319c
```

CPU samples are taken at 100 Hz, so a span under about 10 ms usually has no samples. Trace links are for slow spans.

### On hosting, in the Reliant app

Open **Apps**, pick your project and environment, and choose **Profiles** in the Observability section. It draws a flamegraph of a service's CPU, memory or goroutines over the time range you pick, with a table of the heaviest functions beside it. Opening it from a trace narrows it to that trace's samples. [Where the telemetry goes](https://docs.reliantlabs.io/hosting/observability/where-it-goes.md#reading-it-in-apps) describes the rest of the Apps navigation.

The page is a view over the control plane's `GetFlamegraph` call, which returns a service's aggregated profile as speedscope JSON. Reading it requires sign-in. `forge obs profile` reads only the local stack: its `--env` flag keeps the samples whose environment name matches, and does not export a hosted environment's profiles.

## Related topics

* [Where the telemetry goes](https://docs.reliantlabs.io/hosting/observability/where-it-goes.md): the local store, the hosted store, retention, profile volume and opting out.
* [Querying with forge obs](https://docs.reliantlabs.io/hosting/observability/forge-obs.md): the other `forge obs` verbs, including `check`, `status` and `trace`.
* [Hosted dashboards, alerts and issues](https://docs.reliantlabs.io/hosting/observability/dashboards-and-alerts.md): the rest of what Apps shows for an environment.
* [Observability billing](https://docs.reliantlabs.io/hosting/observability-billing.md): included amounts, the cap and what stops when it is reached.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.