> ## Documentation Index
> Fetch the complete documentation index at: https://docs.reliantlabs.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Loop an AI agent until your tests pass

> Have an AI agent fix code, run your test command and try again until the tests pass, with a cap on attempts so a run that cannot succeed stops on its own.

By the end of this guide you have a workflow in which an agent fixes your code, your own test command runs, and the agent tries again until the command exits `0`. A cap on attempts ends a run that cannot succeed, so it does not keep spending.

The agent does not decide when it is done. A `run` node executes your test command, and the loop and the final edge route on its `exit_code`.

## Before you begin

* Read [Write your first multi-agent workflow](https://docs.reliantlabs.io/guides/workflows/first-multi-agent-workflow.md) for the basics of a workflow file, `validate`, `lint` and scenarios. This guide does not repeat them.
* A git repository with a test command that fails today. This guide uses a Python `slug.py` whose three unittest tests fail.
* For running it live: a signed-in CLI ([Sign in and tokens](https://docs.reliantlabs.io/cli/sign-in.md)) and a connected machine that holds the repository ([Run a machine from the terminal](https://docs.reliantlabs.io/cli/run-a-machine.md)).

**1. Create the workflow file**

```yaml .reliant/workflows/fix-until-green.yaml theme={null}
name: fix-until-green
title: Fix until the tests pass
description: An agent changes the code, your test command runs, and the loop repeats until the tests pass or the attempts run out.

inputs:
  test_command:
    type: string
    default: python3 -m unittest
    description: The command that must exit 0
  max_attempts:
    type: integer
    default: 3
    min: 1
    max: 5
    description: Most fix-and-test rounds before the run gives up

entry: [attempts]

nodes:
  - id: attempts
    type: loop
    # Do-while: the body runs once, then repeats while the tests fail
    # and attempts remain. iter.iteration counts finished rounds here.
    while: outputs.exit_code != 0 && iter.iteration < inputs.max_attempts
    inline:
      entry: [fix]
      outputs:
        exit_code: "{{nodes.test.exit_code}}"
        test_output: "{{nodes.test.stdout + nodes.test.stderr}}"
      nodes:
        - id: fix
          type: workflow
          ref: builtin://agent
          args:
            mode: auto
          thread:
            mode: fork
            memo: true   # the agent keeps its own context from round to round
            inject:
              role: user
              content: |
                Attempt {{iter.iteration + 1}} of {{inputs.max_attempts}}. Make `{{inputs.test_command}}` pass.
                Fix the code, not the tests.
                {{iter.iteration > 0 ? 'The tests still fail. Their output:\n' + outputs.test_output : ''}}

        - id: test
          type: run
          command: "{{inputs.test_command}}"
          timeout: 5m   # bound a hung test run; the default is 30 days
      edges:
        - from: fix
          default: test

  - id: passed
    type: save_message
    outcome: success
    args:
      role: assistant
      content: "Tests pass after {{nodes.attempts._iterations}} attempt(s)."

  - id: gave_up
    type: save_message
    outcome: failure
    args:
      role: assistant
      display_style: warning
      content: |
        Tests still fail after {{nodes.attempts._iterations}} attempt(s). Last output:
        {{nodes.attempts.test_output}}

edges:
  - from: attempts
    cases:
      - to: passed
        condition: nodes.attempts.exit_code == 0
        label: green
    default: gave_up
```

* **`test`** is a [`run` node](https://docs.reliantlabs.io/reference/nodes#run-command). A non-zero exit does not fail it: `exit_code`, `stdout` and `stderr` are outputs you route on.
* **`attempts`** is a [`loop` node](https://docs.reliantlabs.io/reference/nodes#loop). The body declares `exit_code` and `test_output` as outputs, so `while` can read them after each round. Loops are do-while: the first round always runs, and `iter.iteration` is the number of finished rounds when `while` is checked ([CEL loop conditions](https://docs.reliantlabs.io/reference/cel-expressions.md#loop-while-conditions)).
* **`fix`** gets the previous round's test output in its prompt from round 2 on. `memo: true` keeps the agent's own conversation across rounds, so it knows what it already tried.
* **The final edge** is a CEL condition on the loop's last `exit_code`. It sends a green run to `passed` and anything else to `gave_up`, and their `outcome:` sets the run's verdict.

**2. Check the cap with lint**

`while` has two halves. `outputs.exit_code != 0` ends the loop when the tests pass, and `iter.iteration < inputs.max_attempts` ends it when they never do. Lint warns when the second half is missing. With it removed:

```bash terminal theme={null}
reliant workflow lint
```

```text theme={null}
  ✓ fix-until-green.yaml (fix-until-green)
      Lint warning: [loop-unbounded] attempts: the loop's while never reads iter.iteration and it has no items, so nothing bounds it (add `&& iter.iteration < N` to while) https://docs.reliantlabs.io/workflows/lint#loop-unbounded

1 workflow(s): 0 lint error(s), 1 warning(s)
```

Lint also flags a `run` node with no `timeout:` ([`run-no-timeout`](https://docs.reliantlabs.io/workflows/lint#run-no-timeout)). And it flags a `timeout:` on the agent node, which bounds nothing ([`timeout-ignored`](https://docs.reliantlabs.io/workflows/lint#timeout-ignored)). With the file as written, `reliant workflow validate` reports `✅ All workflows valid` and lint reports `0 lint error(s), 0 warning(s)`.

**3. Test every ending with scenarios**

A [scenario](https://docs.reliantlabs.io/workflows/testing.md) mocks the agent and the test command by node path, so you check the loop without spending tokens. This one fails twice and passes on round 3:

```yaml .reliant/workflows/fix-until-green/scenarios/green_on_third_try.yaml theme={null}
name: green_on_third_try
description: The tests fail twice, and the third fix makes them pass

events:
  - node: attempts.fix
    output: {response_text: "Lowercased the title."}
  - node: attempts.test
    output: {exit_code: 1, stderr: "FAIL: test_drops_punctuation\n"}
  - node: attempts.fix
    output: {response_text: "Dropped punctuation."}
  - node: attempts.test
    output: {exit_code: 1, stderr: "FAIL: test_collapses_spaces\n"}
  - node: attempts.fix
    output: {response_text: "Split on whitespace and joined with single hyphens."}
  - node: attempts.test
    output: {exit_code: 0, stderr: "OK\n"}

expect:
  outcome: completed
  completed: [attempts, passed]
  not_reached: [gave_up]
  node_outputs:
    attempts: {exit_code: 0, _iterations: 3}
  messages:
    passed:
      content_contains: ["Tests pass after 3 attempt(s)."]
```

Add `green_first_try.yaml`, with one fix and one `exit_code: 0`, expecting `_iterations: 1`. Then add `gives_up_at_max.yaml`, with `inputs: {max_attempts: 2}`, two failing rounds, and `expect: {outcome: failed, completed: [attempts, gave_up]}`. That one proves the cap stops the run:

```bash terminal theme={null}
reliant workflow scenario run
```

```text theme={null}
Running 3 scenario(s) across 1 workflow(s)

  ✓ fix-until-green/gives_up_at_max (491ms)
  ✓ fix-until-green/green_first_try (8ms)
  ✓ fix-until-green/green_on_third_try (13ms)
...
✅ All scenarios passed
```

**4. Run it on your repository**

Commit first, so `git diff` shows only what the agent changed:

```bash terminal theme={null}
reliant workflow run fix-until-green --project-path . -i max_attempts=3 \
  -m "The slugify tests in test_slug.py fail. Fix slug.py." --follow --timeout 20m
```

```text theme={null}
Workflow "fix-until-green" triggered
  Chat/Execution ID: e9b75db6-4942-49c9-8b1d-4b20f8993733
...
{"event":"node_completed",...,"node_id":"test",...,"new_state":"completed",...,"exit_code":0}
{"event":"node_completed",...,"node_id":"passed",...}
{"event":"workflow_completed",...,"new_state":"completed",...,"outcome":"success"}
```

`--follow` exits `0` when the run succeeds and `1` when it fails, so a script can act on the verdict.

**5. Verify the result**

```bash terminal theme={null}
reliant workflow status e9b75db6-4942-49c9-8b1d-4b20f8993733
git diff --stat -- slug.py && python3 -m unittest
```

```text theme={null}
  Status:    COMPLETED
  Outcome:   ✓ SUCCESS
...
    test                     ExecuteRunStep             1         ok        752
    passed                   SaveMessage                1         ok        41
 slug.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)
Ran 3 tests in 0.000s

OK
```

`test` ran once, so the first fix passed. To watch the cap work, give the loop a command that can never pass. `-i 'test_command="exit 1"' -i max_attempts=2` ended after two rounds at `gave_up`, with `Outcome: ✗ FAILURE` and `test` listed as `2 runs, 2 failed`.

## What bounds the cost

* **The attempt cap.** At most `max_attempts` agent rounds run. The input's `max: 5` refuses a higher value before the run starts: `input 'max_attempts' must be <= 5`.
* **The exit condition.** The loop stops at the first green round, so a fix that works on round 1 costs one round.
* **The test timeout.** A hung test command stops after `timeout: 5m`, instead of running for the 30-day default.

A single round is not capped. `builtin://agent` works until it finishes its turn, and a `timeout:` on that node bounds nothing.

## What this does not do

* **It does not stop the agent from editing tests.** The prompt asks it not to, but the gate is only your command's exit code. Read the diff before you keep it.
* **It does not commit, push or open a pull request.** Add a [`git_publish`](https://docs.reliantlabs.io/reference/nodes#publish-branch) node for that.
* **It does not review the change.** Passing tests are the only check. To add a reviewer agent, lint and build gates, use [Get It Right](https://docs.reliantlabs.io/guides/get-it-right.md).

## Troubleshooting

> [accordion] no machine has synced this project's workflows yet
>
> The first `reliant workflow run --project-path` on a new folder creates the project before its machine has synced `.reliant/workflows/`. Wait about 20 seconds and run the same command again.

> [accordion] input 'test\_command' expects string, got bool
>
> `-i` reads `true`, `false`, `null` and numbers as JSON values. Quote the value to send it as text: `-i 'test_command="false"'`.

> [accordion] \[loop-unbounded] attempts: the loop's while never reads iter.iteration
>
> Add `&& iter.iteration < inputs.max_attempts` to `while`. Without it, only a passing test ends the loop.

## Next steps

> [card] [Write your first multi-agent workflow](https://docs.reliantlabs.io/guides/workflows/first-multi-agent-workflow.md)
>
> A planner, an implementer and a reviewer that sends work back.

> [card] [Get It Right](https://docs.reliantlabs.io/guides/get-it-right.md)
>
> The built-in loop with lint, test and build gates and a reviewer.

> [card] [How workflows run](https://docs.reliantlabs.io/deep-dives/how-workflows-run.md)
>
> How loops, edges and outcomes are evaluated at run time.

> [card] [CEL expressions](https://docs.reliantlabs.io/reference/cel-expressions.md)
>
> Every namespace a `while` or edge condition can read.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.