on_failure run as they would in production (How workflows run).
Before you begin
- The
reliantCLI, v1.21 or later (reliant version). Scenarios need no account and no network. - The basics of a workflow file, from Write your first multi-agent workflow.
1
Start from a workflow
This workflow runs the test suite every night. When the tests fail, an agent reads the output and says why.
.reliant/workflows/nightly-tests.yaml
reliant workflow lint passes it with one warning: [scenario-none] the workflow has no scenarios, so none of its triggers or endings is tested.2
Cover the success path
Scenarios live in A
.reliant/workflows/<name>/scenarios/, one per file, where <name> is the workflow’s name:. Each event mocks one node by its path. trigger: starts the run the way the nightly schedule would:.reliant/workflows/nightly-tests/scenarios/green_overnight.yaml
run node’s mock is its whole output: exit_code, stdout, stderr. A ref: node such as diagnose is mocked as a unit by an event on the node itself.3
Cover the failure path
With only the first scenario, lint names the ending nothing tests yet:
.reliant/workflows/nightly-tests/scenarios/red_tests_get_a_diagnosis.yaml
outcome: failed means the run ended at a node that declares outcome: failure. on_failure checks that the handler ran and which node it saw.4
Run the scenarios
terminal
reliant workflow lint now reports 0 lint error(s), 0 warning(s): every trigger and every ending is covered.5
Write the next expectation first
When the test tool is not installed, the command exits The second mismatch is the one you wrote. The first shows what an unmocked node does:
127, and the workflow pays an agent to read command not found. Write the behavior you want before you change the workflow:.reliant/workflows/nightly-tests/scenarios/missing_tool_skips_the_agent.yaml
terminal
diagnose has no event in this scenario, so it returned an empty output, and red could not read response_text. Without --verbose, only the first mismatch prints, followed by ... and 1 more.6
Fix the workflow and run again
Add a node for the missing tool and a case that routes exit Then change the scenario’s expectation to
127 to it:.reliant/workflows/nightly-tests.yaml
completed: [test, misconfigured], so it also covers the new ending.7
Verify the whole suite
terminal
0 when every scenario passes and 1 when any fails, so it can gate a commit or a CI job. --json prints one result per scenario, with its mismatches. To narrow a run, pass a workflow file (reliant workflow scenario run .reliant/workflows/nightly-tests.yaml) or --filter <text>, which matches scenario names. --fail-fast stops at the first failure. reliant workflow scenario list shows what would run.Learn from the built-in workflows
In reliant v1.21.3 the built-in workflows ship 217 scenarios, with mocked approvals, GitHub actions, loops and triggers.--include-builtins adds them to scenario list and scenario run. The scenario format and the scenario schema document every field.
What this does not do
- It never runs anything real. A passing scenario proves the routing, the CEL and the messages. It does not prove the agent can find the bug or that the command exists on your machine. Run the workflow once for real before you put it on a schedule.
- It does not check what only a live run enforces. For example, an
ask_questionnode without"type":"ask_user"in its metadata passesvalidate,lintand every scenario, but the live question shows the person no options. - It trusts your mocks. A node you do not mock returns an empty output. An output key you misspell is ignored: a
runmock of{exitcode: 1}is read as exit0and takes the green path, andscenario runprints no warning. Copy output field names from the nodes reference.
Troubleshooting
validation failed: event[1]: unknown node "diagnos"
validation failed: event[1]: unknown node "diagnos"
The event names a node the workflow does not have. The message lists the valid paths, for example
(valid nodes: diagnose, green, misconfigured, on_failure.report, red, test). Nodes inside a loop or an inline workflow are <parent>.<id>. The scenario counts under Errors, not Failed.validation failed: expect.completed: unknown node
validation failed: expect.completed: unknown node
An expectation names a node that does not exist yet. Add the node to the workflow first, or leave it out of the expectation until you do.
CEL evaluation error: no such key: response_text
CEL evaluation error: no such key: response_text
A later node reads an output that a node you did not mock never produced. Add an event for that node, such as
diagnose with output: {response_text: "..."}.Next steps
Catch workflows that break unattended
Lint rules for timeouts, unbounded loops and concurrency slots.
Loop an agent until tests pass
Scenarios for a loop that fails twice and then passes.
Require human approval
Mock an approval as approved, denied or timed out.
How workflows run
The engine scenarios run on, and what is mocked.