Testing Workflows

Reliant provides two levels of workflow testing: static validation catches structural problems in your workflow definition, and scenario testing verifies runtime behavior with simulated events. Together, they ensure your workflows are both well-formed and behave correctly.

Static Validation

Reliant validates every workflow when it’s loaded. This catches structural problems before your workflow ever runs.

What Gets Validated

Static validation checks for:

Entry point validity: Every node listed in entry must exist in the workflow
Node references: Edges can’t reference non-existent node IDs
Input completeness: Every input must have either a default value or required: true
Loop structure: Loop nodes must have either a workflow reference or an inline definition
Edge consistency: Edge targets must be valid node IDs within the workflow
Reserved names: Input groups can’t use reserved names like default
CEL expression syntax: Template expressions and conditions are checked for valid syntax

Common Validation Errors

Error	Cause	Fix
”input X has no default and is not required”	Input missing both `default` and `required: true`	Add one of the two
”node X not found”	Edge references a non-existent node ID	Check spelling of node IDs
”entry node X not found”	`entry` lists a node that doesn’t exist	Match entry names to node IDs
”loop must have either ‘workflow’ or ‘inline‘“	Loop node missing its body	Add an `inline` definition or `workflow` reference

Static validation runs automatically—there’s no separate command to invoke it. If your workflow has validation errors, Reliant reports them when loading the workflow and it won’t appear in the workflow picker. You can also trigger validation explicitly using the validate_workflow tool in the workflow builder, which is useful when iterating on a workflow definition.

When Static Validation Isn’t Enough

Static validation confirms your workflow is structurally sound, but it can’t tell you whether the workflow actually does what you intend. A workflow can pass all validation checks and still route to the wrong node, loop forever, or produce unexpected outputs. That’s where scenario testing comes in.

Scenario Testing

Scenario testing lets you verify workflow runtime behavior without running real LLM calls or tool executions. You define simulated events—mocked LLM responses and tool results—then assert expected outcomes like which nodes were reached, what outputs were produced, and whether the workflow completed successfully.

How It Works

The scenario simulator executes your workflow’s node graph, but instead of calling real activities, it uses your mocked events to produce node outputs. It follows the same edge routing, condition evaluation, and loop logic as the real executor.

Scenario Structure

A scenario YAML file defines:

name: my_test_scenario
description: "What this tests"

# Optional: override workflow inputs
inputs:
  max_turns: 5

# Optional: start execution at a specific node (for partial testing)
start_at: execute_tools

# Optional: pre-populate node outputs (for partial testing)
state:
  call_llm:
    tool_calls: [{name: bash, input: {command: ls}}]

# Simulated events (required)
events:
  - type: llm_response
    text: "Hello!"

# Expected outcome (optional but recommended)
expect:
  outcome: completed
  reached: [call_llm, save_result]

Event Types

Type	Fields	Description
`llm_response`	`text`, `tool_calls`	Simulates an LLM returning a response
`tool_result`	`tool`, `output`	Simulates a tool returning results
`tool_error`	`tool`, `error`	Simulates a tool failing
`llm_error`	`error`	Simulates an LLM API error
`user_input`	`user_text`	Simulates user sending a message

Event Fields

events:
  # LLM responds with text only (no tool calls)
  - type: llm_response
    text: "I'm done, here's the answer."

  # LLM responds with tool calls
  - type: llm_response
    text: "Let me check that..."
    tool_calls:
      - name: bash
        input: {command: "ls -la"}
      - name: read_file
        input: {path: "/tmp/test.txt"}

  # Tool returns successfully
  - type: tool_result
    tool: bash
    output: {result: "file1.txt\nfile2.txt"}

  # Tool returns an error
  - type: tool_error
    tool: bash
    error: "Permission denied"

  # User sends input
  - type: user_input
    user_text: "Yes, please continue"

Targeting Nodes

By default, events are consumed sequentially — the first triggered node gets the first event, and so on. Use the node field to target a specific node:

events:
  - node: call_llm
    type: llm_response
    text: "Hello!"
  - node: execute_tools
    type: tool_result
    tool: bash
    output: {result: "ok"}

Multiple events for the same node are consumed in order each time that node is triggered.

Inner Nodes (Loops and Sub-Workflows)

For workflows with inline loops or inline sub-workflows using type: workflow with inline:, you can target individual nodes inside them using dot-separated qualified IDs:

events:
  # Target the call_llm node inside agent_loop
  - node: agent_loop.call_llm
    type: llm_response
    tool_calls: [{name: bash, input: {command: ls}}]

  # Target execute_tools inside agent_loop
  - node: agent_loop.execute_tools
    type: tool_result
    tool: bash
    output: {result: "file.txt"}

  # Second iteration: call_llm responds without tools (exits loop)
  - node: agent_loop.call_llm
    type: llm_response
    text: "All done!"

The same dot-notation works for inline workflow nodes:

events:
  # Target the plan node inside a planning sub-workflow
  - node: planning.plan
    output:
      response_text: "Here is the plan."

  # Target the criticize node inside the same sub-workflow
  - node: planning.criticize
    output:
      response_text: "Critique of the plan."

For nested structures, chain the IDs:

events:
  - node: outer_loop.inner_loop.call_llm
    type: llm_response
    text: "Inner step done"

How inline simulation works

For inline loops with an inline: definition, the simulator:

Creates a sub-state-machine for the inline workflow
Executes each inner node individually per iteration
Evaluates conditions on inner nodes and skips those with false conditions
Evaluates edges within the loop to determine node ordering
Calls the mocker with qualified IDs like loop_id.inner_node_id
Evaluates the while condition after each iteration
If the sub-workflow declares outputs:, evaluates them for the while condition

For inline workflow nodes with type: workflow and inline:, the simulator:

Creates a sub-state-machine for the inline workflow
Evaluates conditions on each inner node and tracks false conditions as skipped
Executes inner nodes using qualified IDs like workflow_id.inner_node_id
Evaluates the sub-workflow’s outputs: expressions if declared

For referenced loops or workflows with ref:, the simulator uses black-box mocking with the ref name by default. When the scenario targets qualified inner nodes and the runner can resolve the referenced workflow, it executes that referenced workflow internally instead.

Expectations

The expect section defines assertions:

expect:
  # Expected outcome
  outcome: completed  # or "error"

  # Nodes that must be executed
  reached:
    - call_llm
    - agent_loop.execute_tools  # inner loop nodes supported
    - planning.plan             # inline workflow inner nodes supported
    - save_result

  # Nodes that must NOT be executed
  not_reached:
    - error_handler
    - agent_loop.fallback_step

  # Nodes that must have been skipped (condition evaluated to false)
  skipped:
    - planning.criticize        # skipped because 'critique' not in steps
    - planning.revise
    - impl_loop.lint            # skipped because 'lint' not in steps

  # Assert specific node output values
  node_outputs:
    call_llm:
      tool_calls: []
    agent_loop.call_llm:  # inner loop nodes supported
      message:
        text: "Done!"

  # Assert error message content
  error_contains: "CEL evaluation"

  # Assert which node produced the error
  error_node: call_llm

Partial Testing

Use start_at and state to test from a specific point in the workflow:

name: test_from_execute_tools
start_at: execute_tools

# Pre-populate prior node outputs
state:
  call_llm:
    tool_calls:
      - name: bash
        input: {command: ls}
    message:
      role: assistant
      text: "Let me check"

events:
  - type: tool_result
    tool: bash
    output: {result: "file.txt"}

expect:
  outcome: completed
  reached: [execute_tools, save_result]
  not_reached: [call_llm]  # skipped because we started after it

Loop Testing Patterns

Single Iteration (Exit Immediately)

name: loop_exits_immediately
events:
  - node: agent_loop.call_llm
    type: llm_response
    text: "No tools needed"
expect:
  outcome: completed
  reached: [agent_loop.call_llm, agent_loop.save_result]
  not_reached: [agent_loop.execute_tools]

Multi-Iteration Loop

Provide events for each iteration. The simulator consumes events per-node sequentially:

name: loop_three_iterations
events:
  # Iteration 1: calls tool
  - node: agent_loop.call_llm
    type: llm_response
    tool_calls: [{name: bash, input: {command: "echo 1"}}]
  - node: agent_loop.execute_tools
    type: tool_result
    tool: bash
    output: {result: "1"}

  # Iteration 2: calls another tool
  - node: agent_loop.call_llm
    type: llm_response
    tool_calls: [{name: bash, input: {command: "echo 2"}}]
  - node: agent_loop.execute_tools
    type: tool_result
    tool: bash
    output: {result: "2"}

  # Iteration 3: exits loop
  - node: agent_loop.call_llm
    type: llm_response
    text: "All done!"

expect:
  outcome: completed
  reached: [agent_loop, agent_loop.call_llm, agent_loop.execute_tools]

Testing Loop Output Routing

When a loop’s inline workflow declares outputs:, those evaluated outputs are available for downstream edge conditions:

# In the workflow YAML:
# nodes:
#   - id: agent_loop
#     type: loop
#     while: "size(outputs.tool_calls) > 0"
#     inline:
#       outputs:
#         status: "{{nodes.call_llm.message.text}}"
#       ...
# edges:
#   - from: agent_loop
#     cases:
#       - to: success
#         condition: "nodes.agent_loop.status == 'done'"
#     default: fallback

name: loop_routes_to_success
events:
  - node: agent_loop.call_llm
    type: llm_response
    text: "done"
expect:
  reached: [agent_loop, success]
  not_reached: [fallback]

Using Scenario Tools

In the workflow builder, use these tools to manage scenarios:

Tool	Usage
`list_scenarios`	See all scenarios and their status
`view_scenario`	Examine a scenario’s full YAML
`create_scenario`	Create a new scenario and run it automatically
`edit_scenario`	Make targeted edits to a scenario
`run_scenario`	Re-run a scenario after workflow changes

Scenarios are stored alongside the workflow and run instantly without network calls or LLM API usage.

Getting Started

Workflows

Workspace & Git

Agent Features

Settings

Architecture

Reference

Help

About

Testing Workflows

Static Validation

What Gets Validated

Common Validation Errors

When Static Validation Isn’t Enough

Scenario Testing

How It Works

Scenario Structure

Event Types

Event Fields

Targeting Nodes

Inner Nodes (Loops and Sub-Workflows)

How inline simulation works

Expectations

Partial Testing

Loop Testing Patterns

Single Iteration (Exit Immediately)

Multi-Iteration Loop

Testing Loop Output Routing

Using Scenario Tools

​Static Validation

​What Gets Validated

​Common Validation Errors

​When Static Validation Isn’t Enough

​Scenario Testing

​How It Works

​Scenario Structure

​Event Types

​Event Fields

​Targeting Nodes

​Inner Nodes (Loops and Sub-Workflows)

​How inline simulation works

​Expectations

​Partial Testing

​Loop Testing Patterns

​Single Iteration (Exit Immediately)

​Multi-Iteration Loop

​Testing Loop Output Routing

​Using Scenario Tools

Static Validation

What Gets Validated

Common Validation Errors

When Static Validation Isn’t Enough

Scenario Testing

How It Works

Scenario Structure

Event Types

Event Fields

Targeting Nodes

Inner Nodes (Loops and Sub-Workflows)

How inline simulation works

Expectations

Partial Testing

Loop Testing Patterns

Single Iteration (Exit Immediately)

Multi-Iteration Loop

Testing Loop Output Routing

Using Scenario Tools