Graph Engine¶

Most agent frameworks are built around a loop: call the model, run the tools it asks for, feed the results back, repeat until done. ModexAgent's built-in ReAct agent replaces that loop with a graph-driven execution engine.

That engine lives in modex_graph — a standalone Python package (per ADR-0033) whose only runtime dependency is Pydantic. It is a sibling of modex_agent, not a submodule: modex_agent depends on modex_graph, the reverse is forbidden and enforced by an architecture guard test. Any project can adopt the graph engine without pulling in the agent framework.

The core abstraction: you declare a Graph[S] where S is a GraphState (a Pydantic BaseModel subclass). You register Node[S] instances and wire topology edges between them — edges declare which nodes can connect, nothing more. compile() validates the topology and produces an immutable CompiledGraph[S]; a GraphEngine then drives execution by delegating to a scheduler. Routing is decided at runtime by the nodes themselves: a node calls deliver(content, next_node, ctx) to hand work to a downstream node. START and END are real registered nodes — the entry node delivers the graph input downstream, and END aggregates everything delivered to it. That one design buys suspension, resumption, typed shared state, controlled exits, graph-internal continuation, and — when you opt in — parallel fan-out with fan-in joins.

Which pools run the graph?

The graph engine is the runtime of react pools — the default. Other pool shapes exist: external coding agent pools (OpenCode, per ADR-0022) run their own CLI harness and do not use the graph at all. When this page says "the agent" or "the runtime", it means a ReAct agent on a graph.

The ReAct runtime as six nodes¶

The built-in ReAct agent is a small graph with six nodes, assembled by build_react_graph() in modex_agent/agents/react/graph.py. Edges declare topology only — two engine edges wire __start__ to the react entry node and the react END to __end__ — and every edge below is taken at runtime via a node's deliver() call:

flowchart TD
    GSTART(["__start__"])
    START["START<br/><small>route: fresh or resume</small>"]
    BEFORE["BEFORE<br/><small>new turn attempt</small>"]
    LLM["LLM<br/><small>call the model</small>"]
    TOOL["TOOL<br/><small>execute tool calls</small>"]
    AFTER["AFTER<br/><small>result + continuation gate</small>"]
    ENDN["END<br/><small>emit, mark completed</small>"]
    GEND(["__end__"])

    GSTART --> START
    START -->|"fresh turn"| BEFORE
    START -.->|"resume_target<br/><small>(approval)</small>"| TOOL
    BEFORE --> LLM
    LLM -->|"tool calls"| TOOL
    TOOL -->|"tools done"| LLM
    LLM -->|"no tools / budget / error"| AFTER
    TOOL -->|"cancel / error"| AFTER
    AFTER -->|"CONTINUATION_REQUEST"| BEFORE
    AFTER -->|"terminal"| ENDN
    ENDN --> GEND

The graph contains two nested loops. The inner loop (LLM ↔ TOOL) is one ReAct reasoning-and-acting iteration. The outer loop (BEFORE → … → AFTER → BEFORE) is the turn attempt: every terminal path — normal completion, cancellation, error, exhausted iteration budget — flows through AFTER, so result construction and the continuation gate run on every exit, not just the happy one.

Node	Responsibility
START	Entry routing. Fresh turn: initialize phase, deliver to BEFORE. Resumed turn: read `state.resume_target`, clear it, deliver straight to that node (TOOL).
BEFORE	Opens a turn attempt: increments `state.turn_attempt`, resets the iteration budget (`state.iteration = 0`), dispatches `BEFORE_TURN`, routes to LLM.
LLM	Calls the model with the current context, handles streaming, dispatches iteration hooks and interceptors. Delivers to TOOL on tool calls, to AFTER otherwise.
TOOL	Executes the requested tool calls. Suspends for approval before risky calls. Delivers back to LLM, or to AFTER on cancellation/error.
AFTER	Constructs the preliminary `AgentResult` from turn state (cancel / error / normal / max-iteration branches) and writes `state.result`. Dispatches `AFTER_TURN`, then runs the continuation gate.
END	Reads `state.result` (constructed by AFTER), emits the matching completion event, marks the turn completed, delivers to `__end__`.

Graph-internal continuation. When an after-turn hook decides the turn is not done — e.g. the agent finished a graph-node turn without calling the deliver tool — it sets a CONTINUATION_REQUEST flag in turn state. The gate in AFTER consumes the flag (one-shot boolean: presence means "continue", the value is never read — and it is popped regardless of the decision) and routes back to BEFORE, which increments turn_attempt and resets the iteration budget for the new attempt. The turn budget is MAX_TURNS — 1 by default in the framework, 3 for graph node turns — and continuation is capped by turn_attempt < MAX_TURNS. A second flag, CONTINUATION_RENEW_MAX_TURNS, is the watchdog: when set (today only by TodoContinuationHook), the gate extends the budget by exactly 1 — once — so a still-progressing agent is not cut off at the ceiling.

The continuation request carries its explanation as a reminder appended to history after the assistant message — ordering [assistant_message, reminder] — so the next attempt's LLM call sees the reminder as the latest message. DeliverRetryHook, for example, fires at AFTER_TURN (the assistant message is already in history by then), appends a <system-reminder> telling the agent it must call deliver, and sets the flag only when the turn budget allows another attempt.

Hook dispatch: four levels¶

Business AOP rides the graph through GraphRuntime — an ABC with default no-op implementations, subclassed by ReactGraphRuntime to bridge hooks, interceptors, governance, and approval into the engine. Only before_node / after_node are engine-auto-invoked; everything else is dispatched explicitly at the exact code points that define it:

Level	Hook points	Dispatched
Graph	`BEFORE_GRAPH` / `AFTER_GRAPH` / `FINALLY_GRAPH`	the agent's `run()` around the turn — once per logical turn; `FINALLY_GRAPH` fires in the `finally` block regardless of outcome
Node	`START_NODE_TURN` / `END_NODE_TURN`	inside `StartNode` / `EndNode` via `ctx.runtime.dispatch_hook()`
Turn attempt	`BEFORE_TURN` / `AFTER_TURN`	inside `BeforeTurnNode` / `AfterTurnNode` via `ctx.runtime.dispatch_hook()`
Iteration	`BEFORE_ITERATION` / `AFTER_ITERATION` / `AFTER_LLM_RESPONSE` / tool hooks	inside `LLMNode` / `ToolNode`

Two consequences worth noting:

Turn-attempt hooks live inside the graph nodes — an earlier design dispatched them from a closure outside the engine; the nodes are now the single source of timing truth. They fire once per turn attempt, so a continuation (a new attempt) re-fires them — by design: KnowledgeHook documents "each turn attempt, including continuations".
Graph-level hooks fire once per run() call. A graph-internal continuation happens inside one engine run, so it does not re-fire BEFORE_GRAPH / AFTER_GRAPH / FINALLY_GRAPH.

Approval: suspend without re-execution¶

Any node can suspend execution mid-turn by calling ctx.interrupt(value), which raises a GraphInterrupt — a member of the GraphBubbleUp cooperative-control exception family. The engine never swallows these; they propagate to the caller verbatim. The interrupt carries a payload (e.g. an ApprovalTransaction awaiting a human decision). Before raising, the suspending node sets state.resume_target, so the entry node knows where to re-enter; already-applied state updates and side effects persist across the interrupt boundary.

On resume the graph re-enters at the entry node — not by re-running the interrupted node body. START reads resume_target, clears it, and delivers straight to TOOL, bypassing BEFORE entirely: the resumed execution is the same turn attempt — turn_attempt is not incremented and the iteration budget is not reset — and the model is not called again just to repeat the tool call it already asked for. This is suspend-without-re-execution: graph topology carries the resume logic.

sequenceDiagram
    autonumber
    actor U as User
    participant G as GraphEngine
    participant T as TOOL node
    participant A as Approval pipeline

    U->>G: message (START → BEFORE → LLM → TOOL)
    T->>T: sensitive tool call. state.resume_target = TOOL
    T-->>A: ctx.interrupt(tx) → GraphInterrupt
    A->>U: render approval card (WebUI / IM)
    Note over T,U: turn suspended — state persisted
    U->>A: /approve (or /deny)
    A->>G: resume — re-enter at START
    G->>T: START reads resume_target → delivers to TOOL
    T->>G: tools execute → LLM → AFTER → END

Never swallow GraphInterrupt

GraphInterrupt is control flow, not an error. It must never be caught and swallowed, or a paused approval would silently vanish.

Routing: deliver-only¶

Node.execute(ctx, integrated_input) returns void — it never returns a route. A node routes by calling deliver() during execution:

async def execute(self, ctx, integrated_input) -> None:
    result = await do_work(ctx)
    self.deliver(result, "next_node", ctx)   # explicit target
    self.deliver(result, None, ctx)          # resolve via topology

deliver() persists the content as a STAGED row in the target node's deliver store; when the source invocation completes, the row is promoted to PENDING and the target is fired as a scheduling wakeup. Content flows through the store — never through the dispatch signal. This replaces an earlier transition/Command(goto=...) routing model with one uniform mechanism, and it buys:

Conditional routing is just an if — the node picks its target in ordinary Python.
Fan-out is free — call deliver() several times with different targets and the parallel scheduler dispatches each group concurrently.
Crash-safe data flow — every delivered payload is a persisted row, so recovery replays inputs from the store instead of replaying nodes.

Node.run wraps this in a fixed lifecycle: begin an invocation, integrate upstream deliveries into integrated_input (collect the consumable rows, mark them consumed), call execute(), complete the invocation, promote the staged outputs, dispatch the wakeups, then promote the consumed inputs. Delivering with next_node=None resolves via topology: the single downstream edge, or END when there is none; ambiguity is an error — a node with several downstream edges must name its target, because unconditional fan-out would loop forever. A node that produces no delivers is a dead end: the graph terminates with reached_end = False and the run is judged FAILED.

Node identity and typed graph I/O¶

Every node carries a machine-facing node_id alongside its human-facing name: node_ plus a 26-character sortable body (12 hex timestamp digits + 14 random alphanumerics). The node_id is the persistence key — invocation records, deliver rows, and scratch regions are keyed by it — while names stay graph-local labels used for topology validation and readable diagnostics. Names can collide across graphs; IDs cannot.

START and END are real registered nodes with their own IDs, not sentinels: Graph registers a StartNode (delivers the graph's user input downstream) and an EndNode on construction, and the declarative compiler creates defaults when a spec omits them. There is no END short-circuit — deliveries to __end__ are persisted and consumed like any other input, and END runs through the same invocation lifecycle as every other node. Its default implementation aggregates every delivered payload into state.result: list[GraphPayload] and is forced to the ON_ALL_PREDS trigger so parallel delivers concatenate instead of overwriting.

GraphPayload is the frozen, strict content type for static-graph I/O: the graph's user input (ctx.user_input), agent-node delivers, and the list[GraphPayload] result. An instance's GraphMetadata persists an immutable node_id_map (name → ID), frozen at creation and copied across re-invocation versions — recovery re-binds the persisted identities instead of minting fresh ones, so a recovered graph addresses the same rows the crashed one wrote.

Two schedulers: linear and parallel¶

Graph.compile(scheduler=...) picks the execution model; GraphEngine is a thin delegator that forwards run_async / run to it (ADR-0034):

LinearScheduler (default) — sequential execution: one node at a time; the first recorded dispatch target becomes the next node. ReAct and all simple graphs use this and pay no tax for the parallel machinery.
ParallelScheduler (opt-in) — a parallel scheduling engine for fan-out/fan-in workloads (MapReduce-style graphs, multi-arm pipelines).

The parallel scheduler uses continuous, event-driven scheduling rather than BSP-style supersteps: a node instance starts the moment its dependencies are satisfied — no batch barrier, so a fast branch never waits for a slow one. Three concepts drive it:

Node instances. Every execution of a node is an independent instance ({node_name}#{seq}) with a lifecycle: DORMANT → PENDING → READY → RUNNING → COMPLETED. Loops simply produce new instances (body#0, body#1, ...) instead of resetting state.
Trigger modes. ON_ALL_PREDS (default, stable) waits until every activated predecessor has dispatched to it — with a reachability check so a join never fires early while another branch could still reach it. ON_RECEIVE is deprecated/experimental: Graph.compile() warns, and the declarative GraphSpec API rejects it outright.
Shared state. Parallel instances get their own context shells but share the same ctx.state; asyncio's single-thread model serializes the synchronous lifecycle segments, so there is no merge step and no conflict-detection machinery. The graph ends when no instance is ready or running; __end__ implicitly waits for all branches.

Both schedulers start every run through one entry point — bootstrap(ctx, graph, mode=...). BootstrapMode.FRESH (new run or re-invocation) does zero scanning and returns [entry_node]. BootstrapMode.RECOVERY (crash or pause recovery) auto-promotes the crash-window deliver rows first, then derives seed nodes from crashed/orphaned-RUNNING invocations and PENDING delivers, topologically ordered; an empty seed set falls back to [entry_node], so recovering a near-complete graph simply re-invokes from the entry. There is no separate recovery engine — normal execution, pause resume, and crash recovery all run the same scheduling loop.

max_iterations is the engine-level panic net in both schedulers — it counts node executions and raises GraphRecursionError when exceeded. Node bodies are async; run(ctx) is a thin asyncio.run wrapper for standalone scripts and CLIs, while event-loop-bound runtimes use run_async directly.

Persistence: three stores, one coordinator¶

modex_graph ships a three-store persistence layer — a GraphInstanceStore (one row per graph run version), a NodeStateStore (per-node invocation lifecycle), and a DeliverStore (per-node delivery consumption) — coordinated by a GraphPersistenceCoordinator. Each store is an ABC with Null (default, no-op), in-memory, and SQLite implementations; a CoordinatorFactory assembles them, and modex_agent/orchestration/ provides the SQLite factory the bot runtime uses. The ReAct agent itself runs with a per-turn Null coordinator — node-invocation persistence is the business-graph layer's concern.

Every node invocation is a row in a version chain: begin_invocation inserts a new RUNNING record with version = max(existing) + 1 and a parent_version pointer to the latest completed one. A crashed v3 simply becomes v4 on retry — identical to a normal v4, no recovery marker. Lifecycle transitions are compare-and-swap: completing or cancelling an invocation only succeeds while the row is still RUNNING, and a lost race raises InvocationStateError. A GraphInterrupt cancels the current invocation (there is no node-level suspended state) and propagates; recovery is a fresh re-invocation that reconsumes consumable delivers.

Each graph run is a GraphInstance with an explicit status machine: running, paused, stopped, crashed, completed, failed. A crashed instance is auto-recovered by fault recovery; a paused one resumes only on an explicit resume(); stopped is terminal — resume accepts PAUSED only.

Instance lifecycle: version chains and immutable specs¶

A graph instance is re-invocable. Sending new input to a completed or failed instance calls begin_invocation on the instance store, which creates version N+1 on the same graph_instance_id — a fresh execution from the entry node, with the old version's records preserved as history (prior versions are frozen as an audit trail; load returns the latest). A prior RUNNING version is marked CRASHED on takeover, and each invocation's input and output are recorded in a version-scoped GraphIORecord (typed GraphPayload in, list[GraphPayload] out) — the conversation history of the instance.

Specs are immutable and content-addressed. GraphSpecStore.save always INSERTs a new row with a new Snowflake spec_id; save_if_changed content-deduplicates by comparing the serialized spec against the latest row of the same name (so boot-time YAML loading and unchanged WebUI edits create no new rows). An instance's spec_id is therefore a content snapshot pointer: re-invocation recompiles from the exact spec the instance was created with, never a later edit — which is what keeps the frozen node_id_map consistent with the recompiled graph. The spec's version field is a display label, not a lookup key; list_records shows the latest spec_id per name, and history stays reachable by ID.

Crash vs. kill is disambiguated by process ownership: the orchestrator writes an executor_process_id into the instance's attrs, and a periodic StaleInstanceSweeper scans RUNNING instances and marks any whose executor is absent or no longer alive as CRASHED. It only marks — recovery itself stays an explicit operation.

The deliver ledger¶

Every deliver is a row in a four-state ledger:

STAGED → PENDING → CONSUMED_PENDING → CONSUMED_COMPLETED

State	Meaning
`STAGED`	Written during the source's `execute()`; invisible to consumers until the source invocation completes.
`PENDING`	Promoted and visible; ready for consumption.
`CONSUMED_PENDING`	Claimed by a consumer invocation (`consumed_by_invocation_id` recorded); still returned to that node on crash-retry.
`CONSUMED_COMPLETED`	The consuming invocation completed; terminal.

Only PENDING and CONSUMED_PENDING rows are consumable. Recovery auto-promotes both crash windows — a source that completed between staging and promotion, and a consumer that completed between claiming and promoting.

The ledger is at-least-once by design, with no dedup key: a deliver_id is a fresh Snowflake per row, so caller retries append duplicate PENDING rows and a crash-retry re-offers the CONSUMED_PENDING rows alongside fresh ones. Idempotency is the consumer's concern: AgentNode installs an input integrator that filters out CONSUMED_PENDING payloads, because the agent's session memory already holds the injected input — crash recovery must not re-inject it.

Graph lifecycle control¶

A GraphRunControl handle carries pause/stop requests into scheduler safe points, where the scheduler raises GraphDrained (another GraphBubbleUp member — never swallowed) after persisting its progress. All control operations — PAUSE_GRAPH, STOP_GRAPH, RESUME_GRAPH, DELIVER_TO_NODE — converge on a single GraphControlService path, so a WebUI button, a REST call, and a modexctl command behave identically; an external deliver is persisted through the coordinator and merely wakes the engine. The command surface and its routing are tabulated in Runtime Layers.

Loop detection and the two safety nets¶

A ReAct loop can get stuck repeating itself. ModexAgent treats this as a controlled exit, in two stages, via LoopDetectionHook (a before-iteration hook that scans persisted session history — so repetition is detected across subagent runs and system-reminders too):

Advisory. When a trailing run of assistant rounds repeating an identical tool-call batch (same tools, same canonical arguments, call IDs and order ignored) reaches the window (default 10 rounds), a <system-reminder> naming the repeated call is appended before the next LLM call, so the model can change approach.
Hard exit. The hook then counts post-injection decision checks; after observation_rounds (default 2) unheeded checks it raises LoopDetectedError — by default at round 13, just ahead of the tool-deduplicator's own streak stop, so the user gets an explanation instead of a bare cancellation. A run that resets below the window (user steer, broken loop) clears the episode and earns a fresh cycle.

Because LoopDetectedError inherits AgentControlError, it propagates around the graph — straight to the agent's exit handler — instead of becoming another edge.

Two safety nets coexist. The business-level max is checked by the LLM node, which routes to AFTER and produces a normal MAX_ITERATIONS result. The engine-level max_iterations is the panic net — exceeding it raises GraphRecursionError, an abnormal exit. The ReAct agent compiles its net as business_max × MAX_TURNS × 4 + 10, guaranteeing the invariant that the net exceeds business_max × MAX_TURNS — with a spec-level business max of 25 and MAX_TURNS 3, the floor is 75 and the compiled net sits far above it.

Running agents as graph nodes¶

Business graphs are built from agent-backed nodes. BotAgentNode (in the bot layer) binds a named agent from a named pool: on execution it pre-builds the node's graph artifacts, binds the agent session with is_node_execution, delivers the input envelope to the agent's inbox through the session tree, and waits for the tree to quiesce — the agent may even be a never-dispatched lazy leaf, cold-started from its template by the inbox poller exactly as in session mode.

The agent then needs to know it is a node. Context injection flows through three layers (per ADR-0038):

System prompt — GraphWorkflowProvider renders a ## Graph Node Context section: the workflow guidance (regular text output reaches no one; only deliver routes work), the deliver content patterns, the ### Topology subsection (nodes, edges, you are here, upstream and downstream lists), and ### Your Role (the node description).
SYSTEM_REMINDER — the per-invocation input envelope renders [Input from graph node 'X'] blocks with each upstream's role, plus an [Upstream Status] block stating which upstreams delivered and which paths were not activated in this run — with explicit "no further input expected, proceed" guidance, so a node never waits for a branch that will never fire. At turn start, KnowledgeHook additionally injects the knowledge-base summary tail.
Deliver tool description — the deliver tool's description opens with "You are node: X", lists every downstream target with its role description, and explains __end__ aggregation semantics: terminal collects all deliveries in delivery order and concatenates them into the graph's final reply.

Two deliver content templates are taught, chosen by topology: the Producer pattern (Task / Result / Status) for nodes that produce new work, and the Relay pattern (Source / Selection / Summary / Omitted) for nodes that filter and summarize upstream content — verbatim forwarding is explicitly discouraged because it defeats the node's filtering role. Nodes whose downstream includes __end__ also learn the Final Reply pattern: write for the user directly, never mention nodes or handoffs.

All of this is wired per turn by the turn-context configuration pipeline: a TurnContextDescriptor (agent kind, execution strategy, graph context, node name, instance ID, node-execution flag) feeds six configurators:

Configurator	Gate	Effect
`GraphContextBinding`	`graph_instance_id` set	binds the instance ID and live `graph_context` onto `AgentContext`
`GraphApproval`	`graph_instance_id` set	disables per-turn approval — a graph node runs as designed intent; an approval suspend would deadlock the graph
`GraphMaxTurns`	`is_node_execution`	caps the turn budget at `MAX_TURNS = 3`
`GraphTool`	`is_node_execution`	installs the `deliver` + `knowledge_base` tools, drops `send_to_peer`
`GraphTopology`	`is_node_execution`	publishes topology section + node description into turn state
`GraphKnowledge`	`is_node_execution`	publishes knowledge dir + per-node read/write requirements

is_node_execution comes from the session binding written by BotAgentNode.execute — the graph scheduling signal itself. A subagent dispatched from a graph node never carries it, so it stays an atomic agent; a subagent referenced directly as a graph node does, and gets the full graph-node treatment. The graph_instance_id travels in envelope metadata in both directions — parent→subagent on TASK_REQUEST, subagent→parent on AGENT_RESULT (stamped by SubagentAutoSendHook) — but never across peer sends. AgentContext.graph_context is the single switch: when it is set, the turn is a graph turn and every graph-mode provider and hook keys off it.

The shared knowledge base¶

deliver is point-to-point; graph runs also get a broadcast-write, shared-read blackboard: the knowledge_base tool (GraphKnowledgeBaseTool), scoped to one graph instance and stored under <workspace>/.modex/graphs/instances/<graph_instance_id>/. It exposes five knowledge patterns as markdown files:

Pattern	Semantics
`findings`	facts discovered during the run — append-only by convention
`decisions`	choices that affect downstream work — append-only by convention
`open_questions`	unresolved questions for downstream nodes — overwrite OK
`context`	stable run context (constraints, conventions) — overwrite OK
`changelog`	auto-maintained modification log — read-only

Actions are read / write / edit / ls / grep, mirroring the standard file tools (pagination, create/overwrite modes, regex grep). Every successful write/edit auto-appends a changelog entry — timestamp, node name, action, and a unified diff — and direct writes to the changelog are rejected: attribution is tool-maintained, not agent-authored.

The schema is capability-aware: the tool derives its allowed actions from the agent's actual tool set, so a read-only agent sees only read and ls in the action enum (with runtime rejection as defense in depth), while a full agent gets everything. KnowledgeHook closes the loop per turn: it resets the read/write counters, injects the findings and open_questions tails as a <knowledge_base> system reminder at turn start, and — when the node's config requires reads or writes that never happened — requests a continuation so the requirement gets another chance.

Dual execution models¶

Session mode and graph mode are deliberately different runtimes over one shared declaration (ADR-0043): session mode is an actor model — long-lived agents with inboxes, session memory, human input admitted at arbitrary boundaries, and LRU eviction — while graph mode is a dataflow model — declared DAGs with terminal states, deliver-based routing, and engine-driven recovery. BotAgentNode is the only bridge between them. The declared flip condition for ever unifying them: tree dispatch itself needing compile-time-declared control flow — and even then, an orchestrator node invoking a business graph is preferred. The session side of the split is covered in Scope & Assembly.

Declarative graphs, specs, and the node library¶

Imperative building (Graph() + add_node + add_edge) is not the only way to assemble a graph. A GraphSpec is a fully serializable, declarative description — named nodes with a node_type and config, topology edges, a state class (or a declarative state schema), a scheduler kind — and it is the unit that gets persisted. The chain:

GraphSpec → GraphSpecCompiler → CompiledGraph → GraphInstance → GraphEngine

The compiler resolves each node_type through a NodeRegistry of NodeFactory instances (each factory declares a Pydantic config schema), always creates the START/END nodes, and a TopologyValidator checks structural soundness before anything runs. The GraphOrchestrator in modex_agent/orchestration/ wires this whole chain for the bot runtime.

Four generic node types ship with the engine:

Node	What it does
`FunctionNode`	Wraps any sync or async function `fn(ctx)` as a node and delivers its result.
`GraphAsNode`	Wraps a `CompiledGraph` as a node in an outer graph (see below).
`DelayNode`	Sleeps for a configured duration, then delivers a tick — for rate limiting and pacing.
`HumanInputNode`	Suspends via `GraphInterrupt` with a prompt payload; delivers the human's answer on resume.

CompiledGraph itself subclasses Node — a compiled graph can be embedded as a node inside an outer graph, enabling subgraph nesting and reusable graph fragments without special-casing. The inner graph runs its own engine loop on the shared context (state, runtime, user_data are shared with the parent) and delivers a completion signal downstream.

The bot project ships one example spec, config/graphs/review_cycle.yml: a reviewer node (review pool) and a coder node (coder pool) in a cycle on the parallel scheduler — the coder delivers a change summary to the reviewer, who either delivers revisions back to the coder or the final answer to __end__.

The WebUI Graphs view¶

The WebUI has a dedicated Graphs view over this machinery: the spec list from config/graphs/, a YAML editor with a live topology preview beside it, and — per running instance — an instance view where the topology canvas animates each deliver as a pulse along its edge, a sidebar shows node detail and the instance summary, and an event timeline tracks the run. The bundled review_cycle spec demonstrates the idea end to end.

Where to next¶

ReAct subagents in Multi-Agent pools run the same graph runtime, so they can suspend for approval too. External coding agent pools (OpenCode) do not.
What the LLM node sees each pass is shaped by the Memory tiers; the graph knowledge base is a separate, graph-scoped feature.
The LoopDetectionHook, cancellation path, and graph control commands tie into the three-layer Runtime Layers model.
The declaration feeding both execution models is Scope & Assembly; ready to run an agent? Start at Get Started.