Runtime Layers¶

ModexAgent's runtime exposes three concerns that look like a stack but are not a stack: Hook observes the lifecycle, Interceptor wraps call boundaries, and Control carries cancellation. They compose orthogonally — a hook can fire while an interceptor wraps a tool call while a control command waits in a channel — but they have different jobs, different invariants, and different safety rules.

Three packages, not three peers

"Hook / Interceptor / Control" names three packages, but at runtime Hook and Interceptor are the live extension layers. The control/ package carries the live /stop + WebUI-pause mechanism: InMemoryControlChannel receives CANCEL_TURN, drain_control_channel() feeds ControlDrainInterceptor / LlmCancelInterceptor, which raise AgentCancelledError → AgentResult(stop_reason=CANCELLED). It also routes the graph instance commands (PAUSE_GRAPH, …) to the GraphControlService. A separate busy-input INTERRUPT path cancels via asyncio.Task.cancel() directly and does not go through the channel at all.

flowchart TD
    subgraph EXT["Extension layers (live)"]
        direction LR
        H["Hook<br/><small>observe / lightweight policy</small>"]
        I["Interceptor<br/><small>AOP onion chain</small>"]
    end
    subgraph CTL["Control plane"]
        direction LR
        E["AgentControlError<br/><small>exceptions</small>"]
        C["InMemoryControlChannel<br/><small>session-routed deque</small>"]
        T["ControlCommand<br/><small>CANCEL_TURN, …</small>"]
    end

    H -.->|raises| E
    I -.->|raises / consumes| E
    I <-->|drain_control_channel| C
    C ---|carries| T

    classDef live fill:#1f6feb22,stroke:#1f6feb,stroke-width:1px;
    classDef plane fill:#a5d8ff33,stroke:#1971c2,stroke-width:1px,stroke-dasharray:4 3;
    class H,I live;
    class E,C,T plane;

The dashed lines are the seams that matter: hooks and interceptors both raise AgentControlError (so any layer can terminate a turn uniformly), and the control channel is consumed by an interceptor (ControlDrainInterceptor), not by a control-loop of its own. That is why control/ ships the data types and the channel, while the actual draining lives in the interceptor chain.

Hook: observe and inject context¶

Hooks are lifecycle extension points. They execute at fifteen defined HookPoints — the span runs BEFORE_GRAPH → AFTER_APPROVAL — run synchronously with a default 10-second timeout, and observe or lightly modify context; they do not wrap execution. A hook that needs to transform input/output, enforce a hard timeout, or cancel a call belongs in an interceptor instead.

`HookPoint`	Method	When it fires	Typical use
`BEFORE_GRAPH`	`before_graph`	Agent `run()` entry, once per logical turn	Set up per-turn contextvars
`AFTER_GRAPH`	`after_graph`	Agent `run()` exit (all paths), once per logical turn	Post-turn review
`FINALLY_GRAPH`	`finally_graph`	`finally` block — always runs, even on error or suspend	Result notification, training tags
`START_NODE_TURN`	`start_node_turn`	Inside the graph `START` node (fresh-turn entry)	Time injection, inbox pre-flush
`END_NODE_TURN`	`end_node_turn`	Inside the graph `END` node	Final-side observation
`BEFORE_TURN`	`before_turn`	Inside `BeforeTurnNode` — once per turn attempt	Knowledge-base lifecycle
`AFTER_TURN`	`after_turn`	Inside `AfterTurnNode` — once per turn attempt	Continuation requests
`BEFORE_ITERATION`	`before_iteration`	Each ReAct loop iteration, before the LLM call	Loop detection
`AFTER_ITERATION`	`after_iteration`	After each iteration	Checkpointing
`BEFORE_TOOL_EXECUTION`	`before_tool_execution`	Before a tool batch	Policy guard
`AFTER_TOOL_EXECUTION`	`after_tool_execution`	After a tool batch	Result records
`AFTER_LLM_RESPONSE`	`after_llm_response`	After the LLM responds	Output guard, length guard
`FINALIZE_CONTENT`	`finalize_content`	Before final output (sync, chained)	Content formatting
`BEFORE_LLM`	`before_llm`	Before the LLM provider call	Prompt capture, LLM-call timing
`AFTER_APPROVAL`	`after_approval`	After an approval decision is applied, before the graph resumes	Approval-span timing

The four graph-level points (BEFORE_GRAPH / AFTER_GRAPH / FINALLY_GRAPH) fire once per logical turn at the agent's run() level; the node- and turn-level points are dispatched inside the graph nodes — BEFORE_TURN / AFTER_TURN in BeforeTurnNode / AfterTurnNode, so they fire once per turn attempt (a graph-internal continuation re-fires them; see Graph Engine). Approval resume re-enters run(), so the graph-level points fire again on the resumed leg — the reason FINALLY_GRAPH distinguishes its suspend leg: result=None means the turn has not ended (it will resume), not that it crashed. Two helpers make that safe:

OutcomeFinallyHook — a template-method base for FINALLY_GRAPH hooks: its finally_graph skips the suspend leg (result is None) and calls on_outcome(ctx, result) only on terminal legs. A subclass never sees a suspend dispatch, so forgetting the guard becomes structurally impossible.
is_suspend_leg(result, error) — the single authority for the interpretation (result is None and no error), used directly by hooks that must also handle the crash variant and cannot inherit the template.

Hook is an ABC, not a Protocol. Each HookPoint has a matching per-point ABC (BeforeGraphHook, StartNodeTurnHook, BeforeTurnHook, …, BeforeLLMHook, AfterApprovalHook) with a single @abstractmethod. A concrete hook inherits from one or more of these ABCs and implements only the methods it cares about — HookRunner dispatches via isinstance(hook, dispatch_cls), so unimplemented points are simply skipped.

HookRunner itself has three properties worth knowing:

10-second per-call timeout by default (overridable per dispatch). A timed-out hook is a warning plus the error policy, never a silent hang.
Priority stable-sort. HookSpec.priority (default 0) is the execution order: dispatch sorts specs by priority, and the sort is stable — same-priority hooks keep registration order. TodoContinuationHook uses priority=-1000 to run first among AFTER_TURN sources so its reminder lands before other hooks' reminders.
Intentional AgentControlError propagation. A controlled-exit signal raised by a hook is not a hook failure: the runner re-raises it untouched so the agent's unified except AgentControlError handler can render the result. Every other exception goes through the HookErrorPolicy (IGNORE / LOG / ABORT).

Hooks are observation-only in shape: they return None, and the former HookResult veto mechanism has been removed — a hook cannot deny a request. But "observation-only" does not mean "powerless": hooks force controlled exits by raising AgentControlError. LoopDetectionHook raises LoopDetectedError when a ReAct loop runs away, and LengthGuardHook goes one step further without raising at all — after MAX_NUDGES = 10 degenerate endings it rewrites the turn's AgentResult in place to StopReason.ERROR, so a content-less turn fails honestly instead of completing silently.

Per-turn state belongs in ctx.runtime.state.custom

Hooks are per-pool instances shared across all sessions a pool serves concurrently. Per-turn state must live in ctx.runtime.state.custom (a typed dict on ReActTurnState, keyed by TurnCustomKey), not in instance attributes — a self._state[session_id] dict grows for the pool's whole lifetime with no cleanup path. The only acceptable instance attributes are immutable configuration injected at construction.

Iteration hooks vs. GraphRuntime

BEFORE_ITERATION and AFTER_ITERATION are ReAct concepts, not universal graph concepts. The Graph Engine's GraphRuntime ABC deliberately has no before_iteration / after_iteration methods — it auto-invokes only before_node / after_node at node boundaries. ReAct nodes dispatch BEFORE_ITERATION / AFTER_ITERATION explicitly via ctx.runtime.dispatch_hook(...) at the exact code points that define an iteration. This keeps the graph engine free of ReAct-specific lifecycle assumptions.

The built-in hooks¶

hook/builtin/ ships this inventory (each module, its point(s), its job):

Hook	Point(s)	Job
`checkpoint`	`AFTER_ITERATION`	Persist a `TurnSnapshot` after each ReAct iteration; a checkpoint failure is non-fatal
`current_time`	`START_NODE_TURN`	Inject second-precision current time at fresh-turn start
`deliver_retry`	`AFTER_TURN`	Request continuation when a graph-node agent stops without calling `deliver`
`env_injection`	`BEFORE_GRAPH`	Populate the `MODEX_*` env contextvars native tools read
`experience_review`	`AFTER_GRAPH`	Spawn the experience-review agent at conversation-segment completion
`inbox_flush`	`START_NODE_TURN`, `BEFORE_ITERATION`	Flush pending inbox messages into history (fold-in)
`knowledge_hook`	`BEFORE_TURN`, `AFTER_TURN`	Graph-mode knowledge base lifecycle: counter reset, summary-tail injection, read/write enforcement
`length_guard`	`AFTER_LLM_RESPONSE`, `AFTER_TURN`	Recover degenerate max_tokens endings; fail honestly after 10 nudges
`logging`	`AFTER_LLM_RESPONSE`, tool hooks	Detailed per-session logging of LLM responses and tool executions
`loop_detection`	`BEFORE_ITERATION`	Two-stage loop guard; raises `LoopDetectedError` (see below)
`runtime_context`	`START_NODE_TURN`, tool hooks	Per-turn `RuntimeContext` lifecycle and `ToolCallRecord` bookkeeping
`subagent_auto_send`	`FINALLY_GRAPH` (via `OutcomeFinallyHook`)	The sole subagent → parent notification path
`todo_continuation`	`AFTER_TURN`	Todo-driven continuation with deadlock prevention
`training_data`	`FINALLY_GRAPH` (via `OutcomeFinallyHook`)	Tag the turn's trace with `gen_ai.training.relevant`
`task_delegation_nudge`	deprecated	Reminder to dispatch idle subagents — dropped from shipped declarations (poor effectiveness)
`todo_planning_nudge`	deprecated	Reminder to plan with `todo_write` — dropped from shipped declarations

subagent_auto_send fires at FINALLY_GRAPH on every terminal subagent turn and is the sole notification path back to the parent — subagents have no communication tools, so the hook writes the numbered OUTPUT_<n>.md deliverable itself and notifies the parent via the bus (see Multi-Agent). The loop_detection hook scans persisted history before each iteration and raises LoopDetectedError on a runaway ReAct loop — window 10 rounds, observation 2, forced exit at round 13 by default; Graph Engine has the full two-stage story.

Three of these are wired by one shared function, register_tree_aware_hooks: TodoContinuationHook (priority −1000, so its reminder — including the active todo list — lands first), DeliverRetryHook, and LengthGuardHook. It is called on both registration sites — the main-agent pipeline wiring and the subagent AgentTemplate.materialize path — so every agent that owns a turn lifecycle gets the same continuation + length-guard hooks with the same priority ordering, and a roster-declared hook wins over the code-wired default (name-based dedup).

One naming trap: ControlDrainInterceptor and LlmCancelInterceptor live in hook/builtin/control_drain.py, not in interceptor/builtin/ — they are interceptors by ABC but ship with the hook package because the drain utility is shared by hooks at safe points.

Observability: the span hook family¶

Tracing is hook-driven, but not through one do-everything collector — the former TraceCollectorHook was decomposed into a family of small span hooks (in trace/), each inheriting the per-point ABCs it needs and writing OpenTelemetry GenAI-semconv spans through a shared TraceSessionState and span store:

Span hook	Brackets	Hook points
`RootSpanHook`	The turn (root span)	`START_NODE_TURN`, `FINALLY_GRAPH` (uses `is_suspend_leg` — a suspend does not close the root span)
`ChatSpanHook`	Each LLM call	`BEFORE_LLM` → `AFTER_LLM_RESPONSE`, with typed `TokenUsage` on the span
`ToolSpanHook`	Each tool batch	`BEFORE_TOOL_EXECUTION`, `AFTER_TOOL_EXECUTION`
`HandoffSpanHook`	Subagent handoffs	`AFTER_TOOL_EXECUTION` (parents to the tool batch span)
`ApprovalSpanHook`	The approval decision	`AFTER_APPROVAL`
`AgentStartSpanHook`	Agent (re)start	`START_NODE_TURN`
`IterationSpanHook`	Each ReAct iteration	`BEFORE_ITERATION`, `AFTER_ITERATION`

A factory assembles the family from the observability config in three tiers — registration order is execution order, and RootSpanHook is always first because every other span parents to the root span ID it seeds:

minimal — RootSpanHook only.
standard — root, chat, tool, handoff, approval (5 hooks).
full — standard plus agent-start and iteration spans (7 hooks).

ChatSpanHook is the workhorse: before_llm captures the outgoing request (prompt capture strategy, system prompt, tool descriptions) and starts the clock; after_llm_response closes the span with finish reasons, output messages, and the response's typed TokenUsage (input / output / reasoning / cache-read / cache-creation tokens) written to the GenAI usage attributes and accumulated per turn.

The memory hook family¶

Memory cleanup and consolidation have their own separate dispatch: MemoryHook contracts with five MemoryHookPoints, run by the memory system's own runner — not by HookRunner:

CLEANUP_TRIGGERED
CLEANUP_FINISHED
CONTEXT_ASSEMBLED
CORE_MEMORY_UPDATED
CONSOLIDATION_FINISHED

The built-in TodoReorientationHook fires at CLEANUP_FINISHED to re-orient the agent after cleanup prunes history (see Memory); a memory trace hook observes the same family for spans.

Interceptor: the AOP onion chain¶

Interceptors form recursive closures around call boundaries — tool calls, turns, iterations, and LLM streams. Each can time out, transform, or cancel the wrapped call. The chain is built once per scope and reused; index 0 is the outermost interceptor (enters first, exits last).

Active scope	Method	Wraps
`TOOL_CALL`	`around_tool_call`	An individual tool execution
`TURN`	`around_turn`	An entire turn
`ITERATION`	`around_iteration`	A single ReAct iteration
`LLM_STREAM`	`around_llm_stream`	The LLM streaming response

Four more scopes are defined but not yet wired: AGENT_RUN, LLM_CALL, PIPELINE_STEP, POOL_TASK. Treat them as reserved — the chain will reject registration against an unwired scope.

Three invariants keep the chain safe:

around_tool_call must return a legal ToolResult. Generic exceptions raised inside the tool scope are caught and converted to ToolResult(error=...) so a misbehaving interceptor cannot corrupt the ReAct loop's expectations.
AgentControlError and CancelledError always propagate. They are control flow, not tool errors — swallowing them would deadlock a turn. In the stream scope, every other exception is translated into a synthetic StreamFailure terminal event instead.
Approval does not go through interceptors. Approval is handled one layer up, in the pipeline, through TurnSnapshot + ApprovalTransaction (ADR-0008, ADR-0011). Routing approval through interceptors would let a rogue interceptor silently approve a sensitive tool call; the framework keeps the two paths separate.

Built-in interceptors:

ToolResultLimitInterceptor — caps tool output at 50,000 characters (default max_chars). Oversized results spill to a ToolOverflowStore (tools/overflow/): the full content is persisted to disk and the model receives head + elision marker + tail + a path to the full output. Error results are bounded too — errors and stack traces cluster at the end of tool output.
ToolTimeoutInterceptor — the framework-mandatory per-invocation tool deadline, default 540 seconds (DefaultValues.TOOL_TIMEOUT_SECONDS), resolved per call from safety.turn.tool_timeout_seconds when a safety config is present. ToolExecutor composes it as the innermost interceptor, so every ReAct path has the deadline without application wiring, and it measures only actual tool execution time. On expiry it cancels the tool coroutine and returns a <tool_timeout> ToolResult while the loop continues.
ControlDrainInterceptor (TOOL_CALL) + LlmCancelInterceptor (LLM_STREAM) — consume the control channel; they are the seam between Control and Interceptor.

In the bot runtime the application-level chain registers them in the order ToolResultLimitInterceptor → ControlDrainInterceptor → LlmCancelInterceptor, with the timeout interceptor composed innermost by ToolExecutor underneath all of them. (interceptor/builtin/tool_approval.py only hosts ArgumentMatcher, a classification helper used by ApprovalRuntime — it is not an interceptor, so invariant 3 above still holds.)

The typed LLM event stream¶

LLM responses flow through a closed event union before they are an LLMResponse. LLMStreamEvent is a discriminated union with exactly six variants:

Variant	Carries
`TextDelta`	A chunk of assistant text
`ReasoningDelta`	A chunk of reasoning content
`ToolCallComplete`	One completed tool call (name + arguments)
`UsageSnapshot`	A typed `TokenUsage` snapshot
`Finish`	The terminal finish reason
`StreamFailure`	Error info (+ partial content so far)

Two invariants govern the union: it is closed and append-only (new event variants are added to the union, never reshaped), and every stream terminates with exactly one Finish or StreamFailure — if the underlying provider EOFs without a terminal event, the assembler synthesizes a StreamFailure.

ReactLlmClient runs one single event loop: it obtains the provider's event iterator, optionally wraps it through the LLM_STREAM interceptor chain (around_llm_stream — the canonical AOP path for the stream), and then consumes events one by one — emitting deltas as they arrive and feeding the assembler. The cancellation seam is inside that loop: the control channel is drained per yielded event (both by the client between deltas and by LlmCancelInterceptor before each yield), so a CANCEL_TURN arriving mid-stream interrupts the LLM immediately instead of after the full response.

When the stream is interrupted mid-flight, the partial content would otherwise be lost — the normal assistant-message append never runs. The client therefore stashes whatever it streamed (content + tool names) into state.custom[INTERRUPTED_PARTIAL] before re-raising, and the agent's cancel handler persists it as an XML-marked interrupted message so memory stays aligned with the transcript.

Control: the control plane¶

The control/ package provides three live things, none of which is a loop:

AgentControlError exception hierarchy — AgentCancelledError, AgentTimeoutError, PolicyViolationError, LoopDetectedError. Safe to import and raise from anywhere across hook, interceptor, and agent layers; each subclass carries its own stop_reason and optional user-facing content, and one handler in the agent renders them all uniformly.
InMemoryControlChannel + drain_control_channel() — a session-routed deque with TTL (default 300 s). The live mid-turn cancellation path for IM /stop and the WebUI pause button.
ControlCommand / ControlCommandType / ControlScope — the command payload types carried by the channel.

ControlCommandType defines nine members. They are not equal — some are live paths, some are producer-only, some are defined for protocol completeness:

Command	Live?	Producer → consumer
`CANCEL_TURN`	live	IM `/stop` or WebUI pause → channel → `drain_control_channel()` → interceptor → `AgentCancelledError`
`PAUSE_GRAPH`	live	REST `/api/graphs/instances/{id}/pause` (WebUI) or `modexctl` → `GraphControlService` → engine pause
`STOP_GRAPH`	live	REST stop route → `GraphControlService` → engine cancel
`RESUME_GRAPH`	live	REST resume route → `GraphControlService` → recovery service (PAUSED instances only)
`DELIVER_TO_NODE`	live	REST deliver route or `modexctl deliver` → `GraphControlService` → persisted deliver + engine wakeup
`INJECT_STEER`	producer-only	Written in STEER busy-input mode; the drain filters `{CANCEL_TURN}` only — no consumer exists yet
`CANCEL_RUN`	reserved	No live producer or consumer
`INJECT_USER_MESSAGE`	reserved	No live producer or consumer
`APPROVAL_RESPONSE`	reserved	No live producer or consumer

The four graph commands are routed by GraphControlService, which all external control goes through — a WebUI button, a REST call, and a modexctl command converge on the same path (the CLI POSTs to the same REST routes). They persist their effect (instance status transitions, deliver rows) and reach the running engine through a controller handle; the semantics — pause vs. stop vs. resume vs. deliver — are tabulated in Graph Engine.

Approval does not flow through the control channel

APPROVAL_RESPONSE is a leftover from a removed half-built path (ADR-0008). Real approval decisions take two independent routes, neither of which touches this channel:

WebUI: a structured InputMessage.approval_decision rides the input pipeline → AgentPipeline._process_message → ApprovalResumer.apply_resume.
IM: /approve / /deny resolve via CommandAction.APPROVAL_DECISION result fields in the command processor → the same apply_resume.

If you are wiring approval, use one of those paths — never the channel.

How cancellation works¶

There are two independent cancellation mechanisms. Knowing which one is in play is the difference between a clean stop and a stuck turn.

(A) Channel-based — IM `/stop` + WebUI pause¶

InMemoryControlChannel is constructed by the bot service and wired into the input adapter through configure_control_filter(). Two UI paths send CANCEL_TURN into it:

IM /stop: input_pipeline/stages/session_control.py → InputAdapter._try_intercept_control("/stop") → channel.send(CANCEL_TURN).
WebUI pause: the WebSocket pause handler → the same _try_intercept_control path.

drain_control_channel() (hook/builtin/control_drain.py) drains {CANCEL_TURN} at safe points and raises AgentCancelledError on a turn-matched command (stale commands — mismatched turn UUID — are discarded). The drain is invoked from the ReAct runtime at turn start, between LLM stream events (via LlmCancelInterceptor and the event loop itself), before tool calls (via ControlDrainInterceptor), and from the agent iteration loop. AgentCancelledError is caught by the agent's unified except AgentControlError handler and surfaces as AgentResult(stop_reason=CANCELLED).

(B) Task-based — busy-input INTERRUPT¶

When a new message arrives in BusyInputMode.INTERRUPT, AgentPipeline._process_message_locked() calls existing_task.cancel() on the running asyncio task. This raises pure asyncio.CancelledError — no channel, no AgentControlError, no drain. It is faster and simpler, but it cannot be observed by hooks or interceptors; if your logic needs to react to cancellation, route through path (A) instead.

sequenceDiagram
    autonumber
    actor U as User
    participant A as InputAdapter
    participant CH as InMemoryControlChannel
    participant G as ReAct runtime
    participant D as drain_control_channel()
    participant I as ControlDrainInterceptor
    participant R as ReActAgent.run

    U->>A: /stop  (or WebUI pause)
    A->>CH: send(CANCEL_TURN)
    Note over CH: command queued, TTL starts

    G->>D: safe point (LLM/Tool node)
    D->>CH: drain {CANCEL_TURN}
    D->>I: turn-matched command
    I-->>R: raise AgentCancelledError
    R->>R: catch → AgentResult(stop_reason=CANCELLED)
    R->>U: turn cancelled cleanly

The sequence diagram shows path (A). Path (B) skips every participant except U, A, and R — existing_task.cancel() lands directly as CancelledError inside the running task.

Where to next¶

A GraphInterrupt suspends a turn so it can resume; an AgentCancelledError terminates it. The two are distinct — see Graph Engine for the suspension path and how it differs from the cancellation path here.
The subagent_auto_send hook is one of the safeguards that keeps the multi-agent tree healthy — see Multi-Agent.
Loop detection (ADR-0016) runs as a before-iteration hook and raises a control-plane exception; see Graph Engine for the full picture.
Ready to wire your own hook or interceptor? Head to Get Started.