Runs and resumption

Verdog records each local workflow execution as a run, with its own ID, output directory, and launch arguments. A checkpoint records the workflow state needed to continue execution after an interruption.

• Resume continues the same run from its latest completed checkpoint.
• Fork creates a new run from a selected checkpoint.
• Restart creates a new run at the workflow entry, using the recorded arguments or replacement arguments.

Fork and restart can either branch existing agent conversations or begin fresh conversations.

Checkpoints and operations

A checkpoint includes node and feature state, parameters, the value in transit, the remaining transition budget, nested calls, and persistent agent sessions. These values must be serializable for the checkpoint to be restorable. Checkpoints reference files in the run's output directory; those files must remain unchanged.

Operation	Run identity	Workflow state	Arguments	Conversations	Output directory
Resume	unchanged	latest completed checkpoint	checkpoint	restored with copy-on-write	unchanged
Fork	new child run	selected checkpoint	checkpoint	branch or fresh	new
Restart	new child run	initial	reused or overridden	branch or fresh	new

A fork or restart does not change the source run's workflow state, checkpoints, or artifacts. New runs record their parent operation and checkpoint so that run history shows their relationship to the source run.

Checkpoint policy

Runs started from VS Code use automatic checkpointing. Programmatic callers select a CheckpointPolicy when calling Dispatcher.run():

Policy	Meaning
OFF	Execute without recording resumable workflow state.
AUTO	Record every serializable completed checkpoint; report a non-restorable checkpoint without failing the workflow.
REQUIRED	Require every checkpoint to be restorable; serialization failure fails the run immediately.

Checkpoints are recorded after a node completes and routing selects its successor. They can also record progress within a nested call. A TERMINAL checkpoint may mark the completion of a child graph; use the run status to determine whether the entire workflow has finished.

The Runs view in VS Code groups runs by workflow and lineage. It shows status, the latest checkpoint, persistent-session availability, and recorded launch arguments. Use Verdog: Refresh Runs to update the listing or Verdog: Open Run Output to inspect a run's files.

Resume the same run

Select a run in the Runs view and choose Resume Run. Resume uses the latest completed checkpoint, provided that it is restorable, and writes subsequent visits into the original output directory. It restores typed node and feature state, parameters, the value in transit, visit numbers, the transition budget, nested calls, and persistent-session bindings. A succeeded run cannot be resumed; fork a checkpoint or restart it instead.

Before restoring state, Verdog verifies that every referenced file is still present with its recorded contents and permissions. A missing or modified file causes an integrity error.

Files written after the latest checkpoint remain available when the workflow resumes. An unfinished ordinary node uses its next visit directory. Nested calls continue from their recorded state and preserve earlier output. Once a child's result is recorded, replay evaluates only the call adapter. Nodes should keep durable inputs in checkpointed artifacts; files from an unfinished attempt do not establish that its side effects completed.

An agent-provider process can be interrupted after the provider accepted a request but before Verdog recorded its reply. Verdog refuses an ambiguous retry by default. Inspect the invocation artifacts before choosing Retry incomplete invocation in VS Code or passing retry_incomplete=True to Dispatcher.resume(). Retrying may repeat an external side effect or provider charge; it does not establish whether the earlier request was processed.

Restart from the beginning

Select a run and choose Restart Run. For persistent sessions, select Branch conversations to continue from recorded conversation context, or Start fresh conversations to create new conversations. Then choose Reuse recorded arguments or Override arguments. Replacement arguments are entered as a JSON array of strings; [] supplies no arguments.

Restart begins with initial workflow state, including fresh counters and a fresh transition budget. It can reuse conversation context without restoring the previous workflow state.

When branching conversations, restart uses the newest checkpoint that can branch them independently. A newer checkpoint from a provider without conversation branching does not hide an older usable checkpoint. To select a particular checkpoint for its conversation state, use the programmatic source_checkpoint parameter described below.

Fork a checkpoint

Select a run and choose Fork Run, then select a checkpoint and the conversation policy. Branch conversations continues each persistent conversation from its recorded state in an independent provider branch. Start fresh conversations retains workflow state but clears provider conversation identities.

A fork validates and copies the selected checkpoint's recorded artifacts into a new output directory. Files created later in the source run are excluded. The new run is independent of the source files after the copy. It starts a fresh trace and regenerates runtime reports. Verdog restores the selected workflow state and updates run-owned artifact paths to the new directory. Recorded exception objects are opaque to path rebasing.

Available conversation policies depend on the checkpoint and provider. A provider that cannot create an independent conversation branch may still permit a fork with fresh conversations.

Programmatic API

verdog_runtime.interpreter provides synchronous execution, the same three resumption operations, and checkpoint and session policies:

from pathlib import Path

from verdog_runtime.interpreter import (
    CheckpointPolicy,
    Dispatcher,
    SessionPolicy,
)

dispatcher = Dispatcher(project_root=project_root)

result = dispatcher.run(
    definition,
    input,
    output_dir=Path(".verdog/runs/experiment"),
    params=params,
    runtime_options=runtime_options,
    checkpointing=CheckpointPolicy.AUTO,
    workflow_arguments=argv,
)

result = dispatcher.resume(
    definition,
    output_dir=source_output,
    retry_incomplete=False,
)

result = dispatcher.restart(
    definition,
    input,
    source_output_dir=source_output,
    output_dir=new_output,
    sessions=SessionPolicy.BRANCH,
    source_checkpoint=None,
    params=params,
    runtime_options=runtime_options,
    workflow_arguments=argv,
    arguments_mode="reused",
)

result = dispatcher.fork(
    definition,
    source_output_dir=source_output,
    checkpoint=17,
    output_dir=new_output,
    sessions=SessionPolicy.FRESH,
)

The complete public lifecycle surface is:

Call	Public configuration
Dispatcher(...)	execution_handler=None, transition_limit=10_000, cancellation=None, project_root=None
run(definition, input, ...)	output_dir, params, run_id, runtime_options, checkpointing, workflow_arguments
resume(definition, ...)	output_dir, retry_incomplete=False
restart(definition, input, ...)	source_output_dir, output_dir, sessions, source_checkpoint=None, params, runtime_options, workflow_arguments, arguments_mode
fork(definition, ...)	source_output_dir, checkpoint, output_dir, sessions

Programmatic run defaults to CheckpointPolicy.OFF, which creates no durable run record. Pass AUTO or REQUIRED explicitly to obtain resumable state. resume accepts no replacement input, parameters, or arguments because the checkpoint owns them. fork has the same restriction. restart instead receives a new input, parameters, and runtime options. Its workflow_arguments and arguments_mode describe that new launch in lineage metadata.

With SessionPolicy.BRANCH, restart.source_checkpoint selects only the committed conversation anchors; None selects the newest branchable checkpoint. With SessionPolicy.FRESH, it is ignored and no source conversation identity is copied. A restart always begins with initial workflow state.

Cancellation and deadlines

Dispatcher.run(), resume(), restart(), and fork() are blocking calls. Programmatic callers can share one thread-safe CancellationToken with a dispatcher and request cancellation from another thread or control handler:

from verdog_runtime.interpreter import (
    CancellationToken,
    CheckpointPolicy,
    Dispatcher,
    ExecutionCancelled,
)

token = CancellationToken.with_timeout(300.0)
dispatcher = Dispatcher(
    project_root=project_root,
    cancellation=token,
)

# At any time, a supervisor thread may instead request cancellation explicitly:
# token.cancel()

try:
    result = dispatcher.run(
        definition,
        input,
        output_dir=output_dir,
        checkpointing=CheckpointPolicy.AUTO,
    )
except ExecutionCancelled:
    # Resume the latest restorable checkpoint when appropriate.
    ...

with_timeout() converts the relative timeout to an absolute monotonic deadline when the token is created. A plain CancellationToken() has no deadline; cancel() remains effective once called.

Cancellation is cooperative. Verdog checks the token between visits and calls, and while waiting for a managed provider or Workflow process. On interruption, it terminates the managed process tree and waits for it to exit. Authored visit code cannot be preempted; it must return, raise, or enter a managed Agent or call invocation before cancellation takes effect. The built-in process invokers inspect AgentRequest.cancellation, while a custom invoker must cooperate with that token itself.

ExecutionCancelled is a BaseException, so ordinary except Exception recovery does not turn cancellation into graph success. Ctrl+C follows the same interruption and process-cleanup path through KeyboardInterrupt. If checkpointing is enabled, a later resume begins at the latest completed, restorable boundary; Verdog does not claim that arbitrary external side effects were rolled back.

Conversation copy-on-write

Persistent sessions use copy-on-write whenever checkpointing is enabled. The checkpoint stores the provider conversation as an immutable anchor. The first later invocation asks the provider to create a new conversation branch; after that, the new provider session advances normally. Consequently, resuming or forking does not mutate the conversation represented by the checkpoint.

The built-in Codex and Claude command providers expose this capability. A custom provider declares whether it supports branching. REQUIRED checkpointing rejects a persistent session that cannot satisfy the guarantee; AUTO records why branch-based operations are unavailable. A provider must return a new, non-empty session ID for a fork—the source ID is not accepted as evidence of a branch.

A custom invoker advertises AgentSessionCapabilities(fork_latest=True) through its session_capabilities property. When AgentRequest.provider_session_action is AgentSessionAction.FORK, it must branch from the supplied provider_session_id and return the independent identity in AgentReply.provider_session_id. For CONTINUE, it resumes normally. An invoker without this capability remains usable, but branch-based resume, restart, and fork are unavailable at checkpoints containing its established persistent sessions.

See Session for ordinary within-run persistence.

Serializable workflow state

Restoring a checkpoint requires that:

• the graph, node, feature, and parameter addresses still match;
• every node-local value has exactly its declared immutable state type;
• every feature value satisfies its declared feature kind; and
• the runtime, authored source, and isolated environment match the recorded versions and contents.

Inputs, parameters, values in transit, node-local state, and durable call requests and outcomes must be serializable. Prefer frozen, slotted dataclasses whose fields are ordinary values or pathlib.Path objects. With AUTO, an unsupported value makes that checkpoint visible but non-restorable. With REQUIRED, it is a run error.

Changing authored code or installed dependencies after a checkpoint is not an in-place migration. Restore the earlier source/environment to resume exactly, or restart under the new version.

The Python implementation, ABI, minor version, and platform must also match the checkpoint. These compatibility checks apply separately to the environment of each isolated Workflow call.

Checkpoint state is trusted local data. It uses Python object serialization and is not safe to load from an untrusted run directory. The recorded hashes detect accidental corruption; they are not signatures and do not establish who created the files. Do not resume or fork a run copied from an untrusted source.

Resumable call nodes

A generated call visit exposes one synchronous, typed implementation:

def visit_impl(input, state, context, /):
    try:
        child_output = context.invoke(
            ChildInput(...),
            params=context.child_params,
        )
    except ChildDomainError:
        return Success(output=recover(...), state=state)
    return Success(output=adapt(child_output), state=state)

The first evaluation captures and journals the child request. The runtime then runs or resumes the child. After the child returns, or after restoring a checkpoint, the runtime evaluates visit_impl from its beginning and makes the same context.invoke() return the journaled output. A journaled child exception is raised there, preserving ordinary Python try/except, exception transformation, and re-raising. Completed child work is not repeated, but code in the adapter can be. Only the child execution and outcome are journaled; both pre- and post-invocation adapter logic remain subject to replay.

Call adapters must satisfy the following requirements:

1. Every successful evaluation reaches exactly one context.invoke().
2. Every evaluation produces the same child input and parameter override.
3. The adapter performs no unjournaled I/O or process launches, reads no clock or random source, and neither reads nor mutates mutable global state.
4. Construction before the invocation and result or exception adaptation after it are deterministic and replay-safe.
5. The adapter catches concrete Exception types, not BaseException, which Verdog uses for cancellation and execution control.

Side effects belong in ordinary nodes or the child graph, where checkpoints give them explicit execution boundaries. Verdog compares a replayed child request with the journaled request and fails on a mismatch. This prevents a source or environmental change from silently redirecting an active call.

Subroutine calls and Workflow calls use the same adapter contract. An ambiguous external agent request still requires an explicit retry decision, as described under Resume the same run.

When a fork inherits an in-flight call, replay of that already-captured adapter retains its source CallContext.run_id so its journaled request stays stable. The forked child and run records use the new run ID, as do call adapters first entered after the fork.

Run files

Managed outputs live under .verdog/runs/; custom output directories are registered in .verdog/run-registry.json. Run metadata and checkpoints are stored under the output directory's .verdog/ folder. Leave these files under runtime control and use the Runs view to inspect them.

<output>/
├── .verdog/
│   ├── run.json
│   └── checkpoints/
├── trace.log
└── <call-node>/
    └── 000001/
        └── <child-node>/000001/

Root nodes live directly beneath the run output. Both Subroutine calls and Workflow calls place child nodes directly beneath their call visit; see Logging for retries and deeper calls. Parent config.md and stats.md reports link to child reports. Directory paths grow with call depth and are subject to filesystem path limits. Resuming a run preserves its existing artifact paths.

Checkpointed file contents and permissions must remain unchanged. Directories may acquire new children: each checkpoint records the files and directories present at that point. Resume and fork verify that the recorded artifacts are intact before restoring state.

Every .verdog directory is reserved for private data and excluded from artifact inventories, including nested visit workspaces. The root trace.log and runtime-owned config.md and stats.md reports are also excluded because they change as execution continues. User artifacts with those names elsewhere, public invocation artifacts, and failure stack traces remain durable.

Resume an older checkpoint with the runtime and environment that created it. Unsupported checkpoint formats are rejected without modifying the run directory; there is no automatic migration. Output directories without .verdog/run.json are not listed or resumable.

Limits of the guarantee

Checkpointing does not roll back external side effects. File writes outside the run output, network mutations, subprocess side effects, and provider requests may already have occurred when a process stops. Visits that perform such effects should be idempotent or maintain their own operation identifiers and reconciliation logic.