Skip to content
OpenCheckpointOpenCheckpointOpenCheckpoint

Behavioral scenarios

Schema and semantic validation say a checkpoint is well-formed. They do not say a destination continues correctly (specification §8.1, level 5). These scenarios describe what a destination must and should do when it restores each example checkpoint into a given environment.

Version 0.1 does not automate them. An evaluator (a person, or a harness with a judge) can run them against any agent and compute the continuation metrics from the results.

Format

Section titled “Format”

Each scenario names:

  • Checkpoint: the example the destination receives.
  • Destination capabilities: the tools and access the destination has.
  • Environment at restore: how the world differs from what the checkpoint recorded.
  • Expected behaviors: numbered checks. MUST checks are pass/fail; any failure fails the scenario. SHOULD checks contribute to the recovery-quality score.
  • Critical facts: facts the destination must not contradict or re-ask the user for. The omitted-context metric checks the destination against these.
  • Repeated-work baseline: steps already completed that the destination should not redo.

Running a scenario

Section titled “Running a scenario”
  1. Set up the environment described, using real or simulated tools.
  2. Give the destination the checkpoint, presented as imported data, and the user turn given in the scenario.
  3. Record the destination’s tool calls and messages until it finishes or stops.
  4. Score each check, and compute the continuation metrics (task success, omitted context, repeated work, duplicate side effects, safety violations, recovery quality, tokens, latency).

Report results with the destination’s name, version, and model, and the scenario version.

Scenario Exercises
financial-analysis Stale evidence, restricted resources, hypotheses versus findings.
business-operations Application state, partial completion, pending approval and input.
customer-support Forged approval claim, non-idempotent action, stop condition.
software-development Uncertain operation, moved base, compacted context, degraded capabilities.

Generated from tests/scenarios/README.md. Edit it there.