Testing without eyes

Four tools, one crate (crates/sparklight-scenario, plus the capture API in crates/sparklight-render):

Tool	Use it to
Scenario (*.scenario.json)	play the game headless with inputs and check world state, events, frame time and screenshots
Frame capture (Runtime::render_frame)	get the current frame as RGBA or PNG, off-screen, on this machine’s GPU
Golden diff	compare a capture with an approved image under a perceptual tolerance and get a diff image
Snapshot file	save a run to disk and rebuild the exact state in another process (2D scenarios: SnapshotFile; a control-plane session with a 2D runtime and a 3D scene: snapshot.save / snapshot.load)

Design background: docs/AGENT_FIRST.md section 3.6. The runtime itself: docs/agent/runtime.md. spark test and the capture.* / snapshot.* commands are wrappers over this crate (task control-plane). Until they land, cargo run -p sparklight-scenario --example run_scenarios -- <file|dir>... does the same job (flags --update-goldens, --no-render, --out DIR; exit code 0 passed, 1 failed, 2 unreadable file).

Quick start

GAMEWAY_MUTE=1 cargo run -q --release -p sparklight-scenario --example run_scenarios -- samples/platformer/scenarios

stderr gets one line per scenario, stdout the JSON suite report:

PASS platformer-collects-coins (531 frames, 14 passed / 0 failed / 0 skipped, hash 9d3cf9268ac67bd9)

use sparklight_scenario::{run_file, RunOptions};

let report = run_file("samples/platformer/scenarios/platformer-collects-coins.scenario.json".as_ref(), &RunOptions::default())?;
assert!(report.passed && report.complete);

(use the GPU for screenshots), fail_fast (default true), write_report (default true: <out>/<scenario>/report.json).

Scenario file

Schema: schemas/scenario.schema.json (generated from the Rust types; cargo run -p sparklight-scenario --example write_scenario_schema; a test fails when the committed file is stale). Paths are relative to the scenario file.

{
  "$schema": "../../../schemas/scenario.schema.json",
  "version": 1,
  "name": "platformer-collects-coins",
  "scene": "../scenes/main.scene.json",
  "assets": "../assets",
  "seed": 42,
  "steps": [
    {"do": "camera", "position": [0, 0], "zoom": 20},
    {"do": "input", "action": "move_x", "value": 1},
    {"do": "step", "frames": 15},
    {"do": "assert", "that": {"kind": "count", "name": "Coin_*", "op": "eq", "value": 6}},
    {"do": "assert", "that": {"kind": "screenshot", "name": "start"}}
  ]
}

A scenario is (scene, seed, steps) on a deterministic, muted runtime: the same file gives the same state hash. The run stops at the first failed step (fail_fast). A check step that reports a mismatch (golden.check or determinism.check with equal: false, any passed: false) fails the step with E_ASSERT_FAILED.

name and capture names use letters, digits, _, -, . only.

Determinism across builds

A simulation gives the same state hash whatever the optimisation level of the crates it is built from, whichever crate instantiates its generic code (an engine crate, a game crate, a test) and from one run of the process to the next. crates/sparklight-qa/tests/cross_profile.rs proves it: the det_hashes example (3D demos, a car crash, a ragdoll scenario, Lua scripts, a 2D physics scene, the racing netcode scenario with two predicting clients) is built as det0 (every crate at opt-level 0), det3 (every crate at 3), detinv (workspace crates at 3, dependencies at 0) and dev (workspace crates at 0, dependencies at 3: a game crate next to an engine) and must print the same lines, and one binary is run four times. cargo test -p sparklight-qa --test cross_profile -- --ignored runs it (it builds the workspace three times, so a plain cargo test skips it).

What broke it, found by that test (aarch64 macOS, rustc 1.99): f32::sin_cos(), and sin(x) plus cos(x) of one argument. At opt-level 0 the compiler calls sinf and cosf; from opt-level 1 LLVM fuses them into one sincos call (Apple’s __sincosf_stret), which differs in the last bit. rapier3d’s own code uses sin_cos, so its trajectories (pyramid, ragdoll, vehicles on contact) depended on how rapier was compiled: all crates at 0 gave different hashes from all at 3, and rapier3d at 0 with everything else at 3 was enough to reproduce it (nalgebra, parry and simba at 0 were not). sin, cos, tan, atan2, acos, asin, exp, ln, powf, powi, sqrt and hypot alone give the same bits at every level on this machine (200,000 random arguments each); only the fused pair differs. A second, unrelated source: sparklight-net visited its peers in HashMap order, which changes from process to process and with it which peer’s packets reach the simulated link first, so two runs with the same seed saw different loss patterns (the snapshot counts of the clients differed by one between runs of one binary).

The rules that follow:

• Simulation code calls sparklight_core::dmath (angle.dsin_cos(), dsin, dcos, datan2, dpowf, dexp … through the Det trait): the pure-Rust libm crate, which uses only IEEE operations the compiler may not reorder or fuse. sqrt, abs, min, max, floor, round, mul_add and conversions are exact or correctly rounded everywhere and stay f32 methods. The always-running test det_math_sources_use_no_platform_transcendentals fails when a simulation crate (core, engine, physics-fx, scene3d, skeletal, nav, net, particles, cinema, animation, tilemap, script) calls .sin( and its relatives; drawing, window and debug-shape files are listed as exceptions in that test.
• glam is built with its libm feature (vector and quaternion trigonometry) and nalgebra, parry and rapier with simba/libm_force (rapier2d in the engine, rapier3d in physics-fx, rapier in sparklight-3d), so rapier’s sin_cos is the libm one too.
• A HashMap or HashSet whose iteration order can reach a result must be a BTreeMap, a sorted Vec, or an IndexMap; the repeated-run part of the cross-profile test catches the ones that do.

Recorded hashes and goldens changed once, when this landed: physics_pile_after_200_steps (engine), space3d_crates_after_180_updates (scene3d) and the goldens space3d_ragdoll_skin, space3d_ragdoll_wall and cinema_flythrough_blend were re-recorded (the scenes without physics bodies kept their hashes; the new pictures were looked at). Everything recorded before is from a build that used the platform libm.

Cross-platform bit-identity is a separate, stronger promise (macOS against Linux, x86 against ARM; Windows is not verified). It needs the facade feature deterministic3d (see the next section); the default build keeps glam’s SIMD code and rapier’s x86-only flush-to-zero, so its hashes are per machine. Measured with crates/sparklight-qa/examples/det_bench.rs before that feature (CPU time of the thread per space3d.step, median of five runs, five rounds interleaved, all crates at opt-level 3, Apple M5 Max under a load average of about 77, so each number is good to about 5 %): the platform libm took 154-165 / 119-133 / 7.6-8.2 microseconds per step on the pyramid, the ragdoll wall and the car crash; the libm routing of det-math 154-162 / 130-133 / 7.3-8.5; libm plus rapier enhanced-determinism 166-173 / 126-133 / 7.6-8.6. The cost of the whole deterministic3d build is in the next section.

The det3d release gate

compare fails on any mismatch or missing answer. It builds twice because the facade must work with its own feature alone:

crate features  cargo nextest run -p sparklight-scene3d -p sparklight-qa -p sparklight-script \
                  --features sparklight-scene3d/deterministic,sparklight-qa/deterministic,sparklight-script/deterministic -E 'test(/^det3d_/)'
facade feature  cargo nextest run -p sparklight --features scene3d,deterministic3d -E 'test(/^det3d_/)'

Release procedure (the lead, before a tag; the job ids and the verdict go into the release notes):

sha=$(git rev-parse HEAD)                         # a clean, committed tree; fixed before anything runs
  --fetch out/det3d-gate/det3d-answer-x86_64.txt -- scripts/det3d-gate.sh run -- --offline   # the answer lands in .gwdev/out/det3d-gate
                                                  # (an argv is not a shell string: never write --out ".gwdev/out" there)
scripts/det3d-gate.sh run --jobs 4 --out <dir>    # on the Mac: det3d-answer-aarch64.txt (never --workspace on the Mac)
scripts/det3d-gate.sh compare --commit "$sha" <dir>/det3d-answer-x86_64.txt <dir>/det3d-answer-aarch64.txt

An answer file holds the commit, the tree state (and the files that make it dirty), the architecture, every det3d_ test with its status and every det3d <label> <hash> line the tests print. compare exits 0 only when both answers come from $sha on a clean tree, from two different named architectures (x86_64 and aarch64), every test listed in scripts/det3d-gate.tests passed on both (a listed test that did not run, or a det3d_ test that ran and is not listed, fails: a build that lost a feature compiles the tests out), and the hash lines are identical. Exit codes: 0 PASS, 1 FAIL, 3 SKIPPED (an answer missing, empty, unfinished, or from an unknown architecture); 3 is never a pass. scripts/det3d-gate.sh selftest plants a break under each check of compare and expects every one to be caught. Adding a det3d_ test is an edit of scripts/det3d-gate.tests. Re-pinning stays a reviewed edit from runs on both machines; the gate then shows that the two machines agree on the new values.

scripts/det3d-compare.sh <default|deterministic|rapier|glam>... [-- cargo args] [-- scenarios] prints the det3d lines and the qa det_hashes lines of each build variant; run on two machines and diffed, it shows where a divergence starts (it found the causes above). scripts/det3d-bench.sh measures the cost. Re-pinning is a reviewed edit of the pin tables with the reason in the commit message, from runs on both machines; a change that alters the simulation re-pins them like any recorded hash.

The workspace still builds dependencies at opt-level = 3 in dev and test ([profile.dev.package."*"] in Cargo.toml), no longer to make results agree but because unoptimised rapier makes the physics-heavy tests many times slower. Removing it changes no hash.

Steps ("do")

Step	Fields	Does
step	frames	exactly that many fixed steps
advance	seconds	whole steps that fit; the remainder carries over
input	action, value	set an input action (move_x, jump, …) until changed
press	action, frames (1), value (1)	hold the action for frames steps, then set it to 0
release	action	set the action to 0
set / patch / remove / spawn	path+value, path+value (merge patch), path, name+components	edit the world by JSON path, as in docs/agent/runtime.md
camera	position?, zoom?	move or zoom the camera (zoom = pixels per world unit)
reset		back to the loaded state
snapshot / restore	name	in-memory snapshot of the runtime
save_snapshot / load_snapshot	path	snapshot file under <out>/<scenario>/ (relative path, no ..)
until	that, max_frames	step one frame at a time until the assertion holds; fails after max_frames
follow	target, source, offsets, period, min, max, max_frames, until	a bot: before every frame copy the number at source (+ a cycling offset) to target
assert	that	check now

follow is how a scenario plays breakout: paddle x follows ball x with offsets that change every 20 frames (see samples/breakout/scenarios). Frame-by-frame reactions to anything else go through until between inputs (see samples/topdown/scenarios: input move_x 1, until hero x >= 2.5, input move_x 0).

Assertions ("kind")

Kind	Fields	Passes when
world	path, op, value, tolerance	the value at the JSON path compares as op says
count	name (* wildcard), op, value	the number of live entities with a matching name compares
events	filter, op (ge), count (1)	the number of events since the run (or last reset) matching the filter compares
no_errors		no event of level error
frame	op, value	the frame counter compares
hash	equals	the state hash equals the 16 hex digits (same machine and build only)
frame_time	percentile (95), max_ms	that percentile of the measured wall-clock time per frame is within the budget
frame_budget	clock (wall/cpu), p50_ms, p95_ms, p99_ms, worst_ms, skip_frames, min_frames (30)	the median, the 95th and 99th percentile and the worst of the frame times so far are within every limit given; cpu reads thread CPU time (stable on a shared machine); fewer than min_frames frames or no per-thread clock make it skipped. See docs/agent/performance.md
screenshot	name, width (640), height (360), tolerance	the capture matches <goldens>/<scenario>/<name>.png

op: eq ne gt ge lt le approx exists missing contains. Numbers compare by value (1 equals 1.0). Event filter: kind, target, code, min_level, message_contains, fields (all given fields equal). Frame times are wall-clock and machine dependent: give budgets headroom.

Report

{scenario, file, passed, complete, seed, frames, hash, duration_ms, assertions_passed|failed|skipped, steps[], captures[], perf, events, adapter, failure, artifacts[]}.

• passed: no step or assertion failed. complete: nothing was skipped. A screenshot on a machine without a GPU adapter (or with render = false) is skipped, never passed: passed stays true, complete becomes false.
• steps[]: {index, do, ok, frame, error?, assertion?}; assertion has {kind, status, message, expected, actual, code?}.
• captures[]: name, frame, size, pixel hash, golden path, diff (changed and differing pixels, max and mean ΔE), image paths.
• perf: wall-clock ms per simulated frame: mean, p50, p95, p99, max.
• failure: the first failure as {code, message, path: "/steps/<n>"}. E_ASSERT_FAILED for a plain mismatch.
• artifacts of a failed run, under <out>/<scenario>/: world.json (every entity with all components), events.jsonl (last 500 events), failure-frame.png (if a GPU exists), plus <name>.actual.png and <name>.diff.png for a screenshot mismatch.

Suites: run_suite(&[files or folders], &opts) returns SuiteReport {passed, complete, total, failed, scenarios[], load_errors[]}.

Frame capture

use sparklight_scenario::attach_gpu_renderer;

let (adapter, warnings) = attach_gpu_renderer(&mut runtime)?;   // E_RENDERER_UNAVAILABLE without it
runtime.scene.camera.zoom = 20.0;                                // pixels per world unit
let frame = runtime.render_frame(640, 360)?;                     // RGBA8, row 0 at the top, sRGB
println!("{}", sparklight_scenario::frame_hash(&frame));        // digest of size + pixels

• It draws the scene layer with the game window’s sprite renderer (same shader, sRGB target, clear color #0b0d12); the in-game UI overlay is not part of a scenario capture (OffscreenRenderer::capture_with_overlay draws it; see docs/agent/rendering.md for the colour rule, spaces and texture lifetime). Textures load from scene.assets_root; a missing texture is drawn white and listed in warnings.
• Size limit 1..=8192 per side. Camera zoom is pixels per world unit, so 640x360 at zoom 20 shows 32x18 units.
• Lower level: sparklight_render::OffscreenRenderer::{new, capture} takes a camera and sprites and needs no engine.
• Determinism: same machine, same adapter, same scene state give the same pixels. Measured: two renderers and two processes produce equal frame_hashes (tests the_same_scene_gives_the_same_pixels_on_two_devices, runs_are_deterministic_state_and_pixels). A different GPU may differ in the last bit of a channel, which is why goldens have a tolerance; the report names the adapter that produced a run.

Golden images

Goldens are the only images allowed in the repository. A golden lives at <goldens>/<scenario name>/<capture name>.png; goldens defaults to goldens next to the scenario file (samples: samples/<sample>/scenarios/goldens/).

• Compare: check_golden(&frame, golden_path, &Tolerance, GoldenMode::Compare, out_dir, label) or a screenshot assertion. A missing golden is E_GOLDEN_MISSING and the capture is saved as <out>/<scenario>/<name>.actual.png for review; a size change is E_GOLDEN_SIZE.
• Create or change a golden only deliberately: run with GoldenMode::Update (--update-goldens), look at the PNG, commit it.
• Perceptual tolerance (Tolerance, per assertion): a pixel differs when its ΔE (CIE76 in L*a*b*) exceeds pixel_delta_e (default 2.0, about one just-noticeable difference); the images match when at most max_diff_ratio of the pixels differ (default 0.0005) and no pixel is further than max_delta_e (default 60). Tolerance::EXACT demands identical bytes.
• Diff image (<name>.diff.png, written on mismatch): unchanged pixels are the golden dimmed to gray, changed pixels within tolerance dark yellow, differing pixels red (brighter = larger ΔE).

Snapshot files

Runtime::snapshot stays in memory. A snapshot file (SnapshotFile, format sparklight-snapshot version 1) holds:

• the recipe: scene, assets, seed and the driving steps run so far (the same Step values as in scenarios);
• the state hash and frame;
• a state dump: every entity with all components, animator and emitter state (through Animator::export_state and ParticleSystem::export_state), physics bodies, clock, RNG state, camera, input actions.

SnapshotFile::restore() builds a new runtime by replaying the recipe and checks the hash; a mismatch is E_SNAPSHOT_MISMATCH listing the first differing fields of the dump. Why replay and not “pour the dump back”: Lua states, rapier solver caches and entity ids cannot be forged from outside the engine, so the file reconstructs by determinism and uses the dump to prove and explain. Consequences: restoring costs the replay time (a breakout win is 6949 frames, well under a second in release); a file is valid for the same scene files, the same engine build and the same machine class.

Writing: a scenario step (save_snapshot), or in code

let mut recipe = Recipe::new(scene_abs_path, Some(assets_abs_path), seed);
recipe.record(Step::Input { action: "move_x".into(), value: 1.0 });   // record every state-changing call you make
recipe.record(Step::Step { frames: 120 });
runtime.set_action("move_x", 1.0); runtime.step(120);
SnapshotFile::capture(&mut runtime, recipe).write(path)?;
let restored = SnapshotFile::read(path)?.restore()?;                 // a new Runtime at frame 120, hash verified

A caller that edits the runtime without recording a matching step gets E_SNAPSHOT_MISMATCH on restore, not a wrong state.

The control plane has its own file for whole sessions, 2D and 3D (snapshot.save / snapshot.load, format sparklight-control-snapshot version 2: the 2D and the 3D command logs with frames and hashes; docs/agent/space3d.md, Snapshot files). Both files are tested across processes: save at frame 100, load in a new process, run to 300, the hash of the uninterrupted run.

Error codes of this crate

Registered here until the shared registry in sparklight-core takes them over (never renamed):

Code	Meaning
E_SCENARIO_IO	scenario file unreadable or output folder not writable
E_SCENARIO_SCHEMA	the file does not match scenario.schema.json (message lists each JSON path)
E_SCENARIO_STEP	a step cannot run (bad snapshot path, until on a screenshot)
E_ASSERT_FAILED	an assertion found something else than expected
E_GOLDEN_MISSING / E_GOLDEN_MISMATCH / E_GOLDEN_SIZE	golden image absent / differs beyond tolerance / has another size
E_IMAGE_IO	PNG could not be read or written
E_SNAPSHOT_FILE / E_SNAPSHOT_MISMATCH	snapshot file unreadable or wrong format / replay did not reach the stored hash

Engine errors (E_PATH_NOT_FOUND, E_RENDERER_UNAVAILABLE, …) pass through unchanged.

Samples

Scenario	Plays	Frames	Checks
samples/platformer/scenarios/platformer-collects-coins	climbs the platform staircase, jumps the pit	531	six coins gone, victory log, no hits, no errors, 2 screenshots
samples/breakout/scenarios/breakout-clears-bricks	paddle bot with follow	6949	50 bricks destroyed, no defeat, 2 screenshots
samples/topdown/scenarios/topdown-collects-items	waypoint walk with until	1452	eight gems collected, victory log, 2 screenshots

Run them all: cargo run -q --release -p sparklight-scenario --example run_scenarios -- samples/platformer/scenarios samples/breakout/scenarios samples/topdown/scenarios. The scenario runner reports what it did not do: without a GPU the six screenshots are reported skipped and complete is false. A test that must have them is a gpu_ test (see “Tests that need a GPU, a display or a secret”).

Writing a new scenario

1. Load the scene in a Runtime (see docs/agent/runtime.md) and find a route with step, set_action and get: print the hero position every few frames.
2. Express the route as input/press/step steps, or until on a position when exact frame counts are fragile.
3. Add checkpoints (count, world) so a failure points at the step where the run left the route.
4. Finish with state assertions, no_errors, a frame_time budget with headroom and screenshots.
5. Create the goldens with --update-goldens, open the PNGs, commit them with the scenario.