Testing and reproducibility
Turn the counter tutorial into a regression test before adding a renderer or a remote client. A useful test checks game behavior and repeatability: two equally wrong runs can still have matching checksums.
Prerequisite: the counter's complete src/main.rs and manifest, or a repository checkout. No server or GPU is needed.
Run the packaged tests
From the repository:
cargo test -p bevy_agent_runner --example counter --lockedExpected: 2 passed; 0 failed. The tests ship with the runner's counter example. If you copied the tutorial into your own project, append the following module to src/main.rs, then run cargo test:
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn reset_repeats_the_same_episode() -> anyhow::Result<()> {
let mut env = AgentApp::new(build_app)?;
let mut runs = Vec::new();
for _ in 0..2 {
env.reset(ResetOptions {
seed: Some(42),
..Default::default()
})?;
let mut checksums = Vec::new();
for expected in 1..=8 {
let step = env.step(AgentAction::Noop)?;
assert_eq!(step.tick, expected);
assert_eq!(env.world().resource::<Counter>().0, expected);
checksums.push(step.checksum);
}
runs.push(checksums);
}
assert_eq!(runs[0], runs[1]);
Ok(())
}
#[test]
fn restored_history_repeats_the_next_transition() -> anyhow::Result<()> {
let mut env = AgentApp::new(build_app)?;
env.reset(ResetOptions::default())?;
env.step(AgentAction::Noop)?;
let saved = env.snapshot()?;
let expected = env.step(AgentAction::Noop)?;
env.step(AgentAction::Noop)?;
env.restore(saved.snapshot_id)?;
assert_eq!(env.world().resource::<Counter>().0, 1);
let repeated = env.step(AgentAction::Noop)?;
assert_eq!(expected.checksum, repeated.checksum);
env.restore_tick(1)?;
assert_eq!(env.world().resource::<Counter>().0, 1);
let replayed = env.step(AgentAction::Noop)?;
assert_eq!(expected.checksum, replayed.checksum);
Ok(())
}
}What each test proves
| Test | Failure it can reveal |
|---|---|
| Reset and run the same eight actions twice | State left over from the previous episode, tick drift, inconsistent state hashes |
| Assert the counter at each tick | An incorrect increment, skipped gameplay, or an unexpected extra tick |
| Restore and repeat the next action | Missing snapshot state that affects the next transition |
| Reconstruct tick 1 and repeat | History reconstruction disagreeing with the original transition |
The counter has no random gameplay. The seed is explicit to show where a randomized game controls its RNG; this example does not test random distributions or cross-platform behavior.
Adapt this to your game
- Replace
build_appwith your headless app builder andNoopwith a short legal action sequence. - Assert an observable game outcome: position, score, inventory, a collision, or an episode ending. Include boundary cases, not only successful movement.
- Compare checksums at every completed tick to locate the first divergence.
- Save just before an interesting action. Restore, repeat it, and compare both the result and hidden state that affects future play.
- Repeat with several fixed seeds when your game uses randomness. Keep a failing seed and action sequence as a regression case.
Stop an action sequence when done or truncated is true. Test terminal behavior separately; a game need not accept another action after the episode ends.
Diagnose a mismatch
The game assertion fails but hashes match: both runs are consistently wrong. Inspect the gameplay system and action semantics.
Reset runs disagree: look for wall-clock time, unseeded randomness, unordered systems, entity iteration order, or state that reset did not initialize.
Reset runs agree but restore diverges: inspect snapshot registration first. Hidden state such as a last movement direction, cooldown, RNG state, or required resource often explains the mismatch.
Checksums match while visible behavior differs: the checksum may omit authoritative fields, or presentation may be reading stale state. A checksum only covers what your extractor includes.
Reproduce a remote failure
Save the environment/version, seed, action sequence, first failing tick, and returned error or observation. A portable replay bundle includes the checkpoints required to reconstruct recorded history. Keep it with the matching game schema and compatible runtime.
Test the same sequence through HTTP and the CLI after local simulation tests pass. Transport validation covers authentication, serialization, capabilities, and uncertain mutation outcomes; it complements the gameplay assertions above.