Skip to content

Testing and reproducibility ​

Turn the counter tutorial into a regression test before adding a renderer or a remote client. A useful test checks game behavior and repeatability: two equally wrong runs can still have matching checksums.

Prerequisite: the counter's complete src/main.rs and manifest, or a repository checkout. No server or GPU is needed.

Run the packaged tests ​

From the repository:

sh
cargo test -p bevy_agent_runner --example counter --locked

Expected: 2 passed; 0 failed. The tests ship with the runner's counter example. If you copied the tutorial into your own project, append the following module to src/main.rs, then run cargo test:

rs
#[cfg(test)]
mod tests {
    use super::*;

    #[test]
    fn reset_repeats_the_same_episode() -> anyhow::Result<()> {
        let mut env = AgentApp::new(build_app)?;
        let mut runs = Vec::new();
        for _ in 0..2 {
            env.reset(ResetOptions {
                seed: Some(42),
                ..Default::default()
            })?;
            let mut checksums = Vec::new();
            for expected in 1..=8 {
                let step = env.step(AgentAction::Noop)?;
                assert_eq!(step.tick, expected);
                assert_eq!(env.world().resource::<Counter>().0, expected);
                checksums.push(step.checksum);
            }
            runs.push(checksums);
        }
        assert_eq!(runs[0], runs[1]);
        Ok(())
    }

    #[test]
    fn restored_history_repeats_the_next_transition() -> anyhow::Result<()> {
        let mut env = AgentApp::new(build_app)?;
        env.reset(ResetOptions::default())?;
        env.step(AgentAction::Noop)?;
        let saved = env.snapshot()?;
        let expected = env.step(AgentAction::Noop)?;
        env.step(AgentAction::Noop)?;

        env.restore(saved.snapshot_id)?;
        assert_eq!(env.world().resource::<Counter>().0, 1);
        let repeated = env.step(AgentAction::Noop)?;
        assert_eq!(expected.checksum, repeated.checksum);

        env.restore_tick(1)?;
        assert_eq!(env.world().resource::<Counter>().0, 1);
        let replayed = env.step(AgentAction::Noop)?;
        assert_eq!(expected.checksum, replayed.checksum);
        Ok(())
    }
}

What each test proves ​

TestFailure it can reveal
Reset and run the same eight actions twiceState left over from the previous episode, tick drift, inconsistent state hashes
Assert the counter at each tickAn incorrect increment, skipped gameplay, or an unexpected extra tick
Restore and repeat the next actionMissing snapshot state that affects the next transition
Reconstruct tick 1 and repeatHistory reconstruction disagreeing with the original transition

The counter has no random gameplay. The seed is explicit to show where a randomized game controls its RNG; this example does not test random distributions or cross-platform behavior.

Adapt this to your game ​

  1. Replace build_app with your headless app builder and Noop with a short legal action sequence.
  2. Assert an observable game outcome: position, score, inventory, a collision, or an episode ending. Include boundary cases, not only successful movement.
  3. Compare checksums at every completed tick to locate the first divergence.
  4. Save just before an interesting action. Restore, repeat it, and compare both the result and hidden state that affects future play.
  5. Repeat with several fixed seeds when your game uses randomness. Keep a failing seed and action sequence as a regression case.

Stop an action sequence when done or truncated is true. Test terminal behavior separately; a game need not accept another action after the episode ends.

Diagnose a mismatch ​

The game assertion fails but hashes match: both runs are consistently wrong. Inspect the gameplay system and action semantics.

Reset runs disagree: look for wall-clock time, unseeded randomness, unordered systems, entity iteration order, or state that reset did not initialize.

Reset runs agree but restore diverges: inspect snapshot registration first. Hidden state such as a last movement direction, cooldown, RNG state, or required resource often explains the mismatch.

Checksums match while visible behavior differs: the checksum may omit authoritative fields, or presentation may be reading stale state. A checksum only covers what your extractor includes.

Reproduce a remote failure ​

Save the environment/version, seed, action sequence, first failing tick, and returned error or observation. A portable replay bundle includes the checkpoints required to reconstruct recorded history. Keep it with the matching game schema and compatible runtime.

Test the same sequence through HTTP and the CLI after local simulation tests pass. Transport validation covers authentication, serialization, capabilities, and uncertain mutation outcomes; it complements the gameplay assertions above.

Released under MIT or Apache-2.0.