snapshots, property and simulation testing

Your suite passes. Coverage is 90%. You deploy. It breaks.

Coverage counted lines for me, not bugs. Unit tests checked pieces on their own, not how they behaved together, and they only caught the cases I had already thought of. The best way to avoid a bug is to not write it, but I keep writing them, so I want tests that actually find them.

Most people treat testing as assertions on business rules. That is a small part of it. As I said in this is ci, correctness needs validating too, and assertions plus the three methods below are how I do it.

Unit tests are not enough

The pyramid puts unit tests at the base. I think that is backwards. Isolated tests missed the interactions that broke things in production for me. Mock-heavy tests checked my mocks. I have sat at 100% coverage and still shipped a correctness bug, which is the worst version of it, because the number told me I was safe.

matklad says it better than I can in unit and integration tests: unit tests are fast and they miss what matters. His how to test is the methodology I follow. Test the right thing at the right level instead of testing everything at the smallest one. Testing on the Toilet had the culture right; the wrong kind of testing still buys you false confidence. Google’s risk-driven testing comes at it from the other side: spend the effort where being wrong costs the most.

Correctness is hard

By correctness I mean the program cannot reach an invalid state, whatever the input, whatever the path. Any piece of state can be corrupted, overflowed, or misused.

Where I have seen the violations come from:

• type and data misuse: a u8 wrapping from 255 back to 0, a string treated as a number
• out-of-bounds access: reads and writes past an array boundary, undefined behavior, sometimes a security hole
• reference safety: dangling or null references, crashes, quiet corruption. Rust puts rails around this one

What I defend with:

• invariants: properties that must hold at a given point. “no balance is ever negative”. “a reference never points to freed memory”. Break one and you are in an impossible state
• types: they try to make invalid states unrepresentable, but most type systems are not strong enough. An int will not stop a division by zero or keep an index in range
• assertions: the runtime checks that carry the invariants your types cannot express

The broader version of this is error handling: make the bad state unreachable instead of catching it after it happens.

Snapshot, property and simulation testing make those violations visible.

Snapshot testing

A snapshot test captures the output of something and compares it against a stored copy. I find it best on API contracts, using insta in Rust.

The problem it solved for me was maintenance. Change a response format and every test that spells out the shape breaks, whether or not the change was intentional:

#[test]
fn test_user_api_traditional() {
    let response = get_user(123);
    assert_eq!(response.status, 200);
    // Manual maintenance: update this when API adds fields
    assert_eq!(response.data.name, "John");
    assert_eq!(response.data.email, "john@example.com");
    // What happens when API adds response.data.last_login?
}

Capture the whole thing instead:

#[test]
fn test_user_api_as_snapshot() {
    let response = get_user(123);
    assert_eq!(response.status, 200);

    // Automatically detects any changes to the response structure
    insta::assert_snapshot!(response.data);
}

The tool shows you the diff and you accept or reject it. That makes API evolution explicit instead of accidental. Jane Street calls this testing with expectations and runs it on everything up to hardware designs.

Property testing

A property test states something that should always hold, then throws generated inputs at it until it does not. proptest is the one I reach for.

My hand-written tests only ever covered the cases I thought of. The payment code was fine under $1000. What about 2,147,483,647?

#[cfg(test)]
mod tests {
    use proptest::prelude::*;

    proptest! {
        #[test]
        fn test_payment_amount_properties(
            amount in 0..i32::MAX
        ) {
            let payment = Payment::new(amount);

            // Property: payment amount should never be negative
            prop_assert!(payment.amount() >= 0);

            // Property: payment should handle edge cases without overflow
            let doubled = payment.amount() * 2;
            prop_assert!(doubled >= payment.amount());

            // This will catch integer overflow with large numbers
            // like 2,147,483,647 (i32::MAX)
        }
    }
}

That found overflow, and array sizes I never tried, and combinations I would never have written by hand. Look at real bugs and most of them need a specific combination before they show at all. Pair it with fuzzing when the input space is wide.

Simulation testing

Some bugs only showed up in a particular sequence of operations, and a property test on one call never found them. So you build a model, generate random sequences, run them against both the model and the real thing, and check the invariants after every step. Phil Eaton’s deterministic simulation testing is the best introduction I know.

#[cfg(test)]
mod tests {
    use proptest::prelude::*;

    proptest! {
        #[test]
        fn test_bounded_array_simulation(
            operations in prop::collection::vec(any::<Operation>(), 0..1000)
        ) {
            let mut array = BoundedArray::new(100);

            for op in operations {
                match op {
                    Operation::Push(item) => {
                        if !array.full() {
                            array.push(item);
                            // Property: count never exceeds capacity
                            prop_assert!(array.count() <= array.capacity());
                        }
                    }
                    Operation::Pop => {
                        if !array.empty() {
                            array.pop();
                            // Property: count never goes negative
                            prop_assert!(array.count() >= 0);
                        }
                    }
                }
            }
        }
    }
}

It also sidesteps the database teardown problem. You exercise real interactions without rebuilding the world between every test.

TigerBeetle runs this against their database invariants under sequences no human would write, and their descent into the vortex shows how far it goes. A simulation can run for hours and cover millions of combinations, and it is the only thing here that has ever found me an ordering bug.

Putting it together

• snapshot tests catch changes to contracts and to output
• property tests catch broken invariants, preconditions and postconditions
• simulation tests walk a tree of states you would never enumerate

None of it worked for me without assertions underneath:

pub fn push(array: *BoundedArray, item: T) void {
    assert(!array.full());  // Precondition
    array.buffer[array.count_u32] = item;
    array.count_u32 += 1;
    assert(array.count() == array.count_u32);  // Postcondition
}

Assertions make impossible states impossible. Tests make changes visible. CI makes both somebody’s problem before it becomes a user’s problem.

I still ship bugs. Fewer of them are correctness bugs.