Skip to content

Allow testing ∃x: p(x): a test passes at least once out of N tries #235

@Janiczek

Description

@Janiczek

Normal fuzz tests check that a fuzzer passes the test every time (∀x: p(x)).
Sometimes it's helpful to test whether a fuzzer passes the test at least once (∃x: p(x)).

API sketch (with bad naming):

Test.fuzz        : Fuzzer a -> String -> (a -> Expectation) -> Test
Test.fuzzCanPass : Fuzzer a -> String -> (a -> Expectation) -> Test

This can be inefficiently done in userspace like this:

{-| Will run the fuzzer up to 100 times and check if it passes the test at least once.
-}
canPass : Fuzzer a -> (a -> Bool) -> Expectation
canPass fuzzer pred =
    let
        tries =
            100

        go n nLeft =
            if nLeft <= 0 then
                Expect.fail <| "Expected the fuzzer to pass the test at least once out of " ++ String.fromInt tries ++ " tries."

            else
                let
                    seenItPass =
                        Fuzz.examples n fuzzer
                            |> List.any pred
                in
                if seenItPass then
                    Expect.pass

                else
                    go (n + 1) (nLeft - n)
    in
    go 1 tries

since the exposed API lacks the ability to run a single fuzzer with a varying seed. And it's kinda abusing Fuzz.examples.

The library would be able to stop fuzzing as soon as it sees a pass; the userspace version generates in chunks and likely throws some of that work away after it finds a passing value.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions