-
Notifications
You must be signed in to change notification settings - Fork 40
Open
Description
Normal fuzz tests check that a fuzzer passes the test every time (∀x: p(x)).
Sometimes it's helpful to test whether a fuzzer passes the test at least once (∃x: p(x)).
API sketch (with bad naming):
Test.fuzz : Fuzzer a -> String -> (a -> Expectation) -> Test
Test.fuzzCanPass : Fuzzer a -> String -> (a -> Expectation) -> TestThis can be inefficiently done in userspace like this:
{-| Will run the fuzzer up to 100 times and check if it passes the test at least once.
-}
canPass : Fuzzer a -> (a -> Bool) -> Expectation
canPass fuzzer pred =
let
tries =
100
go n nLeft =
if nLeft <= 0 then
Expect.fail <| "Expected the fuzzer to pass the test at least once out of " ++ String.fromInt tries ++ " tries."
else
let
seenItPass =
Fuzz.examples n fuzzer
|> List.any pred
in
if seenItPass then
Expect.pass
else
go (n + 1) (nLeft - n)
in
go 1 triessince the exposed API lacks the ability to run a single fuzzer with a varying seed. And it's kinda abusing Fuzz.examples.
The library would be able to stop fuzzing as soon as it sees a pass; the userspace version generates in chunks and likely throws some of that work away after it finds a passing value.
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels