Test failures in test_variants_decompress_into with recent hypothesis versions

[musicinmybrain]

Since the Rust toolchain was updated to 1.84 in Fedora, the python-cramjam package has been failing to build from source due to test failures in test_variants_decompress_into. It’s consistent that some parameterizations of this test fail, but the particular ones that fail vary from run to run. The problem still occurs if I update to a snapshot of 61564e7 and ensure experimental codecs are disabled.

For example:

=================================== FAILURES ===================================
___________ test_variants_decompress_into[snappy-Buffer-memoryview] ____________
  + Exception Group Traceback (most recent call last):
  |   File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 341, in from_call
  |     result: TResult | None = func()
  |                              ~~~~^^
[… many lines omitted for brevity … ]
  | hypothesis.errors.FlakyFailure: Hypothesis test_variants_decompress_into(variant_str='snappy', input_type=memoryview, output_type=Buffer, tmp_path_factory=TempPathFactory(_given_basetemp=None,
  |  _trace=<pluggy._tracing.TagTracerSub object at 0x7f0cbaabed70>,
  |  _basetemp=PosixPath('/tmp/pytest-of-mockbuild/pytest-0'),
  |  _retention_count=3,
  |  _retention_policy='all'), raw_data=b'\xd0', is_pypy=False) produces unreliable results: Falsified on the first call but did not on a subsequent one (1 sub-exception)
  | Falsifying example: test_variants_decompress_into(
  |     variant_str='snappy',
  |     input_type=memoryview,
  |     output_type=Buffer,
  |     tmp_path_factory=TempPathFactory(_given_basetemp=None,
  |      _trace=<pluggy._tracing.TagTracerSub object at 0x7f0cbaabed70>,
  |      _basetemp=PosixPath('/tmp/pytest-of-mockbuild/pytest-0'),
  |      _retention_count=3,
  |      _retention_policy='all'),
  |     is_pypy=False,
  |     raw_data=b'\xd0',
  | )
  | Failed to reproduce exception. Expected:
  | variant_str = 'snappy', input_type = <class 'memoryview'>
  | output_type = <class 'builtins.Buffer'>
  | tmp_path_factory = TempPathFactory(_given_basetemp=None, _trace=<pluggy._tracing.TagTracerSub object at 0x7f0cbaabed70>, _basetemp=PosixPath('/tmp/pytest-of-mockbuild/pytest-0'), _retention_count=3, _retention_policy='all')
  | raw_data = b'0', is_pypy = False
[… many lines omitted for brevity … ]
  | >       assert same_same(output, raw_data)
  | E       AssertionError: assert False   
  | E        +  where False = same_same(b'\xd0', b'0')
  |
  | tests/test_variants.py:231: AssertionError
  |
  +-+---------------- 1 ----------------   
    | AssertionError: assert False
    |  +  where False = same_same(b'\xd0', b'0')
    +------------------------------------  
[… many lines omitted for brevity … ]
=========================== short test summary info ============================
FAILED tests/test_variants.py::test_variants_decompress_into[snappy-Buffer-memoryview]
FAILED tests/test_variants.py::test_variants_decompress_into[lz4-Buffer-Buffer]
FAILED tests/test_variants.py::test_variants_decompress_into[zlib-File-memoryview]
================== 3 failed, 719 passed, 1 skipped in 19.04s ===================

I refrained from immediately filing an issue upstream because I couldn’t reproduce the problem in a git checkout, but something changed at some point, and now I can reproduce it, or at least something similar. Working on Fedora 41:

$ git clone https://github.com/milesgranger/cramjam.git
$ cd cramjam
$ uv venv _e
$ . _e/bin/activate
(_e) $ uv pip install -e .[dev]
(_e) $ python -m pytest --ignore=benchmarks/test_bench.py -n auto
[…]
INTERNALERROR>         self, node: WorkerController, formatted_error: str
INTERNALERROR>     ) -> None:
INTERNALERROR>         """
INTERNALERROR>         pytest_internalerror() was called on the worker.
INTERNALERROR>
INTERNALERROR>         pytest_internalerror() arguments are an excinfo and an excrepr, which can't
INTERNALERROR>         be serialized, so we go with a poor man's solution of raising an exception
INTERNALERROR>         here ourselves using the formatted message.
INTERNALERROR>         """
INTERNALERROR>         self._active_nodes.remove(node)
INTERNALERROR>         try:
INTERNALERROR> >           assert False, formatted_error
INTERNALERROR> E           AssertionError: Traceback (most recent call last):
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR> E                 session.exitstatus = doit(config, session) or 0
INTERNALERROR> E                                      ~~~~^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR> E                 config.hook.pytest_runtestloop(session=session)
INTERNALERROR> E                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E                        ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E                        ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/logging.py", line 803, in pytest_runtestloop
INTERNALERROR> E                 return (yield)  # Run all the tests.
INTERNALERROR> E                         ^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/terminal.py", line 673, in pytest_runtestloop
INTERNALERROR> E                 result = yield
INTERNALERROR> E                          ^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/xdist/remote.py", line 174, in pytest_runtestloop
INTERNALERROR> E                 self.run_one_test()
INTERNALERROR> E                 ~~~~~~~~~~~~~~~~~^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/xdist/remote.py", line 195, in run_one_test
INTERNALERROR> E                 self.config.hook.pytest_runtest_protocol(item=item, nextitem=nextitem)
INTERNALERROR> E                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E                        ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E                        ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR> E                 raise exception.with_traceback(exception.__traceback__)
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/warnings.py", line 112, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E                         ^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/assertion/__init__.py", line 176, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E                         ^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/unittest.py", line 429, in pytest_runtest_protocol
INTERNALERROR> E                 res = yield
INTERNALERROR> E                       ^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR> E                 teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR> E                 ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/faulthandler.py", line 88, in pytest_runtest_protocol
INTERNALERROR> E                 return (yield)
INTERNALERROR> E                         ^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR> E                 res = hook_impl.function(*args)
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/runner.py", line 113, in pytest_runtest_protocol
INTERNALERROR> E                 runtestprotocol(item, nextitem=nextitem)
INTERNALERROR> E                 ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/runner.py", line 137, in runtestprotocol
INTERNALERROR> E                 reports.append(call_and_report(item, "teardown", log, nextitem=nextitem))
INTERNALERROR> E                                ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/runner.py", line 244, in call_and_report
INTERNALERROR> E                 report: TestReport = ihook.pytest_runtest_makereport(item=item, call=call)
INTERNALERROR> E                                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR> E                 return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
INTERNALERROR> E                        ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR> E                 return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR> E                        ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 156, in _multicall
INTERNALERROR> E                 teardown[0].send(outcome)
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_hypothesis_pytestplugin.py", line 368, in pytest_runtest_makereport
INTERNALERROR> E                 from hypothesis.extra._patching import FAIL_MSG, get_patch_for  
INTERNALERROR> E               File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
INTERNALERROR> E               File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
INTERNALERROR> E               File "<frozen importlib._bootstrap>", line 935, in _load_unlocked 
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/assertion/rewrite.py", line 175, in exec_module
INTERNALERROR> E                 source_stat, co = _rewrite_test(fn, self.config)
INTERNALERROR> E                                   ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/assertion/rewrite.py", line 355, in _rewrite_test
INTERNALERROR> E                 tree = ast.parse(source, filename=strfn)
INTERNALERROR> E               File "/usr/lib64/python3.13/ast.py", line 54, in parse
INTERNALERROR> E                 return compile(source, filename, mode, flags,
INTERNALERROR> E                                _feature_version=feature_version, optimize=optimize)
INTERNALERROR> E               File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/hypothesis/extra/_patching.py", line 167
INTERNALERROR> E                 the_call = node.body[0].body[0].value
INTERNALERROR> E             SyntaxError: could not convert string to float: '�' - Consider hexadecimal for huge integer literals to avoid decimal conversion limits.
INTERNALERROR> E           assert False
INTERNALERROR>
INTERNALERROR> _e/lib64/python3.13/site-packages/xdist/dsession.py:232: AssertionError
..INTERNALERROR> Traceback (most recent call last):
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/main.py", line 283, in wrap_session
INTERNALERROR>     session.exitstatus = doit(config, session) or 0
INTERNALERROR>                          ~~~~^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/main.py", line 337, in _main
INTERNALERROR>     config.hook.pytest_runtestloop(session=session)
INTERNALERROR>     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_hooks.py", line 513, in __call__
INTERNALERROR>     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult) 
INTERNALERROR>            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
INTERNALERROR>     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
INTERNALERROR>            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
INTERNALERROR>     raise exception.with_traceback(exception.__traceback__)
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/logging.py", line 803, in pytest_runtestloop
INTERNALERROR>     return (yield)  # Run all the tests.
INTERNALERROR>             ^^^^^
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 122, in _multicall
INTERNALERROR>     teardown.throw(exception)  # type: ignore[union-attr]
INTERNALERROR>     ~~~~~~~~~~~~~~^^^^^^^^^^^
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/_pytest/terminal.py", line 673, in pytest_runtestloop
INTERNALERROR>     result = yield
INTERNALERROR>              ^^^^^
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pluggy/_callers.py", line 103, in _multicall
INTERNALERROR>     res = hook_impl.function(*args)
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/xdist/dsession.py", line 138, in pytest_runtestloop
INTERNALERROR>     self.loop_once()
INTERNALERROR>     ~~~~~~~~~~~~~~^^
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/xdist/dsession.py", line 163, in loop_once
INTERNALERROR>     call(**kwargs)
INTERNALERROR>     ~~~~^^^^^^^^^^
INTERNALERROR>   File "/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/xdist/dsession.py", line 217, in worker_workerfinished
INTERNALERROR>     assert not crashitem, (crashitem, node)
INTERNALERROR> AssertionError: ('tests/test_variants.py::test_variants_decompress_into[zstd-Buffer-bytearray]', <WorkerController gw13>)
INTERNALERROR> assert not 'tests/test_variants.py::test_variants_decompress_into[zstd-Buffer-bytearray]'

============================================================================= 1 failed, 595 passed, 1 skipped, 1 warning in 5.36s =============================================================================

Ok, that’s not exactly the same, but it looks like it could be the same root cause presenting differently due to slightly different versions of pytest, hypothesis, etc.

Repeating the same command shows the failure is flaky:

$ python -m pytest --ignore=benchmarks/test_bench.py -n auto
/home/ben/src/forks/cramjam/_e/lib64/python3.13/site-packages/pytest_benchmark/logger.py:39: PytestBenchmarkWarning: Benchmarks are automatically disabled because xdist plugin is active.Benchmarks cannot be performed reliably in a parallelized environment.
  warner(PytestBenchmarkWarning(text))
============================================================================================= test session starts =============================================================================================
platform linux -- Python 3.13.1, pytest-8.3.4, pluggy-1.5.0
benchmark: 5.1.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/ben/src/forks/cramjam
configfile: pyproject.toml
plugins: hypothesis-6.124.9, xdist-3.6.1, benchmark-5.1.0
16 workers [722 items]   ipped                                                                                                                                                                                
....................................................................................................................................................................................................... [ 27%]
....................................................................................................................................................................................................... [ 55%]
....................................................................................................................................................................................................... [ 82%]
.............................................................................................................................                                                                           [100%]
======================================================================================= 722 passed, 1 skipped in 3.36s ========================================================================================

It seems I got “lucky” catching this in the virtualenv, because I didn’t see any more failures in 10-20 subsequent attempts. I don’t know why the problem is so much easier to reproduce in the Fedora packages – compiler flags? dependency versions? rustc patches? release vs. debug builds?

I’m happy to do any experiments or provide any data that would help.

I’m at my wits’ end with these test failures. I am concerned that they may reflect possible flaky/racy data corruption in python-cramjam package in Fedora, but I have no idea how to find the root cause. If I can’t find a way to resolve the test failures, I’m reluctantly considering orphaning python-cramjam and related packages sometime well before the Fedora 42 final freeze, which would probably lead to these packages being retired from the distribution.

[milesgranger]

thanks for this, I can understand your frustration and wouldn't blame you for orphaning the cramjam project. As I mentioned before, I've also had much less bandwidth to deal with maintain this project and trying in my limited capacity to reduce it's maintenance burden by removing the experimental modules, and I also want to remove hypothesis since it's shown in the past some nondeterministic problems and almost unsurprisingly in the mix here.

Is it also reproducible if the xdist isn't used? Sometimes I think that xdist with hypothesis does weird things.

[musicinmybrain]

thanks for this, I can understand your frustration and wouldn't blame you for orphaning the cramjam project. As I mentioned before, I've also had much less bandwidth to deal with maintain this project and trying in my limited capacity to reduce it's maintenance burden by removing the experimental modules, and I also want to remove hypothesis since it's shown in the past some nondeterministic problems and almost unsurprisingly in the mix here.

What I’m concerned about is that hypothesis may be successfully revealing nondeterministic problems like race conditions in python-cramjam. If that’s true, removing the hypothesis tests may wallpaper over the problem, but may leave users of the Fedora package open to data corruption in production. I can’t easily prove that’s the case, but the possibility worries me enough that I’m not comfortable just skipping test_variants_decompress_into.

Is it also reproducible if the xdist isn't used? Sometimes I think that xdist with hypothesis does weird things.

That’s a reasonable idea. I had already tried removing -n auto from the pytest invocation without any effect, but I just went back and tried removing the pytest-xdist dependency entirely so the xdist plugin wouldn’t even be loaded. The result (based on a snapshot of 61564e7) was still the same:

=========================== short test summary info ============================
FAILED tests/test_variants.py::test_variants_decompress_into[brotli-numpy-File]
FAILED tests/test_variants.py::test_variants_decompress_into[xz-File-File] - ...
================== 2 failed, 720 passed, 1 skipped in 18.67s ===================

[milesgranger]

Ya, I find myself agreeing with the removal of hypothesis probably not good.
I'm reminded of a weird effect I found when working with the PyPy stuff (pypy/pypy#4918 (comment)), excerpt here:

If I try to maintain this behavior it kinda works. Very strange behavior in that when bytes is used, on this line in the tests, output will remain the original value. (For example b'0' * 23) But when debugging it'll quickly change to the expected value (ie b'\xff\x06\x00\x00sNaPpY\x01\x05\x00\x00\xd2\x8f%I\x00').
Even more entertainment can be discovered if I adapt the test to use a while loop which keeps checking the assertion until eventually output is updated to the expected value.
But okay, that's maybe an issue in and of itself for PyPy as this currently works in CPython.. something strange going on there but I'm less interested in going down that route since it's the wrong thing to do anyhow.

As a last debugging interest, if you have interactivity to these, setting --pdb and checking if the values are in fact remaining different or after a slight delay the variable actually updates to the expected value. But given this isn't happening on bytes exclusively that probably isn't the case unless that more reliably exposes some deeper problem when using PyPy.

[milesgranger]

And then, the parameters failing do they always include either the Buffer or File objects as input/output? That might be interesting as those are the only two implemented on the Rust side. From your list of failing tests, it seems like this might be the case.

[musicinmybrain]

And then, the parameters failing do they always include either the Buffer or File objects as input/output? That might be interesting as those are the only two implemented on the Rust side. From your list of failing tests, it seems like this might be the case.

Unfortunately, no. I haven’t detected any pattern. Here are the results of a few successive runs on my workstation (still without pytest-xdist, not that it seems to make a significant difference):

=========================== short test summary info ============================
FAILED tests/test_variants.py::test_variants_decompress_into[xz-numpy-bytearray]
================== 1 failed, 721 passed, 1 skipped in 17.29s ===================

=========================== short test summary info ============================
FAILED tests/test_variants.py::test_variants_decompress_into[zstd-File-bytes]
FAILED tests/test_variants.py::test_variants_decompress_into[zlib-numpy-numpy]
================== 2 failed, 720 passed, 1 skipped in 17.49s ===================

======================= 722 passed, 1 skipped in 17.37s ========================

=========================== short test summary info ============================
FAILED tests/test_variants.py::test_variants_decompress_into[zlib-Buffer-Buffer]
================== 1 failed, 721 passed, 1 skipped in 17.28s ===================

=========================== short test summary info ============================
FAILED tests/test_variants.py::test_variants_decompress_into[brotli-File-bytearray]
================== 1 failed, 721 passed, 1 skipped in 18.64s ===================

As you can see, the problems really seem to be all over the place with respect to all three parameters.

As a last debugging interest, if you have interactivity to these, setting --pdb and checking if the values are in fact remaining different or after a slight delay the variable actually updates to the expected value. But given this isn't happening on bytes exclusively that probably isn't the case unless that more reliably exposes some deeper problem when using PyPy.

I can try running the tests manually in the chroot with --pdb. I’ll see if I can figure out what to do once I get there.

[musicinmybrain]

Just some notes on reproducing this.

I tried running the tests manually in the mock chroot that was left after a build in which the tests had failed:

<mock-chroot> sh-5.2# cd /builddir/build/BUILD/python-cramjam-2.10.0_20250104git61564e7-build/cramjam-61564e7761e38e5ec55e7939ccd6a276c2c55d11
<mock-chroot> sh-5.2# PYTHONPATH=/builddir/build/BUILD/python-cramjam-2.10.0_20250104git61564e7-build/BUILDROOT/usr/lib64/python3.13/site-packages/ python3 -m pytest --ignore=benchmarks/test_bench.py -v

… and I got a larger number of failures, different from the ones in the original build:

=========================================================================================== short test summary info ===========================================================================================
FAILED tests/test_variants.py::test_variants_decompress_into[snappy-numpy-Buffer] - hypothesis.errors.FlakyFailure: Hypothesis test_variants_decompress_into(variant_str='snappy', input_type=Buffer, output_type='numpy', tmp_path_factory=TempPathFactory(_given_basetemp=None,
FAILED tests/test_variants.py::test_variants_decompress_into[snappy-Buffer-Buffer] - hypothesis.errors.FlakyFailure: Hypothesis test_variants_decompress_into(variant_str='snappy', input_type=Buffer, output_type=Buffer, tmp_path_factory=TempPathFactory(_given_basetemp=None,
FAILED tests/test_variants.py::test_variants_decompress_into[snappy-Buffer-File] - hypothesis.errors.FlakyFailure: Hypothesis test_variants_decompress_into(variant_str='snappy', input_type=File, output_type=Buffer, tmp_path_factory=TempPathFactory(_given_basetemp=None,
FAILED tests/test_variants.py::test_variants_decompress_into[brotli-File-Buffer] - hypothesis.errors.FlakyFailure: Hypothesis test_variants_decompress_into(variant_str='brotli', input_type=Buffer, output_type=File, tmp_path_factory=TempPathFactory(_given_basetemp=None,
FAILED tests/test_variants.py::test_variants_decompress_into[bzip2-File-numpy] - hypothesis.errors.FlakyFailure: Hypothesis test_variants_decompress_into(variant_str='bzip2', input_type='numpy', output_type=File, tmp_path_factory=TempPathFactory(_given_basetemp=None,
FAILED tests/test_variants.py::test_variants_decompress_into[zstd-File-bytes] - hypothesis.errors.FlakyFailure: Hypothesis test_variants_decompress_into(variant_str='zstd', input_type=bytes, output_type=File, tmp_path_factory=TempPathFactory(_given_basetemp=None,
================================================================================== 6 failed, 716 passed, 1 skipped in 19.50s ==================================================================================

Then, re-rerunning the same command in the same place, the tests always passed after failing in that initial run.

If I did a fresh build in a new chroot and then repeated the exercise,

This is fairly similar to the original report in this issue, where the tests failed once in a git checkout and didn’t fail in any subsequent attempts.

Maybe this has something to do with hypothesis persisting state, but rm -rf .hypothesis/ .pytest_cache/ doesn’t make the tests start failing again after the initial failure.

Next I will try adding --pdb to that initial failing test run in a fresh chroot.

[musicinmybrain]

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
  + Exception Group Traceback (most recent call last):
  |   File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 341, in from_call
  |     result: TResult | None = func()
  |                              ~~~~^^
  |   File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 242, in <lambda>
  |     lambda: runtest_hook(item=item, **kwds), when=when, reraise=reraise
  |             ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/pluggy/_hooks.py", line 513, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 182, in _multicall
  |     return outcome.get_result()
  |            ~~~~~~~~~~~~~~~~~~^^
  |   File "/usr/lib/python3.13/site-packages/pluggy/_result.py", line 100, in get_result
  |     raise exc.with_traceback(exc.__traceback__)
  |   File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/_pytest/threadexception.py", line 92, in pytest_runtest_call
  |     yield from thread_exception_runtest_hook()
  |   File "/usr/lib/python3.13/site-packages/_pytest/threadexception.py", line 68, in thread_exception_runtest_hook
  |     yield
  |   File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/_pytest/unraisableexception.py", line 95, in pytest_runtest_call
  |     yield from unraisable_exception_runtest_hook()
  |   File "/usr/lib/python3.13/site-packages/_pytest/unraisableexception.py", line 70, in unraisable_exception_runtest_hook
  |     yield
  |   File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/_pytest/logging.py", line 846, in pytest_runtest_call
  |     yield from self._runtest_for(item, "call")
  |   File "/usr/lib/python3.13/site-packages/_pytest/logging.py", line 829, in _runtest_for
  |     yield
  |   File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/_pytest/capture.py", line 880, in pytest_runtest_call
  |     return (yield)
  |             ^^^^^
  |   File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 167, in _multicall
  |     teardown.throw(outcome._exception)
  |     ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/_pytest/skipping.py", line 257, in pytest_runtest_call
  |     return (yield)
  |             ^^^^^
  |   File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 103, in _multicall
  |     res = hook_impl.function(*args)
  |   File "/usr/lib/python3.13/site-packages/_pytest/runner.py", line 174, in pytest_runtest_call
  |     item.runtest()
  |     ~~~~~~~~~~~~^^
  |   File "/usr/lib/python3.13/site-packages/_pytest/python.py", line 1627, in runtest
  |     self.ihook.pytest_pyfunc_call(pyfuncitem=self)
  |     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/pluggy/_hooks.py", line 513, in __call__
  |     return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
  |            ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/pluggy/_manager.py", line 120, in _hookexec
  |     return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
  |            ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 139, in _multicall
  |     raise exception.with_traceback(exception.__traceback__)
  |   File "/usr/lib/python3.13/site-packages/pluggy/_callers.py", line 103, in _multicall
  |     res = hook_impl.function(*args)
  |   File "/usr/lib/python3.13/site-packages/_pytest/python.py", line 159, in pytest_pyfunc_call
  |     result = testfunction(**testargs)
  |   File "/builddir/build/BUILD/python-cramjam-2.10.0_20250104git61564e7-build/cramjam-61564e7761e38e5ec55e7939ccd6a276c2c55d11/tests/test_variants.py", line 171, in test_variants_decompress_into
  |     "input_type", (bytes, bytearray, "numpy", cramjam.Buffer, cramjam.File, memoryview)
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/lib/python3.13/site-packages/hypothesis/core.py", line 1787, in wrapped_test
  |     raise the_error_hypothesis_found
  | hypothesis.errors.FlakyFailure: Hypothesis test_variants_decompress_into(variant_str='zstd', input_type='numpy', output_type='numpy', tmp_path_factory=TempPathFactory(_given_basetemp=None,
  |  _trace=<pluggy._tracing.TagTracerSub object at 0x7fcae92563f0>,
  |  _basetemp=PosixPath('/tmp/pytest-of-root/pytest-0'),
  |  _retention_count=3,
  |  _retention_policy='all'), raw_data=b'\xf6', is_pypy=False) produces unreliable results: Falsified on the first call but did not on a subsequent one (1 sub-exception)
  | Falsifying example: test_variants_decompress_into(
  |     variant_str='zstd',
  |     input_type='numpy',
  |     output_type='numpy',
  |     tmp_path_factory=TempPathFactory(_given_basetemp=None,
  |      _trace=<pluggy._tracing.TagTracerSub object at 0x7fcae92563f0>,
  |      _basetemp=PosixPath('/tmp/pytest-of-root/pytest-0'),
  |      _retention_count=3,
  |      _retention_policy='all'),
  |     is_pypy=False,
  |     raw_data=b'\xf6',
  | )
  | Failed to reproduce exception. Expected: 
  | variant_str = 'zstd', input_type = 'numpy', output_type = 'numpy'
  | tmp_path_factory = TempPathFactory(_given_basetemp=None, _trace=<pluggy._tracing.TagTracerSub object at 0x7fcae92563f0>, _basetemp=PosixPath('/tmp/pytest-of-root/pytest-0'), _retention_count=3, _retention_policy='all')
  | raw_data = b'0', is_pypy = False
  | 
  |     @pytest.mark.parametrize(
  |         "input_type", (bytes, bytearray, "numpy", cramjam.Buffer, cramjam.File, memoryview)
  |     )
  |     @pytest.mark.parametrize(
  |         "output_type", (bytes, bytearray, "numpy", cramjam.Buffer, cramjam.File, memoryview)
  |     )
  |     @pytest.mark.parametrize("variant_str", VARIANTS)
  |     @given(raw_data=st.binary())
  |     def test_variants_decompress_into(
  |         variant_str, input_type, output_type, tmp_path_factory, raw_data, is_pypy
  |     ):
  |         if variant_str == "izlib" and output_type == "memoryview":
  |             pytest.skip("See issue https://github.com/milesgranger/cramjam/issues/193")
  |     
  |         variant = getattr(cramjam, variant_str)
  |     
  |         compressed = variant.compress(raw_data)
  |     
  |         # Setup input
  |         if input_type == "numpy":
  |             input = np.frombuffer(compressed, dtype=np.uint8)
  |         elif input_type == cramjam.File:
  |             path = tmp_path_factory.mktemp("tmp").joinpath("input.txt")
  |             path.touch()
  |             input = cramjam.File(str(path))
  |             input.write(compressed)
  |             input.seek(0)
  |         elif input_type == cramjam.Buffer:
  |             input = cramjam.Buffer()
  |             input.write(compressed)
  |             input.seek(0)
  |         else:
  |             input = input_type(compressed)
  |     
  |         # Setup output buffer
  |         if output_type == "numpy":
  |             output = np.zeros(len(raw_data), dtype=np.uint8)
  |         elif output_type == cramjam.File:
  |             path = tmp_path_factory.mktemp("tmp").joinpath("output.txt")
  |             path.touch()
  |             output = cramjam.File(str(path))
  |         elif output_type == cramjam.Buffer:
  |             output = cramjam.Buffer()
  |         else:
  |             output = output_type(b"0" * len(raw_data))
  |     
  |         if is_pypy and isinstance(output, (bytes, memoryview)):
  |             pytest.xfail(
  |                 reason="PyPy de/compress_into w/ bytes or memoryview is a bit flaky behavior"
  |             )
  |     
  |         n_bytes = variant.decompress_into(input, output)
  |         assert n_bytes == len(raw_data)
  |     
  |         if hasattr(output, "read"):
  |             output.seek(0)
  |             output = output.read()
  |         elif hasattr(output, "tobytes"):
  |             output = output.tobytes()
  |         else:
  |             output = bytes(output)
  | >       assert same_same(output, raw_data)
  | E       AssertionError: assert False
  | E        +  where False = same_same(b'\xf6', b'0')
  | 
  | tests/test_variants.py:231: AssertionError
  | 
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/usr/lib/python3.13/site-packages/hypothesis/core.py", line 1064, in _execute_once_for_engine
    |     result = self.execute_once(data)
    |   File "/usr/lib/python3.13/site-packages/hypothesis/core.py", line 1003, in execute_once
    |     result = self.test_runner(data, run)
    |   File "/usr/lib/python3.13/site-packages/hypothesis/core.py", line 713, in default_executor
    |     return function(data)
    |   File "/usr/lib/python3.13/site-packages/hypothesis/core.py", line 978, in run
    |     return test(*args, **kwargs)
    |   File "/builddir/build/BUILD/python-cramjam-2.10.0_20250104git61564e7-build/cramjam-61564e7761e38e5ec55e7939ccd6a276c2c55d11/tests/test_variants.py", line 171, in test_variants_decompress_into
    |     "input_type", (bytes, bytearray, "numpy", cramjam.Buffer, cramjam.File, memoryview)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/lib/python3.13/site-packages/hypothesis/core.py", line 878, in test
    |     return self.test(*args, **kwargs)
    |            ~~~~~~~~~^^^^^^^^^^^^^^^^^
    |   File "/builddir/build/BUILD/python-cramjam-2.10.0_20250104git61564e7-build/cramjam-61564e7761e38e5ec55e7939ccd6a276c2c55d11/tests/test_variants.py", line 231, in test_variants_decompress_into
    |     assert same_same(output, raw_data)
    | AssertionError: assert False
    |  +  where False = same_same(b'\xf6', b'0')
    +------------------------------------
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>

>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /builddir/build/BUILD/python-cramjam-2.10.0_20250104git61564e7-build/cramjam-61564e7761e38e5ec55e7939ccd6a276c2c55d11/tests/test_variants.py(171)test_variants_decompress_into()
-> "input_type", (bytes, bytearray, "numpy", cramjam.Buffer, cramjam.File, memoryview)
(Pdb) 

I’m not too handy with pdb. What should I try?

[milesgranger]

I suggest just typing in output and see if it's equal to what was actually in the error and not equal to the raw_data; as silly as it might sound that was exactly the problem in that PyPy issue, there was some weird delay between what the error, and what the value of the variable was, so by the time I dropped into pdb and checked it out, it had changed to the expected value. Thus if I added a loop it would eventually pass.

I've also been running the tests many many time now for the last hour and haven't had a single failure like you're seeing here. So interesting as well it only happens on the first go but not subsequent tries.

Is it possible to have a nix shell config for the environment you've got? I'm not sure how far into this we want to go though. 😅 I'll keep trying, but in my mind once I slim down things a bit more 2.10 will be the last release for a long while until time opens up again.

[musicinmybrain]

I suggest just typing in output and see if it's equal to what was actually in the error and not equal to the raw_data; as silly as it might sound that was exactly the problem in that PyPy issue, there was some weird delay between what the error, and what the value of the variable was, so by the time I dropped into pdb and checked it out, it had changed to the expected value. Thus if I added a loop it would eventually pass.

Hmm:

(Pdb) output
*** NameError: name 'output' is not defined
(Pdb) dir()
['f', 'input_type', 'is_pypy', 'output_type', 'tmp_path_factory', 'variant_str']

I've also been running the tests many many time now for the last hour and haven't had a single failure like you're seeing here. So interesting as well it only happens on the first go but not subsequent tries.

I did try this repeatedly on Fedora 41:

rm -rf ~/.cache/pytest/* && rm -rf cramjam && git clone https://github.com/milesgranger/cramjam.git && ( cd cramjam && uv venv _e && . _e/bin/activate && uv pip install -e .[dev] && python -m pytest --ignore=benchmarks/test_bench.py -n auto )

…and I was able to get a failure like the one in the original report, with the INTERNALERRORs. I didn’t get it every time – more like one in five times – and I didn’t try enough variations to see which, if any, of the rm’s were necessary, but it’s something.

Is it possible to have a nix shell config for the environment you've got? I'm not sure how far into this we want to go though. 😅 I'll keep trying, but in my mind once I slim down things a bit more 2.10 will be the last release for a long while until time opens up again.

Unfortunately, I have never worked with nix, so I’m not sure what this would require.

[milesgranger]

I think you've banged your head enough on this for now, I appreciate the tips. I have a Fedora 41 Workstation myself, so I'll try to use your steps soon to reproduce.

[milesgranger]

FYI, using your last command as a repoducer, I can get it to happen every single time on my machine. But same thing, after the first failure run, it's happy after that. Very weird.

[milesgranger]

After poking at it for some time, I notice that the raw_data that it's failing on is always 0, but if I edit your command to first edit the file and set the min_size of the st.binary input to that test to be at least min_size=2? then I can no longer get it to fail. Continues to be weird, but I'm becoming less convinced that it's something wrong with cramjam but instead something else. Not neccessarily b/c of themin_size>=2 making it no longer fail, but because I also cannot get it to fail after running the tests for the first time; hinting there is just something about the first run that hypothesis/pytest is doing that makes things sad.

I'll try to debug some more, but after more than an hour of testing I haven't been able to pin anything; I've also removed all allow_threads bounds inside of the decompress_into impls, so there isn't much else in the way. I'm having the sense there is ya, something else going on. But it does feel super weird.

---

edit:
Puished the min_size=2 change to debugging branch, so the command now I cannot get to fail is:

rm -rf ~/.cache/pytest/* && rm -rf cramjam && git clone https://github.com/milesgranger/cramjam.git && ( cd cramjam && git checkout origin/debugging && uv venv _e && . _e/bin/activate && uv pip install -e .[dev] && CI=1 python -m pytest --ignore=benchmarks/test_bench.py -n auto )

[milesgranger]

Oh, now I see if instead setting max_size=1_000_000 also seems to be fine, can't get it to fail then. I'm starting to think this is not cramjam. I mean especially since if I do the setup in your command all the way to after the pip install then simply manually run the tests it also doesn't fail; only when it's ran in the same command.. Now given these knobs being turned on hypothesis results in it passing or not seems suspicious. I'm tempted again to remove hypothesis and roll my own random data generator for sanity sake.

[musicinmybrain]

I’ve noticed that this doesn’t happen in real RPM builds for Fedora 41, where python-hypothesis is still at 6.104.2 instead of 6.123.0, and in fact now I remember that the python-cramjam regression showed up in an impact check for that update. So it’s fair to say that this is related to the hypothesis version instead of (or, less likely, in addition to) the rustc version. That still doesn’t necessarily mean there’s a regression in hypothesis, as opposed to it doing a better job of finding the right conditions for an issue that already existed…