Missing use cases for binops on array vs. scalar

[crusaderky]

After reading hh.scalars, I understand that array-api-tests currently never generates the use cases of a binop where one argument is an Array with float or complex dtype and the other argument is

• a pure-python integer;
• a scalar inf, -inf, or nan;
• a scalar outside of the range [1/64, 64]

Unless I missed something and the tests for these cases are somewhere else?

array-api-tests/array_api_tests/hypothesis_helpers.py

Lines 451 to 481 in c1e5cda

	@composite
	def scalars(draw, dtypes, finite=False, **kwds):
	"""
	Strategy to generate a scalar that matches a dtype strategy
	dtypes should be one of the shared_* dtypes strategies.
	"""
	dtype = draw(dtypes)
	if dh.is_int_dtype(dtype):
	m, M = dh.dtype_ranges[dtype]
	return draw(integers(m, M))
	elif dtype == bool_dtype:
	return draw(booleans())
	elif dtype == float64:
	if finite:
	return draw(floats(allow_nan=False, allow_infinity=False, **kwds))
	return draw(floats(), **kwds)
	elif dtype == float32:
	if finite:
	return draw(floats(width=32, allow_nan=False, allow_infinity=False, **kwds))
	return draw(floats(width=32, **kwds))
	elif dtype == complex64:
	if finite:
	return draw(complex_numbers(width=32, allow_nan=False, allow_infinity=False))
	return draw(complex_numbers(width=32))
	elif dtype == complex128:
	if finite:
	return draw(complex_numbers(allow_nan=False, allow_infinity=False))
	return draw(complex_numbers())
	else:
	raise ValueError(f"Unrecognized dtype {dtype}")

array-api-tests/array_api_tests/hypothesis_helpers.py

Lines 590 to 597 in c1e5cda

	@composite
	def array_and_py_scalar(draw, dtypes):
	"""Draw a pair: (array, scalar) or (scalar, array)."""
	dtype = draw(sampled_from(dtypes))
	scalar_var = draw(scalars(just(dtype), finite=True,
	**{'min_value': 1/ (2<<5), 'max_value': 2<<5}
	))

Note that the [1/64, 64] range exclusively applies to float/complex. The code could use a clarification there as it is not obvious upon first read that integers ignore it.

e.g. pytest -k minimum returns this:

array_api_tests/test_has_names.py::test_has_names[elementwise-minimum] PASSED                                                                                    [ 20%]
array_api_tests/test_operators_and_elementwise_functions.py::test_minimum PASSED                                                                                 [ 40%]
array_api_tests/test_operators_and_elementwise_functions.py::test_binary_with_scalars_real[minimum] SKIPPED (requires ARRAY_API_TESTS_VERSION >= 2024.12)        [ 60%]
array_api_tests/test_signatures.py::test_func_signature[minimum] PASSED                                                                                          [ 80%]
array_api_tests/test_special_cases.py::test_binary[minimum(x1_i is NaN or x2_i is NaN) -> NaN] PASSED   

To my understanding

• in test_binary_with_scalars_real, when one arg is Array[float64] or Array[float32], the other arg is always in the [1/64, 64] range;
• test_special_cases::test_binary only tests arrays containing NaN, not scalar NaN;
• none tests inf or arrays containing inf

related:

• test_clip: make sure min and max have the same dtype as x #360
• clip: ambiguous wording for scalar min/max array-api#925

[ev-br]

In test_binary_with_scalars_real, when one arg is Array[float64] or Array[float32], the other arg is always in the [1/64, 64] range;

Yes. This is a shortcut of course, made to get some testing going.

There are several irksome bits to take into account when pondering about undoing it:

• Hypothesis likes to generate edge cases. If you don't limit the range, you are bound to get denormal numbers, values close to the min/max values etc.
• There is no way to xfail a part of a test. If a test fails on backend X because of an edge case around FLOAT_MAX, we have to xfail the whole test for this backend.

In general, value testing is very fiddly.

• test_special_cases needs separate attention, Special case tests need to be updated #284

---

My strategy so far was to first get test some coverage for 2024 additions, however rough; and use #299 and #301 to track the holes.

[ev-br]

#410 adds testing of float_array, int_scalar combinations.

[ev-br]

#410 plugged the biggest hole, float array <op> int scalar. The rest of this is issue is in the realm of value testing: inf/nans/large values (complex arrays are not tested either). Better value testing would be nice indeed, improvements are most welcome.