Skip to content

BUG: apply NEP 50 scalar rules in concatenate and choose, and make casting="no" match "equiv" - #32510

Closed
Rohan143-mp wants to merge 1 commit into
numpy:mainfrom
Rohan143-mp:Rohan-contri
Closed

Rohan143-mp wants to merge 1 commit into
numpy:mainfrom
Rohan143-mp:Rohan-contri

Conversation

@Rohan143-mp

Copy link
Copy Markdown

Closes #32491

Problem & Background

In ufuncs and np.copyto, passing casting="no" with a Python scalar whose target dtype differs from its default dtype (e.g. int64 for Python int, float64 for Python float) was previously accepted, while casting="equiv" raised a TypeError:

np.copyto(np.zeros(3, dtype="uint8"), 3, casting="no") # SUCCEEDED
np.copyto(np.zeros(3, dtype="uint8"), 3, casting="equiv") # FAILED with TypeError

Because "no" casting is strictly more restrictive than "equiv", "no" should never succeed where "equiv" fails.

Root Cause
In numpy/_core/src/multiarray/abstractdtypes.c, npy_update_operand_for_scalar specifically checked for casting == NPY_EQUIV_CASTING. When casting == NPY_NO_CASTING was passed, it fell through, recreated the temporary array with the target descriptor, and the subsequent cast-safety check compared identical dtypes.

Additionally:

np.concatenate(axis=None) and np.choose did not convert Python scalars using NEP 50 value bounds, causing out-of-bounds integers to silently wrap instead of raising OverflowError (e.g., np.concatenate((np.ones(2, "int8"), 300), axis=None) returned 44 for 300).
In abstractdtypes.c, npy_update_operand_if_pystr was only handling Python str, but the same scalar re-conversion logic was needed for all Python literals/scalars (NPY_ARRAY_WAS_PYTHON_LITERAL).
Changes Made

  1. Multiarray Core C Engine

abstractdtypes.c:
Included convert_datatype.h for npy_casting_to_string.
Changed condition in npy_update_operand_for_scalar from casting == NPY_EQUIV_CASTING to casting <= NPY_EQUIV_CASTING, rejecting both "no" and "equiv" when the scalar's default dtype does not match the target. Formatted the exception message dynamically using npy_casting_to_string(casting).
Generalized npy_update_operand_if_pystr to npy_update_operand_if_pyscalar, handling NPY_ARRAY_WAS_PYTHON_LITERAL | NPY_ARRAY_WAS_PYTHON_STR and forwarding the operation's casting argument.

abstractdtypes.h:
Updated prototype for npy_update_operand_if_pyscalar.

convert_datatype.c:
Updated PyArray_ConvertToCommonType (used in np.choose) to call npy_update_operand_if_pyscalar(&mps[i], op, i, common_descr, NPY_SAFE_CASTING).

multiarraymodule.c:
Updated PyArray_ConcatenateFlattenedArrays (np.concatenate(..., axis=None)) to call npy_update_operand_if_pyscalar(&arrays[iarrays], op, iarrays, PyArray_DESCR(ret), casting).

  1. Documentation & Release Notes
    Added Towncrier release notes:
    doc/release/upcoming_changes/32497.compatibility.rst
    doc/release/upcoming_changes/32497.improvement.rst
    Updated doc/source/glossary.rst casting definition with reference to scalar rules.
    Added a dedicated section Cast safety of Python scalars in doc/source/reference/arrays.promotion.rst documenting kind-based promotion, precision loss, out-of-bounds OverflowError, and "no" / "equiv" restrictions.

  2. Unit Tests
    test_api.py: Added assertions in test_copyto_cast_safety verifying that casting="no" raises TypeError when target dtypes differ.
    test_multiarray.py: Fixed in-bounds scalar values in test_output_dtype and added test_pyscalar_out_of_bounds for np.choose.
    test_shape_base.py: Added test_pyscalar_out_of_bounds, test_pyscalar_casting_matches_copyto, and test_pyscalar_casting_matches_ufunc.
    test_stringdtype.py: Removed obsolete assertions in test_pystr_scalar_concatenate_preserves_nulls that expected "no" and "equiv" to succeed for Python strings.
    test_ufunc.py: Added casting="no" tests for scalar operands in test_cast_safety_scalar and test_resolve_dtypes_basic.

Result

  1. Consistent Casting Rules:
    casting="no" and casting="equiv" now behave consistently: if converting a scalar to a non-default dtype fails under "equiv", it also fails under "no".

np.copyto(np.zeros(3, dtype="uint8"), 3, casting="no")
TypeError: cannot cast Python int to uint8 under the casting rule 'no'

  1. Safe Python Scalar Handling in concatenate and choose:
    Out-of-bounds scalar values now consistently raise OverflowError rather than silently overflowing or wrapping:

np.concatenate((np.ones(2, "int8"), 300), axis=None)
OverflowError: Python integer 300 out of bounds for int8

np.choose([0], (-1, np.array([1], dtype=np.uint8)))
OverflowError: Python integer -1 out of bounds for uint8

Full Parity Across Operations:
np.concatenate(axis=None), np.choose, np.copyto, and ufuncs now share the exact same scalar casting and conversion behavior.

@Rohan143-mp

Copy link
Copy Markdown
Author

Hi @ngoldbaum, I've implemented the fix for #32491 along with unit tests and docs. Whenever you have time, I'd appreciate your review and feedback. Thanks!

@ngoldbaum ngoldbaum closed this Sep 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ENH: make "no" and "equiv" casting consistent for Python scalars in ufuncs and copyto

2 participants