ENH: add NEP-50-style string promotion outside ufuncs by ngoldbaum · Pull Request #32356 · numpy/numpy

ngoldbaum · 2026-08-19T21:45:35Z

PR summary

Followup for #32040

This adds NEP-50 style promotion for strings to copyto and where, which have explicit handling for numeric scalars. It also adds new promotion handling for concatenate and choose. These weren't needed before in concatenate and choose because numeric scalars don't need to do anything the the scalar value, only type metadata. Concatenate and choose correctly propagate that metadata but don't correctly propagate trailing NULLs.

The net effect is that trailing NULLs are now preserved for StringDType in all four operations, see tests.

AI Disclosure

I iterated on this with an AI model.

mhvk

Looks good! Most comments just nits about organization.

p.s. There's part of me that still wonders whether str shouldn't be treated more just like the numerical scalars, but I guess we can always change that if needed.

mhvk · 2026-08-23T01:19:23Z

        Py_DECREF(item);
    }

+    /* Explicit axes reject the 0-d str before resolving a descriptor */


This can be inside if (axis == NPY_RAVEL_AXIS) just below

mhvk · 2026-08-23T01:20:42Z

+    }
+
    if (axis == NPY_RAVEL_AXIS) {
        ret = PyArray_ConcatenateFlattenedArrays(


I guess for numerical scalers, the check is done inside ConcatenateFlattenedArrays -- it would seem more logical to do the same for str as well (if for some reason that's just unhandy, maybe add a comment here that the same checks for numerical scalars are done in ConcatenateFlattenedArrays).

mhvk · 2026-08-23T01:22:02Z

    /* 'narrays' was set to how far we got in the conversion */
    for (iarrays = 0; iarrays < narrays; ++iarrays) {
-        Py_DECREF(arrays[iarrays]);
+        Py_XDECREF(arrays[iarrays]);


Why is this change necessary?

I rewrote the PR so changing this isn't necessary anymore

mhvk · 2026-08-23T01:26:59Z

+    dst = np.empty(2, dtype=dtype)
+    np.copyto(dst, scalar)
+    assert dst[0] == scalar
+    np.copyto(dst, scalar, casting="unsafe")


I think you need to recreate the array to be sure the tests checks that copyto actually worked (since there was a copy already). Or is the casting="unsafe" needed if the array is already filled? If so, do a quick pytest.raises before with regular casting (and add a comment!).

mhvk · 2026-08-23T01:28:30Z

+
+    res = np.concatenate((arr, scalar), axis=None)
+    assert res.dtype == dtype
+    assert_array_equal(res, np.array(["y", scalar], dtype=dtype))


If you add strict=True, you do not really need to assert the dtype above (though perhaps you prefer it for clarity).

ngoldbaum · 2026-08-26T20:21:43Z

@mhvk thanks for the suggestions! I think I've addressed your comments, mostly to reorganize and delete code that you were commenting on. This is a lot simpler now. I'd appreciate it if you could give it another look.

mhvk

@ngoldbaum - it has indeed become a nice surgical addition! One remaining nitpick, but also a question about casting that struck me in the tests. I may just be missing something obvious, though!

mhvk · 2026-08-27T00:58:08Z


    for (iarrays = 0; iarrays < narrays; ++iarrays) {
-        Py_DECREF(arrays[iarrays]);
+        Py_XDECREF(arrays[iarrays]);


I thought I saw come by that this change was no longer needed. Did you forget to push the change? Or was that in another piece of code. Anyway, no big deal if it is needed, but if not, maybe nice to undo.

I removed one in the failure path, but this is the success path cleanup so it's still needed. It needs to be an XDECREF because I rearranged how the NULL checking works above, which means that arrays[iarrays] might be NULL now.

But actually looking closer I think I can rearrange again so that this doesn't need to change.

Sounds good, though don't go overboard - I don't mind it, just wanted to be sure it wasn't a change that was no longer needed.

mhvk · 2026-08-27T01:03:05Z


+    # A Python str adopts a StringDType's semantics in the same way
+    np.copyto(np.empty(3, dtype=np.dtypes.StringDType()), "x", casting="no")
+    with pytest.raises(TypeError):


I realize I'm actually confused: how can "no" casting be OK, but "equiv" not? I understand the ones above, where one goes up in stringency, from "safe" to "equiv", but here you go down!

Nice catch! I didn't think too hard about this and just added a test that documented the behavior, but you're right we can do better here. See next push coming shortly.

There's also a similar issue in concatenate...

So it turns out this inherits the behavior for ints:

>>> np.copyto(np.arange(3, dtype='uint8'), 3, casting='no') >>> np.copyto(np.arange(3, dtype='uint8'), 3, casting='equiv') Traceback (most recent call last): File "<python-input-3>", line 1, in <module> np.copyto(np.arange(3, dtype='uint8'), 3, casting='equiv') ~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: cannot cast Python int to uint8 under the casting rule 'equiv'

This is happening because npy_update_operand_for_scalar special-cases equiv:

numpy/numpy/_core/src/multiarray/abstractdtypes.c

Lines 366 to 388 in b265412

if (PyArray_EquivTypes(PyArray_DESCR(*operand), descr)) {

/*

* TODO: This is an unfortunate work-around for legacy type resolvers

* (see `convert_ufunc_arguments` in `ufunc_object.c`), that

* currently forces us to replace the array.

*/

if (!(PyArray_FLAGS(*operand) & NPY_ARRAY_WAS_PYTHON_INT)) {

return 0;

}

}

else if (NPY_UNLIKELY(casting == NPY_EQUIV_CASTING) &&

descr->type_num != NPY_OBJECT) {

/*

* incredibly niche, but users could pass equiv casting and we

* actually need to cast. Let object pass (technically correct) but

* in all other cases, we don't technically consider equivalent.

* NOTE(seberg): I don't think we should be beholden to this logic.

*/

PyErr_Format(PyExc_TypeError,

"cannot cast Python %s to %S under the casting rule 'equiv'",

Py_TYPE(scalar)->tp_name, descr);

return -1;

}

So I think we'd need to have a bigger discussion about changing the rules here.

For now I'll change the test to use "safe" casting and open a followup issue about this "no"/"equiv" mess. I'll also fix the inconcistency with ints in concatenate...

OK, indeed best to punt it to a separate issue.

… and choose

ngoldbaum · 2026-09-03T14:29:48Z

I opened #32491 to track the no/equiv casting issue. Let's bring this in. Thank for the review @mhvk.

ngoldbaum added 01 - Enhancement component: numpy.strings String dtypes and functions labels Aug 19, 2026

ngoldbaum force-pushed the fix-promotion-leftovers branch from 70812c6 to 53e0462 Compare August 19, 2026 21:46

mhvk reviewed Aug 23, 2026

View reviewed changes

ngoldbaum mentioned this pull request Aug 25, 2026

BUG: StringDType loses trailing NULs from str scalars in Python wrappers #32431

Open

ngoldbaum force-pushed the fix-promotion-leftovers branch from 53e0462 to b7dda5a Compare August 26, 2026 20:19

mhvk reviewed Aug 27, 2026

View reviewed changes

ngoldbaum added 4 commits September 2, 2026 10:12

ENH: add NEP-50 StringDType promotion for copyto, where, concatenate,…

cdccb3b

… and choose

MAINT: simplify per Marten's suggestions

6c11aea

MAINT: simplify concatenate

b329891

MAINT: respond to code review comments

99a87f0

ngoldbaum force-pushed the fix-promotion-leftovers branch from b7dda5a to 99a87f0 Compare September 2, 2026 16:13

ngoldbaum mentioned this pull request Sep 3, 2026

ENH: make "no" and "equiv" casting consistent for Python scalars in ufuncs and copyto #32491

Open

ngoldbaum merged commit 617a92d into numpy:main Sep 3, 2026
91 checks passed

ngoldbaum mentioned this pull request Sep 3, 2026

BUG: apply the NEP 50 scalar rules in concatenate and choose, and make casting="no" match "equiv" #32497

Open

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 8, 2026

ENH: add NEP-50-style string promotion outside ufuncs (numpy#32356)

ef1360c

ngoldbaum added a commit to ngoldbaum/numpy that referenced this pull request Sep 8, 2026

ENH: add NEP-50-style string promotion outside ufuncs (numpy#32356)

2854158

	if (PyArray_EquivTypes(PyArray_DESCR(*operand), descr)) {
	/*
	* TODO: This is an unfortunate work-around for legacy type resolvers
	* (see `convert_ufunc_arguments` in `ufunc_object.c`), that
	* currently forces us to replace the array.
	*/
	if (!(PyArray_FLAGS(*operand) & NPY_ARRAY_WAS_PYTHON_INT)) {
	return 0;
	}
	}
	else if (NPY_UNLIKELY(casting == NPY_EQUIV_CASTING) &&
	descr->type_num != NPY_OBJECT) {
	/*
	* incredibly niche, but users could pass equiv casting and we
	* actually need to cast. Let object pass (technically correct) but
	* in all other cases, we don't technically consider equivalent.
	* NOTE(seberg): I don't think we should be beholden to this logic.
	*/
	PyErr_Format(PyExc_TypeError,
	"cannot cast Python %s to %S under the casting rule 'equiv'",
	Py_TYPE(scalar)->tp_name, descr);
	return -1;
	}

Uh oh!

Uh oh!

Conversation

ngoldbaum commented Aug 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR summary

AI Disclosure

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngoldbaum commented Aug 26, 2026

Uh oh!

mhvk left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngoldbaum commented Sep 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ngoldbaum commented Aug 19, 2026 •

edited

Loading