BUG: ensure that casting to/from structured is properly checked. - #13648

@eric-wieser - I changed it to allow casting to multiple fields. Also realized I had to put it further up, since the changes hold also for unsafe casting. Since I think this is getting close, I also updated the docstrings. I think can_cast is now consistent with actually executing a.astype(...).

@seberg, @eric-wieser, thanks for all the comments!

On the casting, I'm a bit torn. I agree with @eric-wieser that we should try to make can_cast consistent with actual casting behaviour in, e.g., astype, to the extent possible. The current behaviour of astype and __setitem__ (see below) can be summarized as follows:

• Structured to simple: only allowed for a single field and when casting="unsafe"; can_cast is inconsistent with this in always returning True for "unsafe", independent of the number of fields. This is an easy fix.
• Simple to structured: always done for any casting but "equiv" and "no"; this is the bug I hoped to fix.

My own sense is that symmetry suggests that we do not allow simple to structured at all for any casting but "unsafe". But for "unsafe", it does seem fine to just allow the cast to all fields, especially as this is what happens also in __setitem__.

This seems to make sufficient sense that I'll go ahead and change it accordingly...

Current behaviour

Going from structured to simple:

import numpy as np
# Multiple fields to simple
i8 = np.array([1000])
i1i1 = np.array([(1,2)], dtype='i1,i1')
np.can_cast('i1,i1', 'i8', casting='unsafe')
# True # and False for any other casting mode
i1i1.astype('i8', casting='unsafe')
# ValueError: Can't cast from structure to non-structure, except if the structure only has a single field.
i8[...] = i1i1
# ValueError: Can't cast from structure to non-structure, except if the structure only has a single field.

# Single field to simple
i1 = np.array([(1,)], dtype=[('i', 'i1')])
np.can_cast([('i', 'i1')], 'i8', casting='unsafe')
# True  # and False for any other casting mode
i1.astype('i8', casting='unsafe')
# array([1])  # note that casting='unsafe' is default
i8[...] = i1
# Passes -> i8=1
np.array([(1,)], dtype=[('i', 'i1')]).astype('i8', casting='safe')
# TypeError: Cannot cast array from dtype([('i', 'i1')]) to dtype('int64') according to the rule 'safe'

So, for this case, astype behaves sensibly in allowing a cast from structured to simple only when there is a single field, and even then only when casting="unsafe" - the current PR would extend this to allow any case where the single field can be cast to the simple type, but I think I should revert that and instead just make can_cast consistent with astype, i.e., change its behaviour for "unsafe" to not allow the case of multiple fields.

Now turning to casting from simple to structured

import numpy as np
i8 = np.array([1000])
# Simple to multiple, non-safe fields
np.can_cast('i8', 'i1,i1', casting='safe')
# True  # only 'no' and 'equiv' give False
i8.astype('i1,i1', casting='safe')
# array([(-24, -24)], dtype=[('f0', 'i1'), ('f1', 'i1')])
i1i1[...] = i8
# Passes -> same array.
# only 'no' and 'equiv' error
# Behaviour identical for safe fields and for a single field!

So, the current behaviour is that casting to a structured type is never allowed for "equiv" and always allowed for "safe" and beyond.

p.s. I think there actually is another inconsistency for structured to structured casting, in that can_cast looks at field names while astype and __setitem__ go by field order. Probably a leftover of #6053, so cc @ahaldane (and let's leave it out of this PR...)

EDIT: see gh-13667

Rewritten as noted above. Quite a bit simpler PR, which suggests it may be better..

OK, pushed another update. Am still trying to keep the impact minimal. Another worry I have is whether user types can have fields - though at least this should be better than it was before...

Damn, astype is really quite inconsistent too in what it does for subarrays. For now, focussing on getting the structured from right...

Also failing doctest at numpy.doc.structured_arrays.

@seberg, @eric-wieser - what do you think we should do with this PR? Overall, I feel it at least fixes one obvious glaring inconsistency. On the other, it leaves many others. But those generally need more thought at developing a better set of tests. Personally, I'd prefer to approach this incrementally (mostly as I don't think I've got the time to do the larger overhaul...).

@mhvk sorry, meant to look through this for a bit now, but then did not when I saw we had put 1.18, and thinking I can't merge it anyway ;).
I hope more restructuring of casting can/will happen, but I am ok with doing some small improvements before the end. Plus anything that works and is tested now, is something we can't break when we try to do larger changes.

Ideally this would still go in 1.17...

The 1.18 just means it isn't a blocker for 1.17, it can certainly go in 1.17.

OK, made the changes, and added a release note.

Cool, thanks Marten.