ENH: fixes for StringDType sorting and comparisons for missing data - #32564

PR summary

Fixes a couple bugs in the StringDType implementation, where missing data doesn't properly implement equality operators.

For nan-like missing data, it's the != operator:

>>> import numpy as np
>>> arr = np.array(["hello", np.nan, "world"], dtype=np.dtypes.StringDType(na_object=np.nan))
>>> arr == "hello"
array([ True, False, False])
>>> arr != "hello"
array([False, False,  True])

The answer for the last expression should be array([False, True, True]). The current answer is a straight-up bug. Unfortunately it also changes operations like arr[data != sentinel], so we should probably not backport it.

This also fixes this issue:

>>> arr = np.array([None, "", "hello"], dtype=np.dtypes.StringDType(na_object=None))
>>> arr == ""
array([ True,  True, False])
>>> arr != ""
array([False, False,  True])

NumPy says None == "" here! Under this PR, you get:

>>> arr == ""
array([False,  True, False])
>>> arr != ""
array([ True, False,  True])

As a result of these two fixes, also fixes issues with the np.unique code paths that use the sort-based implementation, see the tests.

This also includes a bugfix for numpy's testing utilities. At some point I added a second implementation of StringDType's missing value classification, which is broken in a way that the tests added here expose. The fix is to defer to the rules the dtype actually uses.

AI Disclosure

I iterated with an AI on this.