Serilization of np integers and floats #304

Activity

1. mfeurer commented on Sep 2, 2017

   

   mfeurer

   on Sep 2, 2017

   Collaborator

   More actions

   Correct, but why can't you use normal integers?
2. mfeurer

   added

   Feature request

   on Sep 29, 2017
3. PGijsbers commented on Mar 3, 2019

   

   PGijsbers

   on Mar 3, 2019

   Last edited by PGijsbers

   Collaborator

   More actions

   Edit: This comment is edited heavily multiple times while working on this issue. I finally edited all parts out not related to this issue, and it is now final.

   I ran into this as well. In my case I wanted to specify all possible labels for the scikit-learn OrdinalEncoder, and the possible labels were retrieved from the dataset, which was stored as numpy floats. Extending for numpy types should be relatively simple, given we can retrieve them programmatically:

   >>> np.sctypes
   {'int': [<class 'numpy.int8'>, <class 'numpy.int16'>, <class 'numpy.int32'>, <class 'numpy.int64'>],
   'uint': [<class 'numpy.uint8'>, <class 'numpy.uint16'>, <class 'numpy.uint32'>, <class 'numpy.uint64'>],
   'float': [<class 'numpy.float16'>, <class 'numpy.float32'>, <class 'numpy.float64'>],
   'complex': [<class 'numpy.complex64'>, <class 'numpy.complex128'>],
   'others': [<class 'bool'>, <class 'object'>, <class 'bytes'>, <class 'str'>, <class 'numpy.void'>]}
   

   e.g.

   simple_numpy_types = [nptype for type_cat, nptypes in np.sctypes.items()
                                for nptype in nptypes
                                if type_cat != 'others']
   simple_types = tuple([bool, int, float, str] + simple_numpy_types)
   

   and then
   elif isinstance(o, simple_types) or o is None:
4. PGijsbers commented on Mar 3, 2019

   

   PGijsbers

   on Mar 3, 2019

   Collaborator

   More actions

   First off, a (artificial) minimal example that exposes the issue:

   import numpy as np
   import openml
   from sklearn.tree import DecisionTreeClassifier
   
   openml.flows.sklearn_to_flow(DecisionTreeClassifier(max_depth=np.float32(3.0))
   

   this currently raises

   TypeError: (3.0, <class 'numpy.float32'>)
   

   After patching in the simple_types check of the previous comment, the following error is raised instead:

   TypeError: Object of type 'float32' is not JSON serializable
   

   So we could extend the sklearn_to_flow by casting to the native data types:

   elif isinstance(o, simple_types) or o is None:
       if type(o) in simple_numpy_types:
           o = o.item()
       # base parameter values
       rval = o
   

   This seems to work as expected.
5. PGijsbers

   self-assigned this

   on Mar 3, 2019
6. mfeurer commented on Mar 4, 2019

   

   mfeurer

   on Mar 4, 2019

   Collaborator

   More actions

   It appears like this is a common feature request. The only thing I'm a bit worried about is that we cannot easily do the de-serialization and get a numpy dtype again. Maybe that is not an issue for simple scalars but only for arrays?
7. PGijsbers commented on Mar 4, 2019

   

   PGijsbers

   on Mar 4, 2019

   Collaborator

   More actions

   For the types that would now be included for serialization (various bit-length ints, uints, floats and complex), I don't foresee an issue. Could you come up with a concrete issue where it would matter? Otherwise I think it it's smarter to add this feature and tackle the problem if it actually comes up.
8. mfeurer commented on Mar 4, 2019

   

   mfeurer

   on Mar 4, 2019

   Collaborator

   More actions

   Makes sense. Please assign me to your PR once it is ready.
9. mfeurer commented on Mar 5, 2019

   

   mfeurer

   on Mar 5, 2019

   Collaborator

   More actions

   Closed via #635.
10. mfeurer

    closed this as completedon Mar 5, 2019