Error with nulls in categorical columns #1036

Activity

1. Neeratyoy commented on Mar 11, 2021

   

   Neeratyoy

   on Mar 11, 2021

   Contributor

   More actions

   Hi @SuryaThiru

   openml.datasets.get_dataset("penguins") appears to work (for me) with the dev branch, with the dependencies (for numpy, pandas especially) as you listed under versions. Could you kindly confirm the same?

   If it works, please use the dev branch. We are planning to release a new version soon which then should fix this issue!
2. SuryaThiru commented on Mar 11, 2021

   

   SuryaThiru

   on Mar 11, 2021

   Author

   More actions

   Installed 0.11.1dev using this command on colab (internet a bit slow to clone openml repo locally):

   pip install git+https://github.com/openml/openml-python@develop
   

   Now this command seems to work:

   import openml
   data = openml.datasets.get_dataset('penguins')
   data

   ==============
   Name..........: penguins
   Version.......: 1
   Format........: ARFF
   Upload Date...: 2020-07-22 18:34:52
   Licence.......: CC0
   Download URL..: https://www.openml.org/data/v1/download/21854866/penguins.arff
   OpenML URL....: https://www.openml.org/d/42585
   # of features.: 7
   # of instances: 344
   

   However, this throws the same error

   >>> data.get_data()
   
   ---------------------------------------------------------------------------
   ValueError                                Traceback (most recent call last)
   <ipython-input-10-b4e1f6df3b81> in <module>()
   ----> 1 data.get_data()
   
   9 frames
   /usr/local/lib/python3.7/dist-packages/openml/datasets/dataset.py in get_data(self, target, include_row_id, include_ignore_attribute, dataset_format)
       691             List of attribute names.
       692         """
   --> 693         data, categorical, attribute_names = self._load_data()
       694 
       695         to_exclude = []
   
   /usr/local/lib/python3.7/dist-packages/openml/datasets/dataset.py in _load_data(self)
       529 
       530             file_to_load = self.data_file if self.parquet_file is None else self.parquet_file
   --> 531             return self._cache_compressed_file_from_file(file_to_load)
       532 
       533         # helper variable to help identify where errors occur
   
   /usr/local/lib/python3.7/dist-packages/openml/datasets/dataset.py in _cache_compressed_file_from_file(self, data_file)
       486 
       487         if data_file.endswith(".arff"):
   --> 488             data, categorical, attribute_names = self._parse_data_from_arff(data_file)
       489         elif data_file.endswith(".pq"):
       490             try:
   
   /usr/local/lib/python3.7/dist-packages/openml/datasets/dataset.py in _parse_data_from_arff(self, arff_file_path)
       443                 if attribute_dtype[column_name] in ("categorical", "boolean"):
       444                     col.append(
   --> 445                         self._unpack_categories(X[column_name], categories_names[column_name])
       446                     )
       447                 elif attribute_dtype[column_name] in ("floating", "integer"):
   
   /usr/local/lib/python3.7/dist-packages/openml/datasets/dataset.py in _unpack_categories(series, categories)
       648         # We require two lines to create a series of categories as detailed here:
       649         # https://pandas.pydata.org/pandas-docs/version/0.24/user_guide/categorical.html#series-creation  # noqa E501
   --> 650         raw_cat = pd.Categorical(col, ordered=True, categories=categories)
       651         return pd.Series(raw_cat, index=series.index, name=series.name)
       652 
   
   /usr/local/lib/python3.7/dist-packages/pandas/core/arrays/categorical.py in __init__(self, values, categories, ordered, dtype, fastpath)
       306 
       307         dtype = CategoricalDtype._from_values_or_dtype(
   --> 308             values, categories, ordered, dtype
       309         )
       310         # At this point, dtype is always a CategoricalDtype, but
   
   /usr/local/lib/python3.7/dist-packages/pandas/core/dtypes/dtypes.py in _from_values_or_dtype(cls, values, categories, ordered, dtype)
       275             # Note: This could potentially have categories=None and
       276             # ordered=None.
   --> 277             dtype = CategoricalDtype(categories, ordered)
       278 
       279         return dtype
   
   /usr/local/lib/python3.7/dist-packages/pandas/core/dtypes/dtypes.py in __init__(self, categories, ordered)
       162 
       163     def __init__(self, categories=None, ordered: Ordered = False):
   --> 164         self._finalize(categories, ordered, fastpath=False)
       165 
       166     @classmethod
   
   /usr/local/lib/python3.7/dist-packages/pandas/core/dtypes/dtypes.py in _finalize(self, categories, ordered, fastpath)
       316 
       317         if categories is not None:
   --> 318             categories = self.validate_categories(categories, fastpath=fastpath)
       319 
       320         self._categories = categories
   
   /usr/local/lib/python3.7/dist-packages/pandas/core/dtypes/dtypes.py in validate_categories(categories, fastpath)
       490 
       491             if categories.hasnans:
   --> 492                 raise ValueError("Categorical categories cannot be null")
       493 
       494             if not categories.is_unique:
   
   ValueError: Categorical categories cannot be null
3. PGijsbers commented on Mar 12, 2021

   

   PGijsbers

   on Mar 12, 2021

   Collaborator

   More actions

   For those that experience this issue, the dataset is still downloaded to ~\.openml\cache\org\openml\www\datasets\42585. So if you only care about the arff file that might be of help. I'll do my best to set up a PR this afternoon.
4. PGijsbers

   added

   bug

   priority

   on Mar 12, 2021
5. PGijsbers

   self-assigned this

   on Mar 12, 2021
6. PGijsbers commented on Mar 12, 2021

   

   PGijsbers

   on Mar 12, 2021

   Collaborator

   More actions

   Fixed in #1037. Please use the develop branch for now, we plan to do a new PyPI release soon.
7. PGijsbers

   closed this as completedon Mar 12, 2021