Repository navigation
Exception with KeyError: 'oml:tag' when fetching benchmarks #641
Description
Description
I followed the instruction on https://docs.openml.org/benchmark/#openml100 to fetch the details of benchmarks. but I encountered KeyError: 'oml:tag' for both open ml 100 and openml-cc18.
I use the latest openml package and Python 3.6
Steps/Code to Reproduce
Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 03:37:03) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
import openml
benchmark_suite = openml.study.get_study('OpenML-CC18','tasks')
Traceback (most recent call last):
File "", line 1, in
File "E:\Python\env\lib\site-packages\openml\study\functions.py", line 33, in get_study
for tag in _multitag_to_list(result_dict, 'oml:tag'):
File "E:\Python\env\lib\site-packages\openml\study\functions.py", line 8, in _multitag_to_list
if isinstance(result_dict[tag], list):
KeyError: 'oml:tag'
Expected Results
No error is thrown.
Actual Results
Traceback (most recent call last):
File "", line 1, in
File "E:\Python\env\lib\site-packages\openml\study\functions.py", line 33, in get_study
for tag in _multitag_to_list(result_dict, 'oml:tag'):
File "E:\Python\env\lib\site-packages\openml\study\functions.py", line 8, in _multitag_to_list
if isinstance(result_dict[tag], list):
KeyError: 'oml:tag'
Versions
Python 3.6.6, openml 0.8.0
Activity
joaquinvanschoren commentedon Mar 18, 2019
@janvanrijn @mfeurer
For the 'new' studies the tag that identifies the study is no longer returned (and the benchmarking studies are 'new' studies): https://www.openml.org/api/v1/study/99
@janvanrijn: is this by design? Then the openml-python package needs to be updated (and probably other bindings as well). If not then this is likely a bug in the API?
bailuding commentedon Mar 19, 2019
How can I avoid the exception and get the benchmark in Python then?
janvanrijn commentedon Mar 19, 2019
Currently, studies do no longer return tags by default (only legacy studies do, as announced in email thread on benchmark suites).
This is indeed a problem in openmlpython not covered by the unit tests. We should update this.
joaquinvanschoren commentedon Mar 19, 2019
OK, so what is the most practical way to proceed?
Do we make a new minor release of the python API with a fix?
Or do we make the server return the tag until the APIs are updated and released?
joaquinvanschoren commentedon Mar 19, 2019
I checked and it this change doesn't affect the R API.
@janvanrijn: I assume that you checked the Java API?
I'll make a PR for the python API now. We still need to decide whether we do a minor release of update the REST API.
mfeurer commentedon Mar 19, 2019
Thanks for checking this @joaquinvanschoren
We can also make a full release of the python API once this is integrated as there have been plenty of changes already. I guess we only need to wait for a fix to #646 prior to doing so to maintain backwards compatibility.
joaquinvanschoren commentedon Mar 19, 2019
Well, it looks like @janvanrijn already fixed this in commit a2a4ade
So that means that the development version should work
joaquinvanschoren commentedon Mar 19, 2019
@bailuding To work around this, you can install the development version of the python API until we made a new release: https://openml.github.io/openml-python/master/contributing.html#installation
janvanrijn commentedon Mar 19, 2019
That PR should be in the latest release already (
0.8.0) if I am not mistaken.joaquinvanschoren commentedon Mar 19, 2019
Hmm, I don't see it here: #617
So I guess it's not in 0.8.0
I added a PR with the new unit test: #649
mfeurer commentedon Mar 19, 2019
Fixed via #649. Please wait for the next release @bailuding for a fix on pypi.
bailuding commentedon Mar 19, 2019
Hi @joaquinvanschoren
I installed the develop branch, and the 'oml:tag' exception is gone. However, now I got another exception when trying to get the X and y:
'numpy.ndarray' object has no attribute 'columns'
Traceback (most recent call last):
File ".\test.py", line 532, in download_openml_benchmark
X, y = task.get_X_and_y() # get the data (not used in this example)
File "E:\Python\env\lib\site-packages\openml-0.8.0-py3.6.egg\openml\tasks\task.py", line 125, in get_X_and_y
dataset_format='array', target=self.target_name
File "E:\Python\env\lib\site-packages\openml-0.8.0-py3.6.egg\openml\datasets\dataset.py", line 538, in get_data
x = self._convert_array_format(x, dataset_format, attribute_names)
File "E:\Python\env\lib\site-packages\openml-0.8.0-py3.6.egg\openml\datasets\dataset.py", line 378, in _convert_array_format
for column_name in data.columns
AttributeError: 'numpy.ndarray' object has no attribute 'columns'
Do you know how I can work around this?
Package version:
numpy==1.15.0+mkl
openml==0.8.0
I am following this instruction to get the task (the Python example):
https://docs.openml.org/benchmark/#openml100
joaquinvanschoren commentedon Mar 20, 2019
@bailuding
Looks like this is related to #646
In short, you can't use cached datasets from the 0.8.0 with the development version (at least not now). Workaround is to delete your cache and try again. Most likely you just need to run:
bailuding commentedon Mar 20, 2019
@joaquinvanschoren
Thanks for the response! I use Windows with virtualenv, and I did not find the cache directory. Where shall I look for it?
joaquinvanschoren commentedon Mar 20, 2019
bailuding commentedon Mar 20, 2019
@joaquinvanschoren
Thanks! This works for me!