Repository navigation
Still not able to use non-sklearn estimators without wrapping them in a pipeline #734
Description
Hi there, I raised this in issue #724. I have been trying to run experiments with a fairly new sklearn-extra classifier (https://github.com/Alex7Li/scikit-learn-extra/tree/master/sklearn_extra). The classifier runs fine on a local dataset. However, when I am trying to run it on an openml task, I am getting an error.
Here is a minimal example:
# define classifier
from sklearn_extra.fast_kernel import FKC_EigenPro
clf = FKC_EigenPro()
# get task
task = openml.tasks.get_task(3)
# run model on task
run = openml.runs.run_model_on_task(clf, task)
# publish run on openml
run.publish()AttributeError Traceback (most recent call last)
in
4 task = openml.tasks.get_task(3)
5 # run model on task
----> 6 run = openml.runs.run_model_on_task(clf, task)
7 # publish run on openml
8 run.publish()
/miniconda3/lib/python3.7/site-packages/openml/runs/functions.py in run_model_on_task(model, task, avoid_duplicate_runs, flow_tags, seed, add_local_measures, upload_flow, return_flow)
104 seed=seed,
105 add_local_measures=add_local_measures,
--> 106 upload_flow=upload_flow,
107 )
108 if return_flow:
/miniconda3/lib/python3.7/site-packages/openml/runs/functions.py in run_flow_on_task(flow, task, avoid_duplicate_runs, flow_tags, seed, add_local_measures, upload_flow)
172 task, flow = flow, task
173
--> 174 flow.model = flow.extension.seed_model(flow.model, seed=seed)
175
176 # We only need to sync with the server right now if we want to upload the flow,
AttributeError: 'NoneType' object has no attribute 'seed_model'
You mentioned that this should be fixed via #722, but I am still encountering the same error.
Activity
amueller commentedon Jul 22, 2019
Easier way to reproduce:
So
get_extension_by_modelreturnssklearnbecauseisinstance(MyLR(), BaseEstimator)- which is also not the correct test btw but whatever.The problem is that the flow that is created from that model is not an sklearn extension flow, because that's created by
get_extension_by_flow, and the sklearn.extension module doesn't set include sklearn in itsexternal_version, it's only including the sklearn version in the tags:openml-python/openml/extensions/sklearn/extension.py
Line 420 in 56fcc00
There are two obvious fixes:
a) When creating the flow, allow setting the extension directly, because we know what the extension is supposed to be.
b) include the sklearn version in the external version
I feel we should be doing both possibly?
mfeurer commentedon Jul 23, 2019
what would you test for? The interface?
that would definitively be helpful
I hope that this won't be necessary, but we should keep it in mind in case this problem persists.
amueller commentedon Oct 14, 2019
This issue still persists with older flows:
I would really like to run flow 7777 because it's used in the definition of CC-18, but I can't because it contains the ConditionalImputer, which can't be reinstantiated with current openml (I tried to use older openml and failed as well).
mfeurer commentedon Oct 14, 2019
Two questions:
amueller commentedon Oct 15, 2019
Yes. The error is that we can only instantiate sklearn flows. This was fixed in add sklearn version to external version in sklearn flows, #742 for new flows, but this is an old flow and doesn't have sklearn in the external version, so the sklearn extension can't detect that it handles it.
That the pipeline is instantiated. The rest can be instantiated.
We can decide that we basically abandon all third-party flows that were created before #742, or we need to change how an extension detects if it can handle a flow. Hacky solution: add
study14to the modules to check for in the module to know that the extension can handle a flow.amueller commentedon Oct 15, 2019
cc @janvanrijn
amueller commentedon Oct 16, 2019
fixed in #830