Repository navigation
Custom OpenMLFlow #753
Description
I want to register a flow referring to the setup we used to conduct the automl benchmark presented at the ICML workshop this year. We want to do this to more easily share results through OpenML. I want to run this by you all.
Now, I know that creating your own flow in openml-python is discouraged ('Flows should not be generated manually'), but for this scenario I don't really see a better way? Other than using e.g. openml-r.
I want to describe that we used the code at our Github repo, and a specific tag, and provide a url to the download for the code. I figured to tag the used AutoML tool (in this case auto-sklearn).
There is no direct way to instantiate the script with the right settings currently, but flow below should help/be enough to configure a reproducible run. Again, the flow is created mainly to link and share results, and provide the best possible pointers for reproducing the flow.
Sketching out how I think the flow should look like:
auto_sklearn_flow = openml.flows.get_flow(15275) # auto-sklearn 0.5.1
amlb_flow = openml.flows.OpenMLFlow(
name='automlbenchmark_autosklearn',
description='Auto-sklearn as set up by the AutoML Benchmark',
external_version='amlb==0.9',
parameters=OrderedDict(
time='240',
memory='32',
cores='8'
),
parameters_meta_info=OrderedDict(
time=dict(data_type='int', description='time in minutes'),
memory=dict(data_type='int', description='memory in gigabytes'),
cores=dict(data_type='int', description='number of available cores')
),
language='English',
components=OrderedDict(automl_tool=auto_sklearn_flow),
)I would still need to find a place for the following information:
source: https://github.com/openml/automlbenchmark/releases/tag/v0.9
subfolder of particular interest: /frameworks/autosklearn
I do see url fields in OpenMLFlow.__init__ but they are set by the server.
I think there are a couple of questions here:
- Do you think I should even try doing this through the Python API?
And if so,
- Should I create only one flow for the entire benchmark, and take the 'automl tool' as a parameter? I think it's more generic, but might make the link to the automl tool's flow harder to find.
- Where to include the code-url.
- Any other things I need to know?
Activity
mfeurer commentedon Aug 5, 2019
In my opinion yes, it should be flexible enough to allow this to an experienced user. We should update the docs based on this experience.
I'd find that a bit weird.
Maybe that would be a good addition to the OpenML schema?
PGijsbers commentedon Aug 5, 2019
Good to hear! I agree. Due to the delay I went ahead and tried some of this already last week anyway (I figured I could delete flows if needed).
An example of a flow I created here. I opted to create a new flow for each tool used in the benchmark as they are set up in different ways (hopefully matching your intuition of doing all in one flow being 'a bit weird'). I did however include the automl frameworks as subflow wherever we used a Python tool. The script used to create the flows is here.
I think so, it would make it much clearer/easier to share flows that represent custom scripts.
I also uploaded some run results, but that was a little bit more difficult (required two minor changes).
It might be due to me trying to provide the predictions through
data_contentinstead of an arff-file.I can look into that more, and/or try if the problems also persist if I try to share arff-files directly instead.
PGijsbers commentedon Aug 12, 2019
Post for later reference.
When uploading runs for said flows, I have to circumvent the following checks:
run.publishrequires the flow to have a model. In my case I don't have the model anymore, just the predictions and parameter settings. Parameter settings were already provided through the OpenMLRun constructor. I think this should be sufficient.run.publishassumes the flow to have an extension, when dumping the arff file. I don't think this is a good assumption, users should be able to upload flows of frameworks which do not (yet) have an extension.Commenting out both publishes the run just fine, example.
Feature #753 (#932)
mfeurer commentedon Sep 2, 2020
Added via #932.