Repository navigation
New evaluation listing call that includes hyperparameters #737
Description
Currently, if one wants to download all hyperparameters and evaluations of a specific flow, they need to first download the list of evaluations (fast), and then download runs one by one to get the hyperparameters (slow).
This is a problem for the current Frontend, plus it also complicates downloading meta-data from the server (last time I tried this it worked but took a very long time).
Since this is such a common use case, should we add an API call for that? Basically something
returning something like:
{
"evaluations": {
"evaluation": [
{
"run_id": "1",
"task_id": "68",
"flow_id": "61",
"function": "area_under_roc_curve",
"upload_time": "2014-04-06 23:30:40",
"value": "0.839359",
"array_data": "[0,0.99113,0.898048,0.874862,0.791282,0.807343,0.820674]",
"setup_parameters": "parameter": [
{
"full_name": "weka.JRip(1)_F",
"parameter_name": "F",
"value": "3"
},
{
"full_name": "weka.JRip(1)_N",
"parameter_name": "N",
"value": "2.0"
},
...]
}
Different options:
- Add it as an option
return_setups=trueto the current evaluation list call. It would only work if there is a specific flow and function, otherwise return an error. - Add it as a new call `list_evaluations_and_setup(flow_id,function)
- Use the current setup/id call, but that has the same issue that you need to ask for the setups one by one, plus the evaluation list does not return the setup id.
Thoughts?
Activity
joaquinvanschoren commentedon Jul 9, 2019
In the meeting today Jan argued to not do this in the REST API but to download the evaluations and setup separately and merge them.
This could done (e.g. in python) by:
janvanrijn commentedon Jul 9, 2019
This function could be implemented in Python API. A more elaborate and useful version of this is available in the openml-python-contrib package and wrapped in the sklearn-bot.
joaquinvanschoren commentedon Jul 9, 2019
I agree that this is a lot more powerful.
For the frontend, we just need something really simple and quick to generate an overview plot like this one: https://www.openml.org/f/65, ideally without any other dependencies.
joaquinvanschoren commentedon Jul 9, 2019
Small observation about the evaluation list. Example: https://www.openml.org/api/v1/xml/setup/list/flow/65
The flow_id and flow name is repeated with every hyperparameter instead of once with the setup.
Would it make sense to only output these once to speed up the I/O?
janvanrijn commentedon Jul 9, 2019
Hyper parameter flow id != flow id, because of subflows
joaquinvanschoren commentedon Jul 9, 2019
I created the documentation for the setup list (working backwards from the API code):
https://www.openml.org/api_docs#!/setup/get_setup_list_filters
Please check.
Merge pull request #747 from openml/add_#737
mfeurer commentedon Aug 5, 2019
@PGijsbers @sahithyaravi1493 is this closed by #747?
PGijsbers commentedon Aug 5, 2019
I wasn't sure. The added function kind of allows this, but how parameters are specified might not be ideal.
Ideally I think we'd like to also see
But perhaps @joaquinvanschoren is best to judge if it should be closed?
joaquinvanschoren commentedon Aug 12, 2019
OK, I see two good solutions:
{1: {hp1: value, hp2: value},
2: {hp1: value, hp2: value}}
@PGijsbers what do you think?
PGijsbers commentedon Aug 12, 2019
I think the first would be a good default, with the second being allowed through an optional argument if exactly one flow is specified? (e.g.
parameter_values_in_separate_columns, which raises aValueErrorif it is set toTruewith more than one flow)I think the one column per hyperparameter will be a common use-case when only a single flow is retrieved (e.g. in studies that study hyperparameter importance of a flow).
mfeurer commentedon Aug 19, 2019
Additional request for this issue: we need an example on how to use this API call.
joaquinvanschoren commentedon Aug 20, 2019
It looks like the setup API does not allow us to see the structure of subflows:
https://www.openml.org/api/v1/setup/1
You could have many complex nested subflows but there is no information about which flow is a child of which parent.
Add #737 (#772)
mfeurer commentedon Sep 13, 2019
@joaquinvanschoren @PGijsbers is this solved via #772?
PGijsbers commentedon Sep 16, 2019
Yes, good catch. Sorry I did not close the issue on merge.