Roadmap

@janvanrijn

As we have a new student research assistant starting on the OpenML python connector, I briefly thought of a roadmap on how to proceed with the python interface:

0.6: Bugfixes and first pypi release, ideally done before the workshop in the beginning of April
0.7: Improved upload of datasets and tasks, ideally done after the workshop in the beginning of April
0.8: Full support of pandas: handling of strings and correct handling of categorical attributes, returning DataFrames instead of dictionaries of dictionaries in listing calls.
0.9: Supporting tasks other than classification (regression, clustering)
1.0: Providing a clean interface to let other packages talk to the OpenML connector.

@janvanrijn @amueller @joaquinvanschoren @ArlindKadra what do you guys think?

What are the particular bugfixes you had in mind for the 0.6 version? Isn't 0.6 the current version btw?

@ArlindKadra

Good catch, so everything should be incremented (and instead of 1.0 and 1.1 we'd have 0.10 and 1.0). Fixes I had in mind (and @ArlindKadra will work on them):

It would be great to support regression as well: #312

Step 1 would be to upload a single regression run. The definition for the predictions is in the regression task type: https://www.openml.org/api_docs#!/tasktype/get_tasktype_id

Step 2 is to document how traces can be uploaded as well: openml/OpenML#413

@joaquinvanschoren

@joaquinvanschoren I think regression and other tasks are a separate work package and worth getting done rather sooner than later, but in my opinion, right now it's more important to have the current functionality in a bug-free state again.

Another large milestone: better handling of uploads when running flows and/or runs: #457, #219

dataset upload and pandas integration seem essential, generally this looks good.

I just updated the roadmap given the recent 0.8 release. Please feel free to coment.

0.9:
- Handling of regression and clustering tasks.
- Removal of python2.
- Start using type annotations.
- Upload/creation of tasks.
- Full support of pandas: handling of strings and correct handling of categorical attributes, returning DataFrames instead of dictionaries of dictionaries in listing calls.
- Providing a clean interface to let other packages talk to the OpenML connector.
0.10: Better handling of uploads when running flows and/or runs: Why does run_flow_or_task always upload the flow? #457, Cache runs for later upload #219

Questions regarding refactoring for Python3:

Do we want to maintain support for passing both strings and ints as arguments (for e.g. ids). If so, I assume we document this with Union[string, int] type hints?
Do we want to start using .format to format strings instead of the % operator?

Do we want to maintain support for passing both strings and ints as arguments (for e.g. ids). If so, I assume we document this with Union[string, int] type hints?

Sounds good to me.

* Do we want to start using `.format` to format strings instead of the `%` operator?

Is there any advantage doing so? If not I do not see a reason to adapt the code.

Is there any advantage doing so? If not I do not see a reason to adapt the code.

Mostly in dealing with tuples and dictionaries (see the docs). I believe I have read % is officially deprecated though without any plans to actually remove it, but I can not find a first hand source for this easily.
In my opinion, the fact that the % string-operator is unique to python while a fine legible alternative is present also weighs against using it. It removes extra nuisance for people coming from other languages. I am not necessarily in favor of refactoring everything in one swoop (though possible), but at least in favor of phasing them out as work is being done on the package.

Thanks for bringing up these arguments. I think these are good reasons to change the practice. Let's try to do this as we work on the package.

Is there any advantage doing so? If not I do not see a reason to adapt the code.

Mostly in dealing with tuples and dictionaries (see the docs). I believe I have read % is officially deprecated though without any plans to actually remove it, but I can not find a first hand source for this easily.
In my opinion, the fact that the % string-operator is unique to python while a fine legible alternative is present also weighs against using it. It removes extra nuisance for people coming from other languages. I am not necessarily in favor of refactoring everything in one swoop (though possible), but at least in favor of phasing them out as work is being done on the package.

Although I understand the benefits, I am going to mention some of the drawbacks that I see. Feel free to correct me if I am wrong.

Regarding the part about people coming from other languages. Previous to python, I mostly programmed in Java and C.
% is used in both programming languages and is common. In fact I would say that the 'old style' formatting in python was inspired by C. For the above reasons the old style formatting became familiar right away.

https://bugs.python.org/issue14123
https://lists.gt.net/python/dev/969817

The discussion in the above links seems to show that the old style formatting will be there for a long time.

Speed comparison between both:

timeit.timeit("{:d}".format(5), number=10000)
0.00022622925007453887
timeit.timeit('%d' % 5, number=10000)
0.00010349239801143995

Addressing the legible part:
Old style:
'Variable %d' % a
New Style:
'Variable {0}'.format(a)

Looking at the code, on the first one I have a clear idea that a is an integer, on the second one I do not.

Thanks for the feedback. And while I am fine with either % or .format, I still want to address your points:

% is used in both programming languages and is common.

It has been a while for me programming C, but I don't recall the % operator? I recall it in the format-string, but otherwise isn't the signature printf(char * format, ...)? Is it different for Java? I am specifically talking about the % operator, that is the % between the string and tuple. Don't get me wrong, I find both alternatives perfectly legible, but I've firsthand witnessed students getting confused over the % operator.

For the code example, why not compare it to 'Variable {:d}'.format(a)?

The speed is a valid argument for using %, but how many strings do we realistically generate in a short time-span? I think the most realistic case is querying for many tasks/datasets, and in those scenarios I imagine the time spent on server round trips, downloading data and/or loading from disk will dwarf the 1ms advantage. Though we would need some actual measurements on our package to see how relevant it is.

Thanks for the feedback. And while I am fine with either % or .format, I still want to address your points:

% is used in both programming languages and is common.

It has been a while for me programming C, but I don't recall the % operator? I recall it in the format-string, but otherwise isn't the signature printf(char * format, ...)? Is it different for Java? I am specifically talking about the % operator, that is the % between the string and tuple. Don't get me wrong, I find both alternatives perfectly legible, but I've firsthand witnessed students getting confused over the % operator.

It has been a while for me too, with what I recall, there is no % operator in both languages between the string and the vars. I was referring to the % in the format-string as you mentioned, that is familiar between all.

For the code example, why not compare it to 'Variable {:d}'.format(a)?

Because unless we are adding it to a code guideline regarding contributions, it can be an observed scenario.

The speed is a valid argument for using %, but how many strings do we realistically generate in a short time-span? I think the most realistic case is querying for many tasks/datasets, and in those scenarios I imagine the time spent on server round trips, downloading data and/or loading from disk will dwarf the 1ms advantage. Though we would need some actual measurements on our package to see how relevant it is.

Yes, I agree on this :)

Personally, I am fine with either % or .format.

@PGijsbers

Thanks for the comments. @PGijsbers would you like to add an update to the contribution guide on how to format the strings?

Will do after finishing #516.

I think everything in this roadmap is now done, and we have just released an arXiv paper https://arxiv.org/abs/1911.02490 so I will close this for now until we have some more large-scale projects to put into the Python API.

Navigation Menu

Repository navigation

Roadmap #410

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Roadmap #410

Description

Activity

janvanrijn commented on Mar 1, 2018

mfeurer commented on Mar 5, 2018

joaquinvanschoren commented on Mar 12, 2018

mfeurer commented on Mar 13, 2018

mfeurer commented on Apr 25, 2018

amueller commented on Apr 30, 2018

mfeurer commented on Feb 18, 2019

PGijsbers commented on Feb 21, 2019

mfeurer commented on Feb 22, 2019

PGijsbers commented on Feb 22, 2019

mfeurer commented on Feb 22, 2019

ArlindKadra commented on Feb 22, 2019

PGijsbers commented on Feb 23, 2019

ArlindKadra commented on Feb 23, 2019

mfeurer commented on Feb 26, 2019

PGijsbers commented on Feb 27, 2019

mfeurer commented on Nov 11, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions