Short name for flows (pipelines)

The flow names are currently very descriptive - which is good - but is often also troublesome to read.

E.g. if I try to plot an overview of sklearn pipelines on a task I get this:

Maybe we can implement a simple 'short name' for pipelines? E.g. this:

sklearn.pipeline.Pipeline(imputation=openmlstudy14.preprocessing.ConditionalImputer,hotencoding=sklearn.preprocessing.data.OneHotEncoder,variencethreshold=sklearn.feature_selection.variance_threshold.VarianceThreshold,classifier=sklearn.ensemble.forest.RandomForestClassifier)

could become this (only show the names of the components):
ConditionalImputer-OneHotEncoder-VarianceThreshold-RandomForestClassifier

You mean using the field custom_name?

Looks like we can create a custom name and provide it to the generated flow for scikit-learn flows here.

I was thinking more about a way to shorten the existing names. If there is a custom name, great, but if we have these long automatic names we also need a way to shorten if for proper display.

Here's a gist with an implementation: link.
Running it gives the output:

Flow name 6651: sklearn.linear_model.logistic.LogisticRegression
Short name: sklearn.LogisticRegression
Flow name 6969: sklearn.pipeline.Pipeline(imputation=openmlstudy14.preprocessing.ConditionalImputer,hotencoding=sklearn.preprocessing.data.OneHotEncoder,variencethreshold=sklearn.feature_selection.variance_threshold.VarianceThreshold,classifier=sklearn.ensemble.forest.RandomForestClassifier)
Short name: sklearn.Pipeline(ConditionalImputer,OneHotEncoder,VarianceThreshold,RandomForestClassifier)
Flow name 6621: sklearn.model_selection._search.RandomizedSearchCV(estimator=sklearn.ensemble.forest.ExtraTreesClassifier)
Short name: sklearn.RandomizedSearchCV(ExtraTreesClassifier)
Flow name 7116: sklearn.model_selection._search.RandomizedSearchCV(estimator=sklearn.pipeline.Pipeline(imputation=openmlstudy14.preprocessing.ConditionalImputer,hotencoding=sklearn.preprocessing.data.OneHotEncoder,variencethreshold=sklearn.feature_selection.variance_threshold.VarianceThreshold,classifier=sklearn.ensemble.weight_boosting.AdaBoostClassifier(base_estimator=sklearn.tree.tree.DecisionTreeClassifier)))
Short name: sklearn.RandomizedSearchCV(Pipeline(...,AdaBoostClassifier))

As this is highly package dependent I would suggest to make this part of the extension interface.
I resorted to string parsing as opposed to re-generating a short name from the model because the latter might/will not work if there are version differences in your installed package and the flow's model's versions.

How do we integrate this nicely?

Hey, I would go one step further and suggest that the different extensions themselves have to create the short name themselves and not do this via the extension interface.

Could you please explain the issue with version differences? Since a flow is uniquely defined by the full name and the dependencies, it doesn't matter if two flows have the same short name, right?

Finally, I would create the short name in the function _serialize_model(self, model: Any) next to the creation of the full name (i.e. leave out the package name) instead of altering the full name (which implies that older flows will not have a short name, but we also shouldn't change older flows).

Could you please explain the issue with version differences?

In the case where you would generate a shortname based on the python object, you need to be able to instantiate the python object. If e.g. a user has scikit-learn 0.20 installed, but the flow uses a scikit-learn 0.18 component, it can not locally instantiate the model. However, as the full name is loaded, we can use this and some string parsing to obtain a short name.

There was the explicit requirement for older flows to also be able to have this short name generated. In either case we are not changing the original name of the flow, so I don't see the harm.

We will add this function to the sklearn extension module. For new flows the shortname will be generated and added to the flow. For old flows we will not retroactively add the short names but they can nonetheless by generated by this function (such that it can be used to generate the short names for e.g. the website).

I think we should use a separate field like a custom name for a short display.
Or think about the semantics of what the name is?

The display on openml.org certainly needs to be improved. But I'm hesitant to entangle display issues with the backend.
Are the names required to be unique? Can you remind me how name and flow correspond? I know we have several flows for the same name when the version of the libraries change. But is that the only case?
As many times before: it would be so great if we had a written down schema of how openml works. Without it it's really hard to reason about any of these things. I'm not sure what the name is used for right now on the backend and in the python wrapper. From what I remember, the name is actually used to store the class information (?!?)

Another possible solution for now would be to have the front-end shorten the name. That seems the easiest and fastest solution.
The backend has more information about the flow, though, so as a long-term solution I think having an alias or short name would make sense.

In the case where you would generate a shortname based on the python object, you need to be able to instantiate the python object. If e.g. a user has scikit-learn 0.20 installed, but the flow uses a scikit-learn 0.18 component, it can not locally instantiate the model. However, as the full name is loaded, we can use this and some string parsing to obtain a short name.

I don't understand this concern. We're talking about generating the short name at the time the flow is generated, right? At that point you have to have these object instantiated and you are in the environment the flow is running in.

@joaquinvanschoren

@joaquinvanschoren you could dramatically shorten the names on the frontend by just removing the module names.

Are the names required to be unique?

The name together with external_version must be unique (see also flow xsd).
I don't think it is enforced that two flows with the same name (but different external_version) are related in any way.

From what I remember, the name is actually used to store the class information (?!?)

This isn't the case (anymore?). Relevant information such as class and parameter values are stored as flow attributes, see deserialization and serialization.

I don't understand this concern. We're talking about generating the short name at the time the flow is generated, right?

It should also work for old flows as we want to display them on the website in a more concise manner. In that case you do not have an instantiated object.

I think we should use a separate field like a custom name for a short display.

There is, it is called custom_name (https://github.com/openml/OpenML/blob/master/openml_OS/views/pages/api_new/v1/xsd/openml.implementation.upload.xsd).

From what I remember, the name is actually used to store the class information

The name is used to provide a unique string representation of the flow. I think we have used the name for deserialization, but changed that at some point when class_name and custom_name were added to OpenML.

Ok but if they have to be unique we can't shorten them (in the main field, but we can in the custom_name), right?

Yes. We cannot shorten the main field (name), but we can shorten it in the field custom_name.

Answering comments above: we could indeed shorten it in the frontend, but I believe it is nice to have short names in any kind of plot/table/report based on the python API.

but I believe it is nice to have short names in any kind of plot/table/report based on the python API.

I do fully agree to this

Closed by #723.

Navigation Menu

Repository navigation

Short name for flows (pipelines) #412

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Short name for flows (pipelines) #412

Description

Activity

mfeurer commented on Jun 14, 2018

PGijsbers commented on Apr 15, 2019

joaquinvanschoren commented on May 13, 2019

PGijsbers commented on Jun 4, 2019

mfeurer commented on Jun 13, 2019

PGijsbers commented on Jun 13, 2019

PGijsbers commented on Jun 15, 2019

amueller commented on Jun 19, 2019

amueller commented on Jun 19, 2019

amueller commented on Jun 19, 2019

PGijsbers commented on Jun 20, 2019

mfeurer commented on Jul 8, 2019

amueller commented on Jul 9, 2019

mfeurer commented on Jul 9, 2019

joaquinvanschoren commented on Jul 10, 2019

mfeurer commented on Jul 12, 2019

mfeurer commented on Jul 24, 2019

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions