Repository navigation
Quick summaries of openml objects #653
Description
Description
I often see users struggle with interpreting what an openml object (e.g. a dataset or a pipeline) really is and what it contains. For beginning users, it is also unclear what they can do to get more insight. Nobody likes to run print(vars(flow), and going to the website to get more information is a bit troublesome (and not obvious).
What about we provide a summary() function for dataset, tasks, flows, and runs that provides a quick but rich overview of the object?
Expected Results
For instance, running something like
import openml
data = openml.flows.get_dataset(1)
data.summary()could output a snippet of the description, a small table with statistics, maybe even a small table with the first couple of rows and columns.
Likewise, running
flow = openml.flows.get_flow(1)
flow.summary()Could print out a list (or graph) of the pipeline components.
Likewise, task.summary() could return the task details, and run.summary() could return some run details and evaluation results.
In a notebook environment, you could even include some nice pandas tables and plots, but that will always work.
Activity
janvanrijn commentedon Mar 26, 2019
I agree that we need a string representation of objects, the absence of these seemed to be confusing for our aspiring new users on the tutorial last Saturday.
Isn't this what the
__str__and__repr__functions do? It is for example implemented forOpenMLRunobjects, but as far as I know not for other objects.joaquinvanschoren commentedon Mar 26, 2019
If the output is pure text, then we can do it with standard str. I just wanted to think more broadly about what would be really useful and intuitive.
print(run) currently returns:
[run id: 2, task id: 72, flow id: 75, flow name: weka.AdaBoostM1_DecisionSt...]
What I would love to see is something like:
Task: ID: 72, breast-cancer (version 1)
Flow: ID: 75, weka.AdaBoostM1_DecisionStump() ...
Uploader: Jan van Rijn
AUC: 0.8
Accuracy: 0.9
mfeurer commentedon Mar 27, 2019
How would that be different than the output of
__repr__or__str__(under the assumption that someone implements that)?Potential duplicate of #75.
PGijsbers commentedon Mar 28, 2019
It definitely overlaps with #75. Though I think a
__repr__implementation should be more succinct and this should be either__str__or its own method. To me, the differences could be e.g.:run
__repr__:{Run {id}|Local run} of {Flow {id}|local flow} on Task {id} (by uploader {username}).Example:
Run 2 of Flow 75 on Task 72 by Jan van Rijn.__str__: as suggested by @joaquinvanschorentask
__repr__:Task {id}, {tasktype} on dataset {id} evaluating {metric} for {target} with {estimation procedure}.Example:
Task 59, Supervised Classification on dataset 61 evaluating predictive_accuracy for class with 10-fold Crossvalidation.__str__: As this is more verbose again, I'll just give an exampleIn general
__repr__should succinctly show the most relevant information for debugging/identification purposes, and__str__would basically show what you would find going to the openml webpage.Note that
__repr__will be used to represent the object when e.g. looking at it in a notebook or console by just specifying the variable, and the__str__representation is used withprintor explicitstrcall:nok commentedon Apr 29, 2019
A method like
summaryremember me to Tensorflow where you see a constructed architecture of a neural network which is immutable.For me the methods
__str__and__repr__print the current internal state of an instance which is mutable. Further I found a discussion on stackoverflow: https://stackoverflow.com/a/2626364/1293700In general I like the idea 👍 .
mfeurer commentedon Apr 30, 2019
I just checked, but Tensorflow's summary is a module, and not a method of a network/graph, right?
BTW: this issue is up for grabs ;)
nok commentedon Apr 30, 2019
My bad, it was Keras 🤦♂ https://stackoverflow.com/a/38484187/1293700
PGijsbers commentedon May 2, 2019
Can you specify in particular how this would deviate from the examples above? From your stackoverflow reference description of
__str__:I feel the proposal of essentially a text representation of the website information achieves these goals?
Exactly recreating a task based on it's
__repr__is going to be complicated as you'll need to specify splits, which I think is a bad idea. From the information to be used proposed above, you can kind of recreate the task on a functional level.PGijsbers commentedon Jun 12, 2019
Added in #701