float precision #559

Activity

1. joaquinvanschoren commented on Sep 27, 2018

   

   joaquinvanschoren

   on Sep 27, 2018

   ContributorAuthor

   More actions

   Another solution could be to store the exact values (string or decimal) as a component of the Dataset object when it first reads the data. That would ensure that you could pickle and cache it and still get the exact values back?
2. mfeurer commented on Sep 28, 2018

   

   mfeurer

   on Sep 28, 2018

   Collaborator

   More actions

   Thanks for figuring this out. Indeed, the issue is the conversion of the target column into floating point representation.

   Another solution could be to store the exact values (string or decimal) as a component of the Dataset object when it first reads the data.

   That's a great idea, but doing it exactly like this will probably not work because the dataset won't know that it is a regression dataset. However, having a function called Dataset.get_column() which returns the values of a column as raw python data types could help here (and also caches it). This would then be called by the task class with the necessary argument (the column). What do you think of this solution?
3. PGijsbers commented on Apr 15, 2019

   

   PGijsbers

   on Apr 15, 2019

   Collaborator

   More actions

   An actual run on the live server is e.g. 10154904.
4. mfeurer

   added

   module:RunOpenML conceptOpenML concept

   on Feb 20, 2023
5. mfeurer

   assigned

   LennartPurucker

   on Feb 24, 2023
6. LennartPurucker commented on Feb 25, 2023

   

   LennartPurucker

   on Feb 25, 2023

   Contributor

   More actions

   So I looked into this again, and I am somewhat sure that the problem described above was a result of the problem fixed in #1209

   I am not able to reproduce the problem for task 738. The ground truth in the arff file seems to have the correct precision.

   I used the following code (with the openml's version from the current dev branch):

   from sklearn.dummy import DummyRegressor
   from openml import tasks, runs
   import openml
   
   openml.config.server = "https://test.openml.org/api/v1/xml"
   
   reg = DummyRegressor()
   task = tasks.get_task(738)
   run = runs.run_model_on_task(reg, task, avoid_duplicate_runs=False)
   run.to_filesystem("./tmp_dir")

   The exported ground truth values and their precision match the values of the downloaded dataset.

   It might be that this only looked like a precision problem because a better model than the DummyRegressor quickly approaches the ground truth with its predictions (the task is very simple to solve). Or this problem was already fixed at some point by updates to the precision of other libraries (e.g., pandas).

   Do we have another case that I could look into where this might happen/has happened?
   Otherwise, I am not sure how to proceed here.

   👍React with 👍1Reacted by Pieter Gijsbers
7. PGijsbers commented on Feb 27, 2023

   

   PGijsbers

   on Feb 27, 2023

   Last edited by PGijsbers

   Collaborator

   More actions

   I can't currently verify with that particular task (test server failure), but testing it with a task on production (id: 5514) the target features do indeed seem to behave. Both the dataset and the prediction file produce the column with :.1f formatting. Model predictions do still have arbitrary decimal points (as expected, and this should not be an issue).

   It is indeed likely that the swapped columns already existed when this issue was opened, and we were looking at the model's predictions rather than ground truth.

   Either way, ground truth is reported correctly now (on dev/in the next release), so I am closing this issue. Feel free to re-open if you find a reproducible example where the error persists. I don't think we should look into any kind of "fixing" the precision for model predictions, we should leave it as is until we move to parquet for model predictions IMO.

   👍React with 👍1Reacted by Lennart Purucker
8. PGijsbers

   closed this as completedon Feb 27, 2023