Repository navigation
openml-python does not recognize corrupt dataset.pkl.py3 file #779
Description
People are experiencing issues downloading dataset 41081 here.
Investigation is on-going, but it looks like the Python process gets killed when pickling the dataset to disk.
Subsequent loads of the cached data fail because the dataset.pkl.py3 file is created (and sits at 0 bytes). Because the file exists it is attempted to be loaded without further sanity checks. I think the correct behavior would to check for this faulty file and if it exists either (1) raise an error or (2) delete the file, attempt to pickle is again, and raise an error only if this fails. I would prefer option 2.
edit: to clarify, this issue is about trying to mitigate the effect of corrupt files in cache. Not about the fact that the process gets killed (possibly during pickling). That warrants a separate issue which I will open after we have some more information from the linked issue.
Activity
mfeurer commentedon Sep 12, 2019
Checking if the file is corrupt is a great idea. One could then delete the corrupt file and try to pickle it again. However, that'll most likely fail, too, so we need to figure out a different way to handle this.
PGijsbers commentedon Sep 12, 2019
A follow up from the student confirmed that the pickling process also fails if the
arfffile is also present (which is expected, but good to know for sure). Because the reported issue is the process being killed, I am not sure that there is anything we can do to mitigate issues immediately. We will have to report the corrupt file before retrying to pickle.mfeurer commentedon Sep 12, 2019
Could you ask the student to put a try/except around the pickle command and print the error message? Maybe it's a MemoryError and we can actually catch it and continue?
PGijsbers commentedon Sep 18, 2019
If the file is corrupt, we could also just load the data from the arff file (in addition to removing the file and giving a warning)?
mfeurer commentedon Sep 20, 2019
Yes, that would be a good first step. The question though is why the file is corrupt and whether we would try to pickle it a second time. Maybe we can add a flag for that?
PGijsbers commentedon Sep 23, 2019
The file would be corrupt due to the process getting killed while pickling happens.
I think by default you probably don't want to retry pickling.
If you would retry it by default and the errors are as severe as the process getting killed, you'd need to first persist something to disk noting that you've tried to pickle it (that way if it does get killed next time around we can know that). I think having the (few) people that are affected by this load the dataset slightly slower from cache because they are parsing the arff file is the better solution while we work on a fix of the underlying issue.
mfeurer commentedon Sep 26, 2019
Okay, fine by me. If we add a warning message, the user can decide on its own whether to delete the corrupt file. Will you go ahead and create a PR?
PGijsbers commentedon Sep 26, 2019
Ok, I'll work on it tomorrow.