Skip to content

Add task rollback if task download fails - #338

Merged
amueller merged 2 commits into
developfrom
fix_302
Oct 11, 2017
Merged

amueller merged 2 commits into
developfrom
fix_302

Conversation

@mfeurer

@mfeurer mfeurer commented Oct 9, 2017

Copy link
Copy Markdown
Collaborator

This PR adds a rollback if task download fails (fixes #302).

@mfeurer
mfeurer requested review from amueller and janvanrijn October 9, 2017 15:35
@codecov-io

codecov-io commented Oct 9, 2017 •

Copy link
Copy Markdown

Codecov Report

Merging #338 into develop will increase coverage by 0.62%.
The diff coverage is 88.67%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop     #338      +/-   ##
===========================================
+ Coverage    89.72%   90.35%   +0.62%     
===========================================
  Files           32       32              
  Lines         2510     2684     +174     
===========================================
+ Hits          2252     2425     +173     
- Misses         258      259       +1
Impacted Files Coverage Δ
openml/tasks/functions.py 86.73% <88.09%> (+0.16%) ⬆️
openml/datasets/functions.py 91.15% <90.9%> (+0.51%) ⬆️
openml/runs/functions.py 90.48% <0%> (+2.59%) ⬆️
openml/evaluations/functions.py 93.33% <0%> (+3.08%) ⬆️
openml/setups/functions.py 93.84% <0%> (+3.36%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 6c25b86...922554c. Read the comment docs.

Comment thread openml/tasks/functions.py Outdated
----------
"""
try:
os.rmdir(tid_cache_dir)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why two different tries? Why not just shutil?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Historic reasons... I changed this, and also changed this for the dataset functions.

Comment thread openml/tasks/functions.py
os.makedirs(task_cache_dir)
except (OSError, IOError):
# TODO add debug information!
pass

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why should this pass? That seems like a severe error! (unless it already exists, but we're not testing that).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. I just changed this, also for the dataset functions.

Comment thread openml/tasks/functions.py
try:
with io.open(xml_file, encoding='utf8') as fh:
task = _create_task_from_xml(fh.read())
task = _get_task_description(task_id)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't get why the logic changed so much here...

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make it look like getting a dataset. Things are now in their own little functions.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hm ok. I can try to go through the details. I mean it looks ok but I haven't looked through the details.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The tests are passing and I just shuffled some lines, so don't think there's something weird going on here.

@amueller
amueller merged commit 6c53531 into develop Oct 11, 2017
@mfeurer
mfeurer deleted the fix_302 branch October 11, 2017 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cache mechanism can become corrupt

3 participants