Skip to content

feat(suite benchmarking): add progress tracking - #1498

Closed
anujagrawal380 wants to merge 2 commits into
openml:developfrom
anujagrawal380:feat/add-progress-tracking-suite-benchmarking
Closed

anujagrawal380 wants to merge 2 commits into
openml:developfrom
anujagrawal380:feat/add-progress-tracking-suite-benchmarking

Conversation

@anujagrawal380

Copy link
Copy Markdown

Description

Adds run_suite_with_progress() function to provide real-time progress feedback when benchmarking models across OpenML suites.

Motivation

When running benchmarks on large suites (e.g., OpenML-CC18), users currently have no visibility into progress, making it difficult to:

  • Monitor long-running experiments
  • Estimate completion time
  • Debug stuck/slow tasks

This is particularly important for the AI-on-Demand Platform where researchers run multi-hour benchmarks.

Related Issue

#1497

Changes

  • New function: openml.study.run_suite_with_progress()
  • Progress bar with tqdm showing completion, ETA, and speed
  • Full backwards compatibility (no breaking changes)
  • Comprehensive tests (3 test cases)
  • Tutorial with examples

Example Usage

from sklearn.ensemble import RandomForestClassifier
from openml.study import run_suite_with_progress

clf = RandomForestClassifier(n_estimators=10, random_state=42)
runs = run_suite_with_progress(suite_id=334, model=clf)

Output

Benchmarking OpenML-Tiny: 100%|████████████| 7/7 [01:35<00:00, 13.6s/task]

@PGijsbers

Copy link
Copy Markdown
Collaborator

Sorry for the slow response. I think at this time we are not looking to increase the surface area of the API for features which are easily taken care of by a user (such as writing a loop with tqdm progress bar). If we were to do something like this in the future, I think it would be more likely that we would consider expanding run_model_on_task to take a task list as opposed to adding a new separate function.

But for now I will close this PR, but thanks for your effort.
If you are still looking to contribute, we are starting to have some more availability to engage in discussions on the issues, which is advised before moving on the create a PR.

@PGijsbers PGijsbers closed this Dec 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants