You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[MNT] diagnose and address long test runtimes #1633
"production" and "sklearn" included tests are responsible for the major time sink
some tests are taking too long, though they involve a single simple API call
it could be something with the server, which is not able to hold the load of multiple jobs from multiple PRs
though I believe it should be sporadic, but some specific tests are taking too long, even when run locally, these single tests can take up to 30-60 mins to exexute, here are some noted from a few recent jobs
The slowest tests should take no longer than 1-2 minutes. These are tests that:
run some local experiment that's time consuming (we should probably find lightweight equivalents)
require a cronjob to run on the server (e.g., uploading a dataset and checking it's processed correctly by the server)
There should be no more than a few of those in the entire test suite. Any test that takes longer, especially such as reported above, is caused by connectivity issues or server-sided issues.
I think it would be useful to look at longer running tests (such as the ones that require the server cronjob) and make sure they can run without a server (e.g., through mocking). This is beneficial twice: our tests are more consistent and faster, but also those tests typically poll the server and for that reason may be more likely to cause server-sided issues in the first place.
please mention down first how you are going to fix it, which test cases you are going to cover and what will you do to reduce their time
Hi, Id like to try tackling this issue.
I have gone through the code base and have some questions and approaches.
Issues as ive tried to understand
Some tests call list_flows(), list_runs()etc. with no size limit so it is triggering _list_all to fetch the entire production database in batches of 10,000 records. For example, test_list_flows fetches all 15,000+ flows just to assert len >= 1500.
The as_robot fixture in conftest.py applies n_retries=20 with a sigmoid delay (approaching 60s/retry) to the entire test session. A single network error on a production test can stall and cause large delays
Several @pytest.mark.production_server tests hit the live server only because the test server lacks that data (study 34/99, setups, clustering tasks)not because they need production behavior. Can't these be replaced with mock fixtures using the create_request_response pattern already in the codebase?
Approach i'm planning:
Cap uncapped listing calls with size=N where the assertion doesn't actually need the full dataset
Reduce n_retries in the as_robot fixture
Replace single-GET production tests with XML fixture mocks following the existing pattern in test_dataset_functions.py
Questions:
For tests like test_list_flows where the assertion is len >= 1500 is it okay to reduce this to something like len >= 100 with size=200, or would you prefer keeping the original assertion and instead just mocking the full response?
For the as_robot retry count is n_retries=3 an acceptable floor, or is this intentional?
Because as mentioned above in thread I also think such a long delay is not required. From what I understand, such a long delay should only be for prod environment, isnt it fine if for just test suite, it fails faster?
Should the mocked tests keep the @pytest.mark.production_server marker (just using a mock instead of the real server), or should that marker be removed entirely if run without production server?
Happy to discuss further before diving in!
@kronzter yes you are looking in the right direction
For tests like test_list_flows where the assertion is len >= 1500 is it okay to reduce this to something like len >= 100 with size=200, or would you prefer keeping the original assertion and instead just mocking the full response?
Ultimately for any given test-case it's better to mock than rely on live server call (unless it's an explicit integration test)
For mocking this is the relevant issue: #1649
CC: @PGijsbers
For the as_robot retry count is n_retries=3 an acceptable floor, or is this intentional?
Because as mentioned above in thread I also think such a long delay is not required. From what I understand, such a long delay should only be for prod environment, isnt it fine if for just test suite, it fails faster?
I would prefer a lower number and would be fine with n_retries=3, but would leave this question for @PGijsbers since they set it up to this value in the first place.
Should the mocked tests keep the @pytest.mark.production_server marker (just using a mock instead of the real server), or should that marker be removed entirely if run without production server?
Happy to discuss further before diving in!
yes you can remove the marker if it's not sending an api call to the production server
@kronzter yes you are looking in the right direction
For tests like test_list_flows where the assertion is len >= 1500 is it okay to reduce this to something like len >= 100 with size=200, or would you prefer keeping the original assertion and instead just mocking the full response?
Ultimately for any given test-case it's better to mock than rely on live server call (unless it's an explicit integration test) For mocking this is the relevant issue: #1649 CC: @PGijsbers
For the as_robot retry count is n_retries=3 an acceptable floor, or is this intentional?
Because as mentioned above in thread I also think such a long delay is not required. From what I understand, such a long delay should only be for prod environment, isnt it fine if for just test suite, it fails faster?
I would prefer a lower number and would be fine with n_retries=3, but would leave this question for @PGijsbers since they set it up to this value in the first place.
Should the mocked tests keep the @pytest.mark.production_server marker (just using a mock instead of the real server), or should that marker be removed entirely if run without production server?
Happy to discuss further before diving in!
yes you can remove the marker if it's not sending an api call to the production server
Will await confirmation regarding retry count and then push a fix.
@kronzter please see this comment #1633 (comment). Can you collect the relevant tests mentioned there? You can run these tests locally using the markers "production_server" and "test_server" to run specific tests.
As for the lagging tests in the previous threads, only some are the same.
But even then, they are comparatively faster
Some examples are : test_get_study_old 3740s to 17.8s test_list_clustering_task 2094s to 6.3s test_get_runs_list_by_tag 1980s to 22s
ps: not sure why pastebin doesnt work. i have accordingly added gists.
Hi @fkiraly,
I’d like to work on this issue.
My plan is to profile the test suite using pytest --durations to identify the slowest tests and then investigate whether repeated API calls or fixture setup are causing the delay.
Please let me know if this approach aligns with your expectations.
This issue was automatically closed by #1692 but that only added some guidelines on how to identify long running tests and does not actually reduce test time, so I am keeping this issue open for reference.
Activity
geetu040 commentedon Feb 16, 2026
Some info from this run: https://github.com/openml/openml-python/actions/runs/22063995363/:
"production" and "sklearn" included tests are responsible for the major time sink
some tests are taking too long, though they involve a single simple API call
it could be something with the server, which is not able to hold the load of multiple jobs from multiple PRs
though I believe it should be sporadic, but some specific tests are taking too long, even when run locally, these single tests can take up to 30-60 mins to exexute, here are some noted from a few recent jobs
fkiraly commentedon Feb 16, 2026
yes, looks like it is coming from "production" - here is a list with runtimes from the
durationsflag:PGijsbers commentedon Feb 18, 2026
The slowest tests should take no longer than 1-2 minutes. These are tests that:
There should be no more than a few of those in the entire test suite. Any test that takes longer, especially such as reported above, is caused by connectivity issues or server-sided issues.
I think it would be useful to look at longer running tests (such as the ones that require the server cronjob) and make sure they can run without a server (e.g., through mocking). This is beneficial twice: our tests are more consistent and faster, but also those tests typically poll the server and for that reason may be more likely to cause server-sided issues in the first place.
geetu040 commentedon Feb 18, 2026
also since the connection_n_retries is set to 5 with a longer delay with each retry, a failing test will take even longer to fail
would it be fine to set retries to possibly 3?
areebahmeddd commentedon Feb 19, 2026
Hi, could i pick this issue up? thanks!
geetu040 commentedon Feb 19, 2026
please mention down first how you are going to fix it, which test cases you are going to cover and what will you do to reduce their time
kronzter commentedon Feb 19, 2026
Hi, Id like to try tackling this issue.
I have gone through the code base and have some questions and approaches.
Issues as ive tried to understand
list_flows(), list_runs()etc. with no size limit so it is triggering_list_allto fetch the entire production database in batches of 10,000 records. For example,test_list_flowsfetches all 15,000+ flows just to assertlen >= 1500.as_robotfixture in conftest.py appliesn_retries=20with a sigmoid delay (approaching 60s/retry) to the entire test session. A single network error on a production test can stall and cause large delays@pytest.mark.production_servertests hit the live server only because the test server lacks that data (study 34/99, setups, clustering tasks)not because they need production behavior. Can't these be replaced with mock fixtures using thecreate_request_responsepattern already in the codebase?Approach i'm planning:
size=Nwhere the assertion doesn't actually need the full datasetn_retriesin theas_robotfixturetest_dataset_functions.pyQuestions:
test_list_flowswhere the assertion islen >= 1500is it okay to reduce this to something likelen >= 100with size=200, or would you prefer keeping the original assertion and instead just mocking the full response?as_robotretry count isn_retries=3an acceptable floor, or is this intentional?Because as mentioned above in thread I also think such a long delay is not required. From what I understand, such a long delay should only be for prod environment, isnt it fine if for just test suite, it fails faster?
@pytest.mark.production_server marker(just using a mock instead of the real server), or should that marker be removed entirely if run without production server?Happy to discuss further before diving in!
geetu040 commentedon Feb 19, 2026
@kronzter yes you are looking in the right direction
Ultimately for any given test-case it's better to mock than rely on live server call (unless it's an explicit integration test)
For mocking this is the relevant issue: #1649
CC: @PGijsbers
I would prefer a lower number and would be fine with
n_retries=3, but would leave this question for @PGijsbers since they set it up to this value in the first place.yes you can remove the marker if it's not sending an api call to the production server
kronzter commentedon Feb 19, 2026
Will await confirmation regarding retry count and then push a fix.
geetu040 commentedon Feb 20, 2026
@kronzter please see this comment #1633 (comment). Can you collect the relevant tests mentioned there? You can run these tests locally using the markers "production_server" and "test_server" to run specific tests.
kronzter commentedon Feb 20, 2026
Hi @geetu040 ,
I've grouped them basis of
test_serverandproduction_serverWith 40 longest durations :
Check snippet
test_server
Production_server
Except the one that failed looks like only one has run durations even reaching 90 secs
Rest are under a minute.
Please let me know if ive missed something.
geetu040 commentedon Feb 20, 2026
The link is not working. It's better to drop the longest durations, we already had some in the initial comments, are they the same?
kronzter commentedon Feb 20, 2026
for test_server marked tests:
87.80s call tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_gridsearch69.04s call tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_data_edit_critical_field63.33s call tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_maskedarrays47.97s call tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_column_transformer_pipeline44.03s call tests/test_runs/test_run.py::TestRun::test_offline_and_online_run_identical41.60s call tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_randomsearch38.38s call tests/test_runs/test_run.py::TestRun::test_publish_with_local_loaded_flow33.33s call tests/test_datasets/test_dataset_functions.py::TestOpenMLDataset::test_data_edit_cannot_edit_critical_field_if_dataset_has_task30.58s call tests/test_runs/test_run_functions.py::TestRun::test_learning_curve_task_229.56s call tests/test_runs/test_run_functions.py::TestRun::test__run_exists28.93s call tests/test_runs/test_run_functions.py::TestRun::test_run_and_upload_linear_regressionfull log
for production_server marked tests:
47.16s call tests/test_flows/test_flow_functions.py::TestFlowFunctions::test_list_flows41.10s call tests/test_flows/test_flow_functions.py::TestFlowFunctions::test_list_flows_output_format38.24s call tests/test_runs/test_run_functions.py::TestRun::test_online_run_metric_score27.05s call tests/test_evaluations/test_evaluation_functions.py::TestEvaluationFunctions::test_evaluation_list_sort23.92s call tests/test_evaluations/test_evaluation_functions.py::TestEvaluationFunctions::test_list_evaluations_setups_filter_flow22.28s call tests/test_runs/test_run_functions.py::TestRun::test_get_runs_list_by_tag17.83s call tests/test_study/test_study_functions.py::TestStudyFunctions::test_get_study_old14.34s call tests/test_runs/test_run_functions.py::TestRun::test_get_runs_pagination14.19s call tests/test_flows/test_flow_functions.py::TestFlowFunctions::test_list_flows_paginate13.72s call tests/test_runs/test_run_functions.py::TestRun::test_get_runs_list_by_task13.22s call tests/test_runs/test_run_functions.py::TestRun::test_get_runs_list_by_filtersfull log
As for the lagging tests in the previous threads, only some are the same.
But even then, they are comparatively faster
Some examples are :
test_get_study_old3740s to 17.8stest_list_clustering_task2094s to 6.3stest_get_runs_list_by_tag1980s to 22sps: not sure why pastebin doesnt work. i have accordingly added gists.
Abhishek9639 commentedon Feb 21, 2026
Hi @fkiraly,
I’d like to work on this issue.
My plan is to profile the test suite using pytest --durations to identify the slowest tests and then investigate whether repeated API calls or fixture setup are causing the delay.
Please let me know if this approach aligns with your expectations.
[MNT] Diagnose and address long test runtimes (openml#1633)
[MNT] Diagnose and address long test runtimes (openml#1633)
[MNT] Diagnose and address long test runtimes (openml#1633)
[DOC] Add instructions for investigating slow tests (#1633) (#1692)
PGijsbers commentedon May 22, 2026
This issue was automatically closed by #1692 but that only added some guidelines on how to identify long running tests and does not actually reduce test time, so I am keeping this issue open for reference.