Tests: Flakyness with "active servers" logic within test suite

Hi,

on [1] and [2], I just observed a flakyness on a single job when running the tests on #397 and #402.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209']

With kind regards,
Andreas.

[1] https://github.com/crate/crate-python/actions/runs/687059879
[2] https://github.com/crate/crate-python/runs/2195227267#step:5:199

Maybe this happens because the retry interval is set to a very low value when running the tests.

crate-python/src/crate/client/test_http.py

Lines 311 to 313 in afb3b0b

    
           def setUp(self): 
        
               self.client = Client(self.servers) 
        
               self.client.retry_interval = 0.1  # faster retry

So, the invalid_host might have gotten swapped out and back in again too quickly.

I actually recognized that this specific job was executing very slow, so it probably has been scheduled on a CI machine currently experiencing more load than others.

Observed this again when running the tests on behalf of #402. So, I added 89507a0 there.

The same thing happened again at https://github.com/crate/crate-python/runs/2297326550#step:6:184.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209']

Now also, slightly different but the same issue, at https://github.com/crate/crate-python/runs/2297736340#step:6:166.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 61, in http.txt
Failed example:
    http_client._get_server()
Expected:
    'http://invalid_host:9999'
Got:
    'http://even_more_invalid_host:9999'

Same thing happens again ¹²:

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209/']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209/']

I did not observe this flaw recently, so I am closing this issue.

It still happens when runners seem to be slow.

File "/home/runner/work/crate-python/crate-python/src/crate/client/doctests/http.txt", line 41, in http.txt
Failed example:
    http_client._active_servers
Expected:
    ['http://127.0.0.1:44209/']
Got:
    ['http://invalid_host:9999', 'http://127.0.0.1:44209/']

-- https://github.com/crate/crate-python/actions/runs/3198965176/jobs/5224159502#step:4:338

Tests: Use small timeouts for server selection tests in `http.txt`

This one still happens from time to time.

-- https://github.com/crate/crate-python/actions/runs/3700515171/jobs/6269005827#step:4:565

Still happening occasionally. One re-run usually fixes the fluke.

-- https://github.com/crate/crate-python/actions/runs/9545350897/job/26305926214#step:4:651

This issue has not been observed recently, so I am closing this.

Tests: Use small timeouts for server selection tests in `http.txt`

@surister

@surister shared an evaluation about this fluke at #747 (comment):

My hypothesis is that either:

The tests are not correctly written to take into account the driver's multi-threading behavior.

There is a subtle bug in the multi-threading part of the code.

Having the right result but different order in multi-threading python code is common imo.

Thanks, that's an excellent assessment. Do you see a chance to fix this issue positively?

>       self.assertRegex(layer.crate_servers[0], r"[http://127.0.0.1:442\d\d](http://127.0.0.1:442/d/d)")
                         ^^^^^^^^^^^^^^^^^^^^^^
E       IndexError: list index out of range

-- https://github.com/crate/crate-python/actions/runs/20072360014/job/57577711112?pr=741#step:6:235

>       sql_uri = layer.crate_servers[0] + "/_sql"
                  ^^^^^^^^^^^^^^^^^^^^^^
E       IndexError: list index out of range

-- https://github.com/crate/crate-python/actions/runs/20072361183/job/57577715457?pr=745#step:6:250

Apparently, after a few refactorings, the problem is mostly happening on macOS now, see also GH-772?

Another sample from GH-750, also occurring on macOS.

FAILED tests/testing/test_layer.py::LayerTest::test_cluster - AssertionError: 1 != 2
FAILED tests/testing/test_layer.py::LayerTest::test_default_settings - AssertionError: 'http://127.0.0.1:4201/' != 'http://127.0.0.1:4200/'
- http://127.0.0.1:4201/
?                     ^
+ http://127.0.0.1:4200/
?                     ^

-- https://github.com/crate/crate-python/actions/runs/21177084017/job/60908800502?pr=750#step:6:4296

I think this spot is a leftover. Let's close this issue, and carry things forward on the new one.

Tests: LayerTest.test_dynamic_http_port is flaky #772

Navigation Menu

Repository navigation

Tests: Flakyness with "active servers" logic within test suite #404

1 remaining item

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	def setUp(self):
	self.client = Client(self.servers)
	self.client.retry_interval = 0.1 # faster retry

Tests: Flakyness with "active servers" logic within test suite #404

Description

Activity

amotl commented on Mar 25, 2021

amotl commented on Mar 26, 2021

amotl commented on Apr 8, 2021

amotl commented on Apr 8, 2021

amotl commented on Apr 28, 2022

Footnotes

amotl commented on Sep 29, 2022

amotl commented on Oct 6, 2022

1 remaining item

amotl commented on Dec 16, 2022

amotl commented on Jun 17, 2024

amotl commented on Oct 25, 2024

amotl commented on Dec 10, 2025

amotl commented on Dec 16, 2025

amotl commented on Jan 20, 2026

amotl commented on Jan 20, 2026

amotl commented on Feb 7, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions