Skip to content

Latest commit

 

History

History

loadtest

README.md

Load tests

Manual transport comparison scripts for the Runloop Python SDK. These hit the real API and are not part of pytest; run on demand.

End-to-end transport comparison (real API)

All scripts require RUNLOOP_API_KEY. Set RUNLOOP_BASE_URL to override the default endpoint (https://api.runloop.ai).

Every script sends devboxes.create requests against a deliberately nonexistent blueprint (bp_nonexistent_loadtest_00000), so every request fails fast server-side (HTTP 400) and no devboxes are created — isolating client + server request handling from provisioning.

Script Transport under test
loadtest.py The SDK itself (AsyncRunloop). USE_HTTP2=0 → HTTP/1.1; default / USE_HTTP2=1 → HTTP/2 (shared httpx pool). Always runs against the installed package in this checkout.
h2_test.py Raw httpx HTTP/2, bypassing the SDK. Configurable connection count.
h2_single_conn.py Raw httpx HTTP/2 on a single warmed connection (50-request burst).
raw_fetch_test.py Raw httpx HTTP/1.1 keep-alive baseline.
alpn_check.py Confirms the origin negotiates h2 via TLS ALPN.
pool_check.py Verifies that multiple sync Runloop instances share a single connection pool (transport object identity check, no real requests).

The raw-transport probes compare httpx HTTP/2 multiplexing against HTTP/1.1 directly — the same comparison that motivates the SDK's shared HTTP/2 pool. They're kept so the comparison stays reproducible.

pool_check.py is a lightweight sanity check (no API key, no real requests) that confirms multiple sync Runloop instances reuse a single underlying transport object — preventing file-descriptor exhaustion when many clients are instantiated.

# Install the SDK from this checkout (editable):
cd /path/to/api-client-python && uv sync

# SDK: HTTP/2 (default) vs HTTP/1.1, 2000-request burst
source ~/env && REQUEST_COUNT=2000 uv run python loadtest/loadtest.py            # HTTP/2
source ~/env && REQUEST_COUNT=2000 USE_HTTP2=0 uv run python loadtest/loadtest.py # HTTP/1.1

# Raw httpx HTTP/2 vs HTTP/1.1 comparison
source ~/env && uv run python loadtest/h2_test.py
source ~/env && uv run python loadtest/raw_fetch_test.py

# Single-connection burst
source ~/env && uv run python loadtest/h2_single_conn.py

# ALPN check
source ~/env && uv run python loadtest/alpn_check.py

HTTP/1.1 opens a socket per in-flight request; for large bursts raise the file-descriptor limit (ulimit -n 65536) or keep REQUEST_COUNT small.

Environment variables

Variable Default Description
RUNLOOP_API_KEY (required) API key
RUNLOOP_BASE_URL https://api.runloop.ai Override API endpoint
REQUEST_COUNT 100000 (loadtest.py) / 10000 (h2_test.py) / 500 (raw_fetch_test.py) Total requests
NUM_CONNECTIONS 10 (h2_test.py) / 20 (raw_fetch_test.py) Parallel connections
USE_HTTP2 1 0 to force HTTP/1.1 in loadtest.py