fix(proxy): run SMTP send_email off the event loop with a connection timeout - #38473

TLDR

Problem this solves:

• Sending an email could freeze the whole proxy
• SMTP ran synchronously on the event loop with no timeout
• An unreachable SMTP host stalled /health/liveliness, so k8s killed the pod

How it solves it:

• SMTP connections now get a timeout (SMTP_TIMEOUT env var, default 30s)
• The whole SMTP session runs in a worker thread via asyncio.to_thread
• Also stops carrying forward the dead circleci codecov flag, whose stale line maps failed codecov/patch here

User Flow

Before: an operator whose SMTP server hangs sees the whole proxy stop responding and the pod get killed

1. The proxy sends any email (key created invite, budget alert, or GET https://litellm-domain/health/services?service=email) while the SMTP host accepts TCP but never answers
2. Every request to the proxy hangs, including GET https://litellm-domain/health/liveliness which times out instead of returning "I'm alive!"
3. Kubernetes liveness probes fail and the pod is killed with exit 137 and no traceback

After: the same hung SMTP server only affects the email itself

1. The proxy sends the same email while the SMTP host accepts TCP but never answers
2. GET https://litellm-domain/health/liveliness keeps returning 200 "I'm alive!" the whole time
3. After 30 seconds (or SMTP_TIMEOUT) the email attempt gives up and the error is logged; the pod stays up

Relevant issues

Linear ticket

Resolves LIT-4992

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Setup shared by both runs: a local banner-less SMTP server (accepts TCP on 127.0.0.1:2525, never sends the SMTP greeting), and the proxy started with SMTP_HOST=127.0.0.1 SMTP_PORT=2525 SMTP_TLS=False [email protected] [email protected] on port 4000. The email is triggered through /health/services?service=email, which goes through the key-created email path and ends in the shared SMTP send helper, the same helper every email path (invites, budget alerts, enterprise SMTP logger) ends in

Before (807ee7f)

1. curl -s -m 5 http://localhost:4000/health/liveliness returns "I'm alive!"
2. curl -s -m 5 "http://localhost:4000/health/services?service=email" -H "Authorization: Bearer sk-1234" & starts the test email; the SMTP connect blocks the event loop
3. curl -s -m 5 http://localhost:4000/health/liveliness while the email is in flight: no response, curl exits 28 (timed out). This is the state where the kubelet kills the pod

After (75fdd7f)

1. curl -s -m 5 http://localhost:4000/health/liveliness returns "I'm alive!"
2. curl -s -m 65 "http://localhost:4000/health/services?service=email" -H "Authorization: Bearer sk-1234" & starts the same test email against the same hung SMTP server
3. curl -s -m 5 http://localhost:4000/health/liveliness 2s into the hung SMTP send: "I'm alive!", curl exit 0
4. curl -s -m 5 http://localhost:4000/health/liveliness 12s in: "I'm alive!", curl exit 0
5. The email endpoint returns {"status":"success","message":"Mock Email Alert sent, verify Email Alert Received"} and 30s after the send started the proxy logs An error occurred while sending the email:Connection unexpectedly closed: timed out (TimeoutError: timed out in the traceback), matching the default SMTP_TIMEOUT=30

Type

🐛 Bug Fix

Caveats (if any)

Low

• Failed sends now surface after SMTP_TIMEOUT instead of hanging forever

Link to Devin session: https://app.devin.ai/sessions/d28d0fbec50b4b709720d86bca5bc416
Open in Devin Desktop: https://app.devin.ai/desktop/session/d28d0fbec50b4b709720d86bca5bc416?variant=devin
Requested by: @yassin-berriai