fix(proxy): classify Azure Speech short audio behind a prefixed api base - #41882

TLDR

Problem this solves:

• Short audio behind a path-prefixed AZURE_SPEECH_API_BASE was billed as batch at $0
• Route family was read from the start of the upstream path

How it solves it:

• Classify by the last /speech/ vs /speechtotext/ segment in the path
• Prefixed short audio and fast transcription now price; batch stays $0

User Flow

Before: a proxy admin fronts Azure Speech with an API gateway at AZURE_SPEECH_API_BASE=https://apim.example.com/speech-proxy. Short audio calls succeed but never count against key budgets

1. Admin creates a key with max_budget: 0.001 via POST https://litellm-domain/key/generate
2. A developer sends POST https://litellm-domain/azure_speech/speech/recognition/conversation/cognitiveservices/v1?language=en-US with a 5 second wav and that key, and gets 200 with the transcript
3. They send the same request again and get 200 again
4. https://litellm-domain/spend/logs?api_key=... shows both calls as azure_speech/batch-transcription at spend 0, and the key's spend stays 0

After: the same calls are billed as short audio and the budget trips

1. Admin creates a key with max_budget: 0.001 via POST https://litellm-domain/key/generate
2. A developer sends POST https://litellm-domain/azure_speech/speech/recognition/conversation/cognitiveservices/v1?language=en-US with a 5 second wav and that key, and gets 200 with the transcript
3. They send the same request again and get 429 Budget has been exceeded
4. https://litellm-domain/spend/logs?api_key=... shows the first call as azure_speech/short-audio with non-zero spend, and the key's spend matches it

Relevant issues

Follow-up to #41557 (Bugbot finding on the merged PR)

Affected release

Linear ticket

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Setup shared by both runs: a real Azure Speech subscription key in AZURE_SPEECH_API_KEY, region swedencentral, Postgres attached, master_key: sk-1234. To stand in for an operator's API gateway, a 40 line forwarder listens on 127.0.0.1:8089, strips the /speech-proxy prefix, and forwards the request unchanged over TLS to the real swedencentral.stt.speech.microsoft.com or swedencentral.api.cognitive.microsoft.com host. Both proxies run with AZURE_SPEECH_API_BASE=http://127.0.0.1:8089/speech-proxy. Every Azure call below hit Azure and was billed there. The wav is tests/proxy_unit_tests/eagle.wav (5.06 s). Before ran on port 4011, After on 4012; $H is http://localhost:<port>

Before (6759f28)

1. VKEY=$(curl -s -X POST $H/key/generate -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"max_budget":0.001}' | jq -r .key)
2. curl -s -w "\nHTTP %{http_code}\n" "$H/azure_speech/speech/recognition/conversation/cognitiveservices/v1?language=en-US" -H "Authorization: Bearer $VKEY" -H "Content-Type: audio/wav" --data-binary @tests/proxy_unit_tests/eagle.wav

   {"RecognitionStatus":"Success","Offset":9700000,"Duration":89500000,"DisplayText":"Britain Tranquility Base. Here the eagle has landed."}
   HTTP 200
   

3. Same command again, 15 s later

   {"RecognitionStatus":"Success","Offset":9700000,"Duration":89500000,"DisplayText":"Britain Tranquility Base. Here the eagle has landed."}
   HTTP 200
   

4. curl -s -w "\nHTTP %{http_code}\n" "$H/azure_speech/speechtotext/transcriptions:transcribe?api-version=2024-11-15" -H "Authorization: Bearer $VKEY" -F "audio=@tests/proxy_unit_tests/eagle.wav" -F 'definition={"locales":["en-US"]}'

   {"durationMilliseconds":5061,"combinedPhrases":[{"text":"Listen, Tranquility Base here. The Eagle has landed."}], ...
   HTTP 200
   

5. curl -s "$H/spend/logs?api_key=$VKEY" -H "Authorization: Bearer sk-1234" | jq -r '.[] | select(.model != "") | [.model, .spend, .api_base] | @tsv'

   azure_speech/fast-transcription	0.0014058334458	http://127.0.0.1:8089/speech-proxy/speechtotext/transcriptions:transcribe?api-version=2024-11-15
   azure_speech/batch-transcription	0	http://127.0.0.1:8089/speech-proxy/speech/recognition/conversation/cognitiveservices/v1?language=en-US
   azure_speech/batch-transcription	0	http://127.0.0.1:8089/speech-proxy/speech/recognition/conversation/cognitiveservices/v1?language=en-US
   

6. curl -s "$H/key/info?key=$VKEY" -H "Authorization: Bearer sk-1234" | jq .info.spend

   0.0014058334458
   

   Two short audio calls behind the prefixed base were logged as batch at $0 and never touched the budget

After (f1b9642)

1. VKEY=$(curl -s -X POST $H/key/generate -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" -d '{"max_budget":0.001}' | jq -r .key)
2. curl -s -w "\nHTTP %{http_code}\n" "$H/azure_speech/speech/recognition/conversation/cognitiveservices/v1?language=en-US" -H "Authorization: Bearer $VKEY" -H "Content-Type: audio/wav" --data-binary @tests/proxy_unit_tests/eagle.wav

   {"RecognitionStatus":"Success","Offset":9700000,"Duration":89500000,"DisplayText":"Britain Tranquility Base. Here the eagle has landed."}
   HTTP 200
   

3. Same command again, 15 s later

   {"error":{"message":"Budget has been exceeded! Key=key (sk-...yxIg) Current cost: 0.002755555776, Max budget: 0.001","type":"budget_exceeded","param":null,"code":"429"}}
   HTTP 429
   

4. curl -s -w "\nHTTP %{http_code}\n" "$H/azure_speech/speechtotext/transcriptions:transcribe?api-version=2024-11-15" -H "Authorization: Bearer $VKEY" -F "audio=@tests/proxy_unit_tests/eagle.wav" -F 'definition={"locales":["en-US"]}'

   {"error":{"message":"Budget has been exceeded! Key=key (sk-...yxIg) Current cost: 0.0027555557759999997, Max budget: 0.001","type":"budget_exceeded","param":null,"code":"429"}}
   HTTP 429
   

5. curl -s "$H/spend/logs?api_key=$VKEY" -H "Authorization: Bearer sk-1234" | jq -r '.[] | select(.model != "") | [.model, .spend, .api_base] | @tsv'

   azure_speech/short-audio	0.002755555776	http://127.0.0.1:8089/speech-proxy/speech/recognition/conversation/cognitiveservices/v1?language=en-US
   

6. curl -s "$H/key/info?key=$VKEY" -H "Authorization: Bearer sk-1234" | jq .info.spend

   0.002755555776
   

   The first short audio call is billed as short audio and the budget blocks the rest

Gateway log for the whole run, showing the prefix stripped and the real Azure hosts hit:

POST /speech-proxy/speech/recognition/conversation/cognitiveservices/v1?language=en-US -> https://swedencentral.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=en-US 200
POST /speech-proxy/speech/recognition/conversation/cognitiveservices/v1?language=en-US -> https://swedencentral.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=en-US 200
POST /speech-proxy/speech/recognition/conversation/cognitiveservices/v1?language=en-US -> https://swedencentral.stt.speech.microsoft.com/speech/recognition/conversation/cognitiveservices/v1?language=en-US 200
POST /speech-proxy/speechtotext/transcriptions:transcribe?api-version=2024-11-15 -> https://swedencentral.api.cognitive.microsoft.com/speechtotext/transcriptions:transcribe?api-version=2024-11-15 200

Type

🐛 Bug Fix

Caveats (if any)

Low

• Fast transcription was already billed correctly behind a prefix, since it matches on the path suffix
• An api base whose prefix itself ends in /speechtotext/ is not distinguishable from batch; unlikely in practice
• The only red check at f1b9642 is the non-required proxy-infra / Run tests, failing on test_login_throttle_settings_are_not_hot_applied_from_the_database, which fails the same way on main and touches nothing in this diff

Review gate at f1b9642

Greptile 5/5, Bugbot found no new issues, Veria found no security issues, codecov patch 100%, every required check green

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR

Link to Devin session: https://app.devin.ai/sessions/2c12f8499dd642e188af5d3e207590d6
Open in Devin Desktop: https://app.devin.ai/desktop/session/2c12f8499dd642e188af5d3e207590d6?variant=devin
Requested by: @yassin-berriai