fix(proxy): /key/bulk_update writes only the fields each item carries - #41949

TLDR

Problem this solves:

• A /key/bulk_update item without max_budget set the key's max_budget to null
• Same for team_id and budget_id: bulk tagging a team key detached it from the team
• An item carrying a field the bulk route does not know (e.g. object_permission) got 200, the field was dropped, and the budget was wiped

How it solves it:

• Each bulk item now writes only the fields it carries
• A field sent explicitly, null included, is applied exactly as /key/update applies it
• A bulk item can now carry object_permission, applied the same way /key/update applies it
• That object_permission is checked against the key's team the way /key/update checks it, so a team key cannot be bulk-granted an MCP server, search tool, or vector store its team does not allow; such an item lands under failed_updates with the same reason /key/update returns

User Flow

Before: an admin who bulk-tags a budgeted key finds its budget wiped, and a team key detached from its team

1. They send POST https://litellm-domain/key/generate with {"key_alias": "team-a-key", "max_budget": 100} and get back a key with "max_budget": 100.0
2. They send POST https://litellm-domain/key/bulk_update with {"keys": [{"key": "<that key>", "tags": ["team-a"]}]} and get HTTP 200 with the key under successful_updates, its key_info already showing "max_budget": null
3. They send GET https://litellm-domain/key/info?key= and see "max_budget": null next to "metadata": {"tags": ["team-a"]}: the budget is gone
4. The same bulk call on a key that belongs to a team also comes back with "team_id": null, so the key is no longer the team's
5. A bulk item carrying object_permission gets HTTP 200 as well; the permission is never applied and that key's budget is wiped too
6. A bulk item granting a team key a search tool outside its team's allowlist also gets HTTP 200 with an empty failed_updates, where the same body on POST https://litellm-domain/key/update is refused with HTTP 403

After: the same bulk tag call changes only the tags, and a bulk item can grant an object permission

1. They send POST https://litellm-domain/key/generate with {"key_alias": "team-a-key", "max_budget": 100} and get back a key with "max_budget": 100.0
2. They send POST https://litellm-domain/key/bulk_update with {"keys": [{"key": "<that key>", "tags": ["team-a"]}]} and get HTTP 200 with the key under successful_updates, its key_info showing "max_budget": 100.0
3. They send GET https://litellm-domain/key/info?key= and see "max_budget": 100.0 next to "metadata": {"tags": ["team-a"]}
4. The same bulk call on a key that belongs to a team keeps its "team_id"
5. A bulk item carrying object_permission gets HTTP 200, and GET https://litellm-domain/key/info?key= shows it under object_permission with "max_budget": 100.0 still in place
6. A bulk item granting a team key a search tool outside its team's allowlist gets HTTP 200 with that key under failed_updates carrying the same "not allowed by team" reason POST https://litellm-domain/key/update returns as HTTP 403, and GET https://litellm-domain/key/info?key= shows the key unchanged

Relevant issues

Affected release

Linear ticket

Resolves LIT-7685

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

Screenshots / Proof of Fix

Shared setup for both sides, run the way a multi-pod deployment runs: two proxy instances (A and B) booted from the same checkout with python litellm/proxy/proxy_cli.py --config config.yaml --port $PORT --num_workers 2 --use_prisma_db_push, each with 2 uvicorn workers, both pointed at one fresh Postgres database, and every request alternated between the two instances (the key is generated on one instance, bulk-updated on the other, and read back on the first). LITELLM_MASTER_KEY=sk-lit7685-master, OPENAI_API_KEY and LITELLM_LICENSE are set (tags on a key are an enterprise field, so the license is load-bearing), no Redis, and this config.yaml:

model_list:
  - model_name: gpt-5.6
    litellm_params:
      model: openai/gpt-5.6
      api_key: os.environ/OPENAI_API_KEY

general_settings:
  store_model_in_db: true

The observed outputs below are the raw responses reduced to the relevant fields with python3 -c 'import json,sys; ...', the way the run printed them. AUTH stands for -H 'Authorization: Bearer sk-lit7685-master' -H 'Content-Type: application/json', $A and $B for the two instances' http://127.0.0.1:<port> base URLs.

Before (5fc510a)

Instance A on port 54387, instance B on port 57234, 2 workers each.

Ticket flow: bulk item with only tags

1. curl -s $A/key/generate $AUTH -d '{"key_alias":"team-a-key-25731","max_budget":100}'
{"key_alias": "team-a-key-25731", "max_budget": 100.0, "team_id": null, "budget_id": null}
2. curl -s -w '\nHTTP %{http_code}\n' $B/key/bulk_update $AUTH -d '{"keys":[{"key":"'$KEY'","tags":["team-a"]}]}'
HTTP 200
{"total_requested": 1, "successful_keys": ["sk-Ai77kcvad..."], "successful_max_budget": [null], "failed": []}
3. curl -s "$A/key/info?key=$KEY" $AUTH
{"key_alias": "team-a-key-25731", "max_budget": null, "team_id": null, "budget_id": null, "metadata": {"tags": ["team-a"]}}

Control: single /key/update with only tags

1. curl -s $B/key/generate $AUTH -d '{"key_alias":"single-update-25731","max_budget":100}'
{"key_alias": "single-update-25731", "max_budget": 100.0}
2. curl -s -o /dev/null -w 'HTTP %{http_code}\n' $A/key/update $AUTH -d '{"key":"'$KEY2'","tags":["team-a"]}'
HTTP 200
3. curl -s "$B/key/info?key=$KEY2" $AUTH
{"key_alias": "single-update-25731", "max_budget": 100.0, "metadata": {"tags": ["team-a"]}}

Control: bulk item with only max_budget

1. curl -s $A/key/generate $AUTH -d '{"key_alias":"bulk-budget-only-25731","max_budget":100,"tags":["keep-me"]}'
{"key_alias": "bulk-budget-only-25731", "max_budget": 100.0, "tags": null}
2. curl -s -o /dev/null -w 'HTTP %{http_code}\n' $B/key/bulk_update $AUTH -d '{"keys":[{"key":"'$KEY3'","max_budget":50}]}'
HTTP 200
3. curl -s "$A/key/info?key=$KEY3" $AUTH
{"key_alias": "bulk-budget-only-25731", "max_budget": 50.0, "metadata": {"tags": ["keep-me"]}}

Team key: bulk item with only tags

1. curl -s $A/team/new $AUTH -d '{"team_alias":"lit7685-team-25731"}'
team_id 3fe52997-308b-4b19-b316-09345408c8aa
2. curl -s $B/key/generate $AUTH -d '{"key_alias":"team-key-25731","max_budget":100,"team_id":"'$TEAM'"}'
{"key_alias": "team-key-25731", "max_budget": 100.0, "team_id": "3fe52997-308b-4b19-b316-09345408c8aa"}
3. curl -s -w '\nHTTP %{http_code}\n' $A/key/bulk_update $AUTH -d '{"keys":[{"key":"'$KEY4'","tags":["team-a"]}]}'
HTTP 200 successful: 1 failed: []
4. curl -s "$B/key/info?key=$KEY4" $AUTH
{"key_alias": "team-key-25731", "max_budget": null, "team_id": null, "metadata": {"tags": ["team-a"]}}

Bulk item carrying object_permission

1. curl -s $B/key/generate $AUTH -d '{"key_alias":"objperm-25731","max_budget":100}'
{"key_alias": "objperm-25731", "max_budget": 100.0}
2. curl -s -w '\nHTTP %{http_code}\n' $A/key/bulk_update $AUTH -d '{"keys":[{"key":"'$KEY5'","object_permission":{"vector_stores":["vs-1"]}}]}'
HTTP 200
{"successful_keys": ["sk--q1CCOH3e..."], "successful_max_budget": [null], "failed": []}
3. curl -s "$B/key/info?key=$KEY5" $AUTH
{"key_alias": "objperm-25731", "max_budget": null, "object_permission_id": null, "object_permission": null}

Team key: bulk item granting a search tool outside the team allowlist

1. curl -s $A/team/new $AUTH -d '{"team_alias":"lit7685-scoped-25731","object_permission":{"search_tools":["team-search"]}}'
team_id fc5c3674-434e-41b2-ae30-87eae4fd585b
2. curl -s $B/key/generate $AUTH -d '{"key_alias":"scoped-bulk-25731","max_budget":100,"team_id":"'$TEAM6'"}' and the same with key_alias scoped-single-25731
{"key_alias": "scoped-bulk-25731", "max_budget": 100.0, "team_id": "fc5c3674-434e-41b2-ae30-87eae4fd585b"}
{"key_alias": "scoped-single-25731", "max_budget": 100.0, "team_id": "fc5c3674-434e-41b2-ae30-87eae4fd585b"}
3. curl -s -w '\nHTTP %{http_code}\n' $A/key/bulk_update $AUTH -d '{"keys":[{"key":"'$KEY6'","object_permission":{"search_tools":["other-search"]}}]}'
HTTP 200
{"successful_max_budget": [null], "failed": []}
4. curl -s -w '\nHTTP %{http_code}\n' $B/key/update $AUTH -d '{"key":"'$KEY7'","object_permission":{"search_tools":["other-search"]}}'
HTTP 403
{"message": "{'error': \"Key requests search tools not allowed by team 'fc5c3674-434e-41b2-ae30-87eae4fd585b': ['other-search']. Team allows: ['team-search'].\"}", "type": "auth_error", "param": "None", "code": "403"}
5. curl -s "$A/key/info?key=$KEY6" $AUTH and the same for $KEY7
{"key_alias": "scoped-bulk-25731", "max_budget": null, "team_id": null, "search_tools": null}
{"key_alias": "scoped-single-25731", "max_budget": 100.0, "team_id": "fc5c3674-434e-41b2-ae30-87eae4fd585b", "search_tools": null}

After (df6a222)

Run at df6a222. The tip dc02e5f since then changes only the unit test file (a stubbed team lookup in the sibling policy tests), so this leg stands at the tip. Instance A on port 34759, instance B on port 29679, 2 workers each.

Ticket flow: bulk item with only tags

1. curl -s $A/key/generate $AUTH -d '{"key_alias":"team-a-key-26743","max_budget":100}'
{"key_alias": "team-a-key-26743", "max_budget": 100.0, "team_id": null, "budget_id": null}
2. curl -s -w '\nHTTP %{http_code}\n' $B/key/bulk_update $AUTH -d '{"keys":[{"key":"'$KEY'","tags":["team-a"]}]}'
HTTP 200
{"total_requested": 1, "successful_keys": ["sk-oUTvz6SIq..."], "successful_max_budget": [100.0], "failed": []}
3. curl -s "$A/key/info?key=$KEY" $AUTH
{"key_alias": "team-a-key-26743", "max_budget": 100.0, "team_id": null, "budget_id": null, "metadata": {"tags": ["team-a"]}}

Control: single /key/update with only tags

1. curl -s $B/key/generate $AUTH -d '{"key_alias":"single-update-26743","max_budget":100}'
{"key_alias": "single-update-26743", "max_budget": 100.0}
2. curl -s -o /dev/null -w 'HTTP %{http_code}\n' $A/key/update $AUTH -d '{"key":"'$KEY2'","tags":["team-a"]}'
HTTP 200
3. curl -s "$B/key/info?key=$KEY2" $AUTH
{"key_alias": "single-update-26743", "max_budget": 100.0, "metadata": {"tags": ["team-a"]}}

Control: bulk item with only max_budget

1. curl -s $A/key/generate $AUTH -d '{"key_alias":"bulk-budget-only-26743","max_budget":100,"tags":["keep-me"]}'
{"key_alias": "bulk-budget-only-26743", "max_budget": 100.0, "tags": null}
2. curl -s -o /dev/null -w 'HTTP %{http_code}\n' $B/key/bulk_update $AUTH -d '{"keys":[{"key":"'$KEY3'","max_budget":50}]}'
HTTP 200
3. curl -s "$A/key/info?key=$KEY3" $AUTH
{"key_alias": "bulk-budget-only-26743", "max_budget": 50.0, "metadata": {"tags": ["keep-me"]}}

Team key: bulk item with only tags

1. curl -s $A/team/new $AUTH -d '{"team_alias":"lit7685-team-26743"}'
team_id 6374dbe9-08d7-4428-a963-c393ad22a152
2. curl -s $B/key/generate $AUTH -d '{"key_alias":"team-key-26743","max_budget":100,"team_id":"'$TEAM'"}'
{"key_alias": "team-key-26743", "max_budget": 100.0, "team_id": "6374dbe9-08d7-4428-a963-c393ad22a152"}
3. curl -s -w '\nHTTP %{http_code}\n' $A/key/bulk_update $AUTH -d '{"keys":[{"key":"'$KEY4'","tags":["team-a"]}]}'
HTTP 200 successful: 1 failed: []
4. curl -s "$B/key/info?key=$KEY4" $AUTH
{"key_alias": "team-key-26743", "max_budget": 100.0, "team_id": "6374dbe9-08d7-4428-a963-c393ad22a152", "metadata": {"tags": ["team-a"]}}

Bulk item carrying object_permission

1. curl -s $B/key/generate $AUTH -d '{"key_alias":"objperm-26743","max_budget":100}'
{"key_alias": "objperm-26743", "max_budget": 100.0}
2. curl -s -w '\nHTTP %{http_code}\n' $A/key/bulk_update $AUTH -d '{"keys":[{"key":"'$KEY5'","object_permission":{"vector_stores":["vs-1"]}}]}'
HTTP 200
{"successful_keys": ["sk-PJpAv7pMM..."], "successful_max_budget": [100.0], "failed": []}
3. curl -s "$B/key/info?key=$KEY5" $AUTH
{"key_alias": "objperm-26743", "max_budget": 100.0, "object_permission_id": "893fd61d-f62b-4cf5-a87a-497aaef15da1", "object_permission": {"object_permission_id": "893fd61d-f62b-4cf5-a87a-497aaef15da1", "mcp_servers": [], "mcp_access_groups": [], "mcp_tool_permissions": null, "vector_stores": ["vs-1"], "agents": [], "agent_access_groups": [], "models": [], "blocked_tools": [], "mcp_toolsets": [], "search_tools": [], "mcp_tool_search_enabled": null, "skills": [], "teams": null, "projects": null, "verification_tokens": null, "organizations": null, "users": null, "end_users": null, "agents_table": null}}

Team key: bulk item granting a search tool outside the team allowlist

1. curl -s $A/team/new $AUTH -d '{"team_alias":"lit7685-scoped-26743","object_permission":{"search_tools":["team-search"]}}'
team_id 85b23254-7946-4021-b687-19320539f9e0
2. curl -s $B/key/generate $AUTH -d '{"key_alias":"scoped-bulk-26743","max_budget":100,"team_id":"'$TEAM6'"}' and the same with key_alias scoped-single-26743
{"key_alias": "scoped-bulk-26743", "max_budget": 100.0, "team_id": "85b23254-7946-4021-b687-19320539f9e0"}
{"key_alias": "scoped-single-26743", "max_budget": 100.0, "team_id": "85b23254-7946-4021-b687-19320539f9e0"}
3. curl -s -w '\nHTTP %{http_code}\n' $A/key/bulk_update $AUTH -d '{"keys":[{"key":"'$KEY6'","object_permission":{"search_tools":["other-search"]}}]}'
HTTP 200
{"successful_max_budget": [], "failed": ["Key requests search tools not allowed by team '85b23254-7946-4021-b687-19320539f9e0': ['other-search']. Team allows: ['team-search']."]}
4. curl -s -w '\nHTTP %{http_code}\n' $B/key/update $AUTH -d '{"key":"'$KEY7'","object_permission":{"search_tools":["other-search"]}}'
HTTP 403
{"message": "{'error': \"Key requests search tools not allowed by team '85b23254-7946-4021-b687-19320539f9e0': ['other-search']. Team allows: ['team-search'].\"}", "type": "auth_error", "param": "None", "code": "403"}
5. curl -s "$A/key/info?key=$KEY6" $AUTH and the same for $KEY7
{"key_alias": "scoped-bulk-26743", "max_budget": 100.0, "team_id": "85b23254-7946-4021-b687-19320539f9e0", "search_tools": null}
{"key_alias": "scoped-single-26743", "max_budget": 100.0, "team_id": "85b23254-7946-4021-b687-19320539f9e0", "search_tools": null}

Before, the bulk item in the last case came back as a success with the permission dropped and the key's max_budget and team_id nulled, while /key/update refused the same body with a 403. After, the bulk item lands under failed_updates carrying the same reason and the key is left as it was.

Observed next to the fix, on both sides, left alone by this PR:

• /key/generate echoes "tags": null although the tags are stored
• Before: a tags-only bulk item also nulled the key's team_id

Dependents driven live, base vs head (df6a222)

Run at df6a222; the tip dc02e5f since then changes only the unit test file, so this run stands at the tip. Same two-instance, two-worker topology per side, fresh database per side, and both key hooks loaded from the config so the readers of the per-key request are observed rather than reasoned about: custom_key_update records data.model_fields_set and allows, custom_key_policy records the effective key and denies an update whose effective max_budget is null

Not driven: /team/key/bulk_update already builds its request with exclude_unset, the Admin UI has no caller of /key/bulk_update, and no main commit since the merge base touches either changed file

Type

🐛 Bug Fix

Caveats (if any)

Low

• Fields outside the item model are still ignored, as on /key/update
  • Rejecting them would turn today's 200s into 422s, so left as is
• object_permission with a non-object value (a number, a list, a string) now fails the whole batch with 422; before, the value was ignored
  • Same validation as /key/update; accepting the field is the option the ticket offered
• custom_key_update now sees only the fields a bulk item carries in model_fields_set, and custom_key_policy sees an effective key that keeps the omitted fields
  • This is the contract the hook docs describe; a hook keyed on an explicit max_budget: null no longer fires on items that left the field out
• tags: null and object_permission: null on an item are left in place rather than cleared, as on /key/update
• On a bulk item, custom_key_update runs before the team-scope check on object_permission, where /key/update validates first and then calls the hook
  • The bulk route already calls the hook ahead of its team-change and team-limit checks, so the new check keeps that order; the hook only returns a decision, and a refused item is never written
• A bulk item refused by the team-scope check counts under failed_updates with HTTP 200 for the batch, where /key/update returns HTTP 403
  • That is how every per-item failure is reported on /key/bulk_update; the reason string is the one /key/update returns
• CircleCI at dc02e5f (pipeline 89874): integration 7/7, build_and_test 41/44 with three fleet-wide reds, none touching key management: google_generate_content_endpoint_testing is a Vertex 429 quota hit and the Bedrock invalid beta flag e2e tests in proxy_e2e_anthropic_messages_tests fail the same way on main's scheduled pipeline 89873 (12:09 UTC today), and test_models_by_provider in litellm_utils_testing failed on main's pipelines 89818 and 89834 until main's 044f88e registered the transcribe provider, which this branch's merge base predates, so the merge commit picks the fix up

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR
• b746ac4 passes /live-pr-risk
• df6a222 passes /live-pr-risk