TLDR

Problem this solves:

• Responses joined from streamed chunks carry no response cost
• DSPy's track_usage records cost: None for every streamed call
• Streamed Anthropic citations are dropped when chunks are joined

How it solves it:

• The joined response now carries its response cost in hidden params
• Provider-reported usage cost wins; otherwise litellm prices the usage, falling back to the bare model name in the model map when the declared provider can't price it (models aliased through a LiteLLM proxy)
• Citation deltas are collected into the final message's citations list
• Block-list citation deltas (Databricks) join without extra nesting, matching the non-streamed shape
• Internal joins that feed the proxy's logging pipeline (mid-stream failure spend recovery) now declare their logging object, so failure spend keeps coming from the full cost pipeline (custom pricing, corrected model) instead of a cost stamped from the raw chunk model

User Flow

Before: a DSPy developer who turns on streaming loses cost tracking and citations

1. They wrap their program in dspy.streamify(dspy.Predict("question -> answer")) with track_usage=True and dspy.LM("anthropic/claude-opus-5", cache=False)
2. DSPy sends POST https://api.anthropic.com/v1/messages with "stream": true and the answer streams back fine
3. lm.history[-1]["cost"] reads None, so their spend tracking records nothing, while the identical non-streamed call records 0.00306
4. With a document attached and citations enabled, they watch citation deltas arrive in the stream, yet the final prediction's citations come back empty, while the identical non-streamed call returns the citation

After: the same streamed program reports real spend and keeps its citations

1. They wrap their program in dspy.streamify(dspy.Predict("question -> answer")) with track_usage=True and dspy.LM("anthropic/claude-opus-5", cache=False)
2. DSPy sends POST https://api.anthropic.com/v1/messages with "stream": true and the answer streams back fine
3. lm.history[-1]["cost"] now reads 0.00318, in line with the non-streamed call
4. The final prediction carries every citation that streamed by

Relevant issues

Linear ticket

Resolves LIT-6376

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

• I have added meaningful tests
• The handful of test files covering my change pass locally, e.g. uv run pytest tests/test_litellm/<your_test_file>.py -v. Leave the suites (make test-unit-*, make test-unit) to CI: it finishes in ~15 minutes where a laptop takes an hour or more
• My PR passes all required CI/CD checks (e.g., lint, schema.d.ts sync check, etc.)
• My PR's scope is as isolated as possible; it only solves 1 specific problem
• I have received a Greptile Confidence Score of at least 4/5 before requesting a maintainer review (Greptile reviews automatically once the PR is opened; only comment @greptileai to re-request a review after pushing changes)

Screenshots / Proof of Fix

Legs: before at the merge base 936e07b, after at this PR's head 9b1b8e7. Same client both sides: DSPy 3.3.1 driving real Anthropic, OpenAI, Gemini, and Bedrock APIs, no mocks. SDK cases call providers directly. Proxy cases boot one proxy instance from the leg's commit with --num_workers 2 (no DB) on a random localhost port, with the client litellm at the same commit and models exposed as OpenAI-compatible aliases (gpt-5.6, claude-opus-5; the after leg also drives gemini-3.7-flash). stream_chunk_builder only joins chat-completions-shaped streams (SDK joining and proxy chat logging); /v1/messages and /v1/responses have their own assemblers and never ride this code, so those endpoints are out of scope here

Streamed cost, run with DSPy exactly as an end user would:

python - <<'PY'
import asyncio
import dspy

lm = dspy.LM("anthropic/claude-opus-5", cache=False)
dspy.configure(lm=lm, track_usage=True)
program = dspy.streamify(dspy.Predict("question -> answer"))

async def run():
    async for _ in program(question="Why is the sky blue? One sentence."):
        pass

asyncio.run(run())
print("cost:", lm.history[-1]["cost"])
PY

Same thing through a LiteLLM proxy, pointing DSPy at the alias:

lm = dspy.LM("openai/claude-opus-5", api_base="http://localhost:<port>/v1", api_key=MASTER_KEY, cache=False)

Streamed citations, run against the live Anthropic API:

python - <<'PY'
import litellm

doc = {
    "type": "document",
    "source": {"type": "text", "media_type": "text/plain",
               "data": "The Eiffel Tower was completed in 1889. It is 330 metres tall."},
    "title": "facts",
    "citations": {"enabled": True},
}
chunks = list(litellm.completion(
    model="anthropic/claude-opus-5",
    messages=[{"role": "user", "content": [doc, {"type": "text", "text": "How tall is the Eiffel Tower and when was it completed? Cite the document."}]}],
    stream=True,
))
built = litellm.stream_chunk_builder(chunks)
print("citations:", (built.choices[0].message.provider_specific_fields or {}).get("citations"))
print("response_cost:", built._hidden_params.get("response_cost"))
PY

Before (936e07b)

Streamed cost lost

1. Ran the DSPy snippet per provider (anthropic/claude-opus-5, gpt-5.6, gemini/gemini-3.7-flash, bedrock/us.anthropic.claude-sonnet-5)
2. Observed cost: None on all four, while the identical non-streamed calls priced fine
3. Through the proxy (2 workers), streamed cost: None on both the gpt-5.6 and claude-opus-5 aliases, while the identical non-streamed calls read 0.001164 and 0.003125

Streamed citations lost

1. Ran the citations snippet: 3 citation deltas arrived in the stream, then citations: None and response_cost: None on the joined response
2. DSPy's streaming Citations program direct to Anthropic ended with 0 citations on the final prediction, where the identical non-streamed run returned 1
3. Through the proxy, the joined response likewise carried 0 citations

After (9b1b8e7)

Streamed cost lost

1. Ran the same DSPy snippet per provider
2. Observed real spend on all four: anthropic 0.00355, openai 0.001204, gemini 0.00108375, bedrock 0.001276
3. Through the proxy (2 workers), streamed cost now reads on the aliases: gpt-5.6 0.001148 (nonstream 0.001164), claude-opus-5 0.00295 (nonstream 0.003125), gemini-3.7-flash 0.000996 (nonstream 0.001065)

Streamed citations lost

1. Ran the same citations snippet: 1 citation delta arrived and the joined response carries 1 citation (char_location on the document) instead of None
2. DSPy's streaming Citations program direct to Anthropic ended with 1 citation on the final prediction, matching the non-streamed run (cost 0.007655 streamed, 0.00763 non-streamed)
3. Through the proxy, the joined response carries 1 citation (non-streamed also 1) and the streamed DSPy Citations run reports cost 0.009825

Databricks streamed citations A/B (SDK direct)

Same joined-stream flow against the real Databricks workspace, model databricks/databricks-claude-sonnet-5, no mocks:

python - <<'PY'
import json
import litellm

document = {
    "type": "document",
    "source": {
        "type": "text",
        "media_type": "text/plain",
        "data": "The sky is blue because of Rayleigh scattering. Grass is green because of chlorophyll.",
    },
    "title": "Science facts",
    "citations": {"enabled": True},
}

chunks = list(
    litellm.completion(
        model="databricks/databricks-claude-sonnet-5",
        messages=[{"role": "user", "content": [document, {"type": "text", "text": "Why is the sky blue, and why is grass green? Cite the document for each fact."}]}],
        stream=True,
    )
)
built = litellm.stream_chunk_builder(chunks)
print(json.dumps((built.choices[0].message.provider_specific_fields or {}).get("citations"), default=str))
PY

1. Before (936e07b): the stream delivered 2 citation deltas, and the joined response printed null; the citations never reach the joined message
2. After (9b1b8e7): both citations survive as one per-block list, [[c1, c2]] with char_location entries, no double nesting. This live run emitted singular citation deltas; the block-list delta shape from the original report stays locked by the regression tests in tests/test_litellm/test_stream_chunk_builder_citations.py

Observations from the runs:

• Proxy openai/ aliases reject reasoning_effort client-side; pre-existing, left alone
• Streamed citation dicts lack supported_text; pre-existing, left alone
• Streaming responses carry no plain response-cost header; pre-existing, left alone
• DSPy's own citation collection masked the loss via proxy; unaffected
• Alias missing from the model map streams cost 0.0
• litellm's wrapper logging already stamps 0.0 there; pre-existing, unchanged

Live PR risk: CHECKED at 9b1b8e7. Two findings were fixed in-PR, each with a regression test: mid-stream failure spend priced off the raw chunk model, and Databricks block-list citation deltas double-nested. CircleCI ran under the run-ci label; the logging_testing red is pre-existing (it fails on staging commits c11a1f0 and 3daf7a3 and on unrelated PR #38440, with identical failure sets at head vs merge base locally). A follow-up live Databricks A/B (above) closed the citation-join gap that had only unit-level coverage

Type

🐛 Bug Fix

Caveats (if any)

Low

• Streamed citation dicts lack supported_text; non-streamed ones carry it
• Without a provider-reported usage cost, pricing falls back to litellm's model map, and a model aliased through a proxy prices by its bare model name, which assumes the alias matches the upstream model
• An alias matching nothing in the model map prices streamed calls at 0.0 (litellm's default zero-cost info for unknown OpenAI-compatible models; the non-streamed call reads the real cost from the proxy's cost header, which streams don't carry)

Final Attestation

• The tests check the right things, including the edge cases, and regressions in the respective real-world customer use-cases are not possible after this PR