feat(json): assemble a streamed chat completion into the non-streaming answer by bernardladenthin · Pull Request #595 · bernardladenthin/java-llama.cpp

bernardladenthin · 2026-10-29T21:12:25Z

Summary

New public json.ChatStreamAssembler. It turns an OpenAI streaming chat completion back into the answer the same request gives without stream.
- Input is either a Server-Sent Events body (parseStream, or line by line with acceptLine) or the chunk strings LlamaModel.streamChatCompletion hands its sink (acceptChunk).
- Per choice it joins content and reasoning_content, rebuilds the tool calls and keeps the last finish reason. From the stream as a whole it keeps the first id/created/model and the last usage and timings.
- Output goes through ChatResponseParser (toResponse) or comes back as the non-streaming JSON (toCompletionJson).
- It is strict where leniency would turn a broken stream into a plausible short answer: a data line that is not JSON, an error chunk, or a body without data: [DONE] throws InvalidStreamException. Heartbeat comments and the other SSE fields are skipped.
ToolCallDeltaAccumulator moves from server to json and becomes public. The server's protocol translators (Ollama, Anthropic, Responses) and the assembler now share one implementation. The SpotBugs exclusion moves with it.
Why: the server side of the stream contract was complete; the client side was missing. srcmorph now streams from OpenAI-compatible servers (feat(provider): stream openai-compatible answers as Server-Sent Events srcmorph#257, needed for the GitHub Copilot API Gateway) and assembles the events itself for now. With this class, every jllama consumer has one interface for the same job.

Test plan

Affected unit / integration tests pass locally (JDK 25, Windows, local jllama.dll): 210 tests across ChatStreamAssemblerTest (22), ToolCallDeltaAccumulatorTest, OpenAiCompatServerHttpTest, OllamaApiSupportTest, the two stream translators, LlamaArchitectureTest and OpenAiCompatServerIntegrationTest.
- The server's own SSE stream assembles into its non-streaming answer (FakeBackend, with and without heartbeats).
- With Qwen3-0.6B, a greedy request streamed and not streamed gives the same text, reasoning, finish reason and token counts (aStreamedChatAssemblesIntoTheNonStreamedAnswer).
PIT: ChatStreamAssembler added to the gate, 50/50 mutations killed.
mvn -pl llama verify -DskipTests: SpotBugs, Javadoc and the other static checks pass.
CI is green on this branch
Docs / CHANGELOG updated where applicable (CHANGELOG Unreleased/Added, CLAUDE.md server section)

Related issues / PRs

Refs bernardladenthin/srcmorph#257. srcmorph can switch to this class once a jllama release contains it.

Checklist

I have read CONTRIBUTING.md and CODE_OF_CONDUCT.md
My commits follow Conventional Commits
No security-sensitive changes (if there are, I have notified the maintainer privately per SECURITY.md)

🤖 Generated with Claude Code

https://claude.ai/code/session_01DuAkLtHrCTx81vP5Mt4jad

Summary by CodeRabbit

New Features
- Added support for assembling streamed chat responses into standard chat responses or JSON, including content, reasoning, tool calls, usage, and timing details.
- Added a reusable tool-call accumulation utility for reconstructing complete calls from streamed fragments.
Bug Fixes
- Invalid stream data and streams that end without a completion marker now produce clear errors.

…g answer json.ChatStreamAssembler turns an OpenAI streaming chat completion back into the answer of the same request without `stream`: from a Server-Sent Events body (parseStream / acceptLine) or from the chunk strings LlamaModel.streamChatCompletion hands its sink (acceptChunk). Per choice it joins content and reasoning_content, rebuilds the tool calls and keeps the last finish reason; of the stream it keeps the first id/created/model and the last usage and timings. A data line that is not JSON, an error chunk and a body without `data: [DONE]` throw InvalidStreamException instead of passing for a short answer. ToolCallDeltaAccumulator moves from `server` to `json` and becomes public, so the server's protocol translators and the assembler share one implementation. Tests: unit tests (PIT 50/50, added to the gate), the server's own SSE stream assembled against its non-streaming answer (FakeBackend, with and without heartbeats), and Qwen3-0.6B greedy: streamed and not streamed give the same text, reasoning, finish reason and token counts. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01DuAkLtHrCTx81vP5Mt4jad

coderabbitai · 2026-10-29T21:12:42Z

Walkthrough

Adds ChatStreamAssembler to parse SSE chat streams into non-streaming responses. Moves ToolCallDeltaAccumulator into the JSON package and makes its API public. Adds unit, HTTP, and integration tests for stream parsing and response equivalence.

Changes: Chat stream assembly

Layer / File(s)	Summary
Shared tool-call accumulator API `llama/src/main/java/net/ladenthin/llama/json/ToolCallDeltaAccumulator.java`, `llama/src/main/java/net/ladenthin/llama/server/*`, `llama/src/test/java/net/ladenthin/llama/json/ToolCallDeltaAccumulatorTest.java`, `llama/src/test/java/net/ladenthin/llama/server/OllamaApiSupportTest.java`, `llama/spotbugs-exclude.xml`	Moves `ToolCallDeltaAccumulator` to the JSON package, makes its constructor and methods public, and adds `acceptToolCalls(JsonNode)`. Updates server imports, package documentation, and the related test package.
SSE parsing and response assembly `llama/src/main/java/net/ladenthin/llama/json/ChatStreamAssembler.java`, `llama/src/test/java/net/ladenthin/llama/json/ChatStreamAssemblerTest.java`, `CHANGELOG.md`, `CLAUDE.md`, `llama/pom.xml`, `llama/spotbugs-exclude.xml`	Adds SSE line and chunk parsing, per-choice content and tool-call accumulation, completion JSON generation, and conversion to `ChatResponse`. Rejects invalid chunks and streams without `[DONE]`. Adds unit tests and supporting changelog, documentation, mutation-testing, and SpotBugs updates.
Server stream equivalence checks `llama/src/test/java/net/ladenthin/llama/server/OpenAiCompatServerHttpTest.java`, `llama/src/test/java/net/ladenthin/llama/server/OpenAiCompatServerIntegrationTest.java`	Adds HTTP and integration tests that compare assembled streaming responses with non-streaming responses, including content and selected usage fields.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant SSEBody
  participant ChatStreamAssembler
  participant ChatResponseParser
  SSEBody->>ChatStreamAssembler: Provide SSE lines
  ChatStreamAssembler->>ChatStreamAssembler: Parse chunks and accumulate choices
  ChatStreamAssembler->>ChatResponseParser: Parse assembled completion JSON

Suggested labels: java

Suggested reviewers: claude

Merge Risk: Minimal · up to 610a8

No concrete issue was identified that needs correction before merge. Complete the normal CI checks.

Security architecture risk: Low · up to 610a8

The change reconstructs responses without granting new execution privileges. Whole-body parsing rejects interrupted streams and reported errors, while incremental callers remain responsible for completion and failure handling. Risk is low, with downstream use not fully established.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

inferred — The demonstrated new exposure is in-process interpretation of caller-supplied stream data into response objects. Neither the assembler nor the public accumulator executes reconstructed tools. Any privileged effect depends on a separate consumer; no automatic execution path from this new API was established.

Trust Boundaries and Controls

inferred — Parsing crosses a representation boundary, not an authority boundary. Provider-controlled content and tool-call fields remain untrusted response data after assembly. The unchanged accumulator can replace tool identity fields supplied in later fragments, but the inspected comparison does not establish a newly bypassed authorization control.

Resilience and Maintainability Implications

observed — Incremental ingestion has no permanent failed state or sealed snapshot. SSE lines stop affecting state after DONE, while direct chunk calls remain available and extraction serializes current state. The documented single-stream, non-thread-safe model therefore requires callers to propagate errors and avoid reusing a failed instance or replaying fragments.

Hardening Proposals

proposed — For consumers that require validated completion before acting on a response, document an explicit integration rule: discard the assembler after ingestion failure, require DONE in line-fed mode, and keep tool authorization independent of parsing success. A separate validated-finalization operation could make that distinction harder to misuse without removing partial snapshots.

Pre-merge checks: 4 · 1

Failed checks: 1 warning

Check name	Status	Explanation	Resolution
Docstring Coverage		Docstring coverage is 72.73% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documenta…	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 4 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title names the main change: a new public json.ChatStreamAssembler that assembles a streamed chat completion into the non-streaming answer. The changeset matches this. It adds ChatStreamAssembler …
Linked Issues check		Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check		Check skipped because no linked issues were found for this pull request.

Full details: Title check

Explanation

The title names the main change: a new public json.ChatStreamAssembler that assembles a streamed chat completion into the non-streaming answer. The changeset matches this. It adds ChatStreamAssembler with SSE parsing, chunk accumulation, and toResponse/toCompletionJson, plus tests. The title follows the conventional-commit form with a clear scope.

Full details: Docstring Coverage

Explanation

Docstring coverage is 72.73% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff with cyclomatic complexity of at least 2. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 11 functions across 12 files. (4 skipped: 4 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 2

📝 Generate docstrings: 💡

Commit to this branch
Create a new PR

🛠️ Fix failing CI checks: 💡

Commit to this branch
Create a new PR

🧪 Generate unit tests (beta)

Commit to this branch
Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

sonarqubecloud · 2026-10-29T21:14:38Z

Quality Gate passed

Issues
10 New issues
0 Accepted issues

Measures
0 Security Hotspots
99.5% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

…treamAssembler `OpenAiCompatibleProviderSupport.assembleStream` and its constants copy what java-llama.cpp ships as `json.ChatStreamAssembler` (bernardladenthin/java-llama.cpp#595), which is in no release yet. The new marker annotation `support.ReplaceWithLibrary` names the library class and the release expected to carry it. `ReplaceWithLibraryTest` fails as soon as that class is on the classpath, i.e. when net.ladenthin:llama is raised to that release, and lists the marked elements. It also fails once no marker is left, so the annotation and the test are deleted with the last copy. TODO.md describes the switch. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01DuAkLtHrCTx81vP5Mt4jad

bernardladenthin deployed to startgate October 29, 2026 21:12 — with GitHub Actions Active

bernardladenthin mentioned this pull request Oct 29, 2026

feat(provider): stream openai-compatible answers as Server-Sent Events bernardladenthin/srcmorph#257

Merged

14 of 15 tasks

bernardladenthin deployed to unsigned October 29, 2026 22:36 — with GitHub Actions Active

bernardladenthin merged commit 6bb7e65 into main Oct 30, 2026
86 of 90 checks passed

bernardladenthin deleted the feat/chat-stream-assembler branch October 30, 2026 09:06

Navigation Menu

Repository navigation

feat(json): assemble a streamed chat completion into the non-streaming answer - #595

feat(json): assemble a streamed chat completion into the non-streaming answer#595
bernardladenthin merged 1 commit into
mainfrom
feat/chat-stream-assembler

bernardladenthin commented Oct 29, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 29, 2026 •

edited

Loading

Sequence Diagram(s)

Uh oh!

sonarqubecloud Bot commented Oct 29, 2026

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

bernardladenthin commented Oct 29, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Related issues / PRs

Checklist

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Sequence Diagram(s)

Uh oh!

sonarqubecloud Bot commented Oct 29, 2026

Quality Gate passed

Uh oh!

Uh oh!

This branch was successfully deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

bernardladenthin commented Oct 29, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 29, 2026 •

edited

Loading