Skip to content

fix: escape SSE-unsafe Unicode line separators in streamed payloads - #1433

Closed
devinhuang2001 wants to merge 1 commit into
a2aproject:mainfrom
devinhuang2001:fix/sse-unicode-line-separators
Closed

devinhuang2001 wants to merge 1 commit into
a2aproject:mainfrom
devinhuang2001:fix/sse-unicode-line-separators

Conversation

@devinhuang2001

@devinhuang2001 devinhuang2001 commented Nov 7, 2026 •

Copy link
Copy Markdown

json_utils.dumps() writes SSE data: payloads with ensure_ascii=False, so
U+2028 / U+2029 / U+0085 go on the wire as raw UTF-8. httpx.aiter_lines()
uses str.splitlines(), which treats those three as line breaks: one data:
line becomes several and json.loads() raises JSONDecodeError. Re-subscribing
sends the same snapshot, so the task stays unreadable over streaming.

This change escapes exactly those three characters as \uXXXX in dumps().
Escapes are ASCII, so the frame stays on one line, json.loads() restores the
original text, and CJK / emoji still pass through unescaped. dumps() is the
serializer used by rest_dispatcher.py, jsonrpc_dispatcher.py and the v0.3
REST adapter for streamed events, so all three streaming paths are covered.

Verification

  • tests/utils/test_json_utils.py + tests/client/transports/: 130 passed
    (the 12 existing aiter_lines mocks are untouched and still pass).
  • New end-to-end cases drive a real httpx.Response through parse_sse_stream
    for all three separators at chunk sizes 1 / 7 / 4096.
  • Mutation check: reverting dumps() to upstream makes all 13 new assertions
    fail; restoring it makes 26/26 pass in the two touched test files.
  • Call chain checked against current main 28f8a62f25. sse-starlette's own
    framing was not re-tested here (the issue reporter covered it).

Known limitation

The client parser still uses httpx.aiter_lines(), so a third-party A2A server
that emits these characters raw can still split a line. Fixing that means
swapping to aiter_text() plus a CR/LF-only decoder and migrating the existing
aiter_lines mocks across the transport tests. I left that out of this bug-fix
PR to keep the fix small; happy to follow up if maintainers prefer the
client-side approach instead.

Checklist

  • Follow the CONTRIBUTING Guide
  • PR title follows Conventional Commits (fix:)
  • Tests added
  • Formatting / lint / type checks: the repo's CI Lint Code Base
    (ruff check, ruff format --check, ty check) passes on this head.
    The formatter was not run locally because the ruff wheel would not
    download here; CI is the authoritative check.
  • No docs change needed

Fixes #1432 🦕

@devinhuang2001
devinhuang2001 requested a review from a team as a code owner November 7, 2026 16:40
`json_utils.dumps()` serializes SSE `data:` payloads with
`ensure_ascii=False`, so U+2028, U+2029 and U+0085 go on the wire as raw UTF-8.
httpx's `aiter_lines()` follows `str.splitlines()`, which treats those three as
line endings, so a single `data:` line is split and `json.loads()` fails with
`JSONDecodeError`. Re-subscribing sends the same snapshot, so the task stays
unreadable over streaming.

Escape exactly those three characters as `\uXXXX` in `dumps()`. JSON escapes are
ASCII, so the payload stays on one line; `json.loads()` restores the original
characters, and all other non-ASCII text (CJK, emoji) still goes out raw.

Verified against upstream main 28f8a62. See the PR body for the evidence
table and the remaining client-side limitation.

Fixes a2aproject#1432

Signed-off-by: devinhuang2001 <[email protected]>
@devinhuang2001
devinhuang2001 force-pushed the fix/sse-unicode-line-separators branch from 871c16f to fc3254d Compare November 7, 2026 16:45
@github-actions

github-actions Bot commented Nov 7, 2026

Copy link
Copy Markdown

🧪 Code Coverage (vs main)

⬇️ Download Full Report

No coverage changes.

Generated by coverage-comment.yml

@Iwaniukooo11 Iwaniukooo11 self-assigned this Nov 10, 2026
@Iwaniukooo11

Iwaniukooo11 commented Nov 10, 2026 •

Copy link
Copy Markdown
Member

Thanks, the diagnosis is spot on

That behavior started after merging #1132, which was mine.

We'd rather not take the server-side change though. Escaping in dumps() doesn't fix what #1432 reports: a Python client still breaks against an a2a-js server, which sends these characters raw. And it puts an unconditional .translate() on every streamed frame, which is O(N).

The fix belongs in parse_sse_stream - aiter_lines() follows str.splitlines(), SSE ends lines only at CR/LF/CRLF, so the client is the non-conformant side.

Closing this one and #1432 stays open.

Please let me know if you want to proceed with creating the next PR on client side

Once again, thanks for contributing and the PR is good, but fixing the client would resolve the issue at its root

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: SSE client splits data lines on U+2028, U+2029 and U+0085, so streamed events fail to parse

2 participants