csv: support non-ASCII Unicode dialect characters by deronnax · Pull Request #9367 · RustPython/RustPython

deronnax · 2026-10-25T12:31:46Z

delimiter, quotechar and escapechar were stored as u8, so valid one-character values outside the byte range (e.g. €) were rejected. They are now stored as char, and the reader state machine and the writer operate on code points. Empty and multi-character values are still rejected with the same messages.

Notes:

The field size limit is now counted in characters instead of bytes, as in CPython.
Lone surrogates are still not supported as dialect characters or data (unchanged: they already failed with UnicodeDecodeError in the data path).
Lib/test/test_csv.py: test_dialect_apply (which uses a Λ delimiter) now passes, so its expectedFailure is removed.
Added test_non_ascii_dialect_chars to the existing extra_tests/snippets/stdlib_csv.py.
The first commit makes scripts/check_redundant_patches.py executable: the redundant-test-patches hook uses language: script and fails on any commit touching Lib/test otherwise. It is independent and can be dropped if preferred.

Tested locally: cargo run --release -- -m test test_csv, the snippet, cargo clippy -p rustpython-stdlib --all-targets, cargo test -p rustpython-stdlib.

AI assistance: written with Claude (see Assisted-by trailers).

🤖 Generated with Claude Code

Summary by CodeRabbit

New Features
- CSV readers and writers now support non-ASCII delimiter, quote, and escape characters.
- Field size limits count characters rather than UTF-8 bytes.
Bug Fixes
- CSV input is decoded as UTF-8, with invalid input reported as a Unicode decode error.
- CSV writing handles Unicode characters in fields and line terminators correctly.

The redundant-test-patches pre-commit hook uses `language: script`, which requires the executable bit, and fails on every commit touching Lib/test. Assisted-by: Claude:claude-sonnet-5-5

Store delimiter, quotechar and escapechar as `char` instead of `u8`, and run the reader state machine and the writer over code points, so that a single non-ASCII character such as `€` is accepted like in CPython. The field size limit is now counted in characters rather than bytes. Fixes RustPython#8408 Assisted-by: Claude:claude-sonnet-5-5

coderabbitai · 2026-10-25T12:31:55Z

No actionable comments were generated in the recent review.

Recent review info

Run configuration

Source mode: Auto
Repository: RustPython/RustPython
Branch: main
File: .coderabbit.yml
Commit: d1c3cb7
Layers: Repository YAML
Review profile: CHILL
Plan: Advanced
Run ID: a02aedbc-7789-40a3-a503-e0be57d7ad98

Commits

Reviewing files that changed from the base of the PR and between d1c3cb7 and eea7622.

Files ignored due to path filters: 1

Lib/test/test_csv.py is excluded by !Lib/**

Files selected for processing: 3

crates/stdlib/src/csv.rs
extra_tests/snippets/stdlib_csv.py
scripts/check_redundant_patches.py

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

Walkthrough

CSV dialect attributes, parser input, field storage, and writer output now use Unicode characters. The reader decodes UTF-8 input, and field limits count characters. Tests cover non-ASCII dialect characters and invalid delimiter values.

Changes: Unicode CSV support

Layer / File(s)	Summary
Unicode dialect attributes `crates/stdlib/src/csv.rs`	Dialect character attributes use `char` values. Parsing and validation retain checks for empty or multi-character values and line-break characters.
Unicode CSV input and parsing `crates/stdlib/src/csv.rs`	The reader decodes input as UTF-8. The parser processes characters, stores fields in a `String`, and counts field limits by character.
Unicode CSV output and validation `crates/stdlib/src/csv.rs`, `extra_tests/snippets/stdlib_csv.py`	The writer builds and writes a `String`, with character-based quoting and escaping. Tests cover non-ASCII dialect characters and invalid delimiter values.

Priority: Normal

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix · Severity of issue fixed: Low

Suggested reviewers: widehyo1, hyoinandout

Merge Risk: Minimal · up to eea76

No concrete issue requiring a fix before merge is established by the available evidence.

Pre-merge checks: 3 · 2

Failed checks: 2 warnings

Check name	Status	Explanation	Resolution
Out of Scope Changes check		The pull request description identifies making `scripts/check_redundant_patches.py` executable as an independent change. This change does not implement or test issue `#8408`'s CSV requirements.	Remove the independent executable-bit change from this pull request, or move it to a separate pull request.
Docstring Coverage		Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 21 functions across 2 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

Passed checks: 3 passed

Check name	Status	Explanation
Description Check		Check skipped - CodeRabbit’s high-level summary is enabled.
Title check		The title "csv: support non-ASCII Unicode dialect characters" matches the main change. The PR changes CSV delimiter, quotechar, and escapechar from u8 to char, and the reader and writer now operate on…
Linked Issues check		Issue `#8408` requires full Unicode code points for `delimiter`, `quotechar`, and `escapechar`, while preserving empty and multi-character validation. The summary reports `char`/`String` storage, charac…

Full details: Title check

Explanation

The title "csv: support non-ASCII Unicode dialect characters" matches the main change. The PR changes CSV delimiter, quotechar, and escapechar from u8 to char, and the reader and writer now operate on code points. The added test covers non-ASCII delimiter, quote, and escape characters. The title is concise, specific, and follows the linked issue #8408 scope. Empty and multi-character values remain rejected, as the title implies.

Full details: Linked Issues check

Explanation

Issue #8408 requires full Unicode code points for delimiter, quotechar, and escapechar, while preserving empty and multi-character validation. The summary reports char/String storage, character-based reader and writer processing, and rejection of invalid lengths. The added test_non_ascii_dialect_chars covers reader and writer round trips with €, invalid delimiter lengths, and delimiter/quote conflicts. The reported behavior satisfies the issue requirements.

Finishing Touches

🧪 Generate unit tests (beta)

Create a new PR

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-10-25T14:50:45Z

📦 Library Dependencies

The following Lib/ modules were modified. Here are their dependencies:

[x] lib: cpython/Lib/traceback.py
[x] test: cpython/Lib/test/test_traceback.py (TODO: 3)

dependencies:

traceback

dependent tests: (170 tests)

[ ] test: cpython/Lib/test/test_external_inspection.py

dependencies:

dependent tests: (no tests depend on external_inspection)

[ ] lib: cpython/Lib/profiling
[ ] test: cpython/Lib/test/test_profiling

dependencies:

profiling (native: main, _child_monitor, _css_utils, _format_utils, _lsprof, _remote_debugging, binary_collector, binary_reader, cli, collector, constants, display, dump, errors, gecko_collector, heatmap_collector, importlib.machinery, importlib.resources, importlib.util, itertools, jsonl_collector, live_collector, marshal, math, module_utils, opcode_utils, profiling.tracing, profiling.tracing._utils, pstats_collector, sample, stack_collector, string_table, sys, time, trend_tracker, widgets)
- curses (native: _curses, _curses_panel, curses.ascii, has_key, sys)
- io (native: _io, _thread, errno, msvcrt, sys)
- opcode (native: _opcode, builtins)
- platform (native: _wmi, itertools, sys, vms_lib, winreg)
- site (native: _io, _pyrepl.main, _pyrepl.pager, _pyrepl.readline, _pyrepl.unix_console, _pyrepl.windows_console, atexit, builtins, errno, readline, sitecustomize, sys, usercustomize)
- sysconfig (native: _sysconfig, _winapi, importlib.machinery, importlib.util, os.path, sys)
- _colorize, abc, argparse, base64, collections, contextlib, dataclasses, functools, html, json, linecache, locale, optparse, os, pathlib, pstats, re, runpy, selectors, socket, statistics, subprocess, tempfile, threading, traceback, types, typing, uuid, webbrowser

dependent tests: (3 tests)

profiling: test_profiling
- cProfile: test_cprofile test_pstats

[ ] test: cpython/Lib/test/test_set.py (TODO: 2)

dependencies:

dependent tests: (no tests depend on set)

[x] lib: cpython/Lib/csv.py
[x] test: cpython/Lib/test/test_csv.py

dependencies:

csv

dependent tests: (4 tests)

csv: test_csv test_genericalias
- importlib.metadata: test_importlib test_zoneinfo

Legend:

[+] path exists in CPython
[x] up-to-date, [ ] outdated

chatgpt-codex-connector · 2026-10-25T15:17:41Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	⚠️ Failed 2026-10-25T15:18:52.401592Z We couldn’t complete this review. Comment `@codex review` to try again.	`eea7622`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

youknowone · 2026-10-26T01:10:16Z

@deronnax Thank you for contributing! welcome to RustPython project

@widehyo1 could you please review this PR?

deronnax · 2026-10-26T12:55:40Z

That would require some sort of speed benchmark/testing, because I think perf should change a bit. But it might go from « unnoticeable » to « significant ». But anyway, the previous behavior was incorrect (delimiter chars never meant to be 1 byte). And also, by doing this change, it fixes few existing unicode mis-handlings.

widehyo1 · 2026-10-27T12:42:00Z

Thanks for the fix! The implementation looks good to me.

youknowone

Thanks!

deronnax added 2 commits October 25, 2026 13:31

scripts: make check_redundant_patches.py executable

f36e065

The redundant-test-patches pre-commit hook uses `language: script`, which requires the executable bit, and fails on every commit touching Lib/test. Assisted-by: Claude:claude-sonnet-5-5

deronnax marked this pull request as ready for review October 25, 2026 15:17

youknowone approved these changes Oct 27, 2026

View reviewed changes

youknowone merged commit a463449 into RustPython:main Oct 27, 2026
21 checks passed

Navigation Menu

Repository navigation

csv: support non-ASCII Unicode dialect characters - #9367

csv: support non-ASCII Unicode dialect characters#9367
youknowone merged 2 commits into
RustPython:mainfrom
deronnax:fix/csv-unicode-dialect-chars

deronnax commented Oct 25, 2026 •

edited by coderabbitai Bot

Loading

Uh oh!

coderabbitai Bot commented Oct 25, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 25, 2026

Uh oh!

chatgpt-codex-connector Bot commented Oct 25, 2026 •

edited

Loading

Uh oh!

youknowone commented Oct 26, 2026

Uh oh!

deronnax commented Oct 26, 2026

Uh oh!

widehyo1 commented Oct 27, 2026

Uh oh!

youknowone left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

deronnax commented Oct 25, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary by CodeRabbit

Uh oh!

coderabbitai Bot commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 25, 2026

📦 Library Dependencies

Uh oh!

chatgpt-codex-connector Bot commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

youknowone commented Oct 26, 2026

Uh oh!

deronnax commented Oct 26, 2026

Uh oh!

widehyo1 commented Oct 27, 2026

Uh oh!

youknowone left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

deronnax commented Oct 25, 2026 •

edited by coderabbitai Bot

Loading

coderabbitai Bot commented Oct 25, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Oct 25, 2026 •

edited

Loading