Repository navigation
Conversation
provinzkraut
left a comment
provinzkraut
left a comment
There was a problem hiding this comment.
I feel rather uneasy with such a big change, and adding this amount of code to the repo. There is an existing pcre2 package: https://pypi.org/project/pcre2/. Can't we simply support this, possibly building upon the work of #925?
|
Also, can you share some benchmarks so we can weigh whether or not this is worth adding? |
|
Yes, the vendored PCRE2 code ended up in this PR by mistake. I started with the existing Then I created another branch for a native integration and started copying PCRE2 into it. Somewhere along the way I mixed up the branches. I thought I was benchmarking the version with the external package installed as an optional extra. I got roughly a 30% improvement and pushed it without checking what was actually in the diff :( I reran the benchmark with the external I'll keep investigating. |
In that case, it might be worth it to consider doing some work upstream to provide a capsule for that package, so we could call into it straight from C. |
| } | ||
|
|
||
| static bool | ||
| ms_pattern_eligible(PyObject *pattern) { |
There was a problem hiding this comment.
Since this does not sit in a hot loop, does it really need to happen in C? The code is rather complex, so having it in Python might make it a bit easier and less error-prone
There was a problem hiding this comment.
Thx, I agree. I moved the eligibility check, imports of the optional package and compilation preparation into Python. Capsule ABI checks, native matcher lifetime and matching each value remain in C.
Benchmarks of the Python-preparation prototype gave these complete JSON decode, validate and encode cycles per second, with validators prepared beforehand:
| Variant | Page of 100 cities | Array of 299 identifiers | Array of 16 country codes |
|---|---|---|---|
| msgspec + PCRE2, C preparation | 8,685 | 38,136 | 806,516 |
| msgspec + PCRE2, Python preparation | 8,767 | 38,308 | 808,399 |
| Pydantic + RustRegex | 5,655 | 67,650 | 1,008,374 |
Throughput with Python preparation differed from C preparation by less than 2% in each JSON workload in both run orders. Warm construction of an identifier-array validator dropped from about 190k/s with C preparation to 137k/s with Python preparation, about 28% fewer constructions per second. That setup is outside the JSON timings above.
These measurements include JSON processing; the city-page path also includes model construction and custom validation. The environment was local Docker, Linux ARM64, CPython 3.14.7 and Pydantic 2.13.5. Both PCRE2 variants used the same experimental pcre2 0.7.1/PCRE2 10.47 build with the capsule API proposed in pcre2.py#29.
|
Thx for the suggestion. I've removed the bundled PCRE2 sources and updated this PR to integrate the existing I've opened grtetrault/pcre2.py#29 with API examples, benchmarks and a reproducible benchmark archive. In the two tested patterns, the C API prototype achieved:
These numbers measure individual regex checks, not the full JSON decoding/validation/encoding path. Both pcre2 variants used JIT and the same patched binary. The comparison combines avoiding Python calls and Match creation with reusing match storage; it does not isolate their individual costs. |
Use the optional
pcre2package throughpcre2._cy._MATCHER_API_v1from pcre2.py#30. Pattern eligibility, imports and compilation preparation remain in Python; ABI checks, matcher ownership and per-value boolean matching remain in C. Accept compatible larger v1 API tables and test this with the real SDK.Preserve Python
resyntax and search semantics. Missing packages or capsules, unsupported patterns/inputs and free-threaded Python usere; invalid providers and unexpected errors propagate. No bundled PCRE2 engine or required dependency. The API is not in released pcre2 0.7.1, so acceleration still depends on an upstream release.Fresh Linux/macOS unit checks passed with the actual API, no SDK and released pcre2 without the API. Typing compatibility, stubtest, doctests, docs and sdist-install checks passed.
mypy src/msgspecretains the same 91 diagnostics as current main.Local Docker Linux aarch64, CPython 3.14.7, PCRE2 10.47 with JIT, release builds. Mean microseconds per warm JSON decode/validate/encode cycle, with validators prepared beforehand:
Two opposite run orders with pyperf, 3 workers per pass and 5 values per worker: 30 values per variant/workload. Model-page variability was about 16% CV in the native-API cell. These are representative local measurements.
Repeated warm identifier-validator construction, measured separately, cost 0.95 us on main, 7.96 us with the API, 3.82 us with released pcre2 and 26.0 us without pcre2. Similar setup costs already existed in the published PR and were confirmed independently on macOS. Small runtime differences in no-pattern/fallback controls varied across runs.
Related to #925.