Repository navigation
json.dumps takes significant CPU time in data heavy workloads like ETL or streaming #689
Description
Hi Team,
I use the crate-python driver during CDC process that writes data to CrateDB.
During benchmarking and optimizations of the process I recoded CPU flamegraphs that show that a lot of CPU time was spent during json serialization
crate-python/src/crate/client/http.py
Line 337 in c6892d5
There are publicly available benchmarks reporting that json.dumps is very slow, and changing the library to ujson or orjson can make a huge difference. I can confirm that swapping json to ujson reduces time spent on serialization by around 30-40%.
I didn't check how this impacts correctness since the current implementation uses JSON.dumps(cls=) to provide custom transformation logic.
Cheers,
Gabriel
Activity
amotl commentedon Jan 15, 2025
Thanks for your suggestion. Switching to orjson seems to work well without much ado, as it doesn't break the test suite.
orjsonto improve JSON marshalling performance #691You can test the package in your application by using
crate==2.0.0.dev0.amotl commentedon Jan 16, 2025
If you are using SQLAlchemy, please also check this patch, which will unlock a communication path not using JSON marshalling at all.
asyncpgandpsycopg3drivers sqlalchemy-cratedb#11Just yesterday @seut emphasized within another conversation that the communication path using the PostgreSQL wire protocol has many benefits when compared to the HTTP/JSON-based communication, specifically about efficiency matters, especially when shuffling large data around in the context of streaming query results from the database cluster to the client.
/cc @karynzv, @surister, @wierdvanderhaar, @hlcianfagna, @hammerhead, @simonprickett, @kneth
amotl commentedon Jan 16, 2025
@seut: Do you agree to include GH-691 into a release version 2.0.0, as proposed?
@widmogrod: Because we received approval of GH-691, may I kindly ask you to test
crate>=2.0.0.dev0already in a downstream application (actually, as many as possible would be excellent), in order to validate that nothing goes south?widmogrod commentedon Jan 16, 2025
Thank you for your fast response 🚀 .
I'm looking at the change.
I will let you know how things work, when I re-run benchmarks with the current and new versions.
amotl commentedon Jan 16, 2025
widmogrod commentedon Jan 16, 2025
I used poetry to fetch
crate = "^2.0.0.dev1"and run benchmarkThis is how a section of the flame chart looks like
crate = "^1.0.0"crate = "^2.0.0.dev1"This part of the code is much more faster:
amotl commentedon Jan 17, 2025
Update 2.0.0.dev5
Hi again. The improvements in GH-691 have been shipped per pre-release packages. Version 2.0.0.dev5 is the most recent one, and it is available on PyPI, together with an SQLAlchemy dialect package that permits installing it.
They can be used to be slotted into downstream applications and frameworks of our and 3rd party realms, in order to give them more eyeballs.
amotl commentedon Jan 18, 2025
Update 2.0.0.dev6
Hi again. The improvements in GH-691 have been updated and shipped per pre-release package version 2.0.0.dev6, again together with a corresponding SQLAlchemy dialect package.
Please use them in your downstream applications and frameworks, we will be happy to hear back about any outcomes.
widmogrod commentedon Jan 20, 2025
I can confirm that on version dev6, performance benefits remain consistent
amotl commentedon Jan 30, 2025
Hi again. Version 2.0.0 has been released, which includes the improvement you have been waiting for. So, I am closing this.