Optimize memory usage of Datom type#263
Conversation
|
Wow, lots of effort here! I don’t think I see how something like this could be merged into DataScript database. My concerns are existing clients, breaking changes, compatibility and JS portability. But I’m happy to see it exists as a separate project on its own. Couple of questions though, purely out of curiosity:
Add to Then run (in |
|
Thanks for the feedback. I will close this PR.
It's the only number type available on JavaScript Engines.
I measured memory usage in Chrome with the following code: Regular Datascript (167 MB) (def db
(d/init-db (for [i (range 1e6)]
(d/datom i :name i))
{:name {:db/order 0}}))With optimized Datom type (76 MB)(def db
(d/init-db (for [i (range 1e6)]
(d/datom i 0 i))
{:name {:db/order 0}})) |
As I would like to use Datascript for large datasets, I am trying to optimise the memory usage.
This patch should reduce memory usage by at least 50%.
[tuned-datascript "0.16.5-SNAPSHOT"]There are some changes to the API:
Internally the new Datom type is implemented as a pointer to :v and a number, which stores Eid and Aid in the lower 32 bits. The remaining 21 Bits in the double number are used for the added flag and the transaction ID.
Needs some work on the JS wrapper and I could not figure out how to benchmark Datascript.