Storing many records with the indexeddb-source is extremely slow in Google Chrome.
Interestingly Firefox only takes seconds, while Google Chrome takes a minute(!) for storing ~5.000 records.
In my setup I have a transform with 5k records. The key problem seems to be my schema in that each record has 3 relationships. So it effectively stores the 5k records + 15k records about relationship infos in __inverseRels__.
After debugging, it seems that writing to idb (setRecordAsync) is not the problem, but reading (getRecordAsync):
(with latest master; a transform with 5.000 records; in the schema each record has 3 relationships)
// Google Chrome v87
Memory Source 0.23s
IndexedDB 49.28s
- getRecordAsync 39.12s (~20.000 times invoked)
- setRecordAsync 4.30s ( ~5.000 times invoked)
- addInverseRelationshipsAsync 2.51s ( ~5.000 times invoked)
// Firefox v83
Memory Source 1.2s
IndexedDB 5.90s
- getRecordAsync 3.22s (~20.000 times invoked)
- setRecordAsync 0.96s ( ~5.000 times invoked)
- addInverseRelationshipsAsync 0.34s ( ~5.000 times invoked)
After some more digging I found this:
https://chromestatus.com/feature/5730701489995776
And using { durability: "relaxed } for each idb transaction, the numbers got down - but it still took Chrome ~20seconds.
We also have some rare cases where we want to store ~10k records - and it ends up taking ~1,5 minutes.
A tempting workaround would be to ignore the cache processors via the cache settings:
- AsyncSchemaValidationProcessor
- AsyncSchemaConsistencyProcessor
- AsyncCacheIntegrityProcessor
-> What are the effects of this?
If I understand correctly, these processors are used to prevent the database of getting into a corrupted state, right? What would such scenario... e.g. that the schema or api has changed?
Another thought was to use source.cache.setRecordsAsync directly - as I'd be able to skip the processors - but they're still active when making smaller changes later...
(without async-record-cache processors)
// Google Chrome
Memory Source 0.1s
IndexedDB 19.80s
- getRecordAsync 15.91s (~10.000 times invoked)
- setRecordAsync 3.29s (~5.000 times invoked)
- addInverseRelationshipsAsync 0s (0 times invoked)
(without async-record-cache processors)
And in combination with the relaxed durability flag, it even comes close to Firefox:
(without async-record-cache processors; transactions with "relaxed durability")
// Google Chrome
Memory Source 0.1s
IndexedDB 4.79s
- getRecordAsync 3.02s (~10.000 times invoked)
- setRecordAsync 1.51s (~5.000 times invoked)
- addInverseRelationshipsAsync 0s (0 times invoked)
Storing many records with the indexeddb-source is extremely slow in Google Chrome.
Interestingly Firefox only takes seconds, while Google Chrome takes a minute(!) for storing ~5.000 records.
In my setup I have a transform with 5k records. The key problem seems to be my schema in that each record has 3 relationships. So it effectively stores the 5k records + 15k records about relationship infos in
__inverseRels__.After debugging, it seems that writing to idb (setRecordAsync) is not the problem, but reading (getRecordAsync):
After some more digging I found this:
https://chromestatus.com/feature/5730701489995776
And using
{ durability: "relaxed }for each idb transaction, the numbers got down - but it still took Chrome ~20seconds.We also have some rare cases where we want to store ~10k records - and it ends up taking ~1,5 minutes.
A tempting workaround would be to ignore the cache processors via the cache settings:
-> What are the effects of this?
If I understand correctly, these processors are used to prevent the database of getting into a corrupted state, right? What would such scenario... e.g. that the schema or api has changed?
Another thought was to use
source.cache.setRecordsAsyncdirectly - as I'd be able to skip the processors - but they're still active when making smaller changes later...And in combination with the relaxed durability flag, it even comes close to Firefox: