I have an entity kind with tens of thousands of entities in Cloud Firestore in Datastore mode. (Python 3.6 & 3.7; google-cloud-ndb==1.7.1) When I try to count the results of queries, sometimes a few entities are missed. Here is an example:
query = Product.query(
Product.is_cool == True,
Product.ingred_list.ingred_key == ndb.Key('Ingredient', 'default', 'Ingredient', 42),
ancestor=ndb.Key('Product', 'default'))
print(len(query.fetch(keys_only=True)))
print(query.count())
The expected output would be:
But instead I get:
I put some print statements into _datastore_query._count_by_skipping and tweaked the logic to see all the batches returned from the backend:
batch {
entity_result_type: KEY_ONLY
skipped_cursor: "cursor1"
end_cursor: "cursor2"
more_results: NOT_FINISHED
skipped_results: 2
snapshot_version: 1606244957318020
}
batch {
entity_result_type: KEY_ONLY
end_cursor: "cursor3"
more_results: NOT_FINISHED
snapshot_version: 1606244957551338
}
batch {
entity_result_type: KEY_ONLY
skipped_cursor: "cursor4"
end_cursor: "cursor5"
more_results: NO_MORE_RESULTS
skipped_results: 1
snapshot_version: 1606244957805378
}
It looks like that the second batch triggers the bail-out logic for the Datastore Emulator, so according to the current logic, the third batch would never be counted. This actually looks to me like a variant of #386
Based on the above results, I've created a test case in a PR: #574 Sorry, no fix, just a broken test.
(And once again, the actual data to reproduce this error is huge: it's 10000+ entities and if I delete some of them the issue fixes itself.)
I have an entity kind with tens of thousands of entities in Cloud Firestore in Datastore mode. (Python 3.6 & 3.7; google-cloud-ndb==1.7.1) When I try to count the results of queries, sometimes a few entities are missed. Here is an example:
The expected output would be:
But instead I get:
I put some
printstatements into_datastore_query._count_by_skippingand tweaked the logic to see all the batches returned from the backend:It looks like that the second batch triggers the bail-out logic for the Datastore Emulator, so according to the current logic, the third batch would never be counted. This actually looks to me like a variant of #386
Based on the above results, I've created a test case in a PR: #574 Sorry, no fix, just a broken test.
(And once again, the actual data to reproduce this error is huge: it's 10000+ entities and if I delete some of them the issue fixes itself.)