feat: SQL COUNT(DISTINCT column) support via HyperLogLog Sketch#368
Merged
milindsrivastava1997 merged 3 commits intoMay 26, 2026
Conversation
…b) and CARDINALITY routing
milindsrivastava1997
approved these changes
May 26, 2026
milindsrivastava1997
deleted the
367-feat-sql-countdistinct-column-support-via-hyperloglog-sketch
branch
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds end-to-end support for
COUNT(DISTINCT column)on the precompute streaming path using a HyperLogLog (HLL) sketch (asap_sketchlib::HllSketch). Enables NetFlow-style queries such as distinct dstip per srcip with ORDER BY / LIMIT, and supports benchmarking against ClickHouse on the same ingest window.COUNT(DISTINCT col)is parsed as CARDINALITY, routed toStatistic::Cardinality, matched toAggregationType::HLL, updated via HllAccumulatorUpdater, stored with lossless msgpack serde, and answered at query time withHllSketch::estimate().Code Changes
SQL Parsing & Validation (
sql_utilities)sqlpattern_parser.rsCOUNT(DISTINCT single_column)→ aggregation name "CARDINALITY"; reject multi-columnDISTINCTandDISTINCTon non-COUNTaggregatessqlpattern_matcher.rslegal_aggregations; for CARDINALITY only, accept distinct target inmetadata_columnsorvalue_columnssqlparser_test.rsORDER BY/LIMIT, SpatioTemporal classificationStatistic Routing (
promql_utilities)enums.rsAggregationOperator::Cardinality→Statistic::Cardinality(FromStr,to_statistics,as_str,is_approximate)Capability Matching (
asap_types)capability_matching.rscompatible_agg_types(Cardinality)includesAggregationType::HLL(single-population; no paired key aggregation)Precompute Engine (
query_engine_rust)precompute_operators/hll_accumulator.rsasap_sketchlib::HllSketch; implementsAggregateCore,SerializableToSink,SingleSubpopulationAggregate,MergeableAccumulatorprecompute_operators/mod.rshll_accumulatorprecompute_engine/accumulator_factory.rsHllAccumulatorUpdater,hll_precision_param(default 14, clamp 4–18),AggregationType::HLLdispatchengines/physical/accumulator_serde.rsSketchType::HLLindeserialize_accumulatoranddeserialize_single_subpopulationTests
33 new unit tests across: