Skip to content

Remove approximate indexes for ORDER BY queries without a LIMIT - #1031

Open
oliness wants to merge 1 commit into
pgvector:masterfrom
oliness:feature/846-fix
Open

oliness wants to merge 1 commit into
pgvector:masterfrom
oliness:feature/846-fix

Conversation

@oliness

@oliness oliness commented Sep 21, 2026

Copy link
Copy Markdown

An HNSW or IVFFlat scan stops after ef_search rows (or the probed lists), so the planner shouldn't pick one for a query that needs every row.

@ankane

ankane commented Sep 23, 2026

Copy link
Copy Markdown
Member

Hi @oliness, thanks for the PR.

This seems to work well for some queries I've tested (better than limit_tuples in #424), but I'm worried it's more risky / brittle / could break other queries. For instance, tuple_fraction is set to zero in a number of situations.

@jkatz @hlinnaka any thoughts?

HNSW and IVFFlat scans return a limited number of tuples, so an ordered
index scan chosen for a query that needs every row (no LIMIT) silently
returned truncated results. Treat root->tuple_fraction <= 0 like the
no-ORDER-BY case in hnswcostestimate and ivfflatcostestimate, unless
enable_seqscan is off (the documented way to force an index).

Add TAP tests for queries planned with tuple_fraction = 0, including
subqueries under an outer aggregate, GROUP BY, HAVING, DISTINCT, ORDER BY
or join, and for queries that should still use the index.
@oliness

oliness commented Sep 23, 2026

Copy link
Copy Markdown
Author

I've added TAP tests for the tuple_fraction = 0 case: test/t/049_hnsw_no_limit.pl and test/t/050_ivfflat_no_limit.pl, one per index type. They run with enable_sort = off, so the index is chosen whenever it's allowed.

For queries planned with tuple_fraction = 0, they check that the index isn't used and that the results match a scan with enable_indexscan = off:

  • no LIMIT
  • a subquery without its own LIMIT, under an outer aggregate, GROUP BY, ROLLUP, HAVING, DISTINCT, ORDER BY on another column, or join
  • the same kind of subquery when the outer query only reads the first rows: an outer ORDER BY on the same distance with a LIMIT, or a join ordered by distance with a LIMIT

They also check that these queries still use the index:

  • a LIMIT
  • an outer LIMIT with nothing else in the outer query
  • a subquery with its own LIMIT, under an aggregate, ORDER BY, or join
  • no LIMIT with enable_seqscan = off, at the top level and in a subquery

On master, 17 of the 27 checks in each file fail, including every "no index scan" check. The checks for queries that should still use the index pass on master too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants