Skip to content

fix: Kepler table-layer dataset identity handling for DuckDB-backed maps - #703

Merged
ilyabo merged 7 commits into
mainfrom
keplerTableIdentity
Jun 10, 2026
Merged

ilyabo merged 7 commits into
mainfrom
keplerTableIdentity

Conversation

@ilyabo

@ilyabo ilyabo commented Jun 9, 2026 •

Copy link
Copy Markdown
Collaborator

Changes:

Added keplerTableIdentity.ts to centralize Kepler dataset-id policy for DuckDB tables.
Use bare dataset ids for tables in the current DuckDB database/schema.
Use qualified table references for attached databases or non-current schemas.
Added restore resolution that accepts both bare current-schema ids and qualified table references.
Updated CustomLayerManager Add Layer options to use the shared identity helper.
Improved Add Layer labels to omit current DB/schema and avoid SQL identifier quotes.
Re-exported the helper APIs from @sqlrooms/kepler.

Why:

Kepler persists layer config by dataId, while table data is reloaded later from DuckDB. Keeping this identity policy centralized prevents map layers from disappearing on project reload and avoids collisions with attached databases.

Summary by CodeRabbit

  • New Features

    • Configurable table-selection policy to control which DB tables are exposed and how dataset IDs/labels are generated/resolved.
    • New table/source picker UI and selector for adding layers/filters, showing loaded vs unloaded state and supporting on-demand loading.
    • addTableToMap now accepts an object-style parameter (documented API change).
  • Tests

    • Added comprehensive tests for ID/label generation, resolution, filtering, and customization.
  • Documentation

    • Updated README and upgrade guide with usage and migration notes.

@ilyabo ilyabo changed the title fix: keplerTableIdentity fix: Layer persistence in maps Jun 9, 2026
@coderabbitai

coderabbitai Bot commented Jun 9, 2026 •

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 20bc1b3f-12d3-4fc2-be7c-3e5566623b00

📥 Commits

Reviewing files that changed from the base of the PR and between 3266f3e and 1a357c7.

📒 Files selected for processing (6)
  • docs/upgrade-guide.md
  • packages/kepler/README.md
  • packages/kepler/src/KeplerSlice.ts
  • packages/kepler/src/components/CustomLayerManager.tsx
  • packages/kepler/src/components/CustomSourceDataSelector.tsx
  • packages/kepler/src/index.ts
✅ Files skipped from review due to trivial changes (1)
  • docs/upgrade-guide.md
🚧 Files skipped from review as they are similar to previous changes (3)
  • packages/kepler/src/index.ts
  • packages/kepler/src/components/CustomLayerManager.tsx
  • packages/kepler/src/components/CustomSourceDataSelector.tsx

📝 Walkthrough

Walkthrough

Kepler adds a policy-driven table-selection module and stores resolved tableSelection in slice state. Dataset sync and add/load flows now resolve saved dataset ids to DuckDB tables using the new helpers, and UI selectors present merged loaded/unloaded table options. Tests and Jest config are included.

Changes

Table identity consistency across schemas

Layer / File(s) Summary
Table identity helpers and types
packages/kepler/src/keplerTableSelection.ts
New exported types and helpers implement configurable table exposure, dataset-id generation, display labels, dataset-id→table resolution, and option builders for selectors.
Slice state and initialization with schema resolution
packages/kepler/src/KeplerSlice.ts
CreateKeplerSliceOptions/KeplerSliceState accept tableSelection; slice initialization resolves defaultDbSchema from DB context when omitted, persists resolved tableSelection, updates addTableToMap label logic, and changes sync logic to resolve dataset ids via the identity helpers.
UI selectors and add/load flows
packages/kepler/src/components/*, packages/kepler/src/components/KeplerTableSourceSelector.tsx, packages/kepler/src/components/CustomSourceDataSelector.tsx, packages/kepler/src/components/CustomLayerManager.tsx, packages/kepler/src/components/CustomFilterManager.tsx, packages/kepler/src/components/KeplerInjector.tsx
Replaces prior typeahead/dropdown wiring with KeplerTableSourceSelector driven by buildKeplerTableSourceOptions; Add Layer/Add Filter now use KeplerTableSourceOption values; adds a SourceDataSelector factory that can load unloaded tables; injector registers the custom factory.
Public API exports
packages/kepler/src/index.ts
Re-exports table-selection helpers and types (buildKeplerTableLayerOptions, findKeplerTableForDatasetId, getKeplerDatasetIdForTable, getKeplerTableLabel, shouldIncludeKeplerTable, and related types).
Comprehensive test coverage and Jest infrastructure
packages/kepler/__tests__/keplerTableSelection.test.ts, packages/kepler/jest.config.js, packages/kepler/package.json
Adds Jest config, test scripts, devDependencies, and tests covering dataset-id generation, label formatting, resolution logic, inclusion filtering, deduplication, and custom callback hooks.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

  • sqlrooms/sqlrooms#533: Both PRs modify packages/kepler/src/KeplerSlice.ts—specifically syncKeplerDatasets and the addTableToMap-driven table/dataset resolution logic—to ensure only the intended DuckDB tables/dataIds are added to Kepler.
  • sqlrooms/sqlrooms#671: Both PRs modify packages/kepler/src/KeplerSlice.ts around how addTableToMap derives dataset labels from (possibly fully qualified) table names and how syncKeplerDatasets discovers/loads referenced tables.
  • sqlrooms/sqlrooms#349: Both PRs modify packages/kepler/src/components/KeplerInjector.tsx’s injector/recipe wiring so that custom factory replacements get registered and resolved at runtime.

Suggested reviewers

  • bdjulbic
  • lixun910

Poem

🐇 I hop through catalogs and names,
Matching tables in schema games.
Bare when default, quoted when not,
Labels tidy, duplicates dropped.
Hooray — the selector finds the spot! 🥕

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 48.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately describes the main change: introducing dataset identity handling for DuckDB-backed Kepler maps, which is the core objective of centralizing the Kepler dataset-id policy.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch keplerTableIdentity

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@ilyabo ilyabo changed the title fix: Layer persistence in maps fix: Kepler table-layer dataset identity handling for DuckDB-backed maps. Jun 9, 2026
@ilyabo ilyabo changed the title fix: Kepler table-layer dataset identity handling for DuckDB-backed maps. fix: Kepler table-layer dataset identity handling for DuckDB-backed maps Jun 9, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
packages/kepler/src/keplerTableIdentity.ts (1)

117-124: 💤 Low value

Consider checking currentDatabase in addition to currentSchema.

The filter on line 118 only checks schema equality, but isCurrentTable checks both database and schema. If an attached database has the same schema name (e.g., "main"), those tables would appear in the dropdown—which may be unintended.

This inconsistency likely won't cause functional issues since getKeplerDatasetIdForTable will still produce correct qualified IDs for cross-database tables, but it may show unexpected entries in the dropdown.

Proposed fix for consistency
   for (const table of dbTables) {
-    if (table.table.schema !== context.currentSchema) continue;
+    if (
+      table.table.database !== context.currentDatabase ||
+      table.table.schema !== context.currentSchema
+    ) {
+      continue;
+    }

     const value = getKeplerDatasetIdForTable(table, context);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/kepler/src/keplerTableIdentity.ts` around lines 117 - 124, The loop
building optionsByValue filters tables by schema only (context.currentSchema)
which is inconsistent with isCurrentTable that checks both database and schema;
update the filter in the for loop over dbTables to also verify the table's
database matches context.currentDatabase (or treat attached DBs appropriately)
before calling getKeplerDatasetIdForTable and getKeplerTableLabel so dropdown
entries only include tables from the current database+schema scope.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@packages/kepler/src/keplerTableIdentity.ts`:
- Around line 117-124: The loop building optionsByValue filters tables by schema
only (context.currentSchema) which is inconsistent with isCurrentTable that
checks both database and schema; update the filter in the for loop over dbTables
to also verify the table's database matches context.currentDatabase (or treat
attached DBs appropriately) before calling getKeplerDatasetIdForTable and
getKeplerTableLabel so dropdown entries only include tables from the current
database+schema scope.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d9002ff7-bbc5-4544-8cc3-908f91f6f2cf

📥 Commits

Reviewing files that changed from the base of the PR and between ba95750 and 40a5efa.

📒 Files selected for processing (4)
  • packages/kepler/src/KeplerSlice.ts
  • packages/kepler/src/components/CustomLayerManager.tsx
  • packages/kepler/src/index.ts
  • packages/kepler/src/keplerTableIdentity.ts

@ilyabo

ilyabo commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
packages/kepler/src/KeplerSlice.ts (1)

606-645: ⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Separate dataset ID from SQL table reference in addTableToMap.

tableName is currently used as both Kepler dataset id (info.id) and SQL query target. With custom tableSelection.getDatasetIdForTable / findTableForDatasetId, dataset ids can be aliases that are not queryable SQL identifiers, so restore/add flows can fail even after successful table resolution.

💡 Proposed fix
         addTableToMap: async (mapId, tableName, options = {}, config = {}) => {
+          const resolvedTable = findKeplerTableForDatasetId(
+            get().db.tables,
+            tableName,
+            get().kepler.tableSelection,
+          );
+          const sqlTableName = resolvedTable?.table.toString() ?? tableName;
           const connector = await get().db.getConnector();
           const duckDbColumns = await getDuckDBColumnTypes(
             {
               query: (query: string) => connector.query(query).result,
             } as DatabaseConnection,
-            tableName,
+            sqlTableName,
           );
           const tableDuckDBTypes = getDuckDBColumnTypesMap(duckDbColumns);
           const adjustedQuery = castDuckDBTypesForKepler(
-            tableName,
+            sqlTableName,
             duckDbColumns,
           );
           const arrowResult = await connector.query(adjustedQuery).result;
@@
             const datasets: AddDataToMapPayload['datasets'] = {
               data: {fields, cols, rows: [], arrowTable: arrowResult},
               info: {label, id: tableName},
-              metadata: {tableName},
+              metadata: {tableName: sqlTableName},
             };
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/kepler/src/KeplerSlice.ts` around lines 606 - 645, addTableToMap
currently reuses the incoming tableName as the Kepler dataset id (info.id) which
breaks when dataset ids are aliases; fix by separating SQL reference from
dataset id: keep the original tableName as the SQL source in metadata.tableName,
then derive datasetId via the table selection API (e.g. if a table object is
found via findKeplerTableForDatasetId(...) use
get().kepler.tableSelection.getDatasetIdForTable(table) or otherwise fall back
to getUnqualifiedSqlIdentifier(String(tableName)) || tableName), and set
datasets.info.id = datasetId; ensure getKeplerTableLabel(...) still uses the
resolved table object for label logic and do not replace metadata.tableName with
the dataset id.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/kepler/src/components/CustomSourceDataSelector.tsx`:
- Around line 90-114: Select the raw state.kepler.map with useStoreWithKepler
and remove the heavy logic from the selector; then compute mapId and mapDatasets
from that raw map using useMemo. Concretely: replace the current selector that
computes mapId (the Object.entries + layers.some logic) with a selector that
returns state.kepler.map only (via useStoreWithKepler), then add two useMemo
hooks that derive mapId (iterate entries and find layerId match) and mapDatasets
(lookup mapId -> visState.datasets) so derivation is outside the Zustand
selector.

---

Outside diff comments:
In `@packages/kepler/src/KeplerSlice.ts`:
- Around line 606-645: addTableToMap currently reuses the incoming tableName as
the Kepler dataset id (info.id) which breaks when dataset ids are aliases; fix
by separating SQL reference from dataset id: keep the original tableName as the
SQL source in metadata.tableName, then derive datasetId via the table selection
API (e.g. if a table object is found via findKeplerTableForDatasetId(...) use
get().kepler.tableSelection.getDatasetIdForTable(table) or otherwise fall back
to getUnqualifiedSqlIdentifier(String(tableName)) || tableName), and set
datasets.info.id = datasetId; ensure getKeplerTableLabel(...) still uses the
resolved table object for label logic and do not replace metadata.tableName with
the dataset id.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 192ecf5b-1d52-4bb1-8340-689f5d436435

📥 Commits

Reviewing files that changed from the base of the PR and between e50f4c2 and 059333d.

📒 Files selected for processing (7)
  • packages/kepler/src/KeplerSlice.ts
  • packages/kepler/src/components/CustomFilterManager.tsx
  • packages/kepler/src/components/CustomLayerManager.tsx
  • packages/kepler/src/components/CustomSourceDataSelector.tsx
  • packages/kepler/src/components/KeplerInjector.tsx
  • packages/kepler/src/components/KeplerTableSourceSelector.tsx
  • packages/kepler/src/keplerTableSelection.ts

Comment thread packages/kepler/src/components/CustomSourceDataSelector.tsx Outdated
@ilyabo

ilyabo commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@ilyabo

ilyabo commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@ilyabo

ilyabo commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.

@ilyabo

ilyabo commented Jun 9, 2026

Copy link
Copy Markdown
Collaborator Author

@codex review

@ilyabo
ilyabo merged commit aa35765 into main Jun 10, 2026
4 checks passed
@ilyabo
ilyabo deleted the keplerTableIdentity branch June 10, 2026 13:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant