Skip to content

Substitute references by position so escapes survive interpolation - #6489

Merged
denik merged 4 commits into
mainfrom
denik/escape-substitute-by-position
Sep 2, 2026
Merged

denik merged 4 commits into
mainfrom
denik/escape-substitute-by-position

Conversation

@denik

@denik denik commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Follow-up to #6484. Two bugs, both needing a $${...} escape and a real reference in the same string.

"$${foo} ${foo}" resolved to "$VALUE ${foo}" — interpolation used strings.Replace(..., 1), which found the escaped occurrence first. Ref now carries byte spans and substitution goes by offset.

An escaped placeholder beside a ${resources.*} reference failed the deploy with "unresolved references": structvar.ResolveRef wrote the escaped template back into the typed state, undoing the unescape done at plan time, and ContainsVariableReference counted the literal as still pending. It is now escape-aware, and ResolveRef substitutes via dynvar.ReplaceRef, which skips escaped occurrences.

acceptance/bundle/resource_deps/escaped_ref gains mixed and same_name fields for the two shapes.

Two bugs in the $${...} escape support added by #6484, both requiring an escape
and a real reference in the same string:

1. dynvar.Resolve interpolated with strings.Replace(..., 1), which finds the
   first occurrence of the match text. An escaped occurrence of the same
   reference earlier in the string absorbed the substitution, so
   "$${foo} ${foo}" resolved to "$VALUE ${foo}" -- literal corrupted, real
   reference left unresolved. Ref now carries byte spans and interpolation
   substitutes by offset.

2. An escaped placeholder sharing a field with a ${resources.*} reference failed
   the deploy with "unresolved references". structvar.ResolveRef wrote the
   escaped template back into the typed state, undoing the unescape done at
   plan time, and ContainsVariableReference counted the literal as still
   pending so the entry never cleared.

ContainsVariableReference is now escape-aware, and ResolveRef substitutes via
the new dynvar.ReplaceRef (which skips escaped occurrences), storing the
unescaped result in the struct while keeping the escaped template in Refs so the
pending check can still tell a literal from a real reference.

Co-authored-by: Isaac
@denik
denik force-pushed the denik/escape-substitute-by-position branch from 611f67d to 036358c Compare September 2, 2026 19:45
@denik
denik enabled auto-merge September 2, 2026 20:30
@denik
denik added this pull request to the merge queue Sep 2, 2026
Merged via the queue into main with commit b15124d Sep 2, 2026
33 checks passed
@denik
denik deleted the denik/escape-substitute-by-position branch September 2, 2026 21:12
deco-sdk-tagging Bot added a commit that referenced this pull request Sep 3, 2026
## Release v1.15.0

### CLI

 * When `uv python install` fails, `databricks environments setup-local` now falls back to a compatible Python interpreter already installed on the machine. ([#6457](#6457))
 * Allow `databricks environments setup-local` to update `pyproject.toml` files containing TOML multi-line strings. ([#6445](#6445))

### Bundles

 * Before committing the automatic terraform→direct migration, run a deployment plan against the converted state; if the plan fails the migration is abandoned. ([#6486](#6486))
 * The `dbt-sql` bundle template now uses Databricks Runtime 16.4 LTS (up from 15.4 LTS) for classic (non-serverless) compute. ([#6418](#6418))
 * Fixed the direct engine silently ignoring edits to duration and timestamp fields, such as a Lakebase endpoint's `suspend_timeout_duration`. Such a change planned `0 to change` and was never applied. ([#6377](#6377))
 * Fixed `$${...}` not escaping a literal `${...}` on the direct engine, which failed with an `invalid dependency` error. ([#6484](#6484), [#6489](#6489))
 * direct: Fix deploying an update to `postgres_projects.default_endpoint_settings`. ([#6440](#6440))
 * direct: Fix deploying an update to `postgres_endpoints.settings.pg_settings`. ([#6441](#6441))
 * direct: Fix deploying an update to `expire_time`, `ttl` or `suspend_timeout_duration` on Lakebase resources. ([#6443](#6443))
 * Added PyDABs (Python) support for catalogs: `Resources.add_catalog` and the `catalog_mutator` decorator. ([#6408](#6408))
 * Bundle templates now use serverless [environment version 5](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/five), which offers better performance, and `databricks-connect` 16.4. ([#6378](#6378))
 * Fixed a job with a `table_update` trigger never converging on the direct engine. ([#6442](#6442))

### Dependency Updates

 * Bump Go toolchain to 1.26.8. ([#6476](#6476))
deco-sdk-tagging Bot added a commit that referenced this pull request Sep 3, 2026
## Release v1.15.0

### CLI

 * When `uv python install` fails, `databricks environments setup-local` now falls back to a compatible Python interpreter already installed on the machine. ([#6457](#6457))
 * Allow `databricks environments setup-local` to update `pyproject.toml` files containing TOML multi-line strings. ([#6445](#6445))

### Bundles

 * Before committing the automatic terraform→direct migration, run a deployment plan against the converted state; if the plan fails the migration is abandoned. ([#6486](#6486))
 * The `dbt-sql` bundle template now uses Databricks Runtime 16.4 LTS (up from 15.4 LTS) for classic (non-serverless) compute. ([#6418](#6418))
 * Fixed the direct engine silently ignoring edits to duration and timestamp fields, such as a Lakebase endpoint's `suspend_timeout_duration`. Such a change planned `0 to change` and was never applied. ([#6377](#6377))
 * Fixed `$${...}` not escaping a literal `${...}` on the direct engine, which failed with an `invalid dependency` error. ([#6484](#6484), [#6489](#6489))
 * Remove forward_user_access_token from update_mask for Apps because it's not supported. Fixes regression in 1.14.1. ([#6510](#6510))
 * direct: Fix deploying an update to `postgres_projects.default_endpoint_settings`. ([#6440](#6440))
 * direct: Fix deploying an update to `postgres_endpoints.settings.pg_settings`. ([#6441](#6441))
 * direct: Fix deploying an update to `expire_time`, `ttl` or `suspend_timeout_duration` on Lakebase resources. ([#6443](#6443))
 * Added PyDABs (Python) support for catalogs: `Resources.add_catalog` and the `catalog_mutator` decorator. ([#6408](#6408))
 * Bundle templates now use serverless [environment version 5](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/five), which offers better performance, and `databricks-connect` 16.4. ([#6378](#6378))
 * Fixed a job with a `table_update` trigger never converging on the direct engine. ([#6442](#6442))

### Dependency Updates

 * Bump Go toolchain to 1.26.8. ([#6476](#6476))
janniklasrose pushed a commit that referenced this pull request Sep 15, 2026
…6489)

Follow-up to #6484. Two bugs, both needing a `$${...}` escape and a real
reference in the same string.

`"$${foo} ${foo}"` resolved to `"$VALUE ${foo}"` — interpolation used
`strings.Replace(..., 1)`, which found the escaped occurrence first.
`Ref` now carries byte spans and substitution goes by offset.

An escaped placeholder beside a `${resources.*}` reference failed the
deploy with "unresolved references": `structvar.ResolveRef` wrote the
escaped template back into the typed state, undoing the unescape done at
plan time, and `ContainsVariableReference` counted the literal as still
pending. It is now escape-aware, and `ResolveRef` substitutes via
`dynvar.ReplaceRef`, which skips escaped occurrences.

`acceptance/bundle/resource_deps/escaped_ref` gains `mixed` and
`same_name` fields for the two shapes.
janniklasrose pushed a commit that referenced this pull request Sep 15, 2026
## Release v1.15.0

### CLI

 * When `uv python install` fails, `databricks environments setup-local` now falls back to a compatible Python interpreter already installed on the machine. ([#6457](#6457))
 * Allow `databricks environments setup-local` to update `pyproject.toml` files containing TOML multi-line strings. ([#6445](#6445))

### Bundles

 * Before committing the automatic terraform→direct migration, run a deployment plan against the converted state; if the plan fails the migration is abandoned. ([#6486](#6486))
 * The `dbt-sql` bundle template now uses Databricks Runtime 16.4 LTS (up from 15.4 LTS) for classic (non-serverless) compute. ([#6418](#6418))
 * Fixed the direct engine silently ignoring edits to duration and timestamp fields, such as a Lakebase endpoint's `suspend_timeout_duration`. Such a change planned `0 to change` and was never applied. ([#6377](#6377))
 * Fixed `$${...}` not escaping a literal `${...}` on the direct engine, which failed with an `invalid dependency` error. ([#6484](#6484), [#6489](#6489))
 * direct: Fix deploying an update to `postgres_projects.default_endpoint_settings`. ([#6440](#6440))
 * direct: Fix deploying an update to `postgres_endpoints.settings.pg_settings`. ([#6441](#6441))
 * direct: Fix deploying an update to `expire_time`, `ttl` or `suspend_timeout_duration` on Lakebase resources. ([#6443](#6443))
 * Added PyDABs (Python) support for catalogs: `Resources.add_catalog` and the `catalog_mutator` decorator. ([#6408](#6408))
 * Bundle templates now use serverless [environment version 5](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/five), which offers better performance, and `databricks-connect` 16.4. ([#6378](#6378))
 * Fixed a job with a `table_update` trigger never converging on the direct engine. ([#6442](#6442))

### Dependency Updates

 * Bump Go toolchain to 1.26.8. ([#6476](#6476))
janniklasrose pushed a commit that referenced this pull request Sep 15, 2026
## Release v1.15.0

### CLI

 * When `uv python install` fails, `databricks environments setup-local` now falls back to a compatible Python interpreter already installed on the machine. ([#6457](#6457))
 * Allow `databricks environments setup-local` to update `pyproject.toml` files containing TOML multi-line strings. ([#6445](#6445))

### Bundles

 * Before committing the automatic terraform→direct migration, run a deployment plan against the converted state; if the plan fails the migration is abandoned. ([#6486](#6486))
 * The `dbt-sql` bundle template now uses Databricks Runtime 16.4 LTS (up from 15.4 LTS) for classic (non-serverless) compute. ([#6418](#6418))
 * Fixed the direct engine silently ignoring edits to duration and timestamp fields, such as a Lakebase endpoint's `suspend_timeout_duration`. Such a change planned `0 to change` and was never applied. ([#6377](#6377))
 * Fixed `$${...}` not escaping a literal `${...}` on the direct engine, which failed with an `invalid dependency` error. ([#6484](#6484), [#6489](#6489))
 * Remove forward_user_access_token from update_mask for Apps because it's not supported. Fixes regression in 1.14.1. ([#6510](#6510))
 * direct: Fix deploying an update to `postgres_projects.default_endpoint_settings`. ([#6440](#6440))
 * direct: Fix deploying an update to `postgres_endpoints.settings.pg_settings`. ([#6441](#6441))
 * direct: Fix deploying an update to `expire_time`, `ttl` or `suspend_timeout_duration` on Lakebase resources. ([#6443](#6443))
 * Added PyDABs (Python) support for catalogs: `Resources.add_catalog` and the `catalog_mutator` decorator. ([#6408](#6408))
 * Bundle templates now use serverless [environment version 5](https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/five), which offers better performance, and `databricks-connect` 16.4. ([#6378](#6378))
 * Fixed a job with a `table_update` trigger never converging on the direct engine. ([#6442](#6442))

### Dependency Updates

 * Bump Go toolchain to 1.26.8. ([#6476](#6476))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants