Conversation
The bigtable integration suite was taking > 1 hour to run with the presubmit's profile list (`.kokoro/presubmit/bigtable-integration.cfg`). Measured from a full run: emulator-it 66s + prod-it 2414s + prod-batch-it 905s ≈ 56 min wall clock. Three separate problems, all fixed here. ### 1. The suite was running single threaded `google-cloud-bigtable/pom.xml` sets `forkCount=4` on failsafe, but the `enable-verbose-grpc-logs` profile overrode it back to `forkCount=1`. Profile plugin config wins, so every run with that profile serialized the entire IT suite into one JVM. The forked-JVM banner in the logs only ever mentions "forked JVM 1", confirming it. The override dates back to googleapis#1004 / googleapis#2295, when `TestEnvRule` installed a JUL appender that wrote grpc logs to the console and interleaved output across forks. That appender is gone — `TestEnvRule.grpcLogHandler` is declared and read in `teardownLogging()` but never assigned — and `redirectTestOutputToFile=true` now gives each fork its own `-output.txt`. So cross-process forking is safe; only in-JVM `parallel` needs to stay off for log attributability. The profile now keeps `<parallel>none</parallel>` and `<redirectTestOutputToFile>true</redirectTestOutputToFile>` but no longer pins `forkCount`. Fork count is also now a property (`-Dbigtable.it.fork-count=8`) since the ITs are RPC bound and can go well above the core count. ### 2. The prod-batch pass was redundant `bigtable-prod-batch-it` re-ran the entire `data.v2.it` package against `batch-bigtable.googleapis.com` — 905s, ~27% of the presubmit's wall clock. It was added in 484b62a (googleapis#892, "fix: jwt authentication on batch-bigtable.googleapis.com") to prove end to end that self-signed JWT auth worked against an endpoint whose service host and JWT audience diverge. That rationale no longer holds: - `EnhancedBigtableStubSettings.Builder.setJwtAudienceMapping` — the endpoint-to-audience table the profile guarded — is now `@Deprecated` and a no-op that just returns `this`. - `batch-bigtable.googleapis.com` appears nowhere in main source. - The profile passed `bigtable.data-jwt-audience=${bigtable.cfe-jwt-audience}`, the same value as `prod-it`, and that property defaults to empty. So the only difference between the two passes was the data endpoint hostname. The auth behaviour for that endpoint is still covered hermetically by `EnhancedBigtableStubTest`, which points the stub at `batch-bigtable.googleapis.com:443` with JWT credentials against an in-process server and asserts the emitted authorization header. Removes the `bigtable-prod-batch-it` profile, the `internal-bigtable-prod-batch-it-prop-helper` helper profile and the `bigtable.cfe-data-batch-endpoint` property. > **Note:** `.kokoro/presubmit/bigtable-integration.cfg` still lists > `bigtable-prod-batch-it` in `INTEGRATION_TEST_ARGS` and needs the profile > dropped from that list. It is the only CI config referencing it > (`bigtable-integration-dp.cfg` uses `bigtable-directpath-it` and is > unaffected). ### 3. Slow tests - **`LargeRowIT.testSkipLargeRow`** (242s prod / 79s batch) sent 200 sequential 3 MiB `mutateRow` calls — 600 MiB of serial round trips. The writes target distinct qualifiers and are independent, so they now go out with a bounded 16-deep in-flight window. - **`BigtableBackupIT`** (485s) had two flat 2-minute sleeps. `RestoreTable` only kicks off the optimize-restored-table operation once the backup is a couple of minutes old, and each restore test slept through that window itself. The backups are now created in `setUpClass()` and the tests only wait for whatever is left of the window after the rest of the class has run. - **`BigtableCmekIT`** (422s) polled key status on a `{5, 10, 50, 100, 150, 200, 250, 300}` second backoff, so it could oversleep 95s+ past the point where the status was already OK. Now a 10s fixed interval against a 360s deadline. - **`BigtableMaterializedViewIT`, `BigtableLogicalViewIT`, `BigtableSchemaBundleIT`, `BigtableAuthorizedViewIT`** each waited for a delete to propagate on a doubling `{2, 4, ... 1024}` backoff (2046s worst case). The resource normally disappears within seconds, so these are now a 2s interval with a 60 attempt cap. ### Follow-ups not done here - `reuseForks=false` costs ~20s of JVM boot per test class (~763s across 36 cloud forks). The pom comment says `BuiltinMetricsIT` mutates process-global metrics state, so enabling reuse means first splitting that test into its own isolated execution. - `TestEnvRule.cleanUpStale()` runs after every test class (36+ times per run) and does a project-wide `listInstances` + `listClusters` + `listTables` + `listAppProfiles` scan, but only deletes resources older than a day. It is logically a once-per-run job. - The admin view ITs (`BigtableMaterializedViewIT`, `BigtableLogicalViewIT`, `BigtableSchemaBundleIT`) create and delete a fresh table in `@Before`/ `@After` for every test method. No test mutates that table — it is only referenced by name in the view query — so one table per class would do.
…JsonCallableFactory (googleapis#14242) This change wires up the minimal Callable implementation from googleapis#14241 into the relevant existing factory classes.
…s#14353) RequestIdTargetTracker keys its cache on the logical request key, but it accepted the x-goog-spanner-request-id header value and derived that key by calling XGoogSpannerRequestId.of(String), which runs a six-group regex plus four Long.parseLong calls. Every caller already holds the parsed XGoogSpannerRequestId in the gRPC CallOptions. HeaderInterceptor paid this twice per RPC (a get on the first response and a remove on close). KeyAwareChannel added a third by rendering the object it already held back into a header string with getHeaderValue(), only for the tracker to parse it again. Location-aware routing is the only producer of tracked targets and is disabled unless the instance type is OMNI, so for most clients the cache stays empty for the lifetime of the process and all of that work is wasted. The tracker now accepts the parsed request id (or the logical key, where the caller already has one) and short circuits on a volatile flag that is set the first time a target is recorded. This also fixes a latent bug in KeyAwareChannel.onClose, which passed an already-normalized key into remove(String). The regex never matched it, so every RPC on that path constructed and threw an IllegalStateException, including its concatenated message and stack trace fill-in, before the catch block fell back to returning the raw string. Measured per RPC (JMH, 2 forks x 8 x 1s, -prof gc): ``` before 581.2 ns 1376 B after, default config 2.0 ns 0 B after, location-aware on 42.5 ns 56 B ```
…n to copy_and_rename (googleapis#14350) Update pinned librarian version to v0.42.1-0.20260910200322-b1deabc79631 and migrate method_operations actions from duplicate to copy_and_rename. For googleapis/librarian#7073
…ead API and nested types (googleapis#14332) b/544839155 This PR enables picosecond precision (`TIMESTAMP(12)`) support across the BigQuery Storage Read API (Arrow stream) and nested data structures (`ARRAY`, `STRUCT`, and `RANGE`) when `EnableTimestampPicos=true`. ### Key Changes - **Storage Read API Session Configuration**: Configured `TableReadOptions.arrowSerializationOptions` with `TIMESTAMP_PRECISION_PICOS` in `BigQueryStatement` when `EnableTimestampPicos` is active. - **Arrow Deserialization & Formatting**: - Wired `enableTimestampPicos` into `BigQueryArrowResultSet`, formatting picosecond timestamps directly to 12 fractional digits for column retrieval. - Normalized Arrow `Text` instances to `String` across hot paths to prevent downstream `BigQueryTypeRegistry` conversion failures. - Formatted `RANGE` and nested `RANGE` bounds using schema-derived element types and precision flags. - **Nested Types Propagation**: - Propagated precision flags to `BigQueryArrowArray` and `BigQueryArrowStruct`. - Overrode `getTargetClass()` in `BigQueryArrowArray` to return `String.class` when picoseconds are enabled, preventing `Array.newInstance` mismatch exceptions. - **Unit Tests**: - Added unit test suites verifying picosecond vs. microsecond behavior across `BigQueryArrowResultSet`, `BigQueryArrowArrayOfPrimitives`, `BigQueryArrowStruct`, and `BigQueryStatement`.
…types (googleapis#14334) b/503296051 This PR extends picosecond timestamp precision (`TIMESTAMP(12)`) support to the REST JSON query execution path and nested BigQuery data structures (`ARRAY`, `STRUCT`, `RANGE`). ### Key Changes 1. **Centralized Schema Precision Helper**: - Added `BigQueryTemporalUtility.isPicosecondTimestamp(Field)` as the single source of truth for scale `> 6` timestamp fields. - Delegated `BigQueryArrowResultSet.isPicosecondTimestamp` directly to this utility. 2. **REST JSON Result Set (`BigQueryJsonResultSet`)**: - Eagerly binds `enableTimestampPicos` from the parent `BigQueryStatement`. - Formats picosecond timestamp primitives into canonical 12-decimal UTC strings (`YYYY-MM-DD HH:MM:SS.SSSSSSSSSSSS`). - Added shared `formatRangeTimestamp(FieldValue)` helper for `RANGE<TIMESTAMP>` picosecond formatting, while preserving existing `BigQueryTypeRegistry` fallthrough for non-pico queries. - Flattened `getObject(int)` control flow using early returns and removed redundant defensive index logic. 3. **Nested Structures (`BigQueryJsonArray`, `BigQueryJsonStruct`)**: - Propagated `enableTimestampPicos` across constructors and nested factories. - Overrode `BigQueryJsonArray.getTargetClass()` to return `String.class` when picoseconds are enabled, preventing `Array.newInstance` type mismatch crashes. - Formatted nested picosecond timestamps and timestamp ranges in `getCoercedValue()` and `getValue()`. 4. **Testing & Verification**: - Added comprehensive test coverage across `BigQueryJsonResultSetTest`, `BigQueryJsonArrayOfPrimitivesTest`, and `BigQueryJsonStructTest`. - Verified backward compatibility when `enableTimestampPicos=false`.
Bumps versions of google-cloud-retail to 2.100.0-SNAPSHOT, google-cloud-java to 1.92.0-SNAPSHOT, google-cloud-bom to 0.270.0-SNAPSHOT, and libraries-bom to 26.89.0-SNAPSHOT after partial release (googleapis#14342).
…ry (googleapis#14360) Update `librarian.yaml` to place the v1beta1 schema protos under `excluded_protos` and regenerate java-aiplatform via Librarian, removing the unused schema proto files and updating reflect-config.json. For googleapis/librarian#5661
…pis#14362) Restores `google-cloud-spanner-jdbc` to `google-cloud-bom` and dependency convergence checks. It was erroneously removed in googleapis#13777 during a partial release. Fixes googleapis#14347
…ion check (googleapis#14356) The scheduled job is redundant as the presubmit check is required. Without this change, there is a bug in the workflow that also creates an issue when this check fails at presubmit (see googleapis#14298). Instead of fixing the immediate issue, we are deleting the cron job and simplify this workflow.
… ResumableUploadCallable (googleapis#14251) This is needed to meet the resumable upload requirement to allow custom headers and other settings to be applied on a per-call basis.
…ta` and `DatabaseMetaData` (googleapis#14358) b/556665374 This PR implements metadata reflection for picosecond timestamps across `ResultSetMetaData` and `DatabaseMetaData.getColumns()` when `EnableTimestampPicos` is active. ### Key Changes 1. **Result Set Metadata (`BigQueryResultSetMetadata`)**: - Reflects picosecond timestamp columns as `Types.VARCHAR`, type name `TIMESTAMP_PICOSECONDS`, `String.class`, display size `32`, precision `32`, and scale `12` when enabled. - Evaluates `enableTimestampPicos` once during initialization. - Aligned standard timestamp `getColumnDisplaySize` fallback from legacy `16` to spec-compliant `26` (`"YYYY-MM-DD HH:MM:SS.ffffff"`). 2. **Database Metadata (`BigQueryDatabaseMetaData`)**: - Updated `mapBigQueryTypeToJdbc` to return `ColumnTypeInfo(Types.VARCHAR, TIMESTAMP_PICOSECONDS, 32, 12, null)` for non-repeated picosecond timestamp fields when `connection.isEnableTimestampPicos()` is active. - Ensured repeated timestamp fields continue reflecting as `Types.ARRAY`. 3. **Centralized Constant (`BigQueryTemporalUtility`)**: - Defined `TIMESTAMP_PICOSECONDS_TYPE_NAME = "TIMESTAMP_PICOSECONDS"` to eliminate duplicate magic string literals across metadata components. 4. **Testing & Verification**: - Added tests in `BigQueryResultSetMetadataTest` and `BigQueryDatabaseMetaDataTest` verifying metadata reflection for enabled, disabled, repeated array, and standard timestamp columns.
…apis#14372) Align `com.google.cloud.spanner.connection.it.ITTransactionRetryTest` with the existing skip precedent in `ITRetryDmlAsPartitionedDmlTest `and `com.google.cloud.spanner.ITTransactionRetryTest` by skipping on Spanner Omni due to known issue b/441255724. Also correct the copy-pasted DirectPath bug reference in the other tests.
Updated googleapis commitish in librarian.yaml to googleapis/googleapis@efc9e8f 💡 **Note:** If this PR is still open when the daily update workflow runs next, it will be closed and replaced with a new PR containing the latest updates.
…eapis#13735) This PR implements publish hedging in the Java Cloud Pub/Sub Publisher. Specifically, it adds support for scheduling "hedged" publish attempts when a publish call is slow to respond. A token-based method is utilized to rate-limit hedged requests and a coordinator is used to manage/cancel concurrent requests to prevent duplicate publishes. --------- Co-authored-by: Tony Cui <[email protected]>
…apis#14365) Remove redundant min_java_version field from spanner's library configuration. The field is obsolete and defaults to 8 in librarian. For googleapis/librarian#6693
…ogleapis#14366) In .kokoro/build.sh, changed_file_list used a two-dot git diff (${BASE_SHA} ${HEAD_SHA}), which compares BASE_SHA directly to HEAD_SHA. When BASE_SHA moves ahead on main after the PR branch diverges, files merged into main on other modules were erroneously included in the diff. Switch to a three-dot git diff (${BASE_SHA}...${HEAD_SHA}) to diff against the merge base between BASE_SHA and HEAD_SHA, matching .kokoro/common.sh get_modified_files() behavior. Logs from lint job in googleapis#14354 which only touched a spanner file: ``` java-bigquery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/BigQueryArrowArray.java java-bigquery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/BigQueryArrowResultSet.java java-bigquery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/BigQueryArrowStruct.java java-bigquery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/BigQueryBaseArray.java java-bigquery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/BigQueryBaseStruct.java java-bigquery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/BigQueryStatement.java java-bigquery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/BigQueryTemporalUtility.java java-bigquery-jdbc/src/test/java/com/google/cloud/bigquery/jdbc/BigQueryArrowArrayOfPrimitivesTest.java java-bigquery-jdbc/src/test/java/com/google/cloud/bigquery/jdbc/BigQueryArrowResultSetTest.java java-bigquery-jdbc/src/test/java/com/google/cloud/bigquery/jdbc/BigQueryArrowStructTest.java java-bigquery-jdbc/src/test/java/com/google/cloud/bigquery/jdbc/BigQueryJdbcCustomLoggerTest.java java-bigquery-jdbc/src/test/java/com/google/cloud/bigquery/jdbc/BigQueryStatementTest.java java-bigquery-jdbc/src/test/java/com/google/cloud/bigquery/jdbc/BigQueryTemporalUtilityTest.java java-spanner/samples/snippets/src/main/java/com/example/spanner/QueueSample.java Matched: java-bigquery-jdbc/src/main/java/com/google/cloud/bigquery/jdbc/BigQueryArrowArray.java Changed Modules: java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc java-bigquery-jdbc Formatting only changed modules: java-bigquery-jdbc ```
…gleapis#14374) Remove obsolete min_java_version field from spanner's .repo-metadata.json, this was earlier missed in googleapis#14365. For googleapis/librarian#6693
Adds the ArrowDeserializer class which handles decoding serialized Arrow schemas and record batches into standard FieldValueList rows.
…tings (googleapis#14253) This allows users of resumable upload -powered methods to set an overall timeout on the entire operation
…load RPCs (googleapis#14317) Adds the `isResumableUpload()` property to the generator's `Method` model and wires pattern matching in `Parser.java` against an initially empty allowlist. The showcase test proto is also added with a hermetic test adapter to enable isolated unit testing across subsequent composers.
…#13642) Adds a Java code sample demonstrating hierarchical namespace recursive folder delete. Fixes: b/530058464
Update librarian version to v0.44.0. ## Debug Info ### System Details - JDK version: javac 17.0.20.1 - Maven version: �[1mApache Maven 3.9.12�[m - OS: Linux 6.18.14-1rodete4-amd64 x86_64 - Librarian version: v0.44.0 ### Commands Executed - `go run github.com/googleapis/librarian/cmd/librarian@latest update version` - `go run github.com/googleapis/librarian/cmd/[email protected] tidy` - `go run github.com/googleapis/librarian/cmd/[email protected] install` - `go run github.com/googleapis/librarian/cmd/[email protected] generate --all` - `git add .` - `git commit -m "chore: update librarian version to v0.44.0"`
…loop (googleapis#14370) The resume loop in ResumableStreamIterator restarted a broken stream indefinitely for any retryable error, ignoring the maxAttempts and totalTimeout configured in the retry settings for ExecuteStreamingSql. A streaming query could therefore retry forever when the server kept returning a retryable error, for example when a user configured DEADLINE_EXCEEDED as a retryable code and every attempt timed out. The loop now counts consecutive failed attempts and stops retrying, rethrowing the last exception, when the configured maxAttempts is reached. It also enforces the configured totalTimeout as a wall-clock budget for a sequence of consecutive failed attempts, measured from the first failure of the sequence: a retry is only allowed when the retry delay still fits in the remaining budget. This applies both to delays from the exponential backoff and to server-supplied retry delays (RetryInfo), which previously bypassed the backoff completely. A totalTimeout of zero means that no time budget has been set, in which case only maxAttempts limits the retries, mirroring GAX. Both limits only bind for custom retry settings: the default streaming retry settings do not set maxAttempts and keep the existing unbounded resume behavior. Progress on the stream resets both budgets, where progress means receiving a resume token that differs from the last seen token, so long-running streams that regularly make progress are not terminated by an occasional transient error, while a stream that keeps returning the same token cannot reset the budget indefinitely.
…oogleapis#14396) Wait for all backend replicas to establish active transport connections before issuing the warmup query in waitForReplicaRoutedRead. Previously, waitForReplicaRoutedRead exited as soon as the first replica handled the initial warmup read. In fast test executions, secondary replicas could still be completing their background Netty channel handshakes initiated by EndpointLifecycleManager probing. When the primary replica subsequently failed, secondary replicas were evaluated as unhealthy (channel state not yet READY) and skipped, causing traffic to unexpectedly fall back to defaultReplica. Tracking active transport connections via ServerTransportFilter in SharedBackendReplicaHarness ensures all replica channels are connected and ready before test assertions begin.
…ow result streaming (googleapis#13944) This PR introduces the `ArrowQueryResult` interface and `ArrowQueryResultImpl` class for streaming and paginating query results backed by Apache Arrow record batches. It provides zero-copy `VectorSchemaRoot` streaming and implements the standard `TableResult` iteration contract over Arrow batches.
Bumps `java-bigquery-jdbc` to version 1.4.1 and updates SNAPSHOT dependencies it references to their previously released versions: - `google-cloud-bigquery-jdbc`: bumped to `1.4.1` - `google-cloud-jar-parent`: `1.92.0-SNAPSHOT` -> `1.91.0` - `google-cloud-bigquery`: `2.72.0-SNAPSHOT` -> `2.71.0` - `google-cloud-bigquerystorage`: `3.34.0-SNAPSHOT` -> `3.33.0` - `google-cloud-logging`: `3.39.0-SNAPSHOT` -> `3.38.0` - `proto-google-cloud-bigquerystorage-v1`: `3.34.0-SNAPSHOT` -> `3.33.0` - `google-cloud-trace`: `2.98.0-SNAPSHOT` -> `2.97.0`
…s#14399) ## Summary - **ReadIT**: adds three single-row read tests covering: - `readSingleRowWithReadRow` — `readRow(tableId, rowKey)` API - `readSingleRowWithRowKeyQuery` — `readRows(Query.rowKey(key))` - `readSingleRowWithRowRangeQuery` — `readRows(Query.range(closed/closed same key))` - Each test writes a row and asserts the exact key, family, qualifier, timestamp, and value returned ## Test plan - [ ] All three `ReadIT` single-row tests passed locally (`Tests run: 3, Failures: 0, Errors: 0`)
…leapis#14407) Onboard google/cloud/networkservices/v1beta1 API (networkservices). Internal tracking bug http://b/561920907 ## Debug Info ### System Details - JDK version: javac 17.0.20.1 - Maven version: �[1mApache Maven 3.9.12�[m - OS: Linux 6.18.14-1rodete4-amd64 x86_64 - Librarian version: v0.44.0 ### Commands Executed - `MAVEN_SKIP_RC=true go run github.com/googleapis/librarian/cmd/[email protected] install` - `go run github.com/googleapis/librarian/cmd/[email protected] add google/cloud/networkservices/v1beta1` - `go run github.com/googleapis/librarian/cmd/[email protected] generate networkservices` - `mvn -B -ntp compile -pl :google-cloud-networkservices-parent -am` - `go run github.com/googleapis/librarian/cmd/[email protected] tidy` - `git add .` - `git commit -m "feat(google/cloud/networkservices/v1beta1): add networkservices"`
Onboard google/cloud/backupdr/v1beta API (backupdr). Internal tracking bug http://b/561460918 ## Debug Info ### System Details - Java version: openjdk version "17.0.20.1" 2026-08-18 - OS: Linux 6.18.14-1rodete4-amd64 x86_64 - Librarian version: v0.44.0 ### Commands Executed - `go run github.com/googleapis/librarian/cmd/[email protected] install` - `go run github.com/googleapis/librarian/cmd/[email protected] add google/cloud/backupdr/v1beta` - `go run github.com/googleapis/librarian/cmd/[email protected] generate -v backupdr` - `go run github.com/googleapis/librarian/cmd/[email protected] tidy` - `git add .` - `git commit -m "feat(google/cloud/backupdr/v1beta): add backupdr"`
…#14397) Re-applies the changes from googleapis#13642 (which was reverted in googleapis#14391 due to googleapis#14387) and includes librarian generation updates so that the librarian generate diff check succeeds. [Generated-by: AI]
googleapis#14401) Required to fix b/557150710
Bumps `java-bigquery-jdbc` and its dependencies back to SNAPSHOT versions after release 1.4.1: - `google-cloud-bigquery-jdbc`: `1.4.1` -> `1.5.0-SNAPSHOT` - `google-cloud-jar-parent`: `1.91.0` -> `1.92.0-SNAPSHOT` - `google-cloud-bigquery`: `2.71.0` -> `2.72.0-SNAPSHOT` - `google-cloud-bigquerystorage`: `3.33.0` -> `3.34.0-SNAPSHOT` - `google-cloud-logging`: `3.38.0` -> `3.39.0-SNAPSHOT` - `proto-google-cloud-bigquerystorage-v1`: `3.33.0` -> `3.34.0-SNAPSHOT` - `google-cloud-trace`: `2.97.0` -> `2.98.0-SNAPSHOT` - `versions.txt`: `google-cloud-bigquery-jdbc:1.4.1:1.5.0-SNAPSHOT`
…ters and batching (googleapis#14373) b/545231211 This PR adds picosecond timestamp support for `PreparedStatement` parameters and batch execution (`EnableTimestampPicos=1`), refactors statement pre-flight validation, and optimizes temporal string parsing. ### Changes * **`BigQueryParameterHandler`**: Supported `enableTimestampPicos` to preserve up to 12 fractional digits without premature microsecond truncation, cleaned up constructors into a `final`-field telescoping chain, and deleted the unused 2-arg `formatValueForQueryParameter` overload. * **`BigQueryPreparedStatement`**: Propagated `isEnableTimestampPicos` across statement initialization and batch parameter handling (`getStandardBatchJobConfiguration`). * **`BigQueryStatement`**: Converted `getJobConfig` into a pure configuration builder and extracted statement checks into `validateExecution()`. * **`BigQueryTemporalUtility`**: Consolidated fractional seconds truncation into a single helper (`truncateFractionalSeconds`) across all temporal parsers. * **Unit Tests**: Added coverage for high-precision parameter binding and batching, while deduplicating helper tests and removing dead mock stubs. ### Key Architectural Decisions #### 1. Legacy SQL Exception & Pre-Flight Validation (`BigQueryStatement`) * **Fail-Fast over Silent Data Loss**: Legacy SQL cannot represent 12-digit picoseconds. Silently falling back to Legacy SQL would drop 6 decimal places of precision without warning. To prevent silent data corruption and maintain parity with other drivers, we fail execution explicitly. * **Pre-Flight Placement**: This check previously lived inside `getJobConfig`. A configuration builder should not enforce fatal query-blocking logic. Moving it to `validateExecution()` alongside `checkClosed()` ensures we fail fast at statement execution entry—avoiding false "Executing query..." logs and unnecessary OpenTelemetry trace spans. * **Exception Classification**: We intentionally use `BigQueryJdbcException` (general `SQLException`) instead of `SQLSyntaxErrorException`. The query itself (e.g., `SELECT 1`) is syntactically valid; the failure is a driver/session configuration conflict, not a SQL grammar defect. #### 2. ASCII Character Scanning Optimization (`BigQueryTemporalUtility`) * In `truncateFractionalSeconds`, we replaced `Character.isDigit(c)` with a direct ASCII check (`c >= '0' && c <= '9'`) and cached `str.length()`. * Because ISO-8601 and JDBC timestamp literals are strictly ASCII, avoiding `Character.isDigit` eliminates repeated Unicode table lookups and branching in the fractional scanning loop, reducing CPU overhead on hot result-set parsing paths.
…leapis#14392) b/556664087 This PR adds ITs and and fixes one gap the ITs uncovered. ## Production fix `BigQueryConnection.getBigQueryConnection()` now sets `DataFormatOptions.TimestampFormatOptions.ISO8601_STRING` when `EnableTimestampPicos=1`. The REST JSON read path was silently truncating to microseconds. BigQuery serializes `TIMESTAMP` as FLOAT64 epoch seconds by default, which cannot represent sub-microsecond digits; `ISO8601_STRING` is the only output format that carries the full 12-digit fraction. The Arrow path was already correct, which is why unit tests and metadata assertions passed while REST values did not. Gated on the property so the wire format is unchanged for existing users. ## Tests New `ITJdbcTimestampPicosTest` (11 tests): Arrow and REST read paths, complex types (`ARRAY`/`STRUCT`), `ResultSetMetaData`, `DatabaseMetaData.getColumns`, `PreparedStatement` round trips, Legacy SQL rejection, and timezone invariance. Registered in the presubmit, nightly, and driver-agnostic suites. Six tests carry `@Tag("advanced")` where behavior is driver-specific; the remaining five were verified to pass against other drivers. ## Known limitation BigQuery truncates `TIMESTAMP`-typed query parameters to microseconds even against a `TIMESTAMP(12)` column (b/419328655), so `setTimestamp`/`setObject` cannot write picosecond values. `setString` works, since the value is coerced server side. Documented in `USER_GUIDE.md` and asserted in the tests so the expectations flip loudly when the backend fix lands.
…ing heap leak (googleapis#14417) - Add `StandaloneBenchmarkServer` and update `ReadBenchmark` (along with the Maven benchmark profile) to connect to an external gRPC mock server when `jmh.spanner.server.port` or `SPANNER_PORT` is specified. This allows running the mock server and JMH client in separate JVMs pinned to disjoint CPU cores. - Add `MockSpannerServiceImpl.setRecordRequests(boolean)` to bypass global synchronized (lock) contention and prevent unbounded retention of 20+ million request protobufs (~21.5 GB Old-Gen heap leak) during 20-minute benchmark runs. - Fix `BenchmarkValidator` so latency improvements (actual < baseline) do not fail validation, and include explicit units (us/op), allowed ceiling, percentage delta, and host SAR CPU telemetry (avg_busy, avg_steal, max_steal) in failure messages.
…ot streaming (googleapis#14402) This PR introduces the `queryArrow` API on the `BigQuery` client veneer. It enables applications to stream query results as Apache Arrow `VectorSchemaRoot` batches with zero memory copies, supporting fast-path query execution and automatic fallback to BigQuery Storage Read API sessions for larger queries.
…oogleapis#14403) This PR adds integration tests verifying the `queryArrow` client API against the live BigQuery service. It tests single-page and multi-page Arrow stream consumption, validating row counts and VectorSchemaRoot iteration over live queries.
…nation (googleapis#14404) This PR introduces `ArrowQueryPageFetcher` to handle pagination for queries executing with `QueryResultsFormat.ARROW`. ### Summary of Changes - Implements `ArrowQueryPageFetcher` implementing `NextPageFetcher<FieldValueList>`. - Connects to the default storage read stream to fetch subsequent row pages. - Leverages `ArrowDeserializer.loadArrowRows` to deserialize Arrow record batches into `FieldValueList` collections with offset and `maxResults` bounding. - Adds `ArrowQueryPageFetcherTest` covering single-page, multi-page, max-results, and serialization behaviors.
…oogleapis#14405) Enables Apache Arrow wire acceleration for the traditional `BigQuery.query()` API returning row-based `TableResult`. Part of the BigQuery Apache Arrow support stack. Based on googleapis#14404 (page fetcher).
… query() (googleapis#14409) This PR implements slow-path execution fallback for row-based queries requesting Arrow results format (`QueryResultsFormat.ARROW`). When queries cannot be evaluated via the fast query path (such as queries writing to destination tables or exceeding fast-path limits), BigQuery job execution is triggered and table results are streamed via the BigQuery Storage Read API in Arrow format. Follow-up PR stacked on top of googleapis#14405.
…w format (googleapis#14413) Add integration tests verifying row-based `query()` execution using `QueryResultsFormat.ARROW` for both single-page and multi-page results in `ITBigQueryTest`.
…tub (googleapis#14321) Modified and introduces composers to emit an internal, dedicated REST stub and settings for resumable upload-powered services
…client (googleapis#14324) Synchronizes GAPIC Showcase to version 0.44.0, imports the `resumable_upload.proto` schema, and generates the baseline unary client files. This establishes an unmodified reference baseline before enabling resumable upload code generation.
Promote last failure status to top level.
…pool (googleapis#14431) This PR fixes session replacement and pool sizing issues in the Jetstream session pool: - Preserve pre-failure state on `forceClose()` (`SessionImpl`): `forceClose()` previously set state directly to `WAIT_SERVER_CLOSE`, tricking `SessionPoolImpl` into treating abnormal closures (handshake timeouts, missed heartbeats) as graceful closes and skipping replacement. It now captures the originating state (`STARTING` or `READY`) and reports it to the pool listener. - Idempotent cleanup in `SessionList.onSessionClosed()`: Switched from checking `prevState` to inspecting handle state (`!afe.isPresent()` for starting count; `afeHandle.sessions.remove()` for ready count). This prevents counter leaks on unexpected closure sequences and avoids double-decrements. - Multi-session scale-up on non-hot paths (`SessionPoolImpl`): Updated `start()`, `onSessionClose()`, `onSessionGoAway()`, and retry scheduling to create sessions while `poolSizer.getScaleDelta() > 0` and budget allows, while intentionally keeping the request hot path (`PendingVRpc.start()`) to a single session creation to avoid caller latency overhead.
…tubs (googleapis#14322) Wires generated transport stubs (`GrpcServiceStub`, `HttpJsonServiceStub`) to delegate resumable upload methods to the internal HTTP upload stub, and excludes upload RPCs from the main stubs' method descriptors and callables.
…able on TPC (googleapis#14435) This PR fixes `ITJdbcTimestampPicosTest` failures in Kokoro ### Changes - **Time zone (`integration_test_continuous`, `standalone_it_continuous`)** Four assertions compared hardcoded UTC literals against `getString()` on plain `TIMESTAMP` columns, which the driver renders in the JVM default time zone. The tests only passed under a UTC JVM; the Kokoro VM runs `America/New_York`, giving the observed `12:34:56` vs `07:34:56` mismatch. Added an `atJvmZone()` helper that converts the UTC literal to the JVM zone, mirroring the existing pattern in `ITBigQueryJDBCTest.validateGetString`. `TIMESTAMP(12)` assertions are unchanged. Those values are returned as verbatim UTC strings and are already zone invariant. - **TPC (`tpc_integration_tests_continuous`)** TPC backends reject the `TIMESTAMP(12)` type parameter, so `@BeforeAll` failed and the whole class errored. Added a class-level `@Tag("disable_tpc")`
Generates the public client surface and base stub contracts for resumable upload RPCs, providing both an asynchronous `ResumableUploadCallable` and a synchronous `InputStream` method. Test composers are updated with placeholder `@Ignore`d tests to reflect that meaningful testing is only exercised via integration tests. This is due to insufficient mock capabilities, which may make sense to augment for resumable uploads in the future. Sample code generation for resumable uploads is disabled; this also probably makes sense to add in the future.
Address review comment by replacing the manual try-catch block with JUnit assertThrows(ExecutionException.class, ...).
The topLevelStatus variable in the MutateRowsException handler had incorrect 2-space continuation indentation; reformatted to 4-space per google-java-format as flagged by the Librarian CI check. Change-Id: I66b9be44c220581c16cc465a331d6fb1a0e2daba
mutianf
force-pushed
the
fix-mutaterows-pitfall-5-6
branch
from
September 21, 2026 18:36
6b64115 to
f2d5417
Compare
Missing entries complete the attempt future successfully via handleAttemptSuccess, not exceptionally. Inspect the returned MutateRowsAttemptResult directly instead of expecting an ExecutionException.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
17 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
partialOmissionMultiEntryTestinMutateRowsAttemptCallableTest.javainside thegoogle-cloud-javamonorepo (java-bigtable/google-cloud-bigtable/...), verifying that when a 3-entryMutateRowsRequest([0, 1, 2]) receives only[Entry(0, OK), Entry(2, OK)]on anOKstream,MutateRowsAttemptCallablesurfaces a non-retryableMutateRowsExceptionwithCode.INTERNALfor omitted entry1.mutianf/java-bigtable#1.