Skip to content
This repository was archived by the owner on Mar 6, 2026. It is now read-only.

feat: add reference_file_schema_uri to LoadJobConfig, ExternalConfig - #1399

Merged
aribray merged 29 commits into
googleapis:mainfrom
aribray:aribray--federated-formats
Nov 14, 2022
Merged

feat: add reference_file_schema_uri to LoadJobConfig, ExternalConfig#1399
aribray merged 29 commits into
googleapis:mainfrom
aribray:aribray--federated-formats

Conversation

@aribray

@aribray aribray commented Nov 4, 2022 •
edited
Loading

Copy link
Copy Markdown
Contributor

Current behavior:

  • for load jobs from federated formats like AVRO, PARQUET, and ORC, BigQuery uses the schema of whichever file is lexicographically last.

Example:

source_uris = [
    "gs://{project}/{bucket_name}/c-file.avro", 
    "gs://{project}/{bucket_name}/b-file.avro",
    "gs://{project}/{bucket_name}/r-file.avro",
]

"gs://{project}/{bucket_name}/r-file.avro" is lexicographically last

New behavior:

  • The reference_file_schema_uri field allows users to specify the schema
  • The reference_file_schema_uri does not have to be a file from the source_uris list
  • To prevent data loss, the reference_file_schema_uri should be a superset of the schemas in the source_uris list

Googlers see 246809557

@product-auto-label product-auto-label Bot added size: m Pull request size is medium. api: bigquery Issues related to the googleapis/python-bigquery API. labels Nov 4, 2022
@aribray
aribray marked this pull request as ready for review November 4, 2022 15:56
@aribray
aribray requested review from a team and Neenu1995 November 4, 2022 15:56
@product-auto-label product-auto-label Bot added size: l Pull request size is large. and removed size: m Pull request size is medium. labels Nov 4, 2022
leahecole
leahecole approved these changes Nov 9, 2022
View reviewed changes

@leahecole leahecole left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah with the nits, I'm honestly torn. Use your best judgment - it's nbd if it's not changed.

Comment thread tests/system/test_client.py Outdated
Comment thread tests/system/test_client.py Outdated
@product-auto-label product-auto-label Bot added size: m Pull request size is medium. and removed size: l Pull request size is large. labels Nov 9, 2022
@product-auto-label product-auto-label Bot removed the size: m Pull request size is medium. label Nov 10, 2022
@aribray aribray added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 10, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 10, 2022
@aribray aribray added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 10, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 10, 2022
@aribray aribray added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 10, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 10, 2022
@aribray aribray added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 11, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 11, 2022
@aribray aribray added the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 13, 2022
@yoshi-kokoro yoshi-kokoro removed the kokoro:force-run Add this label to force Kokoro to re-run the tests. label Nov 13, 2022
@aribray
aribray merged commit 931285f into googleapis:main Nov 14, 2022
@aribray
aribray deleted the aribray--federated-formats branch November 14, 2022 22:26
@release-please release-please Bot mentioned this pull request Nov 14, 2022
Merged
This was referenced Nov 22, 2022
Closed
Closed
Closed
abdelmegahedgoogle pushed a commit to abdelmegahedgoogle/python-bigquery that referenced this pull request Apr 17, 2023
@aribray @abdelmegahedgoogle
feat: add reference_file_schema_uri to LoadJobConfig, ExternalConfig (
b4305b8 
googleapis#1399)

* feat: add 'reference_file_schema_uri' to LoadJobConfig and ExternalConfig
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

2 more reviewers
@leahecole leahecole leahecole approved these changes
@Neenu1995 Neenu1995 Neenu1995 approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

api: bigquery Issues related to the googleapis/python-bigquery API. size: l Pull request size is large.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@aribray @leahecole @Neenu1995 @yoshi-kokoro