Version 3.3.1 has broken blob.upload_from_file(file_obj) with error: Provided CRC32C xMvOLw== doesn't match calculated CRC32C ObqI+Q==

python version 3.10
gcp-cloud run application

Version 3.3.1 of google-cloud-storage

blob.upload_from_file(file_obj)

error thrown:

"Provided CRC32C xMvOLw== doesn't match calculated CRC32C ObqI+Q=="

workaround: revert to Version 3.3.0 or earlier
e.g
"google-cloud-storage>=3.0.0,<3.3.1",

also blob.upload_from_filename() has the same problem

@dougEdwards

Hi @dougEdwards , in version 3.3.1. support was added to include CRC32C checksum of an object. i.e. users can now provide crc32c checksum (calculated by them) while initializing the blob.

This checksum will be matched by the checksum calculated by server, if they don't match, server will send an error. Which is what is happening.

Sample to send checksum of the object while initializing

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)
    blob = bucket.blob(
        destination_blob_name, crc32c_checksum="your_calculated_checksum"
    )

    # Optional: set a generation-match precondition to avoid potential race conditions
    # and data corruptions. The request to upload is aborted if the object's
    # generation number does not match your precondition. For a destination
    # object that does not yet exist, set the if_generation_match precondition to 0.
    # If the destination object already exists in your bucket, set instead a
    # generation-match precondition using its generation number.
    generation_match_precondition = 0

    blob.upload_from_filename(
        source_file_name, if_generation_match=generation_match_precondition
    )

    print(f"File {source_file_name} uploaded to {destination_blob_name}.")

Could you also provide us with a sample to replicated the behaviour you're observing ?

@chandra-siri

@chandra-siri

I'm encountering the same/similar issue via django-storages

The pattern that reproduces the error appears to be:

Create an empty file (probably doesn't have to be empty)
blob.reload() is called, which populates the crc32c checksum property
Attempt to overwrite the file

Here's a reproducible example that succeeds in v3.3.0 but fails in v3.3.1

https://gist.github.com/chrisaiello/6c393cbbb96bb93db9e22ac24c1c391a

Details


Using blob: gs://my_bucket/bug-repro-0d15ff63a6384cecab37a52bf30a06a6.dat
Step 1: Uploading an empty file...
Step 1: Success.

Step 2: Reloading blob metadata from server...
Step 2: Success. Local blob object now has crc32c = 'AAAAAA=='

Step 3: Attempting to overwrite the blob with new data (9437184 bytes)...

--- ERROR REPRODUCED ---
Caught expected google.api_core.exceptions.BadRequest
Status Code: 400
Error Message: PUT https://storage.googleapis.com/upload/storage/v1/b/my_bucket/o?uploadType=resumable&upload_id=ABgVH8--9DCW8DHjWO8XTI7WzQ4fnJfXenfkdZqdh6RlCoocrpjb9VtjSuxESRYEdSML-OKa3xKf6iKCSH4ifDtYjBcI8OgweuTSfyDrdDnG7c5I: {
  "error": {
    "code": 400,
    "message": "Provided CRC32C \"AAAAAA==\" doesn't match calculated CRC32C \"Gs/MBw==\".",
    "errors": [
      {
        "message": "Provided CRC32C \"AAAAAA==\" doesn't match calculated CRC32C \"Gs/MBw==\".",
        "domain": "global",
        "reason": "invalid"
      }
    ]
  }
}
: ('Request failed with status code', 400, 'Expected one of', <HTTPStatus.OK: 200>, <HTTPStatus.PERMANENT_REDIRECT: 308>)
-----------------------------------

Yep me too! Since updating from version 3.3.0 to 3.4.0, I'm also seeing lots of these errors in production:

400 PUT https://storage.googleapis.com/upload/storage/v1/b/f-reports/o?uploadType=resumable&upload_id={{REDACTED}: {
  "error": {
    "code": 400,
    "message": "Provided CRC32C \"AAAAAA==\" doesn't match calculated CRC32C \"2HvlZg==\".",
    "errors": [
      {
        "message": "Provided CRC32C \"AAAAAA==\" doesn't match calculated CRC32C \"2HvlZg==\".",
        "domain": "global",
        "reason": "invalid"
      }
    ]
  }

  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/storage/blob.py", line 3130, in upload_from_string
    self.upload_from_file(
    ^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/storage/blob.py", line 2864, in upload_from_file
    self._prep_and_do_upload(
    ^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/storage/blob.py", line 2735, in _prep_and_do_upload
    _raise_from_invalid_response(exc)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/layers/google.python.pip/pip/lib/python3.12/site-packages/google/cloud/storage/blob.py", line 4925, in _raise_from_invalid_response
    raise exceptions.from_http_status(response.status_code, message, response=response)
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

@chandra-siri

@chandra-siri Do you have an update on a fix for the regression caused in version 3.3.1?

@chrisaiello

Hi all,

We'll soon be fixing this issue, you should see the fix rolled out in < 2 weeks.

Thanks @chrisaiello for providing with the repro script, & thanks @dougEdwards for reporting the bug.

fix: send entire object checksums via upload methods (#1561)

@jacobg

Hi @jacobg , @chrisaiello and @dougEdwards

This issue is now fixed. Please upgrade to the latest version v3.4.1

Navigation Menu

Repository navigation

Version 3.3.1 has broken blob.upload_from_file(file_obj) with error: Provided CRC32C xMvOLw== doesn't match calculated CRC32C ObqI+Q== #1541

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Version 3.3.1 has broken blob.upload_from_file(file_obj) with error: Provided CRC32C xMvOLw== doesn't match calculated CRC32C ObqI+Q== #1541

Description

Activity

chandra-siri commented on Sep 8, 2025

chrisaiello commented on Sep 8, 2025

jacobg commented on Sep 16, 2025

jacobg commented on Sep 16, 2025

chandra-siri commented on Sep 22, 2025

chandra-siri commented on Oct 8, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions