Skip to content

restore-file - Support binary files#9120

Merged
SunsetTechuila merged 11 commits into
mainfrom
restore-bin
Apr 3, 2026
Merged

restore-file - Support binary files#9120
SunsetTechuila merged 11 commits into
mainfrom
restore-bin

Conversation

@SunsetTechuila
Copy link
Copy Markdown
Member

@SunsetTechuila SunsetTechuila commented Mar 25, 2026

@github-actions github-actions Bot added the bug label Mar 25, 2026
Comment thread source/features/restore-file.tsx Outdated
@fregante
Copy link
Copy Markdown
Member

Tests:

  • are large files still supported (restore a large file)
  • are empty files replaced with "\n" (restore a 0-byte file)

@SunsetTechuila
Copy link
Copy Markdown
Member Author

SunsetTechuila commented Mar 25, 2026

are large files still supported

Meh

  • If the requested file's size is:
    • 1 MB or smaller: All features of this endpoint are supported.
    • Between 1-100 MB: Only the raw or object custom media types are supported. Both will work as normal, except that when using the object media type, the content field will be an empty string and the encoding field will be "none". To get the contents of these larger files, use the raw media type.
    • Greater than 100 MB: This endpoint is not supported.

The API itself returns a valid string. The issue must be in how we're processing or storing it

image

@SunsetTechuila
Copy link
Copy Markdown
Member Author

@copilot take a look. Don't commit any changes, just explain what could be the issue and how it can be fixed

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 25, 2026

@copilot take a look. Don't commit any changes, just explain what could be the issue and how it can be fixed

Root cause: api.v3() always reads responses as text

In api.tsx line 118: const textContent = await response.text(). This means there's no binary-safe path. The original code used raw media type + json: false, but response.text() decodes binary bytes as UTF-8, corrupting non-UTF-8 binary content. That's the original issue this PR was trying to fix.

How to fix:

The object approach works for files ≤1MB (returns base64 directly), but fails for larger files. The proper fix is to add binary response support to api.v3() so we can fetch raw content as an ArrayBuffer and convert to base64 correctly:

  1. Add a binary: true option to api.v3() that uses response.arrayBuffer() instead of response.text()
  2. In getFile(), use Accept: application/vnd.github.raw with binary: true
  3. Convert the result: uint8ArrayToBase64(new Uint8Array(arrayBuffer)) from uint8array-extras

This would handle both binary files and large files (up to 100MB) correctly, since the raw binary data never goes through text decoding.

(Note: the empty/0-byte file issue with !file is pre-existing — the original code had the same problem since stringToBase64("") returns "" which is also falsy.)

@SunsetTechuila
Copy link
Copy Markdown
Member Author

  • Add a binary: true option to api.v3() that uses response.arrayBuffer() instead of response.text()

@fregante do we need that option? I think base64 would be more useful

@fregante
Copy link
Copy Markdown
Member

Limiting the whole feature to <1MB is definitely a downgrade, so that's out of the question. A common file I restore is package-lock.json, which is too often very large (monorepos and such)

As for the binary option, it would be ok as long as we don't have to manually toggle between two fetching modes in this feature. I think that would introduce many points of failure. What do you think?

@SunsetTechuila
Copy link
Copy Markdown
Member Author

SunsetTechuila commented Mar 27, 2026

26.3.21 can't discard new files (it throws a 404 error) and changes to 0-byte files (it deletes such files)

@SunsetTechuila

This comment was marked as resolved.

@SunsetTechuila SunsetTechuila marked this pull request as ready for review March 27, 2026 10:00
Comment thread source/features/restore-file.tsx Outdated
@fregante
Copy link
Copy Markdown
Member

fregante commented Apr 2, 2026

Same questions here as the other PR, they're actually more relevant here.

@SunsetTechuila
Copy link
Copy Markdown
Member Author

What's the status of this PR

Ready to be merged

What are the limitations if any?

Other than 100mb file size limit - none

Is the PR title up to date?

Yes

Comment thread source/github-helpers/api.tsx Outdated
if (base64) {
const arrayBuffer = await response.arrayBuffer();
const content = uint8ArrayToBase64(new Uint8Array(arrayBuffer));
apiResponse = {content};
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Base64 is not a useful output for our helper, I think it's best to return the lower-level type instead, Promise<ArrayBuffer>?

Base64 is only useful here because the POST API later requires this format.

Since we're here, are we sure there isn't an even better way to POST the bytes? Would be fantastic if we could pass the raw Response stream straight to the POST api.

Copy link
Copy Markdown
Member Author

@SunsetTechuila SunsetTechuila Apr 2, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Promise<ArrayBuffer>

And what we're gonna do with it?

are we sure there isn't an even better way to POST the bytes?

I'm sure:

https://docs.github.com/en/graphql/reference/input-objects#filechanges

https://docs.github.com/en/rest/git/blobs?apiVersion=2026-03-10#create-a-blob

@fregante
Copy link
Copy Markdown
Member

fregante commented Apr 2, 2026

It's ok if you merge this and possibly revisit the helper later. That part might require larger codebase changes.

@SunsetTechuila

This comment was marked as resolved.

@SunsetTechuila SunsetTechuila reopened this Apr 2, 2026
@SunsetTechuila SunsetTechuila merged commit 5c41c29 into main Apr 3, 2026
9 checks passed
@SunsetTechuila SunsetTechuila deleted the restore-bin branch April 3, 2026 10:35
@cooljeanius
Copy link
Copy Markdown

Thanks for this! I look forward to using it in a tagged release!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

restore-file: work for binary files, too

4 participants