Skip to content

restore-file - Support binary files#9120

Open
SunsetTechuila wants to merge 6 commits intomainfrom
restore-bin
Open

restore-file - Support binary files#9120
SunsetTechuila wants to merge 6 commits intomainfrom
restore-bin

Conversation

@SunsetTechuila
Copy link
Copy Markdown
Member

@SunsetTechuila SunsetTechuila commented Mar 25, 2026

@github-actions github-actions bot added the bug label Mar 25, 2026
@fregante
Copy link
Copy Markdown
Member

Tests:

  • are large files still supported (restore a large file)
  • are empty files replaced with "\n" (restore a 0-byte file)

@SunsetTechuila
Copy link
Copy Markdown
Member Author

SunsetTechuila commented Mar 25, 2026

are large files still supported

Meh

  • If the requested file's size is:
    • 1 MB or smaller: All features of this endpoint are supported.
    • Between 1-100 MB: Only the raw or object custom media types are supported. Both will work as normal, except that when using the object media type, the content field will be an empty string and the encoding field will be "none". To get the contents of these larger files, use the raw media type.
    • Greater than 100 MB: This endpoint is not supported.

The API itself returns a valid string. The issue must be in how we're processing or storing it

image

@SunsetTechuila
Copy link
Copy Markdown
Member Author

@copilot take a look. Don't commit any changes, just explain what could be the issue and how it can be fixed

Copy link
Copy Markdown
Contributor

Copilot AI commented Mar 25, 2026

@copilot take a look. Don't commit any changes, just explain what could be the issue and how it can be fixed

Root cause: api.v3() always reads responses as text

In api.tsx line 118: const textContent = await response.text(). This means there's no binary-safe path. The original code used raw media type + json: false, but response.text() decodes binary bytes as UTF-8, corrupting non-UTF-8 binary content. That's the original issue this PR was trying to fix.

How to fix:

The object approach works for files ≤1MB (returns base64 directly), but fails for larger files. The proper fix is to add binary response support to api.v3() so we can fetch raw content as an ArrayBuffer and convert to base64 correctly:

  1. Add a binary: true option to api.v3() that uses response.arrayBuffer() instead of response.text()
  2. In getFile(), use Accept: application/vnd.github.raw with binary: true
  3. Convert the result: uint8ArrayToBase64(new Uint8Array(arrayBuffer)) from uint8array-extras

This would handle both binary files and large files (up to 100MB) correctly, since the raw binary data never goes through text decoding.

(Note: the empty/0-byte file issue with !file is pre-existing — the original code had the same problem since stringToBase64("") returns "" which is also falsy.)

@SunsetTechuila
Copy link
Copy Markdown
Member Author

  • Add a binary: true option to api.v3() that uses response.arrayBuffer() instead of response.text()

@fregante do we need that option? I think base64 would be more useful

@fregante
Copy link
Copy Markdown
Member

Limiting the whole feature to <1MB is definitely a downgrade, so that's out of the question. A common file I restore is package-lock.json, which is too often very large (monorepos and such)

As for the binary option, it would be ok as long as we don't have to manually toggle between two fetching modes in this feature. I think that would introduce many points of failure. What do you think?

@SunsetTechuila
Copy link
Copy Markdown
Member Author

26.3.21 can't discard new files and changes to 0-byte files

@SunsetTechuila
Copy link
Copy Markdown
Member Author

26.3.21 can't discard new files and changes to 0-byte files

Worth a separate PR

@SunsetTechuila SunsetTechuila marked this pull request as ready for review March 27, 2026 10:00
Comment on lines +34 to +37
const {content} = await api.v3(
`contents/${filePath}?ref=${ref}`,
{
json: false,
base64: true,
Copy link
Copy Markdown
Member Author

@SunsetTechuila SunsetTechuila Mar 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This was the only feature that used textContent. Not the only one that used json: false, though

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

restore-file: work for binary files, too

3 participants