Skip to content

Support parameter files using Zip64 - #144

Merged
jonatanklosko merged 5 commits into
elixir-nx:mainfrom
sitch:main
Jan 9, 2023
Merged

Support parameter files using Zip64#144
jonatanklosko merged 5 commits into
elixir-nx:mainfrom
sitch:main

Conversation

@sitch

@sitch sitch commented Dec 31, 2022

Copy link
Copy Markdown
Contributor

I added a test for the zip64 functionality, but due to requiring a >2GB binary fixture, I think including it in git LFS storage is a bit excessive. More than happy to remove these extra code bits.

@sitch

sitch commented Dec 31, 2022

Copy link
Copy Markdown
Contributor Author

There's also a bit of an annoying warning from unzip about invalid datetimes...

@jonatanklosko

Copy link
Copy Markdown
Member

Hey @sitch, thanks for the PR!

We definitely should fix the warning, in our case it will produce a bunch of logs (since the archive usually has a lot of files) and it will confuse users. I see it's specifically the case where modification time/date is zero, so maybe it would make sense to special case that here and just return the earliest date. Do you want to ask/patch upstream? Alternatively we can try a different package.

As for the test, I don't think it's worth using LFS. We should be good as long as existing tests pass and we know we can read any zip64 file :)

@sitch

sitch commented Jan 4, 2023

Copy link
Copy Markdown
Contributor Author

@jonatanklosko I opened an issue akash-akya/unzip#17 to deal with the superfluous warnings.

As for the test, do you want me to keep the existing code which only runs the test if a user manually runs python test/fixtures/pytorch/generate.py in their local repo? Or should I gut it?

@jonatanklosko

Copy link
Copy Markdown
Member

Thanks! I don't think we need the additional test in this case; as long as we use a library with zip64 support and have tests for regular zips, it should be all good :)

@sitch

sitch commented Jan 9, 2023

Copy link
Copy Markdown
Contributor Author

@jonatanklosko this PR is ready to merge.

unzip upgraded to v0.8.0, and zip64 test removed. Thanks to @akash-akya for the speedy patch!

@jonatanklosko jonatanklosko left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfect, thank you!

@jonatanklosko jonatanklosko changed the title Zip64 Fix Support parameter files using Zip64 Jan 9, 2023
@jonatanklosko
jonatanklosko merged commit c03ea13 into elixir-nx:main Jan 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants