Skip to content

Simplify setupext.download_or_cache. - #15430

Merged
QuLogic merged 1 commit into
matplotlib:masterfrom
anntzer:download_or_cache
Mar 7, 2020
Merged

QuLogic merged 1 commit into
matplotlib:masterfrom
anntzer:download_or_cache

Conversation

@anntzer

@anntzer anntzer commented Oct 16, 2019

Copy link
Copy Markdown
Contributor

Instead of repeatedly unwrapping and rewrapping the file contents in a
BytesIO that needs to be seek()ed, just pass around the bytes object
itself.

PR Summary

PR Checklist

  • Has Pytest style unit tests
  • Code is Flake 8 compliant
  • New features are documented, with examples if plot related
  • Documentation is sphinx and numpydoc compliant
  • Added an entry to doc/users/next_whats_new/ if major new feature (follow instructions in README.rst there)
  • Documented in doc/api/api_changes.rst if API changed in a backward-incompatible way

Comment thread setupext.py
with open(cache_dir / sha, "xb") as fout:
fout.write(data)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
with open(cache_dir / sha, "xb") as fout:
fout.write(data)
(cache_dir / sha).write_bytes(data)

would match reading, I believe.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but then this would not fail if a file already exists with the wrong hash (well, I dunno what we want to do in that case, which is mostly theoretical, but I guess "don't overwrite a file we didn't expect to be there" is a reasonable approach?)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is an unexpected file in the cache, that could cause trouble later when reading from the cache. I think it's a valid position to claim that under this hash there should be exactly data. We can gracefully fix that. But both ways are ok.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At worst what will happen is that the "wrong" file will stay stuck in the cache (we didn't create it, so we can't afford to overwrite it) and later builds will redownload the file and fail to cache it but will otherwise proceed correctly. I think this is the same behavior as before this patch?

Comment thread setupext.py
with open(cache_dir / sha, "xb") as fout:
fout.write(data)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If there is an unexpected file in the cache, that could cause trouble later when reading from the cache. I think it's a valid position to claim that under this hash there should be exactly data. We can gracefully fix that. But both ways are ok.

Instead of repeatedly unwrapping and rewrapping the file contents in a
BytesIO that needs to be seek()ed, just pass around the bytes object
itself.
@anntzer
anntzer force-pushed the download_or_cache branch from da0d435 to ee41349 Compare March 3, 2020 17:11
@anntzer

anntzer commented Mar 3, 2020

Copy link
Copy Markdown
Contributor Author

@QuLogic do you want to re-review? :)

@QuLogic
QuLogic merged commit fe34774 into matplotlib:master Mar 7, 2020
@QuLogic QuLogic added this to the v3.3.0 milestone Mar 7, 2020
@anntzer
anntzer deleted the download_or_cache branch March 7, 2020 13:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants