chore(datasets): improve logging and retry logic for MPUs#387
Merged
cwetherill-ps merged 1 commit intomasterfrom Apr 18, 2022
Merged
Conversation
This adds some basic retry logic for multipart upload parts so we don't choke immediately if a single part fails for any reason. It also adds some decidedly jank logging to update the end user on the progress of large uploads. Unfortunately this can create some really noisy console logs, but the way we have Halo implemented makes it just annoying enough to do a cleaner job of things as to not be worthwhile. The important thing is that users should now get relevant, timely information as to how large uploads are progressing, or an informative error message if the upload fails to complete for any reason.
bbatha
approved these changes
Apr 18, 2022
bbatha
reviewed
Apr 18, 2022
| # better than a task sitting forever, never either | ||
| # completing or emitting an error message. | ||
| if len(parts) % 7 == 0: # About every 100MB | ||
| print( |
Contributor
There was a problem hiding this comment.
Wouldn't you want to use the logger?
cwetherill-ps
deleted the
cwetherill/nb-991-investigate-flakiness-in-cli-mulipart
branch
|
🎉 This PR is included in version 2.0.2 🎉 The release is available on GitHub release Your semantic-release bot 📦🚀 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds some basic retry logic for multipart upload parts so we don't
choke immediately if a single part fails for any reason. It also adds
some decidedly jank logging to update the end user on the progress of
large uploads. Unfortunately this can create some really noisy console
logs, but the way we have Halo implemented makes it just annoying enough
to do a cleaner job of things as to not be worthwhile. The important
thing is that users should now get relevant, timely information as to
how large uploads are progressing, or an informative error message if
the upload fails to complete for any reason.