Proposal: bundled custom transfers RFC - #6306
Conversation
Add a proposal for extending the custom transfer protocol with a new 'bundles' queue processing mode, allowing custom transfer agents to process a bundle of objects as a single unit of work instead of one object at a time. The proposal introduces the custom transfer protocol version 2 with version negotiation on the initiation stage, an opt-in bundles mode driven by the new lfs.customtransfer.<name>.protocol, queueProcessingMode and bundleSize configuration options, and an optional retry mechanism for completion messages. The detailed protocol specification is provided in a companion document. Shipped documentation is intentionally left untouched until an implementation lands. This supersedes the earlier bulk/batch iterations discussed in PR 6119. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Hey, thanks so much for this new proposal! I confess I have not yet had time to do more than skim it quickly, but it certainly looks promising. We should perhaps also consider the ideas outlined in #6242 regarding another way of enhancing the custom transfer protocol. I hope that after the long-delayed v3.8.0 release is done, we'll have some time to review these together and see how they might be compatible, and where they might conflict. Thanks again for separating out the proposal into its own PR; that will make the next steps much easier! |
|
Hey @chrisd8088! Regarding #6242 - yep, i agree, we definetly should consider this proposal. One thing i want to outline specifically is error handling/retries which is explicitly described in #6306 and i have yet to examine it in #6242 closely. Based on my experience of using the current 3.7.1 version with custom transfer - the error handling is a pain point on day-to-day basis. On #6119 discussion you proposed to split different proposals (bundles delivery separate from error handling/retries behaviour) and now seems a good point to pivot the #6303 proposal to error handling approach only having #6242 in mind. It seems possible since the #6242 proposed pipelined mode can cover a lot the bundles has to offer not introducing a separate complicated procotol version. One idea on bundles vs pipelined mode though: bundles are explicitly defined for agent, the pipelined mode defines only the concurrent items via concurrenttransfers. It effectively means there is no well defined way to say "okay, transfer queue has no objects left to offer and agent should proceed with whats left, we will not hit concurrenttransfers number of objects again". The agent has to guess that via time interval waiting and if nothing comes in - flush the actual transfers. Everything above is absolutely discussable, just sharing my perspective. Best regards, Alexander Bogomolets. |
2 participants
Hey git lfs team, hope you're doing great!
This is the proposal-only follow-up to the discussion in #6119, as we agreed
there to develop an RFC first and move to the implementation once the design
is settled.
This PR contains two documents and no code changes:
docs/proposals/custom-transfers-bundling.md- the proposal itself:problem statement, concepts, configuration and compatibility considerations.
docs/proposals/custom-transfers-bundling-protocol.md- the companionspecification of the proposed custom transfer protocol version 2.
The proposal allows custom transfer agents to process a bundle of objects as
a single unit of work instead of a single object per transfer, which enables
transfer strategies like compressing a bundle into a single archive per
download/upload session. This addresses the performance limitations for
repositories with large numbers of small objects and for large well-compressible
objects.
Compared to the state discussed in #6119, the main changes are:
with the Batch API terms.
retryflag for completion messages is included.#6119 is left open as the draft implementation playground for now.
Best regards, Alexander Bogomolets.