tl;dr I am using quiche with curl to POST a long data transfer (1MB) in an experimental setting. I noticed poor performance when limiting packet queue size based on the number of packets, and was looking into why. I think it's because curl is sending many excessive STREAM_DATA_BLOCKED frames, bloating the queue.

Problem: In Wireshark, I observed that the client initially sent some ~MTU-sized packets of ~1200 bytes, followed by many ~45 byte packets. Logging the frames sent in one run showed that of 18164 total frames, 7.8% were STREAM frames, 91.6% STREAM_DATA_BLOCKED frames, and the remainder were ACK, CRYPTO, and PADDING frames. The STREAM_DATA_BLOCKED frames were sent in their own packets, aka not embedded in a STREAM packet like ACK frames were, and were bloating the queue.

I am not super familiar with the details of the QUIC RFC, but 19.13 only says "the sender should send STREAM_DATA_BLOCKED when it wishes to send data but is unable to due to stream-level flow control." The initial send and receive buffer size is (1 * 1024 * 1024) which is 1MB, so that seems fine. I find that the stream capacity is typically limited not by this initial buffer size, but by this variable cwin_available that is based on the cwnd set in quiche's recovery code.

Proposal: I don't see what the peer can do with this frame in a long data transfer within these reasonable parameters, so I made this fix on a fork (ygina@fcc4c3f) that just comments out the check in send_body(). And the RFC only says should and not must. This fix reduced the median time over 10 trials on a 152ms RTT link from 2.6s to 2.1s, and reduced the number of frames from 18164 to 1533.

Questions: I'd like to check if my understanding of the STREAM_DATA_BLOCKED frame is correct. Would this fix be useful in other settings? Is this an issue with the curl code calling send_body() with too large a body and not quiche itself? Can we still send STREAM_DATA_BLOCKED frames but fewer when the stream's capacity is limited, such as by piggybacking this frame on STREAM packets?