Skip to content

MSG_ZEROCOPY + non-temporal stores from AES-GCM#3007

Merged
kazuho merged 81 commits into
masterfrom
kazuho/msg_zerocopy
Jul 1, 2022
Merged

MSG_ZEROCOPY + non-temporal stores from AES-GCM#3007
kazuho merged 81 commits into
masterfrom
kazuho/msg_zerocopy

Conversation

@kazuho

@kazuho kazuho commented Apr 25, 2022 •
edited
Loading

Copy link
Copy Markdown
Member

For sending encrypted bytes to the socket with minimal overhead in terms of both memory bandwidth and CPU clock cycles, uses MSG_ZEROCOPY and non-temporal aes-gcm engine (h2o/picotls#384).

  • Do not mix buffers passed to kernel with MSG_ZEROCOPY and those that are not. At the moment, they share the same allocator: h2o_socket_ssl_buffer_allocator (done in 9bcf40f).
  • Reduce threshold for zerocopy from 16KB to 4KB or something. https://www.kernel.org/doc/html/v5.17/networking/msg_zerocopy.html recommends 10KB but we probably want to use a smaller number, especially when using non-temporal engine and tx-nocache-copy, in which case both the source and destination of the copy becomes main memory (done in 4cba163).
  • Consult if zerocopy is actually used, and report if non-use rate is high. It's too easy to screw up configuration. -> This to be done in a follow-up PR.

Builds on top of h2o/picotls#384, #3031.

kazuho added 18 commits April 23, 2022 18:43
@kazuho
Merge branch 'master' into kazuho/sendvec-knows-how-to-send
4de5400
@kazuho
Merge branch 'kazuho/sendvec-knows-how-to-send+master' into kazuho/ms…
bba8c40 
…g_zerocopy
@kazuho
Merge branch 'kazuho/recycle' into kazuho/msg_zerocopy
179027c
@kazuho
define zerocopy_buffers_t for retaining pointers to zero-copy buffers
203764e
@kazuho
it works, though we still leak at close
b1c6d4a
@kazuho
Merge branch 'kazuho/sendvec-knows-how-to-send' into kazuho/msg_zerocopy
472446c
@kazuho
ack for the first half can be received before sending the latter half
9a7f87e
@kazuho
it works
0220a8e
@kazuho
use recvmmsg to detect drain completion
5f9bc0f
@kazuho
Merge branch 'kazuho/sendvec-knows-how-to-send' into kazuho/msg_zerocopy
b5048aa
@kazuho
attempt to fix compile errors on older versions of linux
c9040bc
@kazuho
"ZEROCOPY" is confusing, use "MSG_ZEROCOPY"
ac963a6
@kazuho
switch to edge trigger when the socket is reset by peer while polled …
97eda7a 
…for neither read or write, as well as suppress mishandled event warning in such case
@kazuho
oops, it is the opposite (amends prev. commit)
56bdec8
@kazuho
dump more information
8361471
@kazuho
reintroduce H2O_SOCKET_FLAG__EPOLL_IS_REGISTERED, which saves us one …
acb02fa 
…syscall in the typical case (where we start reading immediately after accepting a socket)
@kazuho
Merge branch 'kazuho/skip-picotls-per-sni-setup-when-tls12' into kazu…
a180026 
…ho/msg_zerocopy
@kazuho
revert introduction of zerocopy flag directly below listen (b1c6d4a)
5d13146
@kazuho kazuho mentioned this pull request Apr 27, 2022
Merged
@kazuho kazuho changed the title MSG_ZEROCOPY MSG_ZEROCOPY + non-temporal stores from AES-GCM May 11, 2022
kazuho added 10 commits May 26, 2022 17:04
@kazuho
do not rewrite cipher suites used by QUIC
0f31a0d
@kazuho
remove debug printf
3e2936e
@kazuho
Clear zerocopy_owned when a buffer with data still to be sent is re…
3a1a72a 
…leased from zerocopy. Otherwise we leak if the data remaining to be sent is below 16KB, because in such case, zerocopy is not used and hence there would be no reclamation by notification, but as `zerocopy_onwed` is set the buffer is not freed immediately.
@kazuho
reduce threshold
42ba163
@kazuho
tentative fix for crashes due to ntaes engine overwriting a cache lin…
6092129 
…e shared with other threads
@kazuho
extract https://github.com/h2o/picotls @ f9c85cb () at deps/picotls
d221923
@kazuho
Revert "tentative fix for crashes due to ntaes engine overwriting a c…
a6f8d04 
…ache line shared with other threads"

This reverts commit 6092129.
@kazuho
alignment and max_cnt as config parameters of recycler
a23ce03
@kazuho
set alignment of tls buffers to 64B
89d34cf
@kazuho
proper aligned allocation of ssl buffers
ef8ffb7
@kazuho kazuho mentioned this pull request Jun 8, 2022
Merged
kazuho and others added 17 commits June 8, 2022 12:10
@kazuho
Merge branch 'master' into kazuho/msg_zerocopy
8619d33
@kazuho
Merge branch 'kazuho/mem-recycle-autotune' into kazuho/msg_zerocopy
fbf000e
@kazuho
split recyclers
9bcf40f
@kazuho
extract https://github.com/h2o/picotls @ 9dc6982 () at deps/picotls
39d3f91
@kazuho
use zerocopy only when the crypto uses non-temporal stores; otherwise…
dc05350 
… it's faster to use tx-nocache-copy
@deweerdt @kazuho
Use shutdown(RDWR) instead of connect(AF_UNSET) in order to end the T…
13c13e2 
…CP connection while still being able to receive completion messages on the error queue
@deweerdt @kazuho
Don't handle HUP if the socket has been disposed and we're in zerocop…
2ab89a5 
…y mode
@kazuho
Merge branch 'kazuho/mem-recycle-autotune' into kazuho/msg_zerocopy
d57fa75
@kazuho
follow the new design
a03a891
@kazuho
Merge branch 'kazuho/mem-recycle-autotune' into kazuho/msg_zerocopy
c25f5dc
@kazuho
Merge branch 'kazuho/mem-recycle-autotune' into kazuho/msg_zerocopy
fecdd95
@kazuho
initialize sock->_zerocopy based on if a non-temporal engine is used
ad7ebaf
@kazuho
merge two knobs changing one thing
446508a
@kazuho
Merge branch 'master' into kazuho/msg_zerocopy
046b79c
@kazuho
extract https://github.com/h2o/picotls @ 57d4ec5 () at deps/picotls
34be2bf
@kazuho
determine the offload modes to be used
9262f2d
@kazuho
update tests folllowing config knob merge
fd1bc4f
@kazuho
kazuho marked this pull request as ready for review July 1, 2022 05:56
@kazuho
kazuho merged commit 16d0c19 into master Jul 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@kazuho @deweerdt