1f26622b79
Kuniyuki Iwashima says:
====================
The SO_REUSEPORT option allows sockets to listen on the same port and to
accept connections evenly. However, there is a defect in the current
implementation [1]. When a SYN packet is received, the connection is tied
to a listening socket. Accordingly, when the listener is closed, in-flight
requests during the three-way handshake and child sockets in the accept
queue are dropped even if other listeners on the same port could accept
such connections.
This situation can happen when various server management tools restart
server (such as nginx) processes. For instance, when we change nginx
configurations and restart it, it spins up new workers that respect the
new configuration and closes all listeners on the old workers, resulting
in the in-flight ACK of 3WHS is responded by RST.
To avoid such a situation, users have to know deeply how the kernel handles
SYN packets and implement connection draining by eBPF [2]:
1. Stop routing SYN packets to the listener by eBPF.
2. Wait for all timers to expire to complete requests
3. Accept connections until EAGAIN, then close the listener.
or
1. Start counting SYN packets and accept syscalls using the eBPF map.
2. Stop routing SYN packets.
3. Accept connections up to the count, then close the listener.
In either way, we cannot close a listener immediately. However, ideally,
the application need not drain the not yet accepted sockets because 3WHS
and tying a connection to a listener are just the kernel behaviour. The
root cause is within the kernel, so the issue should be addressed in kernel
space and should not be visible to user space. This patchset fixes it so
that users need not take care of kernel implementation and connection
draining. With this patchset, the kernel redistributes requests and
connections from a listener to the others in the same reuseport group
at/after close or shutdown syscalls.
Although some software does connection draining, there are still merits in
migration. For some security reasons, such as replacing TLS certificates,
we may want to apply new settings as soon as possible and/or we may not be
able to wait for connection draining. The sockets in the accept queue have
not started application sessions yet. So, if we do not drain such sockets,
they can be handled by the newer listeners and could have a longer
lifetime. It is difficult to drain all connections in every case, but we
can decrease such aborted connections by migration. In that sense,
migration is always better than draining.
Moreover, auto-migration simplifies user space logic and also works well in
a case where we cannot modify and build a server program to implement the
workaround.
Note that the source and destination listeners MUST have the same settings
at the socket API level; otherwise, applications may face inconsistency and
cause errors. In such a case, we have to use the eBPF program to select a
specific listener or to cancel migration.
Special thanks to Martin KaFai Lau for bouncing ideas and exchanging code
snippets along the way.
Link:
[1] The SO_REUSEPORT socket option
https://lwn.net/Articles/542629/
[2] Re: [PATCH 1/1] net: Add SO_REUSEPORT_LISTEN_OFF socket option as drain mode
https://lore.kernel.org/netdev/1458828813.10868.65.camel@edumazet-glaptop3.roam.corp.google.com/
Changelog:
v8:
* Make reuse const in reuseport_sock_index()
* Don't use __reuseport_add_sock() in reuseport_alloc()
* Change the arg of the second memcpy() in reuseport_grow()
* Fix coding style to use goto in reuseport_alloc()
* Keep sk_refcnt uninitialized in inet_reqsk_clone()
* Initialize ireq_opt and ipv6_opt separately in reqsk_migrate_reset()
[ This series does not include a stats patch suggested by Yuchung Cheng
not to drop Acked-by/Reviewed-by tags and save reviewer's time. I will
post the patch as a follow up after this series is merged. ]
v7:
https://lore.kernel.org/bpf/20210521182104.18273-1-kuniyu@amazon.co.jp/
* Prevent attaching/detaching a bpf prog via shutdowned socket
* Fix typo in commit messages
* Split selftest into subtests
v6:
https://lore.kernel.org/bpf/20210517002258.75019-1-kuniyu@amazon.co.jp/
* Change description in ip-sysctl.rst
* Test IPPROTO_TCP before reading tfo_listener
* Move reqsk_clone() to inet_connection_sock.c and rename to
inet_reqsk_clone()
* Pass req->rsk_listener to inet_csk_reqsk_queue_drop() and
reqsk_queue_removed() in the migration path of receiving ACK
* s/ARG_PTR_TO_SOCKET/PTR_TO_SOCKET/ in sk_reuseport_is_valid_access()
* In selftest, use atomic ops to increment global vars, drop ACK by XDP,
enable force fastopen, use "skel->bss" instead of "skel->data"
v5:
https://lore.kernel.org/bpf/20210510034433.52818-1-kuniyu@amazon.co.jp/
* Move initializtion of sk_node from 6th to 5th patch
* Initialize sk_refcnt in reqsk_clone()
* Modify some definitions in reqsk_timer_handler()
* Validate in which path/state migration happens in selftest
v4:
https://lore.kernel.org/bpf/20210427034623.46528-1-kuniyu@amazon.co.jp/
* Make some functions and variables 'static' in selftest
* Remove 'scalability' from the cover letter
v3:
https://lore.kernel.org/bpf/20210420154140.80034-1-kuniyu@amazon.co.jp/
* Add sysctl back for reuseport_grow()
* Add helper functions to manage socks[]
* Separate migration related logic into functions: reuseport_resurrect(),
reuseport_stop_listen_sock(), reuseport_migrate_sock()
* Clone request_sock to be migrated
* Migrate request one by one
* Pass child socket to eBPF prog
v2:
https://lore.kernel.org/netdev/20201207132456.65472-1-kuniyu@amazon.co.jp/
* Do not save closed sockets in socks[]
* Revert
|
||
---|---|---|
Documentation | ||
LICENSES | ||
arch | ||
block | ||
certs | ||
crypto | ||
drivers | ||
fs | ||
include | ||
init | ||
ipc | ||
kernel | ||
lib | ||
mm | ||
net | ||
samples | ||
scripts | ||
security | ||
sound | ||
tools | ||
usr | ||
virt | ||
.clang-format | ||
.cocciconfig | ||
.get_maintainer.ignore | ||
.gitattributes | ||
.gitignore | ||
.mailmap | ||
COPYING | ||
CREDITS | ||
Kbuild | ||
Kconfig | ||
MAINTAINERS | ||
Makefile | ||
README |
README
Linux kernel ============ There are several guides for kernel developers and users. These guides can be rendered in a number of formats, like HTML and PDF. Please read Documentation/admin-guide/README.rst first. In order to build the documentation, use ``make htmldocs`` or ``make pdfdocs``. The formatted documentation can also be read online at: https://www.kernel.org/doc/html/latest/ There are various text files in the Documentation/ subdirectory, several of them using the Restructured Text markup notation. Please read the Documentation/process/changes.rst file, as it contains the requirements for building and running the kernel, and information about the problems which may result by upgrading your kernel.