Maciej Fijalkowski 68ab769033 xsk: Fix backpressure mechanism on Tx ... [ Upstream commit c00c446168 ] Commit d678cbd2f8 ("xsk: Fix handling of invalid descriptors in XSK TX batching API") fixed batch API usage against set of descriptors with invalid ones but introduced a problem when AF_XDP SW rings are smaller than HW ones. Mismatch of reported Tx'ed frames between HW generator and user space app was observed. It turned out that backpressure mechanism became a bottleneck when the amount of produced descriptors to CQ is lower than what we grabbed from XSK Tx ring. Say that 512 entries had been taken from XSK Tx ring but we had only 490 free entries in CQ. Then callsite (ZC driver) will produce only 490 entries onto HW Tx ring but 512 entries will be released from Tx ring and this is what will be seen by the user space. In order to fix this case, mix XSK Tx/CQ ring interractions by moving around internal functions and changing call order: * pull out xskq_prod_nb_free() from xskq_prod_reserve_addr_batch() up to xsk_tx_peek_release_desc_batch(); ** move xskq_cons_release_n() into xskq_cons_read_desc_batch() After doing so, algorithm can be described as follows: 1. lookup Tx entries 2. use value from 1. to reserve space in CQ (*) 3. Read from Tx ring as much descriptors as value from 2 3a. release descriptors from XSK Tx ring (**) 4. Finally produce addresses to CQ Fixes: d678cbd2f8 ("xsk: Fix handling of invalid descriptors in XSK TX batching API") Signed-off-by: Magnus Karlsson <magnus.karlsson@intel.com> Signed-off-by: Maciej Fijalkowski <maciej.fijalkowski@intel.com> Signed-off-by: Daniel Borkmann <daniel@iogearbox.net> Link: https://lore.kernel.org/bpf/20220830121705.8618-1-maciej.fijalkowski@intel.com Signed-off-by: Sasha Levin <sashal@kernel.org>		2022-10-26 12:34:40 +02:00
..
Kconfig	treewide: Add SPDX license identifier - Makefile/Kconfig	2019-05-21 10:50:46 +02:00
Makefile	xsk: Introduce AF_XDP buffer allocation API	2020-05-21 17:31:26 -07:00
xdp_umem.c	xsk: Use kvcalloc to support large umems	2021-05-25 13:11:50 +02:00
xdp_umem.h	xsk: Fix umem cleanup bug at socket destruct	2020-11-20 15:52:39 +01:00
xsk.c	xsk: Fix backpressure mechanism on Tx	2022-10-26 12:34:40 +02:00
xsk.h	xdp: Add proper __rcu annotations to redirect map entries	2021-06-24 19:41:15 +02:00
xsk_buff_pool.c	xsk: Inherit need_wakeup flag for shared sockets	2022-10-12 09:53:26 +02:00
xsk_diag.c	xsk: Fix possible segfault in xsk umem diagnostics	2020-09-02 16:49:40 +02:00
xsk_queue.c	xsk: Remove MEM_TYPE_ZERO_COPY and corresponding code	2020-05-21 17:31:27 -07:00
xsk_queue.h	xsk: Fix backpressure mechanism on Tx	2022-10-26 12:34:40 +02:00
xskmap.c	xdp: Add proper __rcu annotations to redirect map entries	2021-06-24 19:41:15 +02:00