2005-04-17 02:20:36 +04:00
|
|
|
/*
|
|
|
|
* INET An implementation of the TCP/IP protocol suite for the LINUX
|
|
|
|
* operating system. INET is implemented using the BSD Socket
|
|
|
|
* interface as the means of communication with the user level.
|
|
|
|
*
|
|
|
|
* Implementation of the Transmission Control Protocol(TCP).
|
|
|
|
*
|
2005-05-06 03:16:16 +04:00
|
|
|
* Authors: Ross Biro
|
2005-04-17 02:20:36 +04:00
|
|
|
* Fred N. van Kempen, <waltje@uWalt.NL.Mugnet.ORG>
|
|
|
|
* Mark Evans, <evansmp@uhura.aston.ac.uk>
|
|
|
|
* Corey Minyard <wf-rch!minyard@relay.EU.net>
|
|
|
|
* Florian La Roche, <flla@stud.uni-sb.de>
|
|
|
|
* Charles Hedrick, <hedrick@klinzhai.rutgers.edu>
|
|
|
|
* Linus Torvalds, <torvalds@cs.helsinki.fi>
|
|
|
|
* Alan Cox, <gw4pts@gw4pts.ampr.org>
|
|
|
|
* Matthew Dillon, <dillon@apollo.west.oic.com>
|
|
|
|
* Arnt Gulbrandsen, <agulbra@nvg.unit.no>
|
|
|
|
* Jorge Cwik, <jorge@laser.satlink.net>
|
|
|
|
*/
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Changes:
|
|
|
|
* Pedro Roque : Fast Retransmit/Recovery.
|
|
|
|
* Two receive queues.
|
|
|
|
* Retransmit queue handled by TCP.
|
|
|
|
* Better retransmit timer handling.
|
|
|
|
* New congestion avoidance.
|
|
|
|
* Header prediction.
|
|
|
|
* Variable renaming.
|
|
|
|
*
|
|
|
|
* Eric : Fast Retransmit.
|
|
|
|
* Randy Scott : MSS option defines.
|
|
|
|
* Eric Schenk : Fixes to slow start algorithm.
|
|
|
|
* Eric Schenk : Yet another double ACK bug.
|
|
|
|
* Eric Schenk : Delayed ACK bug fixes.
|
|
|
|
* Eric Schenk : Floyd style fast retrans war avoidance.
|
|
|
|
* David S. Miller : Don't allow zero congestion window.
|
|
|
|
* Eric Schenk : Fix retransmitter so that it sends
|
|
|
|
* next packet on ack of previous packet.
|
|
|
|
* Andi Kleen : Moved open_request checking here
|
|
|
|
* and process RSTs for open_requests.
|
|
|
|
* Andi Kleen : Better prune_queue, and other fixes.
|
2005-11-11 04:13:47 +03:00
|
|
|
* Andrey Savochkin: Fix RTT measurements in the presence of
|
2005-04-17 02:20:36 +04:00
|
|
|
* timestamps.
|
|
|
|
* Andrey Savochkin: Check sequence numbers correctly when
|
|
|
|
* removing SACKs due to in sequence incoming
|
|
|
|
* data segments.
|
|
|
|
* Andi Kleen: Make sure we never ack data there is not
|
|
|
|
* enough room for. Also make this condition
|
|
|
|
* a fatal error if it might still happen.
|
2007-02-09 17:24:47 +03:00
|
|
|
* Andi Kleen: Add tcp_measure_rcv_mss to make
|
2005-04-17 02:20:36 +04:00
|
|
|
* connections with MSS<min(MTU,ann. MSS)
|
2007-02-09 17:24:47 +03:00
|
|
|
* work without delayed acks.
|
2005-04-17 02:20:36 +04:00
|
|
|
* Andi Kleen: Process packets with PSH set in the
|
|
|
|
* fast path.
|
|
|
|
* J Hadi Salim: ECN support
|
|
|
|
* Andrei Gurtov,
|
|
|
|
* Pasi Sarolahti,
|
|
|
|
* Panu Kuhlberg: Experimental audit of TCP (re)transmission
|
|
|
|
* engine. Lots of bugs are found.
|
|
|
|
* Pasi Sarolahti: F-RTO for dealing with spurious RTOs
|
|
|
|
*/
|
|
|
|
|
2012-03-12 11:03:32 +04:00
|
|
|
#define pr_fmt(fmt) "TCP: " fmt
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#include <linux/mm.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 11:04:11 +03:00
|
|
|
#include <linux/slab.h>
|
2005-04-17 02:20:36 +04:00
|
|
|
#include <linux/module.h>
|
|
|
|
#include <linux/sysctl.h>
|
2009-03-21 23:36:17 +03:00
|
|
|
#include <linux/kernel.h>
|
2014-10-12 02:17:29 +04:00
|
|
|
#include <linux/prefetch.h>
|
2008-05-05 09:14:42 +04:00
|
|
|
#include <net/dst.h>
|
2005-04-17 02:20:36 +04:00
|
|
|
#include <net/tcp.h>
|
|
|
|
#include <net/inet_common.h>
|
|
|
|
#include <linux/ipsec.h>
|
|
|
|
#include <asm/unaligned.h>
|
2014-08-05 06:11:50 +04:00
|
|
|
#include <linux/errqueue.h>
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2006-09-23 01:15:41 +04:00
|
|
|
int sysctl_tcp_timestamps __read_mostly = 1;
|
|
|
|
int sysctl_tcp_window_scaling __read_mostly = 1;
|
|
|
|
int sysctl_tcp_sack __read_mostly = 1;
|
|
|
|
int sysctl_tcp_fack __read_mostly = 1;
|
2014-10-28 07:45:24 +03:00
|
|
|
int sysctl_tcp_max_reordering __read_mostly = 300;
|
2006-09-23 01:15:41 +04:00
|
|
|
int sysctl_tcp_dsack __read_mostly = 1;
|
|
|
|
int sysctl_tcp_app_win __read_mostly = 31;
|
2012-05-02 06:28:41 +04:00
|
|
|
int sysctl_tcp_adv_win_scale __read_mostly = 1;
|
2010-07-10 01:22:10 +04:00
|
|
|
EXPORT_SYMBOL(sysctl_tcp_adv_win_scale);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2012-07-17 12:13:05 +04:00
|
|
|
/* rfc5961 challenge ack rate limiting */
|
|
|
|
int sysctl_tcp_challenge_ack_limit = 100;
|
|
|
|
|
2006-09-23 01:15:41 +04:00
|
|
|
int sysctl_tcp_stdurg __read_mostly;
|
|
|
|
int sysctl_tcp_rfc1337 __read_mostly;
|
|
|
|
int sysctl_tcp_max_orphans __read_mostly = NR_FILE;
|
[TCP]: Enable SACK enhanced FRTO (RFC4138) by default
Most of the description that follows comes from my mail to
netdev (some editing done):
Main obstacle to FRTO use is its deployment as it has to be on
the sender side where as wireless link is often the receiver's
access link. Take initiative on behalf of unlucky receivers and
enable it by default in future Linux TCP senders. Also IETF
seems to interested in advancing FRTO from experimental [1].
How does FRTO help?
===================
FRTO detects spurious RTOs and avoids a number of unnecessary
retransmissions and a couple of other problems that can arise
due to incorrect guess made at RTO (i.e., that segments were
lost when they actually got delayed which is likely to occur
e.g. in wireless environments with link-layer retransmission).
Though FRTO cannot prevent the first (potentially unnecessary)
retransmission at RTO, I suspect that it won't cost that much
even if you have to pay for each bit (won't be that high
percentage out of all packets after all :-)). However, usually
when you have a spurious RTO, not only the first segment
unnecessarily retransmitted but the *whole window*. It goes like
this: all cumulative ACKs got delayed due to in-order delivery,
then TCP will actually send 1.5*original cwnd worth of data in
the RTO's slow-start when the delayed ACKs arrive (basically the
original cwnd worth of it unnecessarily). In case one is
interested in minimizing unnecessary retransmissions e.g. due to
cost, those rexmissions must never see daylight. Besides, in the
worst case the generated burst overloads the bottleneck buffers
which is likely to significantly delay the further progress of
the flow. In case of ll rexmissions, ACK compression often
occurs at the same time making the burst very "sharp edged" (in
that case TCP often loses most of the segments above high_seq
=> very bad performance too). When FRTO is enabled, those
unnecessary retransmissions are fully avoided except for the
first segment and the cwnd behavior after detected spurious RTO
is determined by the response (one can tune that by sysctl).
Basic version (non-SACK enhanced one), FRTO can fail to detect
spurious RTO as spurious and falls back to conservative
behavior. ACK lossage is much less significant than reordering,
usually the FRTO can detect spurious RTO if at least 2
cumulative ACKs from original window are preserved (excluding
the ACK that advances to high_seq). With SACK-enhanced version,
the detection is quite robust.
FRTO should remove the need to set a high lower bound for the
RTO estimator due to delay spikes that occur relatively common
in some environments (esp. in wireless/cellular ones).
[1] http://www1.ietf.org/mail-archive/web/tcpm/current/msg02862.html
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-09-20 22:36:37 +04:00
|
|
|
int sysctl_tcp_frto __read_mostly = 2;
|
2015-10-17 07:57:42 +03:00
|
|
|
int sysctl_tcp_min_rtt_wlen __read_mostly = 300;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2010-02-18 07:48:19 +03:00
|
|
|
int sysctl_tcp_thin_dupack __read_mostly;
|
|
|
|
|
2006-09-23 01:15:41 +04:00
|
|
|
int sysctl_tcp_moderate_rcvbuf __read_mostly = 1;
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 14:00:43 +04:00
|
|
|
int sysctl_tcp_early_retrans __read_mostly = 3;
|
tcp: helpers to mitigate ACK loops by rate-limiting out-of-window dupacks
Helpers for mitigating ACK loops by rate-limiting dupacks sent in
response to incoming out-of-window packets.
This patch includes:
- rate-limiting logic
- sysctl to control how often we allow dupacks to out-of-window packets
- SNMP counter for cases where we rate-limited our dupack sending
The rate-limiting logic in this patch decides to not send dupacks in
response to out-of-window segments if (a) they are SYNs or pure ACKs
and (b) the remote endpoint is sending them faster than the configured
rate limit.
We rate-limit our responses rather than blocking them entirely or
resetting the connection, because legitimate connections can rely on
dupacks in response to some out-of-window segments. For example, zero
window probes are typically sent with a sequence number that is below
the current window, and ZWPs thus expect to thus elicit a dupack in
response.
We allow dupacks in response to TCP segments with data, because these
may be spurious retransmissions for which the remote endpoint wants to
receive DSACKs. This is safe because segments with data can't
realistically be part of ACK loops, which by their nature consist of
each side sending pure/data-less ACKs to each other.
The dupack interval is controlled by a new sysctl knob,
tcp_invalid_ratelimit, given in milliseconds, in case an administrator
needs to dial this upward in the face of a high-rate DoS attack. The
name and units are chosen to be analogous to the existing analogous
knob for ICMP, icmp_ratelimit.
The default value for tcp_invalid_ratelimit is 500ms, which allows at
most one such dupack per 500ms. This is chosen to be 2x faster than
the 1-second minimum RTO interval allowed by RFC 6298 (section 2, rule
2.4). We allow the extra 2x factor because network delay variations
can cause packets sent at 1 second intervals to be compressed and
arrive much closer.
Reported-by: Avery Fay <avery@mixpanel.com>
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-02-07 00:04:38 +03:00
|
|
|
int sysctl_tcp_invalid_ratelimit __read_mostly = HZ/2;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
#define FLAG_DATA 0x01 /* Incoming frame contained data. */
|
|
|
|
#define FLAG_WIN_UPDATE 0x02 /* Incoming ACK was a window update. */
|
|
|
|
#define FLAG_DATA_ACKED 0x04 /* This ACK acknowledged new data. */
|
|
|
|
#define FLAG_RETRANS_DATA_ACKED 0x08 /* "" "" some of which was retransmitted. */
|
|
|
|
#define FLAG_SYN_ACKED 0x10 /* This ACK acknowledged SYN. */
|
|
|
|
#define FLAG_DATA_SACKED 0x20 /* New SACK. */
|
|
|
|
#define FLAG_ECE 0x40 /* ECE in this ACK */
|
2015-07-02 00:11:14 +03:00
|
|
|
#define FLAG_LOST_RETRANS 0x80 /* This ACK marks some retransmission lost */
|
2005-04-17 02:20:36 +04:00
|
|
|
#define FLAG_SLOWPATH 0x100 /* Do not skip RFC checks for window update.*/
|
2013-03-20 17:33:00 +04:00
|
|
|
#define FLAG_ORIG_SACK_ACKED 0x200 /* Never retransmitted data are (s)acked */
|
2007-08-03 06:46:58 +04:00
|
|
|
#define FLAG_SND_UNA_ADVANCED 0x400 /* Snd_una was changed (!= FLAG_DATA_ACKED) */
|
2007-10-26 10:03:52 +04:00
|
|
|
#define FLAG_DSACKING_ACK 0x800 /* SACK blocks contained D-SACK info */
|
2007-12-31 15:49:21 +03:00
|
|
|
#define FLAG_SACK_RENEGING 0x2000 /* snd_una advanced to a sacked seq */
|
tcp: call tcp_replace_ts_recent() from tcp_ack()
commit bd090dfc634d (tcp: tcp_replace_ts_recent() should not be called
from tcp_validate_incoming()) introduced a TS ecr bug in slow path
processing.
1 A > B P. 1:10001(10000) ack 1 <nop,nop,TS val 1001 ecr 200>
2 B < A . 1:1(0) ack 1 win 257 <sack 9001:10001,TS val 300 ecr 1001>
3 A > B . 1:1001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
4 A > B . 1001:2001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
(ecr 200 should be ecr 300 in packets 3 & 4)
Problem is tcp_ack() can trigger send of new packets (retransmits),
reflecting the prior TSval, instead of the TSval contained in the
currently processed incoming packet.
Fix this by calling tcp_replace_ts_recent() from tcp_ack() after the
checks, but before the actions.
Reported-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 11:19:48 +04:00
|
|
|
#define FLAG_UPDATE_TS_RECENT 0x4000 /* tcp_replace_ts_recent() */
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
#define FLAG_ACKED (FLAG_DATA_ACKED|FLAG_SYN_ACKED)
|
|
|
|
#define FLAG_NOT_DUP (FLAG_DATA|FLAG_WIN_UPDATE|FLAG_ACKED)
|
|
|
|
#define FLAG_CA_ALERT (FLAG_DATA_SACKED|FLAG_ECE)
|
|
|
|
#define FLAG_FORWARD_PROGRESS (FLAG_ACKED|FLAG_DATA_SACKED)
|
|
|
|
|
|
|
|
#define TCP_REMNANT (TCP_FLAG_FIN|TCP_FLAG_URG|TCP_FLAG_SYN|TCP_FLAG_PSH)
|
2007-05-27 13:04:16 +04:00
|
|
|
#define TCP_HP_BITS (~(TCP_RESERVED_BITS|TCP_FLAG_PSH))
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2016-02-02 21:33:04 +03:00
|
|
|
#define REXMIT_NONE 0 /* no loss recovery to do */
|
|
|
|
#define REXMIT_LOST 1 /* retransmit packets marked lost */
|
|
|
|
#define REXMIT_NEW 2 /* FRTO-style transmit of unsent/new packets */
|
|
|
|
|
2007-02-09 17:24:47 +03:00
|
|
|
/* Adapt the MSS value used to make delayed ack decision to the
|
2005-04-17 02:20:36 +04:00
|
|
|
* real world.
|
2007-02-09 17:24:47 +03:00
|
|
|
*/
|
2008-01-01 01:57:14 +03:00
|
|
|
static void tcp_measure_rcv_mss(struct sock *sk, const struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 07:10:42 +04:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2007-02-09 17:24:47 +03:00
|
|
|
const unsigned int lss = icsk->icsk_ack.last_seg_size;
|
2005-08-10 07:10:42 +04:00
|
|
|
unsigned int len;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2007-02-09 17:24:47 +03:00
|
|
|
icsk->icsk_ack.last_seg_size = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* skb->len may jitter because of SACKs, even if peer
|
|
|
|
* sends good full-sized frames.
|
|
|
|
*/
|
2008-01-01 01:57:14 +03:00
|
|
|
len = skb_shinfo(skb)->gso_size ? : skb->len;
|
2005-08-10 07:10:42 +04:00
|
|
|
if (len >= icsk->icsk_ack.rcv_mss) {
|
|
|
|
icsk->icsk_ack.rcv_mss = len;
|
2005-04-17 02:20:36 +04:00
|
|
|
} else {
|
|
|
|
/* Otherwise, we make more careful check taking into account,
|
|
|
|
* that SACKs block is variable.
|
|
|
|
*
|
|
|
|
* "len" is invariant segment length, including TCP header.
|
|
|
|
*/
|
2007-04-26 05:04:18 +04:00
|
|
|
len += skb->data - skb_transport_header(skb);
|
2009-11-10 12:51:18 +03:00
|
|
|
if (len >= TCP_MSS_DEFAULT + sizeof(struct tcphdr) ||
|
2005-04-17 02:20:36 +04:00
|
|
|
/* If PSH is not set, packet should be
|
|
|
|
* full sized, provided peer TCP is not badly broken.
|
|
|
|
* This observation (if it is correct 8)) allows
|
|
|
|
* to handle super-low mtu links fairly.
|
|
|
|
*/
|
|
|
|
(len >= TCP_MIN_MSS + sizeof(struct tcphdr) &&
|
2007-04-11 08:04:22 +04:00
|
|
|
!(tcp_flag_word(tcp_hdr(skb)) & TCP_REMNANT))) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Subtract also invariant (if peer is RFC compliant),
|
|
|
|
* tcp header plus fixed timestamp option length.
|
|
|
|
* Resulting "len" is MSS free of SACK jitter.
|
|
|
|
*/
|
2005-08-10 07:10:42 +04:00
|
|
|
len -= tcp_sk(sk)->tcp_header_len;
|
|
|
|
icsk->icsk_ack.last_seg_size = len;
|
2005-04-17 02:20:36 +04:00
|
|
|
if (len == lss) {
|
2005-08-10 07:10:42 +04:00
|
|
|
icsk->icsk_ack.rcv_mss = len;
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
2006-09-19 23:52:50 +04:00
|
|
|
if (icsk->icsk_ack.pending & ICSK_ACK_PUSHED)
|
|
|
|
icsk->icsk_ack.pending |= ICSK_ACK_PUSHED2;
|
2005-08-10 07:10:42 +04:00
|
|
|
icsk->icsk_ack.pending |= ICSK_ACK_PUSHED;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
static void tcp_incr_quickack(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 07:10:42 +04:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2012-04-15 09:58:06 +04:00
|
|
|
unsigned int quickacks = tcp_sk(sk)->rcv_wnd / (2 * icsk->icsk_ack.rcv_mss);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
if (quickacks == 0)
|
|
|
|
quickacks = 2;
|
2005-08-10 07:10:42 +04:00
|
|
|
if (quickacks > icsk->icsk_ack.quick)
|
|
|
|
icsk->icsk_ack.quick = min(quickacks, TCP_MAX_QUICKACKS);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2010-09-28 23:30:14 +04:00
|
|
|
static void tcp_enter_quickack_mode(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 07:10:42 +04:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
tcp_incr_quickack(sk);
|
|
|
|
icsk->icsk_ack.pingpong = 0;
|
|
|
|
icsk->icsk_ack.ato = TCP_ATO_MIN;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Send ACKs quickly, if "quick" count is not exhausted
|
|
|
|
* and the session is not interactive.
|
|
|
|
*/
|
|
|
|
|
2015-07-08 03:12:28 +03:00
|
|
|
static bool tcp_in_quickack_mode(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 07:10:42 +04:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2015-07-08 03:12:28 +03:00
|
|
|
const struct dst_entry *dst = __sk_dst_get(sk);
|
2012-05-17 03:15:34 +04:00
|
|
|
|
2015-07-08 03:12:28 +03:00
|
|
|
return (dst && dst_metric(dst, RTAX_QUICKACK)) ||
|
|
|
|
(icsk->icsk_ack.quick && !icsk->icsk_ack.pingpong);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
static void tcp_ecn_queue_cwr(struct tcp_sock *tp)
|
2007-05-27 13:04:16 +04:00
|
|
|
{
|
2008-01-01 01:57:14 +03:00
|
|
|
if (tp->ecn_flags & TCP_ECN_OK)
|
2007-05-27 13:04:16 +04:00
|
|
|
tp->ecn_flags |= TCP_ECN_QUEUE_CWR;
|
|
|
|
}
|
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
static void tcp_ecn_accept_cwr(struct tcp_sock *tp, const struct sk_buff *skb)
|
2007-05-27 13:04:16 +04:00
|
|
|
{
|
|
|
|
if (tcp_hdr(skb)->cwr)
|
|
|
|
tp->ecn_flags &= ~TCP_ECN_DEMAND_CWR;
|
|
|
|
}
|
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
static void tcp_ecn_withdraw_cwr(struct tcp_sock *tp)
|
2007-05-27 13:04:16 +04:00
|
|
|
{
|
|
|
|
tp->ecn_flags &= ~TCP_ECN_DEMAND_CWR;
|
|
|
|
}
|
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
static void __tcp_ecn_check_ce(struct tcp_sock *tp, const struct sk_buff *skb)
|
2007-05-27 13:04:16 +04:00
|
|
|
{
|
2011-09-27 10:20:08 +04:00
|
|
|
switch (TCP_SKB_CB(skb)->ip_dsfield & INET_ECN_MASK) {
|
2011-09-23 00:02:19 +04:00
|
|
|
case INET_ECN_NOT_ECT:
|
2007-05-27 13:04:16 +04:00
|
|
|
/* Funny extension: if ECT is not set on a segment,
|
2011-09-23 00:02:19 +04:00
|
|
|
* and we already seen ECT on a previous segment,
|
|
|
|
* it is probably a retransmit.
|
|
|
|
*/
|
|
|
|
if (tp->ecn_flags & TCP_ECN_SEEN)
|
2007-05-27 13:04:16 +04:00
|
|
|
tcp_enter_quickack_mode((struct sock *)tp);
|
2011-09-23 00:02:19 +04:00
|
|
|
break;
|
|
|
|
case INET_ECN_CE:
|
2014-09-27 00:37:35 +04:00
|
|
|
if (tcp_ca_needs_ecn((struct sock *)tp))
|
|
|
|
tcp_ca_event((struct sock *)tp, CA_EVENT_ECN_IS_CE);
|
|
|
|
|
2012-08-06 15:04:43 +04:00
|
|
|
if (!(tp->ecn_flags & TCP_ECN_DEMAND_CWR)) {
|
|
|
|
/* Better not delay acks, sender can have a very low cwnd */
|
|
|
|
tcp_enter_quickack_mode((struct sock *)tp);
|
|
|
|
tp->ecn_flags |= TCP_ECN_DEMAND_CWR;
|
|
|
|
}
|
2014-09-27 00:37:35 +04:00
|
|
|
tp->ecn_flags |= TCP_ECN_SEEN;
|
|
|
|
break;
|
2011-09-23 00:02:19 +04:00
|
|
|
default:
|
2014-09-27 00:37:35 +04:00
|
|
|
if (tcp_ca_needs_ecn((struct sock *)tp))
|
|
|
|
tcp_ca_event((struct sock *)tp, CA_EVENT_ECN_NO_CE);
|
2011-09-23 00:02:19 +04:00
|
|
|
tp->ecn_flags |= TCP_ECN_SEEN;
|
2014-09-27 00:37:35 +04:00
|
|
|
break;
|
2007-05-27 13:04:16 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
static void tcp_ecn_check_ce(struct tcp_sock *tp, const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
if (tp->ecn_flags & TCP_ECN_OK)
|
|
|
|
__tcp_ecn_check_ce(tp, skb);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void tcp_ecn_rcv_synack(struct tcp_sock *tp, const struct tcphdr *th)
|
2007-05-27 13:04:16 +04:00
|
|
|
{
|
2008-01-01 01:57:14 +03:00
|
|
|
if ((tp->ecn_flags & TCP_ECN_OK) && (!th->ece || th->cwr))
|
2007-05-27 13:04:16 +04:00
|
|
|
tp->ecn_flags &= ~TCP_ECN_OK;
|
|
|
|
}
|
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
static void tcp_ecn_rcv_syn(struct tcp_sock *tp, const struct tcphdr *th)
|
2007-05-27 13:04:16 +04:00
|
|
|
{
|
2008-01-01 01:57:14 +03:00
|
|
|
if ((tp->ecn_flags & TCP_ECN_OK) && (!th->ece || !th->cwr))
|
2007-05-27 13:04:16 +04:00
|
|
|
tp->ecn_flags &= ~TCP_ECN_OK;
|
|
|
|
}
|
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
static bool tcp_ecn_rcv_ecn_echo(const struct tcp_sock *tp, const struct tcphdr *th)
|
2007-05-27 13:04:16 +04:00
|
|
|
{
|
2008-01-01 01:57:14 +03:00
|
|
|
if (th->ece && !th->syn && (tp->ecn_flags & TCP_ECN_OK))
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
|
|
|
return false;
|
2007-05-27 13:04:16 +04:00
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Buffer size and advertised window tuning.
|
|
|
|
*
|
|
|
|
* 1. Tuning sk->sk_sndbuf, when connection enters established state.
|
|
|
|
*/
|
|
|
|
|
2013-10-01 21:23:44 +04:00
|
|
|
static void tcp_sndbuf_expand(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2013-10-01 21:23:44 +04:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int sndmem, per_mss;
|
|
|
|
u32 nr_segs;
|
|
|
|
|
|
|
|
/* Worst case is non GSO/TSO : each frame consumes one skb
|
|
|
|
* and skb->head is kmalloced using power of two area of memory
|
|
|
|
*/
|
|
|
|
per_mss = max_t(u32, tp->rx_opt.mss_clamp, tp->mss_cache) +
|
|
|
|
MAX_TCP_HEADER +
|
|
|
|
SKB_DATA_ALIGN(sizeof(struct skb_shared_info));
|
|
|
|
|
|
|
|
per_mss = roundup_pow_of_two(per_mss) +
|
|
|
|
SKB_DATA_ALIGN(sizeof(struct sk_buff));
|
|
|
|
|
|
|
|
nr_segs = max_t(u32, TCP_INIT_CWND, tp->snd_cwnd);
|
|
|
|
nr_segs = max_t(u32, nr_segs, tp->reordering + 1);
|
|
|
|
|
|
|
|
/* Fast Recovery (RFC 5681 3.2) :
|
|
|
|
* Cubic needs 1.7 factor, rounded to 2 to include
|
|
|
|
* extra cushion (application might react slowly to POLLOUT)
|
|
|
|
*/
|
|
|
|
sndmem = 2 * nr_segs * per_mss;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2011-10-13 22:24:42 +04:00
|
|
|
if (sk->sk_sndbuf < sndmem)
|
|
|
|
sk->sk_sndbuf = min(sndmem, sysctl_tcp_wmem[2]);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* 2. Tuning advertised window (window_clamp, rcv_ssthresh)
|
|
|
|
*
|
|
|
|
* All tcp_full_space() is split to two parts: "network" buffer, allocated
|
|
|
|
* forward and advertised in receiver window (tp->rcv_wnd) and
|
|
|
|
* "application buffer", required to isolate scheduling/application
|
|
|
|
* latencies from network.
|
|
|
|
* window_clamp is maximal advertised window. It can be less than
|
|
|
|
* tcp_full_space(), in this case tcp_full_space() - window_clamp
|
|
|
|
* is reserved for "application" buffer. The less window_clamp is
|
|
|
|
* the smoother our behaviour from viewpoint of network, but the lower
|
|
|
|
* throughput and the higher sensitivity of the connection to losses. 8)
|
|
|
|
*
|
|
|
|
* rcv_ssthresh is more strict window_clamp used at "slow start"
|
|
|
|
* phase to predict further behaviour of this connection.
|
|
|
|
* It is used for two goals:
|
|
|
|
* - to enforce header prediction at sender, even when application
|
|
|
|
* requires some significant "application buffer". It is check #1.
|
|
|
|
* - to prevent pruning of receive queue because of misprediction
|
|
|
|
* of receiver window. Check #2.
|
|
|
|
*
|
|
|
|
* The scheme does not work when sender sends good segments opening
|
2005-11-11 04:13:47 +03:00
|
|
|
* window and then starts to feed us spaghetti. But it should work
|
2005-04-17 02:20:36 +04:00
|
|
|
* in common situations. Otherwise, we have to rely on queue collapsing.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* Slow part of check#2. */
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
static int __tcp_grow_window(const struct sock *sk, const struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Optimize this! */
|
2007-12-21 17:07:53 +03:00
|
|
|
int truesize = tcp_win_from_space(skb->truesize) >> 1;
|
|
|
|
int window = tcp_win_from_space(sysctl_tcp_rmem[2]) >> 1;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
while (tp->rcv_ssthresh <= window) {
|
|
|
|
if (truesize <= skb->len)
|
2005-08-10 07:10:42 +04:00
|
|
|
return 2 * inet_csk(sk)->icsk_ack.rcv_mss;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
truesize >>= 1;
|
|
|
|
window >>= 1;
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2011-10-21 13:22:42 +04:00
|
|
|
static void tcp_grow_window(struct sock *sk, const struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Check #1 */
|
|
|
|
if (tp->rcv_ssthresh < tp->window_clamp &&
|
|
|
|
(int)tp->rcv_ssthresh < tcp_space(sk) &&
|
2015-05-15 22:39:27 +03:00
|
|
|
!tcp_under_memory_pressure(sk)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
int incr;
|
|
|
|
|
|
|
|
/* Check #2. Increase window, if skb with such overhead
|
|
|
|
* will fit to rcvbuf in future.
|
|
|
|
*/
|
|
|
|
if (tcp_win_from_space(skb->truesize) <= skb->len)
|
2008-01-01 01:57:14 +03:00
|
|
|
incr = 2 * tp->advmss;
|
2005-04-17 02:20:36 +04:00
|
|
|
else
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
incr = __tcp_grow_window(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (incr) {
|
2012-04-17 03:28:07 +04:00
|
|
|
incr = max_t(int, incr, 2 * skb->len);
|
2008-01-01 01:57:14 +03:00
|
|
|
tp->rcv_ssthresh = min(tp->rcv_ssthresh + incr,
|
|
|
|
tp->window_clamp);
|
2005-08-10 07:10:42 +04:00
|
|
|
inet_csk(sk)->icsk_ack.quick |= 1;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* 3. Tuning rcvbuf, when connection enters established state. */
|
|
|
|
static void tcp_fixup_rcvbuf(struct sock *sk)
|
|
|
|
{
|
2011-10-21 00:53:56 +04:00
|
|
|
u32 mss = tcp_sk(sk)->advmss;
|
|
|
|
int rcvmem;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-06-12 02:35:32 +04:00
|
|
|
rcvmem = 2 * SKB_TRUESIZE(mss + MAX_TCP_HEADER) *
|
|
|
|
tcp_default_init_rwnd(mss);
|
2011-10-21 00:53:56 +04:00
|
|
|
|
2013-09-21 00:56:58 +04:00
|
|
|
/* Dynamic Right Sizing (DRS) has 2 to 3 RTT latency
|
|
|
|
* Allow enough cushion so that sender is not limited by our window
|
|
|
|
*/
|
|
|
|
if (sysctl_tcp_moderate_rcvbuf)
|
|
|
|
rcvmem <<= 2;
|
|
|
|
|
2011-10-21 00:53:56 +04:00
|
|
|
if (sk->sk_rcvbuf < rcvmem)
|
|
|
|
sk->sk_rcvbuf = min(rcvmem, sysctl_tcp_rmem[2]);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2005-11-11 04:13:47 +03:00
|
|
|
/* 4. Try to fixup all. It is made immediately after connection enters
|
2005-04-17 02:20:36 +04:00
|
|
|
* established state.
|
|
|
|
*/
|
2012-08-31 16:29:11 +04:00
|
|
|
void tcp_init_buffer_space(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int maxwin;
|
|
|
|
|
|
|
|
if (!(sk->sk_userlocks & SOCK_RCVBUF_LOCK))
|
|
|
|
tcp_fixup_rcvbuf(sk);
|
|
|
|
if (!(sk->sk_userlocks & SOCK_SNDBUF_LOCK))
|
2013-10-01 21:23:44 +04:00
|
|
|
tcp_sndbuf_expand(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
tp->rcvq_space.space = tp->rcv_wnd;
|
2013-09-21 00:56:58 +04:00
|
|
|
tp->rcvq_space.time = tcp_time_stamp;
|
|
|
|
tp->rcvq_space.seq = tp->copied_seq;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
maxwin = tcp_full_space(sk);
|
|
|
|
|
|
|
|
if (tp->window_clamp >= maxwin) {
|
|
|
|
tp->window_clamp = maxwin;
|
|
|
|
|
|
|
|
if (sysctl_tcp_app_win && maxwin > 4 * tp->advmss)
|
|
|
|
tp->window_clamp = max(maxwin -
|
|
|
|
(maxwin >> sysctl_tcp_app_win),
|
|
|
|
4 * tp->advmss);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Force reservation of one segment. */
|
|
|
|
if (sysctl_tcp_app_win &&
|
|
|
|
tp->window_clamp > 2 * tp->advmss &&
|
|
|
|
tp->window_clamp + tp->advmss > maxwin)
|
|
|
|
tp->window_clamp = max(2 * tp->advmss, maxwin - tp->advmss);
|
|
|
|
|
|
|
|
tp->rcv_ssthresh = min(tp->rcv_ssthresh, tp->window_clamp);
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* 5. Recalculate window clamp after socket hit its memory bounds. */
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
static void tcp_clamp_window(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-08-10 11:03:31 +04:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2005-08-10 11:03:31 +04:00
|
|
|
icsk->icsk_ack.quick = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2005-11-11 04:11:48 +03:00
|
|
|
if (sk->sk_rcvbuf < sysctl_tcp_rmem[2] &&
|
|
|
|
!(sk->sk_userlocks & SOCK_RCVBUF_LOCK) &&
|
2015-05-15 22:39:27 +03:00
|
|
|
!tcp_under_memory_pressure(sk) &&
|
2011-12-12 01:47:02 +04:00
|
|
|
sk_memory_allocated(sk) < sk_prot_mem_limits(sk, 0)) {
|
2005-11-11 04:11:48 +03:00
|
|
|
sk->sk_rcvbuf = min(atomic_read(&sk->sk_rmem_alloc),
|
|
|
|
sysctl_tcp_rmem[2]);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2005-11-11 04:11:48 +03:00
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf)
|
2008-01-01 01:57:14 +03:00
|
|
|
tp->rcv_ssthresh = min(tp->window_clamp, 2U * tp->advmss);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2006-01-04 03:03:49 +03:00
|
|
|
/* Initialize RCV_MSS value.
|
|
|
|
* RCV_MSS is an our guess about MSS used by the peer.
|
|
|
|
* We haven't any direct information about the MSS.
|
|
|
|
* It's better to underestimate the RCV_MSS rather than overestimate.
|
|
|
|
* Overestimations make us ACKing less frequently than needed.
|
|
|
|
* Underestimations are more easy to detect and fix by tcp_measure_rcv_mss().
|
|
|
|
*/
|
|
|
|
void tcp_initialize_rcv_mss(struct sock *sk)
|
|
|
|
{
|
2011-10-21 13:22:42 +04:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2006-01-04 03:03:49 +03:00
|
|
|
unsigned int hint = min_t(unsigned int, tp->advmss, tp->mss_cache);
|
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
hint = min(hint, tp->rcv_wnd / 2);
|
2009-11-10 12:51:18 +03:00
|
|
|
hint = min(hint, TCP_MSS_DEFAULT);
|
2006-01-04 03:03:49 +03:00
|
|
|
hint = max(hint, TCP_MIN_MSS);
|
|
|
|
|
|
|
|
inet_csk(sk)->icsk_ack.rcv_mss = hint;
|
|
|
|
}
|
2010-07-10 01:22:10 +04:00
|
|
|
EXPORT_SYMBOL(tcp_initialize_rcv_mss);
|
2006-01-04 03:03:49 +03:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Receiver "autotuning" code.
|
|
|
|
*
|
|
|
|
* The algorithm for RTT estimation w/o timestamps is based on
|
|
|
|
* Dynamic Right-Sizing (DRS) by Wu Feng and Mike Fisk of LANL.
|
2010-10-18 13:03:14 +04:00
|
|
|
* <http://public.lanl.gov/radiant/pubs.html#DRS>
|
2005-04-17 02:20:36 +04:00
|
|
|
*
|
|
|
|
* More detail on this code can be found at
|
2010-10-18 13:03:14 +04:00
|
|
|
* <http://staff.psc.edu/jheffner/>,
|
2005-04-17 02:20:36 +04:00
|
|
|
* though this reference is out of date. A new paper
|
|
|
|
* is pending.
|
|
|
|
*/
|
|
|
|
static void tcp_rcv_rtt_update(struct tcp_sock *tp, u32 sample, int win_dep)
|
|
|
|
{
|
|
|
|
u32 new_sample = tp->rcv_rtt_est.rtt;
|
|
|
|
long m = sample;
|
|
|
|
|
|
|
|
if (m == 0)
|
|
|
|
m = 1;
|
|
|
|
|
|
|
|
if (new_sample != 0) {
|
|
|
|
/* If we sample in larger samples in the non-timestamp
|
|
|
|
* case, we could grossly overestimate the RTT especially
|
|
|
|
* with chatty applications or bulk transfer apps which
|
|
|
|
* are stalled on filesystem I/O.
|
|
|
|
*
|
|
|
|
* Also, since we are only going for a minimum in the
|
2005-11-16 02:17:10 +03:00
|
|
|
* non-timestamp case, we do not smooth things out
|
2005-11-11 04:13:47 +03:00
|
|
|
* else with timestamps disabled convergence takes too
|
2005-04-17 02:20:36 +04:00
|
|
|
* long.
|
|
|
|
*/
|
|
|
|
if (!win_dep) {
|
|
|
|
m -= (new_sample >> 3);
|
|
|
|
new_sample += m;
|
2012-04-10 11:59:20 +04:00
|
|
|
} else {
|
|
|
|
m <<= 3;
|
|
|
|
if (m < new_sample)
|
|
|
|
new_sample = m;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
} else {
|
2005-11-11 04:13:47 +03:00
|
|
|
/* No previous measure. */
|
2005-04-17 02:20:36 +04:00
|
|
|
new_sample = m << 3;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (tp->rcv_rtt_est.rtt != new_sample)
|
|
|
|
tp->rcv_rtt_est.rtt = new_sample;
|
|
|
|
}
|
|
|
|
|
|
|
|
static inline void tcp_rcv_rtt_measure(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
if (tp->rcv_rtt_est.time == 0)
|
|
|
|
goto new_measure;
|
|
|
|
if (before(tp->rcv_nxt, tp->rcv_rtt_est.seq))
|
|
|
|
return;
|
2012-04-27 19:29:37 +04:00
|
|
|
tcp_rcv_rtt_update(tp, tcp_time_stamp - tp->rcv_rtt_est.time, 1);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
new_measure:
|
|
|
|
tp->rcv_rtt_est.seq = tp->rcv_nxt + tp->rcv_wnd;
|
|
|
|
tp->rcv_rtt_est.time = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
static inline void tcp_rcv_rtt_measure_ts(struct sock *sk,
|
|
|
|
const struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 07:10:42 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (tp->rx_opt.rcv_tsecr &&
|
|
|
|
(TCP_SKB_CB(skb)->end_seq -
|
2005-08-10 07:10:42 +04:00
|
|
|
TCP_SKB_CB(skb)->seq >= inet_csk(sk)->icsk_ack.rcv_mss))
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_rcv_rtt_update(tp, tcp_time_stamp - tp->rx_opt.rcv_tsecr, 0);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function should be called every time data is copied to user space.
|
|
|
|
* It calculates the appropriate TCP receive buffer space.
|
|
|
|
*/
|
|
|
|
void tcp_rcv_space_adjust(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int time;
|
2013-09-21 00:56:58 +04:00
|
|
|
int copied;
|
2007-02-09 17:24:47 +03:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
time = tcp_time_stamp - tp->rcvq_space.time;
|
2008-01-01 01:57:14 +03:00
|
|
|
if (time < (tp->rcv_rtt_est.rtt >> 3) || tp->rcv_rtt_est.rtt == 0)
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
2007-02-09 17:24:47 +03:00
|
|
|
|
2013-09-21 00:56:58 +04:00
|
|
|
/* Number of bytes copied to user in last RTT */
|
|
|
|
copied = tp->copied_seq - tp->rcvq_space.seq;
|
|
|
|
if (copied <= tp->rcvq_space.space)
|
|
|
|
goto new_measure;
|
|
|
|
|
|
|
|
/* A bit of theory :
|
|
|
|
* copied = bytes received in previous RTT, our base window
|
|
|
|
* To cope with packet losses, we need a 2x factor
|
|
|
|
* To cope with slow start, and sender growing its cwin by 100 %
|
|
|
|
* every RTT, we need a 4x factor, because the ACK we are sending
|
|
|
|
* now is for the next RTT, not the current one :
|
|
|
|
* <prev RTT . ><current RTT .. ><next RTT .... >
|
|
|
|
*/
|
|
|
|
|
|
|
|
if (sysctl_tcp_moderate_rcvbuf &&
|
|
|
|
!(sk->sk_userlocks & SOCK_RCVBUF_LOCK)) {
|
|
|
|
int rcvwin, rcvmem, rcvbuf;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-09-21 00:56:58 +04:00
|
|
|
/* minimal window to cope with packet losses, assuming
|
|
|
|
* steady state. Add some cushion because of small variations.
|
|
|
|
*/
|
|
|
|
rcvwin = (copied << 1) + 16 * tp->advmss;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-09-21 00:56:58 +04:00
|
|
|
/* If rate increased by 25%,
|
|
|
|
* assume slow start, rcvwin = 3 * copied
|
|
|
|
* If rate increased by 50%,
|
|
|
|
* assume sender can use 2x growth, rcvwin = 4 * copied
|
|
|
|
*/
|
|
|
|
if (copied >=
|
|
|
|
tp->rcvq_space.space + (tp->rcvq_space.space >> 2)) {
|
|
|
|
if (copied >=
|
|
|
|
tp->rcvq_space.space + (tp->rcvq_space.space >> 1))
|
|
|
|
rcvwin <<= 1;
|
|
|
|
else
|
|
|
|
rcvwin += (rcvwin >> 1);
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-09-21 00:56:58 +04:00
|
|
|
rcvmem = SKB_TRUESIZE(tp->advmss + MAX_TCP_HEADER);
|
|
|
|
while (tcp_win_from_space(rcvmem) < tp->advmss)
|
|
|
|
rcvmem += 128;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-09-21 00:56:58 +04:00
|
|
|
rcvbuf = min(rcvwin / tp->advmss * rcvmem, sysctl_tcp_rmem[2]);
|
|
|
|
if (rcvbuf > sk->sk_rcvbuf) {
|
|
|
|
sk->sk_rcvbuf = rcvbuf;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-09-21 00:56:58 +04:00
|
|
|
/* Make the window clamp follow along. */
|
|
|
|
tp->window_clamp = rcvwin;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
2013-09-21 00:56:58 +04:00
|
|
|
tp->rcvq_space.space = copied;
|
2007-02-09 17:24:47 +03:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
new_measure:
|
|
|
|
tp->rcvq_space.seq = tp->copied_seq;
|
|
|
|
tp->rcvq_space.time = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* There is something which you must keep in mind when you analyze the
|
|
|
|
* behavior of the tp->ato delayed ack timeout interval. When a
|
|
|
|
* connection starts up, we want to ack as quickly as possible. The
|
|
|
|
* problem is that "good" TCP's do slow start at the beginning of data
|
|
|
|
* transmission. The means that until we send the first few ACK's the
|
|
|
|
* sender will sit on his end and only queue most of his data, because
|
|
|
|
* he can only send snd_cwnd unacked packets at any given time. For
|
|
|
|
* each ACK we send, he increments snd_cwnd and transmits more of his
|
|
|
|
* queue. -DaveM
|
|
|
|
*/
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
static void tcp_event_data_recv(struct sock *sk, struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-08-10 07:10:42 +04:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
u32 now;
|
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
inet_csk_schedule_ack(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_measure_rcv_mss(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
tcp_rcv_rtt_measure(tp);
|
2007-02-09 17:24:47 +03:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
now = tcp_time_stamp;
|
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
if (!icsk->icsk_ack.ato) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* The _first_ data packet received, initialize
|
|
|
|
* delayed ACK engine.
|
|
|
|
*/
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_incr_quickack(sk);
|
|
|
|
icsk->icsk_ack.ato = TCP_ATO_MIN;
|
2005-04-17 02:20:36 +04:00
|
|
|
} else {
|
2005-08-10 07:10:42 +04:00
|
|
|
int m = now - icsk->icsk_ack.lrcvtime;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
if (m <= TCP_ATO_MIN / 2) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* The fastest case is the first. */
|
2005-08-10 07:10:42 +04:00
|
|
|
icsk->icsk_ack.ato = (icsk->icsk_ack.ato >> 1) + TCP_ATO_MIN / 2;
|
|
|
|
} else if (m < icsk->icsk_ack.ato) {
|
|
|
|
icsk->icsk_ack.ato = (icsk->icsk_ack.ato >> 1) + m;
|
|
|
|
if (icsk->icsk_ack.ato > icsk->icsk_rto)
|
|
|
|
icsk->icsk_ack.ato = icsk->icsk_rto;
|
|
|
|
} else if (m > icsk->icsk_rto) {
|
2005-11-11 04:13:47 +03:00
|
|
|
/* Too long gap. Apparently sender failed to
|
2005-04-17 02:20:36 +04:00
|
|
|
* restart window, so that we send ACKs quickly.
|
|
|
|
*/
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_incr_quickack(sk);
|
2007-12-31 11:11:19 +03:00
|
|
|
sk_mem_reclaim(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
2005-08-10 07:10:42 +04:00
|
|
|
icsk->icsk_ack.lrcvtime = now;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
tcp_ecn_check_ce(tp, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (skb->len >= 128)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_grow_window(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Called to compute a smoothed rtt estimate. The data fed to this
|
|
|
|
* routine either comes from timestamps, or from segments that were
|
|
|
|
* known _not_ to have been retransmitted [see Karn/Partridge
|
|
|
|
* Proceedings SIGCOMM 87]. The algorithm is from the SIGCOMM 88
|
|
|
|
* piece by Van Jacobson.
|
|
|
|
* NOTE: the next three routines used to be one big routine.
|
|
|
|
* To save cycles in the RFC 1323 implementation it was better to break
|
|
|
|
* it up into three procedures. -- erics
|
|
|
|
*/
|
2014-02-27 02:02:48 +04:00
|
|
|
static void tcp_rtt_estimator(struct sock *sk, long mrtt_us)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 11:03:31 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2014-02-27 02:02:48 +04:00
|
|
|
long m = mrtt_us; /* RTT */
|
|
|
|
u32 srtt = tp->srtt_us;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* The following amusing code comes from Jacobson's
|
|
|
|
* article in SIGCOMM '88. Note that rtt and mdev
|
|
|
|
* are scaled versions of rtt and mean deviation.
|
2007-02-09 17:24:47 +03:00
|
|
|
* This is designed to be as fast as possible
|
2005-04-17 02:20:36 +04:00
|
|
|
* m stands for "measurement".
|
|
|
|
*
|
|
|
|
* On a 1990 paper the rto value is changed to:
|
|
|
|
* RTO = rtt + 4 * mdev
|
|
|
|
*
|
|
|
|
* Funny. This algorithm seems to be very broken.
|
|
|
|
* These formulae increase RTO, when it should be decreased, increase
|
2005-11-16 02:17:10 +03:00
|
|
|
* too slowly, when it should be increased quickly, decrease too quickly
|
2005-04-17 02:20:36 +04:00
|
|
|
* etc. I guess in BSD RTO takes ONE value, so that it is absolutely
|
|
|
|
* does not matter how to _calculate_ it. Seems, it was trap
|
|
|
|
* that VJ failed to avoid. 8)
|
|
|
|
*/
|
tcp: remove 1ms offset in srtt computation
TCP pacing depends on an accurate srtt estimation.
Current srtt estimation is using jiffie resolution,
and has an artificial offset of at least 1 ms, which can produce
slowdowns when FQ/pacing is used, especially in DC world,
where typical rtt is below 1 ms.
We are planning a switch to usec resolution for linux-3.15,
but in the meantime, this patch removes the 1 ms offset.
All we need is to have tp->srtt minimal value of 1 to differentiate
the case of srtt being initialized or not, not 8.
The problematic behavior was observed on a 40Gbit testbed,
where 32 concurrent netperf were reaching 12Gbps of aggregate
speed, instead of line speed.
This patch also has the effect of reporting more accurate srtt and send
rates to iproute2 ss command as in :
$ ss -i dst cca2
Netid State Recv-Q Send-Q Local Address:Port
Peer Address:Port
tcp ESTAB 0 0 10.244.129.1:56984
10.244.129.2:12865
cubic wscale:6,6 rto:200 rtt:0.25/0.25 ato:40 mss:1448 cwnd:10 send
463.4Mbps rcv_rtt:1 rcv_space:29200
tcp ESTAB 0 390960 10.244.129.1:60247
10.244.129.2:50204
cubic wscale:6,6 rto:200 rtt:0.875/0.75 mss:1448 cwnd:73 ssthresh:51
send 966.4Mbps unacked:73 retrans:0/121 rcv_space:29200
Reported-by: Vytautas Valancius <valas@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-02-07 03:57:10 +04:00
|
|
|
if (srtt != 0) {
|
|
|
|
m -= (srtt >> 3); /* m is now error in rtt est */
|
|
|
|
srtt += m; /* rtt = 7/8 rtt + 1/8 new */
|
2005-04-17 02:20:36 +04:00
|
|
|
if (m < 0) {
|
|
|
|
m = -m; /* m is now abs(error) */
|
2014-02-27 02:02:48 +04:00
|
|
|
m -= (tp->mdev_us >> 2); /* similar update on mdev */
|
2005-04-17 02:20:36 +04:00
|
|
|
/* This is similar to one of Eifel findings.
|
|
|
|
* Eifel blocks mdev updates when rtt decreases.
|
|
|
|
* This solution is a bit different: we use finer gain
|
|
|
|
* for mdev in this case (alpha*beta).
|
|
|
|
* Like Eifel it also prevents growth of rto,
|
|
|
|
* but also it limits too fast rto decreases,
|
|
|
|
* happening in pure Eifel.
|
|
|
|
*/
|
|
|
|
if (m > 0)
|
|
|
|
m >>= 3;
|
|
|
|
} else {
|
2014-02-27 02:02:48 +04:00
|
|
|
m -= (tp->mdev_us >> 2); /* similar update on mdev */
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2014-02-27 02:02:48 +04:00
|
|
|
tp->mdev_us += m; /* mdev = 3/4 mdev + 1/4 new */
|
|
|
|
if (tp->mdev_us > tp->mdev_max_us) {
|
|
|
|
tp->mdev_max_us = tp->mdev_us;
|
|
|
|
if (tp->mdev_max_us > tp->rttvar_us)
|
|
|
|
tp->rttvar_us = tp->mdev_max_us;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
if (after(tp->snd_una, tp->rtt_seq)) {
|
2014-02-27 02:02:48 +04:00
|
|
|
if (tp->mdev_max_us < tp->rttvar_us)
|
|
|
|
tp->rttvar_us -= (tp->rttvar_us - tp->mdev_max_us) >> 2;
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->rtt_seq = tp->snd_nxt;
|
2014-02-27 02:02:48 +04:00
|
|
|
tp->mdev_max_us = tcp_rto_min_us(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
} else {
|
|
|
|
/* no previous measure. */
|
tcp: remove 1ms offset in srtt computation
TCP pacing depends on an accurate srtt estimation.
Current srtt estimation is using jiffie resolution,
and has an artificial offset of at least 1 ms, which can produce
slowdowns when FQ/pacing is used, especially in DC world,
where typical rtt is below 1 ms.
We are planning a switch to usec resolution for linux-3.15,
but in the meantime, this patch removes the 1 ms offset.
All we need is to have tp->srtt minimal value of 1 to differentiate
the case of srtt being initialized or not, not 8.
The problematic behavior was observed on a 40Gbit testbed,
where 32 concurrent netperf were reaching 12Gbps of aggregate
speed, instead of line speed.
This patch also has the effect of reporting more accurate srtt and send
rates to iproute2 ss command as in :
$ ss -i dst cca2
Netid State Recv-Q Send-Q Local Address:Port
Peer Address:Port
tcp ESTAB 0 0 10.244.129.1:56984
10.244.129.2:12865
cubic wscale:6,6 rto:200 rtt:0.25/0.25 ato:40 mss:1448 cwnd:10 send
463.4Mbps rcv_rtt:1 rcv_space:29200
tcp ESTAB 0 390960 10.244.129.1:60247
10.244.129.2:50204
cubic wscale:6,6 rto:200 rtt:0.875/0.75 mss:1448 cwnd:73 ssthresh:51
send 966.4Mbps unacked:73 retrans:0/121 rcv_space:29200
Reported-by: Vytautas Valancius <valas@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-02-07 03:57:10 +04:00
|
|
|
srtt = m << 3; /* take the measured time to be rtt */
|
2014-02-27 02:02:48 +04:00
|
|
|
tp->mdev_us = m << 1; /* make sure rto = 3*rtt */
|
|
|
|
tp->rttvar_us = max(tp->mdev_us, tcp_rto_min_us(sk));
|
|
|
|
tp->mdev_max_us = tp->rttvar_us;
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->rtt_seq = tp->snd_nxt;
|
|
|
|
}
|
2014-02-27 02:02:48 +04:00
|
|
|
tp->srtt_us = max(1U, srtt);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
tcp: TSO packets automatic sizing
After hearing many people over past years complaining against TSO being
bursty or even buggy, we are proud to present automatic sizing of TSO
packets.
One part of the problem is that tcp_tso_should_defer() uses an heuristic
relying on upcoming ACKS instead of a timer, but more generally, having
big TSO packets makes little sense for low rates, as it tends to create
micro bursts on the network, and general consensus is to reduce the
buffering amount.
This patch introduces a per socket sk_pacing_rate, that approximates
the current sending rate, and allows us to size the TSO packets so
that we try to send one packet every ms.
This field could be set by other transports.
Patch has no impact for high speed flows, where having large TSO packets
makes sense to reach line rate.
For other flows, this helps better packet scheduling and ACK clocking.
This patch increases performance of TCP flows in lossy environments.
A new sysctl (tcp_min_tso_segs) is added, to specify the
minimal size of a TSO packet (default being 2).
A follow-up patch will provide a new packet scheduler (FQ), using
sk_pacing_rate as an input to perform optional per flow pacing.
This explains why we chose to set sk_pacing_rate to twice the current
rate, allowing 'slow start' ramp up.
sk_pacing_rate = 2 * cwnd * mss / srtt
v2: Neal Cardwell reported a suspect deferring of last two segments on
initial write of 10 MSS, I had to change tcp_tso_should_defer() to take
into account tp->xmit_size_goal_segs
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Van Jacobson <vanj@google.com>
Cc: Tom Herbert <therbert@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-27 16:46:32 +04:00
|
|
|
/* Set the sk_pacing_rate to allow proper sizing of TSO packets.
|
|
|
|
* Note: TCP stack does not yet implement pacing.
|
|
|
|
* FQ packet scheduler can be used to implement cheap but effective
|
|
|
|
* TCP pacing, to smooth the burst on large writes when packets
|
|
|
|
* in flight is significantly lower than cwnd (or rwin)
|
|
|
|
*/
|
2015-08-22 03:38:02 +03:00
|
|
|
int sysctl_tcp_pacing_ss_ratio __read_mostly = 200;
|
|
|
|
int sysctl_tcp_pacing_ca_ratio __read_mostly = 120;
|
|
|
|
|
tcp: TSO packets automatic sizing
After hearing many people over past years complaining against TSO being
bursty or even buggy, we are proud to present automatic sizing of TSO
packets.
One part of the problem is that tcp_tso_should_defer() uses an heuristic
relying on upcoming ACKS instead of a timer, but more generally, having
big TSO packets makes little sense for low rates, as it tends to create
micro bursts on the network, and general consensus is to reduce the
buffering amount.
This patch introduces a per socket sk_pacing_rate, that approximates
the current sending rate, and allows us to size the TSO packets so
that we try to send one packet every ms.
This field could be set by other transports.
Patch has no impact for high speed flows, where having large TSO packets
makes sense to reach line rate.
For other flows, this helps better packet scheduling and ACK clocking.
This patch increases performance of TCP flows in lossy environments.
A new sysctl (tcp_min_tso_segs) is added, to specify the
minimal size of a TSO packet (default being 2).
A follow-up patch will provide a new packet scheduler (FQ), using
sk_pacing_rate as an input to perform optional per flow pacing.
This explains why we chose to set sk_pacing_rate to twice the current
rate, allowing 'slow start' ramp up.
sk_pacing_rate = 2 * cwnd * mss / srtt
v2: Neal Cardwell reported a suspect deferring of last two segments on
initial write of 10 MSS, I had to change tcp_tso_should_defer() to take
into account tp->xmit_size_goal_segs
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Van Jacobson <vanj@google.com>
Cc: Tom Herbert <therbert@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-27 16:46:32 +04:00
|
|
|
static void tcp_update_pacing_rate(struct sock *sk)
|
|
|
|
{
|
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u64 rate;
|
|
|
|
|
|
|
|
/* set sk_pacing_rate to 200 % of current rate (mss * cwnd / srtt) */
|
2015-08-22 03:38:02 +03:00
|
|
|
rate = (u64)tp->mss_cache * ((USEC_PER_SEC / 100) << 3);
|
|
|
|
|
|
|
|
/* current rate is (cwnd * mss) / srtt
|
|
|
|
* In Slow Start [1], set sk_pacing_rate to 200 % the current rate.
|
|
|
|
* In Congestion Avoidance phase, set it to 120 % the current rate.
|
|
|
|
*
|
|
|
|
* [1] : Normal Slow Start condition is (tp->snd_cwnd < tp->snd_ssthresh)
|
|
|
|
* If snd_cwnd >= (tp->snd_ssthresh / 2), we are approaching
|
|
|
|
* end of slow start and should slow down.
|
|
|
|
*/
|
|
|
|
if (tp->snd_cwnd < tp->snd_ssthresh / 2)
|
|
|
|
rate *= sysctl_tcp_pacing_ss_ratio;
|
|
|
|
else
|
|
|
|
rate *= sysctl_tcp_pacing_ca_ratio;
|
tcp: TSO packets automatic sizing
After hearing many people over past years complaining against TSO being
bursty or even buggy, we are proud to present automatic sizing of TSO
packets.
One part of the problem is that tcp_tso_should_defer() uses an heuristic
relying on upcoming ACKS instead of a timer, but more generally, having
big TSO packets makes little sense for low rates, as it tends to create
micro bursts on the network, and general consensus is to reduce the
buffering amount.
This patch introduces a per socket sk_pacing_rate, that approximates
the current sending rate, and allows us to size the TSO packets so
that we try to send one packet every ms.
This field could be set by other transports.
Patch has no impact for high speed flows, where having large TSO packets
makes sense to reach line rate.
For other flows, this helps better packet scheduling and ACK clocking.
This patch increases performance of TCP flows in lossy environments.
A new sysctl (tcp_min_tso_segs) is added, to specify the
minimal size of a TSO packet (default being 2).
A follow-up patch will provide a new packet scheduler (FQ), using
sk_pacing_rate as an input to perform optional per flow pacing.
This explains why we chose to set sk_pacing_rate to twice the current
rate, allowing 'slow start' ramp up.
sk_pacing_rate = 2 * cwnd * mss / srtt
v2: Neal Cardwell reported a suspect deferring of last two segments on
initial write of 10 MSS, I had to change tcp_tso_should_defer() to take
into account tp->xmit_size_goal_segs
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Van Jacobson <vanj@google.com>
Cc: Tom Herbert <therbert@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-27 16:46:32 +04:00
|
|
|
|
|
|
|
rate *= max(tp->snd_cwnd, tp->packets_out);
|
|
|
|
|
2014-02-27 02:02:48 +04:00
|
|
|
if (likely(tp->srtt_us))
|
|
|
|
do_div(rate, tp->srtt_us);
|
tcp: TSO packets automatic sizing
After hearing many people over past years complaining against TSO being
bursty or even buggy, we are proud to present automatic sizing of TSO
packets.
One part of the problem is that tcp_tso_should_defer() uses an heuristic
relying on upcoming ACKS instead of a timer, but more generally, having
big TSO packets makes little sense for low rates, as it tends to create
micro bursts on the network, and general consensus is to reduce the
buffering amount.
This patch introduces a per socket sk_pacing_rate, that approximates
the current sending rate, and allows us to size the TSO packets so
that we try to send one packet every ms.
This field could be set by other transports.
Patch has no impact for high speed flows, where having large TSO packets
makes sense to reach line rate.
For other flows, this helps better packet scheduling and ACK clocking.
This patch increases performance of TCP flows in lossy environments.
A new sysctl (tcp_min_tso_segs) is added, to specify the
minimal size of a TSO packet (default being 2).
A follow-up patch will provide a new packet scheduler (FQ), using
sk_pacing_rate as an input to perform optional per flow pacing.
This explains why we chose to set sk_pacing_rate to twice the current
rate, allowing 'slow start' ramp up.
sk_pacing_rate = 2 * cwnd * mss / srtt
v2: Neal Cardwell reported a suspect deferring of last two segments on
initial write of 10 MSS, I had to change tcp_tso_should_defer() to take
into account tp->xmit_size_goal_segs
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Van Jacobson <vanj@google.com>
Cc: Tom Herbert <therbert@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-27 16:46:32 +04:00
|
|
|
|
2013-10-10 04:14:52 +04:00
|
|
|
/* ACCESS_ONCE() is needed because sch_fq fetches sk_pacing_rate
|
|
|
|
* without any lock. We want to make sure compiler wont store
|
|
|
|
* intermediate values in this location.
|
|
|
|
*/
|
|
|
|
ACCESS_ONCE(sk->sk_pacing_rate) = min_t(u64, rate,
|
|
|
|
sk->sk_max_pacing_rate);
|
tcp: TSO packets automatic sizing
After hearing many people over past years complaining against TSO being
bursty or even buggy, we are proud to present automatic sizing of TSO
packets.
One part of the problem is that tcp_tso_should_defer() uses an heuristic
relying on upcoming ACKS instead of a timer, but more generally, having
big TSO packets makes little sense for low rates, as it tends to create
micro bursts on the network, and general consensus is to reduce the
buffering amount.
This patch introduces a per socket sk_pacing_rate, that approximates
the current sending rate, and allows us to size the TSO packets so
that we try to send one packet every ms.
This field could be set by other transports.
Patch has no impact for high speed flows, where having large TSO packets
makes sense to reach line rate.
For other flows, this helps better packet scheduling and ACK clocking.
This patch increases performance of TCP flows in lossy environments.
A new sysctl (tcp_min_tso_segs) is added, to specify the
minimal size of a TSO packet (default being 2).
A follow-up patch will provide a new packet scheduler (FQ), using
sk_pacing_rate as an input to perform optional per flow pacing.
This explains why we chose to set sk_pacing_rate to twice the current
rate, allowing 'slow start' ramp up.
sk_pacing_rate = 2 * cwnd * mss / srtt
v2: Neal Cardwell reported a suspect deferring of last two segments on
initial write of 10 MSS, I had to change tcp_tso_should_defer() to take
into account tp->xmit_size_goal_segs
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Cc: Van Jacobson <vanj@google.com>
Cc: Tom Herbert <therbert@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-08-27 16:46:32 +04:00
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Calculate rto without backoff. This is the second half of Van Jacobson's
|
|
|
|
* routine referred to above.
|
|
|
|
*/
|
2013-12-29 23:39:51 +04:00
|
|
|
static void tcp_set_rto(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 07:10:42 +04:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Old crap is replaced with new one. 8)
|
|
|
|
*
|
|
|
|
* More seriously:
|
|
|
|
* 1. If rtt variance happened to be less 50msec, it is hallucination.
|
|
|
|
* It cannot be less due to utterly erratic ACK generation made
|
|
|
|
* at least by solaris and freebsd. "Erratic ACKs" has _nothing_
|
|
|
|
* to do with delayed acks, because at cwnd>2 true delack timeout
|
|
|
|
* is invisible. Actually, Linux-2.4 also generates erratic
|
2005-11-11 04:13:47 +03:00
|
|
|
* ACKs in some circumstances.
|
2005-04-17 02:20:36 +04:00
|
|
|
*/
|
Revert Backoff [v3]: Revert RTO on ICMP destination unreachable
Here, an ICMP host/network unreachable message, whose payload fits to
TCP's SND.UNA, is taken as an indication that the RTO retransmission has
not been lost due to congestion, but because of a route failure
somewhere along the path.
With true congestion, a router won't trigger such a message and the
patched TCP will operate as standard TCP.
This patch reverts one RTO backoff, if an ICMP host/network unreachable
message, whose payload fits to TCP's SND.UNA, arrives.
Based on the new RTO, the retransmission timer is reset to reflect the
remaining time, or - if the revert clocked out the timer - a retransmission
is sent out immediately.
Backoffs are only reverted, if TCP is in RTO loss recovery, i.e. if
there have been retransmissions and reversible backoffs, already.
Changes from v2:
1) Renaming of skb in tcp_v4_err() moved to another patch.
2) Reintroduced tcp_bound_rto() and __tcp_set_rto().
3) Fixed code comments.
Signed-off-by: Damian Lukowski <damian@tvk.rwth-aachen.de>
Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-08-26 04:16:31 +04:00
|
|
|
inet_csk(sk)->icsk_rto = __tcp_set_rto(tp);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* 2. Fixups made earlier cannot be right.
|
|
|
|
* If we do not estimate RTO correctly without them,
|
|
|
|
* all the algo is pure shit and should be replaced
|
2005-11-11 04:13:47 +03:00
|
|
|
* with correct one. It is exactly, which we pretend to do.
|
2005-04-17 02:20:36 +04:00
|
|
|
*/
|
|
|
|
|
2008-12-06 09:43:08 +03:00
|
|
|
/* NOTE: clamping at TCP_RTO_MIN is not required, current algo
|
|
|
|
* guarantees that rto is higher.
|
|
|
|
*/
|
Revert Backoff [v3]: Revert RTO on ICMP destination unreachable
Here, an ICMP host/network unreachable message, whose payload fits to
TCP's SND.UNA, is taken as an indication that the RTO retransmission has
not been lost due to congestion, but because of a route failure
somewhere along the path.
With true congestion, a router won't trigger such a message and the
patched TCP will operate as standard TCP.
This patch reverts one RTO backoff, if an ICMP host/network unreachable
message, whose payload fits to TCP's SND.UNA, arrives.
Based on the new RTO, the retransmission timer is reset to reflect the
remaining time, or - if the revert clocked out the timer - a retransmission
is sent out immediately.
Backoffs are only reverted, if TCP is in RTO loss recovery, i.e. if
there have been retransmissions and reversible backoffs, already.
Changes from v2:
1) Renaming of skb in tcp_v4_err() moved to another patch.
2) Reintroduced tcp_bound_rto() and __tcp_set_rto().
3) Fixed code comments.
Signed-off-by: Damian Lukowski <damian@tvk.rwth-aachen.de>
Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-08-26 04:16:31 +04:00
|
|
|
tcp_bound_rto(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2011-10-21 13:22:42 +04:00
|
|
|
__u32 tcp_init_cwnd(const struct tcp_sock *tp, const struct dst_entry *dst)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
__u32 cwnd = (dst ? dst_metric(dst, RTAX_INITCWND) : 0);
|
|
|
|
|
2010-08-29 23:23:12 +04:00
|
|
|
if (!cwnd)
|
2011-02-03 04:05:11 +03:00
|
|
|
cwnd = TCP_INIT_CWND;
|
2005-04-17 02:20:36 +04:00
|
|
|
return min_t(__u32, cwnd, tp->snd_cwnd_clamp);
|
|
|
|
}
|
|
|
|
|
2007-08-09 16:14:46 +04:00
|
|
|
/*
|
|
|
|
* Packet counting of FACK is based on in-order assumptions, therefore TCP
|
|
|
|
* disables it when reordering is detected
|
|
|
|
*/
|
2012-07-10 03:07:30 +04:00
|
|
|
void tcp_disable_fack(struct tcp_sock *tp)
|
2007-08-09 16:14:46 +04:00
|
|
|
{
|
2007-11-16 06:39:31 +03:00
|
|
|
/* RFC3517 uses different metric in lost marker => reset on change */
|
|
|
|
if (tcp_is_fack(tp))
|
|
|
|
tp->lost_skb_hint = NULL;
|
2011-12-20 17:23:24 +04:00
|
|
|
tp->rx_opt.sack_ok &= ~TCP_FACK_ENABLED;
|
2007-08-09 16:14:46 +04:00
|
|
|
}
|
|
|
|
|
2007-10-26 10:03:52 +04:00
|
|
|
/* Take a notice that peer is sending D-SACKs */
|
2007-08-09 16:14:46 +04:00
|
|
|
static void tcp_dsack_seen(struct tcp_sock *tp)
|
|
|
|
{
|
2011-12-20 17:23:24 +04:00
|
|
|
tp->rx_opt.sack_ok |= TCP_DSACK_SEEN;
|
2007-08-09 16:14:46 +04:00
|
|
|
}
|
|
|
|
|
2005-08-10 11:03:31 +04:00
|
|
|
static void tcp_update_reordering(struct sock *sk, const int metric,
|
|
|
|
const int ts)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 11:03:31 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (metric > tp->reordering) {
|
2008-07-03 12:05:41 +04:00
|
|
|
int mib_idx;
|
|
|
|
|
2014-10-28 07:45:24 +03:00
|
|
|
tp->reordering = min(sysctl_tcp_max_reordering, metric);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* This exciting event is worth to be remembered. 8) */
|
|
|
|
if (ts)
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPTSREORDER;
|
2007-08-09 16:14:46 +04:00
|
|
|
else if (tcp_is_reno(tp))
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPRENOREORDER;
|
2007-08-09 16:14:46 +04:00
|
|
|
else if (tcp_is_fack(tp))
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPFACKREORDER;
|
2005-04-17 02:20:36 +04:00
|
|
|
else
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPSACKREORDER;
|
|
|
|
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), mib_idx);
|
2005-04-17 02:20:36 +04:00
|
|
|
#if FASTRETRANS_DEBUG > 1
|
2012-05-15 18:11:54 +04:00
|
|
|
pr_debug("Disorder%d %d %u f%u s%u rr%d\n",
|
|
|
|
tp->rx_opt.sack_ok, inet_csk(sk)->icsk_ca_state,
|
|
|
|
tp->reordering,
|
|
|
|
tp->fackets_out,
|
|
|
|
tp->sacked_out,
|
|
|
|
tp->undo_marker ? tp->undo_retrans : 0);
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
2007-08-09 16:14:46 +04:00
|
|
|
tcp_disable_fack(tp);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2012-05-02 17:30:03 +04:00
|
|
|
|
|
|
|
if (metric > 0)
|
|
|
|
tcp_disable_early_retrans(tp);
|
2015-10-17 07:57:47 +03:00
|
|
|
tp->rack.reord = 1;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2008-09-21 08:20:20 +04:00
|
|
|
/* This must be called before lost_out is incremented */
|
2008-09-21 08:18:55 +04:00
|
|
|
static void tcp_verify_retransmit_hint(struct tcp_sock *tp, struct sk_buff *skb)
|
|
|
|
{
|
2015-04-03 11:17:26 +03:00
|
|
|
if (!tp->retransmit_skb_hint ||
|
2008-09-21 08:18:55 +04:00
|
|
|
before(TCP_SKB_CB(skb)->seq,
|
|
|
|
TCP_SKB_CB(tp->retransmit_skb_hint)->seq))
|
2008-09-21 08:20:20 +04:00
|
|
|
tp->retransmit_skb_hint = skb;
|
|
|
|
|
|
|
|
if (!tp->lost_out ||
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq, tp->retransmit_high))
|
|
|
|
tp->retransmit_high = TCP_SKB_CB(skb)->end_seq;
|
2008-09-21 08:18:55 +04:00
|
|
|
}
|
|
|
|
|
2008-09-21 08:19:22 +04:00
|
|
|
static void tcp_skb_mark_lost(struct tcp_sock *tp, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
if (!(TCP_SKB_CB(skb)->sacked & (TCPCB_LOST|TCPCB_SACKED_ACKED))) {
|
|
|
|
tcp_verify_retransmit_hint(tp, skb);
|
|
|
|
|
|
|
|
tp->lost_out += tcp_skb_pcount(skb);
|
|
|
|
TCP_SKB_CB(skb)->sacked |= TCPCB_LOST;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-10-17 07:57:47 +03:00
|
|
|
void tcp_skb_mark_lost_uncond_verify(struct tcp_sock *tp, struct sk_buff *skb)
|
2008-09-21 08:20:20 +04:00
|
|
|
{
|
|
|
|
tcp_verify_retransmit_hint(tp, skb);
|
|
|
|
|
|
|
|
if (!(TCP_SKB_CB(skb)->sacked & (TCPCB_LOST|TCPCB_SACKED_ACKED))) {
|
|
|
|
tp->lost_out += tcp_skb_pcount(skb);
|
|
|
|
TCP_SKB_CB(skb)->sacked |= TCPCB_LOST;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* This procedure tags the retransmission queue when SACKs arrive.
|
|
|
|
*
|
|
|
|
* We have three tag bits: SACKED(S), RETRANS(R) and LOST(L).
|
|
|
|
* Packets in queue with these bits set are counted in variables
|
|
|
|
* sacked_out, retrans_out and lost_out, correspondingly.
|
|
|
|
*
|
|
|
|
* Valid combinations are:
|
|
|
|
* Tag InFlight Description
|
|
|
|
* 0 1 - orig segment is in flight.
|
|
|
|
* S 0 - nothing flies, orig reached receiver.
|
|
|
|
* L 0 - nothing flies, orig lost by net.
|
|
|
|
* R 2 - both orig and retransmit are in flight.
|
|
|
|
* L|R 1 - orig is lost, retransmit is in flight.
|
|
|
|
* S|R 1 - orig reached receiver, retrans is still in flight.
|
|
|
|
* (L|S|R is logically valid, it could occur when L|R is sacked,
|
|
|
|
* but it is equivalent to plain S and code short-curcuits it to S.
|
|
|
|
* L|S is logically invalid, it would mean -1 packet in flight 8))
|
|
|
|
*
|
|
|
|
* These 6 states form finite state machine, controlled by the following events:
|
|
|
|
* 1. New ACK (+SACK) arrives. (tcp_sacktag_write_queue())
|
|
|
|
* 2. Retransmission. (tcp_retransmit_skb(), tcp_xmit_retransmit_queue())
|
2012-01-19 18:42:21 +04:00
|
|
|
* 3. Loss detection event of two flavors:
|
2005-04-17 02:20:36 +04:00
|
|
|
* A. Scoreboard estimator decided the packet is lost.
|
|
|
|
* A'. Reno "three dupacks" marks head of queue lost.
|
2012-01-19 18:42:21 +04:00
|
|
|
* A''. Its FACK modification, head until snd.fack is lost.
|
|
|
|
* B. SACK arrives sacking SND.NXT at the moment, when the
|
2005-04-17 02:20:36 +04:00
|
|
|
* segment was retransmitted.
|
|
|
|
* 4. D-SACK added new rule: D-SACK changes any tag to S.
|
|
|
|
*
|
|
|
|
* It is pleasant to note, that state diagram turns out to be commutative,
|
|
|
|
* so that we are allowed not to be bothered by order of our actions,
|
|
|
|
* when multiple events arrive simultaneously. (see the function below).
|
|
|
|
*
|
|
|
|
* Reordering detection.
|
|
|
|
* --------------------
|
|
|
|
* Reordering metric is maximal distance, which a packet can be displaced
|
|
|
|
* in packet stream. With SACKs we can estimate it:
|
|
|
|
*
|
|
|
|
* 1. SACK fills old hole and the corresponding segment was not
|
|
|
|
* ever retransmitted -> reordering. Alas, we cannot use it
|
|
|
|
* when segment was retransmitted.
|
|
|
|
* 2. The last flaw is solved with D-SACK. D-SACK arrives
|
|
|
|
* for retransmitted and already SACKed segment -> reordering..
|
|
|
|
* Both of these heuristics are not used in Loss state, when we cannot
|
|
|
|
* account for retransmits accurately.
|
2007-08-25 09:54:44 +04:00
|
|
|
*
|
|
|
|
* SACK block validation.
|
|
|
|
* ----------------------
|
|
|
|
*
|
|
|
|
* SACK block range validation checks that the received SACK block fits to
|
|
|
|
* the expected sequence limits, i.e., it is between SND.UNA and SND.NXT.
|
|
|
|
* Note that SND.UNA is not included to the range though being valid because
|
2007-10-02 02:28:17 +04:00
|
|
|
* it means that the receiver is rather inconsistent with itself reporting
|
|
|
|
* SACK reneging when it should advance SND.UNA. Such SACK block this is
|
|
|
|
* perfectly valid, however, in light of RFC2018 which explicitly states
|
|
|
|
* that "SACK block MUST reflect the newest segment. Even if the newest
|
|
|
|
* segment is going to be discarded ...", not that it looks very clever
|
|
|
|
* in case of head skb. Due to potentional receiver driven attacks, we
|
|
|
|
* choose to avoid immediate execution of a walk in write queue due to
|
|
|
|
* reneging and defer head skb's loss recovery to standard loss recovery
|
|
|
|
* procedure that will eventually trigger (nothing forbids us doing this).
|
2007-08-25 09:54:44 +04:00
|
|
|
*
|
|
|
|
* Implements also blockage to start_seq wrap-around. Problem lies in the
|
|
|
|
* fact that though start_seq (s) is before end_seq (i.e., not reversed),
|
|
|
|
* there's no guarantee that it will be before snd_nxt (n). The problem
|
|
|
|
* happens when start_seq resides between end_seq wrap (e_w) and snd_nxt
|
|
|
|
* wrap (s_w):
|
|
|
|
*
|
|
|
|
* <- outs wnd -> <- wrapzone ->
|
|
|
|
* u e n u_w e_w s n_w
|
|
|
|
* | | | | | | |
|
|
|
|
* |<------------+------+----- TCP seqno space --------------+---------->|
|
|
|
|
* ...-- <2^31 ->| |<--------...
|
|
|
|
* ...---- >2^31 ------>| |<--------...
|
|
|
|
*
|
|
|
|
* Current code wouldn't be vulnerable but it's better still to discard such
|
|
|
|
* crazy SACK blocks. Doing this check for start_seq alone closes somewhat
|
|
|
|
* similar case (end_seq after snd_nxt wrap) as earlier reversed check in
|
|
|
|
* snd_nxt wrap -> snd_una region will then become "well defined", i.e.,
|
|
|
|
* equal to the ideal case (infinite seqno space without wrap caused issues).
|
|
|
|
*
|
|
|
|
* With D-SACK the lower bound is extended to cover sequence space below
|
|
|
|
* SND.UNA down to undo_marker, which is the last point of interest. Yet
|
2007-10-26 10:03:52 +04:00
|
|
|
* again, D-SACK block must not to go across snd_una (for the same reason as
|
2007-08-25 09:54:44 +04:00
|
|
|
* for the normal SACK blocks, explained above). But there all simplicity
|
|
|
|
* ends, TCP might receive valid D-SACKs below that. As long as they reside
|
|
|
|
* fully below undo_marker they do not affect behavior in anyway and can
|
|
|
|
* therefore be safely ignored. In rare cases (which are more or less
|
|
|
|
* theoretical ones), the D-SACK will nicely cross that boundary due to skb
|
|
|
|
* fragmentation and packet reordering past skb's retransmission. To consider
|
|
|
|
* them correctly, the acceptable range must be extended even more though
|
|
|
|
* the exact amount is rather hard to quantify. However, tp->max_window can
|
|
|
|
* be used as an exaggerated estimate.
|
2005-04-17 02:20:36 +04:00
|
|
|
*/
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_is_sackblock_valid(struct tcp_sock *tp, bool is_dsack,
|
|
|
|
u32 start_seq, u32 end_seq)
|
2007-08-25 09:54:44 +04:00
|
|
|
{
|
|
|
|
/* Too far in future, or reversed (interpretation is ambiguous) */
|
|
|
|
if (after(end_seq, tp->snd_nxt) || !before(start_seq, end_seq))
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2007-08-25 09:54:44 +04:00
|
|
|
|
|
|
|
/* Nasty start_seq wrap-around check (see comments above) */
|
|
|
|
if (!before(start_seq, tp->snd_nxt))
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2007-08-25 09:54:44 +04:00
|
|
|
|
2007-10-26 10:03:52 +04:00
|
|
|
/* In outstanding window? ...This is valid exit for D-SACKs too.
|
2007-08-25 09:54:44 +04:00
|
|
|
* start_seq == snd_una is non-sensical (see comments above)
|
|
|
|
*/
|
|
|
|
if (after(start_seq, tp->snd_una))
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2007-08-25 09:54:44 +04:00
|
|
|
|
|
|
|
if (!is_dsack || !tp->undo_marker)
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2007-08-25 09:54:44 +04:00
|
|
|
|
|
|
|
/* ...Then it's D-SACK, and must reside below snd_una completely */
|
2011-09-19 06:37:34 +04:00
|
|
|
if (after(end_seq, tp->snd_una))
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2007-08-25 09:54:44 +04:00
|
|
|
|
|
|
|
if (!before(start_seq, tp->undo_marker))
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2007-08-25 09:54:44 +04:00
|
|
|
|
|
|
|
/* Too old */
|
|
|
|
if (!after(end_seq, tp->undo_marker))
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2007-08-25 09:54:44 +04:00
|
|
|
|
|
|
|
/* Undo_marker boundary crossing (overestimates a lot). Known already:
|
|
|
|
* start_seq < undo_marker and end_seq >= undo_marker.
|
|
|
|
*/
|
|
|
|
return !before(start_seq, end_seq - tp->max_window);
|
|
|
|
}
|
|
|
|
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_check_dsack(struct sock *sk, const struct sk_buff *ack_skb,
|
|
|
|
struct tcp_sack_block_wire *sp, int num_sacks,
|
|
|
|
u32 prior_snd_una)
|
2007-06-19 09:43:06 +04:00
|
|
|
{
|
2008-07-17 07:29:51 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2008-05-03 03:26:16 +04:00
|
|
|
u32 start_seq_0 = get_unaligned_be32(&sp[0].start_seq);
|
|
|
|
u32 end_seq_0 = get_unaligned_be32(&sp[0].end_seq);
|
2012-05-17 03:15:34 +04:00
|
|
|
bool dup_sack = false;
|
2007-06-19 09:43:06 +04:00
|
|
|
|
|
|
|
if (before(start_seq_0, TCP_SKB_CB(ack_skb)->ack_seq)) {
|
2012-05-17 03:15:34 +04:00
|
|
|
dup_sack = true;
|
2007-08-09 16:14:46 +04:00
|
|
|
tcp_dsack_seen(tp);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPDSACKRECV);
|
2007-06-19 09:43:06 +04:00
|
|
|
} else if (num_sacks > 1) {
|
2008-05-03 03:26:16 +04:00
|
|
|
u32 end_seq_1 = get_unaligned_be32(&sp[1].end_seq);
|
|
|
|
u32 start_seq_1 = get_unaligned_be32(&sp[1].start_seq);
|
2007-06-19 09:43:06 +04:00
|
|
|
|
|
|
|
if (!after(end_seq_0, end_seq_1) &&
|
|
|
|
!before(start_seq_0, start_seq_1)) {
|
2012-05-17 03:15:34 +04:00
|
|
|
dup_sack = true;
|
2007-08-09 16:14:46 +04:00
|
|
|
tcp_dsack_seen(tp);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk),
|
2008-07-17 07:31:16 +04:00
|
|
|
LINUX_MIB_TCPDSACKOFORECV);
|
2007-06-19 09:43:06 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* D-SACK for already forgotten data... Do dumb counting. */
|
2014-07-02 23:07:16 +04:00
|
|
|
if (dup_sack && tp->undo_marker && tp->undo_retrans > 0 &&
|
2007-06-19 09:43:06 +04:00
|
|
|
!after(end_seq_0, prior_snd_una) &&
|
|
|
|
after(end_seq_0, tp->undo_marker))
|
|
|
|
tp->undo_retrans--;
|
|
|
|
|
|
|
|
return dup_sack;
|
|
|
|
}
|
|
|
|
|
2008-12-06 09:42:22 +03:00
|
|
|
struct tcp_sacktag_state {
|
2014-02-27 02:02:48 +04:00
|
|
|
int reord;
|
|
|
|
int fack_count;
|
2015-05-01 02:10:58 +03:00
|
|
|
/* Timestamps for earliest and latest never-retransmitted segment
|
|
|
|
* that was SACKed. RTO needs the earliest RTT to stay conservative,
|
|
|
|
* but congestion control should still get an accurate delay signal.
|
|
|
|
*/
|
|
|
|
struct skb_mstamp first_sackt;
|
|
|
|
struct skb_mstamp last_sackt;
|
2014-02-27 02:02:48 +04:00
|
|
|
int flag;
|
2008-12-06 09:42:22 +03:00
|
|
|
};
|
|
|
|
|
2007-10-12 04:34:25 +04:00
|
|
|
/* Check if skb is fully within the SACK block. In presence of GSO skbs,
|
|
|
|
* the incoming SACK may not exactly match but we can find smaller MSS
|
|
|
|
* aligned portion of it that matches. Therefore we might need to fragment
|
|
|
|
* which may fail and creates some hassle (caller must handle error case
|
|
|
|
* returns).
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
*
|
|
|
|
* FIXME: this could be merged to shift decision code
|
2007-10-12 04:34:25 +04:00
|
|
|
*/
|
2007-10-26 14:57:36 +04:00
|
|
|
static int tcp_match_skb_to_sack(struct sock *sk, struct sk_buff *skb,
|
2012-05-17 03:15:34 +04:00
|
|
|
u32 start_seq, u32 end_seq)
|
2007-10-12 04:34:25 +04:00
|
|
|
{
|
2012-05-17 03:15:34 +04:00
|
|
|
int err;
|
|
|
|
bool in_sack;
|
2007-10-12 04:34:25 +04:00
|
|
|
unsigned int pkt_len;
|
2008-11-25 08:13:50 +03:00
|
|
|
unsigned int mss;
|
2007-10-12 04:34:25 +04:00
|
|
|
|
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq) &&
|
|
|
|
!before(end_seq, TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
|
|
|
if (tcp_skb_pcount(skb) > 1 && !in_sack &&
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq, start_seq)) {
|
2008-11-25 08:13:50 +03:00
|
|
|
mss = tcp_skb_mss(skb);
|
2007-10-12 04:34:25 +04:00
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq);
|
|
|
|
|
2008-11-25 08:13:50 +03:00
|
|
|
if (!in_sack) {
|
2007-10-12 04:34:25 +04:00
|
|
|
pkt_len = start_seq - TCP_SKB_CB(skb)->seq;
|
2008-11-25 08:13:50 +03:00
|
|
|
if (pkt_len < mss)
|
|
|
|
pkt_len = mss;
|
|
|
|
} else {
|
2007-10-12 04:34:25 +04:00
|
|
|
pkt_len = end_seq - TCP_SKB_CB(skb)->seq;
|
2008-11-25 08:13:50 +03:00
|
|
|
if (pkt_len < mss)
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Round if necessary so that SACKs cover only full MSSes
|
|
|
|
* and/or the remaining small portion (if present)
|
|
|
|
*/
|
|
|
|
if (pkt_len > mss) {
|
|
|
|
unsigned int new_len = (pkt_len / mss) * mss;
|
|
|
|
if (!in_sack && new_len < pkt_len) {
|
|
|
|
new_len += mss;
|
2014-06-19 05:15:03 +04:00
|
|
|
if (new_len >= skb->len)
|
2008-11-25 08:13:50 +03:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
pkt_len = new_len;
|
|
|
|
}
|
2014-06-06 18:32:37 +04:00
|
|
|
err = tcp_fragment(sk, skb, pkt_len, mss, GFP_ATOMIC);
|
2007-10-12 04:34:25 +04:00
|
|
|
if (err < 0)
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
return in_sack;
|
|
|
|
}
|
|
|
|
|
2012-02-12 22:37:09 +04:00
|
|
|
/* Mark the given newly-SACKed range as such, adjusting counters and hints. */
|
|
|
|
static u8 tcp_sacktag_one(struct sock *sk,
|
|
|
|
struct tcp_sacktag_state *state, u8 sacked,
|
|
|
|
u32 start_seq, u32 end_seq,
|
2014-02-27 02:02:48 +04:00
|
|
|
int dup_sack, int pcount,
|
|
|
|
const struct skb_mstamp *xmit_time)
|
2007-11-16 06:44:56 +03:00
|
|
|
{
|
2007-12-02 01:48:06 +03:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2008-12-06 09:42:22 +03:00
|
|
|
int fack_count = state->fack_count;
|
2007-11-16 06:44:56 +03:00
|
|
|
|
|
|
|
/* Account D-SACK for retransmitted packet. */
|
|
|
|
if (dup_sack && (sacked & TCPCB_RETRANS)) {
|
2014-07-02 23:07:16 +04:00
|
|
|
if (tp->undo_marker && tp->undo_retrans > 0 &&
|
2012-02-12 22:37:09 +04:00
|
|
|
after(end_seq, tp->undo_marker))
|
2007-11-16 06:44:56 +03:00
|
|
|
tp->undo_retrans--;
|
2007-12-02 01:47:58 +03:00
|
|
|
if (sacked & TCPCB_SACKED_ACKED)
|
2008-12-06 09:42:22 +03:00
|
|
|
state->reord = min(fack_count, state->reord);
|
2007-11-16 06:44:56 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Nothing to do; acked frame is about to be dropped (was ACKed). */
|
2012-02-12 22:37:09 +04:00
|
|
|
if (!after(end_seq, tp->snd_una))
|
2008-12-06 09:42:22 +03:00
|
|
|
return sacked;
|
2007-11-16 06:44:56 +03:00
|
|
|
|
|
|
|
if (!(sacked & TCPCB_SACKED_ACKED)) {
|
2015-10-17 07:57:46 +03:00
|
|
|
tcp_rack_advance(tp, xmit_time, sacked);
|
|
|
|
|
2007-11-16 06:44:56 +03:00
|
|
|
if (sacked & TCPCB_SACKED_RETRANS) {
|
|
|
|
/* If the segment is not tagged as lost,
|
|
|
|
* we do not clear RETRANS, believing
|
|
|
|
* that retransmission is still in flight.
|
|
|
|
*/
|
|
|
|
if (sacked & TCPCB_LOST) {
|
2008-12-06 09:42:22 +03:00
|
|
|
sacked &= ~(TCPCB_LOST|TCPCB_SACKED_RETRANS);
|
2008-11-25 08:14:43 +03:00
|
|
|
tp->lost_out -= pcount;
|
|
|
|
tp->retrans_out -= pcount;
|
2007-11-16 06:44:56 +03:00
|
|
|
}
|
|
|
|
} else {
|
|
|
|
if (!(sacked & TCPCB_RETRANS)) {
|
|
|
|
/* New sack for not retransmitted frame,
|
|
|
|
* which was in hole. It is reordering.
|
|
|
|
*/
|
2012-02-12 22:37:09 +04:00
|
|
|
if (before(start_seq,
|
2007-11-16 06:44:56 +03:00
|
|
|
tcp_highest_sack_seq(tp)))
|
2008-12-06 09:42:22 +03:00
|
|
|
state->reord = min(fack_count,
|
|
|
|
state->reord);
|
2013-03-20 17:33:00 +04:00
|
|
|
if (!after(end_seq, tp->high_seq))
|
|
|
|
state->flag |= FLAG_ORIG_SACK_ACKED;
|
2015-05-01 02:10:58 +03:00
|
|
|
if (state->first_sackt.v64 == 0)
|
|
|
|
state->first_sackt = *xmit_time;
|
|
|
|
state->last_sackt = *xmit_time;
|
2007-11-16 06:44:56 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
if (sacked & TCPCB_LOST) {
|
2008-12-06 09:42:22 +03:00
|
|
|
sacked &= ~TCPCB_LOST;
|
2008-11-25 08:14:43 +03:00
|
|
|
tp->lost_out -= pcount;
|
2007-11-16 06:44:56 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-12-06 09:42:22 +03:00
|
|
|
sacked |= TCPCB_SACKED_ACKED;
|
|
|
|
state->flag |= FLAG_DATA_SACKED;
|
2008-11-25 08:14:43 +03:00
|
|
|
tp->sacked_out += pcount;
|
2016-02-02 21:33:06 +03:00
|
|
|
tp->delivered += pcount; /* Out-of-order packets delivered */
|
2007-11-16 06:44:56 +03:00
|
|
|
|
2008-11-25 08:14:43 +03:00
|
|
|
fack_count += pcount;
|
2007-11-16 06:44:56 +03:00
|
|
|
|
|
|
|
/* Lost marker hint past SACKed? Tweak RFC3517 cnt */
|
2015-04-03 11:17:27 +03:00
|
|
|
if (!tcp_is_fack(tp) && tp->lost_skb_hint &&
|
2012-02-12 22:37:09 +04:00
|
|
|
before(start_seq, TCP_SKB_CB(tp->lost_skb_hint)->seq))
|
2008-11-25 08:14:43 +03:00
|
|
|
tp->lost_cnt_hint += pcount;
|
2007-11-16 06:44:56 +03:00
|
|
|
|
|
|
|
if (fack_count > tp->fackets_out)
|
|
|
|
tp->fackets_out = fack_count;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* D-SACK. We can detect redundant retransmission in S|R and plain R
|
|
|
|
* frames and clear it. undo_retrans is decreased above, L|R frames
|
|
|
|
* are accounted above as well.
|
|
|
|
*/
|
2008-12-06 09:42:22 +03:00
|
|
|
if (dup_sack && (sacked & TCPCB_SACKED_RETRANS)) {
|
|
|
|
sacked &= ~TCPCB_SACKED_RETRANS;
|
2008-11-25 08:14:43 +03:00
|
|
|
tp->retrans_out -= pcount;
|
2007-11-16 06:44:56 +03:00
|
|
|
}
|
|
|
|
|
2008-12-06 09:42:22 +03:00
|
|
|
return sacked;
|
2007-11-16 06:44:56 +03:00
|
|
|
}
|
|
|
|
|
2012-02-12 22:37:10 +04:00
|
|
|
/* Shift newly-SACKed bytes from this skb to the immediately previous
|
|
|
|
* already-SACKed sk_buff. Mark the newly-SACKed bytes as such.
|
|
|
|
*/
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_shifted_skb(struct sock *sk, struct sk_buff *skb,
|
|
|
|
struct tcp_sacktag_state *state,
|
|
|
|
unsigned int pcount, int shifted, int mss,
|
|
|
|
bool dup_sack)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2008-12-06 09:42:41 +03:00
|
|
|
struct sk_buff *prev = tcp_write_queue_prev(sk, skb);
|
2012-02-12 22:37:10 +04:00
|
|
|
u32 start_seq = TCP_SKB_CB(skb)->seq; /* start of newly-SACKed */
|
|
|
|
u32 end_seq = start_seq + shifted; /* end of newly-SACKed */
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
|
|
|
|
BUG_ON(!pcount);
|
|
|
|
|
2012-02-26 14:06:19 +04:00
|
|
|
/* Adjust counters and hints for the newly sacked sequence
|
|
|
|
* range but discard the return value since prev is already
|
|
|
|
* marked. We must tag the range first because the seq
|
|
|
|
* advancement below implicitly advances
|
|
|
|
* tcp_highest_sack_seq() when skb is highest_sack.
|
|
|
|
*/
|
|
|
|
tcp_sacktag_one(sk, state, TCP_SKB_CB(skb)->sacked,
|
2013-07-23 03:20:47 +04:00
|
|
|
start_seq, end_seq, dup_sack, pcount,
|
2014-02-27 02:02:48 +04:00
|
|
|
&skb->skb_mstamp);
|
2012-02-26 14:06:19 +04:00
|
|
|
|
|
|
|
if (skb == tp->lost_skb_hint)
|
2012-02-14 00:22:08 +04:00
|
|
|
tp->lost_cnt_hint += pcount;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
TCP_SKB_CB(prev)->end_seq += shifted;
|
|
|
|
TCP_SKB_CB(skb)->seq += shifted;
|
|
|
|
|
2014-09-24 15:11:22 +04:00
|
|
|
tcp_skb_pcount_add(prev, pcount);
|
|
|
|
BUG_ON(tcp_skb_pcount(skb) < pcount);
|
|
|
|
tcp_skb_pcount_add(skb, -pcount);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
|
|
|
|
/* When we're adding to gso_segs == 1, gso_size will be zero,
|
|
|
|
* in theory this shouldn't be necessary but as long as DSACK
|
|
|
|
* code can come after this skb later on it's better to keep
|
|
|
|
* setting gso_size to something.
|
|
|
|
*/
|
2015-06-11 19:15:18 +03:00
|
|
|
if (!TCP_SKB_CB(prev)->tcp_gso_size)
|
|
|
|
TCP_SKB_CB(prev)->tcp_gso_size = mss;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
|
|
|
|
/* CHECKME: To clear or not to clear? Mimics normal skb currently */
|
2015-06-11 19:15:16 +03:00
|
|
|
if (tcp_skb_pcount(skb) <= 1)
|
2015-06-11 19:15:18 +03:00
|
|
|
TCP_SKB_CB(skb)->tcp_gso_size = 0;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
|
|
|
|
/* Difference in this won't matter, both ACKed by the same cumul. ACK */
|
|
|
|
TCP_SKB_CB(prev)->sacked |= (TCP_SKB_CB(skb)->sacked & TCPCB_EVER_RETRANS);
|
|
|
|
|
|
|
|
if (skb->len > 0) {
|
|
|
|
BUG_ON(!tcp_skb_pcount(skb));
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_SACKSHIFTED);
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Whole SKB was eaten :-) */
|
|
|
|
|
2008-11-25 08:26:56 +03:00
|
|
|
if (skb == tp->retransmit_skb_hint)
|
|
|
|
tp->retransmit_skb_hint = prev;
|
|
|
|
if (skb == tp->lost_skb_hint) {
|
|
|
|
tp->lost_skb_hint = prev;
|
|
|
|
tp->lost_cnt_hint -= tcp_skb_pcount(prev);
|
|
|
|
}
|
|
|
|
|
tcp: do not forget FIN in tcp_shifted_skb()
Yuchung found following problem :
There are bugs in the SACK processing code, merging part in
tcp_shift_skb_data(), that incorrectly resets or ignores the sacked
skbs FIN flag. When a receiver first SACK the FIN sequence, and later
throw away ofo queue (e.g., sack-reneging), the sender will stop
retransmitting the FIN flag, and hangs forever.
Following packetdrill test can be used to reproduce the bug.
$ cat sack-merge-bug.pkt
`sysctl -q net.ipv4.tcp_fack=0`
// Establish a connection and send 10 MSS.
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+.000 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+.000 bind(3, ..., ...) = 0
+.000 listen(3, 1) = 0
+.050 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+.000 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.001 < . 1:1(0) ack 1 win 1024
+.000 accept(3, ..., ...) = 4
+.100 write(4, ..., 12000) = 12000
+.000 shutdown(4, SHUT_WR) = 0
+.000 > . 1:10001(10000) ack 1
+.050 < . 1:1(0) ack 2001 win 257
+.000 > FP. 10001:12001(2000) ack 1
+.050 < . 1:1(0) ack 2001 win 257 <sack 10001:11001,nop,nop>
+.050 < . 1:1(0) ack 2001 win 257 <sack 10001:12002,nop,nop>
// SACK reneg
+.050 < . 1:1(0) ack 12001 win 257
+0 %{ print "unacked: ",tcpi_unacked }%
+5 %{ print "" }%
First, a typo inverted left/right of one OR operation, then
code forgot to advance end_seq if the merged skb carried FIN.
Bug was added in 2.6.29 by commit 832d11c5cd076ab
("tcp: Try to restore large SKBs while SACK processing")
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Cc: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Acked-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-04 21:31:41 +04:00
|
|
|
TCP_SKB_CB(prev)->tcp_flags |= TCP_SKB_CB(skb)->tcp_flags;
|
|
|
|
if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
|
|
|
|
TCP_SKB_CB(prev)->end_seq++;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
if (skb == tcp_highest_sack(sk))
|
|
|
|
tcp_advance_highest_sack(sk, skb);
|
|
|
|
|
tcp: Merge tx_flags and tskey in tcp_shifted_skb
After receiving sacks, tcp_shifted_skb() will collapse
skbs if possible. tx_flags and tskey also have to be
merged.
This patch reuses the tcp_skb_collapse_tstamp() to handle
them.
BPF Output Before:
~~~~~
<no-output-due-to-missing-tstamp-event>
BPF Output After:
~~~~~
<...>-2024 [007] d.s. 88.644374: : ee_data:14599
Packetdrill Script:
~~~~~
+0 `sysctl -q -w net.ipv4.tcp_min_tso_segs=10`
+0 `sysctl -q -w net.ipv4.tcp_no_metrics_save=1`
+0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
0.100 < S 0:0(0) win 32792 <mss 1460,sackOK,nop,nop,nop,wscale 7>
0.100 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 7>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
+0 setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
0.200 write(4, ..., 1460) = 1460
+0 setsockopt(4, SOL_SOCKET, 37, [2688], 4) = 0
0.200 write(4, ..., 13140) = 13140
0.200 > P. 1:1461(1460) ack 1
0.200 > . 1461:8761(7300) ack 1
0.200 > P. 8761:14601(5840) ack 1
0.300 < . 1:1(0) ack 1 win 257 <sack 1461:14601,nop,nop>
0.300 > P. 1:1461(1460) ack 1
0.400 < . 1:1(0) ack 14601 win 257
0.400 close(4) = 0
0.400 > F. 14601:14601(0) ack 1
0.500 < F. 1:1(0) ack 14602 win 257
0.500 > . 14602:14602(0) ack 2
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Soheil Hassas Yeganeh <soheil@google.com>
Cc: Willem de Bruijn <willemb@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Tested-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-20 08:39:29 +03:00
|
|
|
tcp_skb_collapse_tstamp(prev, skb);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
tcp_unlink_write_queue(skb, sk);
|
|
|
|
sk_wmem_free_skb(sk, skb);
|
|
|
|
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_SACKMERGED);
|
2008-11-25 08:27:22 +03:00
|
|
|
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/* I wish gso_size would have a bit more sane initialization than
|
|
|
|
* something-or-zero which complicates things
|
|
|
|
*/
|
2011-10-21 13:22:42 +04:00
|
|
|
static int tcp_skb_seglen(const struct sk_buff *skb)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
{
|
2008-12-06 09:41:26 +03:00
|
|
|
return tcp_skb_pcount(skb) == 1 ? skb->len : tcp_skb_mss(skb);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Shifting pages past head area doesn't work */
|
2011-10-21 13:22:42 +04:00
|
|
|
static int skb_can_shift(const struct sk_buff *skb)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
{
|
|
|
|
return !skb_headlen(skb) && skb_is_nonlinear(skb);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Try collapsing SACK blocks spanning across multiple skbs to a single
|
|
|
|
* skb.
|
|
|
|
*/
|
|
|
|
static struct sk_buff *tcp_shift_skb_data(struct sock *sk, struct sk_buff *skb,
|
2008-12-06 09:42:22 +03:00
|
|
|
struct tcp_sacktag_state *state,
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
u32 start_seq, u32 end_seq,
|
2012-05-17 03:15:34 +04:00
|
|
|
bool dup_sack)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *prev;
|
|
|
|
int mss;
|
|
|
|
int pcount = 0;
|
|
|
|
int len;
|
|
|
|
int in_sack;
|
|
|
|
|
|
|
|
if (!sk_can_gso(sk))
|
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
/* Normally R but no L won't result in plain S */
|
|
|
|
if (!dup_sack &&
|
2008-12-06 09:41:06 +03:00
|
|
|
(TCP_SKB_CB(skb)->sacked & (TCPCB_LOST|TCPCB_SACKED_RETRANS)) == TCPCB_SACKED_RETRANS)
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
goto fallback;
|
|
|
|
if (!skb_can_shift(skb))
|
|
|
|
goto fallback;
|
|
|
|
/* This frame is about to be dropped (was ACKed). */
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, tp->snd_una))
|
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
/* Can only happen with delayed DSACK + discard craziness */
|
|
|
|
if (unlikely(skb == tcp_write_queue_head(sk)))
|
|
|
|
goto fallback;
|
|
|
|
prev = tcp_write_queue_prev(sk, skb);
|
|
|
|
|
|
|
|
if ((TCP_SKB_CB(prev)->sacked & TCPCB_TAGBITS) != TCPCB_SACKED_ACKED)
|
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq) &&
|
|
|
|
!before(end_seq, TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
|
|
|
if (in_sack) {
|
|
|
|
len = skb->len;
|
|
|
|
pcount = tcp_skb_pcount(skb);
|
2008-12-06 09:41:26 +03:00
|
|
|
mss = tcp_skb_seglen(skb);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
|
|
|
|
/* TODO: Fix DSACKs to not fragment already SACKed and we can
|
|
|
|
* drop this restriction as unnecessary
|
|
|
|
*/
|
2008-12-06 09:41:26 +03:00
|
|
|
if (mss != tcp_skb_seglen(prev))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
goto fallback;
|
|
|
|
} else {
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, start_seq))
|
|
|
|
goto noop;
|
|
|
|
/* CHECKME: This is non-MSS split case only?, this will
|
|
|
|
* cause skipped skbs due to advancing loop btw, original
|
|
|
|
* has that feature too
|
|
|
|
*/
|
|
|
|
if (tcp_skb_pcount(skb) <= 1)
|
|
|
|
goto noop;
|
|
|
|
|
|
|
|
in_sack = !after(start_seq, TCP_SKB_CB(skb)->seq);
|
|
|
|
if (!in_sack) {
|
|
|
|
/* TODO: head merge to next could be attempted here
|
|
|
|
* if (!after(TCP_SKB_CB(skb)->end_seq, end_seq)),
|
|
|
|
* though it might not be worth of the additional hassle
|
|
|
|
*
|
|
|
|
* ...we can probably just fallback to what was done
|
|
|
|
* previously. We could try merging non-SACKed ones
|
|
|
|
* as well but it probably isn't going to buy off
|
|
|
|
* because later SACKs might again split them, and
|
|
|
|
* it would make skb timestamp tracking considerably
|
|
|
|
* harder problem.
|
|
|
|
*/
|
|
|
|
goto fallback;
|
|
|
|
}
|
|
|
|
|
|
|
|
len = end_seq - TCP_SKB_CB(skb)->seq;
|
|
|
|
BUG_ON(len < 0);
|
|
|
|
BUG_ON(len > skb->len);
|
|
|
|
|
|
|
|
/* MSS boundaries should be honoured or else pcount will
|
|
|
|
* severely break even though it makes things bit trickier.
|
|
|
|
* Optimize common case to avoid most of the divides
|
|
|
|
*/
|
|
|
|
mss = tcp_skb_mss(skb);
|
|
|
|
|
|
|
|
/* TODO: Fix DSACKs to not fragment already SACKed and we can
|
|
|
|
* drop this restriction as unnecessary
|
|
|
|
*/
|
2008-12-06 09:41:26 +03:00
|
|
|
if (mss != tcp_skb_seglen(prev))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
goto fallback;
|
|
|
|
|
|
|
|
if (len == mss) {
|
|
|
|
pcount = 1;
|
|
|
|
} else if (len < mss) {
|
|
|
|
goto noop;
|
|
|
|
} else {
|
|
|
|
pcount = len / mss;
|
|
|
|
len = pcount * mss;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-03-05 23:35:04 +04:00
|
|
|
/* tcp_sacktag_one() won't SACK-tag ranges below snd_una */
|
|
|
|
if (!after(TCP_SKB_CB(skb)->seq + len, tp->snd_una))
|
|
|
|
goto fallback;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
if (!skb_shift(prev, skb, len))
|
|
|
|
goto fallback;
|
2009-03-01 11:21:36 +03:00
|
|
|
if (!tcp_shifted_skb(sk, skb, state, pcount, len, mss, dup_sack))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
/* Hole filled allows collapsing with the next as well, this is very
|
|
|
|
* useful when hole on every nth skb pattern happens
|
|
|
|
*/
|
|
|
|
if (prev == tcp_write_queue_tail(sk))
|
|
|
|
goto out;
|
|
|
|
skb = tcp_write_queue_next(sk, prev);
|
|
|
|
|
2008-12-06 09:40:47 +03:00
|
|
|
if (!skb_can_shift(skb) ||
|
|
|
|
(skb == tcp_send_head(sk)) ||
|
|
|
|
((TCP_SKB_CB(skb)->sacked & TCPCB_TAGBITS) != TCPCB_SACKED_ACKED) ||
|
2008-12-06 09:41:26 +03:00
|
|
|
(mss != tcp_skb_seglen(skb)))
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
goto out;
|
|
|
|
|
|
|
|
len = skb->len;
|
|
|
|
if (skb_shift(prev, skb, len)) {
|
|
|
|
pcount += tcp_skb_pcount(skb);
|
2009-03-01 11:21:36 +03:00
|
|
|
tcp_shifted_skb(sk, skb, state, tcp_skb_pcount(skb), len, mss, 0);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
out:
|
2008-12-06 09:42:22 +03:00
|
|
|
state->fack_count += pcount;
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
return prev;
|
|
|
|
|
|
|
|
noop:
|
|
|
|
return skb;
|
|
|
|
|
|
|
|
fallback:
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_SACKSHIFTFALLBACK);
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
static struct sk_buff *tcp_sacktag_walk(struct sk_buff *skb, struct sock *sk,
|
|
|
|
struct tcp_sack_block *next_dup,
|
2008-12-06 09:42:22 +03:00
|
|
|
struct tcp_sacktag_state *state,
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
u32 start_seq, u32 end_seq,
|
2012-05-17 03:15:34 +04:00
|
|
|
bool dup_sack_in)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
{
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *tmp;
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
tcp_for_write_queue_from(skb, sk) {
|
|
|
|
int in_sack = 0;
|
2012-05-17 03:15:34 +04:00
|
|
|
bool dup_sack = dup_sack_in;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
|
|
|
|
|
|
|
/* queue is in-order => we can short-circuit the walk early */
|
|
|
|
if (!before(TCP_SKB_CB(skb)->seq, end_seq))
|
|
|
|
break;
|
|
|
|
|
2015-04-03 11:17:27 +03:00
|
|
|
if (next_dup &&
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
before(TCP_SKB_CB(skb)->seq, next_dup->end_seq)) {
|
|
|
|
in_sack = tcp_match_skb_to_sack(sk, skb,
|
|
|
|
next_dup->start_seq,
|
|
|
|
next_dup->end_seq);
|
|
|
|
if (in_sack > 0)
|
2012-05-17 03:15:34 +04:00
|
|
|
dup_sack = true;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
}
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
/* skb reference here is a bit tricky to get right, since
|
|
|
|
* shifting can eat and free both this skb and the next,
|
|
|
|
* so not even _safe variant of the loop is enough.
|
|
|
|
*/
|
|
|
|
if (in_sack <= 0) {
|
2008-12-06 09:42:22 +03:00
|
|
|
tmp = tcp_shift_skb_data(sk, skb, state,
|
|
|
|
start_seq, end_seq, dup_sack);
|
2015-04-03 11:17:27 +03:00
|
|
|
if (tmp) {
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
if (tmp != skb) {
|
|
|
|
skb = tmp;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
in_sack = 0;
|
|
|
|
} else {
|
|
|
|
in_sack = tcp_match_skb_to_sack(sk, skb,
|
|
|
|
start_seq,
|
|
|
|
end_seq);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
if (unlikely(in_sack < 0))
|
|
|
|
break;
|
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
if (in_sack) {
|
2012-02-12 22:37:09 +04:00
|
|
|
TCP_SKB_CB(skb)->sacked =
|
|
|
|
tcp_sacktag_one(sk,
|
|
|
|
state,
|
|
|
|
TCP_SKB_CB(skb)->sacked,
|
|
|
|
TCP_SKB_CB(skb)->seq,
|
|
|
|
TCP_SKB_CB(skb)->end_seq,
|
|
|
|
dup_sack,
|
2013-07-23 03:20:47 +04:00
|
|
|
tcp_skb_pcount(skb),
|
2014-02-27 02:02:48 +04:00
|
|
|
&skb->skb_mstamp);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
|
tcp: Try to restore large SKBs while SACK processing
During SACK processing, most of the benefits of TSO are eaten by
the SACK blocks that one-by-one fragment SKBs to MSS sized chunks.
Then we're in problems when cleanup work for them has to be done
when a large cumulative ACK comes. Try to return back to pre-split
state already while more and more SACK info gets discovered by
combining newly discovered SACK areas with the previous skb if
that's SACKed as well.
This approach has a number of benefits:
1) The processing overhead is spread more equally over the RTT
2) Write queue has less skbs to process (affect everything
which has to walk in the queue past the sacked areas)
3) Write queue is consistent whole the time, so no other parts
of TCP has to be aware of this (this was not the case with
some other approach that was, well, quite intrusive all
around).
4) Clean_rtx_queue can release most of the pages using single
put_page instead of previous PAGE_SIZE/mss+1 calls
In case a hole is fully filled by the new SACK block, we attempt
to combine the next skb too which allows construction of skbs
that are even larger than what tso split them to and it handles
hole per on every nth patterns that often occur during slow start
overshoot pretty nicely. Though this to be really useful also
a retransmission would have to get lost since cumulative ACKs
advance one hole at a time in the most typical case.
TODO: handle upwards only merging. That should be rather easy
when segment is fully sacked but I'm leaving that as future
work item (it won't make very large difference anyway since
this current approach already covers quite a lot of normal
cases).
I was earlier thinking of some sophisticated way of tracking
timestamps of the first and the last segment but later on
realized that it won't be that necessary at all to store the
timestamp of the last segment. The cases that can occur are
basically either:
1) ambiguous => no sensible measurement can be taken anyway
2) non-ambiguous is due to reordering => having the timestamp
of the last segment there is just skewing things more off
than does some good since the ack got triggered by one of
the holes (besides some substle issues that would make
determining right hole/skb even harder problem). Anyway,
it has nothing to do with this change then.
I choose to route some abnormal looking cases with goto noop,
some could be handled differently (eg., by stopping the
walking at that skb but again). In general, they either
shouldn't happen at all or are rare enough to make no difference
in practice.
In theory this change (as whole) could cause some macroscale
regression (global) because of cache misses that are taken over
the round-trip time but it gets very likely better because of much
less (local) cache misses per other write queue walkers and the
big recovery clearing cumulative ack.
Worth to note that these benefits would be very easy to get also
without TSO/GSO being on as long as the data is in pages so that
we can merge them. Currently I won't let that happen because
DSACK splitting at fragment that would mess up pcounts due to
sk_can_gso in tcp_set_skb_tso_segs. Once DSACKs fragments gets
avoided, we have some conditions that can be made less strict.
TODO: I will probably have to convert the excessive pointer
passing to struct sacktag_state... :-)
My testing revealed that considerable amount of skbs couldn't
be shifted because they were cloned (most likely still awaiting
tx reclaim)...
[The rest is considering future work instead since I got
repeatably EFAULT to tcpdump's recvfrom when I added
pskb_expand_head to deal with clones, so I separated that
into another, later patch]
...To counter that, I gave up on the fifth advantage:
5) When growing previous SACK block, less allocs for new skbs
are done, basically a new alloc is needed only when new hole
is detected and when the previous skb runs out of frags space
...which now only happens of if reclaim is fast enough to dispose
the clone before the SACK block comes in (the window is RTT long),
otherwise we'll have to alloc some.
With clones being handled I got these numbers (will be somewhat
worse without that), taken with fine-grained mibs:
TCPSackShifted 398
TCPSackMerged 877
TCPSackShiftFallback 320
TCPSACKCOLLAPSEFALLBACKGSO 0
TCPSACKCOLLAPSEFALLBACKSKBBITS 0
TCPSACKCOLLAPSEFALLBACKSKBDATA 0
TCPSACKCOLLAPSEFALLBACKBELOW 0
TCPSACKCOLLAPSEFALLBACKFIRST 1
TCPSACKCOLLAPSEFALLBACKPREVBITS 318
TCPSACKCOLLAPSEFALLBACKMSS 1
TCPSACKCOLLAPSEFALLBACKNOHEAD 0
TCPSACKCOLLAPSEFALLBACKSHIFT 0
TCPSACKCOLLAPSENOOPSEQ 0
TCPSACKCOLLAPSENOOPSMALLPCOUNT 0
TCPSACKCOLLAPSENOOPSMALLLEN 0
TCPSACKCOLLAPSEHOLE 12
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-11-25 08:20:15 +03:00
|
|
|
if (!before(TCP_SKB_CB(skb)->seq,
|
|
|
|
tcp_highest_sack_seq(tp)))
|
|
|
|
tcp_advance_highest_sack(sk, skb);
|
|
|
|
}
|
|
|
|
|
2008-12-06 09:42:22 +03:00
|
|
|
state->fack_count += tcp_skb_pcount(skb);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
}
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Avoid all extra work that is being done by sacktag while walking in
|
|
|
|
* a normal way
|
|
|
|
*/
|
|
|
|
static struct sk_buff *tcp_sacktag_skip(struct sk_buff *skb, struct sock *sk,
|
2008-12-06 09:42:22 +03:00
|
|
|
struct tcp_sacktag_state *state,
|
|
|
|
u32 skip_to_seq)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
{
|
|
|
|
tcp_for_write_queue_from(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
|
|
|
|
2008-11-25 08:12:28 +03:00
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, skip_to_seq))
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
break;
|
2008-03-03 23:10:16 +03:00
|
|
|
|
2008-12-06 09:42:22 +03:00
|
|
|
state->fack_count += tcp_skb_pcount(skb);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
}
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
|
|
|
static struct sk_buff *tcp_maybe_skipping_dsack(struct sk_buff *skb,
|
|
|
|
struct sock *sk,
|
|
|
|
struct tcp_sack_block *next_dup,
|
2008-12-06 09:42:22 +03:00
|
|
|
struct tcp_sacktag_state *state,
|
|
|
|
u32 skip_to_seq)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
{
|
2015-04-03 11:17:26 +03:00
|
|
|
if (!next_dup)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
return skb;
|
|
|
|
|
|
|
|
if (before(next_dup->start_seq, skip_to_seq)) {
|
2008-12-06 09:42:22 +03:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, state, next_dup->start_seq);
|
|
|
|
skb = tcp_sacktag_walk(skb, sk, NULL, state,
|
|
|
|
next_dup->start_seq, next_dup->end_seq,
|
|
|
|
1);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
return skb;
|
|
|
|
}
|
|
|
|
|
2011-10-21 13:22:42 +04:00
|
|
|
static int tcp_sack_cache_ok(const struct tcp_sock *tp, const struct tcp_sack_block *cache)
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
{
|
|
|
|
return cache < tp->recv_sack_cache + ARRAY_SIZE(tp->recv_sack_cache);
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
static int
|
2011-10-21 13:22:42 +04:00
|
|
|
tcp_sacktag_write_queue(struct sock *sk, const struct sk_buff *ack_skb,
|
2015-05-01 02:10:57 +03:00
|
|
|
u32 prior_snd_una, struct tcp_sacktag_state *state)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2011-10-21 13:22:42 +04:00
|
|
|
const unsigned char *ptr = (skb_transport_header(ack_skb) +
|
|
|
|
TCP_SKB_CB(ack_skb)->sacked);
|
2007-11-16 06:49:47 +03:00
|
|
|
struct tcp_sack_block_wire *sp_wire = (struct tcp_sack_block_wire *)(ptr+2);
|
2008-07-19 11:07:02 +04:00
|
|
|
struct tcp_sack_block sp[TCP_NUM_SACKS];
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
struct tcp_sack_block *cache;
|
|
|
|
struct sk_buff *skb;
|
2008-07-19 11:07:02 +04:00
|
|
|
int num_sacks = min(TCP_NUM_SACKS, (ptr[1] - TCPOLEN_SACK_BASE) >> 3);
|
2007-11-16 06:49:47 +03:00
|
|
|
int used_sacks;
|
2012-05-17 03:15:34 +04:00
|
|
|
bool found_dup_sack = false;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
int i, j;
|
2007-02-05 10:35:57 +03:00
|
|
|
int first_sack_index;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2015-05-01 02:10:57 +03:00
|
|
|
state->flag = 0;
|
|
|
|
state->reord = tp->packets_out;
|
2008-12-06 09:42:22 +03:00
|
|
|
|
2007-03-25 08:03:23 +04:00
|
|
|
if (!tp->sacked_out) {
|
2007-10-08 10:37:55 +04:00
|
|
|
if (WARN_ON(tp->fackets_out))
|
|
|
|
tp->fackets_out = 0;
|
2007-12-02 01:48:06 +03:00
|
|
|
tcp_highest_sack_reset(sk);
|
2007-03-25 08:03:23 +04:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-07-17 07:29:51 +04:00
|
|
|
found_dup_sack = tcp_check_dsack(sk, ack_skb, sp_wire,
|
2007-06-19 09:43:06 +04:00
|
|
|
num_sacks, prior_snd_una);
|
|
|
|
if (found_dup_sack)
|
2015-05-01 02:10:57 +03:00
|
|
|
state->flag |= FLAG_DSACKING_ACK;
|
2007-02-05 10:36:42 +03:00
|
|
|
|
|
|
|
/* Eliminate too old ACKs, but take into
|
|
|
|
* account more or less fresh ones, they can
|
|
|
|
* contain valid SACK info.
|
|
|
|
*/
|
|
|
|
if (before(TCP_SKB_CB(ack_skb)->ack_seq, prior_snd_una - tp->max_window))
|
|
|
|
return 0;
|
|
|
|
|
2007-11-15 02:47:18 +03:00
|
|
|
if (!tp->packets_out)
|
|
|
|
goto out;
|
|
|
|
|
2007-11-16 06:49:47 +03:00
|
|
|
used_sacks = 0;
|
|
|
|
first_sack_index = 0;
|
|
|
|
for (i = 0; i < num_sacks; i++) {
|
2012-05-17 03:15:34 +04:00
|
|
|
bool dup_sack = !i && found_dup_sack;
|
2007-11-16 06:49:47 +03:00
|
|
|
|
2008-05-03 03:26:16 +04:00
|
|
|
sp[used_sacks].start_seq = get_unaligned_be32(&sp_wire[i].start_seq);
|
|
|
|
sp[used_sacks].end_seq = get_unaligned_be32(&sp_wire[i].end_seq);
|
2007-11-16 06:49:47 +03:00
|
|
|
|
|
|
|
if (!tcp_is_sackblock_valid(tp, dup_sack,
|
|
|
|
sp[used_sacks].start_seq,
|
|
|
|
sp[used_sacks].end_seq)) {
|
2008-07-03 12:05:41 +04:00
|
|
|
int mib_idx;
|
|
|
|
|
2007-11-16 06:49:47 +03:00
|
|
|
if (dup_sack) {
|
|
|
|
if (!tp->undo_marker)
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKIGNOREDNOUNDO;
|
2007-11-16 06:49:47 +03:00
|
|
|
else
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKIGNOREDOLD;
|
2007-11-16 06:49:47 +03:00
|
|
|
} else {
|
|
|
|
/* Don't count olds caused by ACK reordering */
|
|
|
|
if ((TCP_SKB_CB(ack_skb)->ack_seq != tp->snd_una) &&
|
|
|
|
!after(sp[used_sacks].end_seq, tp->snd_una))
|
|
|
|
continue;
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPSACKDISCARD;
|
2007-11-16 06:49:47 +03:00
|
|
|
}
|
2008-07-03 12:05:41 +04:00
|
|
|
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), mib_idx);
|
2007-11-16 06:49:47 +03:00
|
|
|
if (i == 0)
|
|
|
|
first_sack_index = -1;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Ignore very old stuff early */
|
|
|
|
if (!after(sp[used_sacks].end_seq, prior_snd_una))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
used_sacks++;
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
/* order SACK blocks to allow in order walk of the retrans queue */
|
|
|
|
for (i = used_sacks - 1; i > 0; i--) {
|
2008-01-01 01:57:14 +03:00
|
|
|
for (j = 0; j < i; j++) {
|
|
|
|
if (after(sp[j].start_seq, sp[j + 1].start_seq)) {
|
2009-03-21 23:36:17 +03:00
|
|
|
swap(sp[j], sp[j + 1]);
|
2005-11-11 04:14:59 +03:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
/* Track where the first SACK block goes to */
|
|
|
|
if (j == first_sack_index)
|
2008-01-01 01:57:14 +03:00
|
|
|
first_sack_index = j + 1;
|
2005-11-11 04:14:59 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
skb = tcp_write_queue_head(sk);
|
2015-05-01 02:10:57 +03:00
|
|
|
state->fack_count = 0;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
i = 0;
|
|
|
|
|
|
|
|
if (!tp->sacked_out) {
|
|
|
|
/* It's already past, so skip checking against it */
|
|
|
|
cache = tp->recv_sack_cache + ARRAY_SIZE(tp->recv_sack_cache);
|
|
|
|
} else {
|
|
|
|
cache = tp->recv_sack_cache;
|
|
|
|
/* Skip empty blocks in at head of the cache */
|
|
|
|
while (tcp_sack_cache_ok(tp, cache) && !cache->start_seq &&
|
|
|
|
!cache->end_seq)
|
|
|
|
cache++;
|
2007-02-05 10:35:57 +03:00
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
while (i < used_sacks) {
|
2007-11-16 06:49:47 +03:00
|
|
|
u32 start_seq = sp[i].start_seq;
|
|
|
|
u32 end_seq = sp[i].end_seq;
|
2012-05-17 03:15:34 +04:00
|
|
|
bool dup_sack = (found_dup_sack && (i == first_sack_index));
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
struct tcp_sack_block *next_dup = NULL;
|
[TCP]: Process DSACKs that reside within a SACK block
DSACK inside another SACK block were missed if start_seq of DSACK
was larger than SACK block's because sorting prioritizes full
processing of the SACK block before DSACK. After SACK block
sorting situation is like this:
SSSSSSSSS
D
SSSSSS
SSSSSSS
Because write_queue is walked in-order, when the first SACK block
has been processed, TCP is already past the skb for which the
DSACK arrived and we haven't taught it to backtrack (nor should
we), so TCP just continues processing by going to the next SACK
block after the DSACK (if any).
Whenever such DSACK is present, do an embedded checking during
the previous SACK block.
If the DSACK is below snd_una, there won't be overlapping SACK
block, and thus no problem in that case. Also if start_seq of
the DSACK is equal to the actual block, it will be processed
first.
Tested this by using netem to duplicate 15% of packets, and
by printing SACK block when found_dup_sack is true and the
selected skb in the dup_sack = 1 branch (if taken):
SACK block 0: 4344-5792 (relative to snd_una 2019137317)
SACK block 1: 4344-5792 (relative to snd_una 2019137317)
equal start seqnos => next_dup = 0, dup_sack = 1 won't occur...
SACK block 0: 5792-7240 (relative to snd_una 2019214061)
SACK block 1: 2896-7240 (relative to snd_una 2019214061)
DSACK skb match 5792-7240 (relative to snd_una)
...and next_dup = 1 case (after the not shown start_seq sort),
went to dup_sack = 1 branch.
Signed-off-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-01 10:09:37 +03:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
if (found_dup_sack && ((i + 1) == first_sack_index))
|
|
|
|
next_dup = &sp[i + 1];
|
2005-04-17 02:20:36 +04:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
/* Skip too early cached blocks */
|
|
|
|
while (tcp_sack_cache_ok(tp, cache) &&
|
|
|
|
!before(start_seq, cache->end_seq))
|
|
|
|
cache++;
|
|
|
|
|
|
|
|
/* Can skip some work by looking recv_sack_cache? */
|
|
|
|
if (tcp_sack_cache_ok(tp, cache) && !dup_sack &&
|
|
|
|
after(end_seq, cache->start_seq)) {
|
|
|
|
|
|
|
|
/* Head todo? */
|
|
|
|
if (before(start_seq, cache->start_seq)) {
|
2015-05-01 02:10:57 +03:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, state,
|
2008-12-06 09:42:22 +03:00
|
|
|
start_seq);
|
2008-01-01 01:57:14 +03:00
|
|
|
skb = tcp_sacktag_walk(skb, sk, next_dup,
|
2015-05-01 02:10:57 +03:00
|
|
|
state,
|
2008-01-01 01:57:14 +03:00
|
|
|
start_seq,
|
|
|
|
cache->start_seq,
|
2008-12-06 09:42:22 +03:00
|
|
|
dup_sack);
|
2007-02-05 10:35:57 +03:00
|
|
|
}
|
2005-11-11 04:14:59 +03:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
/* Rest of the block already fully processed? */
|
2007-11-17 03:17:05 +03:00
|
|
|
if (!after(end_seq, cache->end_seq))
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
goto advance_sp;
|
2007-11-17 03:17:05 +03:00
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
skb = tcp_maybe_skipping_dsack(skb, sk, next_dup,
|
2015-05-01 02:10:57 +03:00
|
|
|
state,
|
2008-12-06 09:42:22 +03:00
|
|
|
cache->end_seq);
|
[TCP]: Process DSACKs that reside within a SACK block
DSACK inside another SACK block were missed if start_seq of DSACK
was larger than SACK block's because sorting prioritizes full
processing of the SACK block before DSACK. After SACK block
sorting situation is like this:
SSSSSSSSS
D
SSSSSS
SSSSSSS
Because write_queue is walked in-order, when the first SACK block
has been processed, TCP is already past the skb for which the
DSACK arrived and we haven't taught it to backtrack (nor should
we), so TCP just continues processing by going to the next SACK
block after the DSACK (if any).
Whenever such DSACK is present, do an embedded checking during
the previous SACK block.
If the DSACK is below snd_una, there won't be overlapping SACK
block, and thus no problem in that case. Also if start_seq of
the DSACK is equal to the actual block, it will be processed
first.
Tested this by using netem to duplicate 15% of packets, and
by printing SACK block when found_dup_sack is true and the
selected skb in the dup_sack = 1 branch (if taken):
SACK block 0: 4344-5792 (relative to snd_una 2019137317)
SACK block 1: 4344-5792 (relative to snd_una 2019137317)
equal start seqnos => next_dup = 0, dup_sack = 1 won't occur...
SACK block 0: 5792-7240 (relative to snd_una 2019214061)
SACK block 1: 2896-7240 (relative to snd_una 2019214061)
DSACK skb match 5792-7240 (relative to snd_una)
...and next_dup = 1 case (after the not shown start_seq sort),
went to dup_sack = 1 branch.
Signed-off-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-01 10:09:37 +03:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
/* ...tail remains todo... */
|
2007-12-02 01:48:06 +03:00
|
|
|
if (tcp_highest_sack_seq(tp) == cache->end_seq) {
|
2007-11-17 03:17:05 +03:00
|
|
|
/* ...but better entrypoint exists! */
|
2007-12-02 01:48:06 +03:00
|
|
|
skb = tcp_highest_sack(sk);
|
2015-04-03 11:17:26 +03:00
|
|
|
if (!skb)
|
2007-12-02 01:48:06 +03:00
|
|
|
break;
|
2015-05-01 02:10:57 +03:00
|
|
|
state->fack_count = tp->fackets_out;
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
cache++;
|
|
|
|
goto walk;
|
[TCP]: Process DSACKs that reside within a SACK block
DSACK inside another SACK block were missed if start_seq of DSACK
was larger than SACK block's because sorting prioritizes full
processing of the SACK block before DSACK. After SACK block
sorting situation is like this:
SSSSSSSSS
D
SSSSSS
SSSSSSS
Because write_queue is walked in-order, when the first SACK block
has been processed, TCP is already past the skb for which the
DSACK arrived and we haven't taught it to backtrack (nor should
we), so TCP just continues processing by going to the next SACK
block after the DSACK (if any).
Whenever such DSACK is present, do an embedded checking during
the previous SACK block.
If the DSACK is below snd_una, there won't be overlapping SACK
block, and thus no problem in that case. Also if start_seq of
the DSACK is equal to the actual block, it will be processed
first.
Tested this by using netem to duplicate 15% of packets, and
by printing SACK block when found_dup_sack is true and the
selected skb in the dup_sack = 1 branch (if taken):
SACK block 0: 4344-5792 (relative to snd_una 2019137317)
SACK block 1: 4344-5792 (relative to snd_una 2019137317)
equal start seqnos => next_dup = 0, dup_sack = 1 won't occur...
SACK block 0: 5792-7240 (relative to snd_una 2019214061)
SACK block 1: 2896-7240 (relative to snd_una 2019214061)
DSACK skb match 5792-7240 (relative to snd_una)
...and next_dup = 1 case (after the not shown start_seq sort),
went to dup_sack = 1 branch.
Signed-off-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-01 10:09:37 +03:00
|
|
|
}
|
|
|
|
|
2015-05-01 02:10:57 +03:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, state, cache->end_seq);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
/* Check overlap against next cached too (past this one already) */
|
|
|
|
cache++;
|
|
|
|
continue;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2007-12-02 01:48:06 +03:00
|
|
|
if (!before(start_seq, tcp_highest_sack_seq(tp))) {
|
|
|
|
skb = tcp_highest_sack(sk);
|
2015-04-03 11:17:26 +03:00
|
|
|
if (!skb)
|
2007-12-02 01:48:06 +03:00
|
|
|
break;
|
2015-05-01 02:10:57 +03:00
|
|
|
state->fack_count = tp->fackets_out;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2015-05-01 02:10:57 +03:00
|
|
|
skb = tcp_sacktag_skip(skb, sk, state, start_seq);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
|
|
|
|
walk:
|
2015-05-01 02:10:57 +03:00
|
|
|
skb = tcp_sacktag_walk(skb, sk, next_dup, state,
|
2008-12-06 09:42:22 +03:00
|
|
|
start_seq, end_seq, dup_sack);
|
2007-11-11 08:24:19 +03:00
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
advance_sp:
|
|
|
|
i++;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
/* Clear the head of the cache sack blocks so we can skip it next time */
|
|
|
|
for (i = 0; i < ARRAY_SIZE(tp->recv_sack_cache) - used_sacks; i++) {
|
|
|
|
tp->recv_sack_cache[i].start_seq = 0;
|
|
|
|
tp->recv_sack_cache[i].end_seq = 0;
|
|
|
|
}
|
|
|
|
for (j = 0; j < used_sacks; j++)
|
|
|
|
tp->recv_sack_cache[i++] = sp[j];
|
|
|
|
|
2015-05-01 02:10:57 +03:00
|
|
|
if ((state->reord < tp->fackets_out) &&
|
2013-03-20 17:32:58 +04:00
|
|
|
((inet_csk(sk)->icsk_ca_state != TCP_CA_Loss) || tp->undo_marker))
|
2015-05-01 02:10:57 +03:00
|
|
|
tcp_update_reordering(sk, tp->fackets_out - state->reord, 0);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2015-04-29 21:28:30 +03:00
|
|
|
tcp_verify_left_out(tp);
|
2007-11-15 02:47:18 +03:00
|
|
|
out:
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#if FASTRETRANS_DEBUG > 0
|
2008-07-26 08:43:18 +04:00
|
|
|
WARN_ON((int)tp->sacked_out < 0);
|
|
|
|
WARN_ON((int)tp->lost_out < 0);
|
|
|
|
WARN_ON((int)tp->retrans_out < 0);
|
|
|
|
WARN_ON((int)tcp_packets_in_flight(tp) < 0);
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
2015-05-01 02:10:57 +03:00
|
|
|
return state->flag;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2008-04-08 09:33:07 +04:00
|
|
|
/* Limits sacked_out so that sum with lost_out isn't ever larger than
|
2012-05-17 03:15:34 +04:00
|
|
|
* packets_out. Returns false if sacked_out adjustement wasn't necessary.
|
2007-02-22 10:01:36 +03:00
|
|
|
*/
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_limit_reno_sacked(struct tcp_sock *tp)
|
2007-05-27 12:52:00 +04:00
|
|
|
{
|
|
|
|
u32 holes;
|
|
|
|
|
|
|
|
holes = max(tp->lost_out, 1U);
|
|
|
|
holes = min(holes, tp->packets_out);
|
|
|
|
|
|
|
|
if ((tp->sacked_out + holes) > tp->packets_out) {
|
|
|
|
tp->sacked_out = tp->packets_out - holes;
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2007-05-27 12:52:00 +04:00
|
|
|
}
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2008-04-08 09:33:07 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* If we receive more dupacks than we expected counting segments
|
|
|
|
* in assumption of absent reordering, interpret this as reordering.
|
|
|
|
* The only another reason could be bug in receiver TCP.
|
|
|
|
*/
|
|
|
|
static void tcp_check_reno_reordering(struct sock *sk, const int addend)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
if (tcp_limit_reno_sacked(tp))
|
|
|
|
tcp_update_reordering(sk, tp->packets_out + addend, 0);
|
2007-05-27 12:52:00 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Emulate SACKs for SACKless connection: account for a new dupack. */
|
|
|
|
|
|
|
|
static void tcp_add_reno_sack(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2016-02-02 21:33:06 +03:00
|
|
|
u32 prior_sacked = tp->sacked_out;
|
|
|
|
|
2007-05-27 12:52:00 +04:00
|
|
|
tp->sacked_out++;
|
|
|
|
tcp_check_reno_reordering(sk, 0);
|
2016-02-02 21:33:06 +03:00
|
|
|
if (tp->sacked_out > prior_sacked)
|
|
|
|
tp->delivered++; /* Some out-of-order packet is delivered */
|
2007-08-09 15:44:16 +04:00
|
|
|
tcp_verify_left_out(tp);
|
2007-05-27 12:52:00 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Account for ACK, ACKing some data in Reno Recovery phase. */
|
|
|
|
|
|
|
|
static void tcp_remove_reno_sacks(struct sock *sk, int acked)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
if (acked > 0) {
|
|
|
|
/* One ACK acked hole. The rest eat duplicate ACKs. */
|
2016-02-02 21:33:06 +03:00
|
|
|
tp->delivered += max_t(int, acked - tp->sacked_out, 1);
|
2008-01-01 01:57:14 +03:00
|
|
|
if (acked - 1 >= tp->sacked_out)
|
2007-05-27 12:52:00 +04:00
|
|
|
tp->sacked_out = 0;
|
|
|
|
else
|
2008-01-01 01:57:14 +03:00
|
|
|
tp->sacked_out -= acked - 1;
|
2007-05-27 12:52:00 +04:00
|
|
|
}
|
|
|
|
tcp_check_reno_reordering(sk, acked);
|
2007-08-09 15:44:16 +04:00
|
|
|
tcp_verify_left_out(tp);
|
2007-05-27 12:52:00 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
static inline void tcp_reset_reno_sack(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
tp->sacked_out = 0;
|
|
|
|
}
|
|
|
|
|
2014-08-23 01:15:22 +04:00
|
|
|
void tcp_clear_retrans(struct tcp_sock *tp)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
tp->retrans_out = 0;
|
|
|
|
tp->lost_out = 0;
|
|
|
|
tp->undo_marker = 0;
|
2014-07-02 23:07:16 +04:00
|
|
|
tp->undo_retrans = -1;
|
2014-08-23 01:15:22 +04:00
|
|
|
tp->fackets_out = 0;
|
|
|
|
tp->sacked_out = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2014-08-23 01:15:22 +04:00
|
|
|
static inline void tcp_init_undo(struct tcp_sock *tp)
|
2007-10-12 04:34:57 +04:00
|
|
|
{
|
2014-08-23 01:15:22 +04:00
|
|
|
tp->undo_marker = tp->snd_una;
|
|
|
|
/* Retransmission still in flight may cause DSACKs later. */
|
|
|
|
tp->undo_retrans = tp->retrans_out ? : -1;
|
2007-10-12 04:34:57 +04:00
|
|
|
}
|
|
|
|
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-05 03:12:29 +04:00
|
|
|
/* Enter Loss state. If we detect SACK reneging, forget all SACK information
|
2005-04-17 02:20:36 +04:00
|
|
|
* and reset tags completely, otherwise preserve SACKs. If receiver
|
|
|
|
* dropped its ofo queue, we will know this due to reneging detection.
|
|
|
|
*/
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-05 03:12:29 +04:00
|
|
|
void tcp_enter_loss(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 11:03:31 +04:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2016-02-03 10:46:52 +03:00
|
|
|
struct net *net = sock_net(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
struct sk_buff *skb;
|
2015-07-13 22:10:20 +03:00
|
|
|
bool new_recovery = icsk->icsk_ca_state < TCP_CA_Recovery;
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-05 03:12:29 +04:00
|
|
|
bool is_reneg; /* is receiver reneging on SACKs? */
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Reduce ssthresh if it has not yet been made inside this window. */
|
2013-03-20 17:33:00 +04:00
|
|
|
if (icsk->icsk_ca_state <= TCP_CA_Disorder ||
|
|
|
|
!after(tp->high_seq, tp->snd_una) ||
|
2005-08-10 11:03:31 +04:00
|
|
|
(icsk->icsk_ca_state == TCP_CA_Loss && !icsk->icsk_retransmits)) {
|
|
|
|
tp->prior_ssthresh = tcp_current_ssthresh(sk);
|
|
|
|
tp->snd_ssthresh = icsk->icsk_ca_ops->ssthresh(sk);
|
|
|
|
tcp_ca_event(sk, CA_EVENT_LOSS);
|
2014-08-23 01:15:22 +04:00
|
|
|
tcp_init_undo(tp);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
tp->snd_cwnd = 1;
|
|
|
|
tp->snd_cwnd_cnt = 0;
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
|
2014-08-23 01:15:22 +04:00
|
|
|
tp->retrans_out = 0;
|
|
|
|
tp->lost_out = 0;
|
2007-10-12 04:34:57 +04:00
|
|
|
|
|
|
|
if (tcp_is_reno(tp))
|
|
|
|
tcp_reset_reno_sack(tp);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-05 03:12:29 +04:00
|
|
|
skb = tcp_write_queue_head(sk);
|
|
|
|
is_reneg = skb && (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED);
|
|
|
|
if (is_reneg) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPSACKRENEGING);
|
2007-10-12 04:34:57 +04:00
|
|
|
tp->sacked_out = 0;
|
|
|
|
tp->fackets_out = 0;
|
2007-09-20 22:37:19 +04:00
|
|
|
}
|
2008-09-21 08:18:32 +04:00
|
|
|
tcp_clear_all_retrans_hints(tp);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2007-03-07 23:12:44 +03:00
|
|
|
tcp_for_write_queue(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
2007-10-12 04:34:57 +04:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
TCP_SKB_CB(skb)->sacked &= (~TCPCB_TAGBITS)|TCPCB_SACKED_ACKED;
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-05 03:12:29 +04:00
|
|
|
if (!(TCP_SKB_CB(skb)->sacked&TCPCB_SACKED_ACKED) || is_reneg) {
|
2005-04-17 02:20:36 +04:00
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_SACKED_ACKED;
|
|
|
|
TCP_SKB_CB(skb)->sacked |= TCPCB_LOST;
|
|
|
|
tp->lost_out += tcp_skb_pcount(skb);
|
2008-09-21 08:20:20 +04:00
|
|
|
tp->retransmit_high = TCP_SKB_CB(skb)->end_seq;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
2007-08-09 15:44:16 +04:00
|
|
|
tcp_verify_left_out(tp);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-08-13 03:41:25 +04:00
|
|
|
/* Timeout in disordered state after receiving substantial DUPACKs
|
|
|
|
* suggests that the degree of reordering is over-estimated.
|
|
|
|
*/
|
|
|
|
if (icsk->icsk_ca_state <= TCP_CA_Disorder &&
|
2016-02-03 10:46:52 +03:00
|
|
|
tp->sacked_out >= net->ipv4.sysctl_tcp_reordering)
|
2013-08-13 03:41:25 +04:00
|
|
|
tp->reordering = min_t(unsigned int, tp->reordering,
|
2016-02-03 10:46:52 +03:00
|
|
|
net->ipv4.sysctl_tcp_reordering);
|
2005-08-10 11:03:31 +04:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Loss);
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->high_seq = tp->snd_nxt;
|
2014-09-29 15:08:30 +04:00
|
|
|
tcp_ecn_queue_cwr(tp);
|
2013-03-20 17:33:00 +04:00
|
|
|
|
|
|
|
/* F-RTO RFC5682 sec 3.1 step 1: retransmit SND.UNA if no previous
|
|
|
|
* loss recovery is underway except recurring timeout(s) on
|
|
|
|
* the same SND.UNA (sec 3.2). Disable F-RTO on path MTU probing
|
|
|
|
*/
|
|
|
|
tp->frto = sysctl_tcp_frto &&
|
|
|
|
(new_recovery || icsk->icsk_retransmits) &&
|
|
|
|
!inet_csk(sk)->icsk_mtup.probe_size;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2007-12-31 15:49:21 +03:00
|
|
|
/* If ACK arrived pointing to a remembered SACK, it means that our
|
|
|
|
* remembered SACKs do not reflect real state of receiver i.e.
|
|
|
|
* receiver _host_ is heavily congested (or buggy).
|
|
|
|
*
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-05 03:12:29 +04:00
|
|
|
* To avoid big spurious retransmission bursts due to transient SACK
|
|
|
|
* scoreboard oddities that look like reneging, we give the receiver a
|
|
|
|
* little time (max(RTT/2, 10ms)) to send us some more ACKs that will
|
|
|
|
* restore sanity to the SACK scoreboard. If the apparent reneging
|
|
|
|
* persists until this RTO then we'll clear the SACK scoreboard.
|
2007-12-31 15:49:21 +03:00
|
|
|
*/
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_check_sack_reneging(struct sock *sk, int flag)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2007-12-31 15:49:21 +03:00
|
|
|
if (flag & FLAG_SACK_RENEGING) {
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-05 03:12:29 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
unsigned long delay = max(usecs_to_jiffies(tp->srtt_us >> 4),
|
|
|
|
msecs_to_jiffies(10));
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_RETRANS,
|
tcp: reduce spurious retransmits due to transient SACK reneging
This commit reduces spurious retransmits due to apparent SACK reneging
by only reacting to SACK reneging that persists for a short delay.
When a sequence space hole at snd_una is filled, some TCP receivers
send a series of ACKs as they apparently scan their out-of-order queue
and cumulatively ACK all the packets that have now been consecutiveyly
received. This is essentially misbehavior B in "Misbehaviors in TCP
SACK generation" ACM SIGCOMM Computer Communication Review, April
2011, so we suspect that this is from several common OSes (Windows
2000, Windows Server 2003, Windows XP). However, this issue has also
been seen in other cases, e.g. the netdev thread "TCP being hoodwinked
into spurious retransmissions by lack of timestamps?" from March 2014,
where the receiver was thought to be a BSD box.
Since snd_una would temporarily be adjacent to a previously SACKed
range in these scenarios, this receiver behavior triggered the Linux
SACK reneging code path in the sender. This led the sender to clear
the SACK scoreboard, enter CA_Loss, and spuriously retransmit
(potentially) every packet from the entire write queue at line rate
just a few milliseconds before the ACK for each packet arrives at the
sender.
To avoid such situations, now when a sender sees apparent reneging it
does not yet retransmit, but rather adjusts the RTO timer to give the
receiver a little time (max(RTT/2, 10ms)) to send us some more ACKs
that will restore sanity to the SACK scoreboard. If the reneging
persists until this RTO then, as before, we clear the SACK scoreboard
and enter CA_Loss.
A 10ms delay tolerates a receiver sending such a stream of ACKs at
56Kbit/sec. And to allow for receivers with slower or more congested
paths, we wait for at least RTT/2.
We validated the resulting max(RTT/2, 10ms) delay formula with a mix
of North American and South American Google web server traffic, and
found that for ACKs displaying transient reneging:
(1) 90% of inter-ACK delays were less than 10ms
(2) 99% of inter-ACK delays were less than RTT/2
In tests on Google web servers this commit reduced reneging events by
75%-90% (as measured by the TcpExtTCPSACKReneging counter), without
any measurable impact on latency for user HTTP and SPDY requests.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-08-05 03:12:29 +04:00
|
|
|
delay, TCP_RTO_MAX);
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2011-10-21 13:22:42 +04:00
|
|
|
static inline int tcp_fackets_out(const struct tcp_sock *tp)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2008-01-01 01:57:14 +03:00
|
|
|
return tcp_is_reno(tp) ? tp->sacked_out + 1 : tp->fackets_out;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2007-11-16 06:39:31 +03:00
|
|
|
/* Heurestics to calculate number of duplicate ACKs. There's no dupACKs
|
|
|
|
* counter when SACK is enabled (without SACK, sacked_out is used for
|
|
|
|
* that purpose).
|
|
|
|
*
|
|
|
|
* Instead, with FACK TCP uses fackets_out that includes both SACKed
|
|
|
|
* segments up to the highest received SACK block so far and holes in
|
|
|
|
* between them.
|
|
|
|
*
|
|
|
|
* With reordering, holes may still be in flight, so RFC3517 recovery
|
|
|
|
* uses pure sacked_out (total number of SACKed segments) even though
|
|
|
|
* it violates the RFC that uses duplicate ACKs, often these are equal
|
|
|
|
* but when e.g. out-of-window ACKs or packet duplication occurs,
|
|
|
|
* they differ. Since neither occurs due to loss, TCP should really
|
|
|
|
* ignore them.
|
|
|
|
*/
|
2011-10-21 13:22:42 +04:00
|
|
|
static inline int tcp_dupack_heuristics(const struct tcp_sock *tp)
|
2007-11-16 06:39:31 +03:00
|
|
|
{
|
|
|
|
return tcp_is_fack(tp) ? tp->fackets_out : tp->sacked_out + 1;
|
|
|
|
}
|
|
|
|
|
2012-05-02 17:30:04 +04:00
|
|
|
static bool tcp_pause_early_retransmit(struct sock *sk, int flag)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
unsigned long delay;
|
|
|
|
|
|
|
|
/* Delay early retransmit and entering fast recovery for
|
|
|
|
* max(RTT/4, 2msec) unless ack has ECE mark, no RTT samples
|
|
|
|
* available, or RTO is scheduled to fire first.
|
|
|
|
*/
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 14:00:43 +04:00
|
|
|
if (sysctl_tcp_early_retrans < 2 || sysctl_tcp_early_retrans > 3 ||
|
2014-02-27 02:02:48 +04:00
|
|
|
(flag & FLAG_ECE) || !tp->srtt_us)
|
2012-05-02 17:30:04 +04:00
|
|
|
return false;
|
|
|
|
|
2014-02-27 02:02:48 +04:00
|
|
|
delay = max(usecs_to_jiffies(tp->srtt_us >> 5),
|
|
|
|
msecs_to_jiffies(2));
|
|
|
|
|
2012-05-02 17:30:04 +04:00
|
|
|
if (!time_after(inet_csk(sk)->icsk_timeout, (jiffies + delay)))
|
|
|
|
return false;
|
|
|
|
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 14:00:43 +04:00
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_EARLY_RETRANS, delay,
|
|
|
|
TCP_RTO_MAX);
|
2012-05-02 17:30:04 +04:00
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Linux NewReno/SACK/FACK/ECN state machine.
|
|
|
|
* --------------------------------------
|
|
|
|
*
|
|
|
|
* "Open" Normal state, no dubious events, fast path.
|
|
|
|
* "Disorder" In all the respects it is "Open",
|
|
|
|
* but requires a bit more attention. It is entered when
|
|
|
|
* we see some SACKs or dupacks. It is split of "Open"
|
|
|
|
* mainly to move some processing from fast path to slow one.
|
|
|
|
* "CWR" CWND was reduced due to some Congestion Notification event.
|
|
|
|
* It can be ECN, ICMP source quench, local device congestion.
|
|
|
|
* "Recovery" CWND was reduced, we are fast-retransmitting.
|
|
|
|
* "Loss" CWND was reduced due to RTO timeout or SACK reneging.
|
|
|
|
*
|
|
|
|
* tcp_fastretrans_alert() is entered:
|
|
|
|
* - each incoming ACK, if state is not "Open"
|
|
|
|
* - when arrived ACK is unusual, namely:
|
|
|
|
* * SACK
|
|
|
|
* * Duplicate ACK.
|
|
|
|
* * ECN ECE.
|
|
|
|
*
|
|
|
|
* Counting packets in flight is pretty simple.
|
|
|
|
*
|
|
|
|
* in_flight = packets_out - left_out + retrans_out
|
|
|
|
*
|
|
|
|
* packets_out is SND.NXT-SND.UNA counted in packets.
|
|
|
|
*
|
|
|
|
* retrans_out is number of retransmitted segments.
|
|
|
|
*
|
|
|
|
* left_out is number of segments left network, but not ACKed yet.
|
|
|
|
*
|
|
|
|
* left_out = sacked_out + lost_out
|
|
|
|
*
|
|
|
|
* sacked_out: Packets, which arrived to receiver out of order
|
|
|
|
* and hence not ACKed. With SACKs this number is simply
|
|
|
|
* amount of SACKed data. Even without SACKs
|
|
|
|
* it is easy to give pretty reliable estimate of this number,
|
|
|
|
* counting duplicate ACKs.
|
|
|
|
*
|
|
|
|
* lost_out: Packets lost by network. TCP has no explicit
|
|
|
|
* "loss notification" feedback from network (for now).
|
|
|
|
* It means that this number can be only _guessed_.
|
|
|
|
* Actually, it is the heuristics to predict lossage that
|
|
|
|
* distinguishes different algorithms.
|
|
|
|
*
|
|
|
|
* F.e. after RTO, when all the queue is considered as lost,
|
|
|
|
* lost_out = packets_out and in_flight = retrans_out.
|
|
|
|
*
|
|
|
|
* Essentially, we have now two algorithms counting
|
|
|
|
* lost packets.
|
|
|
|
*
|
|
|
|
* FACK: It is the simplest heuristics. As soon as we decided
|
|
|
|
* that something is lost, we decide that _all_ not SACKed
|
|
|
|
* packets until the most forward SACK are lost. I.e.
|
|
|
|
* lost_out = fackets_out - sacked_out and left_out = fackets_out.
|
|
|
|
* It is absolutely correct estimate, if network does not reorder
|
|
|
|
* packets. And it loses any connection to reality when reordering
|
|
|
|
* takes place. We use FACK by default until reordering
|
|
|
|
* is suspected on the path to this destination.
|
|
|
|
*
|
|
|
|
* NewReno: when Recovery is entered, we assume that one segment
|
|
|
|
* is lost (classic Reno). While we are in Recovery and
|
|
|
|
* a partial ACK arrives, we assume that one more packet
|
|
|
|
* is lost (NewReno). This heuristics are the same in NewReno
|
|
|
|
* and SACK.
|
|
|
|
*
|
|
|
|
* Imagine, that's all! Forget about all this shamanism about CWND inflation
|
|
|
|
* deflation etc. CWND is real congestion window, never inflated, changes
|
|
|
|
* only according to classic VJ rules.
|
|
|
|
*
|
|
|
|
* Really tricky (and requiring careful tuning) part of algorithm
|
|
|
|
* is hidden in functions tcp_time_to_recover() and tcp_xmit_retransmit_queue().
|
|
|
|
* The first determines the moment _when_ we should reduce CWND and,
|
|
|
|
* hence, slow down forward transmission. In fact, it determines the moment
|
|
|
|
* when we decide that hole is caused by loss, rather than by a reorder.
|
|
|
|
*
|
|
|
|
* tcp_xmit_retransmit_queue() decides, _what_ we should retransmit to fill
|
|
|
|
* holes, caused by lost packets.
|
|
|
|
*
|
|
|
|
* And the most logically complicated part of algorithm is undo
|
|
|
|
* heuristics. We detect false retransmits due to both too early
|
|
|
|
* fast retransmit (reordering) and underestimated RTO, analyzing
|
|
|
|
* timestamps and D-SACKs. When we detect that some segments were
|
|
|
|
* retransmitted by mistake and CWND reduction was wrong, we undo
|
|
|
|
* window reduction and abort recovery phase. This logic is hidden
|
|
|
|
* inside several functions named tcp_try_undo_<something>.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* This function decides, when we should leave Disordered state
|
|
|
|
* and enter Recovery phase, reducing congestion window.
|
|
|
|
*
|
|
|
|
* Main question: may we further continue forward transmission
|
|
|
|
* with the same cwnd?
|
|
|
|
*/
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_time_to_recover(struct sock *sk, int flag)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
__u32 packets_out;
|
2016-02-03 10:46:52 +03:00
|
|
|
int tcp_reordering = sock_net(sk)->ipv4.sysctl_tcp_reordering;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Trick#1: The loss is proven. */
|
|
|
|
if (tp->lost_out)
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Not-A-Trick#2 : Classic rule... */
|
2009-10-27 06:27:21 +03:00
|
|
|
if (tcp_dupack_heuristics(tp) > tp->reordering)
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Trick#4: It is still not OK... But will it be useful to delay
|
|
|
|
* recovery more?
|
|
|
|
*/
|
|
|
|
packets_out = tp->packets_out;
|
|
|
|
if (packets_out <= tp->reordering &&
|
2016-02-03 10:46:52 +03:00
|
|
|
tp->sacked_out >= max_t(__u32, packets_out/2, tcp_reordering) &&
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
!tcp_may_send_now(sk)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* We have nothing to send. This connection is limited
|
|
|
|
* either by receiver window or by application.
|
|
|
|
*/
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2010-02-18 07:48:19 +03:00
|
|
|
/* If a thin stream is detected, retransmit after first
|
|
|
|
* received dupack. Employ only if SACK is supported in order
|
|
|
|
* to avoid possible corner-case series of spurious retransmissions
|
|
|
|
* Use only if there are no unsent data.
|
|
|
|
*/
|
|
|
|
if ((tp->thin_dupack || sysctl_tcp_thin_dupack) &&
|
|
|
|
tcp_stream_is_thin(tp) && tcp_dupack_heuristics(tp) > 1 &&
|
|
|
|
tcp_is_sack(tp) && !tcp_send_head(sk))
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2010-02-18 07:48:19 +03:00
|
|
|
|
2012-05-02 17:30:03 +04:00
|
|
|
/* Trick#6: TCP early retransmit, per RFC5827. To avoid spurious
|
|
|
|
* retransmissions due to small network reorderings, we implement
|
|
|
|
* Mitigation A.3 in the RFC and delay the retransmission for a short
|
|
|
|
* interval if appropriate.
|
|
|
|
*/
|
|
|
|
if (tp->do_early_retrans && !tp->retrans_out && tp->sacked_out &&
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 14:00:43 +04:00
|
|
|
(tp->packets_out >= (tp->sacked_out + 1) && tp->packets_out < 4) &&
|
2012-05-02 17:30:03 +04:00
|
|
|
!tcp_may_send_now(sk))
|
2012-05-02 17:30:04 +04:00
|
|
|
return !tcp_pause_early_retransmit(sk, flag);
|
2012-05-02 17:30:03 +04:00
|
|
|
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2012-01-19 18:42:21 +04:00
|
|
|
/* Detect loss in event "A" above by marking head of queue up as lost.
|
|
|
|
* For FACK or non-SACK(Reno) senders, the first "packets" number of segments
|
|
|
|
* are considered lost. For RFC3517 SACK, a segment is considered lost if it
|
|
|
|
* has at least tp->reordering SACKed seqments above it; "packets" refers to
|
|
|
|
* the maximum SACKed segments to pass before reaching this limit.
|
2007-11-16 06:39:31 +03:00
|
|
|
*/
|
2010-10-14 05:42:30 +04:00
|
|
|
static void tcp_mark_head_lost(struct sock *sk, int packets, int mark_head)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
struct sk_buff *skb;
|
tcp: fix tcp_mark_head_lost to check skb len before fragmenting
This commit fixes a corner case in tcp_mark_head_lost() which was
causing the WARN_ON(len > skb->len) in tcp_fragment() to fire.
tcp_mark_head_lost() was assuming that if a packet has
tcp_skb_pcount(skb) of N, then it's safe to fragment off a prefix of
M*mss bytes, for any M < N. But with the tricky way TCP pcounts are
maintained, this is not always true.
For example, suppose the sender sends 4 1-byte packets and have the
last 3 packet sacked. It will merge the last 3 packets in the write
queue into an skb with pcount = 3 and len = 3 bytes. If another
recovery happens after a sack reneging event, tcp_mark_head_lost()
may attempt to split the skb assuming it has more than 2*MSS bytes.
This sounds very counterintuitive, but as the commit description for
the related commit c0638c247f55 ("tcp: don't fragment SACKed skbs in
tcp_mark_head_lost()") notes, this is because tcp_shifted_skb()
coalesces adjacent regions of SACKed skbs, and when doing this it
preserves the sum of their packet counts in order to reflect the
real-world dynamics on the wire. The c0638c247f55 commit tried to
avoid problems by not fragmenting SACKed skbs, since SACKed skbs are
where the non-proportionality between pcount and skb->len/mss is known
to be possible. However, that commit did not handle the case where
during a reneging event one of these weird SACKed skbs becomes an
un-SACKed skb, which tcp_mark_head_lost() can then try to fragment.
The fix is to simply mark the entire skb lost when this happens.
This makes the recovery slightly more aggressive in such corner
cases before we detect reordering. But once we detect reordering
this code path is by-passed because FACK is disabled.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-01-26 01:01:53 +03:00
|
|
|
int cnt, oldcnt, lost;
|
2008-04-08 09:32:38 +04:00
|
|
|
unsigned int mss;
|
2012-01-19 18:42:21 +04:00
|
|
|
/* Use SACK to deduce losses of new sequences sent during recovery */
|
|
|
|
const u32 loss_high = tcp_is_sack(tp) ? tp->snd_nxt : tp->high_seq;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-07-26 08:43:18 +04:00
|
|
|
WARN_ON(packets > tp->packets_out);
|
2005-11-11 04:14:59 +03:00
|
|
|
if (tp->lost_skb_hint) {
|
|
|
|
skb = tp->lost_skb_hint;
|
|
|
|
cnt = tp->lost_cnt_hint;
|
2010-10-14 05:42:30 +04:00
|
|
|
/* Head already handled? */
|
|
|
|
if (mark_head && skb != tcp_write_queue_head(sk))
|
|
|
|
return;
|
2005-11-11 04:14:59 +03:00
|
|
|
} else {
|
2007-03-07 23:12:44 +03:00
|
|
|
skb = tcp_write_queue_head(sk);
|
2005-11-11 04:14:59 +03:00
|
|
|
cnt = 0;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2007-03-07 23:12:44 +03:00
|
|
|
tcp_for_write_queue_from(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
2005-11-11 04:14:59 +03:00
|
|
|
/* TODO: do this better */
|
|
|
|
/* this is not the most efficient way to do this... */
|
|
|
|
tp->lost_skb_hint = skb;
|
|
|
|
tp->lost_cnt_hint = cnt;
|
2007-11-16 06:39:31 +03:00
|
|
|
|
2012-01-19 18:42:21 +04:00
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, loss_high))
|
2008-04-08 09:32:38 +04:00
|
|
|
break;
|
|
|
|
|
|
|
|
oldcnt = cnt;
|
2008-01-31 07:06:02 +03:00
|
|
|
if (tcp_is_fack(tp) || tcp_is_reno(tp) ||
|
2007-11-16 06:39:31 +03:00
|
|
|
(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED))
|
|
|
|
cnt += tcp_skb_pcount(skb);
|
|
|
|
|
2008-04-08 09:32:38 +04:00
|
|
|
if (cnt > packets) {
|
2010-09-24 17:22:06 +04:00
|
|
|
if ((tcp_is_sack(tp) && !tcp_is_fack(tp)) ||
|
2012-03-03 01:36:51 +04:00
|
|
|
(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED) ||
|
2010-09-24 17:22:06 +04:00
|
|
|
(oldcnt >= packets))
|
2008-04-08 09:32:38 +04:00
|
|
|
break;
|
|
|
|
|
2015-06-11 19:15:18 +03:00
|
|
|
mss = tcp_skb_mss(skb);
|
tcp: fix tcp_mark_head_lost to check skb len before fragmenting
This commit fixes a corner case in tcp_mark_head_lost() which was
causing the WARN_ON(len > skb->len) in tcp_fragment() to fire.
tcp_mark_head_lost() was assuming that if a packet has
tcp_skb_pcount(skb) of N, then it's safe to fragment off a prefix of
M*mss bytes, for any M < N. But with the tricky way TCP pcounts are
maintained, this is not always true.
For example, suppose the sender sends 4 1-byte packets and have the
last 3 packet sacked. It will merge the last 3 packets in the write
queue into an skb with pcount = 3 and len = 3 bytes. If another
recovery happens after a sack reneging event, tcp_mark_head_lost()
may attempt to split the skb assuming it has more than 2*MSS bytes.
This sounds very counterintuitive, but as the commit description for
the related commit c0638c247f55 ("tcp: don't fragment SACKed skbs in
tcp_mark_head_lost()") notes, this is because tcp_shifted_skb()
coalesces adjacent regions of SACKed skbs, and when doing this it
preserves the sum of their packet counts in order to reflect the
real-world dynamics on the wire. The c0638c247f55 commit tried to
avoid problems by not fragmenting SACKed skbs, since SACKed skbs are
where the non-proportionality between pcount and skb->len/mss is known
to be possible. However, that commit did not handle the case where
during a reneging event one of these weird SACKed skbs becomes an
un-SACKed skb, which tcp_mark_head_lost() can then try to fragment.
The fix is to simply mark the entire skb lost when this happens.
This makes the recovery slightly more aggressive in such corner
cases before we detect reordering. But once we detect reordering
this code path is by-passed because FACK is disabled.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-01-26 01:01:53 +03:00
|
|
|
/* If needed, chop off the prefix to mark as lost. */
|
|
|
|
lost = (packets - oldcnt) * mss;
|
|
|
|
if (lost < skb->len &&
|
|
|
|
tcp_fragment(sk, skb, lost, mss, GFP_ATOMIC) < 0)
|
2008-04-08 09:32:38 +04:00
|
|
|
break;
|
|
|
|
cnt = packets;
|
|
|
|
}
|
|
|
|
|
2008-09-21 08:19:22 +04:00
|
|
|
tcp_skb_mark_lost(tp, skb);
|
2010-10-14 05:42:30 +04:00
|
|
|
|
|
|
|
if (mark_head)
|
|
|
|
break;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2007-08-09 15:44:16 +04:00
|
|
|
tcp_verify_left_out(tp);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Account newly detected lost packet(s) */
|
|
|
|
|
2007-11-16 06:39:31 +03:00
|
|
|
static void tcp_update_scoreboard(struct sock *sk, int fast_rexmit)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2007-11-16 06:39:31 +03:00
|
|
|
if (tcp_is_reno(tp)) {
|
2010-10-14 05:42:30 +04:00
|
|
|
tcp_mark_head_lost(sk, 1, 1);
|
2007-11-16 06:39:31 +03:00
|
|
|
} else if (tcp_is_fack(tp)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
int lost = tp->fackets_out - tp->reordering;
|
|
|
|
if (lost <= 0)
|
|
|
|
lost = 1;
|
2010-10-14 05:42:30 +04:00
|
|
|
tcp_mark_head_lost(sk, lost, 0);
|
2005-04-17 02:20:36 +04:00
|
|
|
} else {
|
2007-11-16 06:39:31 +03:00
|
|
|
int sacked_upto = tp->sacked_out - tp->reordering;
|
2010-10-14 05:42:30 +04:00
|
|
|
if (sacked_upto >= 0)
|
|
|
|
tcp_mark_head_lost(sk, sacked_upto, 0);
|
|
|
|
else if (fast_rexmit)
|
|
|
|
tcp_mark_head_lost(sk, 1, 1);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-10-17 07:57:44 +03:00
|
|
|
static bool tcp_tsopt_ecr_before(const struct tcp_sock *tp, u32 when)
|
|
|
|
{
|
|
|
|
return tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
|
|
|
|
before(tp->rx_opt.rcv_tsecr, when);
|
|
|
|
}
|
|
|
|
|
2015-10-17 07:57:46 +03:00
|
|
|
/* skb is spurious retransmitted if the returned timestamp echo
|
|
|
|
* reply is prior to the skb transmission time
|
|
|
|
*/
|
|
|
|
static bool tcp_skb_spurious_retrans(const struct tcp_sock *tp,
|
|
|
|
const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
return (TCP_SKB_CB(skb)->sacked & TCPCB_RETRANS) &&
|
|
|
|
tcp_tsopt_ecr_before(tp, tcp_skb_timestamp(skb));
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Nothing was retransmitted or returned timestamp is less
|
|
|
|
* than timestamp of the first retransmission.
|
|
|
|
*/
|
2012-07-20 01:32:18 +04:00
|
|
|
static inline bool tcp_packet_delayed(const struct tcp_sock *tp)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
return !tp->retrans_stamp ||
|
2015-10-17 07:57:44 +03:00
|
|
|
tcp_tsopt_ecr_before(tp, tp->retrans_stamp);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Undo procedures. */
|
|
|
|
|
2014-11-04 22:15:08 +03:00
|
|
|
/* We can clear retrans_stamp when there are no retransmissions in the
|
|
|
|
* window. It would seem that it is trivially available for us in
|
|
|
|
* tp->retrans_out, however, that kind of assumptions doesn't consider
|
|
|
|
* what will happen if errors occur when sending retransmission for the
|
|
|
|
* second time. ...It could the that such segment has only
|
|
|
|
* TCPCB_EVER_RETRANS set at the present time. It seems that checking
|
|
|
|
* the head skb is enough except for some reneging corner cases that
|
|
|
|
* are not worth the effort.
|
|
|
|
*
|
|
|
|
* Main reason for all this complexity is the fact that connection dying
|
|
|
|
* time now depends on the validity of the retrans_stamp, in particular,
|
|
|
|
* that successive retransmissions of a segment must not advance
|
|
|
|
* retrans_stamp under any conditions.
|
|
|
|
*/
|
|
|
|
static bool tcp_any_retrans_done(const struct sock *sk)
|
|
|
|
{
|
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
if (tp->retrans_out)
|
|
|
|
return true;
|
|
|
|
|
|
|
|
skb = tcp_write_queue_head(sk);
|
|
|
|
if (unlikely(skb && TCP_SKB_CB(skb)->sacked & TCPCB_EVER_RETRANS))
|
|
|
|
return true;
|
|
|
|
|
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#if FASTRETRANS_DEBUG > 1
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
static void DBGUNDO(struct sock *sk, const char *msg)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
struct inet_sock *inet = inet_sk(sk);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
|
2008-04-14 15:09:36 +04:00
|
|
|
if (sk->sk_family == AF_INET) {
|
2012-05-15 18:11:54 +04:00
|
|
|
pr_debug("Undo %s %pI4/%u c%u l%u ss%u/%u p%u\n",
|
|
|
|
msg,
|
|
|
|
&inet->inet_daddr, ntohs(inet->inet_dport),
|
|
|
|
tp->snd_cwnd, tcp_left_out(tp),
|
|
|
|
tp->snd_ssthresh, tp->prior_ssthresh,
|
|
|
|
tp->packets_out);
|
2008-04-14 15:09:36 +04:00
|
|
|
}
|
2011-12-10 13:48:31 +04:00
|
|
|
#if IS_ENABLED(CONFIG_IPV6)
|
2008-04-14 15:09:36 +04:00
|
|
|
else if (sk->sk_family == AF_INET6) {
|
|
|
|
struct ipv6_pinfo *np = inet6_sk(sk);
|
2012-05-15 18:11:54 +04:00
|
|
|
pr_debug("Undo %s %pI6/%u c%u l%u ss%u/%u p%u\n",
|
|
|
|
msg,
|
|
|
|
&np->daddr, ntohs(inet->inet_dport),
|
|
|
|
tp->snd_cwnd, tcp_left_out(tp),
|
|
|
|
tp->snd_ssthresh, tp->prior_ssthresh,
|
|
|
|
tp->packets_out);
|
2008-04-14 15:09:36 +04:00
|
|
|
}
|
|
|
|
#endif
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
#else
|
|
|
|
#define DBGUNDO(x...) do { } while (0)
|
|
|
|
#endif
|
|
|
|
|
2013-05-29 18:20:13 +04:00
|
|
|
static void tcp_undo_cwnd_reduction(struct sock *sk, bool unmark_loss)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 11:03:31 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2013-05-29 18:20:12 +04:00
|
|
|
if (unmark_loss) {
|
|
|
|
struct sk_buff *skb;
|
|
|
|
|
|
|
|
tcp_for_write_queue(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_LOST;
|
|
|
|
}
|
|
|
|
tp->lost_out = 0;
|
|
|
|
tcp_clear_all_retrans_hints(tp);
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
if (tp->prior_ssthresh) {
|
2005-08-10 11:03:31 +04:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
if (icsk->icsk_ca_ops->undo_cwnd)
|
|
|
|
tp->snd_cwnd = icsk->icsk_ca_ops->undo_cwnd(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
else
|
2008-01-01 01:57:14 +03:00
|
|
|
tp->snd_cwnd = max(tp->snd_cwnd, tp->snd_ssthresh << 1);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-05-29 18:20:13 +04:00
|
|
|
if (tp->prior_ssthresh > tp->snd_ssthresh) {
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->snd_ssthresh = tp->prior_ssthresh;
|
2014-09-29 15:08:30 +04:00
|
|
|
tcp_ecn_withdraw_cwr(tp);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
2013-05-29 18:20:13 +04:00
|
|
|
tp->undo_marker = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2012-07-20 01:32:18 +04:00
|
|
|
static inline bool tcp_may_undo(const struct tcp_sock *tp)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2008-01-01 01:57:14 +03:00
|
|
|
return tp->undo_marker && (!tp->undo_retrans || tcp_packet_delayed(tp));
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* People celebrate: "We love our President!" */
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_try_undo_recovery(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
if (tcp_may_undo(tp)) {
|
2008-07-03 12:05:41 +04:00
|
|
|
int mib_idx;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Happy end! We did not retransmit anything
|
|
|
|
* or our original transmission succeeded.
|
|
|
|
*/
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
DBGUNDO(sk, inet_csk(sk)->icsk_ca_state == TCP_CA_Loss ? "loss" : "retrans");
|
2013-05-29 18:20:13 +04:00
|
|
|
tcp_undo_cwnd_reduction(sk, false);
|
2005-08-10 11:03:31 +04:00
|
|
|
if (inet_csk(sk)->icsk_ca_state == TCP_CA_Loss)
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPLOSSUNDO;
|
2005-04-17 02:20:36 +04:00
|
|
|
else
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPFULLUNDO;
|
|
|
|
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), mib_idx);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2007-08-09 16:14:46 +04:00
|
|
|
if (tp->snd_una == tp->high_seq && tcp_is_reno(tp)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Hold old state until something *above* high_seq
|
|
|
|
* is ACKed. For Reno it is MUST to prevent false
|
|
|
|
* fast retransmits (RFC2582). SACK TCP is safe. */
|
2014-11-04 22:15:08 +03:00
|
|
|
if (!tcp_any_retrans_done(sk))
|
|
|
|
tp->retrans_stamp = 0;
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2005-08-10 11:03:31 +04:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Open);
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Try to undo cwnd reduction, because D-SACKs acked all retransmitted data */
|
2013-05-29 18:20:14 +04:00
|
|
|
static bool tcp_try_undo_dsack(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
if (tp->undo_marker && !tp->undo_retrans) {
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
DBGUNDO(sk, "D-SACK");
|
2013-05-29 18:20:13 +04:00
|
|
|
tcp_undo_cwnd_reduction(sk, false);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPDSACKUNDO);
|
2013-05-29 18:20:14 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2013-05-29 18:20:14 +04:00
|
|
|
return false;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2013-03-20 17:33:00 +04:00
|
|
|
/* Undo during loss recovery after partial ACK or using F-RTO. */
|
|
|
|
static bool tcp_try_undo_loss(struct sock *sk, bool frto_undo)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2013-03-20 17:33:00 +04:00
|
|
|
if (frto_undo || tcp_may_undo(tp)) {
|
2013-05-29 18:20:13 +04:00
|
|
|
tcp_undo_cwnd_reduction(sk, true);
|
2005-11-11 04:14:59 +03:00
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
DBGUNDO(sk, "partial loss");
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPLOSSUNDO);
|
2013-03-20 17:33:00 +04:00
|
|
|
if (frto_undo)
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk),
|
|
|
|
LINUX_MIB_TCPSPURIOUSRTOS);
|
2005-08-10 07:10:42 +04:00
|
|
|
inet_csk(sk)->icsk_retransmits = 0;
|
2013-03-20 17:33:00 +04:00
|
|
|
if (frto_undo || tcp_is_sack(tp))
|
2005-08-10 11:03:31 +04:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Open);
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2015-07-02 00:11:15 +03:00
|
|
|
/* The cwnd reduction in CWR and Recovery uses the PRR algorithm in RFC 6937.
|
2012-09-02 21:38:03 +04:00
|
|
|
* It computes the number of packets to send (sndcnt) based on packets newly
|
|
|
|
* delivered:
|
|
|
|
* 1) If the packets in flight is larger than ssthresh, PRR spreads the
|
|
|
|
* cwnd reductions across a full RTT.
|
2015-07-02 00:11:15 +03:00
|
|
|
* 2) Otherwise PRR uses packet conservation to send as much as delivered.
|
|
|
|
* But when the retransmits are acked without further losses, PRR
|
|
|
|
* slow starts cwnd up to ssthresh to speed up the recovery.
|
2012-09-02 21:38:03 +04:00
|
|
|
*/
|
2014-07-14 18:58:32 +04:00
|
|
|
static void tcp_init_cwnd_reduction(struct sock *sk)
|
2012-09-02 21:38:04 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
2013-03-11 14:00:44 +04:00
|
|
|
tp->tlp_high_seq = 0;
|
2012-09-02 21:38:04 +04:00
|
|
|
tp->snd_cwnd_cnt = 0;
|
|
|
|
tp->prior_cwnd = tp->snd_cwnd;
|
|
|
|
tp->prr_delivered = 0;
|
|
|
|
tp->prr_out = 0;
|
2014-07-14 18:58:32 +04:00
|
|
|
tp->snd_ssthresh = inet_csk(sk)->icsk_ca_ops->ssthresh(sk);
|
2014-09-29 15:08:30 +04:00
|
|
|
tcp_ecn_queue_cwr(tp);
|
2012-09-02 21:38:04 +04:00
|
|
|
}
|
|
|
|
|
2016-02-02 21:33:05 +03:00
|
|
|
static void tcp_cwnd_reduction(struct sock *sk, int newly_acked_sacked,
|
|
|
|
int flag)
|
2012-09-02 21:38:03 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int sndcnt = 0;
|
|
|
|
int delta = tp->snd_ssthresh - tcp_packets_in_flight(tp);
|
|
|
|
|
2016-01-06 23:42:38 +03:00
|
|
|
if (newly_acked_sacked <= 0 || WARN_ON_ONCE(!tp->prior_cwnd))
|
|
|
|
return;
|
|
|
|
|
2012-09-02 21:38:04 +04:00
|
|
|
tp->prr_delivered += newly_acked_sacked;
|
2015-07-02 00:11:15 +03:00
|
|
|
if (delta < 0) {
|
2012-09-02 21:38:03 +04:00
|
|
|
u64 dividend = (u64)tp->snd_ssthresh * tp->prr_delivered +
|
|
|
|
tp->prior_cwnd - 1;
|
|
|
|
sndcnt = div_u64(dividend, tp->prior_cwnd) - tp->prr_out;
|
2015-07-02 00:11:15 +03:00
|
|
|
} else if ((flag & FLAG_RETRANS_DATA_ACKED) &&
|
|
|
|
!(flag & FLAG_LOST_RETRANS)) {
|
2012-09-02 21:38:03 +04:00
|
|
|
sndcnt = min_t(int, delta,
|
|
|
|
max_t(int, tp->prr_delivered - tp->prr_out,
|
|
|
|
newly_acked_sacked) + 1);
|
2015-07-02 00:11:15 +03:00
|
|
|
} else {
|
|
|
|
sndcnt = min(delta, newly_acked_sacked);
|
2012-09-02 21:38:03 +04:00
|
|
|
}
|
2016-02-02 21:33:05 +03:00
|
|
|
/* Force a fast retransmit upon entering fast recovery */
|
|
|
|
sndcnt = max(sndcnt, (tp->prr_out ? 0 : 1));
|
2012-09-02 21:38:03 +04:00
|
|
|
tp->snd_cwnd = tcp_packets_in_flight(tp) + sndcnt;
|
|
|
|
}
|
|
|
|
|
2012-09-02 21:38:04 +04:00
|
|
|
static inline void tcp_end_cwnd_reduction(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 11:03:31 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2011-08-22 00:21:57 +04:00
|
|
|
|
2012-09-02 21:38:04 +04:00
|
|
|
/* Reset cwnd to ssthresh in CWR or Recovery (unless it's undone) */
|
|
|
|
if (inet_csk(sk)->icsk_ca_state == TCP_CA_CWR ||
|
|
|
|
(tp->undo_marker && tp->snd_ssthresh < TCP_INFINITE_SSTHRESH)) {
|
|
|
|
tp->snd_cwnd = tp->snd_ssthresh;
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
2011-03-14 13:57:03 +03:00
|
|
|
}
|
2005-08-10 11:03:31 +04:00
|
|
|
tcp_ca_event(sk, CA_EVENT_COMPLETE_CWR);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2012-09-02 21:38:04 +04:00
|
|
|
/* Enter CWR state. Disable cwnd undo since congestion is proven with ECN */
|
2014-07-14 18:58:32 +04:00
|
|
|
void tcp_enter_cwr(struct sock *sk)
|
2012-09-02 21:38:02 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
tp->prior_ssthresh = 0;
|
2012-09-02 21:38:04 +04:00
|
|
|
if (inet_csk(sk)->icsk_ca_state < TCP_CA_CWR) {
|
2012-09-02 21:38:02 +04:00
|
|
|
tp->undo_marker = 0;
|
2014-07-14 18:58:32 +04:00
|
|
|
tcp_init_cwnd_reduction(sk);
|
2012-09-02 21:38:02 +04:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_CWR);
|
|
|
|
}
|
|
|
|
}
|
2015-06-10 20:08:16 +03:00
|
|
|
EXPORT_SYMBOL(tcp_enter_cwr);
|
2012-09-02 21:38:02 +04:00
|
|
|
|
2008-06-04 22:34:22 +04:00
|
|
|
static void tcp_try_keep_open(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int state = TCP_CA_Open;
|
|
|
|
|
2011-11-16 12:58:04 +04:00
|
|
|
if (tcp_left_out(tp) || tcp_any_retrans_done(sk))
|
2008-06-04 22:34:22 +04:00
|
|
|
state = TCP_CA_Disorder;
|
|
|
|
|
|
|
|
if (inet_csk(sk)->icsk_ca_state != state) {
|
|
|
|
tcp_set_ca_state(sk, state);
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2016-02-02 21:33:05 +03:00
|
|
|
static void tcp_try_to_open(struct sock *sk, int flag)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2007-08-09 15:45:17 +04:00
|
|
|
tcp_verify_left_out(tp);
|
|
|
|
|
2013-03-20 17:32:58 +04:00
|
|
|
if (!tcp_any_retrans_done(sk))
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
if (flag & FLAG_ECE)
|
2014-07-14 18:58:32 +04:00
|
|
|
tcp_enter_cwr(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2005-08-10 11:03:31 +04:00
|
|
|
if (inet_csk(sk)->icsk_ca_state != TCP_CA_CWR) {
|
2008-06-04 22:34:22 +04:00
|
|
|
tcp_try_keep_open(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2006-03-21 04:53:41 +03:00
|
|
|
static void tcp_mtup_probe_failed(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
icsk->icsk_mtup.search_high = icsk->icsk_mtup.probe_size - 1;
|
|
|
|
icsk->icsk_mtup.probe_size = 0;
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMTUPFAIL);
|
2006-03-21 04:53:41 +03:00
|
|
|
}
|
|
|
|
|
2009-03-14 17:23:04 +03:00
|
|
|
static void tcp_mtup_probe_success(struct sock *sk)
|
2006-03-21 04:53:41 +03:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
/* FIXME: breaks with very large cwnd */
|
|
|
|
tp->prior_ssthresh = tcp_current_ssthresh(sk);
|
|
|
|
tp->snd_cwnd = tp->snd_cwnd *
|
|
|
|
tcp_mss_to_mtu(sk, tp->mss_cache) /
|
|
|
|
icsk->icsk_mtup.probe_size;
|
|
|
|
tp->snd_cwnd_cnt = 0;
|
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
2010-10-07 08:18:02 +04:00
|
|
|
tp->snd_ssthresh = tcp_current_ssthresh(sk);
|
2006-03-21 04:53:41 +03:00
|
|
|
|
|
|
|
icsk->icsk_mtup.search_low = icsk->icsk_mtup.probe_size;
|
|
|
|
icsk->icsk_mtup.probe_size = 0;
|
|
|
|
tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPMTUPSUCCESS);
|
2006-03-21 04:53:41 +03:00
|
|
|
}
|
|
|
|
|
2008-11-25 08:11:55 +03:00
|
|
|
/* Do a simple retransmit without using the backoff mechanisms in
|
|
|
|
* tcp_timer. This is used for path mtu discovery.
|
|
|
|
* The socket is already locked here.
|
|
|
|
*/
|
|
|
|
void tcp_simple_retransmit(struct sock *sk)
|
|
|
|
{
|
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb;
|
2009-03-14 17:23:05 +03:00
|
|
|
unsigned int mss = tcp_current_mss(sk);
|
2008-11-25 08:11:55 +03:00
|
|
|
u32 prior_lost = tp->lost_out;
|
|
|
|
|
|
|
|
tcp_for_write_queue(skb, sk) {
|
|
|
|
if (skb == tcp_send_head(sk))
|
|
|
|
break;
|
2008-12-06 09:41:26 +03:00
|
|
|
if (tcp_skb_seglen(skb) > mss &&
|
2008-11-25 08:11:55 +03:00
|
|
|
!(TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED)) {
|
|
|
|
if (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_RETRANS) {
|
|
|
|
TCP_SKB_CB(skb)->sacked &= ~TCPCB_SACKED_RETRANS;
|
|
|
|
tp->retrans_out -= tcp_skb_pcount(skb);
|
|
|
|
}
|
|
|
|
tcp_skb_mark_lost_uncond_verify(tp, skb);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
tcp_clear_retrans_hints_partial(tp);
|
|
|
|
|
|
|
|
if (prior_lost == tp->lost_out)
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (tcp_is_reno(tp))
|
|
|
|
tcp_limit_reno_sacked(tp);
|
|
|
|
|
|
|
|
tcp_verify_left_out(tp);
|
|
|
|
|
|
|
|
/* Don't muck with the congestion window here.
|
|
|
|
* Reason is that we do not increase amount of _data_
|
|
|
|
* in network, but units changed and effective
|
|
|
|
* cwnd/ssthresh really reduced now.
|
|
|
|
*/
|
|
|
|
if (icsk->icsk_ca_state != TCP_CA_Loss) {
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
|
|
|
tp->snd_ssthresh = tcp_current_ssthresh(sk);
|
|
|
|
tp->prior_ssthresh = 0;
|
|
|
|
tp->undo_marker = 0;
|
|
|
|
tcp_set_ca_state(sk, TCP_CA_Loss);
|
|
|
|
}
|
|
|
|
tcp_xmit_retransmit_queue(sk);
|
|
|
|
}
|
2010-07-10 01:22:10 +04:00
|
|
|
EXPORT_SYMBOL(tcp_simple_retransmit);
|
2008-11-25 08:11:55 +03:00
|
|
|
|
2012-05-02 17:30:02 +04:00
|
|
|
static void tcp_enter_recovery(struct sock *sk, bool ece_ack)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int mib_idx;
|
|
|
|
|
|
|
|
if (tcp_is_reno(tp))
|
|
|
|
mib_idx = LINUX_MIB_TCPRENORECOVERY;
|
|
|
|
else
|
|
|
|
mib_idx = LINUX_MIB_TCPSACKRECOVERY;
|
|
|
|
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), mib_idx);
|
2012-05-02 17:30:02 +04:00
|
|
|
|
|
|
|
tp->prior_ssthresh = 0;
|
2014-08-23 01:15:22 +04:00
|
|
|
tcp_init_undo(tp);
|
2012-05-02 17:30:02 +04:00
|
|
|
|
2015-07-02 00:11:14 +03:00
|
|
|
if (!tcp_in_cwnd_reduction(sk)) {
|
2012-05-02 17:30:02 +04:00
|
|
|
if (!ece_ack)
|
|
|
|
tp->prior_ssthresh = tcp_current_ssthresh(sk);
|
2014-07-14 18:58:32 +04:00
|
|
|
tcp_init_cwnd_reduction(sk);
|
2012-05-02 17:30:02 +04:00
|
|
|
}
|
|
|
|
tcp_set_ca_state(sk, TCP_CA_Recovery);
|
|
|
|
}
|
|
|
|
|
2013-03-20 17:32:59 +04:00
|
|
|
/* Process an ACK in CA_Loss state. Move to CA_Open if lost data are
|
|
|
|
* recovered or spurious. Otherwise retransmits more on partial ACKs.
|
|
|
|
*/
|
2016-02-02 21:33:04 +03:00
|
|
|
static void tcp_process_loss(struct sock *sk, int flag, bool is_dupack,
|
|
|
|
int *rexmit)
|
2013-03-20 17:32:59 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2013-03-20 17:33:00 +04:00
|
|
|
bool recovered = !before(tp->snd_una, tp->high_seq);
|
2013-03-20 17:32:59 +04:00
|
|
|
|
2015-05-18 22:31:44 +03:00
|
|
|
if ((flag & FLAG_SND_UNA_ADVANCED) &&
|
|
|
|
tcp_try_undo_loss(sk, false))
|
|
|
|
return;
|
|
|
|
|
2013-03-20 17:33:00 +04:00
|
|
|
if (tp->frto) { /* F-RTO RFC5682 sec 3.1 (sack enhanced version). */
|
2014-05-31 02:25:59 +04:00
|
|
|
/* Step 3.b. A timeout is spurious if not all data are
|
|
|
|
* lost, i.e., never-retransmitted data are (s)acked.
|
|
|
|
*/
|
2015-05-18 22:31:44 +03:00
|
|
|
if ((flag & FLAG_ORIG_SACK_ACKED) &&
|
|
|
|
tcp_try_undo_loss(sk, true))
|
2013-03-20 17:33:00 +04:00
|
|
|
return;
|
2014-05-31 02:25:59 +04:00
|
|
|
|
2015-05-18 22:31:45 +03:00
|
|
|
if (after(tp->snd_nxt, tp->high_seq)) {
|
|
|
|
if (flag & FLAG_DATA_SACKED || is_dupack)
|
|
|
|
tp->frto = 0; /* Step 3.a. loss was real */
|
2013-03-20 17:33:00 +04:00
|
|
|
} else if (flag & FLAG_SND_UNA_ADVANCED && !recovered) {
|
|
|
|
tp->high_seq = tp->snd_nxt;
|
2016-02-02 21:33:04 +03:00
|
|
|
/* Step 2.b. Try send new data (but deferred until cwnd
|
|
|
|
* is updated in tcp_ack()). Otherwise fall back to
|
|
|
|
* the conventional recovery.
|
|
|
|
*/
|
|
|
|
if (tcp_send_head(sk) &&
|
|
|
|
after(tcp_wnd_end(tp), tp->snd_nxt)) {
|
|
|
|
*rexmit = REXMIT_NEW;
|
|
|
|
return;
|
|
|
|
}
|
2013-03-20 17:33:00 +04:00
|
|
|
tp->frto = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (recovered) {
|
|
|
|
/* F-RTO RFC5682 sec 3.1 step 2.a and 1st part of step 3.a */
|
2013-03-20 17:32:59 +04:00
|
|
|
tcp_try_undo_recovery(sk);
|
|
|
|
return;
|
|
|
|
}
|
2013-03-20 17:33:00 +04:00
|
|
|
if (tcp_is_reno(tp)) {
|
|
|
|
/* A Reno DUPACK means new data in F-RTO step 2.b above are
|
|
|
|
* delivered. Lower inflight to clock out (re)tranmissions.
|
|
|
|
*/
|
|
|
|
if (after(tp->snd_nxt, tp->high_seq) && is_dupack)
|
|
|
|
tcp_add_reno_sack(sk);
|
|
|
|
else if (flag & FLAG_SND_UNA_ADVANCED)
|
|
|
|
tcp_reset_reno_sack(tp);
|
|
|
|
}
|
2016-02-02 21:33:04 +03:00
|
|
|
*rexmit = REXMIT_LOST;
|
2013-03-20 17:32:59 +04:00
|
|
|
}
|
|
|
|
|
2013-05-29 18:20:12 +04:00
|
|
|
/* Undo during fast recovery after partial ACK. */
|
2016-02-02 21:33:05 +03:00
|
|
|
static bool tcp_try_undo_partial(struct sock *sk, const int acked)
|
2013-05-29 18:20:12 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2013-05-29 18:20:13 +04:00
|
|
|
if (tp->undo_marker && tcp_packet_delayed(tp)) {
|
2013-05-29 18:20:12 +04:00
|
|
|
/* Plain luck! Hole if filled with delayed
|
|
|
|
* packet, rather than with a retransmit.
|
|
|
|
*/
|
2013-05-29 18:20:13 +04:00
|
|
|
tcp_update_reordering(sk, tcp_fackets_out(tp) + acked, 1);
|
|
|
|
|
|
|
|
/* We are getting evidence that the reordering degree is higher
|
|
|
|
* than we realized. If there are no retransmits out then we
|
|
|
|
* can undo. Otherwise we clock out new packets but do not
|
|
|
|
* mark more packets lost or retransmit more.
|
|
|
|
*/
|
2016-02-02 21:33:05 +03:00
|
|
|
if (tp->retrans_out)
|
2013-05-29 18:20:13 +04:00
|
|
|
return true;
|
|
|
|
|
2013-05-29 18:20:12 +04:00
|
|
|
if (!tcp_any_retrans_done(sk))
|
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
|
2013-05-29 18:20:13 +04:00
|
|
|
DBGUNDO(sk, "partial recovery");
|
|
|
|
tcp_undo_cwnd_reduction(sk, true);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPPARTIALUNDO);
|
2013-05-29 18:20:13 +04:00
|
|
|
tcp_try_keep_open(sk);
|
|
|
|
return true;
|
2013-05-29 18:20:12 +04:00
|
|
|
}
|
2013-05-29 18:20:13 +04:00
|
|
|
return false;
|
2013-05-29 18:20:12 +04:00
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Process an event, which can update packets-in-flight not trivially.
|
|
|
|
* Main goal of this function is to calculate new estimate for left_out,
|
|
|
|
* taking into account both packets sitting in receiver's buffer and
|
|
|
|
* packets lost by network.
|
|
|
|
*
|
2016-02-02 21:33:05 +03:00
|
|
|
* Besides that it updates the congestion state when packet loss or ECN
|
|
|
|
* is detected. But it does not reduce the cwnd, it is done by the
|
|
|
|
* congestion control later.
|
2005-04-17 02:20:36 +04:00
|
|
|
*
|
|
|
|
* It does _not_ decide what to send, it is made in function
|
|
|
|
* tcp_xmit_retransmit_queue().
|
|
|
|
*/
|
2013-05-29 18:20:11 +04:00
|
|
|
static void tcp_fastretrans_alert(struct sock *sk, const int acked,
|
2016-02-02 21:33:05 +03:00
|
|
|
bool is_dupack, int *ack_flag, int *rexmit)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 11:03:31 +04:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2016-02-02 21:33:05 +03:00
|
|
|
int fast_rexmit = 0, flag = *ack_flag;
|
2013-05-29 18:20:12 +04:00
|
|
|
bool do_lost = is_dupack || ((flag & FLAG_DATA_SACKED) &&
|
2007-11-16 06:39:31 +03:00
|
|
|
(tcp_fackets_out(tp) > tp->reordering));
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2007-12-31 15:43:32 +03:00
|
|
|
if (WARN_ON(!tp->packets_out && tp->sacked_out))
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->sacked_out = 0;
|
2007-10-09 12:24:15 +04:00
|
|
|
if (WARN_ON(!tp->sacked_out && tp->fackets_out))
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->fackets_out = 0;
|
|
|
|
|
2007-02-09 17:24:47 +03:00
|
|
|
/* Now state machine starts.
|
2005-04-17 02:20:36 +04:00
|
|
|
* A. ECE, hence prohibit cwnd undoing, the reduction is required. */
|
2008-01-01 01:57:14 +03:00
|
|
|
if (flag & FLAG_ECE)
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->prior_ssthresh = 0;
|
|
|
|
|
|
|
|
/* B. In all the states check for reneging SACKs. */
|
2007-12-31 15:49:21 +03:00
|
|
|
if (tcp_check_sack_reneging(sk, flag))
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
|
|
|
|
2012-01-19 18:42:21 +04:00
|
|
|
/* C. Check consistency of the current state. */
|
2007-08-09 15:44:16 +04:00
|
|
|
tcp_verify_left_out(tp);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2012-01-19 18:42:21 +04:00
|
|
|
/* D. Check state exit conditions. State can be terminated
|
2005-04-17 02:20:36 +04:00
|
|
|
* when high_seq is ACKed. */
|
2005-08-10 11:03:31 +04:00
|
|
|
if (icsk->icsk_ca_state == TCP_CA_Open) {
|
2008-07-26 08:43:18 +04:00
|
|
|
WARN_ON(tp->retrans_out != 0);
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
} else if (!before(tp->snd_una, tp->high_seq)) {
|
2005-08-10 11:03:31 +04:00
|
|
|
switch (icsk->icsk_ca_state) {
|
2005-04-17 02:20:36 +04:00
|
|
|
case TCP_CA_CWR:
|
|
|
|
/* CWR is to be held something *above* high_seq
|
|
|
|
* is ACKed for CWR bit to reach receiver. */
|
|
|
|
if (tp->snd_una != tp->high_seq) {
|
2012-09-02 21:38:04 +04:00
|
|
|
tcp_end_cwnd_reduction(sk);
|
2005-08-10 11:03:31 +04:00
|
|
|
tcp_set_ca_state(sk, TCP_CA_Open);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
|
|
|
|
case TCP_CA_Recovery:
|
2007-08-09 16:14:46 +04:00
|
|
|
if (tcp_is_reno(tp))
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_reset_reno_sack(tp);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
if (tcp_try_undo_recovery(sk))
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
2012-09-02 21:38:04 +04:00
|
|
|
tcp_end_cwnd_reduction(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-10-17 07:57:47 +03:00
|
|
|
/* Use RACK to detect loss */
|
|
|
|
if (sysctl_tcp_recovery & TCP_RACK_LOST_RETRANS &&
|
2016-02-02 21:33:05 +03:00
|
|
|
tcp_rack_mark_lost(sk)) {
|
2015-10-17 07:57:47 +03:00
|
|
|
flag |= FLAG_LOST_RETRANS;
|
2016-02-02 21:33:05 +03:00
|
|
|
*ack_flag |= FLAG_LOST_RETRANS;
|
|
|
|
}
|
2015-10-17 07:57:47 +03:00
|
|
|
|
2012-01-19 18:42:21 +04:00
|
|
|
/* E. Process state. */
|
2005-08-10 11:03:31 +04:00
|
|
|
switch (icsk->icsk_ca_state) {
|
2005-04-17 02:20:36 +04:00
|
|
|
case TCP_CA_Recovery:
|
2007-08-03 06:46:58 +04:00
|
|
|
if (!(flag & FLAG_SND_UNA_ADVANCED)) {
|
2007-08-09 16:14:46 +04:00
|
|
|
if (tcp_is_reno(tp) && is_dupack)
|
2005-08-10 11:03:31 +04:00
|
|
|
tcp_add_reno_sack(sk);
|
2013-05-29 18:20:13 +04:00
|
|
|
} else {
|
2016-02-02 21:33:05 +03:00
|
|
|
if (tcp_try_undo_partial(sk, acked))
|
2013-05-29 18:20:13 +04:00
|
|
|
return;
|
|
|
|
/* Partial ACK arrived. Force fast retransmit. */
|
|
|
|
do_lost = tcp_is_reno(tp) ||
|
|
|
|
tcp_fackets_out(tp) > tp->reordering;
|
|
|
|
}
|
2013-05-29 18:20:14 +04:00
|
|
|
if (tcp_try_undo_dsack(sk)) {
|
|
|
|
tcp_try_keep_open(sk);
|
|
|
|
return;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
break;
|
|
|
|
case TCP_CA_Loss:
|
2016-02-02 21:33:04 +03:00
|
|
|
tcp_process_loss(sk, flag, is_dupack, rexmit);
|
2015-07-02 00:11:14 +03:00
|
|
|
if (icsk->icsk_ca_state != TCP_CA_Open &&
|
|
|
|
!(flag & FLAG_LOST_RETRANS))
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
2015-07-02 00:11:14 +03:00
|
|
|
/* Change state if cwnd is undone or retransmits are lost */
|
2005-04-17 02:20:36 +04:00
|
|
|
default:
|
2007-08-09 16:14:46 +04:00
|
|
|
if (tcp_is_reno(tp)) {
|
2007-08-03 06:46:58 +04:00
|
|
|
if (flag & FLAG_SND_UNA_ADVANCED)
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_reset_reno_sack(tp);
|
|
|
|
if (is_dupack)
|
2005-08-10 11:03:31 +04:00
|
|
|
tcp_add_reno_sack(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2011-11-16 12:58:04 +04:00
|
|
|
if (icsk->icsk_ca_state <= TCP_CA_Disorder)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_try_undo_dsack(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2012-05-02 17:30:04 +04:00
|
|
|
if (!tcp_time_to_recover(sk, flag)) {
|
2016-02-02 21:33:05 +03:00
|
|
|
tcp_try_to_open(sk, flag);
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2006-03-21 04:53:41 +03:00
|
|
|
/* MTU probe failure: don't reduce cwnd */
|
|
|
|
if (icsk->icsk_ca_state < TCP_CA_CWR &&
|
|
|
|
icsk->icsk_mtup.probe_size &&
|
2006-03-21 08:32:58 +03:00
|
|
|
tp->snd_una == tp->mtu_probe.probe_seq_start) {
|
2006-03-21 04:53:41 +03:00
|
|
|
tcp_mtup_probe_failed(sk);
|
|
|
|
/* Restores the reduction we did in tcp_mtup_probe() */
|
|
|
|
tp->snd_cwnd++;
|
|
|
|
tcp_simple_retransmit(sk);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Otherwise enter Recovery state */
|
2012-05-02 17:30:02 +04:00
|
|
|
tcp_enter_recovery(sk, (flag & FLAG_ECE));
|
2007-11-16 06:39:31 +03:00
|
|
|
fast_rexmit = 1;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2013-05-17 17:45:05 +04:00
|
|
|
if (do_lost)
|
2007-11-16 06:39:31 +03:00
|
|
|
tcp_update_scoreboard(sk, fast_rexmit);
|
2016-02-02 21:33:04 +03:00
|
|
|
*rexmit = REXMIT_LOST;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2015-10-17 07:57:42 +03:00
|
|
|
/* Kathleen Nichols' algorithm for tracking the minimum value of
|
|
|
|
* a data stream over some fixed time interval. (E.g., the minimum
|
|
|
|
* RTT over the past five minutes.) It uses constant space and constant
|
|
|
|
* time per update yet almost always delivers the same minimum as an
|
|
|
|
* implementation that has to keep all the data in the window.
|
|
|
|
*
|
|
|
|
* The algorithm keeps track of the best, 2nd best & 3rd best min
|
|
|
|
* values, maintaining an invariant that the measurement time of the
|
|
|
|
* n'th best >= n-1'th best. It also makes sure that the three values
|
|
|
|
* are widely separated in the time window since that bounds the worse
|
|
|
|
* case error when that data is monotonically increasing over the window.
|
|
|
|
*
|
|
|
|
* Upon getting a new min, we can forget everything earlier because it
|
|
|
|
* has no value - the new min is <= everything else in the window by
|
|
|
|
* definition and it's the most recent. So we restart fresh on every new min
|
|
|
|
* and overwrites 2nd & 3rd choices. The same property holds for 2nd & 3rd
|
|
|
|
* best.
|
|
|
|
*/
|
|
|
|
static void tcp_update_rtt_min(struct sock *sk, u32 rtt_us)
|
|
|
|
{
|
|
|
|
const u32 now = tcp_time_stamp, wlen = sysctl_tcp_min_rtt_wlen * HZ;
|
|
|
|
struct rtt_meas *m = tcp_sk(sk)->rtt_min;
|
2016-02-11 19:58:18 +03:00
|
|
|
struct rtt_meas rttm = {
|
|
|
|
.rtt = likely(rtt_us) ? rtt_us : jiffies_to_usecs(1),
|
|
|
|
.ts = now,
|
|
|
|
};
|
2015-10-17 07:57:42 +03:00
|
|
|
u32 elapsed;
|
|
|
|
|
|
|
|
/* Check if the new measurement updates the 1st, 2nd, or 3rd choices */
|
|
|
|
if (unlikely(rttm.rtt <= m[0].rtt))
|
|
|
|
m[0] = m[1] = m[2] = rttm;
|
|
|
|
else if (rttm.rtt <= m[1].rtt)
|
|
|
|
m[1] = m[2] = rttm;
|
|
|
|
else if (rttm.rtt <= m[2].rtt)
|
|
|
|
m[2] = rttm;
|
|
|
|
|
|
|
|
elapsed = now - m[0].ts;
|
|
|
|
if (unlikely(elapsed > wlen)) {
|
|
|
|
/* Passed entire window without a new min so make 2nd choice
|
|
|
|
* the new min & 3rd choice the new 2nd. So forth and so on.
|
|
|
|
*/
|
|
|
|
m[0] = m[1];
|
|
|
|
m[1] = m[2];
|
|
|
|
m[2] = rttm;
|
|
|
|
if (now - m[0].ts > wlen) {
|
|
|
|
m[0] = m[1];
|
|
|
|
m[1] = rttm;
|
|
|
|
if (now - m[0].ts > wlen)
|
|
|
|
m[0] = rttm;
|
|
|
|
}
|
|
|
|
} else if (m[1].ts == m[0].ts && elapsed > wlen / 4) {
|
|
|
|
/* Passed a quarter of the window without a new min so
|
|
|
|
* take 2nd choice from the 2nd quarter of the window.
|
|
|
|
*/
|
|
|
|
m[2] = m[1] = rttm;
|
|
|
|
} else if (m[2].ts == m[1].ts && elapsed > wlen / 2) {
|
|
|
|
/* Passed half the window without a new min so take the 3rd
|
|
|
|
* choice from the last half of the window.
|
|
|
|
*/
|
|
|
|
m[2] = rttm;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-07-23 03:20:48 +04:00
|
|
|
static inline bool tcp_ack_update_rtt(struct sock *sk, const int flag,
|
2015-10-17 07:57:42 +03:00
|
|
|
long seq_rtt_us, long sack_rtt_us,
|
|
|
|
long ca_rtt_us)
|
2008-12-06 09:43:26 +03:00
|
|
|
{
|
2013-07-23 03:20:46 +04:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
/* Prefer RTT measured from ACK's timing to TS-ECR. This is because
|
|
|
|
* broken middle-boxes or peers may corrupt TS-ECR fields. But
|
|
|
|
* Karn's algorithm forbids taking RTT if some retransmitted data
|
|
|
|
* is acked (RFC6298).
|
|
|
|
*/
|
2014-02-27 02:02:48 +04:00
|
|
|
if (seq_rtt_us < 0)
|
|
|
|
seq_rtt_us = sack_rtt_us;
|
2013-07-23 03:20:48 +04:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* RTTM Rule: A TSecr value received in a segment is used to
|
|
|
|
* update the averaged RTT measurement only if the segment
|
|
|
|
* acknowledges some new data, i.e., only if it advances the
|
|
|
|
* left edge of the send window.
|
|
|
|
* See draft-ietf-tcplw-high-performance-00, section 3.3.
|
|
|
|
*/
|
2014-02-27 02:02:48 +04:00
|
|
|
if (seq_rtt_us < 0 && tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
|
2013-10-24 19:55:25 +04:00
|
|
|
flag & FLAG_ACKED)
|
2015-10-17 07:57:42 +03:00
|
|
|
seq_rtt_us = ca_rtt_us = jiffies_to_usecs(tcp_time_stamp -
|
|
|
|
tp->rx_opt.rcv_tsecr);
|
2014-02-27 02:02:48 +04:00
|
|
|
if (seq_rtt_us < 0)
|
2013-07-23 03:20:48 +04:00
|
|
|
return false;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2015-10-17 07:57:42 +03:00
|
|
|
/* ca_rtt_us >= 0 is counting on the invariant that ca_rtt_us is
|
|
|
|
* always taken together with ACK, SACK, or TS-opts. Any negative
|
|
|
|
* values will be skipped with the seq_rtt_us < 0 check above.
|
|
|
|
*/
|
|
|
|
tcp_update_rtt_min(sk, ca_rtt_us);
|
2014-02-27 02:02:48 +04:00
|
|
|
tcp_rtt_estimator(sk, seq_rtt_us);
|
2013-07-23 03:20:46 +04:00
|
|
|
tcp_set_rto(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-07-23 03:20:46 +04:00
|
|
|
/* RFC6298: only reset backoff on valid RTT measurement. */
|
|
|
|
inet_csk(sk)->icsk_backoff = 0;
|
2013-07-23 03:20:48 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2013-07-23 03:20:45 +04:00
|
|
|
/* Compute time elapsed between (last) SYNACK and the ACK completing 3WHS. */
|
2015-09-18 21:36:14 +03:00
|
|
|
void tcp_synack_rtt_meas(struct sock *sk, struct request_sock *req)
|
2013-07-23 03:20:45 +04:00
|
|
|
{
|
2015-09-18 21:36:14 +03:00
|
|
|
long rtt_us = -1L;
|
2013-07-23 03:20:45 +04:00
|
|
|
|
2015-09-18 21:36:14 +03:00
|
|
|
if (req && !req->num_retrans && tcp_rsk(req)->snt_synack.v64) {
|
|
|
|
struct skb_mstamp now;
|
2013-10-24 19:44:25 +04:00
|
|
|
|
2015-09-18 21:36:14 +03:00
|
|
|
skb_mstamp_get(&now);
|
|
|
|
rtt_us = skb_mstamp_us_delta(&now, &tcp_rsk(req)->snt_synack);
|
|
|
|
}
|
|
|
|
|
2015-10-17 07:57:42 +03:00
|
|
|
tcp_ack_update_rtt(sk, FLAG_SYN_ACKED, rtt_us, -1L, rtt_us);
|
2013-07-23 03:20:45 +04:00
|
|
|
}
|
|
|
|
|
2015-09-18 21:36:14 +03:00
|
|
|
|
2014-05-03 08:18:05 +04:00
|
|
|
static void tcp_cong_avoid(struct sock *sk, u32 ack, u32 acked)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 11:03:31 +04:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2014-05-03 08:18:05 +04:00
|
|
|
|
|
|
|
icsk->icsk_ca_ops->cong_avoid(sk, ack, acked);
|
2005-08-10 11:03:31 +04:00
|
|
|
tcp_sk(sk)->snd_cwnd_stamp = tcp_time_stamp;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Restart timer after forward progress on connection.
|
|
|
|
* RFC2988 recommends to restart timer to now+rto.
|
|
|
|
*/
|
2012-05-02 17:30:04 +04:00
|
|
|
void tcp_rearm_rto(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 14:00:43 +04:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2012-05-02 17:30:04 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
|
2012-08-31 16:29:13 +04:00
|
|
|
/* If the retrans timer is currently being used by Fast Open
|
|
|
|
* for SYN-ACK retrans purpose, stay put.
|
|
|
|
*/
|
|
|
|
if (tp->fastopen_rsk)
|
|
|
|
return;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
if (!tp->packets_out) {
|
2005-08-10 07:10:42 +04:00
|
|
|
inet_csk_clear_xmit_timer(sk, ICSK_TIME_RETRANS);
|
2005-04-17 02:20:36 +04:00
|
|
|
} else {
|
2012-05-02 17:30:04 +04:00
|
|
|
u32 rto = inet_csk(sk)->icsk_rto;
|
|
|
|
/* Offset the time elapsed after installing regular RTO */
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 14:00:43 +04:00
|
|
|
if (icsk->icsk_pending == ICSK_TIME_EARLY_RETRANS ||
|
|
|
|
icsk->icsk_pending == ICSK_TIME_LOSS_PROBE) {
|
2012-05-02 17:30:04 +04:00
|
|
|
struct sk_buff *skb = tcp_write_queue_head(sk);
|
2014-09-06 02:33:33 +04:00
|
|
|
const u32 rto_time_stamp =
|
|
|
|
tcp_skb_timestamp(skb) + rto;
|
2012-05-02 17:30:04 +04:00
|
|
|
s32 delta = (s32)(rto_time_stamp - tcp_time_stamp);
|
|
|
|
/* delta may not be positive if the socket is locked
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 14:00:43 +04:00
|
|
|
* when the retrans timer fires and is rescheduled.
|
2012-05-02 17:30:04 +04:00
|
|
|
*/
|
|
|
|
if (delta > 0)
|
|
|
|
rto = delta;
|
|
|
|
}
|
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_RETRANS, rto,
|
|
|
|
TCP_RTO_MAX);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2012-05-02 17:30:04 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* This function is called when the delayed ER timer fires. TCP enters
|
|
|
|
* fast recovery and performs fast-retransmit.
|
|
|
|
*/
|
|
|
|
void tcp_resume_early_retransmit(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
tcp_rearm_rto(sk);
|
|
|
|
|
|
|
|
/* Stop if ER is disabled after the delayed ER timer is scheduled */
|
|
|
|
if (!tp->do_early_retrans)
|
|
|
|
return;
|
|
|
|
|
|
|
|
tcp_enter_recovery(sk, false);
|
|
|
|
tcp_update_scoreboard(sk, 1);
|
|
|
|
tcp_xmit_retransmit_queue(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2007-09-20 22:33:43 +04:00
|
|
|
/* If we get here, the whole TSO packet has not been acked. */
|
2007-10-09 12:28:45 +04:00
|
|
|
static u32 tcp_tso_acked(struct sock *sk, struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2007-09-20 22:33:43 +04:00
|
|
|
u32 packets_acked;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2007-09-20 22:33:43 +04:00
|
|
|
BUG_ON(!after(TCP_SKB_CB(skb)->end_seq, tp->snd_una));
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
packets_acked = tcp_skb_pcount(skb);
|
2007-09-20 22:33:43 +04:00
|
|
|
if (tcp_trim_head(sk, skb, tp->snd_una - TCP_SKB_CB(skb)->seq))
|
2005-04-17 02:20:36 +04:00
|
|
|
return 0;
|
|
|
|
packets_acked -= tcp_skb_pcount(skb);
|
|
|
|
|
|
|
|
if (packets_acked) {
|
|
|
|
BUG_ON(tcp_skb_pcount(skb) == 0);
|
2007-09-20 22:33:43 +04:00
|
|
|
BUG_ON(!before(TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq));
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2007-10-09 12:28:45 +04:00
|
|
|
return packets_acked;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2014-10-12 02:17:29 +04:00
|
|
|
static void tcp_ack_tstamp(struct sock *sk, struct sk_buff *skb,
|
|
|
|
u32 prior_snd_una)
|
|
|
|
{
|
|
|
|
const struct skb_shared_info *shinfo;
|
|
|
|
|
|
|
|
/* Avoid cache line misses to get skb_shinfo() and shinfo->tx_flags */
|
2016-04-03 06:08:08 +03:00
|
|
|
if (likely(!TCP_SKB_CB(skb)->txstamp_ack))
|
2014-10-12 02:17:29 +04:00
|
|
|
return;
|
|
|
|
|
|
|
|
shinfo = skb_shinfo(skb);
|
|
|
|
if ((shinfo->tx_flags & SKBTX_ACK_TSTAMP) &&
|
tcp: Fix SOF_TIMESTAMPING_TX_ACK when handling dup acks
Assuming SOF_TIMESTAMPING_TX_ACK is on. When dup acks are received,
it could incorrectly think that a skb has already
been acked and queue a SCM_TSTAMP_ACK cmsg to the
sk->sk_error_queue.
In tcp_ack_tstamp(), it checks
'between(shinfo->tskey, prior_snd_una, tcp_sk(sk)->snd_una - 1)'.
If prior_snd_una == tcp_sk(sk)->snd_una like the following packetdrill
script, between() returns true but the tskey is actually not acked.
e.g. try between(3, 2, 1).
The fix is to replace between() with one before() and one !before().
By doing this, the -1 offset on the tcp_sk(sk)->snd_una can also be
removed.
A packetdrill script is used to reproduce the dup ack scenario.
Due to the lacking cmsg support in packetdrill (may be I
cannot find it), a BPF prog is used to kprobe to
sock_queue_err_skb() and print out the value of
serr->ee.ee_data.
Both the packetdrill and the bcc BPF script is attached at the end of
this commit message.
BPF Output Before Fix:
~~~~~~
<...>-2056 [001] d.s. 433.927987: : ee_data:1459 #incorrect
packetdrill-2056 [001] d.s. 433.929563: : ee_data:1459 #incorrect
packetdrill-2056 [001] d.s. 433.930765: : ee_data:1459 #incorrect
packetdrill-2056 [001] d.s. 434.028177: : ee_data:1459
packetdrill-2056 [001] d.s. 434.029686: : ee_data:14599
BPF Output After Fix:
~~~~~~
<...>-2049 [000] d.s. 113.517039: : ee_data:1459
<...>-2049 [000] d.s. 113.517253: : ee_data:14599
BCC BPF Script:
~~~~~~
#!/usr/bin/env python
from __future__ import print_function
from bcc import BPF
bpf_text = """
#include <uapi/linux/ptrace.h>
#include <net/sock.h>
#include <bcc/proto.h>
#include <linux/errqueue.h>
#ifdef memset
#undef memset
#endif
int trace_err_skb(struct pt_regs *ctx)
{
struct sk_buff *skb = (struct sk_buff *)ctx->si;
struct sock *sk = (struct sock *)ctx->di;
struct sock_exterr_skb *serr;
u32 ee_data = 0;
if (!sk || !skb)
return 0;
serr = SKB_EXT_ERR(skb);
bpf_probe_read(&ee_data, sizeof(ee_data), &serr->ee.ee_data);
bpf_trace_printk("ee_data:%u\\n", ee_data);
return 0;
};
"""
b = BPF(text=bpf_text)
b.attach_kprobe(event="sock_queue_err_skb", fn_name="trace_err_skb")
print("Attached to kprobe")
b.trace_print()
Packetdrill Script:
~~~~~~
+0 `sysctl -q -w net.ipv4.tcp_min_tso_segs=10`
+0 `sysctl -q -w net.ipv4.tcp_no_metrics_save=1`
+0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
0.100 < S 0:0(0) win 32792 <mss 1460,sackOK,nop,nop,nop,wscale 7>
0.100 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 7>
0.200 < . 1:1(0) ack 1 win 257
0.200 accept(3, ..., ...) = 4
+0 setsockopt(4, SOL_TCP, TCP_NODELAY, [1], 4) = 0
+0 setsockopt(4, SOL_SOCKET, 37, [2688], 4) = 0
0.200 write(4, ..., 1460) = 1460
0.200 write(4, ..., 13140) = 13140
0.200 > P. 1:1461(1460) ack 1
0.200 > . 1461:8761(7300) ack 1
0.200 > P. 8761:14601(5840) ack 1
0.300 < . 1:1(0) ack 1 win 257 <sack 1461:2921,nop,nop>
0.300 < . 1:1(0) ack 1 win 257 <sack 1461:4381,nop,nop>
0.300 < . 1:1(0) ack 1 win 257 <sack 1461:5841,nop,nop>
0.300 > P. 1:1461(1460) ack 1
0.400 < . 1:1(0) ack 14601 win 257
0.400 close(4) = 0
0.400 > F. 14601:14601(0) ack 1
0.500 < F. 1:1(0) ack 14602 win 257
0.500 > . 14602:14602(0) ack 2
Signed-off-by: Martin KaFai Lau <kafai@fb.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Soheil Hassas Yeganeh <soheil.kdev@gmail.com>
Cc: Willem de Bruijn <willemb@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Acked-by: Soheil Hassas Yeganeh <soheil@google.com>
Tested-by: Soheil Hassas Yeganeh <soheil@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-19 01:39:53 +03:00
|
|
|
!before(shinfo->tskey, prior_snd_una) &&
|
|
|
|
before(shinfo->tskey, tcp_sk(sk)->snd_una))
|
2014-10-12 02:17:29 +04:00
|
|
|
__skb_tstamp_tx(skb, NULL, sk, SCM_TSTAMP_ACK);
|
|
|
|
}
|
|
|
|
|
2007-09-20 22:33:43 +04:00
|
|
|
/* Remove acknowledged frames from the retransmission queue. If our packet
|
|
|
|
* is before the ack sequence we can discard it as it's confirmed to have
|
|
|
|
* arrived at the other end.
|
|
|
|
*/
|
2008-10-08 01:43:06 +04:00
|
|
|
static int tcp_clean_rtx_queue(struct sock *sk, int prior_fackets,
|
2016-02-02 21:33:07 +03:00
|
|
|
u32 prior_snd_una, int *acked,
|
2015-05-01 02:10:58 +03:00
|
|
|
struct tcp_sacktag_state *sack)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-11-11 03:56:12 +03:00
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
2014-02-27 02:02:48 +04:00
|
|
|
struct skb_mstamp first_ackt, last_ackt, now;
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u32 prior_sacked = tp->sacked_out;
|
|
|
|
u32 reord = tp->packets_out;
|
2013-10-02 16:19:51 +04:00
|
|
|
bool fully_acked = true;
|
2015-05-01 02:10:58 +03:00
|
|
|
long sack_rtt_us = -1L;
|
2014-02-27 02:02:48 +04:00
|
|
|
long seq_rtt_us = -1L;
|
2015-05-01 02:10:58 +03:00
|
|
|
long ca_rtt_us = -1L;
|
2014-02-27 02:02:48 +04:00
|
|
|
struct sk_buff *skb;
|
2007-12-30 15:37:55 +03:00
|
|
|
u32 pkts_acked = 0;
|
2013-10-24 19:59:27 +04:00
|
|
|
bool rtt_update;
|
2014-02-27 02:02:48 +04:00
|
|
|
int flag = 0;
|
|
|
|
|
|
|
|
first_ackt.v64 = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2007-09-20 22:33:43 +04:00
|
|
|
while ((skb = tcp_write_queue_head(sk)) && skb != tcp_send_head(sk)) {
|
2007-02-09 17:24:47 +03:00
|
|
|
struct tcp_skb_cb *scb = TCP_SKB_CB(skb);
|
2007-09-20 22:33:43 +04:00
|
|
|
u8 sacked = scb->sacked;
|
2014-02-27 02:02:48 +04:00
|
|
|
u32 acked_pcount;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2014-10-12 02:17:29 +04:00
|
|
|
tcp_ack_tstamp(sk, skb, prior_snd_una);
|
2014-08-12 22:53:16 +04:00
|
|
|
|
[TCP]: use non-delayed ACK for congestion control RTT
When a delayed ACK representing two packets arrives, there are two RTT
samples available, one for each packet. The first (in order of seq
number) will be artificially long due to the delay waiting for the
second packet, the second will trigger the ACK and so will not itself
be delayed.
According to rfc1323, the SRTT used for RTO calculation should use the
first rtt, so receivers echo the timestamp from the first packet in
the delayed ack. For congestion control however, it seems measuring
delayed ack delay is not desirable as it varies independently of
congestion.
The patch below causes seq_rtt and last_ackt to be updated with any
available later packet rtts which should have less (and hopefully
zero) delack delay. The rtt value then gets passed to
ca_ops->pkts_acked().
Where TCP_CONG_RTT_STAMP was set, effort was made to supress RTTs from
within a TSO chunk (!fully_acked), using only the final ACK (which
includes any TSO delay) to generate RTTs. This patch removes these
checks so RTTs are passed for each ACK to ca_ops->pkts_acked().
For non-delay based congestion control (cubic, h-tcp), rtt is
sometimes used for rtt-scaling. In shortening the RTT, this may make
them a little less aggressive. Delay-based schemes (eg vegas, veno,
illinois) should get a cleaner, more accurate congestion signal,
particularly for small cwnds. The congestion control module can
potentially also filter out bad RTTs due to the delayed ack alarm by
looking at the associated cnt which (where delayed acking is in use)
should probably be 1 if the alarm went off or greater if the ACK was
triggered by a packet.
Signed-off-by: Gavin McCullagh <gavin.mccullagh@nuim.ie>
Acked-by: Ilpo Jrvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-12-30 06:11:21 +03:00
|
|
|
/* Determine how many packets and what bytes were acked, tso and else */
|
2005-04-17 02:20:36 +04:00
|
|
|
if (after(scb->end_seq, tp->snd_una)) {
|
2007-10-09 12:28:45 +04:00
|
|
|
if (tcp_skb_pcount(skb) == 1 ||
|
|
|
|
!after(tp->snd_una, scb->seq))
|
|
|
|
break;
|
|
|
|
|
2007-12-30 15:37:55 +03:00
|
|
|
acked_pcount = tcp_tso_acked(sk, skb);
|
|
|
|
if (!acked_pcount)
|
2007-10-09 12:28:45 +04:00
|
|
|
break;
|
|
|
|
|
2012-05-17 03:15:34 +04:00
|
|
|
fully_acked = false;
|
2007-10-09 12:28:45 +04:00
|
|
|
} else {
|
2014-10-12 02:17:29 +04:00
|
|
|
/* Speedup tcp_unlink_write_queue() and next loop */
|
|
|
|
prefetchw(skb->next);
|
2007-12-30 15:37:55 +03:00
|
|
|
acked_pcount = tcp_skb_pcount(skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2014-10-12 02:17:29 +04:00
|
|
|
if (unlikely(sacked & TCPCB_RETRANS)) {
|
2007-12-30 15:35:27 +03:00
|
|
|
if (sacked & TCPCB_SACKED_RETRANS)
|
2007-12-30 15:37:55 +03:00
|
|
|
tp->retrans_out -= acked_pcount;
|
2007-12-30 15:35:27 +03:00
|
|
|
flag |= FLAG_RETRANS_DATA_ACKED;
|
2015-04-11 03:17:49 +03:00
|
|
|
} else if (!(sacked & TCPCB_SACKED_ACKED)) {
|
2014-02-27 02:02:48 +04:00
|
|
|
last_ackt = skb->skb_mstamp;
|
2014-03-10 04:36:02 +04:00
|
|
|
WARN_ON_ONCE(last_ackt.v64 == 0);
|
2014-02-27 02:02:48 +04:00
|
|
|
if (!first_ackt.v64)
|
|
|
|
first_ackt = last_ackt;
|
|
|
|
|
2015-04-11 03:17:49 +03:00
|
|
|
reord = min(pkts_acked, reord);
|
|
|
|
if (!after(scb->end_seq, tp->high_seq))
|
|
|
|
flag |= FLAG_ORIG_SACK_ACKED;
|
2005-11-11 03:56:12 +03:00
|
|
|
}
|
2007-12-30 15:35:27 +03:00
|
|
|
|
2016-02-02 21:33:06 +03:00
|
|
|
if (sacked & TCPCB_SACKED_ACKED) {
|
2007-12-30 15:37:55 +03:00
|
|
|
tp->sacked_out -= acked_pcount;
|
2016-02-02 21:33:06 +03:00
|
|
|
} else if (tcp_is_sack(tp)) {
|
|
|
|
tp->delivered += acked_pcount;
|
|
|
|
if (!tcp_skb_spurious_retrans(tp, skb))
|
|
|
|
tcp_rack_advance(tp, &skb->skb_mstamp, sacked);
|
|
|
|
}
|
2007-12-30 15:35:27 +03:00
|
|
|
if (sacked & TCPCB_LOST)
|
2007-12-30 15:37:55 +03:00
|
|
|
tp->lost_out -= acked_pcount;
|
2007-12-30 15:35:27 +03:00
|
|
|
|
2007-12-30 15:37:55 +03:00
|
|
|
tp->packets_out -= acked_pcount;
|
|
|
|
pkts_acked += acked_pcount;
|
2007-10-09 12:28:45 +04:00
|
|
|
|
2007-09-20 22:34:38 +04:00
|
|
|
/* Initial outgoing SYN's get put onto the write_queue
|
|
|
|
* just like anything else we transmit. It is not
|
|
|
|
* true data, and if we misinform our callers that
|
|
|
|
* this ACK acks real data, we will erroneously exit
|
|
|
|
* connection startup slow start one packet too
|
|
|
|
* quickly. This is severely frowned upon behavior.
|
|
|
|
*/
|
2014-10-12 02:17:29 +04:00
|
|
|
if (likely(!(scb->tcp_flags & TCPHDR_SYN))) {
|
2007-09-20 22:34:38 +04:00
|
|
|
flag |= FLAG_DATA_ACKED;
|
|
|
|
} else {
|
|
|
|
flag |= FLAG_SYN_ACKED;
|
|
|
|
tp->retrans_stamp = 0;
|
|
|
|
}
|
|
|
|
|
2007-10-09 12:28:45 +04:00
|
|
|
if (!fully_acked)
|
|
|
|
break;
|
|
|
|
|
2007-03-07 23:12:44 +03:00
|
|
|
tcp_unlink_write_queue(skb, sk);
|
2007-12-31 11:11:19 +03:00
|
|
|
sk_wmem_free_skb(sk, skb);
|
2014-10-12 02:17:29 +04:00
|
|
|
if (unlikely(skb == tp->retransmit_skb_hint))
|
2008-09-21 08:25:15 +04:00
|
|
|
tp->retransmit_skb_hint = NULL;
|
2014-10-12 02:17:29 +04:00
|
|
|
if (unlikely(skb == tp->lost_skb_hint))
|
2008-09-21 08:25:52 +04:00
|
|
|
tp->lost_skb_hint = NULL;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2008-10-08 01:43:06 +04:00
|
|
|
if (likely(between(tp->snd_up, prior_snd_una, tp->snd_una)))
|
|
|
|
tp->snd_up = tp->snd_una;
|
|
|
|
|
2007-12-31 15:49:21 +03:00
|
|
|
if (skb && (TCP_SKB_CB(skb)->sacked & TCPCB_SACKED_ACKED))
|
|
|
|
flag |= FLAG_SACK_RENEGING;
|
|
|
|
|
2014-02-27 02:02:48 +04:00
|
|
|
skb_mstamp_get(&now);
|
2015-10-17 07:57:41 +03:00
|
|
|
if (likely(first_ackt.v64) && !(flag & FLAG_RETRANS_DATA_ACKED)) {
|
2014-02-27 02:02:48 +04:00
|
|
|
seq_rtt_us = skb_mstamp_us_delta(&now, &first_ackt);
|
2015-05-01 02:10:58 +03:00
|
|
|
ca_rtt_us = skb_mstamp_us_delta(&now, &last_ackt);
|
|
|
|
}
|
|
|
|
if (sack->first_sackt.v64) {
|
|
|
|
sack_rtt_us = skb_mstamp_us_delta(&now, &sack->first_sackt);
|
|
|
|
ca_rtt_us = skb_mstamp_us_delta(&now, &sack->last_sackt);
|
2014-02-27 02:02:48 +04:00
|
|
|
}
|
|
|
|
|
2015-10-17 07:57:42 +03:00
|
|
|
rtt_update = tcp_ack_update_rtt(sk, flag, seq_rtt_us, sack_rtt_us,
|
|
|
|
ca_rtt_us);
|
2013-07-23 03:20:48 +04:00
|
|
|
|
2007-09-20 22:33:43 +04:00
|
|
|
if (flag & FLAG_ACKED) {
|
2013-10-24 19:59:27 +04:00
|
|
|
tcp_rearm_rto(sk);
|
2009-03-14 17:23:04 +03:00
|
|
|
if (unlikely(icsk->icsk_mtup.probe_size &&
|
|
|
|
!after(tp->mtu_probe.probe_seq_end, tp->snd_una))) {
|
|
|
|
tcp_mtup_probe_success(sk);
|
|
|
|
}
|
|
|
|
|
2007-11-11 08:22:18 +03:00
|
|
|
if (tcp_is_reno(tp)) {
|
|
|
|
tcp_remove_reno_sacks(sk, pkts_acked);
|
|
|
|
} else {
|
2009-02-28 07:44:28 +03:00
|
|
|
int delta;
|
|
|
|
|
2007-11-11 08:22:18 +03:00
|
|
|
/* Non-retransmitted hole got filled? That's reordering */
|
|
|
|
if (reord < prior_fackets)
|
|
|
|
tcp_update_reordering(sk, tp->fackets_out - reord, 0);
|
2008-09-21 08:25:52 +04:00
|
|
|
|
2009-02-28 07:44:28 +03:00
|
|
|
delta = tcp_is_fack(tp) ? pkts_acked :
|
|
|
|
prior_sacked - tp->sacked_out;
|
|
|
|
tp->lost_cnt_hint -= min(tp->lost_cnt_hint, delta);
|
2007-11-11 08:22:18 +03:00
|
|
|
}
|
|
|
|
|
2007-10-09 12:24:15 +04:00
|
|
|
tp->fackets_out -= min(pkts_acked, tp->fackets_out);
|
[TCP]: Rewrite SACK block processing & sack_recv_cache use
Key points of this patch are:
- In case new SACK information is advance only type, no skb
processing below previously discovered highest point is done
- Optimize cases below highest point too since there's no need
to always go up to highest point (which is very likely still
present in that SACK), this is not entirely true though
because I'm dropping the fastpath_skb_hint which could
previously optimize those cases even better. Whether that's
significant, I'm not too sure.
Currently it will provide skipping by walking. Combined with
RB-tree, all skipping would become fast too regardless of window
size (can be done incrementally later).
Previously a number of cases in TCP SACK processing fails to
take advantage of costly stored information in sack_recv_cache,
most importantly, expected events such as cumulative ACK and new
hole ACKs. Processing on such ACKs result in rather long walks
building up latencies (which easily gets nasty when window is
huge). Those latencies are often completely unnecessary
compared with the amount of _new_ information received, usually
for cumulative ACK there's no new information at all, yet TCP
walks whole queue unnecessary potentially taking a number of
costly cache misses on the way, etc.!
Since the inclusion of highest_sack, there's a lot information
that is very likely redundant (SACK fastpath hint stuff,
fackets_out, highest_sack), though there's no ultimate guarantee
that they'll remain the same whole the time (in all unearthly
scenarios). Take advantage of this knowledge here and drop
fastpath hint and use direct access to highest SACKed skb as
a replacement.
Effectively "special cased" fastpath is dropped. This change
adds some complexity to introduce better coveraged "fastpath",
though the added complexity should make TCP behave more cache
friendly.
The current ACK's SACK blocks are compared against each cached
block individially and only ranges that are new are then scanned
by the high constant walk. For other parts of write queue, even
when in previously known part of the SACK blocks, a faster skip
function is used (if necessary at all). In addition, whenever
possible, TCP fast-forwards to highest_sack skb that was made
available by an earlier patch. In typical case, no other things
but this fast-forward and mandatory markings after that occur
making the access pattern quite similar to the former fastpath
"special case".
DSACKs are special case that must always be walked.
The local to recv_sack_cache copying could be more intelligent
w.r.t DSACKs which are likely to be there only once but that
is left to a separate patch.
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-11-16 06:50:37 +03:00
|
|
|
|
2014-02-27 02:02:48 +04:00
|
|
|
} else if (skb && rtt_update && sack_rtt_us >= 0 &&
|
|
|
|
sack_rtt_us > skb_mstamp_us_delta(&now, &skb->skb_mstamp)) {
|
2013-10-24 19:59:27 +04:00
|
|
|
/* Do not re-arm RTO if the sack RTT is measured from data sent
|
|
|
|
* after when the head was last (re)transmitted. Otherwise the
|
|
|
|
* timeout may continue to extend in loss recovery.
|
|
|
|
*/
|
|
|
|
tcp_rearm_rto(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2015-05-01 02:10:59 +03:00
|
|
|
if (icsk->icsk_ca_ops->pkts_acked)
|
|
|
|
icsk->icsk_ca_ops->pkts_acked(sk, pkts_acked, ca_rtt_us);
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#if FASTRETRANS_DEBUG > 0
|
2008-07-26 08:43:18 +04:00
|
|
|
WARN_ON((int)tp->sacked_out < 0);
|
|
|
|
WARN_ON((int)tp->lost_out < 0);
|
|
|
|
WARN_ON((int)tp->retrans_out < 0);
|
2007-08-09 16:14:46 +04:00
|
|
|
if (!tp->packets_out && tcp_is_sack(tp)) {
|
2007-10-09 12:59:42 +04:00
|
|
|
icsk = inet_csk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (tp->lost_out) {
|
2012-05-15 18:11:54 +04:00
|
|
|
pr_debug("Leak l=%u %d\n",
|
|
|
|
tp->lost_out, icsk->icsk_ca_state);
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->lost_out = 0;
|
|
|
|
}
|
|
|
|
if (tp->sacked_out) {
|
2012-05-15 18:11:54 +04:00
|
|
|
pr_debug("Leak s=%u %d\n",
|
|
|
|
tp->sacked_out, icsk->icsk_ca_state);
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->sacked_out = 0;
|
|
|
|
}
|
|
|
|
if (tp->retrans_out) {
|
2012-05-15 18:11:54 +04:00
|
|
|
pr_debug("Leak r=%u %d\n",
|
|
|
|
tp->retrans_out, icsk->icsk_ca_state);
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->retrans_out = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
#endif
|
2016-02-02 21:33:07 +03:00
|
|
|
*acked = pkts_acked;
|
2007-09-20 22:33:43 +04:00
|
|
|
return flag;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
static void tcp_ack_probe(struct sock *sk)
|
|
|
|
{
|
2005-08-10 07:10:42 +04:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Was it a usable window open? */
|
|
|
|
|
2007-12-31 15:48:41 +03:00
|
|
|
if (!after(TCP_SKB_CB(tcp_send_head(sk))->end_seq, tcp_wnd_end(tp))) {
|
2005-08-10 07:10:42 +04:00
|
|
|
icsk->icsk_backoff = 0;
|
|
|
|
inet_csk_clear_xmit_timer(sk, ICSK_TIME_PROBE0);
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Socket must be waked up by subsequent tcp_data_snd_check().
|
|
|
|
* This function is not for random using!
|
|
|
|
*/
|
|
|
|
} else {
|
tcp: adjust window probe timers to safer values
With the advent of small rto timers in datacenter TCP,
(ip route ... rto_min x), the following can happen :
1) Qdisc is full, transmit fails.
TCP sets a timer based on icsk_rto to retry the transmit, without
exponential backoff.
With low icsk_rto, and lot of sockets, all cpus are servicing timer
interrupts like crazy.
Intent of the code was to retry with a timer between 200 (TCP_RTO_MIN)
and 500ms (TCP_RESOURCE_PROBE_INTERVAL)
2) Receivers can send zero windows if they don't drain their receive queue.
TCP sends zero window probes, based on icsk_rto current value, with
exponential backoff.
With /proc/sys/net/ipv4/tcp_retries2 being 15 (or even smaller in
some cases), sender can abort in less than one or two minutes !
If receiver stops the sender, it obviously doesn't care of very tight
rto. Probability of dropping the ACK reopening the window is not
worth the risk.
Lets change the base timer to be at least 200ms (TCP_RTO_MIN) for these
events (but not normal RTO based retransmits)
A followup patch adds a new SNMP counter, as it would have helped a lot
diagnosing this issue.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-05-07 00:26:24 +03:00
|
|
|
unsigned long when = tcp_probe0_when(sk, TCP_RTO_MAX);
|
2014-09-23 00:19:44 +04:00
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_PROBE0,
|
2014-09-23 00:19:44 +04:00
|
|
|
when, TCP_RTO_MAX);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-07-20 01:32:18 +04:00
|
|
|
static inline bool tcp_ack_is_dubious(const struct sock *sk, const int flag)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2010-09-23 00:43:57 +04:00
|
|
|
return !(flag & FLAG_NOT_DUP) || (flag & FLAG_CA_ALERT) ||
|
|
|
|
inet_csk(sk)->icsk_ca_state != TCP_CA_Open;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2013-08-22 04:29:23 +04:00
|
|
|
/* Decide wheather to run the increase function of congestion control. */
|
2012-07-20 01:32:18 +04:00
|
|
|
static inline bool tcp_may_raise_cwnd(const struct sock *sk, const int flag)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2013-08-22 04:29:23 +04:00
|
|
|
/* If reordering is high then always grow cwnd whenever data is
|
|
|
|
* delivered regardless of its ordering. Otherwise stay conservative
|
2013-09-06 02:36:09 +04:00
|
|
|
* and only grow cwnd on in-order delivery (RFC5681). A stretched ACK w/
|
2013-08-22 04:29:23 +04:00
|
|
|
* new SACK or ECE mark may first advance cwnd here and later reduce
|
|
|
|
* cwnd in tcp_fastretrans_alert() based on more states.
|
|
|
|
*/
|
2016-02-03 10:46:52 +03:00
|
|
|
if (tcp_sk(sk)->reordering > sock_net(sk)->ipv4.sysctl_tcp_reordering)
|
2013-08-22 04:29:23 +04:00
|
|
|
return flag & FLAG_FORWARD_PROGRESS;
|
|
|
|
|
2013-09-06 02:36:09 +04:00
|
|
|
return flag & FLAG_DATA_ACKED;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2016-02-02 21:33:09 +03:00
|
|
|
/* The "ultimate" congestion control function that aims to replace the rigid
|
|
|
|
* cwnd increase and decrease control (tcp_cong_avoid,tcp_*cwnd_reduction).
|
|
|
|
* It's called toward the end of processing an ACK with precise rate
|
|
|
|
* information. All transmission or retransmission are delayed afterwards.
|
|
|
|
*/
|
|
|
|
static void tcp_cong_control(struct sock *sk, u32 ack, u32 acked_sacked,
|
|
|
|
int flag)
|
|
|
|
{
|
|
|
|
if (tcp_in_cwnd_reduction(sk)) {
|
|
|
|
/* Reduce cwnd if state mandates */
|
|
|
|
tcp_cwnd_reduction(sk, acked_sacked, flag);
|
|
|
|
} else if (tcp_may_raise_cwnd(sk, flag)) {
|
|
|
|
/* Advance cwnd if state allows */
|
|
|
|
tcp_cong_avoid(sk, ack, acked_sacked);
|
|
|
|
}
|
|
|
|
tcp_update_pacing_rate(sk);
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Check that window update is acceptable.
|
|
|
|
* The function assumes that snd_una<=ack<=snd_next.
|
|
|
|
*/
|
2012-07-20 01:32:18 +04:00
|
|
|
static inline bool tcp_may_update_window(const struct tcp_sock *tp,
|
2008-01-01 01:57:14 +03:00
|
|
|
const u32 ack, const u32 ack_seq,
|
|
|
|
const u32 nwin)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2010-09-23 00:43:57 +04:00
|
|
|
return after(ack, tp->snd_una) ||
|
2005-04-17 02:20:36 +04:00
|
|
|
after(ack_seq, tp->snd_wl1) ||
|
2010-09-23 00:43:57 +04:00
|
|
|
(ack_seq == tp->snd_wl1 && nwin > tp->snd_wnd);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2015-04-29 01:28:17 +03:00
|
|
|
/* If we update tp->snd_una, also update tp->bytes_acked */
|
|
|
|
static void tcp_snd_una_update(struct tcp_sock *tp, u32 ack)
|
|
|
|
{
|
|
|
|
u32 delta = ack - tp->snd_una;
|
|
|
|
|
2015-05-22 07:51:19 +03:00
|
|
|
u64_stats_update_begin(&tp->syncp);
|
2015-04-29 01:28:17 +03:00
|
|
|
tp->bytes_acked += delta;
|
2015-05-22 07:51:19 +03:00
|
|
|
u64_stats_update_end(&tp->syncp);
|
2015-04-29 01:28:17 +03:00
|
|
|
tp->snd_una = ack;
|
|
|
|
}
|
|
|
|
|
2015-04-29 01:28:18 +03:00
|
|
|
/* If we update tp->rcv_nxt, also update tp->bytes_received */
|
|
|
|
static void tcp_rcv_nxt_update(struct tcp_sock *tp, u32 seq)
|
|
|
|
{
|
|
|
|
u32 delta = seq - tp->rcv_nxt;
|
|
|
|
|
2015-05-22 07:51:19 +03:00
|
|
|
u64_stats_update_begin(&tp->syncp);
|
2015-04-29 01:28:18 +03:00
|
|
|
tp->bytes_received += delta;
|
2015-05-22 07:51:19 +03:00
|
|
|
u64_stats_update_end(&tp->syncp);
|
2015-04-29 01:28:18 +03:00
|
|
|
tp->rcv_nxt = seq;
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Update our send window.
|
|
|
|
*
|
|
|
|
* Window update algorithm, described in RFC793/RFC1122 (used in linux-2.2
|
|
|
|
* and in FreeBSD. NetBSD's one is even worse.) is wrong.
|
|
|
|
*/
|
2011-10-21 13:22:42 +04:00
|
|
|
static int tcp_ack_update_window(struct sock *sk, const struct sk_buff *skb, u32 ack,
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
u32 ack_seq)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
int flag = 0;
|
2007-04-11 08:04:22 +04:00
|
|
|
u32 nwin = ntohs(tcp_hdr(skb)->window);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2007-04-11 08:04:22 +04:00
|
|
|
if (likely(!tcp_hdr(skb)->syn))
|
2005-04-17 02:20:36 +04:00
|
|
|
nwin <<= tp->rx_opt.snd_wscale;
|
|
|
|
|
|
|
|
if (tcp_may_update_window(tp, ack, ack_seq, nwin)) {
|
|
|
|
flag |= FLAG_WIN_UPDATE;
|
2009-03-03 09:42:02 +03:00
|
|
|
tcp_update_wl(tp, ack_seq);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (tp->snd_wnd != nwin) {
|
|
|
|
tp->snd_wnd = nwin;
|
|
|
|
|
|
|
|
/* Note, it is the only place, where
|
|
|
|
* fast path is recovered for sending TCP.
|
|
|
|
*/
|
2005-10-27 12:47:46 +04:00
|
|
|
tp->pred_flags = 0;
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_fast_path_check(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
tcp: fix slow start after idle vs TSO/GSO
slow start after idle might reduce cwnd, but we perform this
after first packet was cooked and sent.
With TSO/GSO, it means that we might send a full TSO packet
even if cwnd should have been reduced to IW10.
Moving the SSAI check in skb_entail() makes sense, because
we slightly reduce number of times this check is done,
especially for large send() and TCP Small queue callbacks from
softirq context.
As Neal pointed out, we also need to perform the check
if/when receive window opens.
Tested:
Following packetdrill test demonstrates the problem
// Test of slow start after idle
`sysctl -q net.ipv4.tcp_slow_start_after_idle=1`
0.000 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 65535 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.100 < . 1:1(0) ack 1 win 511
+0 accept(3, ..., ...) = 4
+0 setsockopt(4, SOL_SOCKET, SO_SNDBUF, [200000], 4) = 0
+0 write(4, ..., 26000) = 26000
+0 > . 1:5001(5000) ack 1
+0 > . 5001:10001(5000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10 }%
+.100 < . 1:1(0) ack 10001 win 511
+0 %{ assert tcpi_snd_cwnd == 20, tcpi_snd_cwnd }%
+0 > . 10001:20001(10000) ack 1
+0 > P. 20001:26001(6000) ack 1
+.100 < . 1:1(0) ack 26001 win 511
+0 %{ assert tcpi_snd_cwnd == 36, tcpi_snd_cwnd }%
+4 write(4, ..., 20000) = 20000
// If slow start after idle works properly, we should send 5 MSS here (cwnd/2)
+0 > . 26001:31001(5000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
+0 > . 31001:36001(5000) ack 1
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-21 22:30:00 +03:00
|
|
|
if (tcp_send_head(sk))
|
|
|
|
tcp_slow_start_after_idle_check(sk);
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
if (nwin > tp->max_window) {
|
|
|
|
tp->max_window = nwin;
|
2005-12-14 10:26:10 +03:00
|
|
|
tcp_sync_mss(sk, inet_csk(sk)->icsk_pmtu_cookie);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2015-04-29 01:28:17 +03:00
|
|
|
tcp_snd_una_update(tp, ack);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
return flag;
|
|
|
|
}
|
|
|
|
|
2015-03-17 07:06:20 +03:00
|
|
|
/* Return true if we're currently rate-limiting out-of-window ACKs and
|
|
|
|
* thus shouldn't send a dupack right now. We rate-limit dupacks in
|
|
|
|
* response to out-of-window SYNs or ACKs to mitigate ACK loops or DoS
|
|
|
|
* attacks that send repeated SYNs or ACKs for the same connection. To
|
|
|
|
* do this, we do not send a duplicate SYNACK or ACK if the remote
|
|
|
|
* endpoint is sending out-of-window SYNs or pure ACKs at a high rate.
|
|
|
|
*/
|
|
|
|
bool tcp_oow_rate_limited(struct net *net, const struct sk_buff *skb,
|
|
|
|
int mib_idx, u32 *last_oow_ack_time)
|
|
|
|
{
|
|
|
|
/* Data packets without SYNs are not likely part of an ACK loop. */
|
|
|
|
if ((TCP_SKB_CB(skb)->seq != TCP_SKB_CB(skb)->end_seq) &&
|
|
|
|
!tcp_hdr(skb)->syn)
|
|
|
|
goto not_rate_limited;
|
|
|
|
|
|
|
|
if (*last_oow_ack_time) {
|
|
|
|
s32 elapsed = (s32)(tcp_time_stamp - *last_oow_ack_time);
|
|
|
|
|
|
|
|
if (0 <= elapsed && elapsed < sysctl_tcp_invalid_ratelimit) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(net, mib_idx);
|
2015-03-17 07:06:20 +03:00
|
|
|
return true; /* rate-limited: don't send yet! */
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
*last_oow_ack_time = tcp_time_stamp;
|
|
|
|
|
|
|
|
not_rate_limited:
|
|
|
|
return false; /* not rate-limited: go ahead, send dupack now! */
|
|
|
|
}
|
|
|
|
|
2012-10-21 23:57:11 +04:00
|
|
|
/* RFC 5961 7 [ACK Throttling] */
|
2015-02-07 00:04:40 +03:00
|
|
|
static void tcp_send_challenge_ack(struct sock *sk, const struct sk_buff *skb)
|
2012-10-21 23:57:11 +04:00
|
|
|
{
|
|
|
|
/* unprotected vars, we dont care of overwrites */
|
|
|
|
static u32 challenge_timestamp;
|
|
|
|
static unsigned int challenge_count;
|
2015-02-07 00:04:40 +03:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u32 now;
|
|
|
|
|
|
|
|
/* First check our per-socket dupack rate limit. */
|
|
|
|
if (tcp_oow_rate_limited(sock_net(sk), skb,
|
|
|
|
LINUX_MIB_TCPACKSKIPPEDCHALLENGE,
|
|
|
|
&tp->last_oow_ack_time))
|
|
|
|
return;
|
2012-10-21 23:57:11 +04:00
|
|
|
|
2015-02-07 00:04:40 +03:00
|
|
|
/* Then check the check host-wide RFC 5961 rate limit. */
|
|
|
|
now = jiffies / HZ;
|
2012-10-21 23:57:11 +04:00
|
|
|
if (now != challenge_timestamp) {
|
|
|
|
challenge_timestamp = now;
|
|
|
|
challenge_count = 0;
|
|
|
|
}
|
|
|
|
if (++challenge_count <= sysctl_tcp_challenge_ack_limit) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPCHALLENGEACK);
|
2012-10-21 23:57:11 +04:00
|
|
|
tcp_send_ack(sk);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
tcp: call tcp_replace_ts_recent() from tcp_ack()
commit bd090dfc634d (tcp: tcp_replace_ts_recent() should not be called
from tcp_validate_incoming()) introduced a TS ecr bug in slow path
processing.
1 A > B P. 1:10001(10000) ack 1 <nop,nop,TS val 1001 ecr 200>
2 B < A . 1:1(0) ack 1 win 257 <sack 9001:10001,TS val 300 ecr 1001>
3 A > B . 1:1001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
4 A > B . 1001:2001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
(ecr 200 should be ecr 300 in packets 3 & 4)
Problem is tcp_ack() can trigger send of new packets (retransmits),
reflecting the prior TSval, instead of the TSval contained in the
currently processed incoming packet.
Fix this by calling tcp_replace_ts_recent() from tcp_ack() after the
checks, but before the actions.
Reported-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 11:19:48 +04:00
|
|
|
static void tcp_store_ts_recent(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
tp->rx_opt.ts_recent = tp->rx_opt.rcv_tsval;
|
|
|
|
tp->rx_opt.ts_recent_stamp = get_seconds();
|
|
|
|
}
|
|
|
|
|
|
|
|
static void tcp_replace_ts_recent(struct tcp_sock *tp, u32 seq)
|
|
|
|
{
|
|
|
|
if (tp->rx_opt.saw_tstamp && !after(seq, tp->rcv_wup)) {
|
|
|
|
/* PAWS bug workaround wrt. ACK frames, the PAWS discard
|
|
|
|
* extra check below makes sure this can only happen
|
|
|
|
* for pure ACK frames. -DaveM
|
|
|
|
*
|
|
|
|
* Not only, also it occurs for expired timestamps.
|
|
|
|
*/
|
|
|
|
|
|
|
|
if (tcp_paws_check(&tp->rx_opt, 0))
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2013-03-11 14:00:44 +04:00
|
|
|
/* This routine deals with acks during a TLP episode.
|
tcp: avoid reducing cwnd when ACK+DSACK is received
With TLP, the peer may reply to a probe with an
ACK+D-SACK, with ack value set to tlp_high_seq. In the current code,
such ACK+DSACK will be missed and only at next, higher ack will the TLP
episode be considered done. Since the DSACK is not present anymore,
this will cost a cwnd reduction.
This patch ensures that this scenario does not cause a cwnd reduction, since
receiving an ACK+DSACK indicates that both the initial segment and the probe
have been received by the peer.
The following packetdrill test, from Neal Cardwell, validates this patch:
// Establish a connection.
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.020 < . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
// Send 1 packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1
// Loss probe retransmission.
// packets_out == 1 => schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
// In this case, this means: 1.5*RTT + 200ms = 230ms
+.230 > P. 1:1001(1000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Receiver ACKs at tlp_high_seq with a DSACK,
// indicating they received the original packet and probe.
+.020 < . 1:1(0) ack 1001 win 257 <sack 1:1001,nop,nop>
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Send another packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1001:2001(1000) ack 1
// Receiver ACKs above tlp_high_seq, which should end the TLP episode
// if we haven't already. We should not reduce cwnd.
+.020 < . 1:1(0) ack 2001 win 257
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
Credits:
-Gregory helped in finding that tcp_process_tlp_ack was where the cwnd
got reduced in our MPTCP tests.
-Neal wrote the packetdrill test above
-Yuchung reworked the patch to make it more readable.
Cc: Gregory Detal <gregory.detal@uclouvain.be>
Cc: Nandita Dukkipati <nanditad@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Reviewed-by: Yuchung Cheng <ycheng@google.com>
Reviewed-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: Sébastien Barré <sebastien.barre@uclouvain.be>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-12 12:30:40 +03:00
|
|
|
* We mark the end of a TLP episode on receiving TLP dupack or when
|
|
|
|
* ack is after tlp_high_seq.
|
2013-03-11 14:00:44 +04:00
|
|
|
* Ref: loss detection algorithm in draft-dukkipati-tcpm-tcp-loss-probe.
|
|
|
|
*/
|
|
|
|
static void tcp_process_tlp_ack(struct sock *sk, u32 ack, int flag)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
tcp: avoid reducing cwnd when ACK+DSACK is received
With TLP, the peer may reply to a probe with an
ACK+D-SACK, with ack value set to tlp_high_seq. In the current code,
such ACK+DSACK will be missed and only at next, higher ack will the TLP
episode be considered done. Since the DSACK is not present anymore,
this will cost a cwnd reduction.
This patch ensures that this scenario does not cause a cwnd reduction, since
receiving an ACK+DSACK indicates that both the initial segment and the probe
have been received by the peer.
The following packetdrill test, from Neal Cardwell, validates this patch:
// Establish a connection.
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.020 < . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
// Send 1 packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1
// Loss probe retransmission.
// packets_out == 1 => schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
// In this case, this means: 1.5*RTT + 200ms = 230ms
+.230 > P. 1:1001(1000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Receiver ACKs at tlp_high_seq with a DSACK,
// indicating they received the original packet and probe.
+.020 < . 1:1(0) ack 1001 win 257 <sack 1:1001,nop,nop>
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Send another packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1001:2001(1000) ack 1
// Receiver ACKs above tlp_high_seq, which should end the TLP episode
// if we haven't already. We should not reduce cwnd.
+.020 < . 1:1(0) ack 2001 win 257
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
Credits:
-Gregory helped in finding that tcp_process_tlp_ack was where the cwnd
got reduced in our MPTCP tests.
-Neal wrote the packetdrill test above
-Yuchung reworked the patch to make it more readable.
Cc: Gregory Detal <gregory.detal@uclouvain.be>
Cc: Nandita Dukkipati <nanditad@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Reviewed-by: Yuchung Cheng <ycheng@google.com>
Reviewed-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: Sébastien Barré <sebastien.barre@uclouvain.be>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-12 12:30:40 +03:00
|
|
|
if (before(ack, tp->tlp_high_seq))
|
2013-03-11 14:00:44 +04:00
|
|
|
return;
|
|
|
|
|
tcp: avoid reducing cwnd when ACK+DSACK is received
With TLP, the peer may reply to a probe with an
ACK+D-SACK, with ack value set to tlp_high_seq. In the current code,
such ACK+DSACK will be missed and only at next, higher ack will the TLP
episode be considered done. Since the DSACK is not present anymore,
this will cost a cwnd reduction.
This patch ensures that this scenario does not cause a cwnd reduction, since
receiving an ACK+DSACK indicates that both the initial segment and the probe
have been received by the peer.
The following packetdrill test, from Neal Cardwell, validates this patch:
// Establish a connection.
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.020 < . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
// Send 1 packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1
// Loss probe retransmission.
// packets_out == 1 => schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
// In this case, this means: 1.5*RTT + 200ms = 230ms
+.230 > P. 1:1001(1000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Receiver ACKs at tlp_high_seq with a DSACK,
// indicating they received the original packet and probe.
+.020 < . 1:1(0) ack 1001 win 257 <sack 1:1001,nop,nop>
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Send another packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1001:2001(1000) ack 1
// Receiver ACKs above tlp_high_seq, which should end the TLP episode
// if we haven't already. We should not reduce cwnd.
+.020 < . 1:1(0) ack 2001 win 257
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
Credits:
-Gregory helped in finding that tcp_process_tlp_ack was where the cwnd
got reduced in our MPTCP tests.
-Neal wrote the packetdrill test above
-Yuchung reworked the patch to make it more readable.
Cc: Gregory Detal <gregory.detal@uclouvain.be>
Cc: Nandita Dukkipati <nanditad@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Reviewed-by: Yuchung Cheng <ycheng@google.com>
Reviewed-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: Sébastien Barré <sebastien.barre@uclouvain.be>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-12 12:30:40 +03:00
|
|
|
if (flag & FLAG_DSACKING_ACK) {
|
|
|
|
/* This DSACK means original and TLP probe arrived; no loss */
|
|
|
|
tp->tlp_high_seq = 0;
|
|
|
|
} else if (after(ack, tp->tlp_high_seq)) {
|
|
|
|
/* ACK advances: there was a loss, so reduce cwnd. Reset
|
|
|
|
* tlp_high_seq in tcp_init_cwnd_reduction()
|
|
|
|
*/
|
|
|
|
tcp_init_cwnd_reduction(sk);
|
|
|
|
tcp_set_ca_state(sk, TCP_CA_CWR);
|
|
|
|
tcp_end_cwnd_reduction(sk);
|
|
|
|
tcp_try_keep_open(sk);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk),
|
|
|
|
LINUX_MIB_TCPLOSSPROBERECOVERY);
|
tcp: avoid reducing cwnd when ACK+DSACK is received
With TLP, the peer may reply to a probe with an
ACK+D-SACK, with ack value set to tlp_high_seq. In the current code,
such ACK+DSACK will be missed and only at next, higher ack will the TLP
episode be considered done. Since the DSACK is not present anymore,
this will cost a cwnd reduction.
This patch ensures that this scenario does not cause a cwnd reduction, since
receiving an ACK+DSACK indicates that both the initial segment and the probe
have been received by the peer.
The following packetdrill test, from Neal Cardwell, validates this patch:
// Establish a connection.
0 socket(..., SOCK_STREAM, IPPROTO_TCP) = 3
+0 setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
+0 bind(3, ..., ...) = 0
+0 listen(3, 1) = 0
+0 < S 0:0(0) win 32792 <mss 1000,sackOK,nop,nop,nop,wscale 7>
+0 > S. 0:0(0) ack 1 <mss 1460,nop,nop,sackOK,nop,wscale 6>
+.020 < . 1:1(0) ack 1 win 257
+0 accept(3, ..., ...) = 4
// Send 1 packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1:1001(1000) ack 1
// Loss probe retransmission.
// packets_out == 1 => schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
// In this case, this means: 1.5*RTT + 200ms = 230ms
+.230 > P. 1:1001(1000) ack 1
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Receiver ACKs at tlp_high_seq with a DSACK,
// indicating they received the original packet and probe.
+.020 < . 1:1(0) ack 1001 win 257 <sack 1:1001,nop,nop>
+0 %{ assert tcpi_snd_cwnd == 10 }%
// Send another packet.
+0 write(4, ..., 1000) = 1000
+0 > P. 1001:2001(1000) ack 1
// Receiver ACKs above tlp_high_seq, which should end the TLP episode
// if we haven't already. We should not reduce cwnd.
+.020 < . 1:1(0) ack 2001 win 257
+0 %{ assert tcpi_snd_cwnd == 10, tcpi_snd_cwnd }%
Credits:
-Gregory helped in finding that tcp_process_tlp_ack was where the cwnd
got reduced in our MPTCP tests.
-Neal wrote the packetdrill test above
-Yuchung reworked the patch to make it more readable.
Cc: Gregory Detal <gregory.detal@uclouvain.be>
Cc: Nandita Dukkipati <nanditad@google.com>
Tested-by: Neal Cardwell <ncardwell@google.com>
Reviewed-by: Yuchung Cheng <ycheng@google.com>
Reviewed-by: Eric Dumazet <eric.dumazet@gmail.com>
Signed-off-by: Sébastien Barré <sebastien.barre@uclouvain.be>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-12 12:30:40 +03:00
|
|
|
} else if (!(flag & (FLAG_SND_UNA_ADVANCED |
|
|
|
|
FLAG_NOT_DUP | FLAG_DATA_SACKED))) {
|
|
|
|
/* Pure dupack: original and TLP probe arrived; no loss */
|
2013-03-11 14:00:44 +04:00
|
|
|
tp->tlp_high_seq = 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-09-27 00:37:34 +04:00
|
|
|
static inline void tcp_in_ack_event(struct sock *sk, u32 flags)
|
|
|
|
{
|
|
|
|
const struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
if (icsk->icsk_ca_ops->in_ack_event)
|
|
|
|
icsk->icsk_ca_ops->in_ack_event(sk, flags);
|
|
|
|
}
|
|
|
|
|
2016-02-02 21:33:04 +03:00
|
|
|
/* Congestion control has updated the cwnd already. So if we're in
|
|
|
|
* loss recovery then now we do any new sends (for FRTO) or
|
|
|
|
* retransmits (for CA_Loss or CA_recovery) that make sense.
|
|
|
|
*/
|
|
|
|
static void tcp_xmit_recovery(struct sock *sk, int rexmit)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
if (rexmit == REXMIT_NONE)
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (unlikely(rexmit == 2)) {
|
|
|
|
__tcp_push_pending_frames(sk, tcp_current_mss(sk),
|
|
|
|
TCP_NAGLE_OFF);
|
|
|
|
if (after(tp->snd_nxt, tp->high_seq))
|
|
|
|
return;
|
|
|
|
tp->frto = 0;
|
|
|
|
}
|
|
|
|
tcp_xmit_retransmit_queue(sk);
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* This routine deals with incoming acks, but not outgoing ones. */
|
2011-10-21 13:22:42 +04:00
|
|
|
static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 11:03:31 +04:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2015-05-01 02:10:57 +03:00
|
|
|
struct tcp_sacktag_state sack_state;
|
2005-04-17 02:20:36 +04:00
|
|
|
u32 prior_snd_una = tp->snd_una;
|
|
|
|
u32 ack_seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
u32 ack = TCP_SKB_CB(skb)->ack_seq;
|
2011-11-16 12:58:01 +04:00
|
|
|
bool is_dupack = false;
|
2007-11-11 08:22:18 +03:00
|
|
|
u32 prior_fackets;
|
2013-05-21 19:12:07 +04:00
|
|
|
int prior_packets = tp->packets_out;
|
2016-02-02 21:33:06 +03:00
|
|
|
u32 prior_delivered = tp->delivered;
|
2013-05-29 18:20:11 +04:00
|
|
|
int acked = 0; /* Number of packets newly acked */
|
2016-02-02 21:33:04 +03:00
|
|
|
int rexmit = REXMIT_NONE; /* Flag to (re)transmit to recover losses */
|
2015-05-01 02:10:57 +03:00
|
|
|
|
2015-05-01 02:10:58 +03:00
|
|
|
sack_state.first_sackt.v64 = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2014-10-12 02:17:29 +04:00
|
|
|
/* We very likely will need to access write queue head. */
|
|
|
|
prefetchw(sk->sk_write_queue.next);
|
|
|
|
|
2009-03-23 07:49:57 +03:00
|
|
|
/* If the ack is older than previous acks
|
2005-04-17 02:20:36 +04:00
|
|
|
* then we can probably ignore it.
|
|
|
|
*/
|
2012-10-21 23:57:11 +04:00
|
|
|
if (before(ack, prior_snd_una)) {
|
|
|
|
/* RFC 5961 5.2 [Blind Data Injection Attack].[Mitigation] */
|
|
|
|
if (before(ack, prior_snd_una - tp->max_window)) {
|
2015-02-07 00:04:40 +03:00
|
|
|
tcp_send_challenge_ack(sk, skb);
|
2012-10-21 23:57:11 +04:00
|
|
|
return -1;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
goto old_ack;
|
2012-10-21 23:57:11 +04:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2009-03-23 07:49:57 +03:00
|
|
|
/* If the ack includes data we haven't sent yet, discard
|
|
|
|
* this segment (RFC793 Section 3.9).
|
|
|
|
*/
|
|
|
|
if (after(ack, tp->snd_nxt))
|
|
|
|
goto invalid_ack;
|
|
|
|
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 14:00:43 +04:00
|
|
|
if (icsk->icsk_pending == ICSK_TIME_EARLY_RETRANS ||
|
|
|
|
icsk->icsk_pending == ICSK_TIME_LOSS_PROBE)
|
2012-05-02 17:30:04 +04:00
|
|
|
tcp_rearm_rto(sk);
|
|
|
|
|
2014-08-15 00:13:07 +04:00
|
|
|
if (after(ack, prior_snd_una)) {
|
2007-08-03 06:46:58 +04:00
|
|
|
flag |= FLAG_SND_UNA_ADVANCED;
|
2014-08-15 00:13:07 +04:00
|
|
|
icsk->icsk_retransmits = 0;
|
|
|
|
}
|
2007-08-03 06:46:58 +04:00
|
|
|
|
2007-11-11 08:22:18 +03:00
|
|
|
prior_fackets = tp->fackets_out;
|
|
|
|
|
tcp: call tcp_replace_ts_recent() from tcp_ack()
commit bd090dfc634d (tcp: tcp_replace_ts_recent() should not be called
from tcp_validate_incoming()) introduced a TS ecr bug in slow path
processing.
1 A > B P. 1:10001(10000) ack 1 <nop,nop,TS val 1001 ecr 200>
2 B < A . 1:1(0) ack 1 win 257 <sack 9001:10001,TS val 300 ecr 1001>
3 A > B . 1:1001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
4 A > B . 1001:2001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
(ecr 200 should be ecr 300 in packets 3 & 4)
Problem is tcp_ack() can trigger send of new packets (retransmits),
reflecting the prior TSval, instead of the TSval contained in the
currently processed incoming packet.
Fix this by calling tcp_replace_ts_recent() from tcp_ack() after the
checks, but before the actions.
Reported-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 11:19:48 +04:00
|
|
|
/* ts_recent update must be made after we are sure that the packet
|
|
|
|
* is in window.
|
|
|
|
*/
|
|
|
|
if (flag & FLAG_UPDATE_TS_RECENT)
|
|
|
|
tcp_replace_ts_recent(tp, TCP_SKB_CB(skb)->seq);
|
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
if (!(flag & FLAG_SLOWPATH) && after(ack, prior_snd_una)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Window is constant, pure forward advance.
|
|
|
|
* No more checks are required.
|
|
|
|
* Note, we use the fact that SND.UNA>=SND.WL2.
|
|
|
|
*/
|
2009-03-03 09:42:02 +03:00
|
|
|
tcp_update_wl(tp, ack_seq);
|
2015-04-29 01:28:17 +03:00
|
|
|
tcp_snd_una_update(tp, ack);
|
2005-04-17 02:20:36 +04:00
|
|
|
flag |= FLAG_WIN_UPDATE;
|
|
|
|
|
2014-09-27 00:37:35 +04:00
|
|
|
tcp_in_ack_event(sk, CA_ACK_WIN_UPDATE);
|
2005-06-23 23:19:55 +04:00
|
|
|
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPHPACKS);
|
2005-04-17 02:20:36 +04:00
|
|
|
} else {
|
2014-09-27 00:37:35 +04:00
|
|
|
u32 ack_ev_flags = CA_ACK_SLOWPATH;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
if (ack_seq != TCP_SKB_CB(skb)->end_seq)
|
|
|
|
flag |= FLAG_DATA;
|
|
|
|
else
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPPUREACKS);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
flag |= tcp_ack_update_window(sk, skb, ack, ack_seq);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->sacked)
|
2013-07-23 03:20:47 +04:00
|
|
|
flag |= tcp_sacktag_write_queue(sk, skb, prior_snd_una,
|
2015-05-01 02:10:57 +03:00
|
|
|
&sack_state);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
if (tcp_ecn_rcv_ecn_echo(tp, tcp_hdr(skb))) {
|
2005-04-17 02:20:36 +04:00
|
|
|
flag |= FLAG_ECE;
|
2014-09-27 00:37:35 +04:00
|
|
|
ack_ev_flags |= CA_ACK_ECE;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (flag & FLAG_WIN_UPDATE)
|
|
|
|
ack_ev_flags |= CA_ACK_WIN_UPDATE;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2014-09-27 00:37:35 +04:00
|
|
|
tcp_in_ack_event(sk, ack_ev_flags);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* We passed data and got it acked, remove any soft error
|
|
|
|
* log. Something worked...
|
|
|
|
*/
|
|
|
|
sk->sk_err_soft = 0;
|
tcp: Clear probes_out more aggressively in tcp_ack().
This is based upon an excellent bug report from Eric Dumazet.
tcp_ack() should clear ->icsk_probes_out even if there are packets
outstanding. Otherwise if we get a sequence of ACKs while we do have
packets outstanding over and over again, we'll never clear the
probes_out value and eventually think the connection is too sick and
we'll reset it.
This appears to be some "optimization" added to tcp_ack() in the 2.4.x
timeframe. In 2.2.x, probes_out is pretty much always cleared by
tcp_ack().
Here is Eric's original report:
----------------------------------------
Apparently, we can in some situations reset TCP connections in a couple of seconds when some frames are lost.
In order to reproduce the problem, please try the following program on linux-2.6.25.*
Setup some iptables rules to allow two frames per second sent on loopback interface to tcp destination port 12000
iptables -N SLOWLO
iptables -A SLOWLO -m hashlimit --hashlimit 2 --hashlimit-burst 1 --hashlimit-mode dstip --hashlimit-name slow2 -j ACCEPT
iptables -A SLOWLO -j DROP
iptables -A OUTPUT -o lo -p tcp --dport 12000 -j SLOWLO
Then run the attached program and see the output :
# ./loop
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,1)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,3)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,5)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,7)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,9)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,200ms,11)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,201ms,13)
State Recv-Q Send-Q Local Address:Port Peer Address:Port
ESTAB 0 40 127.0.0.1:54455 127.0.0.1:12000 timer:(persist,188ms,15)
write(): Connection timed out
wrote 890 bytes but was interrupted after 9 seconds
ESTAB 0 0 127.0.0.1:12000 127.0.0.1:54455
Exiting read() because no data available (4000 ms timeout).
read 860 bytes
While this tcp session makes progress (sending frames with 50 bytes of payload, every 500ms), linux tcp stack decides to reset it, when tcp_retries 2 is reached (default value : 15)
tcpdump :
15:30:28.856695 IP 127.0.0.1.56554 > 127.0.0.1.12000: S 33788768:33788768(0) win 32792 <mss 16396,nop,nop,sackOK,nop,wscale 7>
15:30:28.856711 IP 127.0.0.1.12000 > 127.0.0.1.56554: S 33899253:33899253(0) ack 33788769 win 32792 <mss 16396,nop,nop,sackOK,nop,wscale 7>
15:30:29.356947 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 1:61(60) ack 1 win 257
15:30:29.356966 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 61 win 257
15:30:29.866415 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 61:111(50) ack 1 win 257
15:30:29.866427 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 111 win 257
15:30:30.366516 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 111:161(50) ack 1 win 257
15:30:30.366527 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 161 win 257
15:30:30.876196 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 161:211(50) ack 1 win 257
15:30:30.876207 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 211 win 257
15:30:31.376282 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 211:261(50) ack 1 win 257
15:30:31.376290 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 261 win 257
15:30:31.885619 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 261:311(50) ack 1 win 257
15:30:31.885631 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 311 win 257
15:30:32.385705 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 311:361(50) ack 1 win 257
15:30:32.385715 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 361 win 257
15:30:32.895249 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 361:411(50) ack 1 win 257
15:30:32.895266 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 411 win 257
15:30:33.395341 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 411:461(50) ack 1 win 257
15:30:33.395351 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 461 win 257
15:30:33.918085 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 461:511(50) ack 1 win 257
15:30:33.918096 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 511 win 257
15:30:34.418163 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 511:561(50) ack 1 win 257
15:30:34.418172 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 561 win 257
15:30:34.927685 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 561:611(50) ack 1 win 257
15:30:34.927698 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 611 win 257
15:30:35.427757 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 611:661(50) ack 1 win 257
15:30:35.427766 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 661 win 257
15:30:35.937359 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 661:711(50) ack 1 win 257
15:30:35.937376 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 711 win 257
15:30:36.437451 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 711:761(50) ack 1 win 257
15:30:36.437464 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 761 win 257
15:30:36.947022 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 761:811(50) ack 1 win 257
15:30:36.947039 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 811 win 257
15:30:37.447135 IP 127.0.0.1.56554 > 127.0.0.1.12000: P 811:861(50) ack 1 win 257
15:30:37.447203 IP 127.0.0.1.12000 > 127.0.0.1.56554: . ack 861 win 257
15:30:41.448171 IP 127.0.0.1.12000 > 127.0.0.1.56554: F 1:1(0) ack 861 win 257
15:30:41.448189 IP 127.0.0.1.56554 > 127.0.0.1.12000: R 33789629:33789629(0) win 0
Source of program :
/*
* small producer/consumer program.
* setup a listener on 127.0.0.1:12000
* Forks a child
* child connect to 127.0.0.1, and sends 10 bytes on this tcp socket every 100 ms
* Father accepts connection, and read all data
*/
#include <sys/types.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <unistd.h>
#include <stdio.h>
#include <time.h>
#include <sys/poll.h>
int port = 12000;
char buffer[4096];
int main(int argc, char *argv[])
{
int lfd = socket(AF_INET, SOCK_STREAM, 0);
struct sockaddr_in socket_address;
time_t t0, t1;
int on = 1, sfd, res;
unsigned long total = 0;
socklen_t alen = sizeof(socket_address);
pid_t pid;
time(&t0);
socket_address.sin_family = AF_INET;
socket_address.sin_port = htons(port);
socket_address.sin_addr.s_addr = htonl(INADDR_LOOPBACK);
if (lfd == -1) {
perror("socket()");
return 1;
}
setsockopt(lfd, SOL_SOCKET, SO_REUSEADDR, &on, sizeof(int));
if (bind(lfd, (struct sockaddr *)&socket_address, sizeof(socket_address)) == -1) {
perror("bind");
close(lfd);
return 1;
}
if (listen(lfd, 1) == -1) {
perror("listen()");
close(lfd);
return 1;
}
pid = fork();
if (pid == 0) {
int i, cfd = socket(AF_INET, SOCK_STREAM, 0);
close(lfd);
if (connect(cfd, (struct sockaddr *)&socket_address, sizeof(socket_address)) == -1) {
perror("connect()");
return 1;
}
for (i = 0 ; ;) {
res = write(cfd, "blablabla\n", 10);
if (res > 0) total += res;
else if (res == -1) {
perror("write()");
break;
} else break;
usleep(100000);
if (++i == 10) {
system("ss -on dst 127.0.0.1:12000");
i = 0;
}
}
time(&t1);
fprintf(stderr, "wrote %lu bytes but was interrupted after %g seconds\n", total, difftime(t1, t0));
system("ss -on | grep 127.0.0.1:12000");
close(cfd);
return 0;
}
sfd = accept(lfd, (struct sockaddr *)&socket_address, &alen);
if (sfd == -1) {
perror("accept");
return 1;
}
close(lfd);
while (1) {
struct pollfd pfd[1];
pfd[0].fd = sfd;
pfd[0].events = POLLIN;
if (poll(pfd, 1, 4000) == 0) {
fprintf(stderr, "Exiting read() because no data available (4000 ms timeout).\n");
break;
}
res = read(sfd, buffer, sizeof(buffer));
if (res > 0) total += res;
else if (res == 0) break;
else perror("read()");
}
fprintf(stderr, "read %lu bytes\n", total);
close(sfd);
return 0;
}
----------------------------------------
Signed-off-by: David S. Miller <davem@davemloft.net>
2008-07-24 03:38:45 +04:00
|
|
|
icsk->icsk_probes_out = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->rcv_tstamp = tcp_time_stamp;
|
|
|
|
if (!prior_packets)
|
|
|
|
goto no_queue;
|
|
|
|
|
|
|
|
/* See if we can take anything off of the retransmit queue. */
|
2016-02-02 21:33:07 +03:00
|
|
|
flag |= tcp_clean_rtx_queue(sk, prior_fackets, prior_snd_una, &acked,
|
2015-05-01 02:10:58 +03:00
|
|
|
&sack_state);
|
2011-08-22 00:21:57 +04:00
|
|
|
|
2005-08-10 11:03:31 +04:00
|
|
|
if (tcp_ack_is_dubious(sk, flag)) {
|
2011-11-16 12:58:01 +04:00
|
|
|
is_dupack = !(flag & (FLAG_SND_UNA_ADVANCED | FLAG_NOT_DUP));
|
2016-02-02 21:33:05 +03:00
|
|
|
tcp_fastretrans_alert(sk, acked, is_dupack, &flag, &rexmit);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2013-03-11 14:00:44 +04:00
|
|
|
if (tp->tlp_high_seq)
|
|
|
|
tcp_process_tlp_ack(sk, ack, flag);
|
|
|
|
|
2012-07-02 13:21:03 +04:00
|
|
|
if ((flag & FLAG_FORWARD_PROGRESS) || !(flag & FLAG_NOT_DUP)) {
|
|
|
|
struct dst_entry *dst = __sk_dst_get(sk);
|
|
|
|
if (dst)
|
|
|
|
dst_confirm(dst);
|
|
|
|
}
|
tcp: Tail loss probe (TLP)
This patch series implement the Tail loss probe (TLP) algorithm described
in http://tools.ietf.org/html/draft-dukkipati-tcpm-tcp-loss-probe-01. The
first patch implements the basic algorithm.
TLP's goal is to reduce tail latency of short transactions. It achieves
this by converting retransmission timeouts (RTOs) occuring due
to tail losses (losses at end of transactions) into fast recovery.
TLP transmits one packet in two round-trips when a connection is in
Open state and isn't receiving any ACKs. The transmitted packet, aka
loss probe, can be either new or a retransmission. When there is tail
loss, the ACK from a loss probe triggers FACK/early-retransmit based
fast recovery, thus avoiding a costly RTO. In the absence of loss,
there is no change in the connection state.
PTO stands for probe timeout. It is a timer event indicating
that an ACK is overdue and triggers a loss probe packet. The PTO value
is set to max(2*SRTT, 10ms) and is adjusted to account for delayed
ACK timer when there is only one oustanding packet.
TLP Algorithm
On transmission of new data in Open state:
-> packets_out > 1: schedule PTO in max(2*SRTT, 10ms).
-> packets_out == 1: schedule PTO in max(2*RTT, 1.5*RTT + 200ms)
-> PTO = min(PTO, RTO)
Conditions for scheduling PTO:
-> Connection is in Open state.
-> Connection is either cwnd limited or no new data to send.
-> Number of probes per tail loss episode is limited to one.
-> Connection is SACK enabled.
When PTO fires:
new_segment_exists:
-> transmit new segment.
-> packets_out++. cwnd remains same.
no_new_packet:
-> retransmit the last segment.
Its ACK triggers FACK or early retransmit based recovery.
ACK path:
-> rearm RTO at start of ACK processing.
-> reschedule PTO if need be.
In addition, the patch includes a small variation to the Early Retransmit
(ER) algorithm, such that ER and TLP together can in principle recover any
N-degree of tail loss through fast recovery. TLP is controlled by the same
sysctl as ER, tcp_early_retrans sysctl.
tcp_early_retrans==0; disables TLP and ER.
==1; enables RFC5827 ER.
==2; delayed ER.
==3; TLP and delayed ER. [DEFAULT]
==4; TLP only.
The TLP patch series have been extensively tested on Google Web servers.
It is most effective for short Web trasactions, where it reduced RTOs by 15%
and improved HTTP response time (average by 6%, 99th percentile by 10%).
The transmitted probes account for <0.5% of the overall transmissions.
Signed-off-by: Nandita Dukkipati <nanditad@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-03-11 14:00:43 +04:00
|
|
|
|
|
|
|
if (icsk->icsk_pending == ICSK_TIME_RETRANS)
|
|
|
|
tcp_schedule_loss_probe(sk);
|
2016-02-02 21:33:09 +03:00
|
|
|
tcp_cong_control(sk, ack, tp->delivered - prior_delivered, flag);
|
2016-02-02 21:33:04 +03:00
|
|
|
tcp_xmit_recovery(sk, rexmit);
|
2005-04-17 02:20:36 +04:00
|
|
|
return 1;
|
|
|
|
|
|
|
|
no_queue:
|
2011-11-16 12:58:02 +04:00
|
|
|
/* If data was DSACKed, see if we can undo a cwnd reduction. */
|
|
|
|
if (flag & FLAG_DSACKING_ACK)
|
2016-02-02 21:33:05 +03:00
|
|
|
tcp_fastretrans_alert(sk, acked, is_dupack, &flag, &rexmit);
|
2005-04-17 02:20:36 +04:00
|
|
|
/* If this ack opens up a zero window, clear backoff. It was
|
|
|
|
* being used to time the probes, and is probably far higher than
|
|
|
|
* it needs to be for normal retransmission.
|
|
|
|
*/
|
2007-03-07 23:12:44 +03:00
|
|
|
if (tcp_send_head(sk))
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_ack_probe(sk);
|
2013-03-11 14:00:44 +04:00
|
|
|
|
|
|
|
if (tp->tlp_high_seq)
|
|
|
|
tcp_process_tlp_ack(sk, ack, flag);
|
2005-04-17 02:20:36 +04:00
|
|
|
return 1;
|
|
|
|
|
2009-03-23 07:49:57 +03:00
|
|
|
invalid_ack:
|
|
|
|
SOCK_DEBUG(sk, "Ack %u after %u:%u\n", ack, tp->snd_una, tp->snd_nxt);
|
|
|
|
return -1;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
old_ack:
|
2011-11-16 12:58:03 +04:00
|
|
|
/* If data was SACKed, tag it and see if we should send more data.
|
|
|
|
* If data was DSACKed, see if we can undo a cwnd reduction.
|
|
|
|
*/
|
2008-06-04 22:34:22 +04:00
|
|
|
if (TCP_SKB_CB(skb)->sacked) {
|
2013-07-23 03:20:47 +04:00
|
|
|
flag |= tcp_sacktag_write_queue(sk, skb, prior_snd_una,
|
2015-05-01 02:10:57 +03:00
|
|
|
&sack_state);
|
2016-02-02 21:33:05 +03:00
|
|
|
tcp_fastretrans_alert(sk, acked, is_dupack, &flag, &rexmit);
|
2016-02-02 21:33:04 +03:00
|
|
|
tcp_xmit_recovery(sk, rexmit);
|
2008-06-04 22:34:22 +04:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2009-03-23 07:49:57 +03:00
|
|
|
SOCK_DEBUG(sk, "Ack %u before %u:%u\n", ack, tp->snd_una, tp->snd_nxt);
|
2005-04-17 02:20:36 +04:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2015-04-07 00:37:26 +03:00
|
|
|
static void tcp_parse_fastopen_option(int len, const unsigned char *cookie,
|
|
|
|
bool syn, struct tcp_fastopen_cookie *foc,
|
|
|
|
bool exp_opt)
|
|
|
|
{
|
|
|
|
/* Valid only in SYN or SYN-ACK with an even length. */
|
|
|
|
if (!foc || !syn || len < 0 || (len & 1))
|
|
|
|
return;
|
|
|
|
|
|
|
|
if (len >= TCP_FASTOPEN_COOKIE_MIN &&
|
|
|
|
len <= TCP_FASTOPEN_COOKIE_MAX)
|
|
|
|
memcpy(foc->val, cookie, len);
|
|
|
|
else if (len != 0)
|
|
|
|
len = -1;
|
|
|
|
foc->len = len;
|
|
|
|
foc->exp = exp_opt;
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Look for tcp options. Normally only called on SYN and SYNACK packets.
|
|
|
|
* But, this can also be called on packets in the established flow when
|
|
|
|
* the fast version below fails.
|
|
|
|
*/
|
2013-03-17 12:23:34 +04:00
|
|
|
void tcp_parse_options(const struct sk_buff *skb,
|
|
|
|
struct tcp_options_received *opt_rx, int estab,
|
2012-07-19 10:43:05 +04:00
|
|
|
struct tcp_fastopen_cookie *foc)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2011-10-21 13:22:42 +04:00
|
|
|
const unsigned char *ptr;
|
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
2008-01-01 01:57:14 +03:00
|
|
|
int length = (th->doff * 4) - sizeof(struct tcphdr);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2011-10-21 13:22:42 +04:00
|
|
|
ptr = (const unsigned char *)(th + 1);
|
2005-04-17 02:20:36 +04:00
|
|
|
opt_rx->saw_tstamp = 0;
|
|
|
|
|
2007-03-09 07:45:19 +03:00
|
|
|
while (length > 0) {
|
2008-01-01 01:57:14 +03:00
|
|
|
int opcode = *ptr++;
|
2005-04-17 02:20:36 +04:00
|
|
|
int opsize;
|
|
|
|
|
|
|
|
switch (opcode) {
|
2008-01-04 07:36:55 +03:00
|
|
|
case TCPOPT_EOL:
|
|
|
|
return;
|
|
|
|
case TCPOPT_NOP: /* Ref: RFC 793 section 3.1 */
|
|
|
|
length--;
|
|
|
|
continue;
|
|
|
|
default:
|
|
|
|
opsize = *ptr++;
|
|
|
|
if (opsize < 2) /* "silly options" */
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
2008-01-04 07:36:55 +03:00
|
|
|
if (opsize > length)
|
|
|
|
return; /* don't parse partial options */
|
|
|
|
switch (opcode) {
|
|
|
|
case TCPOPT_MSS:
|
|
|
|
if (opsize == TCPOLEN_MSS && th->syn && !estab) {
|
2008-05-03 03:26:16 +04:00
|
|
|
u16 in_mss = get_unaligned_be16(ptr);
|
2008-01-04 07:36:55 +03:00
|
|
|
if (in_mss) {
|
|
|
|
if (opt_rx->user_mss &&
|
|
|
|
opt_rx->user_mss < in_mss)
|
|
|
|
in_mss = opt_rx->user_mss;
|
|
|
|
opt_rx->mss_clamp = in_mss;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2008-01-04 07:36:55 +03:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case TCPOPT_WINDOW:
|
|
|
|
if (opsize == TCPOLEN_WINDOW && th->syn &&
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 07:56:42 +03:00
|
|
|
!estab && sysctl_tcp_window_scaling) {
|
2008-01-04 07:36:55 +03:00
|
|
|
__u8 snd_wscale = *(__u8 *)ptr;
|
|
|
|
opt_rx->wscale_ok = 1;
|
|
|
|
if (snd_wscale > 14) {
|
2012-05-14 01:56:26 +04:00
|
|
|
net_info_ratelimited("%s: Illegal window scaling value %d >14 received\n",
|
|
|
|
__func__,
|
|
|
|
snd_wscale);
|
2008-01-04 07:36:55 +03:00
|
|
|
snd_wscale = 14;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2008-01-04 07:36:55 +03:00
|
|
|
opt_rx->snd_wscale = snd_wscale;
|
|
|
|
}
|
|
|
|
break;
|
|
|
|
case TCPOPT_TIMESTAMP:
|
|
|
|
if ((opsize == TCPOLEN_TIMESTAMP) &&
|
|
|
|
((estab && opt_rx->tstamp_ok) ||
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 07:56:42 +03:00
|
|
|
(!estab && sysctl_tcp_timestamps))) {
|
2008-01-04 07:36:55 +03:00
|
|
|
opt_rx->saw_tstamp = 1;
|
2008-05-03 03:26:16 +04:00
|
|
|
opt_rx->rcv_tsval = get_unaligned_be32(ptr);
|
|
|
|
opt_rx->rcv_tsecr = get_unaligned_be32(ptr + 4);
|
2008-01-04 07:36:55 +03:00
|
|
|
}
|
|
|
|
break;
|
|
|
|
case TCPOPT_SACK_PERM:
|
|
|
|
if (opsize == TCPOLEN_SACK_PERM && th->syn &&
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 07:56:42 +03:00
|
|
|
!estab && sysctl_tcp_sack) {
|
2011-12-20 17:23:24 +04:00
|
|
|
opt_rx->sack_ok = TCP_SACK_SEEN;
|
2008-01-04 07:36:55 +03:00
|
|
|
tcp_sack_reset(opt_rx);
|
|
|
|
}
|
|
|
|
break;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-01-04 07:36:55 +03:00
|
|
|
case TCPOPT_SACK:
|
|
|
|
if ((opsize >= (TCPOLEN_SACK_BASE + TCPOLEN_SACK_PERBLOCK)) &&
|
|
|
|
!((opsize - TCPOLEN_SACK_BASE) % TCPOLEN_SACK_PERBLOCK) &&
|
|
|
|
opt_rx->sack_ok) {
|
|
|
|
TCP_SKB_CB(skb)->sacked = (ptr - 2) - (unsigned char *)th;
|
|
|
|
}
|
|
|
|
break;
|
2006-11-15 06:07:45 +03:00
|
|
|
#ifdef CONFIG_TCP_MD5SIG
|
2008-01-04 07:36:55 +03:00
|
|
|
case TCPOPT_MD5SIG:
|
|
|
|
/*
|
|
|
|
* The MD5 Hash has already been
|
|
|
|
* checked (see tcp_v{4,6}_do_rcv()).
|
|
|
|
*/
|
|
|
|
break;
|
2006-11-15 06:07:45 +03:00
|
|
|
#endif
|
2015-04-07 00:37:26 +03:00
|
|
|
case TCPOPT_FASTOPEN:
|
|
|
|
tcp_parse_fastopen_option(
|
|
|
|
opsize - TCPOLEN_FASTOPEN_BASE,
|
|
|
|
ptr, th->syn, foc, false);
|
|
|
|
break;
|
|
|
|
|
2012-07-19 10:43:05 +04:00
|
|
|
case TCPOPT_EXP:
|
|
|
|
/* Fast Open option shares code 254 using a
|
2015-04-07 00:37:26 +03:00
|
|
|
* 16 bits magic number.
|
2012-07-19 10:43:05 +04:00
|
|
|
*/
|
2015-04-07 00:37:26 +03:00
|
|
|
if (opsize >= TCPOLEN_EXP_FASTOPEN_BASE &&
|
|
|
|
get_unaligned_be16(ptr) ==
|
|
|
|
TCPOPT_FASTOPEN_MAGIC)
|
|
|
|
tcp_parse_fastopen_option(opsize -
|
|
|
|
TCPOLEN_EXP_FASTOPEN_BASE,
|
|
|
|
ptr + 2, th->syn, foc, true);
|
2012-07-19 10:43:05 +04:00
|
|
|
break;
|
|
|
|
|
|
|
|
}
|
2008-01-04 07:36:55 +03:00
|
|
|
ptr += opsize-2;
|
|
|
|
length -= opsize;
|
2007-04-21 04:09:22 +04:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
2010-07-10 01:22:10 +04:00
|
|
|
EXPORT_SYMBOL(tcp_parse_options);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_parse_aligned_timestamp(struct tcp_sock *tp, const struct tcphdr *th)
|
2008-08-23 16:12:29 +04:00
|
|
|
{
|
2011-10-21 13:22:42 +04:00
|
|
|
const __be32 *ptr = (const __be32 *)(th + 1);
|
2008-08-23 16:12:29 +04:00
|
|
|
|
|
|
|
if (*ptr == htonl((TCPOPT_NOP << 24) | (TCPOPT_NOP << 16)
|
|
|
|
| (TCPOPT_TIMESTAMP << 8) | TCPOLEN_TIMESTAMP)) {
|
|
|
|
tp->rx_opt.saw_tstamp = 1;
|
|
|
|
++ptr;
|
|
|
|
tp->rx_opt.rcv_tsval = ntohl(*ptr);
|
|
|
|
++ptr;
|
2013-08-27 12:21:55 +04:00
|
|
|
if (*ptr)
|
|
|
|
tp->rx_opt.rcv_tsecr = ntohl(*ptr) - tp->tsoffset;
|
|
|
|
else
|
|
|
|
tp->rx_opt.rcv_tsecr = 0;
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2008-08-23 16:12:29 +04:00
|
|
|
}
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2008-08-23 16:12:29 +04:00
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Fast parse options. This hopes to only see timestamps.
|
|
|
|
* If it is wrong it falls back on tcp_parse_options().
|
|
|
|
*/
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_fast_parse_options(const struct sk_buff *skb,
|
2013-03-17 12:23:34 +04:00
|
|
|
const struct tcphdr *th, struct tcp_sock *tp)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2009-12-02 21:25:27 +03:00
|
|
|
/* In the spirit of fast parsing, compare doff directly to constant
|
|
|
|
* values. Because equality is used, short doff can be ignored here.
|
|
|
|
*/
|
|
|
|
if (th->doff == (sizeof(*th) / 4)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2005-04-17 02:20:36 +04:00
|
|
|
} else if (tp->rx_opt.tstamp_ok &&
|
2009-12-02 21:25:27 +03:00
|
|
|
th->doff == ((sizeof(*th) + TCPOLEN_TSTAMP_ALIGNED) / 4)) {
|
2008-08-23 16:12:29 +04:00
|
|
|
if (tcp_parse_aligned_timestamp(tp, th))
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2013-02-11 09:50:19 +04:00
|
|
|
|
2013-03-17 12:23:34 +04:00
|
|
|
tcp_parse_options(skb, &tp->rx_opt, 1, NULL);
|
2013-08-27 12:21:55 +04:00
|
|
|
if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
|
2013-02-11 09:50:19 +04:00
|
|
|
tp->rx_opt.rcv_tsecr -= tp->tsoffset;
|
|
|
|
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2008-04-17 07:29:53 +04:00
|
|
|
#ifdef CONFIG_TCP_MD5SIG
|
|
|
|
/*
|
|
|
|
* Parse MD5 Signature option
|
|
|
|
*/
|
2011-10-21 13:22:42 +04:00
|
|
|
const u8 *tcp_parse_md5sig_option(const struct tcphdr *th)
|
2008-04-17 07:29:53 +04:00
|
|
|
{
|
2011-10-21 13:22:42 +04:00
|
|
|
int length = (th->doff << 2) - sizeof(*th);
|
|
|
|
const u8 *ptr = (const u8 *)(th + 1);
|
2008-04-17 07:29:53 +04:00
|
|
|
|
|
|
|
/* If the TCP option is too short, we can short cut */
|
|
|
|
if (length < TCPOLEN_MD5SIG)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
while (length > 0) {
|
|
|
|
int opcode = *ptr++;
|
|
|
|
int opsize;
|
|
|
|
|
2013-12-23 10:37:26 +04:00
|
|
|
switch (opcode) {
|
2008-04-17 07:29:53 +04:00
|
|
|
case TCPOPT_EOL:
|
|
|
|
return NULL;
|
|
|
|
case TCPOPT_NOP:
|
|
|
|
length--;
|
|
|
|
continue;
|
|
|
|
default:
|
|
|
|
opsize = *ptr++;
|
|
|
|
if (opsize < 2 || opsize > length)
|
|
|
|
return NULL;
|
|
|
|
if (opcode == TCPOPT_MD5SIG)
|
2010-08-08 07:24:28 +04:00
|
|
|
return opsize == TCPOLEN_MD5SIG ? ptr : NULL;
|
2008-04-17 07:29:53 +04:00
|
|
|
}
|
|
|
|
ptr += opsize - 2;
|
|
|
|
length -= opsize;
|
|
|
|
}
|
|
|
|
return NULL;
|
|
|
|
}
|
2010-07-10 01:22:10 +04:00
|
|
|
EXPORT_SYMBOL(tcp_parse_md5sig_option);
|
2008-04-17 07:29:53 +04:00
|
|
|
#endif
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Sorry, PAWS as specified is broken wrt. pure-ACKs -DaveM
|
|
|
|
*
|
|
|
|
* It is not fatal. If this ACK does _not_ change critical state (seqs, window)
|
|
|
|
* it can pass through stack. So, the following predicate verifies that
|
|
|
|
* this segment is not used for anything but congestion avoidance or
|
|
|
|
* fast retransmit. Moreover, we even are able to eliminate most of such
|
|
|
|
* second order effects, if we apply some small "replay" window (~RTO)
|
|
|
|
* to timestamp space.
|
|
|
|
*
|
|
|
|
* All these measures still do not guarantee that we reject wrapped ACKs
|
|
|
|
* on networks with high bandwidth, when sequence space is recycled fastly,
|
|
|
|
* but it guarantees that such events will be very rare and do not affect
|
|
|
|
* connection seriously. This doesn't look nice, but alas, PAWS is really
|
|
|
|
* buggy extension.
|
|
|
|
*
|
|
|
|
* [ Later note. Even worse! It is buggy for segments _with_ data. RFC
|
|
|
|
* states that events when retransmit arrives after original data are rare.
|
|
|
|
* It is a blatant lie. VJ forgot about fast retransmit! 8)8) It is
|
|
|
|
* the biggest problem on large power networks even with minor reordering.
|
|
|
|
* OK, let's give it small replay window. If peer clock is even 1hz, it is safe
|
|
|
|
* up to bandwidth of 18Gigabit/sec. 8) ]
|
|
|
|
*/
|
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
static int tcp_disordered_ack(const struct sock *sk, const struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2011-10-21 13:22:42 +04:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
u32 seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
u32 ack = TCP_SKB_CB(skb)->ack_seq;
|
|
|
|
|
|
|
|
return (/* 1. Pure ACK with correct sequence number. */
|
|
|
|
(th->ack && seq == TCP_SKB_CB(skb)->end_seq && seq == tp->rcv_nxt) &&
|
|
|
|
|
|
|
|
/* 2. ... and duplicate ACK. */
|
|
|
|
ack == tp->snd_una &&
|
|
|
|
|
|
|
|
/* 3. ... and does not update window. */
|
|
|
|
!tcp_may_update_window(tp, ack, seq, ntohs(th->window) << tp->rx_opt.snd_wscale) &&
|
|
|
|
|
|
|
|
/* 4. ... and sits in replay window. */
|
2005-08-10 07:10:42 +04:00
|
|
|
(s32)(tp->rx_opt.ts_recent - tp->rx_opt.rcv_tsval) <= (inet_csk(sk)->icsk_rto * 1024) / HZ);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2012-07-20 01:32:18 +04:00
|
|
|
static inline bool tcp_paws_discard(const struct sock *sk,
|
2008-01-01 01:57:14 +03:00
|
|
|
const struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 07:10:42 +04:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
2009-03-14 17:23:03 +03:00
|
|
|
|
|
|
|
return !tcp_paws_check(&tp->rx_opt, TCP_PAWS_WINDOW) &&
|
|
|
|
!tcp_disordered_ack(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* Check segment sequence number for validity.
|
|
|
|
*
|
|
|
|
* Segment controls are considered valid, if the segment
|
|
|
|
* fits to the window after truncation to the window. Acceptability
|
|
|
|
* of data (and SYN, FIN, of course) is checked separately.
|
|
|
|
* See tcp_data_queue(), for example.
|
|
|
|
*
|
|
|
|
* Also, controls (RST is main one) are accepted using RCV.WUP instead
|
|
|
|
* of RCV.NXT. Peer still did not advance his SND.UNA when we
|
|
|
|
* delayed ACK, so that hisSND.UNA<=ourRCV.WUP.
|
|
|
|
* (borrowed from freebsd)
|
|
|
|
*/
|
|
|
|
|
2012-07-20 01:32:18 +04:00
|
|
|
static inline bool tcp_sequence(const struct tcp_sock *tp, u32 seq, u32 end_seq)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
return !before(end_seq, tp->rcv_wup) &&
|
|
|
|
!after(seq, tp->rcv_nxt + tcp_receive_window(tp));
|
|
|
|
}
|
|
|
|
|
|
|
|
/* When we get a reset we do this. */
|
2012-08-31 16:29:11 +04:00
|
|
|
void tcp_reset(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
/* We want the right error as BSD sees it (and indeed as we do). */
|
|
|
|
switch (sk->sk_state) {
|
2008-01-01 01:57:14 +03:00
|
|
|
case TCP_SYN_SENT:
|
|
|
|
sk->sk_err = ECONNREFUSED;
|
|
|
|
break;
|
|
|
|
case TCP_CLOSE_WAIT:
|
|
|
|
sk->sk_err = EPIPE;
|
|
|
|
break;
|
|
|
|
case TCP_CLOSE:
|
|
|
|
return;
|
|
|
|
default:
|
|
|
|
sk->sk_err = ECONNRESET;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2010-09-21 02:42:05 +04:00
|
|
|
/* This barrier is coupled with smp_rmb() in tcp_poll() */
|
|
|
|
smp_wmb();
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD))
|
|
|
|
sk->sk_error_report(sk);
|
|
|
|
|
|
|
|
tcp_done(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Process the FIN bit. This now behaves as it is supposed to work
|
|
|
|
* and the FIN takes effect when it is validly part of sequence
|
|
|
|
* space. Not before when we get holes.
|
|
|
|
*
|
|
|
|
* If we are ESTABLISHED, a received fin moves us to CLOSE-WAIT
|
|
|
|
* (and thence onto LAST-ACK and finally, CLOSE, we never enter
|
|
|
|
* TIME-WAIT)
|
|
|
|
*
|
|
|
|
* If we are in FINWAIT-1, a received FIN indicates simultaneous
|
|
|
|
* close and we go into CLOSING (and later onto TIME-WAIT)
|
|
|
|
*
|
|
|
|
* If we are in FINWAIT-2, a received FIN moves us to TIME-WAIT.
|
|
|
|
*/
|
2016-02-06 22:16:28 +03:00
|
|
|
void tcp_fin(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
inet_csk_schedule_ack(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
sk->sk_shutdown |= RCV_SHUTDOWN;
|
|
|
|
sock_set_flag(sk, SOCK_DONE);
|
|
|
|
|
|
|
|
switch (sk->sk_state) {
|
2008-01-01 01:57:14 +03:00
|
|
|
case TCP_SYN_RECV:
|
|
|
|
case TCP_ESTABLISHED:
|
|
|
|
/* Move to CLOSE_WAIT */
|
|
|
|
tcp_set_state(sk, TCP_CLOSE_WAIT);
|
2015-07-08 03:12:28 +03:00
|
|
|
inet_csk(sk)->icsk_ack.pingpong = 1;
|
2008-01-01 01:57:14 +03:00
|
|
|
break;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
case TCP_CLOSE_WAIT:
|
|
|
|
case TCP_CLOSING:
|
|
|
|
/* Received a retransmission of the FIN, do
|
|
|
|
* nothing.
|
|
|
|
*/
|
|
|
|
break;
|
|
|
|
case TCP_LAST_ACK:
|
|
|
|
/* RFC793: Remain in the LAST-ACK state. */
|
|
|
|
break;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
case TCP_FIN_WAIT1:
|
|
|
|
/* This case occurs when a simultaneous close
|
|
|
|
* happens, we must ack the received FIN and
|
|
|
|
* enter the CLOSING state.
|
|
|
|
*/
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
tcp_set_state(sk, TCP_CLOSING);
|
|
|
|
break;
|
|
|
|
case TCP_FIN_WAIT2:
|
|
|
|
/* Received a FIN -- send ACK and enter TIME_WAIT. */
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
tcp_time_wait(sk, TCP_TIME_WAIT, 0);
|
|
|
|
break;
|
|
|
|
default:
|
|
|
|
/* Only TCP_LISTEN and TCP_CLOSE are left, in these
|
|
|
|
* cases we should never reach this piece of code.
|
|
|
|
*/
|
2012-03-11 22:36:11 +04:00
|
|
|
pr_err("%s: Impossible, sk->sk_state=%d\n",
|
2008-03-06 07:47:47 +03:00
|
|
|
__func__, sk->sk_state);
|
2008-01-01 01:57:14 +03:00
|
|
|
break;
|
2007-04-21 04:09:22 +04:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* It _is_ possible, that we have something out-of-order _after_ FIN.
|
|
|
|
* Probably, we should reset in this case. For now drop them.
|
|
|
|
*/
|
|
|
|
__skb_queue_purge(&tp->out_of_order_queue);
|
2007-08-09 16:14:46 +04:00
|
|
|
if (tcp_is_sack(tp))
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_sack_reset(&tp->rx_opt);
|
2007-12-31 11:11:19 +03:00
|
|
|
sk_mem_reclaim(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD)) {
|
|
|
|
sk->sk_state_change(sk);
|
|
|
|
|
|
|
|
/* Do not send POLL_HUP for half duplex close. */
|
|
|
|
if (sk->sk_shutdown == SHUTDOWN_MASK ||
|
|
|
|
sk->sk_state == TCP_CLOSE)
|
2007-11-26 15:10:50 +03:00
|
|
|
sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_HUP);
|
2005-04-17 02:20:36 +04:00
|
|
|
else
|
2007-11-26 15:10:50 +03:00
|
|
|
sk_wake_async(sk, SOCK_WAKE_WAITD, POLL_IN);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-05-17 03:15:34 +04:00
|
|
|
static inline bool tcp_sack_extend(struct tcp_sack_block *sp, u32 seq,
|
2008-01-01 01:57:14 +03:00
|
|
|
u32 end_seq)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
if (!after(seq, sp->end_seq) && !after(sp->start_seq, end_seq)) {
|
|
|
|
if (before(seq, sp->start_seq))
|
|
|
|
sp->start_seq = seq;
|
|
|
|
if (after(end_seq, sp->end_seq))
|
|
|
|
sp->end_seq = end_seq;
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2008-07-17 07:29:51 +04:00
|
|
|
static void tcp_dsack_set(struct sock *sk, u32 seq, u32 end_seq)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2008-07-17 07:29:51 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 07:56:42 +03:00
|
|
|
if (tcp_is_sack(tp) && sysctl_tcp_dsack) {
|
2008-07-03 12:05:41 +04:00
|
|
|
int mib_idx;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
if (before(seq, tp->rcv_nxt))
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKOLDSENT;
|
2005-04-17 02:20:36 +04:00
|
|
|
else
|
2008-07-03 12:05:41 +04:00
|
|
|
mib_idx = LINUX_MIB_TCPDSACKOFOSENT;
|
|
|
|
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), mib_idx);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
tp->rx_opt.dsack = 1;
|
|
|
|
tp->duplicate_sack[0].start_seq = seq;
|
|
|
|
tp->duplicate_sack[0].end_seq = end_seq;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-07-17 07:29:51 +04:00
|
|
|
static void tcp_dsack_extend(struct sock *sk, u32 seq, u32 end_seq)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2008-07-17 07:29:51 +04:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
if (!tp->rx_opt.dsack)
|
2008-07-17 07:29:51 +04:00
|
|
|
tcp_dsack_set(sk, seq, end_seq);
|
2005-04-17 02:20:36 +04:00
|
|
|
else
|
|
|
|
tcp_sack_extend(tp->duplicate_sack, seq, end_seq);
|
|
|
|
}
|
|
|
|
|
2011-10-21 13:22:42 +04:00
|
|
|
static void tcp_send_dupack(struct sock *sk, const struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
|
|
|
|
before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt)) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_DELAYEDACKLOST);
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_enter_quickack_mode(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
tcp: Revert per-route SACK/DSACK/TIMESTAMP changes.
It creates a regression, triggering badness for SYN_RECV
sockets, for example:
[19148.022102] Badness at net/ipv4/inet_connection_sock.c:293
[19148.022570] NIP: c02a0914 LR: c02a0904 CTR: 00000000
[19148.023035] REGS: eeecbd30 TRAP: 0700 Not tainted (2.6.32)
[19148.023496] MSR: 00029032 <EE,ME,CE,IR,DR> CR: 24002442 XER: 00000000
[19148.024012] TASK = eee9a820[1756] 'privoxy' THREAD: eeeca000
This is likely caused by the change in the 'estab' parameter
passed to tcp_parse_options() when invoked by the functions
in net/ipv4/tcp_minisocks.c
But even if that is fixed, the ->conn_request() changes made in
this patch series is fundamentally wrong. They try to use the
listening socket's 'dst' to probe the route settings. The
listening socket doesn't even have a route, and you can't
get the right route (the child request one) until much later
after we setup all of the state, and it must be done by hand.
This stuff really isn't ready, so the best thing to do is a
full revert. This reverts the following commits:
f55017a93f1a74d50244b1254b9a2bd7ac9bbf7d
022c3f7d82f0f1c68018696f2f027b87b9bb45c2
1aba721eba1d84a2defce45b950272cee1e6c72a
cda42ebd67ee5fdf09d7057b5a4584d36fe8a335
345cda2fd695534be5a4494f1b59da9daed33663
dc343475ed062e13fc260acccaab91d7d80fd5b2
05eaade2782fb0c90d3034fd7a7d5a16266182bb
6a2a2d6bf8581216e08be15fcb563cfd6c430e1e
Signed-off-by: David S. Miller <davem@davemloft.net>
2009-12-16 07:56:42 +03:00
|
|
|
if (tcp_is_sack(tp) && sysctl_tcp_dsack) {
|
2005-04-17 02:20:36 +04:00
|
|
|
u32 end_seq = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
|
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt))
|
|
|
|
end_seq = tp->rcv_nxt;
|
2008-07-17 07:29:51 +04:00
|
|
|
tcp_dsack_set(sk, TCP_SKB_CB(skb)->seq, end_seq);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* These routines update the SACK block as out-of-order packets arrive or
|
|
|
|
* in-order packets close up the sequence space.
|
|
|
|
*/
|
|
|
|
static void tcp_sack_maybe_coalesce(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
int this_sack;
|
|
|
|
struct tcp_sack_block *sp = &tp->selective_acks[0];
|
2008-01-01 01:57:14 +03:00
|
|
|
struct tcp_sack_block *swalk = sp + 1;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* See if the recent change to the first SACK eats into
|
|
|
|
* or hits the sequence space of other SACK blocks, if so coalesce.
|
|
|
|
*/
|
2008-01-01 01:57:14 +03:00
|
|
|
for (this_sack = 1; this_sack < tp->rx_opt.num_sacks;) {
|
2005-04-17 02:20:36 +04:00
|
|
|
if (tcp_sack_extend(sp, swalk->start_seq, swalk->end_seq)) {
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/* Zap SWALK, by moving every further SACK up by one slot.
|
|
|
|
* Decrease num_sacks.
|
|
|
|
*/
|
|
|
|
tp->rx_opt.num_sacks--;
|
2008-01-01 01:57:14 +03:00
|
|
|
for (i = this_sack; i < tp->rx_opt.num_sacks; i++)
|
|
|
|
sp[i] = sp[i + 1];
|
2005-04-17 02:20:36 +04:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
this_sack++, swalk++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static void tcp_sack_new_ofo_skb(struct sock *sk, u32 seq, u32 end_seq)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct tcp_sack_block *sp = &tp->selective_acks[0];
|
|
|
|
int cur_sacks = tp->rx_opt.num_sacks;
|
|
|
|
int this_sack;
|
|
|
|
|
|
|
|
if (!cur_sacks)
|
|
|
|
goto new_sack;
|
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
for (this_sack = 0; this_sack < cur_sacks; this_sack++, sp++) {
|
2005-04-17 02:20:36 +04:00
|
|
|
if (tcp_sack_extend(sp, seq, end_seq)) {
|
|
|
|
/* Rotate this_sack to the first one. */
|
2008-01-01 01:57:14 +03:00
|
|
|
for (; this_sack > 0; this_sack--, sp--)
|
2009-03-21 23:36:17 +03:00
|
|
|
swap(*sp, *(sp - 1));
|
2005-04-17 02:20:36 +04:00
|
|
|
if (cur_sacks > 1)
|
|
|
|
tcp_sack_maybe_coalesce(tp);
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Could not find an adjacent existing SACK, build a new one,
|
|
|
|
* put it at the front, and shift everyone else down. We
|
|
|
|
* always know there is at least one SACK present already here.
|
|
|
|
*
|
|
|
|
* If the sack array is full, forget about the last one.
|
|
|
|
*/
|
2008-07-19 11:07:02 +04:00
|
|
|
if (this_sack >= TCP_NUM_SACKS) {
|
2005-04-17 02:20:36 +04:00
|
|
|
this_sack--;
|
|
|
|
tp->rx_opt.num_sacks--;
|
|
|
|
sp--;
|
|
|
|
}
|
2007-03-09 07:45:19 +03:00
|
|
|
for (; this_sack > 0; this_sack--, sp--)
|
2008-01-01 01:57:14 +03:00
|
|
|
*sp = *(sp - 1);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
new_sack:
|
|
|
|
/* Build the new head SACK, and we're done. */
|
|
|
|
sp->start_seq = seq;
|
|
|
|
sp->end_seq = end_seq;
|
|
|
|
tp->rx_opt.num_sacks++;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* RCV.NXT advances, some SACKs should be eaten. */
|
|
|
|
|
|
|
|
static void tcp_sack_remove(struct tcp_sock *tp)
|
|
|
|
{
|
|
|
|
struct tcp_sack_block *sp = &tp->selective_acks[0];
|
|
|
|
int num_sacks = tp->rx_opt.num_sacks;
|
|
|
|
int this_sack;
|
|
|
|
|
|
|
|
/* Empty ofo queue, hence, all the SACKs are eaten. Clear. */
|
2005-07-09 01:57:23 +04:00
|
|
|
if (skb_queue_empty(&tp->out_of_order_queue)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->rx_opt.num_sacks = 0;
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
for (this_sack = 0; this_sack < num_sacks;) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Check if the start of the sack is covered by RCV.NXT. */
|
|
|
|
if (!before(tp->rcv_nxt, sp->start_seq)) {
|
|
|
|
int i;
|
|
|
|
|
|
|
|
/* RCV.NXT must cover all the block! */
|
2008-07-26 08:43:18 +04:00
|
|
|
WARN_ON(before(tp->rcv_nxt, sp->end_seq));
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Zap this SACK, by moving forward any other SACKS. */
|
2013-12-23 10:37:26 +04:00
|
|
|
for (i = this_sack+1; i < num_sacks; i++)
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->selective_acks[i-1] = tp->selective_acks[i];
|
|
|
|
num_sacks--;
|
|
|
|
continue;
|
|
|
|
}
|
|
|
|
this_sack++;
|
|
|
|
sp++;
|
|
|
|
}
|
2009-03-14 17:23:01 +03:00
|
|
|
tp->rx_opt.num_sacks = num_sacks;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2014-09-19 19:26:20 +04:00
|
|
|
/**
|
|
|
|
* tcp_try_coalesce - try to merge skb to prior one
|
|
|
|
* @sk: socket
|
|
|
|
* @to: prior buffer
|
|
|
|
* @from: buffer to add in queue
|
|
|
|
* @fragstolen: pointer to boolean
|
|
|
|
*
|
|
|
|
* Before queueing skb @from after @to, try to merge them
|
|
|
|
* to reduce overall memory use and queue lengths, if cost is small.
|
|
|
|
* Packets in ofo or receive queues can stay a long time.
|
|
|
|
* Better try to coalesce them right now to avoid future collapses.
|
|
|
|
* Returns true if caller should free @from instead of queueing it
|
|
|
|
*/
|
|
|
|
static bool tcp_try_coalesce(struct sock *sk,
|
|
|
|
struct sk_buff *to,
|
|
|
|
struct sk_buff *from,
|
|
|
|
bool *fragstolen)
|
|
|
|
{
|
|
|
|
int delta;
|
|
|
|
|
|
|
|
*fragstolen = false;
|
|
|
|
|
|
|
|
/* Its possible this segment overlaps with prior segment in queue */
|
|
|
|
if (TCP_SKB_CB(from)->seq != TCP_SKB_CB(to)->end_seq)
|
|
|
|
return false;
|
|
|
|
|
|
|
|
if (!skb_try_coalesce(to, from, fragstolen, &delta))
|
|
|
|
return false;
|
|
|
|
|
|
|
|
atomic_add(delta, &sk->sk_rmem_alloc);
|
|
|
|
sk_mem_charge(sk, delta);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPRCVCOALESCE);
|
2014-09-19 19:26:20 +04:00
|
|
|
TCP_SKB_CB(to)->end_seq = TCP_SKB_CB(from)->end_seq;
|
|
|
|
TCP_SKB_CB(to)->ack_seq = TCP_SKB_CB(from)->ack_seq;
|
|
|
|
TCP_SKB_CB(to)->tcp_flags |= TCP_SKB_CB(from)->tcp_flags;
|
|
|
|
return true;
|
|
|
|
}
|
|
|
|
|
2016-04-01 18:52:19 +03:00
|
|
|
static void tcp_drop(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
sk_drops_add(sk, skb);
|
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* This one checks to see if we can put data from the
|
|
|
|
* out_of_order queue into the receive_queue.
|
|
|
|
*/
|
|
|
|
static void tcp_ofo_queue(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
__u32 dsack_high = tp->rcv_nxt;
|
2014-09-19 19:26:20 +04:00
|
|
|
struct sk_buff *skb, *tail;
|
|
|
|
bool fragstolen, eaten;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
while ((skb = skb_peek(&tp->out_of_order_queue)) != NULL) {
|
|
|
|
if (after(TCP_SKB_CB(skb)->seq, tp->rcv_nxt))
|
|
|
|
break;
|
|
|
|
|
|
|
|
if (before(TCP_SKB_CB(skb)->seq, dsack_high)) {
|
|
|
|
__u32 dsack = dsack_high;
|
|
|
|
if (before(TCP_SKB_CB(skb)->end_seq, dsack_high))
|
|
|
|
dsack_high = TCP_SKB_CB(skb)->end_seq;
|
2008-07-17 07:29:51 +04:00
|
|
|
tcp_dsack_extend(sk, TCP_SKB_CB(skb)->seq, dsack);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2014-09-19 19:26:20 +04:00
|
|
|
__skb_unlink(skb, &tp->out_of_order_queue);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt)) {
|
2010-03-24 10:57:28 +03:00
|
|
|
SOCK_DEBUG(sk, "ofo packet was already received\n");
|
2016-04-01 18:52:19 +03:00
|
|
|
tcp_drop(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
continue;
|
|
|
|
}
|
|
|
|
SOCK_DEBUG(sk, "ofo requeuing : rcv_next %X seq %X - %X\n",
|
|
|
|
tp->rcv_nxt, TCP_SKB_CB(skb)->seq,
|
|
|
|
TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
2014-09-19 19:26:20 +04:00
|
|
|
tail = skb_peek_tail(&sk->sk_receive_queue);
|
|
|
|
eaten = tail && tcp_try_coalesce(sk, tail, skb, &fragstolen);
|
2015-04-29 01:28:18 +03:00
|
|
|
tcp_rcv_nxt_update(tp, TCP_SKB_CB(skb)->end_seq);
|
2014-09-19 19:26:20 +04:00
|
|
|
if (!eaten)
|
|
|
|
__skb_queue_tail(&sk->sk_receive_queue, skb);
|
2014-09-15 15:19:51 +04:00
|
|
|
if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
|
2011-10-21 01:44:03 +04:00
|
|
|
tcp_fin(sk);
|
2014-09-19 19:26:20 +04:00
|
|
|
if (eaten)
|
|
|
|
kfree_skb_partial(skb, fragstolen);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_prune_ofo_queue(struct sock *sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
static int tcp_prune_queue(struct sock *sk);
|
|
|
|
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-08-01 03:44:41 +04:00
|
|
|
static int tcp_try_rmem_schedule(struct sock *sk, struct sk_buff *skb,
|
|
|
|
unsigned int size)
|
2008-04-15 11:33:38 +04:00
|
|
|
{
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) > sk->sk_rcvbuf ||
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-08-01 03:44:41 +04:00
|
|
|
!sk_rmem_schedule(sk, skb, size)) {
|
2008-04-15 11:33:38 +04:00
|
|
|
|
|
|
|
if (tcp_prune_queue(sk) < 0)
|
|
|
|
return -1;
|
|
|
|
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-08-01 03:44:41 +04:00
|
|
|
if (!sk_rmem_schedule(sk, skb, size)) {
|
2008-04-16 07:26:34 +04:00
|
|
|
if (!tcp_prune_ofo_queue(sk))
|
|
|
|
return -1;
|
|
|
|
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-08-01 03:44:41 +04:00
|
|
|
if (!sk_rmem_schedule(sk, skb, size))
|
2008-04-15 11:33:38 +04:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2012-03-18 15:06:44 +04:00
|
|
|
static void tcp_data_queue_ofo(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb1;
|
|
|
|
u32 seq, end_seq;
|
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
tcp_ecn_check_ce(tp, skb);
|
2012-03-18 15:06:44 +04:00
|
|
|
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-08-01 03:44:41 +04:00
|
|
|
if (unlikely(tcp_try_rmem_schedule(sk, skb, skb->truesize))) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPOFODROP);
|
2016-04-01 18:52:19 +03:00
|
|
|
tcp_drop(sk, skb);
|
2012-03-18 15:06:44 +04:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Disable header prediction. */
|
|
|
|
tp->pred_flags = 0;
|
|
|
|
inet_csk_schedule_ack(sk);
|
|
|
|
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPOFOQUEUE);
|
2012-03-18 15:06:44 +04:00
|
|
|
SOCK_DEBUG(sk, "out of order segment: rcv_next %X seq %X - %X\n",
|
|
|
|
tp->rcv_nxt, TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
|
|
|
skb1 = skb_peek_tail(&tp->out_of_order_queue);
|
|
|
|
if (!skb1) {
|
|
|
|
/* Initial out of order segment, build 1 SACK. */
|
|
|
|
if (tcp_is_sack(tp)) {
|
|
|
|
tp->rx_opt.num_sacks = 1;
|
|
|
|
tp->selective_acks[0].start_seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
tp->selective_acks[0].end_seq =
|
|
|
|
TCP_SKB_CB(skb)->end_seq;
|
|
|
|
}
|
|
|
|
__skb_queue_head(&tp->out_of_order_queue, skb);
|
|
|
|
goto end;
|
|
|
|
}
|
|
|
|
|
|
|
|
seq = TCP_SKB_CB(skb)->seq;
|
|
|
|
end_seq = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
|
|
|
|
if (seq == TCP_SKB_CB(skb1)->end_seq) {
|
2012-04-27 04:38:33 +04:00
|
|
|
bool fragstolen;
|
|
|
|
|
|
|
|
if (!tcp_try_coalesce(sk, skb1, skb, &fragstolen)) {
|
2012-04-23 11:11:42 +04:00
|
|
|
__skb_queue_after(&tp->out_of_order_queue, skb1, skb);
|
|
|
|
} else {
|
2013-09-06 21:35:58 +04:00
|
|
|
tcp_grow_window(sk, skb);
|
2012-05-02 11:55:58 +04:00
|
|
|
kfree_skb_partial(skb, fragstolen);
|
2012-03-18 15:07:47 +04:00
|
|
|
skb = NULL;
|
|
|
|
}
|
2012-03-18 15:06:44 +04:00
|
|
|
|
|
|
|
if (!tp->rx_opt.num_sacks ||
|
|
|
|
tp->selective_acks[0].end_seq != seq)
|
|
|
|
goto add_sack;
|
|
|
|
|
|
|
|
/* Common case: data arrive in order after hole. */
|
|
|
|
tp->selective_acks[0].end_seq = end_seq;
|
|
|
|
goto end;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Find place to insert this segment. */
|
|
|
|
while (1) {
|
|
|
|
if (!after(TCP_SKB_CB(skb1)->seq, seq))
|
|
|
|
break;
|
|
|
|
if (skb_queue_is_first(&tp->out_of_order_queue, skb1)) {
|
|
|
|
skb1 = NULL;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
skb1 = skb_queue_prev(&tp->out_of_order_queue, skb1);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Do skb overlap to previous one? */
|
|
|
|
if (skb1 && before(seq, TCP_SKB_CB(skb1)->end_seq)) {
|
|
|
|
if (!after(end_seq, TCP_SKB_CB(skb1)->end_seq)) {
|
|
|
|
/* All the bits are present. Drop. */
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPOFOMERGE);
|
2016-04-01 18:52:19 +03:00
|
|
|
tcp_drop(sk, skb);
|
2012-03-18 15:06:44 +04:00
|
|
|
skb = NULL;
|
|
|
|
tcp_dsack_set(sk, seq, end_seq);
|
|
|
|
goto add_sack;
|
|
|
|
}
|
|
|
|
if (after(seq, TCP_SKB_CB(skb1)->seq)) {
|
|
|
|
/* Partial overlap. */
|
|
|
|
tcp_dsack_set(sk, seq,
|
|
|
|
TCP_SKB_CB(skb1)->end_seq);
|
|
|
|
} else {
|
|
|
|
if (skb_queue_is_first(&tp->out_of_order_queue,
|
|
|
|
skb1))
|
|
|
|
skb1 = NULL;
|
|
|
|
else
|
|
|
|
skb1 = skb_queue_prev(
|
|
|
|
&tp->out_of_order_queue,
|
|
|
|
skb1);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!skb1)
|
|
|
|
__skb_queue_head(&tp->out_of_order_queue, skb);
|
|
|
|
else
|
|
|
|
__skb_queue_after(&tp->out_of_order_queue, skb1, skb);
|
|
|
|
|
|
|
|
/* And clean segments covered by new one as whole. */
|
|
|
|
while (!skb_queue_is_last(&tp->out_of_order_queue, skb)) {
|
|
|
|
skb1 = skb_queue_next(&tp->out_of_order_queue, skb);
|
|
|
|
|
|
|
|
if (!after(end_seq, TCP_SKB_CB(skb1)->seq))
|
|
|
|
break;
|
|
|
|
if (before(end_seq, TCP_SKB_CB(skb1)->end_seq)) {
|
|
|
|
tcp_dsack_extend(sk, TCP_SKB_CB(skb1)->seq,
|
|
|
|
end_seq);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
__skb_unlink(skb1, &tp->out_of_order_queue);
|
|
|
|
tcp_dsack_extend(sk, TCP_SKB_CB(skb1)->seq,
|
|
|
|
TCP_SKB_CB(skb1)->end_seq);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPOFOMERGE);
|
2016-04-01 18:52:19 +03:00
|
|
|
tcp_drop(sk, skb1);
|
2012-03-18 15:06:44 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
add_sack:
|
|
|
|
if (tcp_is_sack(tp))
|
|
|
|
tcp_sack_new_ofo_skb(sk, seq, end_seq);
|
|
|
|
end:
|
2013-09-06 21:35:58 +04:00
|
|
|
if (skb) {
|
|
|
|
tcp_grow_window(sk, skb);
|
2012-03-18 15:06:44 +04:00
|
|
|
skb_set_owner_r(skb, sk);
|
2013-09-06 21:35:58 +04:00
|
|
|
}
|
2012-03-18 15:06:44 +04:00
|
|
|
}
|
|
|
|
|
2012-05-10 05:49:41 +04:00
|
|
|
static int __must_check tcp_queue_rcv(struct sock *sk, struct sk_buff *skb, int hdrlen,
|
2012-05-02 13:58:29 +04:00
|
|
|
bool *fragstolen)
|
|
|
|
{
|
|
|
|
int eaten;
|
|
|
|
struct sk_buff *tail = skb_peek_tail(&sk->sk_receive_queue);
|
|
|
|
|
|
|
|
__skb_pull(skb, hdrlen);
|
|
|
|
eaten = (tail &&
|
|
|
|
tcp_try_coalesce(sk, tail, skb, fragstolen)) ? 1 : 0;
|
2015-04-29 01:28:18 +03:00
|
|
|
tcp_rcv_nxt_update(tcp_sk(sk), TCP_SKB_CB(skb)->end_seq);
|
2012-05-02 13:58:29 +04:00
|
|
|
if (!eaten) {
|
|
|
|
__skb_queue_tail(&sk->sk_receive_queue, skb);
|
|
|
|
skb_set_owner_r(skb, sk);
|
|
|
|
}
|
|
|
|
return eaten;
|
|
|
|
}
|
2012-03-18 15:06:44 +04:00
|
|
|
|
2012-05-10 05:49:41 +04:00
|
|
|
int tcp_send_rcvq(struct sock *sk, struct msghdr *msg, size_t size)
|
|
|
|
{
|
2014-09-17 14:14:42 +04:00
|
|
|
struct sk_buff *skb;
|
2015-11-19 08:03:33 +03:00
|
|
|
int err = -ENOMEM;
|
|
|
|
int data_len = 0;
|
2012-05-10 05:49:41 +04:00
|
|
|
bool fragstolen;
|
|
|
|
|
2012-10-29 09:05:33 +04:00
|
|
|
if (size == 0)
|
|
|
|
return 0;
|
|
|
|
|
2015-11-19 08:03:33 +03:00
|
|
|
if (size > PAGE_SIZE) {
|
|
|
|
int npages = min_t(size_t, size >> PAGE_SHIFT, MAX_SKB_FRAGS);
|
|
|
|
|
|
|
|
data_len = npages << PAGE_SHIFT;
|
|
|
|
size = data_len + (size & ~PAGE_MASK);
|
|
|
|
}
|
|
|
|
skb = alloc_skb_with_frags(size - data_len, data_len,
|
|
|
|
PAGE_ALLOC_COSTLY_ORDER,
|
|
|
|
&err, sk->sk_allocation);
|
2012-05-10 05:49:41 +04:00
|
|
|
if (!skb)
|
|
|
|
goto err;
|
|
|
|
|
2015-11-19 08:03:33 +03:00
|
|
|
skb_put(skb, size - data_len);
|
|
|
|
skb->data_len = data_len;
|
|
|
|
skb->len = size;
|
|
|
|
|
2014-09-17 14:14:42 +04:00
|
|
|
if (tcp_try_rmem_schedule(sk, skb, skb->truesize))
|
netvm: prevent a stream-specific deadlock
This patch series is based on top of "Swap-over-NBD without deadlocking
v15" as it depends on the same reservation of PF_MEMALLOC reserves logic.
When a user or administrator requires swap for their application, they
create a swap partition and file, format it with mkswap and activate it
with swapon. In diskless systems this is not an option so if swap if
required then swapping over the network is considered. The two likely
scenarios are when blade servers are used as part of a cluster where the
form factor or maintenance costs do not allow the use of disks and thin
clients.
The Linux Terminal Server Project recommends the use of the Network Block
Device (NBD) for swap but this is not always an option. There is no
guarantee that the network attached storage (NAS) device is running Linux
or supports NBD. However, it is likely that it supports NFS so there are
users that want support for swapping over NFS despite any performance
concern. Some distributions currently carry patches that support swapping
over NFS but it would be preferable to support it in the mainline kernel.
Patch 1 avoids a stream-specific deadlock that potentially affects TCP.
Patch 2 is a small modification to SELinux to avoid using PFMEMALLOC
reserves.
Patch 3 adds three helpers for filesystems to handle swap cache pages.
For example, page_file_mapping() returns page->mapping for
file-backed pages and the address_space of the underlying
swap file for swap cache pages.
Patch 4 adds two address_space_operations to allow a filesystem
to pin all metadata relevant to a swapfile in memory. Upon
successful activation, the swapfile is marked SWP_FILE and
the address space operation ->direct_IO is used for writing
and ->readpage for reading in swap pages.
Patch 5 notes that patch 3 is bolting
filesystem-specific-swapfile-support onto the side and that
the default handlers have different information to what
is available to the filesystem. This patch refactors the
code so that there are generic handlers for each of the new
address_space operations.
Patch 6 adds an API to allow a vector of kernel addresses to be
translated to struct pages and pinned for IO.
Patch 7 adds support for using highmem pages for swap by kmapping
the pages before calling the direct_IO handler.
Patch 8 updates NFS to use the helpers from patch 3 where necessary.
Patch 9 avoids setting PF_private on PG_swapcache pages within NFS.
Patch 10 implements the new swapfile-related address_space operations
for NFS and teaches the direct IO handler how to manage
kernel addresses.
Patch 11 prevents page allocator recursions in NFS by using GFP_NOIO
where appropriate.
Patch 12 fixes a NULL pointer dereference that occurs when using
swap-over-NFS.
With the patches applied, it is possible to mount a swapfile that is on an
NFS filesystem. Swap performance is not great with a swap stress test
taking roughly twice as long to complete than if the swap device was
backed by NBD.
This patch: netvm: prevent a stream-specific deadlock
It could happen that all !SOCK_MEMALLOC sockets have buffered so much data
that we're over the global rmem limit. This will prevent SOCK_MEMALLOC
buffers from receiving data, which will prevent userspace from running,
which is needed to reduce the buffered data.
Fix this by exempting the SOCK_MEMALLOC sockets from the rmem limit. Once
this change it applied, it is important that sockets that set
SOCK_MEMALLOC do not clear the flag until the socket is being torn down.
If this happens, a warning is generated and the tokens reclaimed to avoid
accounting errors until the bug is fixed.
[davem@davemloft.net: Warning about clearing SOCK_MEMALLOC]
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Signed-off-by: Mel Gorman <mgorman@suse.de>
Acked-by: David S. Miller <davem@davemloft.net>
Acked-by: Rik van Riel <riel@redhat.com>
Cc: Trond Myklebust <Trond.Myklebust@netapp.com>
Cc: Neil Brown <neilb@suse.de>
Cc: Christoph Hellwig <hch@infradead.org>
Cc: Mike Christie <michaelc@cs.wisc.edu>
Cc: Eric B Munson <emunson@mgebm.net>
Cc: Sebastian Andrzej Siewior <sebastian@breakpoint.cc>
Cc: Mel Gorman <mgorman@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2012-08-01 03:44:41 +04:00
|
|
|
goto err_free;
|
|
|
|
|
2015-11-19 08:03:33 +03:00
|
|
|
err = skb_copy_datagram_from_iter(skb, 0, &msg->msg_iter, size);
|
|
|
|
if (err)
|
2012-05-10 05:49:41 +04:00
|
|
|
goto err_free;
|
|
|
|
|
|
|
|
TCP_SKB_CB(skb)->seq = tcp_sk(sk)->rcv_nxt;
|
|
|
|
TCP_SKB_CB(skb)->end_seq = TCP_SKB_CB(skb)->seq + size;
|
|
|
|
TCP_SKB_CB(skb)->ack_seq = tcp_sk(sk)->snd_una - 1;
|
|
|
|
|
2014-09-17 14:14:42 +04:00
|
|
|
if (tcp_queue_rcv(sk, skb, 0, &fragstolen)) {
|
2012-05-10 05:49:41 +04:00
|
|
|
WARN_ON_ONCE(fragstolen); /* should not happen */
|
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
|
|
|
return size;
|
|
|
|
|
|
|
|
err_free:
|
|
|
|
kfree_skb(skb);
|
|
|
|
err:
|
2015-11-19 08:03:33 +03:00
|
|
|
return err;
|
|
|
|
|
2012-05-10 05:49:41 +04:00
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
static void tcp_data_queue(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2012-04-27 04:38:33 +04:00
|
|
|
bool fragstolen = false;
|
2016-04-01 18:52:19 +03:00
|
|
|
int eaten = -1;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2016-04-01 18:52:19 +03:00
|
|
|
if (TCP_SKB_CB(skb)->seq == TCP_SKB_CB(skb)->end_seq) {
|
|
|
|
__kfree_skb(skb);
|
|
|
|
return;
|
|
|
|
}
|
2010-04-29 02:31:51 +04:00
|
|
|
skb_dst_drop(skb);
|
2014-09-24 18:17:02 +04:00
|
|
|
__skb_pull(skb, tcp_hdr(skb)->doff * 4);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
tcp_ecn_accept_cwr(tp, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2009-03-14 17:23:01 +03:00
|
|
|
tp->rx_opt.dsack = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Queue data for delivery to the user.
|
|
|
|
* Packets in sequence go to the receive queue.
|
|
|
|
* Out of sequence packets to the out_of_order_queue.
|
|
|
|
*/
|
|
|
|
if (TCP_SKB_CB(skb)->seq == tp->rcv_nxt) {
|
|
|
|
if (tcp_receive_window(tp) == 0)
|
|
|
|
goto out_of_window;
|
|
|
|
|
|
|
|
/* Ok. In sequence. In window. */
|
|
|
|
if (tp->ucopy.task == current &&
|
|
|
|
tp->copied_seq == tp->rcv_nxt && tp->ucopy.len &&
|
|
|
|
sock_owned_by_user(sk) && !tp->urg_data) {
|
|
|
|
int chunk = min_t(unsigned int, skb->len,
|
2008-01-01 01:57:14 +03:00
|
|
|
tp->ucopy.len);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
__set_current_state(TASK_RUNNING);
|
|
|
|
|
|
|
|
local_bh_enable();
|
2014-11-24 21:26:06 +03:00
|
|
|
if (!skb_copy_datagram_msg(skb, 0, tp->ucopy.msg, chunk)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->ucopy.len -= chunk;
|
|
|
|
tp->copied_seq += chunk;
|
TCP: fix a bug that triggers large number of TCP RST by mistake
This patch fixes a bug that causes TCP RST packets to be generated
on otherwise correctly behaved applications, e.g., no unread data
on close,..., etc. To trigger the bug, at least two conditions must
be met:
1. The FIN flag is set on the last data packet, i.e., it's not on a
separate, FIN only packet.
2. The size of the last data chunk on the receive side matches
exactly with the size of buffer posted by the receiver, and the
receiver closes the socket without any further read attempt.
This bug was first noticed on our netperf based testbed for our IW10
proposal to IETF where a large number of RST packets were observed.
netperf's read side code meets the condition 2 above 100%.
Before the fix, tcp_data_queue() will queue the last skb that meets
condition 1 to sk_receive_queue even though it has fully copied out
(skb_copy_datagram_iovec()) the data. Then if condition 2 is also met,
tcp_recvmsg() often returns all the copied out data successfully
without actually consuming the skb, due to a check
"if ((chunk = len - tp->ucopy.len) != 0) {"
and
"len -= chunk;"
after tcp_prequeue_process() that causes "len" to become 0 and an
early exit from the big while loop.
I don't see any reason not to free the skb whose data have been fully
consumed in tcp_data_queue(), regardless of the FIN flag. We won't
get there if MSG_PEEK is on. Am I missing some arcane cases related
to urgent data?
Signed-off-by: H.K. Jerry Chu <hkchu@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2011-01-26 00:46:30 +03:00
|
|
|
eaten = (chunk == skb->len);
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_rcv_space_adjust(sk);
|
|
|
|
}
|
|
|
|
local_bh_disable();
|
|
|
|
}
|
|
|
|
|
|
|
|
if (eaten <= 0) {
|
|
|
|
queue_and_out:
|
2015-05-15 22:39:29 +03:00
|
|
|
if (eaten < 0) {
|
|
|
|
if (skb_queue_len(&sk->sk_receive_queue) == 0)
|
|
|
|
sk_forced_mem_schedule(sk, skb->truesize);
|
|
|
|
else if (tcp_try_rmem_schedule(sk, skb, skb->truesize))
|
|
|
|
goto drop;
|
|
|
|
}
|
2012-05-02 13:58:29 +04:00
|
|
|
eaten = tcp_queue_rcv(sk, skb, 0, &fragstolen);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2015-04-29 01:28:18 +03:00
|
|
|
tcp_rcv_nxt_update(tp, TCP_SKB_CB(skb)->end_seq);
|
2007-03-09 07:45:19 +03:00
|
|
|
if (skb->len)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_event_data_recv(sk, skb);
|
2014-09-24 18:17:02 +04:00
|
|
|
if (TCP_SKB_CB(skb)->tcp_flags & TCPHDR_FIN)
|
2011-10-21 01:44:03 +04:00
|
|
|
tcp_fin(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2005-07-09 01:57:23 +04:00
|
|
|
if (!skb_queue_empty(&tp->out_of_order_queue)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_ofo_queue(sk);
|
|
|
|
|
|
|
|
/* RFC2581. 4.2. SHOULD send immediate ACK, when
|
|
|
|
* gap in queue is filled.
|
|
|
|
*/
|
2005-07-09 01:57:23 +04:00
|
|
|
if (skb_queue_empty(&tp->out_of_order_queue))
|
2005-08-10 07:10:42 +04:00
|
|
|
inet_csk(sk)->icsk_ack.pingpong = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
if (tp->rx_opt.num_sacks)
|
|
|
|
tcp_sack_remove(tp);
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_fast_path_check(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2012-05-02 11:55:58 +04:00
|
|
|
if (eaten > 0)
|
|
|
|
kfree_skb_partial(skb, fragstolen);
|
2012-09-17 16:51:39 +04:00
|
|
|
if (!sock_flag(sk, SOCK_DEAD))
|
2014-04-12 00:15:36 +04:00
|
|
|
sk->sk_data_ready(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!after(TCP_SKB_CB(skb)->end_seq, tp->rcv_nxt)) {
|
|
|
|
/* A retransmit, 2nd most common case. Force an immediate ack. */
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_DELAYEDACKLOST);
|
2008-07-17 07:29:51 +04:00
|
|
|
tcp_dsack_set(sk, TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
out_of_window:
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_enter_quickack_mode(sk);
|
|
|
|
inet_csk_schedule_ack(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
drop:
|
2016-04-01 18:52:19 +03:00
|
|
|
tcp_drop(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Out of window. F.e. zero window probe. */
|
|
|
|
if (!before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt + tcp_receive_window(tp)))
|
|
|
|
goto out_of_window;
|
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_enter_quickack_mode(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt)) {
|
|
|
|
/* Partial packet, seq < rcv_next < end_seq */
|
|
|
|
SOCK_DEBUG(sk, "partial packet: rcv_next %X seq %X - %X\n",
|
|
|
|
tp->rcv_nxt, TCP_SKB_CB(skb)->seq,
|
|
|
|
TCP_SKB_CB(skb)->end_seq);
|
|
|
|
|
2008-07-17 07:29:51 +04:00
|
|
|
tcp_dsack_set(sk, TCP_SKB_CB(skb)->seq, tp->rcv_nxt);
|
2007-02-09 17:24:47 +03:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* If window is closed, drop tail of packet. But after
|
|
|
|
* remembering D-SACK for its head made in previous line.
|
|
|
|
*/
|
|
|
|
if (!tcp_receive_window(tp))
|
|
|
|
goto out_of_window;
|
|
|
|
goto queue_and_out;
|
|
|
|
}
|
|
|
|
|
2012-03-18 15:06:44 +04:00
|
|
|
tcp_data_queue_ofo(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2008-08-23 16:11:41 +04:00
|
|
|
static struct sk_buff *tcp_collapse_one(struct sock *sk, struct sk_buff *skb,
|
|
|
|
struct sk_buff_head *list)
|
|
|
|
{
|
2009-05-29 08:35:47 +04:00
|
|
|
struct sk_buff *next = NULL;
|
|
|
|
|
|
|
|
if (!skb_queue_is_last(list, skb))
|
|
|
|
next = skb_queue_next(list, skb);
|
2008-08-23 16:11:41 +04:00
|
|
|
|
|
|
|
__skb_unlink(skb, list);
|
|
|
|
__kfree_skb(skb);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPRCVCOLLAPSED);
|
2008-08-23 16:11:41 +04:00
|
|
|
|
|
|
|
return next;
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Collapse contiguous sequence of skbs head..tail with
|
|
|
|
* sequence numbers start..end.
|
2009-05-29 08:35:47 +04:00
|
|
|
*
|
|
|
|
* If tail is NULL, this means until the end of the list.
|
|
|
|
*
|
2005-04-17 02:20:36 +04:00
|
|
|
* Segments with FIN/SYN are not collapsed (only because this
|
|
|
|
* simplifies code)
|
|
|
|
*/
|
|
|
|
static void
|
2005-08-10 06:25:21 +04:00
|
|
|
tcp_collapse(struct sock *sk, struct sk_buff_head *list,
|
|
|
|
struct sk_buff *head, struct sk_buff *tail,
|
|
|
|
u32 start, u32 end)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2009-05-29 08:35:47 +04:00
|
|
|
struct sk_buff *skb, *n;
|
|
|
|
bool end_of_skbs;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2005-11-11 04:13:47 +03:00
|
|
|
/* First, check that queue is collapsible and find
|
2005-04-17 02:20:36 +04:00
|
|
|
* the point where collapsing can be useful. */
|
2009-05-29 08:35:47 +04:00
|
|
|
skb = head;
|
|
|
|
restart:
|
|
|
|
end_of_skbs = true;
|
|
|
|
skb_queue_walk_from_safe(list, skb, n) {
|
|
|
|
if (skb == tail)
|
|
|
|
break;
|
2005-04-17 02:20:36 +04:00
|
|
|
/* No new bits? It is possible on ofo queue. */
|
|
|
|
if (!before(start, TCP_SKB_CB(skb)->end_seq)) {
|
2008-08-23 16:11:41 +04:00
|
|
|
skb = tcp_collapse_one(sk, skb, list);
|
2009-05-29 08:35:47 +04:00
|
|
|
if (!skb)
|
|
|
|
break;
|
|
|
|
goto restart;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
/* The first skb to collapse is:
|
|
|
|
* - not SYN/FIN and
|
|
|
|
* - bloated or contains data before "start" or
|
|
|
|
* overlaps to the next one.
|
|
|
|
*/
|
2014-09-15 15:19:51 +04:00
|
|
|
if (!(TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)) &&
|
2005-04-17 02:20:36 +04:00
|
|
|
(tcp_win_from_space(skb->truesize) > skb->len ||
|
2009-05-29 08:35:47 +04:00
|
|
|
before(TCP_SKB_CB(skb)->seq, start))) {
|
|
|
|
end_of_skbs = false;
|
2005-04-17 02:20:36 +04:00
|
|
|
break;
|
2009-05-29 08:35:47 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
if (!skb_queue_is_last(list, skb)) {
|
|
|
|
struct sk_buff *next = skb_queue_next(list, skb);
|
|
|
|
if (next != tail &&
|
|
|
|
TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(next)->seq) {
|
|
|
|
end_of_skbs = false;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Decided to skip this, advance start seq. */
|
|
|
|
start = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
}
|
2014-09-15 15:19:51 +04:00
|
|
|
if (end_of_skbs ||
|
|
|
|
(TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)))
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
|
|
|
|
|
|
|
while (before(start, end)) {
|
2014-09-15 15:19:53 +04:00
|
|
|
int copy = min_t(int, SKB_MAX_ORDER(0, 0), end - start);
|
2005-04-17 02:20:36 +04:00
|
|
|
struct sk_buff *nskb;
|
|
|
|
|
2014-09-15 15:19:53 +04:00
|
|
|
nskb = alloc_skb(copy, GFP_ATOMIC);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (!nskb)
|
|
|
|
return;
|
2007-03-10 18:47:22 +03:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
memcpy(nskb->cb, skb->cb, sizeof(skb->cb));
|
|
|
|
TCP_SKB_CB(nskb)->seq = TCP_SKB_CB(nskb)->end_seq = start;
|
2008-09-22 08:28:51 +04:00
|
|
|
__skb_queue_before(list, skb, nskb);
|
2007-12-31 11:11:19 +03:00
|
|
|
skb_set_owner_r(nskb, sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Copy data, releasing collapsed skbs. */
|
|
|
|
while (copy > 0) {
|
|
|
|
int offset = start - TCP_SKB_CB(skb)->seq;
|
|
|
|
int size = TCP_SKB_CB(skb)->end_seq - start;
|
|
|
|
|
2006-01-09 09:24:28 +03:00
|
|
|
BUG_ON(offset < 0);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (size > 0) {
|
|
|
|
size = min(copy, size);
|
|
|
|
if (skb_copy_bits(skb, offset, skb_put(nskb, size), size))
|
|
|
|
BUG();
|
|
|
|
TCP_SKB_CB(nskb)->end_seq += size;
|
|
|
|
copy -= size;
|
|
|
|
start += size;
|
|
|
|
}
|
|
|
|
if (!before(start, TCP_SKB_CB(skb)->end_seq)) {
|
2008-08-23 16:11:41 +04:00
|
|
|
skb = tcp_collapse_one(sk, skb, list);
|
2009-05-29 08:35:47 +04:00
|
|
|
if (!skb ||
|
|
|
|
skb == tail ||
|
2014-09-15 15:19:51 +04:00
|
|
|
(TCP_SKB_CB(skb)->tcp_flags & (TCPHDR_SYN | TCPHDR_FIN)))
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Collapse ofo queue. Algorithm: select contiguous sequence of skbs
|
|
|
|
* and tcp_collapse() them until all the queue is collapsed.
|
|
|
|
*/
|
|
|
|
static void tcp_collapse_ofo_queue(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct sk_buff *skb = skb_peek(&tp->out_of_order_queue);
|
|
|
|
struct sk_buff *head;
|
|
|
|
u32 start, end;
|
|
|
|
|
2015-04-03 11:17:26 +03:00
|
|
|
if (!skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
|
|
|
|
|
|
|
start = TCP_SKB_CB(skb)->seq;
|
|
|
|
end = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
head = skb;
|
|
|
|
|
|
|
|
for (;;) {
|
2009-05-29 08:35:47 +04:00
|
|
|
struct sk_buff *next = NULL;
|
|
|
|
|
|
|
|
if (!skb_queue_is_last(&tp->out_of_order_queue, skb))
|
|
|
|
next = skb_queue_next(&tp->out_of_order_queue, skb);
|
|
|
|
skb = next;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Segment is terminated when we see gap or when
|
|
|
|
* we are at the end of all the queue. */
|
2009-05-29 08:35:47 +04:00
|
|
|
if (!skb ||
|
2005-04-17 02:20:36 +04:00
|
|
|
after(TCP_SKB_CB(skb)->seq, end) ||
|
|
|
|
before(TCP_SKB_CB(skb)->end_seq, start)) {
|
2005-08-10 06:25:21 +04:00
|
|
|
tcp_collapse(sk, &tp->out_of_order_queue,
|
|
|
|
head, skb, start, end);
|
2005-04-17 02:20:36 +04:00
|
|
|
head = skb;
|
2009-05-29 08:35:47 +04:00
|
|
|
if (!skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
break;
|
|
|
|
/* Start new segment */
|
|
|
|
start = TCP_SKB_CB(skb)->seq;
|
|
|
|
end = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
} else {
|
|
|
|
if (before(TCP_SKB_CB(skb)->seq, start))
|
|
|
|
start = TCP_SKB_CB(skb)->seq;
|
|
|
|
if (after(TCP_SKB_CB(skb)->end_seq, end))
|
|
|
|
end = TCP_SKB_CB(skb)->end_seq;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-04-15 11:33:38 +04:00
|
|
|
/*
|
|
|
|
* Purge the out-of-order queue.
|
2008-04-16 07:26:34 +04:00
|
|
|
* Return true if queue was pruned.
|
2008-04-15 11:33:38 +04:00
|
|
|
*/
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_prune_ofo_queue(struct sock *sk)
|
2008-04-15 11:33:38 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2012-05-17 03:15:34 +04:00
|
|
|
bool res = false;
|
2008-04-15 11:33:38 +04:00
|
|
|
|
|
|
|
if (!skb_queue_empty(&tp->out_of_order_queue)) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_OFOPRUNED);
|
2008-04-15 11:33:38 +04:00
|
|
|
__skb_queue_purge(&tp->out_of_order_queue);
|
|
|
|
|
|
|
|
/* Reset SACK state. A conforming SACK implementation will
|
|
|
|
* do the same at a timeout based retransmit. When a connection
|
|
|
|
* is in a sad state like this, we care only about integrity
|
|
|
|
* of the connection not performance.
|
|
|
|
*/
|
|
|
|
if (tp->rx_opt.sack_ok)
|
|
|
|
tcp_sack_reset(&tp->rx_opt);
|
|
|
|
sk_mem_reclaim(sk);
|
2012-05-17 03:15:34 +04:00
|
|
|
res = true;
|
2008-04-15 11:33:38 +04:00
|
|
|
}
|
2008-04-16 07:26:34 +04:00
|
|
|
return res;
|
2008-04-15 11:33:38 +04:00
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Reduce allocated memory if we can, trying to get
|
|
|
|
* the socket within its memory limits again.
|
|
|
|
*
|
|
|
|
* Return less than zero if we should start dropping frames
|
|
|
|
* until the socket owning process reads some of the data
|
|
|
|
* to stabilize the situation.
|
|
|
|
*/
|
|
|
|
static int tcp_prune_queue(struct sock *sk)
|
|
|
|
{
|
2007-02-09 17:24:47 +03:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
SOCK_DEBUG(sk, "prune_queue: c=%x\n", tp->copied_seq);
|
|
|
|
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_PRUNECALLED);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) >= sk->sk_rcvbuf)
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_clamp_window(sk);
|
2015-05-15 22:39:27 +03:00
|
|
|
else if (tcp_under_memory_pressure(sk))
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->rcv_ssthresh = min(tp->rcv_ssthresh, 4U * tp->advmss);
|
|
|
|
|
|
|
|
tcp_collapse_ofo_queue(sk);
|
2009-05-29 08:35:47 +04:00
|
|
|
if (!skb_queue_empty(&sk->sk_receive_queue))
|
|
|
|
tcp_collapse(sk, &sk->sk_receive_queue,
|
|
|
|
skb_peek(&sk->sk_receive_queue),
|
|
|
|
NULL,
|
|
|
|
tp->copied_seq, tp->rcv_nxt);
|
2007-12-31 11:11:19 +03:00
|
|
|
sk_mem_reclaim(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* Collapsing did not help, destructive actions follow.
|
|
|
|
* This must not ever occur. */
|
|
|
|
|
2008-04-15 11:33:38 +04:00
|
|
|
tcp_prune_ofo_queue(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (atomic_read(&sk->sk_rmem_alloc) <= sk->sk_rcvbuf)
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
/* If we are really being abused, tell the caller to silently
|
|
|
|
* drop receive data on the floor. It will get retransmitted
|
|
|
|
* and hopefully then we'll have sufficient space.
|
|
|
|
*/
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_RCVPRUNED);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Massive buffer overcommit. */
|
|
|
|
tp->pred_flags = 0;
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2012-05-17 03:15:34 +04:00
|
|
|
static bool tcp_should_expand_sndbuf(const struct sock *sk)
|
2005-07-06 02:21:10 +04:00
|
|
|
{
|
2011-10-21 13:22:42 +04:00
|
|
|
const struct tcp_sock *tp = tcp_sk(sk);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
|
2005-07-06 02:21:10 +04:00
|
|
|
/* If the user specified a specific send buffer setting, do
|
|
|
|
* not modify it.
|
|
|
|
*/
|
|
|
|
if (sk->sk_userlocks & SOCK_SNDBUF_LOCK)
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2005-07-06 02:21:10 +04:00
|
|
|
|
|
|
|
/* If we are under global TCP memory pressure, do not expand. */
|
2015-05-15 22:39:27 +03:00
|
|
|
if (tcp_under_memory_pressure(sk))
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2005-07-06 02:21:10 +04:00
|
|
|
|
|
|
|
/* If we are under soft global TCP memory pressure, do not expand. */
|
2011-12-12 01:47:02 +04:00
|
|
|
if (sk_memory_allocated(sk) >= sk_prot_mem_limits(sk, 0))
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2005-07-06 02:21:10 +04:00
|
|
|
|
|
|
|
/* If we filled the congestion window, do not expand. */
|
2015-02-20 21:33:16 +03:00
|
|
|
if (tcp_packets_in_flight(tp) >= tp->snd_cwnd)
|
2012-05-17 03:15:34 +04:00
|
|
|
return false;
|
2005-07-06 02:21:10 +04:00
|
|
|
|
2012-05-17 03:15:34 +04:00
|
|
|
return true;
|
2005-07-06 02:21:10 +04:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* When incoming ACK allowed to free some skb from write_queue,
|
|
|
|
* we remember this event in flag SOCK_QUEUE_SHRUNK and wake up socket
|
|
|
|
* on the exit from tcp input handler.
|
|
|
|
*
|
|
|
|
* PROBLEM: sndbuf expansion does not work well with largesend.
|
|
|
|
*/
|
|
|
|
static void tcp_new_space(struct sock *sk)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
if (tcp_should_expand_sndbuf(sk)) {
|
2013-10-01 21:23:44 +04:00
|
|
|
tcp_sndbuf_expand(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->snd_cwnd_stamp = tcp_time_stamp;
|
|
|
|
}
|
|
|
|
|
|
|
|
sk->sk_write_space(sk);
|
|
|
|
}
|
|
|
|
|
2006-01-04 03:03:49 +03:00
|
|
|
static void tcp_check_space(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
if (sock_flag(sk, SOCK_QUEUE_SHRUNK)) {
|
|
|
|
sock_reset_flag(sk, SOCK_QUEUE_SHRUNK);
|
2015-04-20 23:05:07 +03:00
|
|
|
/* pairs with tcp_poll() */
|
|
|
|
smp_mb__after_atomic();
|
2005-04-17 02:20:36 +04:00
|
|
|
if (sk->sk_socket &&
|
|
|
|
test_bit(SOCK_NOSPACE, &sk->sk_socket->flags))
|
|
|
|
tcp_new_space(sk);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
static inline void tcp_data_snd_check(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_push_pending_frames(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_check_space(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Check if sending an ack is needed.
|
|
|
|
*/
|
|
|
|
static void __tcp_ack_snd_check(struct sock *sk, int ofo_possible)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
/* More than one full frame received... */
|
2009-11-23 21:41:23 +03:00
|
|
|
if (((tp->rcv_nxt - tp->rcv_wup) > inet_csk(sk)->icsk_ack.rcv_mss &&
|
2005-04-17 02:20:36 +04:00
|
|
|
/* ... and right edge of window advances far enough.
|
|
|
|
* (tcp_recvmsg() will send ACK otherwise). Or...
|
|
|
|
*/
|
2009-11-23 21:41:23 +03:00
|
|
|
__tcp_select_window(sk) >= tp->rcv_wnd) ||
|
2005-04-17 02:20:36 +04:00
|
|
|
/* We ACK each frame or... */
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_in_quickack_mode(sk) ||
|
2005-04-17 02:20:36 +04:00
|
|
|
/* We have out of order data. */
|
2008-01-01 01:57:14 +03:00
|
|
|
(ofo_possible && skb_peek(&tp->out_of_order_queue))) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Then ack it now */
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
} else {
|
|
|
|
/* Else, send delayed ack. */
|
|
|
|
tcp_send_delayed_ack(sk);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2006-01-04 03:03:49 +03:00
|
|
|
static inline void tcp_ack_snd_check(struct sock *sk)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-08-10 07:10:42 +04:00
|
|
|
if (!inet_csk_ack_scheduled(sk)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* We sent a data segment already. */
|
|
|
|
return;
|
|
|
|
}
|
|
|
|
__tcp_ack_snd_check(sk, 1);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This routine is only called when we have urgent data
|
2005-11-11 04:13:47 +03:00
|
|
|
* signaled. Its the 'slow' part of tcp_urg. It could be
|
2005-04-17 02:20:36 +04:00
|
|
|
* moved inline now as tcp_urg is only called from one
|
|
|
|
* place. We handle URGent data wrong. We have to - as
|
|
|
|
* BSD still doesn't use the correction from RFC961.
|
|
|
|
* For 1003.1g we should support a new option TCP_STDURG to permit
|
|
|
|
* either form (or just set the sysctl tcp_stdurg).
|
|
|
|
*/
|
2007-02-09 17:24:47 +03:00
|
|
|
|
2011-10-21 13:22:42 +04:00
|
|
|
static void tcp_check_urg(struct sock *sk, const struct tcphdr *th)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
u32 ptr = ntohs(th->urg_ptr);
|
|
|
|
|
|
|
|
if (ptr && !sysctl_tcp_stdurg)
|
|
|
|
ptr--;
|
|
|
|
ptr += ntohl(th->seq);
|
|
|
|
|
|
|
|
/* Ignore urgent data that we've already seen and read. */
|
|
|
|
if (after(tp->copied_seq, ptr))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Do not replay urg ptr.
|
|
|
|
*
|
|
|
|
* NOTE: interesting situation not covered by specs.
|
|
|
|
* Misbehaving sender may send urg ptr, pointing to segment,
|
|
|
|
* which we already have in ofo queue. We are not able to fetch
|
|
|
|
* such data and will stay in TCP_URG_NOTYET until will be eaten
|
|
|
|
* by recvmsg(). Seems, we are not obliged to handle such wicked
|
|
|
|
* situations. But it is worth to think about possibility of some
|
|
|
|
* DoSes using some hypothetical application level deadlock.
|
|
|
|
*/
|
|
|
|
if (before(ptr, tp->rcv_nxt))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Do we already have a newer (or duplicate) urgent pointer? */
|
|
|
|
if (tp->urg_data && !after(ptr, tp->urg_seq))
|
|
|
|
return;
|
|
|
|
|
|
|
|
/* Tell the world about our new urgent pointer. */
|
|
|
|
sk_send_sigurg(sk);
|
|
|
|
|
|
|
|
/* We may be adding urgent data when the last byte read was
|
|
|
|
* urgent. To do this requires some care. We cannot just ignore
|
|
|
|
* tp->copied_seq since we would read the last urgent byte again
|
|
|
|
* as data, nor can we alter copied_seq until this data arrives
|
2005-11-11 04:13:47 +03:00
|
|
|
* or we break the semantics of SIOCATMARK (and thus sockatmark())
|
2005-04-17 02:20:36 +04:00
|
|
|
*
|
|
|
|
* NOTE. Double Dutch. Rendering to plain English: author of comment
|
|
|
|
* above did something sort of send("A", MSG_OOB); send("B", MSG_OOB);
|
|
|
|
* and expect that both A and B disappear from stream. This is _wrong_.
|
|
|
|
* Though this happens in BSD with high probability, this is occasional.
|
|
|
|
* Any application relying on this is buggy. Note also, that fix "works"
|
|
|
|
* only in this artificial test. Insert some normal data between A and B and we will
|
|
|
|
* decline of BSD again. Verdict: it is better to remove to trap
|
|
|
|
* buggy users.
|
|
|
|
*/
|
|
|
|
if (tp->urg_seq == tp->copied_seq && tp->urg_data &&
|
2008-01-01 01:57:14 +03:00
|
|
|
!sock_flag(sk, SOCK_URGINLINE) && tp->copied_seq != tp->rcv_nxt) {
|
2005-04-17 02:20:36 +04:00
|
|
|
struct sk_buff *skb = skb_peek(&sk->sk_receive_queue);
|
|
|
|
tp->copied_seq++;
|
|
|
|
if (skb && !before(tp->copied_seq, TCP_SKB_CB(skb)->end_seq)) {
|
2005-08-10 06:25:21 +04:00
|
|
|
__skb_unlink(skb, &sk->sk_receive_queue);
|
2005-04-17 02:20:36 +04:00
|
|
|
__kfree_skb(skb);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
tp->urg_data = TCP_URG_NOTYET;
|
|
|
|
tp->urg_seq = ptr;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Disable header prediction. */
|
|
|
|
tp->pred_flags = 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* This is the 'fast' part of urgent handling. */
|
2011-10-21 13:22:42 +04:00
|
|
|
static void tcp_urg(struct sock *sk, struct sk_buff *skb, const struct tcphdr *th)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
/* Check if we get a new urgent pointer - normally not. */
|
|
|
|
if (th->urg)
|
2008-01-01 01:57:14 +03:00
|
|
|
tcp_check_urg(sk, th);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Do we wait for any urgent data? - normally not... */
|
|
|
|
if (tp->urg_data == TCP_URG_NOTYET) {
|
|
|
|
u32 ptr = tp->urg_seq - ntohl(th->seq) + (th->doff * 4) -
|
|
|
|
th->syn;
|
|
|
|
|
2007-02-09 17:24:47 +03:00
|
|
|
/* Is the urgent pointer pointing into this packet? */
|
2005-04-17 02:20:36 +04:00
|
|
|
if (ptr < skb->len) {
|
|
|
|
u8 tmp;
|
|
|
|
if (skb_copy_bits(skb, ptr, &tmp, 1))
|
|
|
|
BUG();
|
|
|
|
tp->urg_data = TCP_URG_VALID | tmp;
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD))
|
2014-04-12 00:15:36 +04:00
|
|
|
sk->sk_data_ready(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
static int tcp_copy_to_iovec(struct sock *sk, struct sk_buff *skb, int hlen)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
int chunk = skb->len - hlen;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
local_bh_enable();
|
2007-04-09 22:59:39 +04:00
|
|
|
if (skb_csum_unnecessary(skb))
|
2014-11-24 21:26:06 +03:00
|
|
|
err = skb_copy_datagram_msg(skb, hlen, tp->ucopy.msg, chunk);
|
2005-04-17 02:20:36 +04:00
|
|
|
else
|
2014-11-24 21:26:06 +03:00
|
|
|
err = skb_copy_and_csum_datagram_msg(skb, hlen, tp->ucopy.msg);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (!err) {
|
|
|
|
tp->ucopy.len -= chunk;
|
|
|
|
tp->copied_seq += chunk;
|
|
|
|
tcp_rcv_space_adjust(sk);
|
|
|
|
}
|
|
|
|
|
|
|
|
local_bh_disable();
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2008-01-01 01:57:14 +03:00
|
|
|
static __sum16 __tcp_checksum_complete_user(struct sock *sk,
|
|
|
|
struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2006-11-15 08:40:42 +03:00
|
|
|
__sum16 result;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (sock_owned_by_user(sk)) {
|
|
|
|
local_bh_enable();
|
|
|
|
result = __tcp_checksum_complete(skb);
|
|
|
|
local_bh_disable();
|
|
|
|
} else {
|
|
|
|
result = __tcp_checksum_complete(skb);
|
|
|
|
}
|
|
|
|
return result;
|
|
|
|
}
|
|
|
|
|
2012-07-20 01:32:18 +04:00
|
|
|
static inline bool tcp_checksum_complete_user(struct sock *sk,
|
2008-01-01 01:57:14 +03:00
|
|
|
struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2007-04-09 22:59:39 +04:00
|
|
|
return !skb_csum_unnecessary(skb) &&
|
2008-01-01 01:57:14 +03:00
|
|
|
__tcp_checksum_complete_user(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2008-08-23 16:10:12 +04:00
|
|
|
/* Does PAWS and seqno based validation of an incoming segment, flags will
|
|
|
|
* play significant role here.
|
|
|
|
*/
|
2012-07-17 05:41:30 +04:00
|
|
|
static bool tcp_validate_incoming(struct sock *sk, struct sk_buff *skb,
|
|
|
|
const struct tcphdr *th, int syn_inerr)
|
2008-08-23 16:10:12 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
|
|
|
/* RFC1323: H1. Apply PAWS check first. */
|
2013-03-17 12:23:34 +04:00
|
|
|
if (tcp_fast_parse_options(skb, th, tp) && tp->rx_opt.saw_tstamp &&
|
2008-08-23 16:10:12 +04:00
|
|
|
tcp_paws_discard(sk, skb)) {
|
|
|
|
if (!th->rst) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_PAWSESTABREJECTED);
|
2015-02-07 00:04:40 +03:00
|
|
|
if (!tcp_oow_rate_limited(sock_net(sk), skb,
|
|
|
|
LINUX_MIB_TCPACKSKIPPEDPAWS,
|
|
|
|
&tp->last_oow_ack_time))
|
|
|
|
tcp_send_dupack(sk, skb);
|
2008-08-23 16:10:12 +04:00
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
/* Reset is accepted even if it did not pass PAWS. */
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Step 1: check sequence number */
|
|
|
|
if (!tcp_sequence(tp, TCP_SKB_CB(skb)->seq, TCP_SKB_CB(skb)->end_seq)) {
|
|
|
|
/* RFC793, page 37: "In all states except SYN-SENT, all reset
|
|
|
|
* (RST) segments are validated by checking their SEQ-fields."
|
|
|
|
* And page 69: "If an incoming segment is not acceptable,
|
|
|
|
* an acknowledgment should be sent in reply (unless the RST
|
|
|
|
* bit is set, if so drop the segment and return)".
|
|
|
|
*/
|
2012-07-17 16:29:30 +04:00
|
|
|
if (!th->rst) {
|
|
|
|
if (th->syn)
|
|
|
|
goto syn_challenge;
|
2015-02-07 00:04:40 +03:00
|
|
|
if (!tcp_oow_rate_limited(sock_net(sk), skb,
|
|
|
|
LINUX_MIB_TCPACKSKIPPEDSEQ,
|
|
|
|
&tp->last_oow_ack_time))
|
|
|
|
tcp_send_dupack(sk, skb);
|
2012-07-17 16:29:30 +04:00
|
|
|
}
|
2008-08-23 16:10:12 +04:00
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Step 2: check RST bit */
|
|
|
|
if (th->rst) {
|
2012-07-17 12:13:05 +04:00
|
|
|
/* RFC 5961 3.2 :
|
|
|
|
* If sequence number exactly matches RCV.NXT, then
|
|
|
|
* RESET the connection
|
|
|
|
* else
|
|
|
|
* Send a challenge ACK
|
|
|
|
*/
|
|
|
|
if (TCP_SKB_CB(skb)->seq == tp->rcv_nxt)
|
|
|
|
tcp_reset(sk);
|
|
|
|
else
|
2015-02-07 00:04:40 +03:00
|
|
|
tcp_send_challenge_ack(sk, skb);
|
2008-08-23 16:10:12 +04:00
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* step 3: check security and precedence [ignored] */
|
|
|
|
|
2012-07-17 05:41:30 +04:00
|
|
|
/* step 4: Check for a SYN
|
2014-10-30 19:48:08 +03:00
|
|
|
* RFC 5961 4.2 : Send a challenge ack
|
2012-07-17 05:41:30 +04:00
|
|
|
*/
|
|
|
|
if (th->syn) {
|
2012-07-17 16:29:30 +04:00
|
|
|
syn_challenge:
|
2008-08-23 16:10:12 +04:00
|
|
|
if (syn_inerr)
|
2016-04-28 02:44:32 +03:00
|
|
|
__TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPSYNCHALLENGE);
|
2015-02-07 00:04:40 +03:00
|
|
|
tcp_send_challenge_ack(sk, skb);
|
2012-07-17 05:41:30 +04:00
|
|
|
goto discard;
|
2008-08-23 16:10:12 +04:00
|
|
|
}
|
|
|
|
|
2012-07-17 05:41:30 +04:00
|
|
|
return true;
|
2008-08-23 16:10:12 +04:00
|
|
|
|
|
|
|
discard:
|
2016-04-01 18:52:19 +03:00
|
|
|
tcp_drop(sk, skb);
|
2012-07-17 05:41:30 +04:00
|
|
|
return false;
|
2008-08-23 16:10:12 +04:00
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/*
|
2007-02-09 17:24:47 +03:00
|
|
|
* TCP receive function for the ESTABLISHED state.
|
2005-04-17 02:20:36 +04:00
|
|
|
*
|
2007-02-09 17:24:47 +03:00
|
|
|
* It is split into a fast path and a slow path. The fast path is
|
2005-04-17 02:20:36 +04:00
|
|
|
* disabled when:
|
|
|
|
* - A zero window was announced from us - zero window probing
|
2007-02-09 17:24:47 +03:00
|
|
|
* is only handled properly in the slow path.
|
2005-04-17 02:20:36 +04:00
|
|
|
* - Out of order segments arrived.
|
|
|
|
* - Urgent data is expected.
|
|
|
|
* - There is no buffer space left
|
|
|
|
* - Unexpected TCP flags/window values/header lengths are received
|
2007-02-09 17:24:47 +03:00
|
|
|
* (detected by checking the TCP header against pred_flags)
|
2005-04-17 02:20:36 +04:00
|
|
|
* - Data is sent in both directions. Fast path only supports pure senders
|
|
|
|
* or pure receivers (this means either the sequence number or the ack
|
|
|
|
* value must stay constant)
|
|
|
|
* - Unexpected TCP option.
|
|
|
|
*
|
2007-02-09 17:24:47 +03:00
|
|
|
* When these conditions are not satisfied it drops into a standard
|
2005-04-17 02:20:36 +04:00
|
|
|
* receive procedure patterned after RFC793 to handle all cases.
|
|
|
|
* The first three cases are guaranteed by proper pred_flags setting,
|
2007-02-09 17:24:47 +03:00
|
|
|
* the rest is checked inline. Fast processing is turned on in
|
2005-04-17 02:20:36 +04:00
|
|
|
* tcp_data_queue when everything is OK.
|
|
|
|
*/
|
2013-09-03 23:23:22 +04:00
|
|
|
void tcp_rcv_established(struct sock *sk, struct sk_buff *skb,
|
|
|
|
const struct tcphdr *th, unsigned int len)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
|
2015-04-03 11:17:26 +03:00
|
|
|
if (unlikely(!sk->sk_rx_dst))
|
2012-08-06 09:09:33 +04:00
|
|
|
inet_csk(sk)->icsk_af_ops->sk_rx_dst_set(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
/*
|
|
|
|
* Header prediction.
|
2007-02-09 17:24:47 +03:00
|
|
|
* The code loosely follows the one in the famous
|
2005-04-17 02:20:36 +04:00
|
|
|
* "30 instruction TCP receive" Van Jacobson mail.
|
2007-02-09 17:24:47 +03:00
|
|
|
*
|
|
|
|
* Van's trick is to deposit buffers into socket queue
|
2005-04-17 02:20:36 +04:00
|
|
|
* on a device interrupt, to call tcp_recv function
|
|
|
|
* on the receive process context and checksum and copy
|
|
|
|
* the buffer to user space. smart...
|
|
|
|
*
|
2007-02-09 17:24:47 +03:00
|
|
|
* Our current scheme is not silly either but we take the
|
2005-04-17 02:20:36 +04:00
|
|
|
* extra cost of the net_bh soft interrupt processing...
|
|
|
|
* We do checksum and copy also but from device to kernel.
|
|
|
|
*/
|
|
|
|
|
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
|
|
|
|
|
|
|
/* pred_flags is 0xS?10 << 16 + snd_wnd
|
2005-11-11 04:13:47 +03:00
|
|
|
* if header_prediction is to be made
|
2005-04-17 02:20:36 +04:00
|
|
|
* 'S' will always be tp->tcp_header_len >> 2
|
|
|
|
* '?' will be 0 for the fast path, otherwise pred_flags is 0 to
|
2007-02-09 17:24:47 +03:00
|
|
|
* turn it off (when there are holes in the receive
|
2005-04-17 02:20:36 +04:00
|
|
|
* space for instance)
|
|
|
|
* PSH flag is ignored.
|
|
|
|
*/
|
|
|
|
|
|
|
|
if ((tcp_flag_word(th) & TCP_HP_BITS) == tp->pred_flags &&
|
2009-03-23 07:49:57 +03:00
|
|
|
TCP_SKB_CB(skb)->seq == tp->rcv_nxt &&
|
|
|
|
!after(TCP_SKB_CB(skb)->ack_seq, tp->snd_nxt)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
int tcp_header_len = tp->tcp_header_len;
|
|
|
|
|
|
|
|
/* Timestamp header prediction: tcp_header_len
|
|
|
|
* is automatically equal to th->doff*4 due to pred_flags
|
|
|
|
* match.
|
|
|
|
*/
|
|
|
|
|
|
|
|
/* Check timestamp */
|
|
|
|
if (tcp_header_len == sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED) {
|
|
|
|
/* No? Slow path! */
|
2008-08-23 16:12:29 +04:00
|
|
|
if (!tcp_parse_aligned_timestamp(tp, th))
|
2005-04-17 02:20:36 +04:00
|
|
|
goto slow_path;
|
|
|
|
|
|
|
|
/* If PAWS failed, check it more carefully in slow path */
|
|
|
|
if ((s32)(tp->rx_opt.rcv_tsval - tp->rx_opt.ts_recent) < 0)
|
|
|
|
goto slow_path;
|
|
|
|
|
|
|
|
/* DO NOT update ts_recent here, if checksum fails
|
|
|
|
* and timestamp was corrupted part, it will result
|
|
|
|
* in a hung connection since we will drop all
|
|
|
|
* future packets due to the PAWS test.
|
|
|
|
*/
|
|
|
|
}
|
|
|
|
|
|
|
|
if (len <= tcp_header_len) {
|
|
|
|
/* Bulk data transfer: sender */
|
|
|
|
if (len == tcp_header_len) {
|
|
|
|
/* Predicted packet is in window by definition.
|
|
|
|
* seq == rcv_nxt and rcv_wup <= rcv_nxt.
|
|
|
|
* Hence, check seq<=rcv_wup reduces to:
|
|
|
|
*/
|
|
|
|
if (tcp_header_len ==
|
|
|
|
(sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED) &&
|
|
|
|
tp->rcv_nxt == tp->rcv_wup)
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
|
|
|
|
/* We know that such packets are checksummed
|
|
|
|
* on entry.
|
|
|
|
*/
|
|
|
|
tcp_ack(sk, skb, 0);
|
2007-02-09 17:24:47 +03:00
|
|
|
__kfree_skb(skb);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_data_snd_check(sk);
|
2013-09-03 23:23:22 +04:00
|
|
|
return;
|
2005-04-17 02:20:36 +04:00
|
|
|
} else { /* Header too small */
|
2016-04-28 02:44:32 +03:00
|
|
|
__TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
|
2005-04-17 02:20:36 +04:00
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
} else {
|
|
|
|
int eaten = 0;
|
2012-05-02 13:58:29 +04:00
|
|
|
bool fragstolen = false;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-12-30 23:37:15 +04:00
|
|
|
if (tp->ucopy.task == current &&
|
|
|
|
tp->copied_seq == tp->rcv_nxt &&
|
|
|
|
len - tcp_header_len <= tp->ucopy.len &&
|
|
|
|
sock_owned_by_user(sk)) {
|
|
|
|
__set_current_state(TASK_RUNNING);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-12-30 23:37:15 +04:00
|
|
|
if (!tcp_copy_to_iovec(sk, skb, tcp_header_len)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Predicted packet is in window by definition.
|
|
|
|
* seq == rcv_nxt and rcv_wup <= rcv_nxt.
|
|
|
|
* Hence, check seq<=rcv_wup reduces to:
|
|
|
|
*/
|
|
|
|
if (tcp_header_len ==
|
|
|
|
(sizeof(struct tcphdr) +
|
|
|
|
TCPOLEN_TSTAMP_ALIGNED) &&
|
|
|
|
tp->rcv_nxt == tp->rcv_wup)
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_rcv_rtt_measure_ts(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
__skb_pull(skb, tcp_header_len);
|
2015-04-29 01:28:18 +03:00
|
|
|
tcp_rcv_nxt_update(tp, TCP_SKB_CB(skb)->end_seq);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk),
|
|
|
|
LINUX_MIB_TCPHPHITSTOUSER);
|
2013-12-30 23:37:15 +04:00
|
|
|
eaten = 1;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
if (!eaten) {
|
|
|
|
if (tcp_checksum_complete_user(sk, skb))
|
|
|
|
goto csum_error;
|
|
|
|
|
2013-03-04 10:23:05 +04:00
|
|
|
if ((int)skb->truesize > sk->sk_forward_alloc)
|
|
|
|
goto step5;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Predicted packet is in window by definition.
|
|
|
|
* seq == rcv_nxt and rcv_wup <= rcv_nxt.
|
|
|
|
* Hence, check seq<=rcv_wup reduces to:
|
|
|
|
*/
|
|
|
|
if (tcp_header_len ==
|
|
|
|
(sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED) &&
|
|
|
|
tp->rcv_nxt == tp->rcv_wup)
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_rcv_rtt_measure_ts(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPHPHITS);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Bulk data transfer: receiver */
|
2012-05-02 13:58:29 +04:00
|
|
|
eaten = tcp_queue_rcv(sk, skb, tcp_header_len,
|
|
|
|
&fragstolen);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_event_data_recv(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (TCP_SKB_CB(skb)->ack_seq != tp->snd_una) {
|
|
|
|
/* Well, only one small jumplet in fast path... */
|
|
|
|
tcp_ack(sk, skb, FLAG_DATA);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_data_snd_check(sk);
|
2005-08-10 07:10:42 +04:00
|
|
|
if (!inet_csk_ack_scheduled(sk))
|
2005-04-17 02:20:36 +04:00
|
|
|
goto no_ack;
|
|
|
|
}
|
|
|
|
|
2013-12-30 23:37:15 +04:00
|
|
|
__tcp_ack_snd_check(sk, 0);
|
2005-04-17 02:20:36 +04:00
|
|
|
no_ack:
|
|
|
|
if (eaten)
|
2012-05-02 13:58:29 +04:00
|
|
|
kfree_skb_partial(skb, fragstolen);
|
2014-04-12 00:15:36 +04:00
|
|
|
sk->sk_data_ready(sk);
|
2013-09-03 23:23:22 +04:00
|
|
|
return;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
slow_path:
|
2008-01-01 01:57:14 +03:00
|
|
|
if (len < (th->doff << 2) || tcp_checksum_complete_user(sk, skb))
|
2005-04-17 02:20:36 +04:00
|
|
|
goto csum_error;
|
|
|
|
|
tcp: Restore RFC5961-compliant behavior for SYN packets
Commit c3ae62af8e755 ("tcp: should drop incoming frames without ACK
flag set") was created to mitigate a security vulnerability in which a
local attacker is able to inject data into locally-opened sockets by
using TCP protocol statistics in procfs to quickly find the correct
sequence number.
This broke the RFC5961 requirement to send a challenge ACK in response
to spurious RST packets, which was subsequently fixed by commit
7b514a886ba50 ("tcp: accept RST without ACK flag").
Unfortunately, the RFC5961 requirement that spurious SYN packets be
handled in a similar manner remains broken.
RFC5961 section 4 states that:
... the handling of the SYN in the synchronized state SHOULD be
performed as follows:
1) If the SYN bit is set, irrespective of the sequence number, TCP
MUST send an ACK (also referred to as challenge ACK) to the remote
peer:
<SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>
After sending the acknowledgment, TCP MUST drop the unacceptable
segment and stop processing further.
By sending an ACK, the remote peer is challenged to confirm the loss
of the previous connection and the request to start a new connection.
A legitimate peer, after restart, would not have a TCB in the
synchronized state. Thus, when the ACK arrives, the peer should send
a RST segment back with the sequence number derived from the ACK
field that caused the RST.
This RST will confirm that the remote peer has indeed closed the
previous connection. Upon receipt of a valid RST, the local TCP
endpoint MUST terminate its connection. The local TCP endpoint
should then rely on SYN retransmission from the remote end to
re-establish the connection.
This patch lets SYN packets through the discard added in c3ae62af8e755,
so that spurious SYN packets are properly dealt with as per the RFC.
The challenge ACK is sent unconditionally and is rate-limited, so the
original vulnerability is not reintroduced by this patch.
Signed-off-by: Calvin Owens <calvinowens@fb.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-21 02:09:53 +03:00
|
|
|
if (!th->ack && !th->rst && !th->syn)
|
2012-12-26 16:44:34 +04:00
|
|
|
goto discard;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
/*
|
|
|
|
* Standard slow path.
|
|
|
|
*/
|
|
|
|
|
2012-07-17 05:41:30 +04:00
|
|
|
if (!tcp_validate_incoming(sk, skb, th, 1))
|
2013-09-03 23:23:22 +04:00
|
|
|
return;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
step5:
|
tcp: call tcp_replace_ts_recent() from tcp_ack()
commit bd090dfc634d (tcp: tcp_replace_ts_recent() should not be called
from tcp_validate_incoming()) introduced a TS ecr bug in slow path
processing.
1 A > B P. 1:10001(10000) ack 1 <nop,nop,TS val 1001 ecr 200>
2 B < A . 1:1(0) ack 1 win 257 <sack 9001:10001,TS val 300 ecr 1001>
3 A > B . 1:1001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
4 A > B . 1001:2001(1000) ack 1 win 227 <nop,nop,TS val 1002 ecr 200>
(ecr 200 should be ecr 300 in packets 3 & 4)
Problem is tcp_ack() can trigger send of new packets (retransmits),
reflecting the prior TSval, instead of the TSval contained in the
currently processed incoming packet.
Fix this by calling tcp_replace_ts_recent() from tcp_ack() after the
checks, but before the actions.
Reported-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: Eric Dumazet <edumazet@google.com>
Cc: Neal Cardwell <ncardwell@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-04-19 11:19:48 +04:00
|
|
|
if (tcp_ack(sk, skb, FLAG_SLOWPATH | FLAG_UPDATE_TS_RECENT) < 0)
|
2009-03-23 07:49:57 +03:00
|
|
|
goto discard;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_rcv_rtt_measure_ts(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* Process urgent data. */
|
|
|
|
tcp_urg(sk, skb, th);
|
|
|
|
|
|
|
|
/* step 7: process the segment text */
|
|
|
|
tcp_data_queue(sk, skb);
|
|
|
|
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_data_snd_check(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_ack_snd_check(sk);
|
2013-09-03 23:23:22 +04:00
|
|
|
return;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
csum_error:
|
2016-04-28 02:44:32 +03:00
|
|
|
__TCP_INC_STATS(sock_net(sk), TCP_MIB_CSUMERRORS);
|
|
|
|
__TCP_INC_STATS(sock_net(sk), TCP_MIB_INERRS);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
discard:
|
2016-04-01 18:52:19 +03:00
|
|
|
tcp_drop(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2010-07-10 01:22:10 +04:00
|
|
|
EXPORT_SYMBOL(tcp_rcv_established);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2012-04-19 07:40:01 +04:00
|
|
|
void tcp_finish_connect(struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
|
|
|
|
|
|
|
tcp_set_state(sk, TCP_ESTABLISHED);
|
|
|
|
|
2015-04-03 11:17:27 +03:00
|
|
|
if (skb) {
|
2012-08-06 09:09:33 +04:00
|
|
|
icsk->icsk_af_ops->sk_rx_dst_set(sk, skb);
|
2012-04-19 07:40:01 +04:00
|
|
|
security_inet_conn_established(sk, skb);
|
2012-06-20 08:22:05 +04:00
|
|
|
}
|
2012-04-19 07:40:01 +04:00
|
|
|
|
|
|
|
/* Make sure socket is routed, for correct metrics. */
|
|
|
|
icsk->icsk_af_ops->rebuild_header(sk);
|
|
|
|
|
|
|
|
tcp_init_metrics(sk);
|
|
|
|
|
|
|
|
tcp_init_congestion_control(sk);
|
|
|
|
|
|
|
|
/* Prevent spurious tcp_cwnd_restart() on first data
|
|
|
|
* packet.
|
|
|
|
*/
|
|
|
|
tp->lsndtime = tcp_time_stamp;
|
|
|
|
|
|
|
|
tcp_init_buffer_space(sk);
|
|
|
|
|
|
|
|
if (sock_flag(sk, SOCK_KEEPOPEN))
|
|
|
|
inet_csk_reset_keepalive_timer(sk, keepalive_time_when(tp));
|
|
|
|
|
|
|
|
if (!tp->rx_opt.snd_wscale)
|
|
|
|
__tcp_fast_path_on(tp, tp->snd_wnd);
|
|
|
|
else
|
|
|
|
tp->pred_flags = 0;
|
|
|
|
|
|
|
|
if (!sock_flag(sk, SOCK_DEAD)) {
|
|
|
|
sk->sk_state_change(sk);
|
|
|
|
sk_wake_async(sk, SOCK_WAKE_IO, POLL_OUT);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2012-07-19 10:43:08 +04:00
|
|
|
static bool tcp_rcv_fastopen_synack(struct sock *sk, struct sk_buff *synack,
|
|
|
|
struct tcp_fastopen_cookie *cookie)
|
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2012-07-19 10:43:11 +04:00
|
|
|
struct sk_buff *data = tp->syn_data ? tcp_write_queue_head(sk) : NULL;
|
2015-04-07 00:37:27 +03:00
|
|
|
u16 mss = tp->rx_opt.mss_clamp, try_exp = 0;
|
|
|
|
bool syn_drop = false;
|
2012-07-19 10:43:08 +04:00
|
|
|
|
|
|
|
if (mss == tp->rx_opt.user_mss) {
|
|
|
|
struct tcp_options_received opt;
|
|
|
|
|
|
|
|
/* Get original SYNACK MSS value if user MSS sets mss_clamp */
|
|
|
|
tcp_clear_options(&opt);
|
|
|
|
opt.user_mss = opt.mss_clamp = 0;
|
2013-03-17 12:23:34 +04:00
|
|
|
tcp_parse_options(synack, &opt, 0, NULL);
|
2012-07-19 10:43:08 +04:00
|
|
|
mss = opt.mss_clamp;
|
|
|
|
}
|
|
|
|
|
2015-04-07 00:37:27 +03:00
|
|
|
if (!tp->syn_fastopen) {
|
|
|
|
/* Ignore an unsolicited cookie */
|
2012-07-19 10:43:11 +04:00
|
|
|
cookie->len = -1;
|
2015-04-07 00:37:27 +03:00
|
|
|
} else if (tp->total_retrans) {
|
|
|
|
/* SYN timed out and the SYN-ACK neither has a cookie nor
|
|
|
|
* acknowledges data. Presumably the remote received only
|
|
|
|
* the retransmitted (regular) SYNs: either the original
|
|
|
|
* SYN-data or the corresponding SYN-ACK was dropped.
|
|
|
|
*/
|
|
|
|
syn_drop = (cookie->len < 0 && data);
|
|
|
|
} else if (cookie->len < 0 && !tp->syn_data) {
|
|
|
|
/* We requested a cookie but didn't get it. If we did not use
|
|
|
|
* the (old) exp opt format then try so next time (try_exp=1).
|
|
|
|
* Otherwise we go back to use the RFC7413 opt (try_exp=2).
|
|
|
|
*/
|
|
|
|
try_exp = tp->syn_fastopen_exp ? 2 : 1;
|
|
|
|
}
|
2012-07-19 10:43:11 +04:00
|
|
|
|
2015-04-07 00:37:27 +03:00
|
|
|
tcp_fastopen_cache_set(sk, mss, cookie, syn_drop, try_exp);
|
2012-07-19 10:43:08 +04:00
|
|
|
|
|
|
|
if (data) { /* Retransmit unacked data in SYN */
|
2012-12-06 12:45:32 +04:00
|
|
|
tcp_for_write_queue_from(data, sk) {
|
|
|
|
if (data == tcp_send_head(sk) ||
|
tcp-tso: do not split TSO packets at retransmit time
Linux TCP stack painfully segments all TSO/GSO packets before retransmits.
This was fine back in the days when TSO/GSO were emerging, with their
bugs, but we believe the dark age is over.
Keeping big packets in write queues, but also in stack traversal
has a lot of benefits.
- Less memory overhead, because write queues have less skbs
- Less cpu overhead at ACK processing.
- Better SACK processing, as lot of studies mentioned how
awful linux was at this ;)
- Less cpu overhead to send the rtx packets
(IP stack traversal, netfilter traversal, drivers...)
- Better latencies in presence of losses.
- Smaller spikes in fq like packet schedulers, as retransmits
are not constrained by TCP Small Queues.
1 % packet losses are common today, and at 100Gbit speeds, this
translates to ~80,000 losses per second.
Losses are often correlated, and we see many retransmit events
leading to 1-MSS train of packets, at the time hosts are already
under stress.
Signed-off-by: Eric Dumazet <edumazet@google.com>
Acked-by: Yuchung Cheng <ycheng@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2016-04-21 20:55:23 +03:00
|
|
|
__tcp_retransmit_skb(sk, data, 1))
|
2012-12-06 12:45:32 +04:00
|
|
|
break;
|
|
|
|
}
|
2012-07-19 10:43:08 +04:00
|
|
|
tcp_rearm_rto(sk);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk),
|
|
|
|
LINUX_MIB_TCPFASTOPENACTIVEFAIL);
|
2012-07-19 10:43:08 +04:00
|
|
|
return true;
|
|
|
|
}
|
2012-10-19 19:14:44 +04:00
|
|
|
tp->syn_data_acked = tp->syn_data;
|
2014-03-04 00:31:36 +04:00
|
|
|
if (tp->syn_data_acked)
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk),
|
|
|
|
LINUX_MIB_TCPFASTOPENACTIVE);
|
2016-02-02 08:03:07 +03:00
|
|
|
|
|
|
|
tcp_fastopen_add_skb(sk, synack);
|
|
|
|
|
2012-07-19 10:43:08 +04:00
|
|
|
return false;
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
static int tcp_rcv_synsent_state_process(struct sock *sk, struct sk_buff *skb,
|
2015-09-29 17:42:40 +03:00
|
|
|
const struct tcphdr *th)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2005-12-14 10:26:10 +03:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2009-12-02 21:25:27 +03:00
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2012-07-19 10:43:08 +04:00
|
|
|
struct tcp_fastopen_cookie foc = { .len = -1 };
|
2009-12-02 21:25:27 +03:00
|
|
|
int saved_clamp = tp->rx_opt.mss_clamp;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-03-17 12:23:34 +04:00
|
|
|
tcp_parse_options(skb, &tp->rx_opt, 0, &foc);
|
2013-08-27 12:21:55 +04:00
|
|
|
if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr)
|
2013-02-11 09:50:19 +04:00
|
|
|
tp->rx_opt.rcv_tsecr -= tp->tsoffset;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (th->ack) {
|
|
|
|
/* rfc793:
|
|
|
|
* "If the state is SYN-SENT then
|
|
|
|
* first check the ACK bit
|
|
|
|
* If the ACK bit is set
|
|
|
|
* If SEG.ACK =< ISS, or SEG.ACK > SND.NXT, send
|
|
|
|
* a reset (unless the RST bit is set, if so drop
|
|
|
|
* the segment and return)"
|
|
|
|
*/
|
2012-07-19 10:43:08 +04:00
|
|
|
if (!after(TCP_SKB_CB(skb)->ack_seq, tp->snd_una) ||
|
|
|
|
after(TCP_SKB_CB(skb)->ack_seq, tp->snd_nxt))
|
2005-04-17 02:20:36 +04:00
|
|
|
goto reset_and_undo;
|
|
|
|
|
|
|
|
if (tp->rx_opt.saw_tstamp && tp->rx_opt.rcv_tsecr &&
|
|
|
|
!between(tp->rx_opt.rcv_tsecr, tp->retrans_stamp,
|
|
|
|
tcp_time_stamp)) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk),
|
|
|
|
LINUX_MIB_PAWSACTIVEREJECTED);
|
2005-04-17 02:20:36 +04:00
|
|
|
goto reset_and_undo;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Now ACK is acceptable.
|
|
|
|
*
|
|
|
|
* "If the RST bit is set
|
|
|
|
* If the ACK was acceptable then signal the user "error:
|
|
|
|
* connection reset", drop the segment, enter CLOSED state,
|
|
|
|
* delete TCB, and return."
|
|
|
|
*/
|
|
|
|
|
|
|
|
if (th->rst) {
|
|
|
|
tcp_reset(sk);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* rfc793:
|
|
|
|
* "fifth, if neither of the SYN or RST bits is set then
|
|
|
|
* drop the segment and return."
|
|
|
|
*
|
|
|
|
* See note below!
|
|
|
|
* --ANK(990513)
|
|
|
|
*/
|
|
|
|
if (!th->syn)
|
|
|
|
goto discard_and_undo;
|
|
|
|
|
|
|
|
/* rfc793:
|
|
|
|
* "If the SYN bit is on ...
|
|
|
|
* are acceptable then ...
|
|
|
|
* (our SYN has been ACKed), change the connection
|
|
|
|
* state to ESTABLISHED..."
|
|
|
|
*/
|
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
tcp_ecn_rcv_synack(tp, th);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2012-08-14 17:30:20 +04:00
|
|
|
tcp_init_wl(tp, TCP_SKB_CB(skb)->seq);
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_ack(sk, skb, FLAG_SLOWPATH);
|
|
|
|
|
|
|
|
/* Ok.. it's good. Set up sequence numbers and
|
|
|
|
* move to established.
|
|
|
|
*/
|
|
|
|
tp->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
|
|
|
|
tp->rcv_wup = TCP_SKB_CB(skb)->seq + 1;
|
|
|
|
|
|
|
|
/* RFC1323: The window in SYN & SYN/ACK segments is
|
|
|
|
* never scaled.
|
|
|
|
*/
|
|
|
|
tp->snd_wnd = ntohs(th->window);
|
|
|
|
|
|
|
|
if (!tp->rx_opt.wscale_ok) {
|
|
|
|
tp->rx_opt.snd_wscale = tp->rx_opt.rcv_wscale = 0;
|
|
|
|
tp->window_clamp = min(tp->window_clamp, 65535U);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (tp->rx_opt.saw_tstamp) {
|
|
|
|
tp->rx_opt.tstamp_ok = 1;
|
|
|
|
tp->tcp_header_len =
|
|
|
|
sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
tp->advmss -= TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
} else {
|
|
|
|
tp->tcp_header_len = sizeof(struct tcphdr);
|
|
|
|
}
|
|
|
|
|
2007-08-09 16:14:46 +04:00
|
|
|
if (tcp_is_sack(tp) && sysctl_tcp_fack)
|
|
|
|
tcp_enable_fack(tp);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2006-03-21 04:53:41 +03:00
|
|
|
tcp_mtup_init(sk);
|
2005-12-14 10:26:10 +03:00
|
|
|
tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_initialize_rcv_mss(sk);
|
|
|
|
|
|
|
|
/* Remember, tcp_poll() does not lock socket!
|
|
|
|
* Change state from SYN-SENT only after copied_seq
|
|
|
|
* is initialized. */
|
|
|
|
tp->copied_seq = tp->rcv_nxt;
|
2009-12-02 21:25:27 +03:00
|
|
|
|
2006-12-07 11:11:33 +03:00
|
|
|
smp_mb();
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2012-04-19 07:40:01 +04:00
|
|
|
tcp_finish_connect(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2012-07-19 10:43:11 +04:00
|
|
|
if ((tp->syn_fastopen || tp->syn_data) &&
|
|
|
|
tcp_rcv_fastopen_synack(sk, skb, &foc))
|
2012-07-19 10:43:08 +04:00
|
|
|
return -1;
|
|
|
|
|
2005-08-10 07:11:56 +04:00
|
|
|
if (sk->sk_write_pending ||
|
|
|
|
icsk->icsk_accept_queue.rskq_defer_accept ||
|
|
|
|
icsk->icsk_ack.pingpong) {
|
2005-04-17 02:20:36 +04:00
|
|
|
/* Save one ACK. Data will be ready after
|
|
|
|
* several ticks, if write_pending is set.
|
|
|
|
*
|
|
|
|
* It may be deleted, but with this feature tcpdumps
|
|
|
|
* look so _wonderfully_ clever, that I was not able
|
|
|
|
* to stand against the temptation 8) --ANK
|
|
|
|
*/
|
2005-08-10 07:10:42 +04:00
|
|
|
inet_csk_schedule_ack(sk);
|
2005-08-10 07:11:56 +04:00
|
|
|
icsk->icsk_ack.lrcvtime = tcp_time_stamp;
|
2005-08-10 07:10:42 +04:00
|
|
|
tcp_enter_quickack_mode(sk);
|
2005-08-10 07:11:08 +04:00
|
|
|
inet_csk_reset_xmit_timer(sk, ICSK_TIME_DACK,
|
|
|
|
TCP_DELACK_MAX, TCP_RTO_MAX);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
discard:
|
2016-04-01 18:52:19 +03:00
|
|
|
tcp_drop(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
return 0;
|
|
|
|
} else {
|
|
|
|
tcp_send_ack(sk);
|
|
|
|
}
|
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* No ACK in the segment */
|
|
|
|
|
|
|
|
if (th->rst) {
|
|
|
|
/* rfc793:
|
|
|
|
* "If the RST bit is set
|
|
|
|
*
|
|
|
|
* Otherwise (no ACK) drop the segment and return."
|
|
|
|
*/
|
|
|
|
|
|
|
|
goto discard_and_undo;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* PAWS check. */
|
2008-01-01 01:57:14 +03:00
|
|
|
if (tp->rx_opt.ts_recent_stamp && tp->rx_opt.saw_tstamp &&
|
2009-03-14 17:23:03 +03:00
|
|
|
tcp_paws_reject(&tp->rx_opt, 0))
|
2005-04-17 02:20:36 +04:00
|
|
|
goto discard_and_undo;
|
|
|
|
|
|
|
|
if (th->syn) {
|
|
|
|
/* We see SYN without ACK. It is attempt of
|
|
|
|
* simultaneous connect with crossed SYNs.
|
|
|
|
* Particularly, it can be connect to self.
|
|
|
|
*/
|
|
|
|
tcp_set_state(sk, TCP_SYN_RECV);
|
|
|
|
|
|
|
|
if (tp->rx_opt.saw_tstamp) {
|
|
|
|
tp->rx_opt.tstamp_ok = 1;
|
|
|
|
tcp_store_ts_recent(tp);
|
|
|
|
tp->tcp_header_len =
|
|
|
|
sizeof(struct tcphdr) + TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
} else {
|
|
|
|
tp->tcp_header_len = sizeof(struct tcphdr);
|
|
|
|
}
|
|
|
|
|
|
|
|
tp->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
|
2015-11-26 19:18:14 +03:00
|
|
|
tp->copied_seq = tp->rcv_nxt;
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->rcv_wup = TCP_SKB_CB(skb)->seq + 1;
|
|
|
|
|
|
|
|
/* RFC1323: The window in SYN & SYN/ACK segments is
|
|
|
|
* never scaled.
|
|
|
|
*/
|
|
|
|
tp->snd_wnd = ntohs(th->window);
|
|
|
|
tp->snd_wl1 = TCP_SKB_CB(skb)->seq;
|
|
|
|
tp->max_window = tp->snd_wnd;
|
|
|
|
|
2014-09-29 15:08:30 +04:00
|
|
|
tcp_ecn_rcv_syn(tp, th);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2006-03-21 04:53:41 +03:00
|
|
|
tcp_mtup_init(sk);
|
2005-12-14 10:26:10 +03:00
|
|
|
tcp_sync_mss(sk, icsk->icsk_pmtu_cookie);
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_initialize_rcv_mss(sk);
|
|
|
|
|
|
|
|
tcp_send_synack(sk);
|
|
|
|
#if 0
|
|
|
|
/* Note, we could accept data and URG from this segment.
|
2012-08-31 16:29:13 +04:00
|
|
|
* There are no obstacles to make this (except that we must
|
|
|
|
* either change tcp_recvmsg() to prevent it from returning data
|
|
|
|
* before 3WHS completes per RFC793, or employ TCP Fast Open).
|
2005-04-17 02:20:36 +04:00
|
|
|
*
|
|
|
|
* However, if we ignore data in ACKless segments sometimes,
|
|
|
|
* we have no reasons to accept it sometimes.
|
|
|
|
* Also, seems the code doing it in step6 of tcp_rcv_state_process
|
|
|
|
* is not flawless. So, discard packet for sanity.
|
|
|
|
* Uncomment this return to process the data.
|
|
|
|
*/
|
|
|
|
return -1;
|
|
|
|
#else
|
|
|
|
goto discard;
|
|
|
|
#endif
|
|
|
|
}
|
|
|
|
/* "fifth, if neither of the SYN or RST bits is set then
|
|
|
|
* drop the segment and return."
|
|
|
|
*/
|
|
|
|
|
|
|
|
discard_and_undo:
|
|
|
|
tcp_clear_options(&tp->rx_opt);
|
|
|
|
tp->rx_opt.mss_clamp = saved_clamp;
|
|
|
|
goto discard;
|
|
|
|
|
|
|
|
reset_and_undo:
|
|
|
|
tcp_clear_options(&tp->rx_opt);
|
|
|
|
tp->rx_opt.mss_clamp = saved_clamp;
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* This function implements the receiving procedure of RFC 793 for
|
2007-02-09 17:24:47 +03:00
|
|
|
* all states except ESTABLISHED and TIME_WAIT.
|
2005-04-17 02:20:36 +04:00
|
|
|
* It's called from both tcp_v4_rcv and tcp_v6_rcv and should be
|
|
|
|
* address independent.
|
|
|
|
*/
|
2007-02-09 17:24:47 +03:00
|
|
|
|
2015-09-29 17:42:41 +03:00
|
|
|
int tcp_rcv_state_process(struct sock *sk, struct sk_buff *skb)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2005-12-14 10:15:52 +03:00
|
|
|
struct inet_connection_sock *icsk = inet_csk(sk);
|
2015-09-29 17:42:41 +03:00
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
2012-08-31 16:29:13 +04:00
|
|
|
struct request_sock *req;
|
2005-04-17 02:20:36 +04:00
|
|
|
int queued = 0;
|
2013-05-24 19:03:54 +04:00
|
|
|
bool acceptable;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
switch (sk->sk_state) {
|
|
|
|
case TCP_CLOSE:
|
|
|
|
goto discard;
|
|
|
|
|
|
|
|
case TCP_LISTEN:
|
2007-03-09 07:45:19 +03:00
|
|
|
if (th->ack)
|
2005-04-17 02:20:36 +04:00
|
|
|
return 1;
|
|
|
|
|
2007-03-09 07:45:19 +03:00
|
|
|
if (th->rst)
|
2005-04-17 02:20:36 +04:00
|
|
|
goto discard;
|
|
|
|
|
2007-03-09 07:45:19 +03:00
|
|
|
if (th->syn) {
|
2011-12-03 03:41:42 +04:00
|
|
|
if (th->fin)
|
|
|
|
goto discard;
|
2005-12-14 10:15:52 +03:00
|
|
|
if (icsk->icsk_af_ops->conn_request(sk, skb) < 0)
|
2005-04-17 02:20:36 +04:00
|
|
|
return 1;
|
|
|
|
|
2016-04-22 08:13:01 +03:00
|
|
|
consume_skb(skb);
|
2007-01-24 07:15:06 +03:00
|
|
|
return 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
goto discard;
|
|
|
|
|
|
|
|
case TCP_SYN_SENT:
|
2016-04-14 08:05:40 +03:00
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
2015-09-29 17:42:40 +03:00
|
|
|
queued = tcp_rcv_synsent_state_process(sk, skb, th);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (queued >= 0)
|
|
|
|
return queued;
|
|
|
|
|
|
|
|
/* Do step6 onward by hand. */
|
|
|
|
tcp_urg(sk, skb, th);
|
|
|
|
__kfree_skb(skb);
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_data_snd_check(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-04-14 08:05:40 +03:00
|
|
|
tp->rx_opt.saw_tstamp = 0;
|
2012-08-31 16:29:13 +04:00
|
|
|
req = tp->fastopen_rsk;
|
2015-04-03 11:17:27 +03:00
|
|
|
if (req) {
|
2012-10-22 15:26:36 +04:00
|
|
|
WARN_ON_ONCE(sk->sk_state != TCP_SYN_RECV &&
|
2012-08-31 16:29:13 +04:00
|
|
|
sk->sk_state != TCP_FIN_WAIT1);
|
|
|
|
|
2015-04-03 11:17:26 +03:00
|
|
|
if (!tcp_check_req(sk, skb, req, true))
|
2012-08-31 16:29:13 +04:00
|
|
|
goto discard;
|
2012-09-22 08:18:57 +04:00
|
|
|
}
|
2012-12-26 16:44:34 +04:00
|
|
|
|
tcp: Restore RFC5961-compliant behavior for SYN packets
Commit c3ae62af8e755 ("tcp: should drop incoming frames without ACK
flag set") was created to mitigate a security vulnerability in which a
local attacker is able to inject data into locally-opened sockets by
using TCP protocol statistics in procfs to quickly find the correct
sequence number.
This broke the RFC5961 requirement to send a challenge ACK in response
to spurious RST packets, which was subsequently fixed by commit
7b514a886ba50 ("tcp: accept RST without ACK flag").
Unfortunately, the RFC5961 requirement that spurious SYN packets be
handled in a similar manner remains broken.
RFC5961 section 4 states that:
... the handling of the SYN in the synchronized state SHOULD be
performed as follows:
1) If the SYN bit is set, irrespective of the sequence number, TCP
MUST send an ACK (also referred to as challenge ACK) to the remote
peer:
<SEQ=SND.NXT><ACK=RCV.NXT><CTL=ACK>
After sending the acknowledgment, TCP MUST drop the unacceptable
segment and stop processing further.
By sending an ACK, the remote peer is challenged to confirm the loss
of the previous connection and the request to start a new connection.
A legitimate peer, after restart, would not have a TCB in the
synchronized state. Thus, when the ACK arrives, the peer should send
a RST segment back with the sequence number derived from the ACK
field that caused the RST.
This RST will confirm that the remote peer has indeed closed the
previous connection. Upon receipt of a valid RST, the local TCP
endpoint MUST terminate its connection. The local TCP endpoint
should then rely on SYN retransmission from the remote end to
re-establish the connection.
This patch lets SYN packets through the discard added in c3ae62af8e755,
so that spurious SYN packets are properly dealt with as per the RFC.
The challenge ACK is sent unconditionally and is rate-limited, so the
original vulnerability is not reintroduced by this patch.
Signed-off-by: Calvin Owens <calvinowens@fb.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Acked-by: Neal Cardwell <ncardwell@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-21 02:09:53 +03:00
|
|
|
if (!th->ack && !th->rst && !th->syn)
|
2012-12-26 16:44:34 +04:00
|
|
|
goto discard;
|
|
|
|
|
2012-09-22 08:18:57 +04:00
|
|
|
if (!tcp_validate_incoming(sk, skb, th, 0))
|
2012-07-17 05:41:30 +04:00
|
|
|
return 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* step 5: check the ACK field */
|
2013-05-24 19:03:54 +04:00
|
|
|
acceptable = tcp_ack(sk, skb, FLAG_SLOWPATH |
|
|
|
|
FLAG_UPDATE_TS_RECENT) > 0;
|
2012-08-31 16:29:13 +04:00
|
|
|
|
2013-05-24 19:03:54 +04:00
|
|
|
switch (sk->sk_state) {
|
|
|
|
case TCP_SYN_RECV:
|
2013-05-24 22:36:13 +04:00
|
|
|
if (!acceptable)
|
|
|
|
return 1;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2015-09-18 21:36:14 +03:00
|
|
|
if (!tp->srtt_us)
|
|
|
|
tcp_synack_rtt_meas(sk, req);
|
|
|
|
|
2013-05-24 22:36:13 +04:00
|
|
|
/* Once we leave TCP_SYN_RECV, we no longer need req
|
|
|
|
* so release it.
|
|
|
|
*/
|
|
|
|
if (req) {
|
|
|
|
tp->total_retrans = req->num_retrans;
|
|
|
|
reqsk_fastopen_remove(sk, req, false);
|
|
|
|
} else {
|
|
|
|
/* Make sure socket is routed, for correct metrics. */
|
|
|
|
icsk->icsk_af_ops->rebuild_header(sk);
|
|
|
|
tcp_init_congestion_control(sk);
|
|
|
|
|
|
|
|
tcp_mtup_init(sk);
|
|
|
|
tp->copied_seq = tp->rcv_nxt;
|
2013-09-21 00:56:58 +04:00
|
|
|
tcp_init_buffer_space(sk);
|
2013-05-24 22:36:13 +04:00
|
|
|
}
|
|
|
|
smp_mb();
|
|
|
|
tcp_set_state(sk, TCP_ESTABLISHED);
|
|
|
|
sk->sk_state_change(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-05-24 22:36:13 +04:00
|
|
|
/* Note, that this wakeup is only for marginal crossed SYN case.
|
|
|
|
* Passively open sockets are not waked up, because
|
|
|
|
* sk->sk_sleep == NULL and sk->sk_socket == NULL.
|
|
|
|
*/
|
|
|
|
if (sk->sk_socket)
|
|
|
|
sk_wake_async(sk, SOCK_WAKE_IO, POLL_OUT);
|
|
|
|
|
|
|
|
tp->snd_una = TCP_SKB_CB(skb)->ack_seq;
|
|
|
|
tp->snd_wnd = ntohs(th->window) << tp->rx_opt.snd_wscale;
|
|
|
|
tcp_init_wl(tp, TCP_SKB_CB(skb)->seq);
|
|
|
|
|
|
|
|
if (tp->rx_opt.tstamp_ok)
|
|
|
|
tp->advmss -= TCPOLEN_TSTAMP_ALIGNED;
|
|
|
|
|
|
|
|
if (req) {
|
|
|
|
/* Re-arm the timer because data may have been sent out.
|
|
|
|
* This is similar to the regular data transmission case
|
|
|
|
* when new data has just been ack'ed.
|
|
|
|
*
|
|
|
|
* (TFO) - we could try to be more aggressive and
|
|
|
|
* retransmitting any data sooner based on when they
|
|
|
|
* are sent out.
|
2013-05-24 19:03:54 +04:00
|
|
|
*/
|
2013-05-24 22:36:13 +04:00
|
|
|
tcp_rearm_rto(sk);
|
|
|
|
} else
|
|
|
|
tcp_init_metrics(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
tcp: initialize passive-side sk_pacing_rate after 3WHS
For passive TCP connections, upon receiving the ACK that completes the
3WHS, make sure we set our pacing rate after we get our first RTT
sample.
On passive TCP connections, when we receive the ACK completing the
3WHS we do not take an RTT sample in tcp_ack(), but rather in
tcp_synack_rtt_meas(). So upon receiving the ACK that completes the
3WHS, tcp_ack() leaves sk_pacing_rate at its initial value.
Originally the initial sk_pacing_rate value was 0, so passive-side
connections defaulted to sysctl_tcp_min_tso_segs (2 segs) in skbuffs
made in the first RTT. With a default initial cwnd of 10 packets, this
happened to be correct for RTTs 5ms or bigger, so it was hard to
see problems in WAN or emulated WAN testing.
Since 7eec4174ff ("pkt_sched: fq: fix non TCP flows pacing"), the
initial sk_pacing_rate is 0xffffffff. So after that change, passive
TCP connections were keeping this value (and using large numbers of
segments per skbuff) until receiving an ACK for data.
Signed-off-by: Neal Cardwell <ncardwell@google.com>
Cc: Eric Dumazet <edumazet@google.com>
Cc: Yuchung Cheng <ycheng@google.com>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2013-10-21 23:40:19 +04:00
|
|
|
tcp_update_pacing_rate(sk);
|
|
|
|
|
2013-05-24 22:36:13 +04:00
|
|
|
/* Prevent spurious tcp_cwnd_restart() on first data packet */
|
|
|
|
tp->lsndtime = tcp_time_stamp;
|
|
|
|
|
|
|
|
tcp_initialize_rcv_mss(sk);
|
|
|
|
tcp_fast_path_on(tp);
|
2013-05-24 19:03:54 +04:00
|
|
|
break;
|
|
|
|
|
2013-05-24 22:06:58 +04:00
|
|
|
case TCP_FIN_WAIT1: {
|
|
|
|
struct dst_entry *dst;
|
|
|
|
int tmo;
|
|
|
|
|
2013-05-24 19:03:54 +04:00
|
|
|
/* If we enter the TCP_FIN_WAIT1 state and we are a
|
|
|
|
* Fast Open socket and this is the first acceptable
|
|
|
|
* ACK we have received, this would have acknowledged
|
|
|
|
* our SYNACK so stop the SYNACK timer.
|
|
|
|
*/
|
2015-04-03 11:17:27 +03:00
|
|
|
if (req) {
|
2013-05-24 19:03:54 +04:00
|
|
|
/* Return RST if ack_seq is invalid.
|
|
|
|
* Note that RFC793 only says to generate a
|
|
|
|
* DUPACK for it but for TCP Fast Open it seems
|
|
|
|
* better to treat this case like TCP_SYN_RECV
|
|
|
|
* above.
|
2012-08-31 16:29:13 +04:00
|
|
|
*/
|
2013-05-24 19:03:54 +04:00
|
|
|
if (!acceptable)
|
|
|
|
return 1;
|
|
|
|
/* We no longer need the request sock. */
|
|
|
|
reqsk_fastopen_remove(sk, req, false);
|
|
|
|
tcp_rearm_rto(sk);
|
|
|
|
}
|
2013-05-24 22:06:58 +04:00
|
|
|
if (tp->snd_una != tp->write_seq)
|
|
|
|
break;
|
2013-05-24 19:03:54 +04:00
|
|
|
|
2013-05-24 22:06:58 +04:00
|
|
|
tcp_set_state(sk, TCP_FIN_WAIT2);
|
|
|
|
sk->sk_shutdown |= SEND_SHUTDOWN;
|
2013-05-24 19:03:54 +04:00
|
|
|
|
2013-05-24 22:06:58 +04:00
|
|
|
dst = __sk_dst_get(sk);
|
|
|
|
if (dst)
|
|
|
|
dst_confirm(dst);
|
2013-05-24 19:03:54 +04:00
|
|
|
|
2013-05-24 22:06:58 +04:00
|
|
|
if (!sock_flag(sk, SOCK_DEAD)) {
|
|
|
|
/* Wake up lingering close() */
|
|
|
|
sk->sk_state_change(sk);
|
|
|
|
break;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-05-24 22:06:58 +04:00
|
|
|
if (tp->linger2 < 0 ||
|
|
|
|
(TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt))) {
|
|
|
|
tcp_done(sk);
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
|
2013-05-24 22:06:58 +04:00
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
|
|
|
|
tmo = tcp_fin_time(sk);
|
|
|
|
if (tmo > TCP_TIMEWAIT_LEN) {
|
|
|
|
inet_csk_reset_keepalive_timer(sk, tmo - TCP_TIMEWAIT_LEN);
|
|
|
|
} else if (th->fin || sock_owned_by_user(sk)) {
|
|
|
|
/* Bad case. We could lose such FIN otherwise.
|
|
|
|
* It is not a big problem, but it looks confusing
|
|
|
|
* and not so rare event. We still can lose it now,
|
|
|
|
* if it spins in bh_lock_sock(), but it is really
|
|
|
|
* marginal case.
|
|
|
|
*/
|
|
|
|
inet_csk_reset_keepalive_timer(sk, tmo);
|
|
|
|
} else {
|
|
|
|
tcp_time_wait(sk, TCP_FIN_WAIT2, tmo);
|
|
|
|
goto discard;
|
2013-05-24 19:03:54 +04:00
|
|
|
}
|
|
|
|
break;
|
2013-05-24 22:06:58 +04:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-05-24 19:03:54 +04:00
|
|
|
case TCP_CLOSING:
|
|
|
|
if (tp->snd_una == tp->write_seq) {
|
|
|
|
tcp_time_wait(sk, TCP_TIME_WAIT, 0);
|
|
|
|
goto discard;
|
|
|
|
}
|
|
|
|
break;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2013-05-24 19:03:54 +04:00
|
|
|
case TCP_LAST_ACK:
|
|
|
|
if (tp->snd_una == tp->write_seq) {
|
|
|
|
tcp_update_metrics(sk);
|
|
|
|
tcp_done(sk);
|
|
|
|
goto discard;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2013-05-24 19:03:54 +04:00
|
|
|
break;
|
2012-12-26 16:44:34 +04:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/* step 6: check the URG bit */
|
|
|
|
tcp_urg(sk, skb, th);
|
|
|
|
|
|
|
|
/* step 7: process the segment text */
|
|
|
|
switch (sk->sk_state) {
|
|
|
|
case TCP_CLOSE_WAIT:
|
|
|
|
case TCP_CLOSING:
|
|
|
|
case TCP_LAST_ACK:
|
|
|
|
if (!before(TCP_SKB_CB(skb)->seq, tp->rcv_nxt))
|
|
|
|
break;
|
|
|
|
case TCP_FIN_WAIT1:
|
|
|
|
case TCP_FIN_WAIT2:
|
|
|
|
/* RFC 793 says to queue data in these states,
|
2007-02-09 17:24:47 +03:00
|
|
|
* RFC 1122 says we MUST send a reset.
|
2005-04-17 02:20:36 +04:00
|
|
|
* BSD 4.4 also does reset.
|
|
|
|
*/
|
|
|
|
if (sk->sk_shutdown & RCV_SHUTDOWN) {
|
|
|
|
if (TCP_SKB_CB(skb)->end_seq != TCP_SKB_CB(skb)->seq &&
|
|
|
|
after(TCP_SKB_CB(skb)->end_seq - th->fin, tp->rcv_nxt)) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPABORTONDATA);
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_reset(sk);
|
|
|
|
return 1;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/* Fall through */
|
2007-02-09 17:24:47 +03:00
|
|
|
case TCP_ESTABLISHED:
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_data_queue(sk, skb);
|
|
|
|
queued = 1;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* tcp_data could move socket to TIME-WAIT */
|
|
|
|
if (sk->sk_state != TCP_CLOSE) {
|
[TCP]: Sed magic converts func(sk, tp, ...) -> func(sk, ...)
This is (mostly) automated change using magic:
sed -e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e '/struct sock \*sk/ N' -e '/struct sock \*sk/ N'
-e 's|struct sock \*sk,[\n\t ]*struct tcp_sock \*tp\([^{]*\n{\n\)|
struct sock \*sk\1\tstruct tcp_sock *tp = tcp_sk(sk);\n|g'
-e 's|struct sock \*sk, struct tcp_sock \*tp|
struct sock \*sk|g' -e 's|sk, tp\([^-]\)|sk\1|g'
Fixed four unused variable (tp) warnings that were introduced.
In addition, manually added newlines after local variables and
tweaked function arguments positioning.
$ gcc --version
gcc (GCC) 4.1.1 20060525 (Red Hat 4.1.1-1)
...
$ codiff -fV built-in.o.old built-in.o.new
net/ipv4/route.c:
rt_cache_flush | +14
1 function changed, 14 bytes added
net/ipv4/tcp.c:
tcp_setsockopt | -5
tcp_sendpage | -25
tcp_sendmsg | -16
3 functions changed, 46 bytes removed
net/ipv4/tcp_input.c:
tcp_try_undo_recovery | +3
tcp_try_undo_dsack | +2
tcp_mark_head_lost | -12
tcp_ack | -15
tcp_event_data_recv | -32
tcp_rcv_state_process | -10
tcp_rcv_established | +1
7 functions changed, 6 bytes added, 69 bytes removed, diff: -63
net/ipv4/tcp_output.c:
update_send_head | -9
tcp_transmit_skb | +19
tcp_cwnd_validate | +1
tcp_write_wakeup | -17
__tcp_push_pending_frames | -25
tcp_push_one | -8
tcp_send_fin | -4
7 functions changed, 20 bytes added, 63 bytes removed, diff: -43
built-in.o.new:
18 functions changed, 40 bytes added, 178 bytes removed, diff: -138
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@helsinki.fi>
Signed-off-by: David S. Miller <davem@davemloft.net>
2007-04-21 09:18:02 +04:00
|
|
|
tcp_data_snd_check(sk);
|
2005-04-17 02:20:36 +04:00
|
|
|
tcp_ack_snd_check(sk);
|
|
|
|
}
|
|
|
|
|
2007-02-09 17:24:47 +03:00
|
|
|
if (!queued) {
|
2005-04-17 02:20:36 +04:00
|
|
|
discard:
|
2016-04-01 18:52:19 +03:00
|
|
|
tcp_drop(sk, skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(tcp_rcv_state_process);
|
2014-06-25 18:10:02 +04:00
|
|
|
|
|
|
|
static inline void pr_drop_req(struct request_sock *req, __u16 port, int family)
|
|
|
|
{
|
|
|
|
struct inet_request_sock *ireq = inet_rsk(req);
|
|
|
|
|
|
|
|
if (family == AF_INET)
|
2014-11-11 21:59:17 +03:00
|
|
|
net_dbg_ratelimited("drop open request from %pI4/%u\n",
|
|
|
|
&ireq->ir_rmt_addr, port);
|
2014-06-28 22:20:54 +04:00
|
|
|
#if IS_ENABLED(CONFIG_IPV6)
|
|
|
|
else if (family == AF_INET6)
|
2014-11-11 21:59:17 +03:00
|
|
|
net_dbg_ratelimited("drop open request from %pI6/%u\n",
|
|
|
|
&ireq->ir_v6_rmt_addr, port);
|
2014-06-28 22:20:54 +04:00
|
|
|
#endif
|
2014-06-25 18:10:02 +04:00
|
|
|
}
|
|
|
|
|
2014-09-29 15:08:29 +04:00
|
|
|
/* RFC3168 : 6.1.1 SYN packets must not have ECT/ECN bits set
|
|
|
|
*
|
|
|
|
* If we receive a SYN packet with these bits set, it means a
|
|
|
|
* network is playing bad games with TOS bits. In order to
|
|
|
|
* avoid possible false congestion notifications, we disable
|
2014-10-30 02:05:06 +03:00
|
|
|
* TCP ECN negotiation.
|
2014-09-29 15:08:29 +04:00
|
|
|
*
|
|
|
|
* Exception: tcp_ca wants ECN. This is required for DCTCP
|
net: dctcp: loosen requirement to assert ECT(0) during 3WHS
One deployment requirement of DCTCP is to be able to run
in a DC setting along with TCP traffic. As Glenn Judd's
NSDI'15 paper "Attaining the Promise and Avoiding the Pitfalls
of TCP in the Datacenter" [1] (tba) explains, one way to
solve this on switch side is to split DCTCP and TCP traffic
in two queues per switch port based on the DSCP: one queue
soley intended for DCTCP traffic and one for non-DCTCP traffic.
For the DCTCP queue, there's the marking threshold K as
explained in commit e3118e8359bb ("net: tcp: add DCTCP congestion
control algorithm") for RED marking ECT(0) packets with CE.
For the non-DCTCP queue, there's f.e. a classic tail drop queue.
As already explained in e3118e8359bb, running DCTCP at scale
when not marking SYN/SYN-ACK packets with ECT(0) has severe
consequences as for non-ECT(0) packets, traversing the RED
marking DCTCP queue will result in a severe reduction of
connection probability.
This is due to the DCTCP queue being dominated by ECT(0) traffic
and switches handle non-ECT traffic in the RED marking queue
after passing K as drops, where K is usually a low watermark
in order to leave enough tailroom for bursts. Splitting DCTCP
traffic among several queues (ECN and non-ECN queue) is being
considered a terrible idea in the network community as it
splits single flows across multiple network paths.
Therefore, commit e3118e8359bb implements this on Linux as
ECT(0) marked traffic, as we argue that marking all packets
of a DCTCP flow is the only viable solution and also doesn't
speak against the draft.
However, recently, a DCTCP implementation for FreeBSD hit also
their mainline kernel [2]. In order to let them play well
together with Linux' DCTCP, we would need to loosen the
requirement that ECT(0) has to be asserted during the 3WHS as
not implemented in FreeBSD. This simplifies the ECN test and
lets DCTCP work together with FreeBSD.
Joint work with Daniel Borkmann.
[1] https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/judd
[2] https://github.com/freebsd/freebsd/commit/8ad879445281027858a7fa706d13e458095b595f
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Glenn Judd <glenn.judd@morganstanley.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-30 22:45:20 +03:00
|
|
|
* congestion control: Linux DCTCP asserts ECT on all packets,
|
|
|
|
* including SYN, which is most optimal solution; however,
|
|
|
|
* others, such as FreeBSD do not.
|
2014-09-29 15:08:29 +04:00
|
|
|
*/
|
|
|
|
static void tcp_ecn_create_request(struct request_sock *req,
|
|
|
|
const struct sk_buff *skb,
|
net: allow setting ecn via routing table
This patch allows to set ECN on a per-route basis in case the sysctl
tcp_ecn is not set to 1. In other words, when ECN is set for specific
routes, it provides a tcp_ecn=1 behaviour for that route while the rest
of the stack acts according to the global settings.
One can use 'ip route change dev $dev $net features ecn' to toggle this.
Having a more fine-grained per-route setting can be beneficial for various
reasons, for example, 1) within data centers, or 2) local ISPs may deploy
ECN support for their own video/streaming services [1], etc.
There was a recent measurement study/paper [2] which scanned the Alexa's
publicly available top million websites list from a vantage point in US,
Europe and Asia:
Half of the Alexa list will now happily use ECN (tcp_ecn=2, most likely
blamed to commit 255cac91c3 ("tcp: extend ECN sysctl to allow server-side
only ECN") ;)); the break in connectivity on-path was found is about
1 in 10,000 cases. Timeouts rather than receiving back RSTs were much
more common in the negotiation phase (and mostly seen in the Alexa
middle band, ranks around 50k-150k): from 12-thousand hosts on which
there _may_ be ECN-linked connection failures, only 79 failed with RST
when _not_ failing with RST when ECN is not requested.
It's unclear though, how much equipment in the wild actually marks CE
when buffers start to fill up.
We thought about a fallback to non-ECN for retransmitted SYNs as another
global option (which could perhaps one day be made default), but as Eric
points out, there's much more work needed to detect broken middleboxes.
Two examples Eric mentioned are buggy firewalls that accept only a single
SYN per flow, and middleboxes that successfully let an ECN flow establish,
but later mark CE for all packets (so cwnd converges to 1).
[1] http://www.ietf.org/proceedings/89/slides/slides-89-tsvarea-1.pdf, p.15
[2] http://ecn.ethz.ch/
Joint work with Daniel Borkmann.
Reference: http://thread.gmane.org/gmane.linux.network/335797
Suggested-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Daniel Borkmann <dborkman@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-03 19:35:03 +03:00
|
|
|
const struct sock *listen_sk,
|
|
|
|
const struct dst_entry *dst)
|
2014-09-29 15:08:29 +04:00
|
|
|
{
|
|
|
|
const struct tcphdr *th = tcp_hdr(skb);
|
|
|
|
const struct net *net = sock_net(listen_sk);
|
|
|
|
bool th_ecn = th->ece && th->cwr;
|
net: dctcp: loosen requirement to assert ECT(0) during 3WHS
One deployment requirement of DCTCP is to be able to run
in a DC setting along with TCP traffic. As Glenn Judd's
NSDI'15 paper "Attaining the Promise and Avoiding the Pitfalls
of TCP in the Datacenter" [1] (tba) explains, one way to
solve this on switch side is to split DCTCP and TCP traffic
in two queues per switch port based on the DSCP: one queue
soley intended for DCTCP traffic and one for non-DCTCP traffic.
For the DCTCP queue, there's the marking threshold K as
explained in commit e3118e8359bb ("net: tcp: add DCTCP congestion
control algorithm") for RED marking ECT(0) packets with CE.
For the non-DCTCP queue, there's f.e. a classic tail drop queue.
As already explained in e3118e8359bb, running DCTCP at scale
when not marking SYN/SYN-ACK packets with ECT(0) has severe
consequences as for non-ECT(0) packets, traversing the RED
marking DCTCP queue will result in a severe reduction of
connection probability.
This is due to the DCTCP queue being dominated by ECT(0) traffic
and switches handle non-ECT traffic in the RED marking queue
after passing K as drops, where K is usually a low watermark
in order to leave enough tailroom for bursts. Splitting DCTCP
traffic among several queues (ECN and non-ECN queue) is being
considered a terrible idea in the network community as it
splits single flows across multiple network paths.
Therefore, commit e3118e8359bb implements this on Linux as
ECT(0) marked traffic, as we argue that marking all packets
of a DCTCP flow is the only viable solution and also doesn't
speak against the draft.
However, recently, a DCTCP implementation for FreeBSD hit also
their mainline kernel [2]. In order to let them play well
together with Linux' DCTCP, we would need to loosen the
requirement that ECT(0) has to be asserted during the 3WHS as
not implemented in FreeBSD. This simplifies the ECN test and
lets DCTCP work together with FreeBSD.
Joint work with Daniel Borkmann.
[1] https://www.usenix.org/conference/nsdi15/technical-sessions/presentation/judd
[2] https://github.com/freebsd/freebsd/commit/8ad879445281027858a7fa706d13e458095b595f
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Cc: Glenn Judd <glenn.judd@morganstanley.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-30 22:45:20 +03:00
|
|
|
bool ect, ecn_ok;
|
tcp: use dctcp if enabled on the route to the initiator
Currently, the following case doesn't use DCTCP, even if it should:
A responder has f.e. Cubic as system wide default, but for a specific
route to the initiating host, DCTCP is being set in RTAX_CC_ALGO. The
initiating host then uses DCTCP as congestion control, but since the
initiator sets ECT(0), tcp_ecn_create_request() doesn't set ecn_ok,
and we have to fall back to Reno after 3WHS completes.
We were thinking on how to solve this in a minimal, non-intrusive
way without bloating tcp_ecn_create_request() needlessly: lets cache
the CA ecn option flag in RTAX_FEATURES. In other words, when ECT(0)
is set on the SYN packet, set ecn_ok=1 iff route RTAX_FEATURES
contains the unexposed (internal-only) DST_FEATURE_ECN_CA. This allows
to only do a single metric feature lookup inside tcp_ecn_create_request().
Joint work with Florian Westphal.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-31 16:58:47 +03:00
|
|
|
u32 ecn_ok_dst;
|
2014-09-29 15:08:29 +04:00
|
|
|
|
|
|
|
if (!th_ecn)
|
|
|
|
return;
|
|
|
|
|
|
|
|
ect = !INET_ECN_is_not_ect(TCP_SKB_CB(skb)->ip_dsfield);
|
tcp: use dctcp if enabled on the route to the initiator
Currently, the following case doesn't use DCTCP, even if it should:
A responder has f.e. Cubic as system wide default, but for a specific
route to the initiating host, DCTCP is being set in RTAX_CC_ALGO. The
initiating host then uses DCTCP as congestion control, but since the
initiator sets ECT(0), tcp_ecn_create_request() doesn't set ecn_ok,
and we have to fall back to Reno after 3WHS completes.
We were thinking on how to solve this in a minimal, non-intrusive
way without bloating tcp_ecn_create_request() needlessly: lets cache
the CA ecn option flag in RTAX_FEATURES. In other words, when ECT(0)
is set on the SYN packet, set ecn_ok=1 iff route RTAX_FEATURES
contains the unexposed (internal-only) DST_FEATURE_ECN_CA. This allows
to only do a single metric feature lookup inside tcp_ecn_create_request().
Joint work with Florian Westphal.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-31 16:58:47 +03:00
|
|
|
ecn_ok_dst = dst_feature(dst, DST_FEATURE_ECN_MASK);
|
|
|
|
ecn_ok = net->ipv4.sysctl_tcp_ecn || ecn_ok_dst;
|
2014-09-29 15:08:29 +04:00
|
|
|
|
tcp: use dctcp if enabled on the route to the initiator
Currently, the following case doesn't use DCTCP, even if it should:
A responder has f.e. Cubic as system wide default, but for a specific
route to the initiating host, DCTCP is being set in RTAX_CC_ALGO. The
initiating host then uses DCTCP as congestion control, but since the
initiator sets ECT(0), tcp_ecn_create_request() doesn't set ecn_ok,
and we have to fall back to Reno after 3WHS completes.
We were thinking on how to solve this in a minimal, non-intrusive
way without bloating tcp_ecn_create_request() needlessly: lets cache
the CA ecn option flag in RTAX_FEATURES. In other words, when ECT(0)
is set on the SYN packet, set ecn_ok=1 iff route RTAX_FEATURES
contains the unexposed (internal-only) DST_FEATURE_ECN_CA. This allows
to only do a single metric feature lookup inside tcp_ecn_create_request().
Joint work with Florian Westphal.
Signed-off-by: Daniel Borkmann <daniel@iogearbox.net>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2015-08-31 16:58:47 +03:00
|
|
|
if ((!ect && ecn_ok) || tcp_ca_needs_ecn(listen_sk) ||
|
|
|
|
(ecn_ok_dst & DST_FEATURE_ECN_CA))
|
2014-09-29 15:08:29 +04:00
|
|
|
inet_rsk(req)->ecn_ok = 1;
|
|
|
|
}
|
|
|
|
|
2015-03-17 07:06:19 +03:00
|
|
|
static void tcp_openreq_init(struct request_sock *req,
|
|
|
|
const struct tcp_options_received *rx_opt,
|
|
|
|
struct sk_buff *skb, const struct sock *sk)
|
|
|
|
{
|
|
|
|
struct inet_request_sock *ireq = inet_rsk(req);
|
|
|
|
|
2015-10-09 05:33:23 +03:00
|
|
|
req->rsk_rcv_wnd = 0; /* So that tcp_send_synack() knows! */
|
2015-03-17 07:06:19 +03:00
|
|
|
req->cookie_ts = 0;
|
|
|
|
tcp_rsk(req)->rcv_isn = TCP_SKB_CB(skb)->seq;
|
|
|
|
tcp_rsk(req)->rcv_nxt = TCP_SKB_CB(skb)->seq + 1;
|
2015-09-18 21:36:14 +03:00
|
|
|
skb_mstamp_get(&tcp_rsk(req)->snt_synack);
|
2015-03-17 07:06:19 +03:00
|
|
|
tcp_rsk(req)->last_oow_ack_time = 0;
|
|
|
|
req->mss = rx_opt->mss_clamp;
|
|
|
|
req->ts_recent = rx_opt->saw_tstamp ? rx_opt->rcv_tsval : 0;
|
|
|
|
ireq->tstamp_ok = rx_opt->tstamp_ok;
|
|
|
|
ireq->sack_ok = rx_opt->sack_ok;
|
|
|
|
ireq->snd_wscale = rx_opt->snd_wscale;
|
|
|
|
ireq->wscale_ok = rx_opt->wscale_ok;
|
|
|
|
ireq->acked = 0;
|
|
|
|
ireq->ecn_ok = 0;
|
|
|
|
ireq->ir_rmt_port = tcp_hdr(skb)->source;
|
|
|
|
ireq->ir_num = ntohs(tcp_hdr(skb)->dest);
|
|
|
|
ireq->ir_mark = inet_request_mark(sk, skb);
|
|
|
|
}
|
|
|
|
|
2015-03-18 04:32:27 +03:00
|
|
|
struct request_sock *inet_reqsk_alloc(const struct request_sock_ops *ops,
|
2015-10-05 07:08:11 +03:00
|
|
|
struct sock *sk_listener,
|
|
|
|
bool attach_listener)
|
2015-03-18 04:32:27 +03:00
|
|
|
{
|
2015-10-05 07:08:11 +03:00
|
|
|
struct request_sock *req = reqsk_alloc(ops, sk_listener,
|
|
|
|
attach_listener);
|
2015-03-18 04:32:27 +03:00
|
|
|
|
|
|
|
if (req) {
|
|
|
|
struct inet_request_sock *ireq = inet_rsk(req);
|
|
|
|
|
|
|
|
kmemcheck_annotate_bitfield(ireq, flags);
|
|
|
|
ireq->opt = NULL;
|
|
|
|
atomic64_set(&ireq->ir_cookie, 0);
|
|
|
|
ireq->ireq_state = TCP_NEW_SYN_RECV;
|
|
|
|
write_pnet(&ireq->ireq_net, sock_net(sk_listener));
|
2015-03-25 07:45:56 +03:00
|
|
|
ireq->ireq_family = sk_listener->sk_family;
|
2015-03-18 04:32:27 +03:00
|
|
|
}
|
|
|
|
|
|
|
|
return req;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(inet_reqsk_alloc);
|
|
|
|
|
2015-03-26 01:08:47 +03:00
|
|
|
/*
|
|
|
|
* Return true if a syncookie should be sent
|
|
|
|
*/
|
2015-09-29 17:42:51 +03:00
|
|
|
static bool tcp_syn_flood_action(const struct sock *sk,
|
2015-03-26 01:08:47 +03:00
|
|
|
const struct sk_buff *skb,
|
|
|
|
const char *proto)
|
|
|
|
{
|
2015-10-02 21:43:25 +03:00
|
|
|
struct request_sock_queue *queue = &inet_csk(sk)->icsk_accept_queue;
|
2015-03-26 01:08:47 +03:00
|
|
|
const char *msg = "Dropping request";
|
|
|
|
bool want_cookie = false;
|
2016-02-03 10:46:51 +03:00
|
|
|
struct net *net = sock_net(sk);
|
2015-03-26 01:08:47 +03:00
|
|
|
|
|
|
|
#ifdef CONFIG_SYN_COOKIES
|
2016-02-03 10:46:51 +03:00
|
|
|
if (net->ipv4.sysctl_tcp_syncookies) {
|
2015-03-26 01:08:47 +03:00
|
|
|
msg = "Sending cookies";
|
|
|
|
want_cookie = true;
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPREQQFULLDOCOOKIES);
|
2015-03-26 01:08:47 +03:00
|
|
|
} else
|
|
|
|
#endif
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_TCPREQQFULLDROP);
|
2015-03-26 01:08:47 +03:00
|
|
|
|
2015-10-02 21:43:25 +03:00
|
|
|
if (!queue->synflood_warned &&
|
2016-02-03 10:46:51 +03:00
|
|
|
net->ipv4.sysctl_tcp_syncookies != 2 &&
|
2015-10-02 21:43:25 +03:00
|
|
|
xchg(&queue->synflood_warned, 1) == 0)
|
2015-03-26 01:08:47 +03:00
|
|
|
pr_info("%s: Possible SYN flooding on port %d. %s. Check SNMP counters.\n",
|
|
|
|
proto, ntohs(tcp_hdr(skb)->dest), msg);
|
2015-09-29 17:42:51 +03:00
|
|
|
|
2015-03-26 01:08:47 +03:00
|
|
|
return want_cookie;
|
|
|
|
}
|
|
|
|
|
2015-05-04 07:34:46 +03:00
|
|
|
static void tcp_reqsk_record_syn(const struct sock *sk,
|
|
|
|
struct request_sock *req,
|
|
|
|
const struct sk_buff *skb)
|
|
|
|
{
|
|
|
|
if (tcp_sk(sk)->save_syn) {
|
|
|
|
u32 len = skb_network_header_len(skb) + tcp_hdrlen(skb);
|
|
|
|
u32 *copy;
|
|
|
|
|
|
|
|
copy = kmalloc(len + sizeof(u32), GFP_ATOMIC);
|
|
|
|
if (copy) {
|
|
|
|
copy[0] = len;
|
|
|
|
memcpy(©[1], skb_network_header(skb), len);
|
|
|
|
req->saved_syn = copy;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2014-06-25 18:10:02 +04:00
|
|
|
int tcp_conn_request(struct request_sock_ops *rsk_ops,
|
|
|
|
const struct tcp_request_sock_ops *af_ops,
|
|
|
|
struct sock *sk, struct sk_buff *skb)
|
|
|
|
{
|
2015-09-25 03:16:05 +03:00
|
|
|
struct tcp_fastopen_cookie foc = { .len = -1 };
|
|
|
|
__u32 isn = TCP_SKB_CB(skb)->tcp_tw_isn;
|
2014-06-25 18:10:02 +04:00
|
|
|
struct tcp_options_received tmp_opt;
|
|
|
|
struct tcp_sock *tp = tcp_sk(sk);
|
2016-02-03 10:46:51 +03:00
|
|
|
struct net *net = sock_net(sk);
|
2015-09-25 03:16:05 +03:00
|
|
|
struct sock *fastopen_sk = NULL;
|
2014-06-25 18:10:02 +04:00
|
|
|
struct dst_entry *dst = NULL;
|
2015-09-25 03:16:05 +03:00
|
|
|
struct request_sock *req;
|
|
|
|
bool want_cookie = false;
|
2014-06-25 18:10:02 +04:00
|
|
|
struct flowi fl;
|
|
|
|
|
|
|
|
/* TW buckets are converted to open requests without
|
|
|
|
* limitations, they conserve resources and peer is
|
|
|
|
* evidently real one.
|
|
|
|
*/
|
2016-02-03 10:46:51 +03:00
|
|
|
if ((net->ipv4.sysctl_tcp_syncookies == 2 ||
|
2014-06-25 18:10:02 +04:00
|
|
|
inet_csk_reqsk_queue_is_full(sk)) && !isn) {
|
|
|
|
want_cookie = tcp_syn_flood_action(sk, skb, rsk_ops->slab_name);
|
|
|
|
if (!want_cookie)
|
|
|
|
goto drop;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
/* Accept backlog is full. If we have already queued enough
|
|
|
|
* of warm entries in syn queue, drop request. It is better than
|
|
|
|
* clogging syn queue with openreqs with exponentially increasing
|
|
|
|
* timeout.
|
|
|
|
*/
|
|
|
|
if (sk_acceptq_is_full(sk) && inet_csk_reqsk_queue_young(sk) > 1) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_LISTENOVERFLOWS);
|
2014-06-25 18:10:02 +04:00
|
|
|
goto drop;
|
|
|
|
}
|
|
|
|
|
2015-10-05 07:08:11 +03:00
|
|
|
req = inet_reqsk_alloc(rsk_ops, sk, !want_cookie);
|
2014-06-25 18:10:02 +04:00
|
|
|
if (!req)
|
|
|
|
goto drop;
|
|
|
|
|
|
|
|
tcp_rsk(req)->af_specific = af_ops;
|
|
|
|
|
|
|
|
tcp_clear_options(&tmp_opt);
|
|
|
|
tmp_opt.mss_clamp = af_ops->mss_clamp;
|
|
|
|
tmp_opt.user_mss = tp->rx_opt.user_mss;
|
|
|
|
tcp_parse_options(skb, &tmp_opt, 0, want_cookie ? NULL : &foc);
|
|
|
|
|
|
|
|
if (want_cookie && !tmp_opt.saw_tstamp)
|
|
|
|
tcp_clear_options(&tmp_opt);
|
|
|
|
|
|
|
|
tmp_opt.tstamp_ok = tmp_opt.saw_tstamp;
|
|
|
|
tcp_openreq_init(req, &tmp_opt, skb, sk);
|
|
|
|
|
2015-03-14 01:51:10 +03:00
|
|
|
/* Note: tcp_v6_init_req() might override ir_iif for link locals */
|
2015-12-17 00:20:44 +03:00
|
|
|
inet_rsk(req)->ir_iif = inet_request_bound_dev_if(sk, skb);
|
2015-03-14 01:51:10 +03:00
|
|
|
|
2014-06-25 18:10:02 +04:00
|
|
|
af_ops->init_req(req, sk, skb);
|
|
|
|
|
|
|
|
if (security_inet_conn_request(sk, skb, req))
|
|
|
|
goto drop_and_free;
|
|
|
|
|
net: allow setting ecn via routing table
This patch allows to set ECN on a per-route basis in case the sysctl
tcp_ecn is not set to 1. In other words, when ECN is set for specific
routes, it provides a tcp_ecn=1 behaviour for that route while the rest
of the stack acts according to the global settings.
One can use 'ip route change dev $dev $net features ecn' to toggle this.
Having a more fine-grained per-route setting can be beneficial for various
reasons, for example, 1) within data centers, or 2) local ISPs may deploy
ECN support for their own video/streaming services [1], etc.
There was a recent measurement study/paper [2] which scanned the Alexa's
publicly available top million websites list from a vantage point in US,
Europe and Asia:
Half of the Alexa list will now happily use ECN (tcp_ecn=2, most likely
blamed to commit 255cac91c3 ("tcp: extend ECN sysctl to allow server-side
only ECN") ;)); the break in connectivity on-path was found is about
1 in 10,000 cases. Timeouts rather than receiving back RSTs were much
more common in the negotiation phase (and mostly seen in the Alexa
middle band, ranks around 50k-150k): from 12-thousand hosts on which
there _may_ be ECN-linked connection failures, only 79 failed with RST
when _not_ failing with RST when ECN is not requested.
It's unclear though, how much equipment in the wild actually marks CE
when buffers start to fill up.
We thought about a fallback to non-ECN for retransmitted SYNs as another
global option (which could perhaps one day be made default), but as Eric
points out, there's much more work needed to detect broken middleboxes.
Two examples Eric mentioned are buggy firewalls that accept only a single
SYN per flow, and middleboxes that successfully let an ECN flow establish,
but later mark CE for all packets (so cwnd converges to 1).
[1] http://www.ietf.org/proceedings/89/slides/slides-89-tsvarea-1.pdf, p.15
[2] http://ecn.ethz.ch/
Joint work with Daniel Borkmann.
Reference: http://thread.gmane.org/gmane.linux.network/335797
Suggested-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Daniel Borkmann <dborkman@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-03 19:35:03 +03:00
|
|
|
if (!want_cookie && !isn) {
|
2014-06-25 18:10:02 +04:00
|
|
|
/* VJ's idea. We save last timestamp seen
|
|
|
|
* from the destination in peer table, when entering
|
|
|
|
* state TIME-WAIT, and check against it before
|
|
|
|
* accepting new connection request.
|
|
|
|
*
|
|
|
|
* If "isn" is not zero, this request hit alive
|
|
|
|
* timewait bucket, so that all the necessary checks
|
|
|
|
* are made in the function processing timewait state.
|
|
|
|
*/
|
2014-08-15 00:06:12 +04:00
|
|
|
if (tcp_death_row.sysctl_tw_recycle) {
|
2014-06-25 18:10:02 +04:00
|
|
|
bool strict;
|
|
|
|
|
|
|
|
dst = af_ops->route_req(sk, &fl, req, &strict);
|
2014-08-15 00:06:12 +04:00
|
|
|
|
2014-06-25 18:10:02 +04:00
|
|
|
if (dst && strict &&
|
2014-08-15 00:06:12 +04:00
|
|
|
!tcp_peer_is_proven(req, dst, true,
|
|
|
|
tmp_opt.saw_tstamp)) {
|
2016-04-28 02:44:39 +03:00
|
|
|
__NET_INC_STATS(sock_net(sk), LINUX_MIB_PAWSPASSIVEREJECTED);
|
2014-06-25 18:10:02 +04:00
|
|
|
goto drop_and_release;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
/* Kill the following clause, if you dislike this way. */
|
2016-02-03 10:46:51 +03:00
|
|
|
else if (!net->ipv4.sysctl_tcp_syncookies &&
|
2014-06-25 18:10:02 +04:00
|
|
|
(sysctl_max_syn_backlog - inet_csk_reqsk_queue_len(sk) <
|
|
|
|
(sysctl_max_syn_backlog >> 2)) &&
|
2014-08-15 00:06:12 +04:00
|
|
|
!tcp_peer_is_proven(req, dst, false,
|
|
|
|
tmp_opt.saw_tstamp)) {
|
2014-06-25 18:10:02 +04:00
|
|
|
/* Without syncookies last quarter of
|
|
|
|
* backlog is filled with destinations,
|
|
|
|
* proven to be alive.
|
|
|
|
* It means that we continue to communicate
|
|
|
|
* to destinations, already remembered
|
|
|
|
* to the moment of synflood.
|
|
|
|
*/
|
|
|
|
pr_drop_req(req, ntohs(tcp_hdr(skb)->source),
|
|
|
|
rsk_ops->family);
|
|
|
|
goto drop_and_release;
|
|
|
|
}
|
|
|
|
|
|
|
|
isn = af_ops->init_seq(skb);
|
|
|
|
}
|
|
|
|
if (!dst) {
|
|
|
|
dst = af_ops->route_req(sk, &fl, req, NULL);
|
|
|
|
if (!dst)
|
|
|
|
goto drop_and_free;
|
|
|
|
}
|
|
|
|
|
net: allow setting ecn via routing table
This patch allows to set ECN on a per-route basis in case the sysctl
tcp_ecn is not set to 1. In other words, when ECN is set for specific
routes, it provides a tcp_ecn=1 behaviour for that route while the rest
of the stack acts according to the global settings.
One can use 'ip route change dev $dev $net features ecn' to toggle this.
Having a more fine-grained per-route setting can be beneficial for various
reasons, for example, 1) within data centers, or 2) local ISPs may deploy
ECN support for their own video/streaming services [1], etc.
There was a recent measurement study/paper [2] which scanned the Alexa's
publicly available top million websites list from a vantage point in US,
Europe and Asia:
Half of the Alexa list will now happily use ECN (tcp_ecn=2, most likely
blamed to commit 255cac91c3 ("tcp: extend ECN sysctl to allow server-side
only ECN") ;)); the break in connectivity on-path was found is about
1 in 10,000 cases. Timeouts rather than receiving back RSTs were much
more common in the negotiation phase (and mostly seen in the Alexa
middle band, ranks around 50k-150k): from 12-thousand hosts on which
there _may_ be ECN-linked connection failures, only 79 failed with RST
when _not_ failing with RST when ECN is not requested.
It's unclear though, how much equipment in the wild actually marks CE
when buffers start to fill up.
We thought about a fallback to non-ECN for retransmitted SYNs as another
global option (which could perhaps one day be made default), but as Eric
points out, there's much more work needed to detect broken middleboxes.
Two examples Eric mentioned are buggy firewalls that accept only a single
SYN per flow, and middleboxes that successfully let an ECN flow establish,
but later mark CE for all packets (so cwnd converges to 1).
[1] http://www.ietf.org/proceedings/89/slides/slides-89-tsvarea-1.pdf, p.15
[2] http://ecn.ethz.ch/
Joint work with Daniel Borkmann.
Reference: http://thread.gmane.org/gmane.linux.network/335797
Suggested-by: Hannes Frederic Sowa <hannes@stressinduktion.org>
Acked-by: Eric Dumazet <edumazet@google.com>
Signed-off-by: Daniel Borkmann <dborkman@redhat.com>
Signed-off-by: Florian Westphal <fw@strlen.de>
Signed-off-by: David S. Miller <davem@davemloft.net>
2014-11-03 19:35:03 +03:00
|
|
|
tcp_ecn_create_request(req, skb, sk, dst);
|
|
|
|
|
|
|
|
if (want_cookie) {
|
|
|
|
isn = cookie_init_sequence(af_ops, sk, skb, &req->mss);
|
|
|
|
req->cookie_ts = tmp_opt.tstamp_ok;
|
|
|
|
if (!tmp_opt.tstamp_ok)
|
|
|
|
inet_rsk(req)->ecn_ok = 0;
|
|
|
|
}
|
|
|
|
|
2014-06-25 18:10:02 +04:00
|
|
|
tcp_rsk(req)->snt_isn = isn;
|
2015-09-16 01:24:20 +03:00
|
|
|
tcp_rsk(req)->txhash = net_tx_rndhash();
|
2014-06-25 18:10:02 +04:00
|
|
|
tcp_openreq_init_rwin(req, sk, dst);
|
2015-10-02 21:43:35 +03:00
|
|
|
if (!want_cookie) {
|
|
|
|
tcp_reqsk_record_syn(sk, req, skb);
|
2015-10-05 07:08:07 +03:00
|
|
|
fastopen_sk = tcp_try_fastopen(sk, skb, req, &foc, dst);
|
2015-10-02 21:43:35 +03:00
|
|
|
}
|
2015-09-25 03:16:05 +03:00
|
|
|
if (fastopen_sk) {
|
2015-10-02 21:43:35 +03:00
|
|
|
af_ops->send_synack(fastopen_sk, dst, &fl, req,
|
2016-04-14 08:05:39 +03:00
|
|
|
&foc, TCP_SYNACK_FASTOPEN);
|
2015-10-05 07:08:07 +03:00
|
|
|
/* Add the child socket directly into the accept queue */
|
|
|
|
inet_csk_reqsk_queue_add(sk, req, fastopen_sk);
|
|
|
|
sk->sk_data_ready(sk);
|
|
|
|
bh_unlock_sock(fastopen_sk);
|
2015-09-25 03:16:05 +03:00
|
|
|
sock_put(fastopen_sk);
|
|
|
|
} else {
|
2015-03-18 04:32:29 +03:00
|
|
|
tcp_rsk(req)->tfo_listener = false;
|
2015-10-02 21:43:35 +03:00
|
|
|
if (!want_cookie)
|
|
|
|
inet_csk_reqsk_queue_hash_add(sk, req, TCP_TIMEOUT_INIT);
|
2016-04-14 08:05:39 +03:00
|
|
|
af_ops->send_synack(sk, dst, &fl, req, &foc,
|
|
|
|
!want_cookie ? TCP_SYNACK_NORMAL :
|
|
|
|
TCP_SYNACK_COOKIE);
|
2016-04-01 18:52:20 +03:00
|
|
|
if (want_cookie) {
|
|
|
|
reqsk_free(req);
|
|
|
|
return 0;
|
|
|
|
}
|
2014-06-25 18:10:02 +04:00
|
|
|
}
|
2015-10-02 21:43:35 +03:00
|
|
|
reqsk_put(req);
|
2014-06-25 18:10:02 +04:00
|
|
|
return 0;
|
|
|
|
|
|
|
|
drop_and_release:
|
|
|
|
dst_release(dst);
|
|
|
|
drop_and_free:
|
|
|
|
reqsk_free(req);
|
|
|
|
drop:
|
2016-04-01 18:52:20 +03:00
|
|
|
tcp_listendrop(sk);
|
2014-06-25 18:10:02 +04:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL(tcp_conn_request);
|