2019-05-27 09:55:01 +03:00
|
|
|
// SPDX-License-Identifier: GPL-2.0-or-later
|
2005-04-17 02:20:36 +04:00
|
|
|
/*
|
|
|
|
* net/sched/cls_u32.c Ugly (or Universal) 32bit key Packet Classifier.
|
|
|
|
*
|
|
|
|
* Authors: Alexey Kuznetsov, <kuznet@ms2.inr.ac.ru>
|
|
|
|
*
|
|
|
|
* The filters are packed to hash tables of key nodes
|
|
|
|
* with a set of 32bit key/mask pairs at every node.
|
|
|
|
* Nodes reference next level hash tables etc.
|
|
|
|
*
|
|
|
|
* This scheme is the best universal classifier I managed to
|
|
|
|
* invent; it is not super-fast, but it is not slow (provided you
|
|
|
|
* program it correctly), and general enough. And its relative
|
|
|
|
* speed grows as the number of rules becomes larger.
|
|
|
|
*
|
|
|
|
* It seems that it represents the best middle point between
|
|
|
|
* speed and manageability both by human and by machine.
|
|
|
|
*
|
|
|
|
* It is especially useful for link sharing combined with QoS;
|
|
|
|
* pure RSVP doesn't need such a general approach and can use
|
|
|
|
* much simpler (and faster) schemes, sort of cls_rsvp.c.
|
|
|
|
*
|
|
|
|
* nfmark match added by Catalin(ux aka Dino) BOIE <catab at umbrella.ro>
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/module.h>
|
include cleanup: Update gfp.h and slab.h includes to prepare for breaking implicit slab.h inclusion from percpu.h
percpu.h is included by sched.h and module.h and thus ends up being
included when building most .c files. percpu.h includes slab.h which
in turn includes gfp.h making everything defined by the two files
universally available and complicating inclusion dependencies.
percpu.h -> slab.h dependency is about to be removed. Prepare for
this change by updating users of gfp and slab facilities include those
headers directly instead of assuming availability. As this conversion
needs to touch large number of source files, the following script is
used as the basis of conversion.
http://userweb.kernel.org/~tj/misc/slabh-sweep.py
The script does the followings.
* Scan files for gfp and slab usages and update includes such that
only the necessary includes are there. ie. if only gfp is used,
gfp.h, if slab is used, slab.h.
* When the script inserts a new include, it looks at the include
blocks and try to put the new include such that its order conforms
to its surrounding. It's put in the include block which contains
core kernel includes, in the same order that the rest are ordered -
alphabetical, Christmas tree, rev-Xmas-tree or at the end if there
doesn't seem to be any matching order.
* If the script can't find a place to put a new include (mostly
because the file doesn't have fitting include block), it prints out
an error message indicating which .h file needs to be added to the
file.
The conversion was done in the following steps.
1. The initial automatic conversion of all .c files updated slightly
over 4000 files, deleting around 700 includes and adding ~480 gfp.h
and ~3000 slab.h inclusions. The script emitted errors for ~400
files.
2. Each error was manually checked. Some didn't need the inclusion,
some needed manual addition while adding it to implementation .h or
embedding .c file was more appropriate for others. This step added
inclusions to around 150 files.
3. The script was run again and the output was compared to the edits
from #2 to make sure no file was left behind.
4. Several build tests were done and a couple of problems were fixed.
e.g. lib/decompress_*.c used malloc/free() wrappers around slab
APIs requiring slab.h to be added manually.
5. The script was run on all .h files but without automatically
editing them as sprinkling gfp.h and slab.h inclusions around .h
files could easily lead to inclusion dependency hell. Most gfp.h
inclusion directives were ignored as stuff from gfp.h was usually
wildly available and often used in preprocessor macros. Each
slab.h inclusion directive was examined and added manually as
necessary.
6. percpu.h was updated not to include slab.h.
7. Build test were done on the following configurations and failures
were fixed. CONFIG_GCOV_KERNEL was turned off for all tests (as my
distributed build env didn't work with gcov compiles) and a few
more options had to be turned off depending on archs to make things
build (like ipr on powerpc/64 which failed due to missing writeq).
* x86 and x86_64 UP and SMP allmodconfig and a custom test config.
* powerpc and powerpc64 SMP allmodconfig
* sparc and sparc64 SMP allmodconfig
* ia64 SMP allmodconfig
* s390 SMP allmodconfig
* alpha SMP allmodconfig
* um on x86_64 SMP allmodconfig
8. percpu.h modifications were reverted so that it could be applied as
a separate patch and serve as bisection point.
Given the fact that I had only a couple of failures from tests on step
6, I'm fairly confident about the coverage of this conversion patch.
If there is a breakage, it's likely to be something in one of the arch
headers which should be easily discoverable easily on most builds of
the specific arch.
Signed-off-by: Tejun Heo <tj@kernel.org>
Guess-its-ok-by: Christoph Lameter <cl@linux-foundation.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Lee Schermerhorn <Lee.Schermerhorn@hp.com>
2010-03-24 11:04:11 +03:00
|
|
|
#include <linux/slab.h>
|
2005-04-17 02:20:36 +04:00
|
|
|
#include <linux/types.h>
|
|
|
|
#include <linux/kernel.h>
|
|
|
|
#include <linux/string.h>
|
|
|
|
#include <linux/errno.h>
|
2014-09-13 07:09:16 +04:00
|
|
|
#include <linux/percpu.h>
|
2005-04-17 02:20:36 +04:00
|
|
|
#include <linux/rtnetlink.h>
|
|
|
|
#include <linux/skbuff.h>
|
2014-07-18 04:34:53 +04:00
|
|
|
#include <linux/bitmap.h>
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
#include <linux/netdevice.h>
|
|
|
|
#include <linux/hash.h>
|
2007-07-03 09:49:07 +04:00
|
|
|
#include <net/netlink.h>
|
2005-04-17 02:20:36 +04:00
|
|
|
#include <net/act_api.h>
|
|
|
|
#include <net/pkt_cls.h>
|
2017-09-25 20:13:51 +03:00
|
|
|
#include <linux/idr.h>
|
2022-12-06 16:55:13 +03:00
|
|
|
#include <net/tc_wrapper.h>
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2011-01-19 22:26:56 +03:00
|
|
|
struct tc_u_knode {
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_knode __rcu *next;
|
2005-04-17 02:20:36 +04:00
|
|
|
u32 handle;
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_hnode __rcu *ht_up;
|
2005-04-17 02:20:36 +04:00
|
|
|
struct tcf_exts exts;
|
2014-01-10 04:14:02 +04:00
|
|
|
int ifindex;
|
2005-04-17 02:20:36 +04:00
|
|
|
u8 fshift;
|
|
|
|
struct tcf_result res;
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_hnode __rcu *ht_down;
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
2014-09-13 07:08:47 +04:00
|
|
|
struct tc_u32_pcnt __percpu *pf;
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
2016-02-26 18:54:39 +03:00
|
|
|
u32 flags;
|
2018-06-26 00:30:08 +03:00
|
|
|
unsigned int in_hw_count;
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_MARK
|
2014-09-13 07:08:47 +04:00
|
|
|
u32 val;
|
|
|
|
u32 mask;
|
|
|
|
u32 __percpu *pcpu_success;
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
2018-05-24 01:26:53 +03:00
|
|
|
struct rcu_work rwork;
|
2014-09-17 22:11:46 +04:00
|
|
|
/* The 'sel' field MUST be the last field in structure to allow for
|
|
|
|
* tc_u32_keys allocated at end of structure.
|
|
|
|
*/
|
|
|
|
struct tc_u32_sel sel;
|
2005-04-17 02:20:36 +04:00
|
|
|
};
|
|
|
|
|
2011-01-19 22:26:56 +03:00
|
|
|
struct tc_u_hnode {
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_hnode __rcu *next;
|
2005-04-17 02:20:36 +04:00
|
|
|
u32 handle;
|
|
|
|
u32 prio;
|
|
|
|
int refcnt;
|
2011-01-19 22:26:56 +03:00
|
|
|
unsigned int divisor;
|
2017-09-25 20:13:51 +03:00
|
|
|
struct idr handle_idr;
|
2018-10-08 13:22:33 +03:00
|
|
|
bool is_root;
|
2014-09-13 07:09:16 +04:00
|
|
|
struct rcu_head rcu;
|
2018-01-24 23:54:22 +03:00
|
|
|
u32 flags;
|
2015-03-10 03:03:40 +03:00
|
|
|
/* The 'ht' field MUST be the last field in structure to allow for
|
|
|
|
* more entries allocated at end of structure.
|
|
|
|
*/
|
net/sched: cls_u32: Replace one-element array with flexible-array member
There is a regular need in the kernel to provide a way to declare having
a dynamically sized set of trailing elements in a structure. Kernel code
should always use “flexible array members”[1] for these cases. The older
style of one-element or zero-length arrays should no longer be used[2].
Refactor the code according to the use of a flexible-array member in
struct tc_u_hnode and use the struct_size() helper to calculate the
size for the allocations. Commit 5778d39d070b ("net_sched: fix struct
tc_u_hnode layout in u32") makes it clear that the code is expected to
dynamically allocate divisor + 1 entries for ->ht[] in tc_uhnode. Also,
based on other observations, as the piece of code below:
1232 for (h = 0; h <= ht->divisor; h++) {
1233 for (n = rtnl_dereference(ht->ht[h]);
1234 n;
1235 n = rtnl_dereference(n->next)) {
1236 if (tc_skip_hw(n->flags))
1237 continue;
1238
1239 err = u32_reoffload_knode(tp, n, add, cb,
1240 cb_priv, extack);
1241 if (err)
1242 return err;
1243 }
1244 }
we can assume that, in general, the code is actually expecting to allocate
that extra space for the one-element array in tc_uhnode, everytime it
allocates memory for instances of tc_uhnode or tc_u_common structures.
That's the reason for passing '1' as the last argument for struct_size()
in the allocation for _root_ht_ and _tp_c_, and 'divisor + 1' in the
allocation code for _ht_.
[1] https://en.wikipedia.org/wiki/Flexible_array_member
[2] https://www.kernel.org/doc/html/v5.9-rc1/process/deprecated.html#zero-length-and-one-element-arrays
Tested-by: kernel test robot <lkp@intel.com>
Link: https://lore.kernel.org/lkml/5f7062af.z3T9tn9yIPv6h5Ny%25lkp@intel.com/
Signed-off-by: Gustavo A. R. Silva <gustavoars@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2020-09-28 18:30:52 +03:00
|
|
|
struct tc_u_knode __rcu *ht[];
|
2005-04-17 02:20:36 +04:00
|
|
|
};
|
|
|
|
|
2011-01-19 22:26:56 +03:00
|
|
|
struct tc_u_common {
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_hnode __rcu *hlist;
|
2018-02-13 14:00:17 +03:00
|
|
|
void *ptr;
|
2005-04-17 02:20:36 +04:00
|
|
|
int refcnt;
|
2017-09-25 20:13:51 +03:00
|
|
|
struct idr handle_idr;
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
struct hlist_node hnode;
|
2018-10-08 13:22:43 +03:00
|
|
|
long knodes;
|
2005-04-17 02:20:36 +04:00
|
|
|
};
|
|
|
|
|
2011-01-19 22:26:56 +03:00
|
|
|
static inline unsigned int u32_hash_fold(__be32 key,
|
|
|
|
const struct tc_u32_sel *sel,
|
|
|
|
u8 fshift)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2011-01-19 22:26:56 +03:00
|
|
|
unsigned int h = ntohl(key & sel->hmask) >> fshift;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
return h;
|
|
|
|
}
|
|
|
|
|
2022-12-06 16:55:13 +03:00
|
|
|
TC_INDIRECT_SCOPE int u32_classify(struct sk_buff *skb,
|
|
|
|
const struct tcf_proto *tp,
|
|
|
|
struct tcf_result *res)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct {
|
|
|
|
struct tc_u_knode *knode;
|
2010-06-02 18:32:42 +04:00
|
|
|
unsigned int off;
|
2005-04-17 02:20:36 +04:00
|
|
|
} stack[TC_U32_MAXDEPTH];
|
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_hnode *ht = rcu_dereference_bh(tp->root);
|
2010-06-02 18:32:42 +04:00
|
|
|
unsigned int off = skb_network_offset(skb);
|
2005-04-17 02:20:36 +04:00
|
|
|
struct tc_u_knode *n;
|
|
|
|
int sdepth = 0;
|
|
|
|
int off2 = 0;
|
|
|
|
int sel = 0;
|
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
|
|
|
int j;
|
|
|
|
#endif
|
|
|
|
int i, r;
|
|
|
|
|
|
|
|
next_ht:
|
2014-09-13 07:09:16 +04:00
|
|
|
n = rcu_dereference_bh(ht->ht[sel]);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
next_knode:
|
|
|
|
if (n) {
|
|
|
|
struct tc_u32_key *key = n->sel.keys;
|
|
|
|
|
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
2014-09-13 07:08:47 +04:00
|
|
|
__this_cpu_inc(n->pf->rcnt);
|
2005-04-17 02:20:36 +04:00
|
|
|
j = 0;
|
|
|
|
#endif
|
|
|
|
|
2016-05-13 03:08:23 +03:00
|
|
|
if (tc_skip_sw(n->flags)) {
|
|
|
|
n = rcu_dereference_bh(n->next);
|
|
|
|
goto next_knode;
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_MARK
|
2014-09-13 07:08:47 +04:00
|
|
|
if ((skb->mark & n->mask) != n->val) {
|
2014-09-13 07:09:16 +04:00
|
|
|
n = rcu_dereference_bh(n->next);
|
2005-04-17 02:20:36 +04:00
|
|
|
goto next_knode;
|
|
|
|
} else {
|
2014-09-13 07:08:47 +04:00
|
|
|
__this_cpu_inc(*n->pcpu_success);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2011-01-19 22:26:56 +03:00
|
|
|
for (i = n->sel.nkeys; i > 0; i--, key++) {
|
2010-08-02 17:44:13 +04:00
|
|
|
int toff = off + key->off + (off2 & key->offmask);
|
2011-02-20 19:14:23 +03:00
|
|
|
__be32 *data, hdata;
|
2010-06-02 18:32:42 +04:00
|
|
|
|
2010-10-04 06:28:36 +04:00
|
|
|
if (skb_headroom(skb) + toff > INT_MAX)
|
2010-08-02 17:44:13 +04:00
|
|
|
goto out;
|
|
|
|
|
2011-02-20 19:14:23 +03:00
|
|
|
data = skb_header_pointer(skb, toff, 4, &hdata);
|
2010-06-02 18:32:42 +04:00
|
|
|
if (!data)
|
|
|
|
goto out;
|
|
|
|
if ((*data ^ key->val) & key->mask) {
|
2014-09-13 07:09:16 +04:00
|
|
|
n = rcu_dereference_bh(n->next);
|
2005-04-17 02:20:36 +04:00
|
|
|
goto next_knode;
|
|
|
|
}
|
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
2014-09-13 07:08:47 +04:00
|
|
|
__this_cpu_inc(n->pf->kcnts[j]);
|
2005-04-17 02:20:36 +04:00
|
|
|
j++;
|
|
|
|
#endif
|
|
|
|
}
|
2014-09-13 07:09:16 +04:00
|
|
|
|
|
|
|
ht = rcu_dereference_bh(n->ht_down);
|
|
|
|
if (!ht) {
|
2005-04-17 02:20:36 +04:00
|
|
|
check_terminal:
|
2011-01-19 22:26:56 +03:00
|
|
|
if (n->sel.flags & TC_U32_TERMINAL) {
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
*res = n->res;
|
2014-01-10 04:14:02 +04:00
|
|
|
if (!tcf_match_indev(skb, n->ifindex)) {
|
2014-09-13 07:09:16 +04:00
|
|
|
n = rcu_dereference_bh(n->next);
|
2005-04-17 02:20:36 +04:00
|
|
|
goto next_knode;
|
|
|
|
}
|
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
2014-09-13 07:08:47 +04:00
|
|
|
__this_cpu_inc(n->pf->rhit);
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
|
|
|
r = tcf_exts_exec(skb, &n->exts, res);
|
|
|
|
if (r < 0) {
|
2014-09-13 07:09:16 +04:00
|
|
|
n = rcu_dereference_bh(n->next);
|
2005-04-17 02:20:36 +04:00
|
|
|
goto next_knode;
|
|
|
|
}
|
|
|
|
|
|
|
|
return r;
|
|
|
|
}
|
2014-09-13 07:09:16 +04:00
|
|
|
n = rcu_dereference_bh(n->next);
|
2005-04-17 02:20:36 +04:00
|
|
|
goto next_knode;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* PUSH */
|
|
|
|
if (sdepth >= TC_U32_MAXDEPTH)
|
|
|
|
goto deadloop;
|
|
|
|
stack[sdepth].knode = n;
|
2010-06-02 18:32:42 +04:00
|
|
|
stack[sdepth].off = off;
|
2005-04-17 02:20:36 +04:00
|
|
|
sdepth++;
|
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
ht = rcu_dereference_bh(n->ht_down);
|
2005-04-17 02:20:36 +04:00
|
|
|
sel = 0;
|
2010-06-02 18:32:42 +04:00
|
|
|
if (ht->divisor) {
|
2011-02-20 19:14:23 +03:00
|
|
|
__be32 *data, hdata;
|
2010-06-02 18:32:42 +04:00
|
|
|
|
|
|
|
data = skb_header_pointer(skb, off + n->sel.hoff, 4,
|
2011-02-20 19:14:23 +03:00
|
|
|
&hdata);
|
2010-06-02 18:32:42 +04:00
|
|
|
if (!data)
|
|
|
|
goto out;
|
|
|
|
sel = ht->divisor & u32_hash_fold(*data, &n->sel,
|
|
|
|
n->fshift);
|
|
|
|
}
|
2011-01-19 22:26:56 +03:00
|
|
|
if (!(n->sel.flags & (TC_U32_VAROFFSET | TC_U32_OFFSET | TC_U32_EAT)))
|
2005-04-17 02:20:36 +04:00
|
|
|
goto next_ht;
|
|
|
|
|
2011-01-19 22:26:56 +03:00
|
|
|
if (n->sel.flags & (TC_U32_OFFSET | TC_U32_VAROFFSET)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
off2 = n->sel.off + 3;
|
2010-06-02 18:32:42 +04:00
|
|
|
if (n->sel.flags & TC_U32_VAROFFSET) {
|
2011-02-20 19:14:23 +03:00
|
|
|
__be16 *data, hdata;
|
2010-06-02 18:32:42 +04:00
|
|
|
|
|
|
|
data = skb_header_pointer(skb,
|
|
|
|
off + n->sel.offoff,
|
2011-02-20 19:14:23 +03:00
|
|
|
2, &hdata);
|
2010-06-02 18:32:42 +04:00
|
|
|
if (!data)
|
|
|
|
goto out;
|
|
|
|
off2 += ntohs(n->sel.offmask & *data) >>
|
|
|
|
n->sel.offshift;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
off2 &= ~3;
|
|
|
|
}
|
2011-01-19 22:26:56 +03:00
|
|
|
if (n->sel.flags & TC_U32_EAT) {
|
2010-06-02 18:32:42 +04:00
|
|
|
off += off2;
|
2005-04-17 02:20:36 +04:00
|
|
|
off2 = 0;
|
|
|
|
}
|
|
|
|
|
2010-06-02 18:32:42 +04:00
|
|
|
if (off < skb->len)
|
2005-04-17 02:20:36 +04:00
|
|
|
goto next_ht;
|
|
|
|
}
|
|
|
|
|
|
|
|
/* POP */
|
|
|
|
if (sdepth--) {
|
|
|
|
n = stack[sdepth].knode;
|
2014-09-13 07:09:16 +04:00
|
|
|
ht = rcu_dereference_bh(n->ht_up);
|
2010-06-02 18:32:42 +04:00
|
|
|
off = stack[sdepth].off;
|
2005-04-17 02:20:36 +04:00
|
|
|
goto check_terminal;
|
|
|
|
}
|
2010-06-02 18:32:42 +04:00
|
|
|
out:
|
2005-04-17 02:20:36 +04:00
|
|
|
return -1;
|
|
|
|
|
|
|
|
deadloop:
|
2012-05-14 01:56:26 +04:00
|
|
|
net_warn_ratelimited("cls_u32: dead loop\n");
|
2005-04-17 02:20:36 +04:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2016-09-18 15:45:33 +03:00
|
|
|
static struct tc_u_hnode *u32_lookup_ht(struct tc_u_common *tp_c, u32 handle)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tc_u_hnode *ht;
|
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
for (ht = rtnl_dereference(tp_c->hlist);
|
|
|
|
ht;
|
|
|
|
ht = rtnl_dereference(ht->next))
|
2005-04-17 02:20:36 +04:00
|
|
|
if (ht->handle == handle)
|
|
|
|
break;
|
|
|
|
|
|
|
|
return ht;
|
|
|
|
}
|
|
|
|
|
2016-09-18 15:45:33 +03:00
|
|
|
static struct tc_u_knode *u32_lookup_key(struct tc_u_hnode *ht, u32 handle)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2011-01-19 22:26:56 +03:00
|
|
|
unsigned int sel;
|
2005-04-17 02:20:36 +04:00
|
|
|
struct tc_u_knode *n = NULL;
|
|
|
|
|
|
|
|
sel = TC_U32_HASH(handle);
|
|
|
|
if (sel > ht->divisor)
|
|
|
|
goto out;
|
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
for (n = rtnl_dereference(ht->ht[sel]);
|
|
|
|
n;
|
|
|
|
n = rtnl_dereference(n->next))
|
2005-04-17 02:20:36 +04:00
|
|
|
if (n->handle == handle)
|
|
|
|
break;
|
|
|
|
out:
|
|
|
|
return n;
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2017-08-05 07:31:43 +03:00
|
|
|
static void *u32_get(struct tcf_proto *tp, u32 handle)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tc_u_hnode *ht;
|
|
|
|
struct tc_u_common *tp_c = tp->data;
|
|
|
|
|
|
|
|
if (TC_U32_HTID(handle) == TC_U32_ROOT)
|
2014-09-13 07:09:16 +04:00
|
|
|
ht = rtnl_dereference(tp->root);
|
2005-04-17 02:20:36 +04:00
|
|
|
else
|
|
|
|
ht = u32_lookup_ht(tp_c, TC_U32_HTID(handle));
|
|
|
|
|
|
|
|
if (!ht)
|
2017-08-05 07:31:43 +03:00
|
|
|
return NULL;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (TC_U32_KEY(handle) == 0)
|
2017-08-05 07:31:43 +03:00
|
|
|
return ht;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2017-08-05 07:31:43 +03:00
|
|
|
return u32_lookup_key(ht, handle);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2017-11-28 20:05:54 +03:00
|
|
|
/* Protected by rtnl lock */
|
2017-09-25 20:13:51 +03:00
|
|
|
static u32 gen_new_htid(struct tc_u_common *tp_c, struct tc_u_hnode *ptr)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2017-11-28 20:05:54 +03:00
|
|
|
int id = idr_alloc_cyclic(&tp_c->handle_idr, ptr, 1, 0x7FF, GFP_KERNEL);
|
|
|
|
if (id < 0)
|
2017-09-25 20:13:51 +03:00
|
|
|
return 0;
|
2017-11-28 20:05:54 +03:00
|
|
|
return (id | 0x800U) << 20;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
static struct hlist_head *tc_u_common_hash;
|
|
|
|
|
|
|
|
#define U32_HASH_SHIFT 10
|
|
|
|
#define U32_HASH_SIZE (1 << U32_HASH_SHIFT)
|
|
|
|
|
2018-02-13 14:00:17 +03:00
|
|
|
static void *tc_u_common_ptr(const struct tcf_proto *tp)
|
|
|
|
{
|
|
|
|
struct tcf_block *block = tp->chain->block;
|
|
|
|
|
|
|
|
/* The block sharing is currently supported only
|
|
|
|
* for classless qdiscs. In that case we use block
|
|
|
|
* for tc_u_common identification. In case the
|
|
|
|
* block is not shared, block->q is a valid pointer
|
|
|
|
* and we can use that. That works for classful qdiscs.
|
|
|
|
*/
|
|
|
|
if (tcf_block_shared(block))
|
|
|
|
return block;
|
|
|
|
else
|
|
|
|
return block->q;
|
|
|
|
}
|
|
|
|
|
2018-10-08 13:22:39 +03:00
|
|
|
static struct hlist_head *tc_u_hash(void *key)
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
{
|
2018-10-08 13:22:39 +03:00
|
|
|
return tc_u_common_hash + hash_ptr(key, U32_HASH_SHIFT);
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
}
|
|
|
|
|
2018-10-08 13:22:39 +03:00
|
|
|
static struct tc_u_common *tc_u_common_find(void *key)
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
{
|
|
|
|
struct tc_u_common *tc;
|
2018-10-08 13:22:39 +03:00
|
|
|
hlist_for_each_entry(tc, tc_u_hash(key), hnode) {
|
|
|
|
if (tc->ptr == key)
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
return tc;
|
|
|
|
}
|
|
|
|
return NULL;
|
|
|
|
}
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
static int u32_init(struct tcf_proto *tp)
|
|
|
|
{
|
|
|
|
struct tc_u_hnode *root_ht;
|
2018-10-08 13:22:39 +03:00
|
|
|
void *key = tc_u_common_ptr(tp);
|
|
|
|
struct tc_u_common *tp_c = tc_u_common_find(key);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
net/sched: cls_u32: Replace one-element array with flexible-array member
There is a regular need in the kernel to provide a way to declare having
a dynamically sized set of trailing elements in a structure. Kernel code
should always use “flexible array members”[1] for these cases. The older
style of one-element or zero-length arrays should no longer be used[2].
Refactor the code according to the use of a flexible-array member in
struct tc_u_hnode and use the struct_size() helper to calculate the
size for the allocations. Commit 5778d39d070b ("net_sched: fix struct
tc_u_hnode layout in u32") makes it clear that the code is expected to
dynamically allocate divisor + 1 entries for ->ht[] in tc_uhnode. Also,
based on other observations, as the piece of code below:
1232 for (h = 0; h <= ht->divisor; h++) {
1233 for (n = rtnl_dereference(ht->ht[h]);
1234 n;
1235 n = rtnl_dereference(n->next)) {
1236 if (tc_skip_hw(n->flags))
1237 continue;
1238
1239 err = u32_reoffload_knode(tp, n, add, cb,
1240 cb_priv, extack);
1241 if (err)
1242 return err;
1243 }
1244 }
we can assume that, in general, the code is actually expecting to allocate
that extra space for the one-element array in tc_uhnode, everytime it
allocates memory for instances of tc_uhnode or tc_u_common structures.
That's the reason for passing '1' as the last argument for struct_size()
in the allocation for _root_ht_ and _tp_c_, and 'divisor + 1' in the
allocation code for _ht_.
[1] https://en.wikipedia.org/wiki/Flexible_array_member
[2] https://www.kernel.org/doc/html/v5.9-rc1/process/deprecated.html#zero-length-and-one-element-arrays
Tested-by: kernel test robot <lkp@intel.com>
Link: https://lore.kernel.org/lkml/5f7062af.z3T9tn9yIPv6h5Ny%25lkp@intel.com/
Signed-off-by: Gustavo A. R. Silva <gustavoars@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2020-09-28 18:30:52 +03:00
|
|
|
root_ht = kzalloc(struct_size(root_ht, ht, 1), GFP_KERNEL);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (root_ht == NULL)
|
|
|
|
return -ENOBUFS;
|
|
|
|
|
|
|
|
root_ht->refcnt++;
|
2017-09-25 20:13:51 +03:00
|
|
|
root_ht->handle = tp_c ? gen_new_htid(tp_c, root_ht) : 0x80000000;
|
2005-04-17 02:20:36 +04:00
|
|
|
root_ht->prio = tp->prio;
|
2018-10-08 13:22:33 +03:00
|
|
|
root_ht->is_root = true;
|
2017-09-25 20:13:51 +03:00
|
|
|
idr_init(&root_ht->handle_idr);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (tp_c == NULL) {
|
net/sched: cls_u32: Replace one-element array with flexible-array member
There is a regular need in the kernel to provide a way to declare having
a dynamically sized set of trailing elements in a structure. Kernel code
should always use “flexible array members”[1] for these cases. The older
style of one-element or zero-length arrays should no longer be used[2].
Refactor the code according to the use of a flexible-array member in
struct tc_u_hnode and use the struct_size() helper to calculate the
size for the allocations. Commit 5778d39d070b ("net_sched: fix struct
tc_u_hnode layout in u32") makes it clear that the code is expected to
dynamically allocate divisor + 1 entries for ->ht[] in tc_uhnode. Also,
based on other observations, as the piece of code below:
1232 for (h = 0; h <= ht->divisor; h++) {
1233 for (n = rtnl_dereference(ht->ht[h]);
1234 n;
1235 n = rtnl_dereference(n->next)) {
1236 if (tc_skip_hw(n->flags))
1237 continue;
1238
1239 err = u32_reoffload_knode(tp, n, add, cb,
1240 cb_priv, extack);
1241 if (err)
1242 return err;
1243 }
1244 }
we can assume that, in general, the code is actually expecting to allocate
that extra space for the one-element array in tc_uhnode, everytime it
allocates memory for instances of tc_uhnode or tc_u_common structures.
That's the reason for passing '1' as the last argument for struct_size()
in the allocation for _root_ht_ and _tp_c_, and 'divisor + 1' in the
allocation code for _ht_.
[1] https://en.wikipedia.org/wiki/Flexible_array_member
[2] https://www.kernel.org/doc/html/v5.9-rc1/process/deprecated.html#zero-length-and-one-element-arrays
Tested-by: kernel test robot <lkp@intel.com>
Link: https://lore.kernel.org/lkml/5f7062af.z3T9tn9yIPv6h5Ny%25lkp@intel.com/
Signed-off-by: Gustavo A. R. Silva <gustavoars@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2020-09-28 18:30:52 +03:00
|
|
|
tp_c = kzalloc(struct_size(tp_c, hlist->ht, 1), GFP_KERNEL);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (tp_c == NULL) {
|
|
|
|
kfree(root_ht);
|
|
|
|
return -ENOBUFS;
|
|
|
|
}
|
2018-10-08 13:22:39 +03:00
|
|
|
tp_c->ptr = key;
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
INIT_HLIST_NODE(&tp_c->hnode);
|
2017-09-25 20:13:51 +03:00
|
|
|
idr_init(&tp_c->handle_idr);
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
|
2018-10-08 13:22:39 +03:00
|
|
|
hlist_add_head(&tp_c->hnode, tc_u_hash(key));
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
tp_c->refcnt++;
|
2014-09-13 07:09:16 +04:00
|
|
|
RCU_INIT_POINTER(root_ht->next, tp_c->hlist);
|
|
|
|
rcu_assign_pointer(tp_c->hlist, root_ht);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
net: sched: cls_u32: fix hnode refcounting
cls_u32.c misuses refcounts for struct tc_u_hnode - it counts references
via ->hlist and via ->tp_root together. u32_destroy() drops the former
and, in case when there had been links, leaves the sucker on the list.
As the result, there's nothing to protect it from getting freed once links
are dropped.
That also makes the "is it busy" check incapable of catching the root
hnode - it *is* busy (there's a reference from tp), but we don't see it as
something separate. "Is it our root?" check partially covers that, but
the problem exists for others' roots as well.
AFAICS, the minimal fix preserving the existing behaviour (where it doesn't
include oopsen, that is) would be this:
* count tp->root and tp_c->hlist as separate references. I.e.
have u32_init() set refcount to 2, not 1.
* in u32_destroy() we always drop the former;
in u32_destroy_hnode() - the latter.
That way we have *all* references contributing to refcount. List
removal happens in u32_destroy_hnode() (called only when ->refcnt is 1)
an in u32_destroy() in case of tc_u_common going away, along with
everything reachable from it. IOW, that way we know that
u32_destroy_key() won't free something still on the list (or pointed to by
someone's ->root).
Reproducer:
tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: protocol ip prio 100 handle 1: \
u32 divisor 1
tc filter add dev eth0 parent ffff: protocol ip prio 200 handle 2: \
u32 divisor 1
tc filter add dev eth0 parent ffff: protocol ip prio 100 \
handle 1:0:11 u32 ht 1: link 801: offset at 0 mask 0f00 shift 6 \
plus 0 eat match ip protocol 6 ff
tc filter delete dev eth0 parent ffff: protocol ip prio 200
tc filter change dev eth0 parent ffff: protocol ip prio 100 \
handle 1:0:11 u32 ht 1: link 0: offset at 0 mask 0f00 shift 6 plus 0 \
eat match ip protocol 6 ff
tc filter delete dev eth0 parent ffff: protocol ip prio 100
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-10-07 14:40:17 +03:00
|
|
|
root_ht->refcnt++;
|
2014-09-13 07:09:16 +04:00
|
|
|
rcu_assign_pointer(tp->root, root_ht);
|
2005-04-17 02:20:36 +04:00
|
|
|
tp->data = tp_c;
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2022-04-13 20:35:41 +03:00
|
|
|
static void __u32_destroy_key(struct tc_u_knode *n)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2018-02-06 00:23:01 +03:00
|
|
|
struct tc_u_hnode *ht = rtnl_dereference(n->ht_down);
|
|
|
|
|
2014-09-25 21:26:37 +04:00
|
|
|
tcf_exts_destroy(&n->exts);
|
2018-02-06 00:23:01 +03:00
|
|
|
if (ht && --ht->refcnt == 0)
|
|
|
|
kfree(ht);
|
2022-04-13 20:35:41 +03:00
|
|
|
kfree(n);
|
|
|
|
}
|
|
|
|
|
|
|
|
static void u32_destroy_key(struct tc_u_knode *n, bool free_pf)
|
|
|
|
{
|
|
|
|
tcf_exts_put_net(&n->exts);
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
2014-09-20 08:50:34 +04:00
|
|
|
if (free_pf)
|
|
|
|
free_percpu(n->pf);
|
2014-09-20 08:50:04 +04:00
|
|
|
#endif
|
|
|
|
#ifdef CONFIG_CLS_U32_MARK
|
2014-09-20 08:50:34 +04:00
|
|
|
if (free_pf)
|
|
|
|
free_percpu(n->pcpu_success);
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
2022-04-13 20:35:41 +03:00
|
|
|
__u32_destroy_key(n);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2014-09-20 08:50:34 +04:00
|
|
|
/* u32_delete_key_rcu should be called when free'ing a copied
|
|
|
|
* version of a tc_u_knode obtained from u32_init_knode(). When
|
|
|
|
* copies are obtained from u32_init_knode() the statistics are
|
|
|
|
* shared between the old and new copies to allow readers to
|
|
|
|
* continue to update the statistics during the copy. To support
|
|
|
|
* this the u32_delete_key_rcu variant does not free the percpu
|
|
|
|
* statistics.
|
|
|
|
*/
|
2017-10-27 04:24:36 +03:00
|
|
|
static void u32_delete_key_work(struct work_struct *work)
|
|
|
|
{
|
2018-05-24 01:26:53 +03:00
|
|
|
struct tc_u_knode *key = container_of(to_rcu_work(work),
|
|
|
|
struct tc_u_knode,
|
|
|
|
rwork);
|
2017-10-27 04:24:36 +03:00
|
|
|
rtnl_lock();
|
2018-10-08 13:22:36 +03:00
|
|
|
u32_destroy_key(key, false);
|
2017-10-27 04:24:36 +03:00
|
|
|
rtnl_unlock();
|
|
|
|
}
|
|
|
|
|
2014-09-20 08:50:34 +04:00
|
|
|
/* u32_delete_key_freepf_rcu is the rcu callback variant
|
|
|
|
* that free's the entire structure including the statistics
|
|
|
|
* percpu variables. Only use this if the key is not a copy
|
|
|
|
* returned by u32_init_knode(). See u32_delete_key_rcu()
|
|
|
|
* for the variant that should be used with keys return from
|
|
|
|
* u32_init_knode()
|
|
|
|
*/
|
2017-10-27 04:24:36 +03:00
|
|
|
static void u32_delete_key_freepf_work(struct work_struct *work)
|
|
|
|
{
|
2018-05-24 01:26:53 +03:00
|
|
|
struct tc_u_knode *key = container_of(to_rcu_work(work),
|
|
|
|
struct tc_u_knode,
|
|
|
|
rwork);
|
2017-10-27 04:24:36 +03:00
|
|
|
rtnl_lock();
|
2018-10-08 13:22:36 +03:00
|
|
|
u32_destroy_key(key, true);
|
2017-10-27 04:24:36 +03:00
|
|
|
rtnl_unlock();
|
|
|
|
}
|
|
|
|
|
2013-12-10 16:55:31 +04:00
|
|
|
static int u32_delete_key(struct tcf_proto *tp, struct tc_u_knode *key)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2018-10-08 13:22:43 +03:00
|
|
|
struct tc_u_common *tp_c = tp->data;
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_knode __rcu **kp;
|
|
|
|
struct tc_u_knode *pkp;
|
2014-09-16 10:30:49 +04:00
|
|
|
struct tc_u_hnode *ht = rtnl_dereference(key->ht_up);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (ht) {
|
2014-09-13 07:09:16 +04:00
|
|
|
kp = &ht->ht[TC_U32_HASH(key->handle)];
|
|
|
|
for (pkp = rtnl_dereference(*kp); pkp;
|
|
|
|
kp = &pkp->next, pkp = rtnl_dereference(*kp)) {
|
|
|
|
if (pkp == key) {
|
|
|
|
RCU_INIT_POINTER(*kp, key->next);
|
2018-10-08 13:22:43 +03:00
|
|
|
tp_c->knodes--;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2014-10-01 03:07:24 +04:00
|
|
|
tcf_unbind_filter(tp, &key->res);
|
2018-04-07 03:19:41 +03:00
|
|
|
idr_remove(&ht->handle_idr, key->handle);
|
2017-11-07 00:47:30 +03:00
|
|
|
tcf_exts_get_net(&key->exts);
|
2018-05-24 01:26:53 +03:00
|
|
|
tcf_queue_work(&key->rwork, u32_delete_key_freepf_work);
|
2005-04-17 02:20:36 +04:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
2008-07-26 08:43:18 +04:00
|
|
|
WARN_ON(1);
|
2005-04-17 02:20:36 +04:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2018-01-24 23:54:23 +03:00
|
|
|
static void u32_clear_hw_hnode(struct tcf_proto *tp, struct tc_u_hnode *h,
|
|
|
|
struct netlink_ext_ack *extack)
|
2016-02-17 08:17:09 +03:00
|
|
|
{
|
2017-10-19 16:50:35 +03:00
|
|
|
struct tcf_block *block = tp->chain->block;
|
2017-08-07 11:15:32 +03:00
|
|
|
struct tc_cls_u32_offload cls_u32 = {};
|
2016-02-17 08:17:09 +03:00
|
|
|
|
2019-05-07 03:24:21 +03:00
|
|
|
tc_cls_common_offload_init(&cls_u32.common, tp, h->flags, extack);
|
2017-10-19 16:50:34 +03:00
|
|
|
cls_u32.command = TC_CLSU32_DELETE_HNODE;
|
|
|
|
cls_u32.hnode.divisor = h->divisor;
|
|
|
|
cls_u32.hnode.handle = h->handle;
|
|
|
|
cls_u32.hnode.prio = h->prio;
|
2017-08-07 11:15:32 +03:00
|
|
|
|
2019-08-26 16:44:59 +03:00
|
|
|
tc_setup_cb_call(block, TC_SETUP_CLSU32, &cls_u32, false, true);
|
2016-02-17 08:17:09 +03:00
|
|
|
}
|
|
|
|
|
2016-09-18 15:45:33 +03:00
|
|
|
static int u32_replace_hw_hnode(struct tcf_proto *tp, struct tc_u_hnode *h,
|
2018-01-20 04:44:45 +03:00
|
|
|
u32 flags, struct netlink_ext_ack *extack)
|
2016-02-17 08:17:09 +03:00
|
|
|
{
|
2017-10-19 16:50:35 +03:00
|
|
|
struct tcf_block *block = tp->chain->block;
|
2017-08-07 11:15:32 +03:00
|
|
|
struct tc_cls_u32_offload cls_u32 = {};
|
2017-10-19 16:50:35 +03:00
|
|
|
bool skip_sw = tc_skip_sw(flags);
|
|
|
|
bool offloaded = false;
|
2016-05-13 03:08:23 +03:00
|
|
|
int err;
|
2016-02-17 08:17:09 +03:00
|
|
|
|
2019-05-07 03:24:21 +03:00
|
|
|
tc_cls_common_offload_init(&cls_u32.common, tp, flags, extack);
|
2017-08-07 11:15:32 +03:00
|
|
|
cls_u32.command = TC_CLSU32_NEW_HNODE;
|
|
|
|
cls_u32.hnode.divisor = h->divisor;
|
|
|
|
cls_u32.hnode.handle = h->handle;
|
|
|
|
cls_u32.hnode.prio = h->prio;
|
2016-02-17 08:17:09 +03:00
|
|
|
|
2019-08-26 16:44:59 +03:00
|
|
|
err = tc_setup_cb_call(block, TC_SETUP_CLSU32, &cls_u32, skip_sw, true);
|
2017-10-19 16:50:35 +03:00
|
|
|
if (err < 0) {
|
2018-01-24 23:54:23 +03:00
|
|
|
u32_clear_hw_hnode(tp, h, NULL);
|
2016-06-06 18:16:48 +03:00
|
|
|
return err;
|
2017-10-19 16:50:35 +03:00
|
|
|
} else if (err > 0) {
|
|
|
|
offloaded = true;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (skip_sw && !offloaded)
|
|
|
|
return -EINVAL;
|
2016-05-13 03:08:23 +03:00
|
|
|
|
|
|
|
return 0;
|
2016-02-17 08:17:09 +03:00
|
|
|
}
|
|
|
|
|
2018-01-24 23:54:23 +03:00
|
|
|
static void u32_remove_hw_knode(struct tcf_proto *tp, struct tc_u_knode *n,
|
|
|
|
struct netlink_ext_ack *extack)
|
2016-02-17 08:17:09 +03:00
|
|
|
{
|
2017-10-19 16:50:35 +03:00
|
|
|
struct tcf_block *block = tp->chain->block;
|
2017-08-07 11:15:32 +03:00
|
|
|
struct tc_cls_u32_offload cls_u32 = {};
|
2016-02-17 08:17:09 +03:00
|
|
|
|
2019-05-07 03:24:21 +03:00
|
|
|
tc_cls_common_offload_init(&cls_u32.common, tp, n->flags, extack);
|
2017-10-19 16:50:34 +03:00
|
|
|
cls_u32.command = TC_CLSU32_DELETE_KNODE;
|
2018-01-17 13:46:50 +03:00
|
|
|
cls_u32.knode.handle = n->handle;
|
2016-02-17 08:17:09 +03:00
|
|
|
|
2019-08-26 16:44:59 +03:00
|
|
|
tc_setup_cb_destroy(block, tp, TC_SETUP_CLSU32, &cls_u32, false,
|
|
|
|
&n->flags, &n->in_hw_count, true);
|
2016-02-17 08:17:09 +03:00
|
|
|
}
|
|
|
|
|
2016-09-18 15:45:33 +03:00
|
|
|
static int u32_replace_hw_knode(struct tcf_proto *tp, struct tc_u_knode *n,
|
2018-01-20 04:44:45 +03:00
|
|
|
u32 flags, struct netlink_ext_ack *extack)
|
2016-02-17 08:17:09 +03:00
|
|
|
{
|
2018-02-02 18:02:22 +03:00
|
|
|
struct tc_u_hnode *ht = rtnl_dereference(n->ht_down);
|
2017-10-19 16:50:35 +03:00
|
|
|
struct tcf_block *block = tp->chain->block;
|
2017-08-07 11:15:32 +03:00
|
|
|
struct tc_cls_u32_offload cls_u32 = {};
|
2017-10-19 16:50:35 +03:00
|
|
|
bool skip_sw = tc_skip_sw(flags);
|
2016-05-13 03:08:23 +03:00
|
|
|
int err;
|
2016-02-17 08:17:09 +03:00
|
|
|
|
2019-05-07 03:24:21 +03:00
|
|
|
tc_cls_common_offload_init(&cls_u32.common, tp, flags, extack);
|
2017-08-07 11:15:32 +03:00
|
|
|
cls_u32.command = TC_CLSU32_REPLACE_KNODE;
|
|
|
|
cls_u32.knode.handle = n->handle;
|
|
|
|
cls_u32.knode.fshift = n->fshift;
|
2016-02-17 08:17:09 +03:00
|
|
|
#ifdef CONFIG_CLS_U32_MARK
|
2017-08-07 11:15:32 +03:00
|
|
|
cls_u32.knode.val = n->val;
|
|
|
|
cls_u32.knode.mask = n->mask;
|
2016-02-17 08:17:09 +03:00
|
|
|
#else
|
2017-08-07 11:15:32 +03:00
|
|
|
cls_u32.knode.val = 0;
|
|
|
|
cls_u32.knode.mask = 0;
|
2016-02-17 08:17:09 +03:00
|
|
|
#endif
|
2017-08-07 11:15:32 +03:00
|
|
|
cls_u32.knode.sel = &n->sel;
|
2018-11-20 02:21:46 +03:00
|
|
|
cls_u32.knode.res = &n->res;
|
2017-08-07 11:15:32 +03:00
|
|
|
cls_u32.knode.exts = &n->exts;
|
2016-06-08 22:11:04 +03:00
|
|
|
if (n->ht_down)
|
2018-02-02 18:02:22 +03:00
|
|
|
cls_u32.knode.link_handle = ht->handle;
|
2016-06-08 22:11:04 +03:00
|
|
|
|
2019-08-26 16:44:59 +03:00
|
|
|
err = tc_setup_cb_add(block, tp, TC_SETUP_CLSU32, &cls_u32, skip_sw,
|
|
|
|
&n->flags, &n->in_hw_count, true);
|
|
|
|
if (err) {
|
2018-01-24 23:54:23 +03:00
|
|
|
u32_remove_hw_knode(tp, n, NULL);
|
2016-06-08 22:11:04 +03:00
|
|
|
return err;
|
2017-10-19 16:50:35 +03:00
|
|
|
}
|
|
|
|
|
2017-11-03 11:09:45 +03:00
|
|
|
if (skip_sw && !(n->flags & TCA_CLS_FLAGS_IN_HW))
|
2017-10-19 16:50:35 +03:00
|
|
|
return -EINVAL;
|
2016-05-13 03:08:23 +03:00
|
|
|
|
|
|
|
return 0;
|
2016-02-17 08:17:09 +03:00
|
|
|
}
|
|
|
|
|
2018-01-24 23:54:23 +03:00
|
|
|
static void u32_clear_hnode(struct tcf_proto *tp, struct tc_u_hnode *ht,
|
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2018-10-08 13:22:43 +03:00
|
|
|
struct tc_u_common *tp_c = tp->data;
|
2005-04-17 02:20:36 +04:00
|
|
|
struct tc_u_knode *n;
|
2011-01-19 22:26:56 +03:00
|
|
|
unsigned int h;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2011-01-19 22:26:56 +03:00
|
|
|
for (h = 0; h <= ht->divisor; h++) {
|
2014-09-13 07:09:16 +04:00
|
|
|
while ((n = rtnl_dereference(ht->ht[h])) != NULL) {
|
|
|
|
RCU_INIT_POINTER(ht->ht[h],
|
|
|
|
rtnl_dereference(n->next));
|
2018-10-08 13:22:43 +03:00
|
|
|
tp_c->knodes--;
|
2014-10-01 03:07:24 +04:00
|
|
|
tcf_unbind_filter(tp, &n->res);
|
2018-01-24 23:54:23 +03:00
|
|
|
u32_remove_hw_knode(tp, n, extack);
|
2017-11-28 17:48:43 +03:00
|
|
|
idr_remove(&ht->handle_idr, n->handle);
|
2017-11-07 00:47:30 +03:00
|
|
|
if (tcf_exts_get_net(&n->exts))
|
2018-05-24 01:26:53 +03:00
|
|
|
tcf_queue_work(&n->rwork, u32_delete_key_freepf_work);
|
2017-11-07 00:47:30 +03:00
|
|
|
else
|
2018-10-08 13:22:36 +03:00
|
|
|
u32_destroy_key(n, true);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-01-24 23:54:23 +03:00
|
|
|
static int u32_destroy_hnode(struct tcf_proto *tp, struct tc_u_hnode *ht,
|
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tc_u_common *tp_c = tp->data;
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_hnode __rcu **hn;
|
|
|
|
struct tc_u_hnode *phn;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
net: sched: cls_u32: fix hnode refcounting
cls_u32.c misuses refcounts for struct tc_u_hnode - it counts references
via ->hlist and via ->tp_root together. u32_destroy() drops the former
and, in case when there had been links, leaves the sucker on the list.
As the result, there's nothing to protect it from getting freed once links
are dropped.
That also makes the "is it busy" check incapable of catching the root
hnode - it *is* busy (there's a reference from tp), but we don't see it as
something separate. "Is it our root?" check partially covers that, but
the problem exists for others' roots as well.
AFAICS, the minimal fix preserving the existing behaviour (where it doesn't
include oopsen, that is) would be this:
* count tp->root and tp_c->hlist as separate references. I.e.
have u32_init() set refcount to 2, not 1.
* in u32_destroy() we always drop the former;
in u32_destroy_hnode() - the latter.
That way we have *all* references contributing to refcount. List
removal happens in u32_destroy_hnode() (called only when ->refcnt is 1)
an in u32_destroy() in case of tc_u_common going away, along with
everything reachable from it. IOW, that way we know that
u32_destroy_key() won't free something still on the list (or pointed to by
someone's ->root).
Reproducer:
tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: protocol ip prio 100 handle 1: \
u32 divisor 1
tc filter add dev eth0 parent ffff: protocol ip prio 200 handle 2: \
u32 divisor 1
tc filter add dev eth0 parent ffff: protocol ip prio 100 \
handle 1:0:11 u32 ht 1: link 801: offset at 0 mask 0f00 shift 6 \
plus 0 eat match ip protocol 6 ff
tc filter delete dev eth0 parent ffff: protocol ip prio 200
tc filter change dev eth0 parent ffff: protocol ip prio 100 \
handle 1:0:11 u32 ht 1: link 0: offset at 0 mask 0f00 shift 6 plus 0 \
eat match ip protocol 6 ff
tc filter delete dev eth0 parent ffff: protocol ip prio 100
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-10-07 14:40:17 +03:00
|
|
|
WARN_ON(--ht->refcnt);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2018-01-24 23:54:23 +03:00
|
|
|
u32_clear_hnode(tp, ht, extack);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
hn = &tp_c->hlist;
|
|
|
|
for (phn = rtnl_dereference(*hn);
|
|
|
|
phn;
|
|
|
|
hn = &phn->next, phn = rtnl_dereference(*hn)) {
|
|
|
|
if (phn == ht) {
|
2018-01-24 23:54:23 +03:00
|
|
|
u32_clear_hw_hnode(tp, ht, extack);
|
2017-09-25 20:13:51 +03:00
|
|
|
idr_destroy(&ht->handle_idr);
|
2017-11-28 17:48:43 +03:00
|
|
|
idr_remove(&tp_c->handle_idr, ht->handle);
|
2014-09-13 07:09:16 +04:00
|
|
|
RCU_INIT_POINTER(*hn, ht->next);
|
|
|
|
kfree_rcu(ht, rcu);
|
2005-04-17 02:20:36 +04:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
return -ENOENT;
|
|
|
|
}
|
|
|
|
|
2019-02-11 11:55:45 +03:00
|
|
|
static void u32_destroy(struct tcf_proto *tp, bool rtnl_held,
|
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tc_u_common *tp_c = tp->data;
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_hnode *root_ht = rtnl_dereference(tp->root);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-07-26 08:43:18 +04:00
|
|
|
WARN_ON(root_ht == NULL);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
net: sched: cls_u32: fix hnode refcounting
cls_u32.c misuses refcounts for struct tc_u_hnode - it counts references
via ->hlist and via ->tp_root together. u32_destroy() drops the former
and, in case when there had been links, leaves the sucker on the list.
As the result, there's nothing to protect it from getting freed once links
are dropped.
That also makes the "is it busy" check incapable of catching the root
hnode - it *is* busy (there's a reference from tp), but we don't see it as
something separate. "Is it our root?" check partially covers that, but
the problem exists for others' roots as well.
AFAICS, the minimal fix preserving the existing behaviour (where it doesn't
include oopsen, that is) would be this:
* count tp->root and tp_c->hlist as separate references. I.e.
have u32_init() set refcount to 2, not 1.
* in u32_destroy() we always drop the former;
in u32_destroy_hnode() - the latter.
That way we have *all* references contributing to refcount. List
removal happens in u32_destroy_hnode() (called only when ->refcnt is 1)
an in u32_destroy() in case of tc_u_common going away, along with
everything reachable from it. IOW, that way we know that
u32_destroy_key() won't free something still on the list (or pointed to by
someone's ->root).
Reproducer:
tc qdisc add dev eth0 ingress
tc filter add dev eth0 parent ffff: protocol ip prio 100 handle 1: \
u32 divisor 1
tc filter add dev eth0 parent ffff: protocol ip prio 200 handle 2: \
u32 divisor 1
tc filter add dev eth0 parent ffff: protocol ip prio 100 \
handle 1:0:11 u32 ht 1: link 801: offset at 0 mask 0f00 shift 6 \
plus 0 eat match ip protocol 6 ff
tc filter delete dev eth0 parent ffff: protocol ip prio 200
tc filter change dev eth0 parent ffff: protocol ip prio 100 \
handle 1:0:11 u32 ht 1: link 0: offset at 0 mask 0f00 shift 6 plus 0 \
eat match ip protocol 6 ff
tc filter delete dev eth0 parent ffff: protocol ip prio 100
Signed-off-by: Al Viro <viro@zeniv.linux.org.uk>
Signed-off-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2018-10-07 14:40:17 +03:00
|
|
|
if (root_ht && --root_ht->refcnt == 1)
|
2018-01-24 23:54:23 +03:00
|
|
|
u32_destroy_hnode(tp, root_ht, extack);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (--tp_c->refcnt == 0) {
|
|
|
|
struct tc_u_hnode *ht;
|
|
|
|
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
hlist_del(&tp_c->hnode);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
while ((ht = rtnl_dereference(tp_c->hlist)) != NULL) {
|
2018-02-06 00:23:01 +03:00
|
|
|
u32_clear_hnode(tp, ht, extack);
|
2014-09-13 07:09:16 +04:00
|
|
|
RCU_INIT_POINTER(tp_c->hlist, ht->next);
|
2018-02-06 00:23:01 +03:00
|
|
|
|
|
|
|
/* u32_destroy_key() will later free ht for us, if it's
|
|
|
|
* still referenced by some knode
|
|
|
|
*/
|
|
|
|
if (--ht->refcnt == 0)
|
|
|
|
kfree_rcu(ht, rcu);
|
2007-04-21 04:09:22 +04:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2017-09-25 20:13:51 +03:00
|
|
|
idr_destroy(&tp_c->handle_idr);
|
2005-04-17 02:20:36 +04:00
|
|
|
kfree(tp_c);
|
|
|
|
}
|
|
|
|
|
|
|
|
tp->data = NULL;
|
|
|
|
}
|
|
|
|
|
2018-01-18 19:20:53 +03:00
|
|
|
static int u32_delete(struct tcf_proto *tp, void *arg, bool *last,
|
2019-02-11 11:55:45 +03:00
|
|
|
bool rtnl_held, struct netlink_ext_ack *extack)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2017-08-05 07:31:43 +03:00
|
|
|
struct tc_u_hnode *ht = arg;
|
2017-04-20 00:21:21 +03:00
|
|
|
struct tc_u_common *tp_c = tp->data;
|
|
|
|
int ret = 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2016-02-17 08:17:09 +03:00
|
|
|
if (TC_U32_KEY(ht->handle)) {
|
2018-01-24 23:54:23 +03:00
|
|
|
u32_remove_hw_knode(tp, (struct tc_u_knode *)ht, extack);
|
2017-04-20 00:21:21 +03:00
|
|
|
ret = u32_delete_key(tp, (struct tc_u_knode *)ht);
|
|
|
|
goto out;
|
2016-02-17 08:17:09 +03:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2018-10-08 13:22:33 +03:00
|
|
|
if (ht->is_root) {
|
2018-01-18 19:20:55 +03:00
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Not allowed to delete root node");
|
2005-04-17 02:20:36 +04:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-04-13 05:37:13 +04:00
|
|
|
if (ht->refcnt == 1) {
|
2018-01-24 23:54:23 +03:00
|
|
|
u32_destroy_hnode(tp, ht, extack);
|
2008-04-13 05:37:13 +04:00
|
|
|
} else {
|
2018-01-18 19:20:55 +03:00
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Can not delete in-use filter");
|
2008-04-13 05:37:13 +04:00
|
|
|
return -EBUSY;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2017-04-20 00:21:21 +03:00
|
|
|
out:
|
2018-10-08 13:22:44 +03:00
|
|
|
*last = tp_c->refcnt == 1 && tp_c->knodes == 0;
|
2017-04-20 00:21:21 +03:00
|
|
|
return ret;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2017-09-25 20:13:51 +03:00
|
|
|
static u32 gen_new_kid(struct tc_u_hnode *ht, u32 htid)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2017-11-28 21:45:02 +03:00
|
|
|
u32 index = htid | 0x800;
|
2017-09-25 20:13:51 +03:00
|
|
|
u32 max = htid | 0xFFF;
|
|
|
|
|
2017-11-28 21:45:02 +03:00
|
|
|
if (idr_alloc_u32(&ht->handle_idr, NULL, &index, max, GFP_KERNEL)) {
|
|
|
|
index = htid + 1;
|
|
|
|
if (idr_alloc_u32(&ht->handle_idr, NULL, &index, max,
|
|
|
|
GFP_KERNEL))
|
|
|
|
index = max;
|
2017-09-25 20:13:51 +03:00
|
|
|
}
|
2014-07-18 04:34:53 +04:00
|
|
|
|
2017-11-28 21:45:02 +03:00
|
|
|
return index;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2008-01-24 07:36:12 +03:00
|
|
|
static const struct nla_policy u32_policy[TCA_U32_MAX + 1] = {
|
|
|
|
[TCA_U32_CLASSID] = { .type = NLA_U32 },
|
|
|
|
[TCA_U32_HASH] = { .type = NLA_U32 },
|
|
|
|
[TCA_U32_LINK] = { .type = NLA_U32 },
|
|
|
|
[TCA_U32_DIVISOR] = { .type = NLA_U32 },
|
|
|
|
[TCA_U32_SEL] = { .len = sizeof(struct tc_u32_sel) },
|
|
|
|
[TCA_U32_INDEV] = { .type = NLA_STRING, .len = IFNAMSIZ },
|
|
|
|
[TCA_U32_MARK] = { .len = sizeof(struct tc_u32_mark) },
|
2016-02-26 18:54:39 +03:00
|
|
|
[TCA_U32_FLAGS] = { .type = NLA_U32 },
|
2008-01-24 07:36:12 +03:00
|
|
|
};
|
|
|
|
|
2013-01-14 09:15:39 +04:00
|
|
|
static int u32_set_parms(struct net *net, struct tcf_proto *tp,
|
2018-10-08 13:22:42 +03:00
|
|
|
unsigned long base,
|
2008-01-23 09:11:33 +03:00
|
|
|
struct tc_u_knode *n, struct nlattr **tb,
|
2021-12-17 21:16:28 +03:00
|
|
|
struct nlattr *est, u32 flags, u32 fl_flags,
|
2018-01-18 19:20:52 +03:00
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2016-08-19 22:36:54 +03:00
|
|
|
int err;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2021-12-17 21:16:28 +03:00
|
|
|
err = tcf_exts_validate_ex(net, tp, tb, est, &n->exts, flags,
|
|
|
|
fl_flags, extack);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (err < 0)
|
|
|
|
return err;
|
|
|
|
|
2008-01-23 09:11:33 +03:00
|
|
|
if (tb[TCA_U32_LINK]) {
|
2008-01-24 07:35:03 +03:00
|
|
|
u32 handle = nla_get_u32(tb[TCA_U32_LINK]);
|
2008-11-19 11:03:09 +03:00
|
|
|
struct tc_u_hnode *ht_down = NULL, *ht_old;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2018-01-18 19:20:55 +03:00
|
|
|
if (TC_U32_KEY(handle)) {
|
|
|
|
NL_SET_ERR_MSG_MOD(extack, "u32 Link handle must be a hash table");
|
2017-08-04 15:29:14 +03:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (handle) {
|
2018-10-08 13:22:42 +03:00
|
|
|
ht_down = u32_lookup_ht(tp->data, handle);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2018-01-18 19:20:55 +03:00
|
|
|
if (!ht_down) {
|
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Link hash table not found");
|
2017-08-04 15:29:14 +03:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
2018-10-08 13:22:34 +03:00
|
|
|
if (ht_down->is_root) {
|
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Not linking to root node");
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
ht_down->refcnt++;
|
|
|
|
}
|
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
ht_old = rtnl_dereference(n->ht_down);
|
|
|
|
rcu_assign_pointer(n->ht_down, ht_down);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-11-19 11:03:09 +03:00
|
|
|
if (ht_old)
|
|
|
|
ht_old->refcnt--;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2008-01-23 09:11:33 +03:00
|
|
|
if (tb[TCA_U32_CLASSID]) {
|
2008-01-24 07:35:03 +03:00
|
|
|
n->res.classid = nla_get_u32(tb[TCA_U32_CLASSID]);
|
2005-04-17 02:20:36 +04:00
|
|
|
tcf_bind_filter(tp, &n->res, base);
|
|
|
|
}
|
|
|
|
|
2008-01-23 09:11:33 +03:00
|
|
|
if (tb[TCA_U32_INDEV]) {
|
2014-01-10 04:14:02 +04:00
|
|
|
int ret;
|
2018-01-18 19:20:54 +03:00
|
|
|
ret = tcf_change_indev(net, tb[TCA_U32_INDEV], extack);
|
2014-01-10 04:14:02 +04:00
|
|
|
if (ret < 0)
|
2017-08-04 15:29:14 +03:00
|
|
|
return -EINVAL;
|
2014-01-10 04:14:02 +04:00
|
|
|
n->ifindex = ret;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2016-09-18 15:45:33 +03:00
|
|
|
static void u32_replace_knode(struct tcf_proto *tp, struct tc_u_common *tp_c,
|
2014-09-20 08:50:34 +04:00
|
|
|
struct tc_u_knode *n)
|
|
|
|
{
|
|
|
|
struct tc_u_knode __rcu **ins;
|
|
|
|
struct tc_u_knode *pins;
|
|
|
|
struct tc_u_hnode *ht;
|
|
|
|
|
|
|
|
if (TC_U32_HTID(n->handle) == TC_U32_ROOT)
|
|
|
|
ht = rtnl_dereference(tp->root);
|
|
|
|
else
|
|
|
|
ht = u32_lookup_ht(tp_c, TC_U32_HTID(n->handle));
|
|
|
|
|
|
|
|
ins = &ht->ht[TC_U32_HASH(n->handle)];
|
|
|
|
|
|
|
|
/* The node must always exist for it to be replaced if this is not the
|
|
|
|
* case then something went very wrong elsewhere.
|
|
|
|
*/
|
|
|
|
for (pins = rtnl_dereference(*ins); ;
|
|
|
|
ins = &pins->next, pins = rtnl_dereference(*ins))
|
|
|
|
if (pins->handle == n->handle)
|
|
|
|
break;
|
|
|
|
|
2017-11-28 17:56:36 +03:00
|
|
|
idr_replace(&ht->handle_idr, n, n->handle);
|
2014-09-20 08:50:34 +04:00
|
|
|
RCU_INIT_POINTER(n->next, pins->next);
|
|
|
|
rcu_assign_pointer(*ins, n);
|
|
|
|
}
|
|
|
|
|
2019-02-21 08:37:42 +03:00
|
|
|
static struct tc_u_knode *u32_init_knode(struct net *net, struct tcf_proto *tp,
|
2014-09-20 08:50:34 +04:00
|
|
|
struct tc_u_knode *n)
|
|
|
|
{
|
2018-02-02 18:02:22 +03:00
|
|
|
struct tc_u_hnode *ht = rtnl_dereference(n->ht_down);
|
2014-09-20 08:50:34 +04:00
|
|
|
struct tc_u32_sel *s = &n->sel;
|
2018-02-02 18:02:22 +03:00
|
|
|
struct tc_u_knode *new;
|
2014-09-20 08:50:34 +04:00
|
|
|
|
2020-06-18 17:53:42 +03:00
|
|
|
new = kzalloc(struct_size(new, sel.keys, s->nkeys), GFP_KERNEL);
|
2014-09-20 08:50:34 +04:00
|
|
|
if (!new)
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
RCU_INIT_POINTER(new->next, n->next);
|
|
|
|
new->handle = n->handle;
|
|
|
|
RCU_INIT_POINTER(new->ht_up, n->ht_up);
|
|
|
|
|
|
|
|
new->ifindex = n->ifindex;
|
|
|
|
new->fshift = n->fshift;
|
|
|
|
new->res = n->res;
|
2016-02-26 18:54:39 +03:00
|
|
|
new->flags = n->flags;
|
2018-02-02 18:02:22 +03:00
|
|
|
RCU_INIT_POINTER(new->ht_down, ht);
|
2014-09-20 08:50:34 +04:00
|
|
|
|
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
|
|
|
/* Statistics may be incremented by readers during update
|
|
|
|
* so we must keep them in tact. When the node is later destroyed
|
|
|
|
* a special destroy call must be made to not free the pf memory.
|
|
|
|
*/
|
|
|
|
new->pf = n->pf;
|
|
|
|
#endif
|
|
|
|
|
|
|
|
#ifdef CONFIG_CLS_U32_MARK
|
|
|
|
new->val = n->val;
|
|
|
|
new->mask = n->mask;
|
|
|
|
/* Similarly success statistics must be moved as pointers */
|
|
|
|
new->pcpu_success = n->pcpu_success;
|
|
|
|
#endif
|
2019-05-01 19:23:15 +03:00
|
|
|
memcpy(&new->sel, s, struct_size(s, keys, s->nkeys));
|
2014-09-20 08:50:34 +04:00
|
|
|
|
2019-02-21 08:37:42 +03:00
|
|
|
if (tcf_exts_init(&new->exts, net, TCA_U32_ACT, TCA_U32_POLICE)) {
|
2016-08-19 22:36:54 +03:00
|
|
|
kfree(new);
|
|
|
|
return NULL;
|
|
|
|
}
|
2014-09-20 08:50:34 +04:00
|
|
|
|
2022-04-13 20:35:42 +03:00
|
|
|
/* bump reference count as long as we hold pointer to structure */
|
|
|
|
if (ht)
|
|
|
|
ht->refcnt++;
|
|
|
|
|
2014-09-20 08:50:34 +04:00
|
|
|
return new;
|
|
|
|
}
|
|
|
|
|
2013-01-14 09:15:39 +04:00
|
|
|
static int u32_change(struct net *net, struct sk_buff *in_skb,
|
2012-05-25 23:42:45 +04:00
|
|
|
struct tcf_proto *tp, unsigned long base, u32 handle,
|
2021-07-30 02:12:14 +03:00
|
|
|
struct nlattr **tca, void **arg, u32 flags,
|
2018-01-18 19:20:51 +03:00
|
|
|
struct netlink_ext_ack *extack)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tc_u_common *tp_c = tp->data;
|
|
|
|
struct tc_u_hnode *ht;
|
|
|
|
struct tc_u_knode *n;
|
|
|
|
struct tc_u32_sel *s;
|
2008-01-23 09:11:33 +03:00
|
|
|
struct nlattr *opt = tca[TCA_OPTIONS];
|
|
|
|
struct nlattr *tb[TCA_U32_MAX + 1];
|
2021-07-30 02:12:14 +03:00
|
|
|
u32 htid, userflags = 0;
|
2018-08-26 08:58:01 +03:00
|
|
|
size_t sel_size;
|
2005-04-17 02:20:36 +04:00
|
|
|
int err;
|
|
|
|
|
2018-01-18 19:20:55 +03:00
|
|
|
if (!opt) {
|
|
|
|
if (handle) {
|
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Filter handle requires options");
|
|
|
|
return -EINVAL;
|
|
|
|
} else {
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
netlink: make validation more configurable for future strictness
We currently have two levels of strict validation:
1) liberal (default)
- undefined (type >= max) & NLA_UNSPEC attributes accepted
- attribute length >= expected accepted
- garbage at end of message accepted
2) strict (opt-in)
- NLA_UNSPEC attributes accepted
- attribute length >= expected accepted
Split out parsing strictness into four different options:
* TRAILING - check that there's no trailing data after parsing
attributes (in message or nested)
* MAXTYPE - reject attrs > max known type
* UNSPEC - reject attributes with NLA_UNSPEC policy entries
* STRICT_ATTRS - strictly validate attribute size
The default for future things should be *everything*.
The current *_strict() is a combination of TRAILING and MAXTYPE,
and is renamed to _deprecated_strict().
The current regular parsing has none of this, and is renamed to
*_parse_deprecated().
Additionally it allows us to selectively set one of the new flags
even on old policies. Notably, the UNSPEC flag could be useful in
this case, since it can be arranged (by filling in the policy) to
not be an incompatible userspace ABI change, but would then going
forward prevent forgetting attribute entries. Similar can apply
to the POLICY flag.
We end up with the following renames:
* nla_parse -> nla_parse_deprecated
* nla_parse_strict -> nla_parse_deprecated_strict
* nlmsg_parse -> nlmsg_parse_deprecated
* nlmsg_parse_strict -> nlmsg_parse_deprecated_strict
* nla_parse_nested -> nla_parse_nested_deprecated
* nla_validate_nested -> nla_validate_nested_deprecated
Using spatch, of course:
@@
expression TB, MAX, HEAD, LEN, POL, EXT;
@@
-nla_parse(TB, MAX, HEAD, LEN, POL, EXT)
+nla_parse_deprecated(TB, MAX, HEAD, LEN, POL, EXT)
@@
expression NLH, HDRLEN, TB, MAX, POL, EXT;
@@
-nlmsg_parse(NLH, HDRLEN, TB, MAX, POL, EXT)
+nlmsg_parse_deprecated(NLH, HDRLEN, TB, MAX, POL, EXT)
@@
expression NLH, HDRLEN, TB, MAX, POL, EXT;
@@
-nlmsg_parse_strict(NLH, HDRLEN, TB, MAX, POL, EXT)
+nlmsg_parse_deprecated_strict(NLH, HDRLEN, TB, MAX, POL, EXT)
@@
expression TB, MAX, NLA, POL, EXT;
@@
-nla_parse_nested(TB, MAX, NLA, POL, EXT)
+nla_parse_nested_deprecated(TB, MAX, NLA, POL, EXT)
@@
expression START, MAX, POL, EXT;
@@
-nla_validate_nested(START, MAX, POL, EXT)
+nla_validate_nested_deprecated(START, MAX, POL, EXT)
@@
expression NLH, HDRLEN, MAX, POL, EXT;
@@
-nlmsg_validate(NLH, HDRLEN, MAX, POL, EXT)
+nlmsg_validate_deprecated(NLH, HDRLEN, MAX, POL, EXT)
For this patch, don't actually add the strict, non-renamed versions
yet so that it breaks compile if I get it wrong.
Also, while at it, make nla_validate and nla_parse go down to a
common __nla_validate_parse() function to avoid code duplication.
Ultimately, this allows us to have very strict validation for every
new caller of nla_parse()/nlmsg_parse() etc as re-introduced in the
next patch, while existing things will continue to work as is.
In effect then, this adds fully strict validation for any new command.
Signed-off-by: Johannes Berg <johannes.berg@intel.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2019-04-26 15:07:28 +03:00
|
|
|
err = nla_parse_nested_deprecated(tb, TCA_U32_MAX, opt, u32_policy,
|
|
|
|
extack);
|
2008-01-24 07:33:32 +03:00
|
|
|
if (err < 0)
|
|
|
|
return err;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2016-05-13 03:08:23 +03:00
|
|
|
if (tb[TCA_U32_FLAGS]) {
|
2021-07-30 02:12:14 +03:00
|
|
|
userflags = nla_get_u32(tb[TCA_U32_FLAGS]);
|
|
|
|
if (!tc_flags_valid(userflags)) {
|
2018-01-18 19:20:55 +03:00
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Invalid filter flags");
|
2016-06-06 18:16:47 +03:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
2016-05-13 03:08:23 +03:00
|
|
|
}
|
2016-02-26 18:54:39 +03:00
|
|
|
|
2017-08-05 07:31:43 +03:00
|
|
|
n = *arg;
|
2011-01-19 22:26:56 +03:00
|
|
|
if (n) {
|
2014-09-20 08:50:34 +04:00
|
|
|
struct tc_u_knode *new;
|
|
|
|
|
2018-01-18 19:20:55 +03:00
|
|
|
if (TC_U32_KEY(n->handle) == 0) {
|
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Key node id cannot be zero");
|
2005-04-17 02:20:36 +04:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2021-07-30 02:12:14 +03:00
|
|
|
if ((n->flags ^ userflags) &
|
2018-02-08 18:10:39 +03:00
|
|
|
~(TCA_CLS_FLAGS_IN_HW | TCA_CLS_FLAGS_NOT_IN_HW)) {
|
2018-01-18 19:20:55 +03:00
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Key node flags do not match passed flags");
|
2016-02-26 18:54:39 +03:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
2016-02-26 18:54:39 +03:00
|
|
|
|
2019-02-21 08:37:42 +03:00
|
|
|
new = u32_init_knode(net, tp, n);
|
2014-09-20 08:50:34 +04:00
|
|
|
if (!new)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
2018-10-08 13:22:42 +03:00
|
|
|
err = u32_set_parms(net, tp, base, new, tb,
|
2021-12-17 21:16:28 +03:00
|
|
|
tca[TCA_RATE], flags, new->flags,
|
|
|
|
extack);
|
2014-09-20 08:50:34 +04:00
|
|
|
|
|
|
|
if (err) {
|
2022-04-13 20:35:41 +03:00
|
|
|
__u32_destroy_key(new);
|
2014-09-20 08:50:34 +04:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2018-01-20 04:44:45 +03:00
|
|
|
err = u32_replace_hw_knode(tp, new, flags, extack);
|
2016-05-13 03:08:23 +03:00
|
|
|
if (err) {
|
2022-04-13 20:35:41 +03:00
|
|
|
__u32_destroy_key(new);
|
2016-05-13 03:08:23 +03:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2017-02-16 11:31:15 +03:00
|
|
|
if (!tc_in_hw(new->flags))
|
|
|
|
new->flags |= TCA_CLS_FLAGS_NOT_IN_HW;
|
|
|
|
|
2014-09-20 08:50:34 +04:00
|
|
|
u32_replace_knode(tp, tp_c, new);
|
2014-10-01 03:07:24 +04:00
|
|
|
tcf_unbind_filter(tp, &n->res);
|
2017-11-07 00:47:30 +03:00
|
|
|
tcf_exts_get_net(&n->exts);
|
2018-05-24 01:26:53 +03:00
|
|
|
tcf_queue_work(&n->rwork, u32_delete_key_work);
|
2014-09-20 08:50:34 +04:00
|
|
|
return 0;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2008-01-23 09:11:33 +03:00
|
|
|
if (tb[TCA_U32_DIVISOR]) {
|
2011-01-19 22:26:56 +03:00
|
|
|
unsigned int divisor = nla_get_u32(tb[TCA_U32_DIVISOR]);
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2018-10-08 13:22:35 +03:00
|
|
|
if (!is_power_of_2(divisor)) {
|
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Divisor is not a power of 2");
|
|
|
|
return -EINVAL;
|
|
|
|
}
|
|
|
|
if (divisor-- > 0x100) {
|
2018-01-18 19:20:55 +03:00
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Exceeded maximum 256 hash buckets");
|
2005-04-17 02:20:36 +04:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
|
|
|
if (TC_U32_KEY(handle)) {
|
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Divisor can only be used on a hash table");
|
2005-04-17 02:20:36 +04:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
net/sched: cls_u32: Replace one-element array with flexible-array member
There is a regular need in the kernel to provide a way to declare having
a dynamically sized set of trailing elements in a structure. Kernel code
should always use “flexible array members”[1] for these cases. The older
style of one-element or zero-length arrays should no longer be used[2].
Refactor the code according to the use of a flexible-array member in
struct tc_u_hnode and use the struct_size() helper to calculate the
size for the allocations. Commit 5778d39d070b ("net_sched: fix struct
tc_u_hnode layout in u32") makes it clear that the code is expected to
dynamically allocate divisor + 1 entries for ->ht[] in tc_uhnode. Also,
based on other observations, as the piece of code below:
1232 for (h = 0; h <= ht->divisor; h++) {
1233 for (n = rtnl_dereference(ht->ht[h]);
1234 n;
1235 n = rtnl_dereference(n->next)) {
1236 if (tc_skip_hw(n->flags))
1237 continue;
1238
1239 err = u32_reoffload_knode(tp, n, add, cb,
1240 cb_priv, extack);
1241 if (err)
1242 return err;
1243 }
1244 }
we can assume that, in general, the code is actually expecting to allocate
that extra space for the one-element array in tc_uhnode, everytime it
allocates memory for instances of tc_uhnode or tc_u_common structures.
That's the reason for passing '1' as the last argument for struct_size()
in the allocation for _root_ht_ and _tp_c_, and 'divisor + 1' in the
allocation code for _ht_.
[1] https://en.wikipedia.org/wiki/Flexible_array_member
[2] https://www.kernel.org/doc/html/v5.9-rc1/process/deprecated.html#zero-length-and-one-element-arrays
Tested-by: kernel test robot <lkp@intel.com>
Link: https://lore.kernel.org/lkml/5f7062af.z3T9tn9yIPv6h5Ny%25lkp@intel.com/
Signed-off-by: Gustavo A. R. Silva <gustavoars@kernel.org>
Signed-off-by: David S. Miller <davem@davemloft.net>
2020-09-28 18:30:52 +03:00
|
|
|
ht = kzalloc(struct_size(ht, ht, divisor + 1), GFP_KERNEL);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (ht == NULL)
|
|
|
|
return -ENOBUFS;
|
2017-09-25 20:13:51 +03:00
|
|
|
if (handle == 0) {
|
|
|
|
handle = gen_new_htid(tp->data, ht);
|
|
|
|
if (handle == 0) {
|
|
|
|
kfree(ht);
|
|
|
|
return -ENOMEM;
|
|
|
|
}
|
|
|
|
} else {
|
2017-11-28 21:45:02 +03:00
|
|
|
err = idr_alloc_u32(&tp_c->handle_idr, ht, &handle,
|
|
|
|
handle, GFP_KERNEL);
|
2017-09-25 20:13:51 +03:00
|
|
|
if (err) {
|
|
|
|
kfree(ht);
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
}
|
2008-04-13 05:37:13 +04:00
|
|
|
ht->refcnt = 1;
|
2005-04-17 02:20:36 +04:00
|
|
|
ht->divisor = divisor;
|
|
|
|
ht->handle = handle;
|
|
|
|
ht->prio = tp->prio;
|
2017-09-25 20:13:51 +03:00
|
|
|
idr_init(&ht->handle_idr);
|
2021-07-30 02:12:14 +03:00
|
|
|
ht->flags = userflags;
|
2016-06-08 22:11:03 +03:00
|
|
|
|
2021-07-30 02:12:14 +03:00
|
|
|
err = u32_replace_hw_hnode(tp, ht, userflags, extack);
|
2016-06-08 22:11:03 +03:00
|
|
|
if (err) {
|
2017-11-28 17:48:43 +03:00
|
|
|
idr_remove(&tp_c->handle_idr, handle);
|
2016-06-08 22:11:03 +03:00
|
|
|
kfree(ht);
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
RCU_INIT_POINTER(ht->next, tp_c->hlist);
|
|
|
|
rcu_assign_pointer(tp_c->hlist, ht);
|
2017-08-05 07:31:43 +03:00
|
|
|
*arg = ht;
|
2016-02-17 08:17:09 +03:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2008-01-23 09:11:33 +03:00
|
|
|
if (tb[TCA_U32_HASH]) {
|
2008-01-24 07:35:03 +03:00
|
|
|
htid = nla_get_u32(tb[TCA_U32_HASH]);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (TC_U32_HTID(htid) == TC_U32_ROOT) {
|
2014-09-13 07:09:16 +04:00
|
|
|
ht = rtnl_dereference(tp->root);
|
2005-04-17 02:20:36 +04:00
|
|
|
htid = ht->handle;
|
|
|
|
} else {
|
|
|
|
ht = u32_lookup_ht(tp->data, TC_U32_HTID(htid));
|
2018-01-18 19:20:55 +03:00
|
|
|
if (!ht) {
|
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Specified hash table not found");
|
2005-04-17 02:20:36 +04:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
} else {
|
2014-09-13 07:09:16 +04:00
|
|
|
ht = rtnl_dereference(tp->root);
|
2005-04-17 02:20:36 +04:00
|
|
|
htid = ht->handle;
|
|
|
|
}
|
|
|
|
|
2018-01-18 19:20:55 +03:00
|
|
|
if (ht->divisor < TC_U32_HASH(htid)) {
|
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Specified hash table buckets exceed configured value");
|
2005-04-17 02:20:36 +04:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (handle) {
|
2018-01-18 19:20:55 +03:00
|
|
|
if (TC_U32_HTID(handle) && TC_U32_HTID(handle ^ htid)) {
|
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Handle specified hash table address mismatch");
|
2005-04-17 02:20:36 +04:00
|
|
|
return -EINVAL;
|
2018-01-18 19:20:55 +03:00
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
handle = htid | TC_U32_NODE(handle);
|
2017-11-28 21:45:02 +03:00
|
|
|
err = idr_alloc_u32(&ht->handle_idr, NULL, &handle, handle,
|
2017-09-25 20:13:51 +03:00
|
|
|
GFP_KERNEL);
|
|
|
|
if (err)
|
|
|
|
return err;
|
2005-04-17 02:20:36 +04:00
|
|
|
} else
|
|
|
|
handle = gen_new_kid(ht, htid);
|
|
|
|
|
2017-09-25 20:13:51 +03:00
|
|
|
if (tb[TCA_U32_SEL] == NULL) {
|
2018-01-18 19:20:55 +03:00
|
|
|
NL_SET_ERR_MSG_MOD(extack, "Selector not specified");
|
2017-09-25 20:13:51 +03:00
|
|
|
err = -EINVAL;
|
|
|
|
goto erridr;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2008-01-23 09:11:33 +03:00
|
|
|
s = nla_data(tb[TCA_U32_SEL]);
|
2018-08-26 08:58:01 +03:00
|
|
|
sel_size = struct_size(s, keys, s->nkeys);
|
|
|
|
if (nla_len(tb[TCA_U32_SEL]) < sel_size) {
|
|
|
|
err = -EINVAL;
|
|
|
|
goto erridr;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2020-07-30 19:03:14 +03:00
|
|
|
n = kzalloc(struct_size(n, sel.keys, s->nkeys), GFP_KERNEL);
|
2017-09-25 20:13:51 +03:00
|
|
|
if (n == NULL) {
|
|
|
|
err = -ENOBUFS;
|
|
|
|
goto erridr;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
2020-07-30 19:03:14 +03:00
|
|
|
n->pf = __alloc_percpu(struct_size(n->pf, kcnts, s->nkeys),
|
|
|
|
__alignof__(struct tc_u32_pcnt));
|
2014-09-13 07:08:47 +04:00
|
|
|
if (!n->pf) {
|
2017-09-25 20:13:51 +03:00
|
|
|
err = -ENOBUFS;
|
|
|
|
goto errfree;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2022-09-27 18:37:01 +03:00
|
|
|
unsafe_memcpy(&n->sel, s, sel_size,
|
|
|
|
/* A composite flex-array structure destination,
|
|
|
|
* which was correctly sized with struct_size(),
|
|
|
|
* bounds-checked against nla_len(), and allocated
|
|
|
|
* above. */);
|
2014-09-16 10:30:49 +04:00
|
|
|
RCU_INIT_POINTER(n->ht_up, ht);
|
2005-04-17 02:20:36 +04:00
|
|
|
n->handle = handle;
|
2007-11-11 08:54:50 +03:00
|
|
|
n->fshift = s->hmask ? ffs(ntohl(s->hmask)) - 1 : 0;
|
2021-07-30 02:12:14 +03:00
|
|
|
n->flags = userflags;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2019-02-21 08:37:42 +03:00
|
|
|
err = tcf_exts_init(&n->exts, net, TCA_U32_ACT, TCA_U32_POLICE);
|
2016-08-19 22:36:54 +03:00
|
|
|
if (err < 0)
|
|
|
|
goto errout;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_MARK
|
2014-09-13 07:08:47 +04:00
|
|
|
n->pcpu_success = alloc_percpu(u32);
|
2014-09-20 08:50:04 +04:00
|
|
|
if (!n->pcpu_success) {
|
|
|
|
err = -ENOMEM;
|
|
|
|
goto errout;
|
|
|
|
}
|
2014-09-13 07:08:47 +04:00
|
|
|
|
2008-01-23 09:11:33 +03:00
|
|
|
if (tb[TCA_U32_MARK]) {
|
2005-04-17 02:20:36 +04:00
|
|
|
struct tc_u32_mark *mark;
|
|
|
|
|
2008-01-23 09:11:33 +03:00
|
|
|
mark = nla_data(tb[TCA_U32_MARK]);
|
2014-09-13 07:08:47 +04:00
|
|
|
n->val = mark->val;
|
|
|
|
n->mask = mark->mask;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
#endif
|
|
|
|
|
2021-12-17 21:16:28 +03:00
|
|
|
err = u32_set_parms(net, tp, base, n, tb, tca[TCA_RATE],
|
|
|
|
flags, n->flags, extack);
|
2005-04-17 02:20:36 +04:00
|
|
|
if (err == 0) {
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_knode __rcu **ins;
|
|
|
|
struct tc_u_knode *pins;
|
|
|
|
|
2018-01-20 04:44:45 +03:00
|
|
|
err = u32_replace_hw_knode(tp, n, flags, extack);
|
2016-05-13 03:08:23 +03:00
|
|
|
if (err)
|
|
|
|
goto errhw;
|
|
|
|
|
2017-02-16 11:31:15 +03:00
|
|
|
if (!tc_in_hw(n->flags))
|
|
|
|
n->flags |= TCA_CLS_FLAGS_NOT_IN_HW;
|
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
ins = &ht->ht[TC_U32_HASH(handle)];
|
|
|
|
for (pins = rtnl_dereference(*ins); pins;
|
|
|
|
ins = &pins->next, pins = rtnl_dereference(*ins))
|
|
|
|
if (TC_U32_NODE(handle) < TC_U32_NODE(pins->handle))
|
2005-04-17 02:20:36 +04:00
|
|
|
break;
|
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
RCU_INIT_POINTER(n->next, pins);
|
|
|
|
rcu_assign_pointer(*ins, n);
|
2018-10-08 13:22:43 +03:00
|
|
|
tp_c->knodes++;
|
2017-08-05 07:31:43 +03:00
|
|
|
*arg = n;
|
2005-04-17 02:20:36 +04:00
|
|
|
return 0;
|
|
|
|
}
|
2014-09-20 08:50:04 +04:00
|
|
|
|
2016-05-13 03:08:23 +03:00
|
|
|
errhw:
|
2014-09-20 08:50:04 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_MARK
|
|
|
|
free_percpu(n->pcpu_success);
|
|
|
|
#endif
|
|
|
|
|
2016-08-19 22:36:54 +03:00
|
|
|
errout:
|
|
|
|
tcf_exts_destroy(&n->exts);
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
2017-09-25 20:13:51 +03:00
|
|
|
errfree:
|
2014-09-13 07:09:16 +04:00
|
|
|
free_percpu(n->pf);
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
|
|
|
kfree(n);
|
2017-09-25 20:13:51 +03:00
|
|
|
erridr:
|
2017-11-28 17:48:43 +03:00
|
|
|
idr_remove(&ht->handle_idr, handle);
|
2005-04-17 02:20:36 +04:00
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
2019-02-11 11:55:45 +03:00
|
|
|
static void u32_walk(struct tcf_proto *tp, struct tcf_walker *arg,
|
|
|
|
bool rtnl_held)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
struct tc_u_common *tp_c = tp->data;
|
|
|
|
struct tc_u_hnode *ht;
|
|
|
|
struct tc_u_knode *n;
|
2011-01-19 22:26:56 +03:00
|
|
|
unsigned int h;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (arg->stop)
|
|
|
|
return;
|
|
|
|
|
2014-09-13 07:09:16 +04:00
|
|
|
for (ht = rtnl_dereference(tp_c->hlist);
|
|
|
|
ht;
|
|
|
|
ht = rtnl_dereference(ht->next)) {
|
2005-04-17 02:20:36 +04:00
|
|
|
if (ht->prio != tp->prio)
|
|
|
|
continue;
|
2022-09-16 05:02:44 +03:00
|
|
|
|
|
|
|
if (!tc_cls_stats_dump(tp, arg, ht))
|
|
|
|
return;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
for (h = 0; h <= ht->divisor; h++) {
|
2014-09-13 07:09:16 +04:00
|
|
|
for (n = rtnl_dereference(ht->ht[h]);
|
|
|
|
n;
|
|
|
|
n = rtnl_dereference(n->next)) {
|
2022-09-16 05:02:44 +03:00
|
|
|
if (!tc_cls_stats_dump(tp, arg, n))
|
2005-04-17 02:20:36 +04:00
|
|
|
return;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-06-26 00:30:08 +03:00
|
|
|
static int u32_reoffload_hnode(struct tcf_proto *tp, struct tc_u_hnode *ht,
|
2019-07-19 19:20:15 +03:00
|
|
|
bool add, flow_setup_cb_t *cb, void *cb_priv,
|
2018-06-26 00:30:08 +03:00
|
|
|
struct netlink_ext_ack *extack)
|
|
|
|
{
|
|
|
|
struct tc_cls_u32_offload cls_u32 = {};
|
|
|
|
int err;
|
|
|
|
|
2019-05-07 03:24:21 +03:00
|
|
|
tc_cls_common_offload_init(&cls_u32.common, tp, ht->flags, extack);
|
2018-06-26 00:30:08 +03:00
|
|
|
cls_u32.command = add ? TC_CLSU32_NEW_HNODE : TC_CLSU32_DELETE_HNODE;
|
|
|
|
cls_u32.hnode.divisor = ht->divisor;
|
|
|
|
cls_u32.hnode.handle = ht->handle;
|
|
|
|
cls_u32.hnode.prio = ht->prio;
|
|
|
|
|
|
|
|
err = cb(TC_SETUP_CLSU32, &cls_u32, cb_priv);
|
|
|
|
if (err && add && tc_skip_sw(ht->flags))
|
|
|
|
return err;
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int u32_reoffload_knode(struct tcf_proto *tp, struct tc_u_knode *n,
|
2019-07-19 19:20:15 +03:00
|
|
|
bool add, flow_setup_cb_t *cb, void *cb_priv,
|
2018-06-26 00:30:08 +03:00
|
|
|
struct netlink_ext_ack *extack)
|
|
|
|
{
|
|
|
|
struct tc_u_hnode *ht = rtnl_dereference(n->ht_down);
|
|
|
|
struct tcf_block *block = tp->chain->block;
|
|
|
|
struct tc_cls_u32_offload cls_u32 = {};
|
|
|
|
|
2019-05-07 03:24:21 +03:00
|
|
|
tc_cls_common_offload_init(&cls_u32.common, tp, n->flags, extack);
|
2018-06-26 00:30:08 +03:00
|
|
|
cls_u32.command = add ?
|
|
|
|
TC_CLSU32_REPLACE_KNODE : TC_CLSU32_DELETE_KNODE;
|
|
|
|
cls_u32.knode.handle = n->handle;
|
|
|
|
|
|
|
|
if (add) {
|
|
|
|
cls_u32.knode.fshift = n->fshift;
|
|
|
|
#ifdef CONFIG_CLS_U32_MARK
|
|
|
|
cls_u32.knode.val = n->val;
|
|
|
|
cls_u32.knode.mask = n->mask;
|
|
|
|
#else
|
|
|
|
cls_u32.knode.val = 0;
|
|
|
|
cls_u32.knode.mask = 0;
|
|
|
|
#endif
|
|
|
|
cls_u32.knode.sel = &n->sel;
|
2018-11-20 02:21:46 +03:00
|
|
|
cls_u32.knode.res = &n->res;
|
2018-06-26 00:30:08 +03:00
|
|
|
cls_u32.knode.exts = &n->exts;
|
|
|
|
if (n->ht_down)
|
|
|
|
cls_u32.knode.link_handle = ht->handle;
|
|
|
|
}
|
|
|
|
|
2020-12-08 15:08:22 +03:00
|
|
|
return tc_setup_cb_reoffload(block, tp, add, cb, TC_SETUP_CLSU32,
|
|
|
|
&cls_u32, cb_priv, &n->flags,
|
|
|
|
&n->in_hw_count);
|
2018-06-26 00:30:08 +03:00
|
|
|
}
|
|
|
|
|
2019-07-19 19:20:15 +03:00
|
|
|
static int u32_reoffload(struct tcf_proto *tp, bool add, flow_setup_cb_t *cb,
|
2018-06-26 00:30:08 +03:00
|
|
|
void *cb_priv, struct netlink_ext_ack *extack)
|
|
|
|
{
|
|
|
|
struct tc_u_common *tp_c = tp->data;
|
|
|
|
struct tc_u_hnode *ht;
|
|
|
|
struct tc_u_knode *n;
|
|
|
|
unsigned int h;
|
|
|
|
int err;
|
|
|
|
|
|
|
|
for (ht = rtnl_dereference(tp_c->hlist);
|
|
|
|
ht;
|
|
|
|
ht = rtnl_dereference(ht->next)) {
|
|
|
|
if (ht->prio != tp->prio)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/* When adding filters to a new dev, try to offload the
|
|
|
|
* hashtable first. When removing, do the filters before the
|
|
|
|
* hashtable.
|
|
|
|
*/
|
|
|
|
if (add && !tc_skip_hw(ht->flags)) {
|
|
|
|
err = u32_reoffload_hnode(tp, ht, add, cb, cb_priv,
|
|
|
|
extack);
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
|
|
|
|
for (h = 0; h <= ht->divisor; h++) {
|
|
|
|
for (n = rtnl_dereference(ht->ht[h]);
|
|
|
|
n;
|
|
|
|
n = rtnl_dereference(n->next)) {
|
|
|
|
if (tc_skip_hw(n->flags))
|
|
|
|
continue;
|
|
|
|
|
|
|
|
err = u32_reoffload_knode(tp, n, add, cb,
|
|
|
|
cb_priv, extack);
|
|
|
|
if (err)
|
|
|
|
return err;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (!add && !tc_skip_hw(ht->flags))
|
|
|
|
u32_reoffload_hnode(tp, ht, add, cb, cb_priv, extack);
|
|
|
|
}
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2020-01-24 03:26:18 +03:00
|
|
|
static void u32_bind_class(void *fh, u32 classid, unsigned long cl, void *q,
|
|
|
|
unsigned long base)
|
net_sched: add reverse binding for tc class
TC filters when used as classifiers are bound to TC classes.
However, there is a hidden difference when adding them in different
orders:
1. If we add tc classes before its filters, everything is fine.
Logically, the classes exist before we specify their ID's in
filters, it is easy to bind them together, just as in the current
code base.
2. If we add tc filters before the tc classes they bind, we have to
do dynamic lookup in fast path. What's worse, this happens all
the time not just once, because on fast path tcf_result is passed
on stack, there is no way to propagate back to the one in tc filters.
This hidden difference hurts performance silently if we have many tc
classes in hierarchy.
This patch intends to close this gap by doing the reverse binding when
we create a new class, in this case we can actually search all the
filters in its parent, match and fixup by classid. And because
tcf_result is specific to each type of tc filter, we have to introduce
a new ops for each filter to tell how to bind the class.
Note, we still can NOT totally get rid of those class lookup in
->enqueue() because cgroup and flow filters have no way to determine
the classid at setup time, they still have to go through dynamic lookup.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-31 00:30:36 +03:00
|
|
|
{
|
|
|
|
struct tc_u_knode *n = fh;
|
|
|
|
|
2022-09-27 15:48:55 +03:00
|
|
|
tc_cls_bind_class(classid, cl, q, &n->res, base);
|
net_sched: add reverse binding for tc class
TC filters when used as classifiers are bound to TC classes.
However, there is a hidden difference when adding them in different
orders:
1. If we add tc classes before its filters, everything is fine.
Logically, the classes exist before we specify their ID's in
filters, it is easy to bind them together, just as in the current
code base.
2. If we add tc filters before the tc classes they bind, we have to
do dynamic lookup in fast path. What's worse, this happens all
the time not just once, because on fast path tcf_result is passed
on stack, there is no way to propagate back to the one in tc filters.
This hidden difference hurts performance silently if we have many tc
classes in hierarchy.
This patch intends to close this gap by doing the reverse binding when
we create a new class, in this case we can actually search all the
filters in its parent, match and fixup by classid. And because
tcf_result is specific to each type of tc filter, we have to introduce
a new ops for each filter to tell how to bind the class.
Note, we still can NOT totally get rid of those class lookup in
->enqueue() because cgroup and flow filters have no way to determine
the classid at setup time, they still have to go through dynamic lookup.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-31 00:30:36 +03:00
|
|
|
}
|
|
|
|
|
2017-08-05 07:31:43 +03:00
|
|
|
static int u32_dump(struct net *net, struct tcf_proto *tp, void *fh,
|
2019-02-11 11:55:45 +03:00
|
|
|
struct sk_buff *skb, struct tcmsg *t, bool rtnl_held)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
2017-08-05 07:31:43 +03:00
|
|
|
struct tc_u_knode *n = fh;
|
2014-09-13 07:09:16 +04:00
|
|
|
struct tc_u_hnode *ht_up, *ht_down;
|
2008-01-24 07:34:11 +03:00
|
|
|
struct nlattr *nest;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (n == NULL)
|
|
|
|
return skb->len;
|
|
|
|
|
|
|
|
t->tcm_handle = n->handle;
|
|
|
|
|
2019-04-26 12:13:06 +03:00
|
|
|
nest = nla_nest_start_noflag(skb, TCA_OPTIONS);
|
2008-01-24 07:34:11 +03:00
|
|
|
if (nest == NULL)
|
|
|
|
goto nla_put_failure;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
if (TC_U32_KEY(n->handle) == 0) {
|
2017-08-05 07:31:43 +03:00
|
|
|
struct tc_u_hnode *ht = fh;
|
2011-01-19 22:26:56 +03:00
|
|
|
u32 divisor = ht->divisor + 1;
|
|
|
|
|
2012-03-29 13:11:39 +04:00
|
|
|
if (nla_put_u32(skb, TCA_U32_DIVISOR, divisor))
|
|
|
|
goto nla_put_failure;
|
2005-04-17 02:20:36 +04:00
|
|
|
} else {
|
2014-09-13 07:08:47 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
|
|
|
struct tc_u32_pcnt *gpf;
|
|
|
|
int cpu;
|
2014-09-16 10:30:26 +04:00
|
|
|
#endif
|
2014-09-13 07:08:47 +04:00
|
|
|
|
2020-07-30 19:03:14 +03:00
|
|
|
if (nla_put(skb, TCA_U32_SEL, struct_size(&n->sel, keys, n->sel.nkeys),
|
2012-03-29 13:11:39 +04:00
|
|
|
&n->sel))
|
|
|
|
goto nla_put_failure;
|
2014-09-13 07:09:16 +04:00
|
|
|
|
|
|
|
ht_up = rtnl_dereference(n->ht_up);
|
|
|
|
if (ht_up) {
|
2005-04-17 02:20:36 +04:00
|
|
|
u32 htid = n->handle & 0xFFFFF000;
|
2012-03-29 13:11:39 +04:00
|
|
|
if (nla_put_u32(skb, TCA_U32_HASH, htid))
|
|
|
|
goto nla_put_failure;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
2012-03-29 13:11:39 +04:00
|
|
|
if (n->res.classid &&
|
|
|
|
nla_put_u32(skb, TCA_U32_CLASSID, n->res.classid))
|
|
|
|
goto nla_put_failure;
|
2014-09-13 07:09:16 +04:00
|
|
|
|
|
|
|
ht_down = rtnl_dereference(n->ht_down);
|
|
|
|
if (ht_down &&
|
|
|
|
nla_put_u32(skb, TCA_U32_LINK, ht_down->handle))
|
2012-03-29 13:11:39 +04:00
|
|
|
goto nla_put_failure;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2016-02-26 18:54:39 +03:00
|
|
|
if (n->flags && nla_put_u32(skb, TCA_U32_FLAGS, n->flags))
|
|
|
|
goto nla_put_failure;
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_MARK
|
2014-09-13 07:08:47 +04:00
|
|
|
if ((n->val || n->mask)) {
|
|
|
|
struct tc_u32_mark mark = {.val = n->val,
|
|
|
|
.mask = n->mask,
|
|
|
|
.success = 0};
|
2014-09-16 10:30:26 +04:00
|
|
|
int cpum;
|
2014-09-13 07:08:47 +04:00
|
|
|
|
2014-09-16 10:30:26 +04:00
|
|
|
for_each_possible_cpu(cpum) {
|
|
|
|
__u32 cnt = *per_cpu_ptr(n->pcpu_success, cpum);
|
2014-09-13 07:08:47 +04:00
|
|
|
|
|
|
|
mark.success += cnt;
|
|
|
|
}
|
|
|
|
|
|
|
|
if (nla_put(skb, TCA_U32_MARK, sizeof(mark), &mark))
|
|
|
|
goto nla_put_failure;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
|
|
|
|
2013-12-16 08:15:07 +04:00
|
|
|
if (tcf_exts_dump(skb, &n->exts) < 0)
|
2008-01-23 09:11:33 +03:00
|
|
|
goto nla_put_failure;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
2014-01-10 04:14:02 +04:00
|
|
|
if (n->ifindex) {
|
|
|
|
struct net_device *dev;
|
|
|
|
dev = __dev_get_by_index(net, n->ifindex);
|
|
|
|
if (dev && nla_put_string(skb, TCA_U32_INDEV, dev->name))
|
|
|
|
goto nla_put_failure;
|
|
|
|
}
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
2020-07-30 19:03:14 +03:00
|
|
|
gpf = kzalloc(struct_size(gpf, kcnts, n->sel.nkeys), GFP_KERNEL);
|
2014-09-13 07:08:47 +04:00
|
|
|
if (!gpf)
|
|
|
|
goto nla_put_failure;
|
|
|
|
|
|
|
|
for_each_possible_cpu(cpu) {
|
|
|
|
int i;
|
|
|
|
struct tc_u32_pcnt *pf = per_cpu_ptr(n->pf, cpu);
|
|
|
|
|
|
|
|
gpf->rcnt += pf->rcnt;
|
|
|
|
gpf->rhit += pf->rhit;
|
|
|
|
for (i = 0; i < n->sel.nkeys; i++)
|
|
|
|
gpf->kcnts[i] += pf->kcnts[i];
|
|
|
|
}
|
|
|
|
|
2020-07-30 19:03:14 +03:00
|
|
|
if (nla_put_64bit(skb, TCA_U32_PCNT, struct_size(gpf, kcnts, n->sel.nkeys),
|
2016-04-26 11:06:18 +03:00
|
|
|
gpf, TCA_U32_PAD)) {
|
2014-09-13 07:08:47 +04:00
|
|
|
kfree(gpf);
|
2012-03-29 13:11:39 +04:00
|
|
|
goto nla_put_failure;
|
2014-09-13 07:08:47 +04:00
|
|
|
}
|
|
|
|
kfree(gpf);
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
|
|
|
}
|
|
|
|
|
2008-01-24 07:34:11 +03:00
|
|
|
nla_nest_end(skb, nest);
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
if (TC_U32_KEY(n->handle))
|
2013-12-16 08:15:07 +04:00
|
|
|
if (tcf_exts_dump_stats(skb, &n->exts) < 0)
|
2008-01-23 09:11:33 +03:00
|
|
|
goto nla_put_failure;
|
2005-04-17 02:20:36 +04:00
|
|
|
return skb->len;
|
|
|
|
|
2008-01-23 09:11:33 +03:00
|
|
|
nla_put_failure:
|
2008-01-24 07:34:11 +03:00
|
|
|
nla_nest_cancel(skb, nest);
|
2005-04-17 02:20:36 +04:00
|
|
|
return -1;
|
|
|
|
}
|
|
|
|
|
2008-01-23 09:10:42 +03:00
|
|
|
static struct tcf_proto_ops cls_u32_ops __read_mostly = {
|
2005-04-17 02:20:36 +04:00
|
|
|
.kind = "u32",
|
|
|
|
.classify = u32_classify,
|
|
|
|
.init = u32_init,
|
|
|
|
.destroy = u32_destroy,
|
|
|
|
.get = u32_get,
|
|
|
|
.change = u32_change,
|
|
|
|
.delete = u32_delete,
|
|
|
|
.walk = u32_walk,
|
2018-06-26 00:30:08 +03:00
|
|
|
.reoffload = u32_reoffload,
|
2005-04-17 02:20:36 +04:00
|
|
|
.dump = u32_dump,
|
net_sched: add reverse binding for tc class
TC filters when used as classifiers are bound to TC classes.
However, there is a hidden difference when adding them in different
orders:
1. If we add tc classes before its filters, everything is fine.
Logically, the classes exist before we specify their ID's in
filters, it is easy to bind them together, just as in the current
code base.
2. If we add tc filters before the tc classes they bind, we have to
do dynamic lookup in fast path. What's worse, this happens all
the time not just once, because on fast path tcf_result is passed
on stack, there is no way to propagate back to the one in tc filters.
This hidden difference hurts performance silently if we have many tc
classes in hierarchy.
This patch intends to close this gap by doing the reverse binding when
we create a new class, in this case we can actually search all the
filters in its parent, match and fixup by classid. And because
tcf_result is specific to each type of tc filter, we have to introduce
a new ops for each filter to tell how to bind the class.
Note, we still can NOT totally get rid of those class lookup in
->enqueue() because cgroup and flow filters have no way to determine
the classid at setup time, they still have to go through dynamic lookup.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-31 00:30:36 +03:00
|
|
|
.bind_class = u32_bind_class,
|
2005-04-17 02:20:36 +04:00
|
|
|
.owner = THIS_MODULE,
|
|
|
|
};
|
|
|
|
|
|
|
|
static int __init init_u32(void)
|
|
|
|
{
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
int i, ret;
|
|
|
|
|
2010-05-12 10:37:05 +04:00
|
|
|
pr_info("u32 classifier\n");
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_CLS_U32_PERF
|
2010-05-12 10:37:05 +04:00
|
|
|
pr_info(" Performance counters on\n");
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
2010-05-12 10:37:05 +04:00
|
|
|
pr_info(" input device check on\n");
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_NET_CLS_ACT
|
2010-05-12 10:37:05 +04:00
|
|
|
pr_info(" Actions configured\n");
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
tc_u_common_hash = kvmalloc_array(U32_HASH_SIZE,
|
|
|
|
sizeof(struct hlist_head),
|
|
|
|
GFP_KERNEL);
|
|
|
|
if (!tc_u_common_hash)
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
for (i = 0; i < U32_HASH_SIZE; i++)
|
|
|
|
INIT_HLIST_HEAD(&tc_u_common_hash[i]);
|
|
|
|
|
|
|
|
ret = register_tcf_proto_ops(&cls_u32_ops);
|
|
|
|
if (ret)
|
|
|
|
kvfree(tc_u_common_hash);
|
|
|
|
return ret;
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
2007-02-09 17:25:16 +03:00
|
|
|
static void __exit exit_u32(void)
|
2005-04-17 02:20:36 +04:00
|
|
|
{
|
|
|
|
unregister_tcf_proto_ops(&cls_u32_ops);
|
net_sched: kill u32_node pointer in Qdisc
It is ugly to hide a u32-filter-specific pointer inside Qdisc,
this breaks the TC layers:
1. Qdisc is a generic representation, should not have any specific
data of any type
2. Qdisc layer is above filter layer, should only save filters in
the list of struct tcf_proto.
This pointer is used as the head of the chain of u32 hash tables,
that is struct tc_u_hnode, because u32 filter is very special,
it allows to create multiple hash tables within one qdisc and
across multiple u32 filters.
Instead of using this ugly pointer, we can just save it in a global
hash table key'ed by (dev ifindex, qdisc handle), therefore we can
still treat it as a per qdisc basis data structure conceptually.
Of course, because of network namespaces, this key is not unique
at all, but it is fine as we already have a pointer to Qdisc in
struct tc_u_common, we can just compare the pointers when collision.
And this only affects slow paths, has no impact to fast path,
thanks to the pointer ->tp_c.
Cc: Jamal Hadi Salim <jhs@mojatatu.com>
Cc: Jiri Pirko <jiri@resnulli.us>
Signed-off-by: Cong Wang <xiyou.wangcong@gmail.com>
Acked-by: Jiri Pirko <jiri@mellanox.com>
Acked-by: Jamal Hadi Salim <jhs@mojatatu.com>
Signed-off-by: David S. Miller <davem@davemloft.net>
2017-08-25 02:51:30 +03:00
|
|
|
kvfree(tc_u_common_hash);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
module_init(init_u32)
|
|
|
|
module_exit(exit_u32)
|
|
|
|
MODULE_LICENSE("GPL");
|