License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 17:07:57 +03:00
|
|
|
/* SPDX-License-Identifier: GPL-2.0 */
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifndef _ASM_GENERIC_PERCPU_H_
|
|
|
|
|
#define _ASM_GENERIC_PERCPU_H_
|
2009-04-22 02:00:29 +04:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#include <linux/compiler.h>
|
2007-05-02 21:27:10 +04:00
|
|
|
#include <linux/threads.h>
|
2009-04-22 02:00:29 +04:00
|
|
|
#include <linux/percpu-defs.h>
|
2008-01-30 15:32:52 +03:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#ifdef CONFIG_SMP
|
|
|
|
|
|
2008-01-30 15:32:52 +03:00
|
|
|
/*
|
|
|
|
|
* per_cpu_offset() is the offset that has to be added to a
|
|
|
|
|
* percpu variable to get to the instance for a certain processor.
|
|
|
|
|
*
|
|
|
|
|
* Most arches use the __per_cpu_offset array for those offsets but
|
|
|
|
|
* some arches have their own ways of determining the offset (x86_64, s390).
|
|
|
|
|
*/
|
|
|
|
|
#ifndef __per_cpu_offset
|
2005-04-17 02:20:36 +04:00
|
|
|
extern unsigned long __per_cpu_offset[NR_CPUS];
|
|
|
|
|
|
2006-07-03 11:24:26 +04:00
|
|
|
#define per_cpu_offset(x) (__per_cpu_offset[x])
|
2008-01-30 15:32:52 +03:00
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Determine the offset for the currently active processor.
|
|
|
|
|
* An arch may define __my_cpu_offset to provide a more effective
|
|
|
|
|
* means of obtaining the offset to the per cpu variables of the
|
|
|
|
|
* current processor.
|
|
|
|
|
*/
|
|
|
|
|
#ifndef __my_cpu_offset
|
|
|
|
|
#define __my_cpu_offset per_cpu_offset(raw_smp_processor_id())
|
2008-02-23 22:40:17 +03:00
|
|
|
#endif
|
|
|
|
|
#ifdef CONFIG_DEBUG_PREEMPT
|
2008-01-30 15:32:52 +03:00
|
|
|
#define my_cpu_offset per_cpu_offset(smp_processor_id())
|
|
|
|
|
#else
|
|
|
|
|
#define my_cpu_offset __my_cpu_offset
|
|
|
|
|
#endif
|
|
|
|
|
|
2014-06-18 03:12:34 +04:00
|
|
|
/*
|
|
|
|
|
* Arch may define arch_raw_cpu_ptr() to provide more efficient address
|
|
|
|
|
* translations for raw_cpu_ptr().
|
|
|
|
|
*/
|
|
|
|
|
#ifndef arch_raw_cpu_ptr
|
|
|
|
|
#define arch_raw_cpu_ptr(ptr) SHIFT_PERCPU_PTR(ptr, __my_cpu_offset)
|
2010-09-09 20:17:26 +04:00
|
|
|
#endif
|
2014-06-18 03:12:34 +04:00
|
|
|
|
2008-01-30 15:33:32 +03:00
|
|
|
#ifdef CONFIG_HAVE_SETUP_PER_CPU_AREA
|
2008-01-30 15:32:52 +03:00
|
|
|
extern void setup_per_cpu_areas(void);
|
|
|
|
|
#endif
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
|
#endif /* SMP */
|
|
|
|
|
|
2009-04-22 02:00:24 +04:00
|
|
|
#ifndef PER_CPU_BASE_SECTION
|
|
|
|
|
#ifdef CONFIG_SMP
|
2010-02-20 03:03:43 +03:00
|
|
|
#define PER_CPU_BASE_SECTION ".data..percpu"
|
2009-04-22 02:00:24 +04:00
|
|
|
#else
|
|
|
|
|
#define PER_CPU_BASE_SECTION ".data"
|
|
|
|
|
#endif
|
|
|
|
|
#endif
|
|
|
|
|
|
2008-01-30 15:32:52 +03:00
|
|
|
#ifndef PER_CPU_ATTRIBUTES
|
|
|
|
|
#define PER_CPU_ATTRIBUTES
|
|
|
|
|
#endif
|
|
|
|
|
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
#define raw_cpu_generic_read(pcp) \
|
|
|
|
|
({ \
|
|
|
|
|
*raw_cpu_ptr(&(pcp)); \
|
|
|
|
|
})
|
|
|
|
|
|
2014-06-18 03:12:39 +04:00
|
|
|
#define raw_cpu_generic_to_op(pcp, val, op) \
|
|
|
|
|
do { \
|
|
|
|
|
*raw_cpu_ptr(&(pcp)) op val; \
|
|
|
|
|
} while (0)
|
|
|
|
|
|
|
|
|
|
#define raw_cpu_generic_add_return(pcp, val) \
|
|
|
|
|
({ \
|
percpu: fix __percpu annotation in asm-generic
The generic implementation of raw_cpu_generic_add_return() is:
#define raw_cpu_generic_add_return(pcp, val) \
({ \
typeof(&(pcp)) __p = raw_cpu_ptr(&(pcp)); \
\
*__p += val; \
*__p; \
})
where the 'pcp' argument is a __percpu lvalue.
There, the variable '__p' is declared as a __percpu pointer
the type of the address of 'pcp') but:
1) the value assigned to it, the return value of raw_cpu_ptr(), is
a plain (__kernel) pointer, not a __percpu one.
2) this variable is dereferenced just after while a __percpu
pointer is implicitly __noderef.
So, fix the declaration of the 'pcp' variable to its correct type:
the plain (non-percpu) pointer corresponding to pcp's address,
using the fact that typeof() ignores the address space and the
'noderef' attribute of its agument.
Same for raw_cpu_generic_xchg(), raw_cpu_generic_cmpxchg() &
raw_cpu_generic_cmpxchg_double().
This removes 209 warnings on ARM, 525 on ARM64, 220 on x86 &
more than 2600 on ppc64 (all of them with the default config).
Cc: Tejun Heo <tj@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Reported-by: Ben Dooks <ben.dooks@codethink.co.uk>
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Acked-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Dennis Zhou <dennis@kernel.org>
2019-12-04 04:06:23 +03:00
|
|
|
typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
\
|
|
|
|
|
*__p += val; \
|
|
|
|
|
*__p; \
|
2014-06-18 03:12:39 +04:00
|
|
|
})
|
|
|
|
|
|
|
|
|
|
#define raw_cpu_generic_xchg(pcp, nval) \
|
2014-06-18 03:12:40 +04:00
|
|
|
({ \
|
percpu: fix __percpu annotation in asm-generic
The generic implementation of raw_cpu_generic_add_return() is:
#define raw_cpu_generic_add_return(pcp, val) \
({ \
typeof(&(pcp)) __p = raw_cpu_ptr(&(pcp)); \
\
*__p += val; \
*__p; \
})
where the 'pcp' argument is a __percpu lvalue.
There, the variable '__p' is declared as a __percpu pointer
the type of the address of 'pcp') but:
1) the value assigned to it, the return value of raw_cpu_ptr(), is
a plain (__kernel) pointer, not a __percpu one.
2) this variable is dereferenced just after while a __percpu
pointer is implicitly __noderef.
So, fix the declaration of the 'pcp' variable to its correct type:
the plain (non-percpu) pointer corresponding to pcp's address,
using the fact that typeof() ignores the address space and the
'noderef' attribute of its agument.
Same for raw_cpu_generic_xchg(), raw_cpu_generic_cmpxchg() &
raw_cpu_generic_cmpxchg_double().
This removes 209 warnings on ARM, 525 on ARM64, 220 on x86 &
more than 2600 on ppc64 (all of them with the default config).
Cc: Tejun Heo <tj@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Reported-by: Ben Dooks <ben.dooks@codethink.co.uk>
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Acked-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Dennis Zhou <dennis@kernel.org>
2019-12-04 04:06:23 +03:00
|
|
|
typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \
|
2014-06-18 03:12:40 +04:00
|
|
|
typeof(pcp) __ret; \
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
__ret = *__p; \
|
|
|
|
|
*__p = nval; \
|
2014-06-18 03:12:40 +04:00
|
|
|
__ret; \
|
2014-06-18 03:12:39 +04:00
|
|
|
})
|
|
|
|
|
|
|
|
|
|
#define raw_cpu_generic_cmpxchg(pcp, oval, nval) \
|
|
|
|
|
({ \
|
percpu: fix __percpu annotation in asm-generic
The generic implementation of raw_cpu_generic_add_return() is:
#define raw_cpu_generic_add_return(pcp, val) \
({ \
typeof(&(pcp)) __p = raw_cpu_ptr(&(pcp)); \
\
*__p += val; \
*__p; \
})
where the 'pcp' argument is a __percpu lvalue.
There, the variable '__p' is declared as a __percpu pointer
the type of the address of 'pcp') but:
1) the value assigned to it, the return value of raw_cpu_ptr(), is
a plain (__kernel) pointer, not a __percpu one.
2) this variable is dereferenced just after while a __percpu
pointer is implicitly __noderef.
So, fix the declaration of the 'pcp' variable to its correct type:
the plain (non-percpu) pointer corresponding to pcp's address,
using the fact that typeof() ignores the address space and the
'noderef' attribute of its agument.
Same for raw_cpu_generic_xchg(), raw_cpu_generic_cmpxchg() &
raw_cpu_generic_cmpxchg_double().
This removes 209 warnings on ARM, 525 on ARM64, 220 on x86 &
more than 2600 on ppc64 (all of them with the default config).
Cc: Tejun Heo <tj@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Reported-by: Ben Dooks <ben.dooks@codethink.co.uk>
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Acked-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Dennis Zhou <dennis@kernel.org>
2019-12-04 04:06:23 +03:00
|
|
|
typeof(pcp) *__p = raw_cpu_ptr(&(pcp)); \
|
2014-06-18 03:12:40 +04:00
|
|
|
typeof(pcp) __ret; \
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
__ret = *__p; \
|
2014-06-18 03:12:40 +04:00
|
|
|
if (__ret == (oval)) \
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
*__p = nval; \
|
2014-06-18 03:12:40 +04:00
|
|
|
__ret; \
|
2014-06-18 03:12:39 +04:00
|
|
|
})
|
|
|
|
|
|
2014-06-18 03:12:40 +04:00
|
|
|
#define raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
|
2014-06-18 03:12:39 +04:00
|
|
|
({ \
|
percpu: fix __percpu annotation in asm-generic
The generic implementation of raw_cpu_generic_add_return() is:
#define raw_cpu_generic_add_return(pcp, val) \
({ \
typeof(&(pcp)) __p = raw_cpu_ptr(&(pcp)); \
\
*__p += val; \
*__p; \
})
where the 'pcp' argument is a __percpu lvalue.
There, the variable '__p' is declared as a __percpu pointer
the type of the address of 'pcp') but:
1) the value assigned to it, the return value of raw_cpu_ptr(), is
a plain (__kernel) pointer, not a __percpu one.
2) this variable is dereferenced just after while a __percpu
pointer is implicitly __noderef.
So, fix the declaration of the 'pcp' variable to its correct type:
the plain (non-percpu) pointer corresponding to pcp's address,
using the fact that typeof() ignores the address space and the
'noderef' attribute of its agument.
Same for raw_cpu_generic_xchg(), raw_cpu_generic_cmpxchg() &
raw_cpu_generic_cmpxchg_double().
This removes 209 warnings on ARM, 525 on ARM64, 220 on x86 &
more than 2600 on ppc64 (all of them with the default config).
Cc: Tejun Heo <tj@kernel.org>
Cc: Nicholas Piggin <npiggin@gmail.com>
Reported-by: Ben Dooks <ben.dooks@codethink.co.uk>
Signed-off-by: Luc Van Oostenryck <luc.vanoostenryck@gmail.com>
Acked-by: Christoph Lameter <cl@linux.com>
Signed-off-by: Dennis Zhou <dennis@kernel.org>
2019-12-04 04:06:23 +03:00
|
|
|
typeof(pcp1) *__p1 = raw_cpu_ptr(&(pcp1)); \
|
|
|
|
|
typeof(pcp2) *__p2 = raw_cpu_ptr(&(pcp2)); \
|
2014-06-18 03:12:39 +04:00
|
|
|
int __ret = 0; \
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
if (*__p1 == (oval1) && *__p2 == (oval2)) { \
|
|
|
|
|
*__p1 = nval1; \
|
|
|
|
|
*__p2 = nval2; \
|
2014-06-18 03:12:39 +04:00
|
|
|
__ret = 1; \
|
|
|
|
|
} \
|
|
|
|
|
(__ret); \
|
|
|
|
|
})
|
|
|
|
|
|
2017-09-26 14:41:52 +03:00
|
|
|
#define __this_cpu_generic_read_nopreempt(pcp) \
|
2014-06-18 03:12:40 +04:00
|
|
|
({ \
|
2020-10-26 18:53:48 +03:00
|
|
|
typeof(pcp) ___ret; \
|
2016-11-03 15:09:24 +03:00
|
|
|
preempt_disable_notrace(); \
|
2020-10-26 18:53:48 +03:00
|
|
|
___ret = READ_ONCE(*raw_cpu_ptr(&(pcp))); \
|
2016-11-03 15:09:24 +03:00
|
|
|
preempt_enable_notrace(); \
|
2020-10-26 18:53:48 +03:00
|
|
|
___ret; \
|
2014-06-18 03:12:39 +04:00
|
|
|
})
|
|
|
|
|
|
2017-09-26 14:41:52 +03:00
|
|
|
#define __this_cpu_generic_read_noirq(pcp) \
|
|
|
|
|
({ \
|
2020-10-26 18:53:48 +03:00
|
|
|
typeof(pcp) ___ret; \
|
|
|
|
|
unsigned long ___flags; \
|
|
|
|
|
raw_local_irq_save(___flags); \
|
|
|
|
|
___ret = raw_cpu_generic_read(pcp); \
|
|
|
|
|
raw_local_irq_restore(___flags); \
|
|
|
|
|
___ret; \
|
2017-09-26 14:41:52 +03:00
|
|
|
})
|
|
|
|
|
|
|
|
|
|
#define this_cpu_generic_read(pcp) \
|
|
|
|
|
({ \
|
|
|
|
|
typeof(pcp) __ret; \
|
|
|
|
|
if (__native_word(pcp)) \
|
|
|
|
|
__ret = __this_cpu_generic_read_nopreempt(pcp); \
|
|
|
|
|
else \
|
|
|
|
|
__ret = __this_cpu_generic_read_noirq(pcp); \
|
|
|
|
|
__ret; \
|
|
|
|
|
})
|
|
|
|
|
|
2014-06-18 03:12:40 +04:00
|
|
|
#define this_cpu_generic_to_op(pcp, val, op) \
|
2014-06-18 03:12:39 +04:00
|
|
|
do { \
|
2014-06-18 03:12:40 +04:00
|
|
|
unsigned long __flags; \
|
|
|
|
|
raw_local_irq_save(__flags); \
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
raw_cpu_generic_to_op(pcp, val, op); \
|
2014-06-18 03:12:40 +04:00
|
|
|
raw_local_irq_restore(__flags); \
|
2014-06-18 03:12:39 +04:00
|
|
|
} while (0)
|
|
|
|
|
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
|
2014-06-18 03:12:40 +04:00
|
|
|
#define this_cpu_generic_add_return(pcp, val) \
|
2014-06-18 03:12:39 +04:00
|
|
|
({ \
|
2014-06-18 03:12:40 +04:00
|
|
|
typeof(pcp) __ret; \
|
|
|
|
|
unsigned long __flags; \
|
|
|
|
|
raw_local_irq_save(__flags); \
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
__ret = raw_cpu_generic_add_return(pcp, val); \
|
2014-06-18 03:12:40 +04:00
|
|
|
raw_local_irq_restore(__flags); \
|
|
|
|
|
__ret; \
|
2014-06-18 03:12:39 +04:00
|
|
|
})
|
|
|
|
|
|
2014-06-18 03:12:40 +04:00
|
|
|
#define this_cpu_generic_xchg(pcp, nval) \
|
|
|
|
|
({ \
|
|
|
|
|
typeof(pcp) __ret; \
|
|
|
|
|
unsigned long __flags; \
|
|
|
|
|
raw_local_irq_save(__flags); \
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
__ret = raw_cpu_generic_xchg(pcp, nval); \
|
2014-06-18 03:12:40 +04:00
|
|
|
raw_local_irq_restore(__flags); \
|
|
|
|
|
__ret; \
|
2014-06-18 03:12:39 +04:00
|
|
|
})
|
|
|
|
|
|
2014-06-18 03:12:40 +04:00
|
|
|
#define this_cpu_generic_cmpxchg(pcp, oval, nval) \
|
2014-06-18 03:12:39 +04:00
|
|
|
({ \
|
2014-06-18 03:12:40 +04:00
|
|
|
typeof(pcp) __ret; \
|
|
|
|
|
unsigned long __flags; \
|
|
|
|
|
raw_local_irq_save(__flags); \
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
__ret = raw_cpu_generic_cmpxchg(pcp, oval, nval); \
|
2014-06-18 03:12:40 +04:00
|
|
|
raw_local_irq_restore(__flags); \
|
|
|
|
|
__ret; \
|
2014-06-18 03:12:39 +04:00
|
|
|
})
|
|
|
|
|
|
2014-06-18 03:12:40 +04:00
|
|
|
#define this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2) \
|
2014-06-18 03:12:39 +04:00
|
|
|
({ \
|
2014-06-18 03:12:40 +04:00
|
|
|
int __ret; \
|
|
|
|
|
unsigned long __flags; \
|
|
|
|
|
raw_local_irq_save(__flags); \
|
|
|
|
|
__ret = raw_cpu_generic_cmpxchg_double(pcp1, pcp2, \
|
2014-06-18 03:12:39 +04:00
|
|
|
oval1, oval2, nval1, nval2); \
|
2014-06-18 03:12:40 +04:00
|
|
|
raw_local_irq_restore(__flags); \
|
|
|
|
|
__ret; \
|
2014-06-18 03:12:39 +04:00
|
|
|
})
|
|
|
|
|
|
2014-06-18 03:12:40 +04:00
|
|
|
#ifndef raw_cpu_read_1
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
#define raw_cpu_read_1(pcp) raw_cpu_generic_read(pcp)
|
2014-06-18 03:12:40 +04:00
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_read_2
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
#define raw_cpu_read_2(pcp) raw_cpu_generic_read(pcp)
|
2014-06-18 03:12:40 +04:00
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_read_4
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
#define raw_cpu_read_4(pcp) raw_cpu_generic_read(pcp)
|
2014-06-18 03:12:40 +04:00
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_read_8
|
percpu: improve generic percpu modify-return implementation
Some architectures require an additional load to find the address of
percpu pointers. In some implemenatations, the C aliasing rules do not
allow the result of that load to be kept over the store that modifies
the percpu variable, which causes additional loads.
Work around this by finding the pointer first, then operating on that.
It's also possible to mark things as restrict and those kind of games,
but that can require larger and arch specific changes.
On powerpc, __this_cpu_inc_return compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
ld 9,48(13)
ldx 3,9,3
With this patch it compiles to:
ld 10,48(13)
ldx 9,3,10
addi 9,9,1
stdx 9,3,10
Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
To: Tejun Heo <tj@kernel.org>
To: Christoph Lameter <cl@linux.com>
Cc: linux-kernel@vger.kernel.org
Cc: linux-arch@vger.kernel.org
Signed-off-by: Tejun Heo <tj@kernel.org>
2016-09-22 18:55:54 +03:00
|
|
|
#define raw_cpu_read_8(pcp) raw_cpu_generic_read(pcp)
|
2014-06-18 03:12:40 +04:00
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef raw_cpu_write_1
|
|
|
|
|
#define raw_cpu_write_1(pcp, val) raw_cpu_generic_to_op(pcp, val, =)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_write_2
|
|
|
|
|
#define raw_cpu_write_2(pcp, val) raw_cpu_generic_to_op(pcp, val, =)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_write_4
|
|
|
|
|
#define raw_cpu_write_4(pcp, val) raw_cpu_generic_to_op(pcp, val, =)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_write_8
|
|
|
|
|
#define raw_cpu_write_8(pcp, val) raw_cpu_generic_to_op(pcp, val, =)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef raw_cpu_add_1
|
|
|
|
|
#define raw_cpu_add_1(pcp, val) raw_cpu_generic_to_op(pcp, val, +=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_add_2
|
|
|
|
|
#define raw_cpu_add_2(pcp, val) raw_cpu_generic_to_op(pcp, val, +=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_add_4
|
|
|
|
|
#define raw_cpu_add_4(pcp, val) raw_cpu_generic_to_op(pcp, val, +=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_add_8
|
|
|
|
|
#define raw_cpu_add_8(pcp, val) raw_cpu_generic_to_op(pcp, val, +=)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef raw_cpu_and_1
|
|
|
|
|
#define raw_cpu_and_1(pcp, val) raw_cpu_generic_to_op(pcp, val, &=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_and_2
|
|
|
|
|
#define raw_cpu_and_2(pcp, val) raw_cpu_generic_to_op(pcp, val, &=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_and_4
|
|
|
|
|
#define raw_cpu_and_4(pcp, val) raw_cpu_generic_to_op(pcp, val, &=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_and_8
|
|
|
|
|
#define raw_cpu_and_8(pcp, val) raw_cpu_generic_to_op(pcp, val, &=)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef raw_cpu_or_1
|
|
|
|
|
#define raw_cpu_or_1(pcp, val) raw_cpu_generic_to_op(pcp, val, |=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_or_2
|
|
|
|
|
#define raw_cpu_or_2(pcp, val) raw_cpu_generic_to_op(pcp, val, |=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_or_4
|
|
|
|
|
#define raw_cpu_or_4(pcp, val) raw_cpu_generic_to_op(pcp, val, |=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_or_8
|
|
|
|
|
#define raw_cpu_or_8(pcp, val) raw_cpu_generic_to_op(pcp, val, |=)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef raw_cpu_add_return_1
|
|
|
|
|
#define raw_cpu_add_return_1(pcp, val) raw_cpu_generic_add_return(pcp, val)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_add_return_2
|
|
|
|
|
#define raw_cpu_add_return_2(pcp, val) raw_cpu_generic_add_return(pcp, val)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_add_return_4
|
|
|
|
|
#define raw_cpu_add_return_4(pcp, val) raw_cpu_generic_add_return(pcp, val)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_add_return_8
|
|
|
|
|
#define raw_cpu_add_return_8(pcp, val) raw_cpu_generic_add_return(pcp, val)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef raw_cpu_xchg_1
|
|
|
|
|
#define raw_cpu_xchg_1(pcp, nval) raw_cpu_generic_xchg(pcp, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_xchg_2
|
|
|
|
|
#define raw_cpu_xchg_2(pcp, nval) raw_cpu_generic_xchg(pcp, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_xchg_4
|
|
|
|
|
#define raw_cpu_xchg_4(pcp, nval) raw_cpu_generic_xchg(pcp, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_xchg_8
|
|
|
|
|
#define raw_cpu_xchg_8(pcp, nval) raw_cpu_generic_xchg(pcp, nval)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef raw_cpu_cmpxchg_1
|
|
|
|
|
#define raw_cpu_cmpxchg_1(pcp, oval, nval) \
|
|
|
|
|
raw_cpu_generic_cmpxchg(pcp, oval, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_cmpxchg_2
|
|
|
|
|
#define raw_cpu_cmpxchg_2(pcp, oval, nval) \
|
|
|
|
|
raw_cpu_generic_cmpxchg(pcp, oval, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_cmpxchg_4
|
|
|
|
|
#define raw_cpu_cmpxchg_4(pcp, oval, nval) \
|
|
|
|
|
raw_cpu_generic_cmpxchg(pcp, oval, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_cmpxchg_8
|
|
|
|
|
#define raw_cpu_cmpxchg_8(pcp, oval, nval) \
|
|
|
|
|
raw_cpu_generic_cmpxchg(pcp, oval, nval)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef raw_cpu_cmpxchg_double_1
|
|
|
|
|
#define raw_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \
|
2014-06-18 03:12:39 +04:00
|
|
|
raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
|
2014-06-18 03:12:40 +04:00
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_cmpxchg_double_2
|
|
|
|
|
#define raw_cpu_cmpxchg_double_2(pcp1, pcp2, oval1, oval2, nval1, nval2) \
|
2014-06-18 03:12:39 +04:00
|
|
|
raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
|
2014-06-18 03:12:40 +04:00
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_cmpxchg_double_4
|
|
|
|
|
#define raw_cpu_cmpxchg_double_4(pcp1, pcp2, oval1, oval2, nval1, nval2) \
|
2014-06-18 03:12:39 +04:00
|
|
|
raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
|
2014-06-18 03:12:40 +04:00
|
|
|
#endif
|
|
|
|
|
#ifndef raw_cpu_cmpxchg_double_8
|
|
|
|
|
#define raw_cpu_cmpxchg_double_8(pcp1, pcp2, oval1, oval2, nval1, nval2) \
|
2014-06-18 03:12:39 +04:00
|
|
|
raw_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
|
2014-06-18 03:12:40 +04:00
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef this_cpu_read_1
|
|
|
|
|
#define this_cpu_read_1(pcp) this_cpu_generic_read(pcp)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_read_2
|
|
|
|
|
#define this_cpu_read_2(pcp) this_cpu_generic_read(pcp)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_read_4
|
|
|
|
|
#define this_cpu_read_4(pcp) this_cpu_generic_read(pcp)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_read_8
|
|
|
|
|
#define this_cpu_read_8(pcp) this_cpu_generic_read(pcp)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef this_cpu_write_1
|
|
|
|
|
#define this_cpu_write_1(pcp, val) this_cpu_generic_to_op(pcp, val, =)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_write_2
|
|
|
|
|
#define this_cpu_write_2(pcp, val) this_cpu_generic_to_op(pcp, val, =)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_write_4
|
|
|
|
|
#define this_cpu_write_4(pcp, val) this_cpu_generic_to_op(pcp, val, =)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_write_8
|
|
|
|
|
#define this_cpu_write_8(pcp, val) this_cpu_generic_to_op(pcp, val, =)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef this_cpu_add_1
|
|
|
|
|
#define this_cpu_add_1(pcp, val) this_cpu_generic_to_op(pcp, val, +=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_add_2
|
|
|
|
|
#define this_cpu_add_2(pcp, val) this_cpu_generic_to_op(pcp, val, +=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_add_4
|
|
|
|
|
#define this_cpu_add_4(pcp, val) this_cpu_generic_to_op(pcp, val, +=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_add_8
|
|
|
|
|
#define this_cpu_add_8(pcp, val) this_cpu_generic_to_op(pcp, val, +=)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef this_cpu_and_1
|
|
|
|
|
#define this_cpu_and_1(pcp, val) this_cpu_generic_to_op(pcp, val, &=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_and_2
|
|
|
|
|
#define this_cpu_and_2(pcp, val) this_cpu_generic_to_op(pcp, val, &=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_and_4
|
|
|
|
|
#define this_cpu_and_4(pcp, val) this_cpu_generic_to_op(pcp, val, &=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_and_8
|
|
|
|
|
#define this_cpu_and_8(pcp, val) this_cpu_generic_to_op(pcp, val, &=)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef this_cpu_or_1
|
|
|
|
|
#define this_cpu_or_1(pcp, val) this_cpu_generic_to_op(pcp, val, |=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_or_2
|
|
|
|
|
#define this_cpu_or_2(pcp, val) this_cpu_generic_to_op(pcp, val, |=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_or_4
|
|
|
|
|
#define this_cpu_or_4(pcp, val) this_cpu_generic_to_op(pcp, val, |=)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_or_8
|
|
|
|
|
#define this_cpu_or_8(pcp, val) this_cpu_generic_to_op(pcp, val, |=)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef this_cpu_add_return_1
|
|
|
|
|
#define this_cpu_add_return_1(pcp, val) this_cpu_generic_add_return(pcp, val)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_add_return_2
|
|
|
|
|
#define this_cpu_add_return_2(pcp, val) this_cpu_generic_add_return(pcp, val)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_add_return_4
|
|
|
|
|
#define this_cpu_add_return_4(pcp, val) this_cpu_generic_add_return(pcp, val)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_add_return_8
|
|
|
|
|
#define this_cpu_add_return_8(pcp, val) this_cpu_generic_add_return(pcp, val)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef this_cpu_xchg_1
|
|
|
|
|
#define this_cpu_xchg_1(pcp, nval) this_cpu_generic_xchg(pcp, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_xchg_2
|
|
|
|
|
#define this_cpu_xchg_2(pcp, nval) this_cpu_generic_xchg(pcp, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_xchg_4
|
|
|
|
|
#define this_cpu_xchg_4(pcp, nval) this_cpu_generic_xchg(pcp, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_xchg_8
|
|
|
|
|
#define this_cpu_xchg_8(pcp, nval) this_cpu_generic_xchg(pcp, nval)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef this_cpu_cmpxchg_1
|
|
|
|
|
#define this_cpu_cmpxchg_1(pcp, oval, nval) \
|
|
|
|
|
this_cpu_generic_cmpxchg(pcp, oval, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_cmpxchg_2
|
|
|
|
|
#define this_cpu_cmpxchg_2(pcp, oval, nval) \
|
|
|
|
|
this_cpu_generic_cmpxchg(pcp, oval, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_cmpxchg_4
|
|
|
|
|
#define this_cpu_cmpxchg_4(pcp, oval, nval) \
|
|
|
|
|
this_cpu_generic_cmpxchg(pcp, oval, nval)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_cmpxchg_8
|
|
|
|
|
#define this_cpu_cmpxchg_8(pcp, oval, nval) \
|
|
|
|
|
this_cpu_generic_cmpxchg(pcp, oval, nval)
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
#ifndef this_cpu_cmpxchg_double_1
|
|
|
|
|
#define this_cpu_cmpxchg_double_1(pcp1, pcp2, oval1, oval2, nval1, nval2) \
|
|
|
|
|
this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_cmpxchg_double_2
|
|
|
|
|
#define this_cpu_cmpxchg_double_2(pcp1, pcp2, oval1, oval2, nval1, nval2) \
|
|
|
|
|
this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_cmpxchg_double_4
|
|
|
|
|
#define this_cpu_cmpxchg_double_4(pcp1, pcp2, oval1, oval2, nval1, nval2) \
|
|
|
|
|
this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
|
|
|
|
|
#endif
|
|
|
|
|
#ifndef this_cpu_cmpxchg_double_8
|
|
|
|
|
#define this_cpu_cmpxchg_double_8(pcp1, pcp2, oval1, oval2, nval1, nval2) \
|
|
|
|
|
this_cpu_generic_cmpxchg_double(pcp1, pcp2, oval1, oval2, nval1, nval2)
|
|
|
|
|
#endif
|
2014-06-18 03:12:39 +04:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif /* _ASM_GENERIC_PERCPU_H_ */
|
