2005-04-17 02:20:36 +04:00
|
|
|
#ifndef __LINUX_CPUMASK_H
|
|
|
|
#define __LINUX_CPUMASK_H
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Cpumasks provide a bitmap suitable for representing the
|
|
|
|
* set of CPU's in a system, one bit position per CPU number.
|
|
|
|
*
|
|
|
|
* See detailed comments in the file linux/bitmap.h describing the
|
|
|
|
* data type on which these cpumasks are based.
|
|
|
|
*
|
2006-10-11 12:21:55 +04:00
|
|
|
* For details of cpumask_scnprintf() and cpumask_parse_user(),
|
|
|
|
* see bitmap_scnprintf() and bitmap_parse_user() in lib/bitmap.c.
|
2005-04-17 02:20:36 +04:00
|
|
|
* For details of cpulist_scnprintf() and cpulist_parse(), see
|
|
|
|
* bitmap_scnlistprintf() and bitmap_parselist(), also in bitmap.c.
|
2005-10-31 02:02:33 +03:00
|
|
|
* For details of cpu_remap(), see bitmap_bitremap in lib/bitmap.c
|
|
|
|
* For details of cpus_remap(), see bitmap_remap in lib/bitmap.c.
|
mempolicy: add bitmap_onto() and bitmap_fold() operations
The following adds two more bitmap operators, bitmap_onto() and bitmap_fold(),
with the usual cpumask and nodemask wrappers.
The bitmap_onto() operator computes one bitmap relative to another. If the
n-th bit in the origin mask is set, then the m-th bit of the destination mask
will be set, where m is the position of the n-th set bit in the relative mask.
The bitmap_fold() operator folds a bitmap into a second that has bit m set iff
the input bitmap has some bit n set, where m == n mod sz, for the specified sz
value.
There are two substantive changes between this patch and its
predecessor bitmap_relative:
1) Renamed bitmap_relative() to be bitmap_onto().
2) Added bitmap_fold().
The essential motivation for bitmap_onto() is to provide a mechanism for
converting a cpuset-relative CPU or Node mask to an absolute mask. Cpuset
relative masks are written as if the current task were in a cpuset whose CPUs
or Nodes were just the consecutive ones numbered 0..N-1, for some N. The
bitmap_onto() operator is provided in anticipation of adding support for the
first such cpuset relative mask, by the mbind() and set_mempolicy() system
calls, using a planned flag of MPOL_F_RELATIVE_NODES. These bitmap operators
(and their nodemask wrappers, in particular) will be used in code that
converts the user specified cpuset relative memory policy to a specific system
node numbered policy, given the current mems_allowed of the tasks cpuset.
Such cpuset relative mempolicies will address two deficiencies
of the existing interface between cpusets and mempolicies:
1) A task cannot at present reliably establish a cpuset
relative mempolicy because there is an essential race
condition, in that the tasks cpuset may be changed in
between the time the task can query its cpuset placement,
and the time the task can issue the applicable mbind or
set_memplicy system call.
2) A task cannot at present establish what cpuset relative
mempolicy it would like to have, if it is in a smaller
cpuset than it might have mempolicy preferences for,
because the existing interface only allows specifying
mempolicies for nodes currently allowed by the cpuset.
Cpuset relative mempolicies are useful for tasks that don't distinguish
particularly between one CPU or Node and another, but only between how many of
each are allowed, and the proper placement of threads and memory pages on the
various CPUs and Nodes available.
The motivation for the added bitmap_fold() can be seen in the following
example.
Let's say an application has specified some mempolicies that presume 16 memory
nodes, including say a mempolicy that specified MPOL_F_RELATIVE_NODES (cpuset
relative) nodes 12-15. Then lets say that application is crammed into a
cpuset that only has 8 memory nodes, 0-7. If one just uses bitmap_onto(),
this mempolicy, mapped to that cpuset, would ignore the requested relative
nodes above 7, leaving it empty of nodes. That's not good; better to fold the
higher nodes down, so that some nodes are included in the resulting mapped
mempolicy. In this case, the mempolicy nodes 12-15 are taken modulo 8 (the
weight of the mems_allowed of the confining cpuset), resulting in a mempolicy
specifying nodes 4-7.
Signed-off-by: Paul Jackson <pj@sgi.com>
Signed-off-by: David Rientjes <rientjes@google.com>
Cc: Christoph Lameter <clameter@sgi.com>
Cc: Andi Kleen <ak@suse.de>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: <kosaki.motohiro@jp.fujitsu.com>
Cc: <ray-lk@madrabbit.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-04-28 13:12:29 +04:00
|
|
|
* For details of cpus_onto(), see bitmap_onto in lib/bitmap.c.
|
|
|
|
* For details of cpus_fold(), see bitmap_fold in lib/bitmap.c.
|
2005-04-17 02:20:36 +04:00
|
|
|
*
|
2008-05-12 23:21:13 +04:00
|
|
|
* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
|
|
|
|
* Note: The alternate operations with the suffix "_nr" are used
|
|
|
|
* to limit the range of the loop to nr_cpu_ids instead of
|
|
|
|
* NR_CPUS when NR_CPUS > 64 for performance reasons.
|
|
|
|
* If NR_CPUS is <= 64 then most assembler bitmask
|
|
|
|
* operators execute faster with a constant range, so
|
|
|
|
* the operator will continue to use NR_CPUS.
|
|
|
|
*
|
|
|
|
* Another consideration is that nr_cpu_ids is initialized
|
|
|
|
* to NR_CPUS and isn't lowered until the possible cpus are
|
|
|
|
* discovered (including any disabled cpus). So early uses
|
|
|
|
* will span the entire range of NR_CPUS.
|
|
|
|
* . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
|
|
|
|
*
|
2005-04-17 02:20:36 +04:00
|
|
|
* The available cpumask operations are:
|
|
|
|
*
|
|
|
|
* void cpu_set(cpu, mask) turn on bit 'cpu' in mask
|
|
|
|
* void cpu_clear(cpu, mask) turn off bit 'cpu' in mask
|
|
|
|
* void cpus_setall(mask) set all bits
|
|
|
|
* void cpus_clear(mask) clear all bits
|
|
|
|
* int cpu_isset(cpu, mask) true iff bit 'cpu' set in mask
|
|
|
|
* int cpu_test_and_set(cpu, mask) test and set bit 'cpu' in mask
|
|
|
|
*
|
|
|
|
* void cpus_and(dst, src1, src2) dst = src1 & src2 [intersection]
|
|
|
|
* void cpus_or(dst, src1, src2) dst = src1 | src2 [union]
|
|
|
|
* void cpus_xor(dst, src1, src2) dst = src1 ^ src2
|
|
|
|
* void cpus_andnot(dst, src1, src2) dst = src1 & ~src2
|
|
|
|
* void cpus_complement(dst, src) dst = ~src
|
|
|
|
*
|
|
|
|
* int cpus_equal(mask1, mask2) Does mask1 == mask2?
|
|
|
|
* int cpus_intersects(mask1, mask2) Do mask1 and mask2 intersect?
|
|
|
|
* int cpus_subset(mask1, mask2) Is mask1 a subset of mask2?
|
|
|
|
* int cpus_empty(mask) Is mask empty (no bits sets)?
|
|
|
|
* int cpus_full(mask) Is mask full (all bits sets)?
|
|
|
|
* int cpus_weight(mask) Hamming weigh - number of set bits
|
2008-05-12 23:21:13 +04:00
|
|
|
* int cpus_weight_nr(mask) Same using nr_cpu_ids instead of NR_CPUS
|
2005-04-17 02:20:36 +04:00
|
|
|
*
|
|
|
|
* void cpus_shift_right(dst, src, n) Shift right
|
|
|
|
* void cpus_shift_left(dst, src, n) Shift left
|
|
|
|
*
|
|
|
|
* int first_cpu(mask) Number lowest set bit, or NR_CPUS
|
|
|
|
* int next_cpu(cpu, mask) Next cpu past 'cpu', or NR_CPUS
|
2008-05-12 23:21:13 +04:00
|
|
|
* int next_cpu_nr(cpu, mask) Next cpu past 'cpu', or nr_cpu_ids
|
2005-04-17 02:20:36 +04:00
|
|
|
*
|
|
|
|
* cpumask_t cpumask_of_cpu(cpu) Return cpumask with bit 'cpu' set
|
2008-07-16 01:14:30 +04:00
|
|
|
*ifdef CONFIG_HAS_CPUMASK_OF_CPU
|
|
|
|
* cpumask_of_cpu_ptr_declare(v) Declares cpumask_t *v
|
|
|
|
* cpumask_of_cpu_ptr_next(v, cpu) Sets v = &cpumask_of_cpu_map[cpu]
|
|
|
|
* cpumask_of_cpu_ptr(v, cpu) Combines above two operations
|
|
|
|
*else
|
|
|
|
* cpumask_of_cpu_ptr_declare(v) Declares cpumask_t _v and *v = &_v
|
|
|
|
* cpumask_of_cpu_ptr_next(v, cpu) Sets _v = cpumask_of_cpu(cpu)
|
|
|
|
* cpumask_of_cpu_ptr(v, cpu) Combines above two operations
|
|
|
|
*endif
|
2005-04-17 02:20:36 +04:00
|
|
|
* CPU_MASK_ALL Initializer - all bits set
|
|
|
|
* CPU_MASK_NONE Initializer - no bits set
|
|
|
|
* unsigned long *cpus_addr(mask) Array of unsigned long's in mask
|
|
|
|
*
|
2008-07-19 05:11:33 +04:00
|
|
|
* CPUMASK_ALLOC kmalloc's a structure that is a composite of many cpumask_t
|
|
|
|
* variables, and CPUMASK_PTR provides pointers to each field.
|
|
|
|
*
|
|
|
|
* The structure should be defined something like this:
|
|
|
|
* struct my_cpumasks {
|
|
|
|
* cpumask_t mask1;
|
|
|
|
* cpumask_t mask2;
|
|
|
|
* };
|
|
|
|
*
|
|
|
|
* Usage is then:
|
|
|
|
* CPUMASK_ALLOC(my_cpumasks);
|
|
|
|
* CPUMASK_PTR(mask1, my_cpumasks);
|
|
|
|
* CPUMASK_PTR(mask2, my_cpumasks);
|
|
|
|
*
|
|
|
|
* --- DO NOT reference cpumask_t pointers until this check ---
|
|
|
|
* if (my_cpumasks == NULL)
|
|
|
|
* "kmalloc failed"...
|
|
|
|
*
|
|
|
|
* References are now pointers to the cpumask_t variables (*mask1, ...)
|
|
|
|
*
|
2008-07-16 01:14:36 +04:00
|
|
|
*if NR_CPUS > BITS_PER_LONG
|
|
|
|
* CPUMASK_ALLOC(m) Declares and allocates struct m *m =
|
2008-07-19 05:11:33 +04:00
|
|
|
* kmalloc(sizeof(*m), GFP_KERNEL)
|
|
|
|
* CPUMASK_FREE(m) Macro for kfree(m)
|
2008-07-16 01:14:36 +04:00
|
|
|
*else
|
|
|
|
* CPUMASK_ALLOC(m) Declares struct m _m, *m = &_m
|
|
|
|
* CPUMASK_FREE(m) Nop
|
|
|
|
*endif
|
2008-07-19 05:11:33 +04:00
|
|
|
* CPUMASK_PTR(v, m) Declares cpumask_t *v = &(m->v)
|
|
|
|
* ------------------------------------------------------------------------
|
2008-07-16 01:14:36 +04:00
|
|
|
*
|
2005-04-17 02:20:36 +04:00
|
|
|
* int cpumask_scnprintf(buf, len, mask) Format cpumask for printing
|
2006-10-11 12:21:55 +04:00
|
|
|
* int cpumask_parse_user(ubuf, ulen, mask) Parse ascii string as cpumask
|
2005-04-17 02:20:36 +04:00
|
|
|
* int cpulist_scnprintf(buf, len, mask) Format cpumask as list for printing
|
|
|
|
* int cpulist_parse(buf, map) Parse ascii string as cpulist
|
2005-10-31 02:02:33 +03:00
|
|
|
* int cpu_remap(oldbit, old, new) newbit = map(old, new)(oldbit)
|
mempolicy: add bitmap_onto() and bitmap_fold() operations
The following adds two more bitmap operators, bitmap_onto() and bitmap_fold(),
with the usual cpumask and nodemask wrappers.
The bitmap_onto() operator computes one bitmap relative to another. If the
n-th bit in the origin mask is set, then the m-th bit of the destination mask
will be set, where m is the position of the n-th set bit in the relative mask.
The bitmap_fold() operator folds a bitmap into a second that has bit m set iff
the input bitmap has some bit n set, where m == n mod sz, for the specified sz
value.
There are two substantive changes between this patch and its
predecessor bitmap_relative:
1) Renamed bitmap_relative() to be bitmap_onto().
2) Added bitmap_fold().
The essential motivation for bitmap_onto() is to provide a mechanism for
converting a cpuset-relative CPU or Node mask to an absolute mask. Cpuset
relative masks are written as if the current task were in a cpuset whose CPUs
or Nodes were just the consecutive ones numbered 0..N-1, for some N. The
bitmap_onto() operator is provided in anticipation of adding support for the
first such cpuset relative mask, by the mbind() and set_mempolicy() system
calls, using a planned flag of MPOL_F_RELATIVE_NODES. These bitmap operators
(and their nodemask wrappers, in particular) will be used in code that
converts the user specified cpuset relative memory policy to a specific system
node numbered policy, given the current mems_allowed of the tasks cpuset.
Such cpuset relative mempolicies will address two deficiencies
of the existing interface between cpusets and mempolicies:
1) A task cannot at present reliably establish a cpuset
relative mempolicy because there is an essential race
condition, in that the tasks cpuset may be changed in
between the time the task can query its cpuset placement,
and the time the task can issue the applicable mbind or
set_memplicy system call.
2) A task cannot at present establish what cpuset relative
mempolicy it would like to have, if it is in a smaller
cpuset than it might have mempolicy preferences for,
because the existing interface only allows specifying
mempolicies for nodes currently allowed by the cpuset.
Cpuset relative mempolicies are useful for tasks that don't distinguish
particularly between one CPU or Node and another, but only between how many of
each are allowed, and the proper placement of threads and memory pages on the
various CPUs and Nodes available.
The motivation for the added bitmap_fold() can be seen in the following
example.
Let's say an application has specified some mempolicies that presume 16 memory
nodes, including say a mempolicy that specified MPOL_F_RELATIVE_NODES (cpuset
relative) nodes 12-15. Then lets say that application is crammed into a
cpuset that only has 8 memory nodes, 0-7. If one just uses bitmap_onto(),
this mempolicy, mapped to that cpuset, would ignore the requested relative
nodes above 7, leaving it empty of nodes. That's not good; better to fold the
higher nodes down, so that some nodes are included in the resulting mapped
mempolicy. In this case, the mempolicy nodes 12-15 are taken modulo 8 (the
weight of the mems_allowed of the confining cpuset), resulting in a mempolicy
specifying nodes 4-7.
Signed-off-by: Paul Jackson <pj@sgi.com>
Signed-off-by: David Rientjes <rientjes@google.com>
Cc: Christoph Lameter <clameter@sgi.com>
Cc: Andi Kleen <ak@suse.de>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: <kosaki.motohiro@jp.fujitsu.com>
Cc: <ray-lk@madrabbit.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-04-28 13:12:29 +04:00
|
|
|
* void cpus_remap(dst, src, old, new) *dst = map(old, new)(src)
|
|
|
|
* void cpus_onto(dst, orig, relmap) *dst = orig relative to relmap
|
|
|
|
* void cpus_fold(dst, orig, sz) dst bits = orig bits mod sz
|
2005-04-17 02:20:36 +04:00
|
|
|
*
|
2008-05-12 23:21:13 +04:00
|
|
|
* for_each_cpu_mask(cpu, mask) for-loop cpu over mask using NR_CPUS
|
|
|
|
* for_each_cpu_mask_nr(cpu, mask) for-loop cpu over mask using nr_cpu_ids
|
2005-04-17 02:20:36 +04:00
|
|
|
*
|
|
|
|
* int num_online_cpus() Number of online CPUs
|
|
|
|
* int num_possible_cpus() Number of all possible CPUs
|
|
|
|
* int num_present_cpus() Number of present CPUs
|
|
|
|
*
|
|
|
|
* int cpu_online(cpu) Is some cpu online?
|
|
|
|
* int cpu_possible(cpu) Is some cpu possible?
|
|
|
|
* int cpu_present(cpu) Is some cpu present (can schedule)?
|
|
|
|
*
|
|
|
|
* int any_online_cpu(mask) First online cpu in mask
|
|
|
|
*
|
2006-03-28 13:56:36 +04:00
|
|
|
* for_each_possible_cpu(cpu) for-loop cpu over cpu_possible_map
|
2005-04-17 02:20:36 +04:00
|
|
|
* for_each_online_cpu(cpu) for-loop cpu over cpu_online_map
|
|
|
|
* for_each_present_cpu(cpu) for-loop cpu over cpu_present_map
|
|
|
|
*
|
|
|
|
* Subtlety:
|
|
|
|
* 1) The 'type-checked' form of cpu_isset() causes gcc (3.3.2, anyway)
|
|
|
|
* to generate slightly worse code. Note for example the additional
|
|
|
|
* 40 lines of assembly code compiling the "for each possible cpu"
|
|
|
|
* loops buried in the disk_stat_read() macros calls when compiling
|
|
|
|
* drivers/block/genhd.c (arch i386, CONFIG_SMP=y). So use a simple
|
|
|
|
* one-line #define for cpu_isset(), instead of wrapping an inline
|
|
|
|
* inside a macro, the way we do the other calls.
|
|
|
|
*/
|
|
|
|
|
|
|
|
#include <linux/kernel.h>
|
|
|
|
#include <linux/threads.h>
|
|
|
|
#include <linux/bitmap.h>
|
|
|
|
|
|
|
|
typedef struct { DECLARE_BITMAP(bits, NR_CPUS); } cpumask_t;
|
|
|
|
extern cpumask_t _unused_cpumask_arg_;
|
|
|
|
|
|
|
|
#define cpu_set(cpu, dst) __cpu_set((cpu), &(dst))
|
|
|
|
static inline void __cpu_set(int cpu, volatile cpumask_t *dstp)
|
|
|
|
{
|
|
|
|
set_bit(cpu, dstp->bits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpu_clear(cpu, dst) __cpu_clear((cpu), &(dst))
|
|
|
|
static inline void __cpu_clear(int cpu, volatile cpumask_t *dstp)
|
|
|
|
{
|
|
|
|
clear_bit(cpu, dstp->bits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_setall(dst) __cpus_setall(&(dst), NR_CPUS)
|
|
|
|
static inline void __cpus_setall(cpumask_t *dstp, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_fill(dstp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_clear(dst) __cpus_clear(&(dst), NR_CPUS)
|
|
|
|
static inline void __cpus_clear(cpumask_t *dstp, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_zero(dstp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* No static inline type checking - see Subtlety (1) above. */
|
|
|
|
#define cpu_isset(cpu, cpumask) test_bit((cpu), (cpumask).bits)
|
|
|
|
|
|
|
|
#define cpu_test_and_set(cpu, cpumask) __cpu_test_and_set((cpu), &(cpumask))
|
|
|
|
static inline int __cpu_test_and_set(int cpu, cpumask_t *addr)
|
|
|
|
{
|
|
|
|
return test_and_set_bit(cpu, addr->bits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_and(dst, src1, src2) __cpus_and(&(dst), &(src1), &(src2), NR_CPUS)
|
|
|
|
static inline void __cpus_and(cpumask_t *dstp, const cpumask_t *src1p,
|
|
|
|
const cpumask_t *src2p, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_and(dstp->bits, src1p->bits, src2p->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_or(dst, src1, src2) __cpus_or(&(dst), &(src1), &(src2), NR_CPUS)
|
|
|
|
static inline void __cpus_or(cpumask_t *dstp, const cpumask_t *src1p,
|
|
|
|
const cpumask_t *src2p, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_or(dstp->bits, src1p->bits, src2p->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_xor(dst, src1, src2) __cpus_xor(&(dst), &(src1), &(src2), NR_CPUS)
|
|
|
|
static inline void __cpus_xor(cpumask_t *dstp, const cpumask_t *src1p,
|
|
|
|
const cpumask_t *src2p, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_xor(dstp->bits, src1p->bits, src2p->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_andnot(dst, src1, src2) \
|
|
|
|
__cpus_andnot(&(dst), &(src1), &(src2), NR_CPUS)
|
|
|
|
static inline void __cpus_andnot(cpumask_t *dstp, const cpumask_t *src1p,
|
|
|
|
const cpumask_t *src2p, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_andnot(dstp->bits, src1p->bits, src2p->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_complement(dst, src) __cpus_complement(&(dst), &(src), NR_CPUS)
|
|
|
|
static inline void __cpus_complement(cpumask_t *dstp,
|
|
|
|
const cpumask_t *srcp, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_complement(dstp->bits, srcp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_equal(src1, src2) __cpus_equal(&(src1), &(src2), NR_CPUS)
|
|
|
|
static inline int __cpus_equal(const cpumask_t *src1p,
|
|
|
|
const cpumask_t *src2p, int nbits)
|
|
|
|
{
|
|
|
|
return bitmap_equal(src1p->bits, src2p->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_intersects(src1, src2) __cpus_intersects(&(src1), &(src2), NR_CPUS)
|
|
|
|
static inline int __cpus_intersects(const cpumask_t *src1p,
|
|
|
|
const cpumask_t *src2p, int nbits)
|
|
|
|
{
|
|
|
|
return bitmap_intersects(src1p->bits, src2p->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_subset(src1, src2) __cpus_subset(&(src1), &(src2), NR_CPUS)
|
|
|
|
static inline int __cpus_subset(const cpumask_t *src1p,
|
|
|
|
const cpumask_t *src2p, int nbits)
|
|
|
|
{
|
|
|
|
return bitmap_subset(src1p->bits, src2p->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_empty(src) __cpus_empty(&(src), NR_CPUS)
|
|
|
|
static inline int __cpus_empty(const cpumask_t *srcp, int nbits)
|
|
|
|
{
|
|
|
|
return bitmap_empty(srcp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_full(cpumask) __cpus_full(&(cpumask), NR_CPUS)
|
|
|
|
static inline int __cpus_full(const cpumask_t *srcp, int nbits)
|
|
|
|
{
|
|
|
|
return bitmap_full(srcp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_weight(cpumask) __cpus_weight(&(cpumask), NR_CPUS)
|
|
|
|
static inline int __cpus_weight(const cpumask_t *srcp, int nbits)
|
|
|
|
{
|
|
|
|
return bitmap_weight(srcp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_shift_right(dst, src, n) \
|
|
|
|
__cpus_shift_right(&(dst), &(src), (n), NR_CPUS)
|
|
|
|
static inline void __cpus_shift_right(cpumask_t *dstp,
|
|
|
|
const cpumask_t *srcp, int n, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_shift_right(dstp->bits, srcp->bits, n, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_shift_left(dst, src, n) \
|
|
|
|
__cpus_shift_left(&(dst), &(src), (n), NR_CPUS)
|
|
|
|
static inline void __cpus_shift_left(cpumask_t *dstp,
|
|
|
|
const cpumask_t *srcp, int n, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_shift_left(dstp->bits, srcp->bits, n, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
|
2008-04-05 05:11:01 +04:00
|
|
|
#ifdef CONFIG_HAVE_CPUMASK_OF_CPU_MAP
|
|
|
|
extern cpumask_t *cpumask_of_cpu_map;
|
2008-07-16 01:14:30 +04:00
|
|
|
#define cpumask_of_cpu(cpu) (cpumask_of_cpu_map[cpu])
|
|
|
|
#define cpumask_of_cpu_ptr(v, cpu) \
|
|
|
|
const cpumask_t *v = &cpumask_of_cpu(cpu)
|
|
|
|
#define cpumask_of_cpu_ptr_declare(v) \
|
|
|
|
const cpumask_t *v
|
|
|
|
#define cpumask_of_cpu_ptr_next(v, cpu) \
|
|
|
|
v = &cpumask_of_cpu(cpu)
|
2008-04-05 05:11:01 +04:00
|
|
|
#else
|
2005-04-17 02:20:36 +04:00
|
|
|
#define cpumask_of_cpu(cpu) \
|
2008-07-16 01:14:30 +04:00
|
|
|
({ \
|
2005-04-17 02:20:36 +04:00
|
|
|
typeof(_unused_cpumask_arg_) m; \
|
|
|
|
if (sizeof(m) == sizeof(unsigned long)) { \
|
|
|
|
m.bits[0] = 1UL<<(cpu); \
|
|
|
|
} else { \
|
|
|
|
cpus_clear(m); \
|
|
|
|
cpu_set((cpu), m); \
|
|
|
|
} \
|
2008-07-16 01:14:30 +04:00
|
|
|
m; \
|
|
|
|
})
|
|
|
|
#define cpumask_of_cpu_ptr(v, cpu) \
|
|
|
|
cpumask_t _##v = cpumask_of_cpu(cpu); \
|
|
|
|
const cpumask_t *v = &_##v
|
|
|
|
#define cpumask_of_cpu_ptr_declare(v) \
|
|
|
|
cpumask_t _##v; \
|
|
|
|
const cpumask_t *v = &_##v
|
|
|
|
#define cpumask_of_cpu_ptr_next(v, cpu) \
|
|
|
|
_##v = cpumask_of_cpu(cpu)
|
2008-04-05 05:11:01 +04:00
|
|
|
#endif
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
#define CPU_MASK_LAST_WORD BITMAP_LAST_WORD_MASK(NR_CPUS)
|
|
|
|
|
|
|
|
#if NR_CPUS <= BITS_PER_LONG
|
|
|
|
|
|
|
|
#define CPU_MASK_ALL \
|
|
|
|
(cpumask_t) { { \
|
|
|
|
[BITS_TO_LONGS(NR_CPUS)-1] = CPU_MASK_LAST_WORD \
|
|
|
|
} }
|
|
|
|
|
2008-04-05 05:11:02 +04:00
|
|
|
#define CPU_MASK_ALL_PTR (&CPU_MASK_ALL)
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#else
|
|
|
|
|
|
|
|
#define CPU_MASK_ALL \
|
|
|
|
(cpumask_t) { { \
|
|
|
|
[0 ... BITS_TO_LONGS(NR_CPUS)-2] = ~0UL, \
|
|
|
|
[BITS_TO_LONGS(NR_CPUS)-1] = CPU_MASK_LAST_WORD \
|
|
|
|
} }
|
|
|
|
|
2008-04-05 05:11:02 +04:00
|
|
|
/* cpu_mask_all is in init/main.c */
|
|
|
|
extern cpumask_t cpu_mask_all;
|
|
|
|
#define CPU_MASK_ALL_PTR (&cpu_mask_all)
|
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
|
|
|
|
|
|
|
#define CPU_MASK_NONE \
|
|
|
|
(cpumask_t) { { \
|
|
|
|
[0 ... BITS_TO_LONGS(NR_CPUS)-1] = 0UL \
|
|
|
|
} }
|
|
|
|
|
|
|
|
#define CPU_MASK_CPU0 \
|
|
|
|
(cpumask_t) { { \
|
|
|
|
[0] = 1UL \
|
|
|
|
} }
|
|
|
|
|
|
|
|
#define cpus_addr(src) ((src).bits)
|
|
|
|
|
2008-07-16 01:14:36 +04:00
|
|
|
#if NR_CPUS > BITS_PER_LONG
|
|
|
|
#define CPUMASK_ALLOC(m) struct m *m = kmalloc(sizeof(*m), GFP_KERNEL)
|
|
|
|
#define CPUMASK_FREE(m) kfree(m)
|
|
|
|
#else
|
2008-07-19 05:11:33 +04:00
|
|
|
#define CPUMASK_ALLOC(m) struct m _m, *m = &_m
|
2008-07-16 01:14:36 +04:00
|
|
|
#define CPUMASK_FREE(m)
|
|
|
|
#endif
|
2008-07-19 05:11:33 +04:00
|
|
|
#define CPUMASK_PTR(v, m) cpumask_t *v = &(m->v)
|
2008-07-16 01:14:36 +04:00
|
|
|
|
2005-04-17 02:20:36 +04:00
|
|
|
#define cpumask_scnprintf(buf, len, src) \
|
|
|
|
__cpumask_scnprintf((buf), (len), &(src), NR_CPUS)
|
|
|
|
static inline int __cpumask_scnprintf(char *buf, int len,
|
|
|
|
const cpumask_t *srcp, int nbits)
|
|
|
|
{
|
|
|
|
return bitmap_scnprintf(buf, len, srcp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
2006-10-11 12:21:55 +04:00
|
|
|
#define cpumask_parse_user(ubuf, ulen, dst) \
|
|
|
|
__cpumask_parse_user((ubuf), (ulen), &(dst), NR_CPUS)
|
|
|
|
static inline int __cpumask_parse_user(const char __user *buf, int len,
|
2005-04-17 02:20:36 +04:00
|
|
|
cpumask_t *dstp, int nbits)
|
|
|
|
{
|
2006-10-11 12:21:55 +04:00
|
|
|
return bitmap_parse_user(buf, len, dstp->bits, nbits);
|
2005-04-17 02:20:36 +04:00
|
|
|
}
|
|
|
|
|
|
|
|
#define cpulist_scnprintf(buf, len, src) \
|
|
|
|
__cpulist_scnprintf((buf), (len), &(src), NR_CPUS)
|
|
|
|
static inline int __cpulist_scnprintf(char *buf, int len,
|
|
|
|
const cpumask_t *srcp, int nbits)
|
|
|
|
{
|
|
|
|
return bitmap_scnlistprintf(buf, len, srcp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpulist_parse(buf, dst) __cpulist_parse((buf), &(dst), NR_CPUS)
|
|
|
|
static inline int __cpulist_parse(const char *buf, cpumask_t *dstp, int nbits)
|
|
|
|
{
|
|
|
|
return bitmap_parselist(buf, dstp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
2005-10-31 02:02:33 +03:00
|
|
|
#define cpu_remap(oldbit, old, new) \
|
|
|
|
__cpu_remap((oldbit), &(old), &(new), NR_CPUS)
|
|
|
|
static inline int __cpu_remap(int oldbit,
|
|
|
|
const cpumask_t *oldp, const cpumask_t *newp, int nbits)
|
|
|
|
{
|
|
|
|
return bitmap_bitremap(oldbit, oldp->bits, newp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_remap(dst, src, old, new) \
|
|
|
|
__cpus_remap(&(dst), &(src), &(old), &(new), NR_CPUS)
|
|
|
|
static inline void __cpus_remap(cpumask_t *dstp, const cpumask_t *srcp,
|
|
|
|
const cpumask_t *oldp, const cpumask_t *newp, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_remap(dstp->bits, srcp->bits, oldp->bits, newp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
mempolicy: add bitmap_onto() and bitmap_fold() operations
The following adds two more bitmap operators, bitmap_onto() and bitmap_fold(),
with the usual cpumask and nodemask wrappers.
The bitmap_onto() operator computes one bitmap relative to another. If the
n-th bit in the origin mask is set, then the m-th bit of the destination mask
will be set, where m is the position of the n-th set bit in the relative mask.
The bitmap_fold() operator folds a bitmap into a second that has bit m set iff
the input bitmap has some bit n set, where m == n mod sz, for the specified sz
value.
There are two substantive changes between this patch and its
predecessor bitmap_relative:
1) Renamed bitmap_relative() to be bitmap_onto().
2) Added bitmap_fold().
The essential motivation for bitmap_onto() is to provide a mechanism for
converting a cpuset-relative CPU or Node mask to an absolute mask. Cpuset
relative masks are written as if the current task were in a cpuset whose CPUs
or Nodes were just the consecutive ones numbered 0..N-1, for some N. The
bitmap_onto() operator is provided in anticipation of adding support for the
first such cpuset relative mask, by the mbind() and set_mempolicy() system
calls, using a planned flag of MPOL_F_RELATIVE_NODES. These bitmap operators
(and their nodemask wrappers, in particular) will be used in code that
converts the user specified cpuset relative memory policy to a specific system
node numbered policy, given the current mems_allowed of the tasks cpuset.
Such cpuset relative mempolicies will address two deficiencies
of the existing interface between cpusets and mempolicies:
1) A task cannot at present reliably establish a cpuset
relative mempolicy because there is an essential race
condition, in that the tasks cpuset may be changed in
between the time the task can query its cpuset placement,
and the time the task can issue the applicable mbind or
set_memplicy system call.
2) A task cannot at present establish what cpuset relative
mempolicy it would like to have, if it is in a smaller
cpuset than it might have mempolicy preferences for,
because the existing interface only allows specifying
mempolicies for nodes currently allowed by the cpuset.
Cpuset relative mempolicies are useful for tasks that don't distinguish
particularly between one CPU or Node and another, but only between how many of
each are allowed, and the proper placement of threads and memory pages on the
various CPUs and Nodes available.
The motivation for the added bitmap_fold() can be seen in the following
example.
Let's say an application has specified some mempolicies that presume 16 memory
nodes, including say a mempolicy that specified MPOL_F_RELATIVE_NODES (cpuset
relative) nodes 12-15. Then lets say that application is crammed into a
cpuset that only has 8 memory nodes, 0-7. If one just uses bitmap_onto(),
this mempolicy, mapped to that cpuset, would ignore the requested relative
nodes above 7, leaving it empty of nodes. That's not good; better to fold the
higher nodes down, so that some nodes are included in the resulting mapped
mempolicy. In this case, the mempolicy nodes 12-15 are taken modulo 8 (the
weight of the mems_allowed of the confining cpuset), resulting in a mempolicy
specifying nodes 4-7.
Signed-off-by: Paul Jackson <pj@sgi.com>
Signed-off-by: David Rientjes <rientjes@google.com>
Cc: Christoph Lameter <clameter@sgi.com>
Cc: Andi Kleen <ak@suse.de>
Cc: Mel Gorman <mel@csn.ul.ie>
Cc: Lee Schermerhorn <lee.schermerhorn@hp.com>
Cc: <kosaki.motohiro@jp.fujitsu.com>
Cc: <ray-lk@madrabbit.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2008-04-28 13:12:29 +04:00
|
|
|
#define cpus_onto(dst, orig, relmap) \
|
|
|
|
__cpus_onto(&(dst), &(orig), &(relmap), NR_CPUS)
|
|
|
|
static inline void __cpus_onto(cpumask_t *dstp, const cpumask_t *origp,
|
|
|
|
const cpumask_t *relmapp, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_onto(dstp->bits, origp->bits, relmapp->bits, nbits);
|
|
|
|
}
|
|
|
|
|
|
|
|
#define cpus_fold(dst, orig, sz) \
|
|
|
|
__cpus_fold(&(dst), &(orig), sz, NR_CPUS)
|
|
|
|
static inline void __cpus_fold(cpumask_t *dstp, const cpumask_t *origp,
|
|
|
|
int sz, int nbits)
|
|
|
|
{
|
|
|
|
bitmap_fold(dstp->bits, origp->bits, sz, nbits);
|
|
|
|
}
|
|
|
|
|
2008-05-12 23:21:13 +04:00
|
|
|
#if NR_CPUS == 1
|
|
|
|
|
|
|
|
#define nr_cpu_ids 1
|
|
|
|
#define first_cpu(src) ({ (void)(src); 0; })
|
|
|
|
#define next_cpu(n, src) ({ (void)(src); 1; })
|
|
|
|
#define any_online_cpu(mask) 0
|
|
|
|
#define for_each_cpu_mask(cpu, mask) \
|
|
|
|
for ((cpu) = 0; (cpu) < 1; (cpu)++, (void)mask)
|
|
|
|
|
|
|
|
#else /* NR_CPUS > 1 */
|
|
|
|
|
|
|
|
extern int nr_cpu_ids;
|
|
|
|
int __first_cpu(const cpumask_t *srcp);
|
|
|
|
int __next_cpu(int n, const cpumask_t *srcp);
|
|
|
|
int __any_online_cpu(const cpumask_t *mask);
|
|
|
|
|
|
|
|
#define first_cpu(src) __first_cpu(&(src))
|
|
|
|
#define next_cpu(n, src) __next_cpu((n), &(src))
|
|
|
|
#define any_online_cpu(mask) __any_online_cpu(&(mask))
|
2008-05-13 13:28:21 +04:00
|
|
|
#define for_each_cpu_mask(cpu, mask) \
|
|
|
|
for ((cpu) = -1; \
|
|
|
|
(cpu) = next_cpu((cpu), (mask)), \
|
|
|
|
(cpu) < NR_CPUS; )
|
2008-05-12 23:21:13 +04:00
|
|
|
#endif
|
|
|
|
|
|
|
|
#if NR_CPUS <= 64
|
|
|
|
|
|
|
|
#define next_cpu_nr(n, src) next_cpu(n, src)
|
|
|
|
#define cpus_weight_nr(cpumask) cpus_weight(cpumask)
|
|
|
|
#define for_each_cpu_mask_nr(cpu, mask) for_each_cpu_mask(cpu, mask)
|
|
|
|
|
|
|
|
#else /* NR_CPUS > 64 */
|
|
|
|
|
|
|
|
int __next_cpu_nr(int n, const cpumask_t *srcp);
|
|
|
|
#define next_cpu_nr(n, src) __next_cpu_nr((n), &(src))
|
|
|
|
#define cpus_weight_nr(cpumask) __cpus_weight(&(cpumask), nr_cpu_ids)
|
2008-05-13 13:28:21 +04:00
|
|
|
#define for_each_cpu_mask_nr(cpu, mask) \
|
|
|
|
for ((cpu) = -1; \
|
|
|
|
(cpu) = next_cpu_nr((cpu), (mask)), \
|
|
|
|
(cpu) < nr_cpu_ids; )
|
2008-05-12 23:21:13 +04:00
|
|
|
|
|
|
|
#endif /* NR_CPUS > 64 */
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
/*
|
|
|
|
* The following particular system cpumasks and operations manage
|
2008-07-15 15:43:49 +04:00
|
|
|
* possible, present, active and online cpus. Each of them is a fixed size
|
2005-04-17 02:20:36 +04:00
|
|
|
* bitmap of size NR_CPUS.
|
|
|
|
*
|
|
|
|
* #ifdef CONFIG_HOTPLUG_CPU
|
2006-02-10 12:51:08 +03:00
|
|
|
* cpu_possible_map - has bit 'cpu' set iff cpu is populatable
|
2005-04-17 02:20:36 +04:00
|
|
|
* cpu_present_map - has bit 'cpu' set iff cpu is populated
|
|
|
|
* cpu_online_map - has bit 'cpu' set iff cpu available to scheduler
|
2008-07-15 15:43:49 +04:00
|
|
|
* cpu_active_map - has bit 'cpu' set iff cpu available to migration
|
2005-04-17 02:20:36 +04:00
|
|
|
* #else
|
|
|
|
* cpu_possible_map - has bit 'cpu' set iff cpu is populated
|
|
|
|
* cpu_present_map - copy of cpu_possible_map
|
|
|
|
* cpu_online_map - has bit 'cpu' set iff cpu available to scheduler
|
|
|
|
* #endif
|
|
|
|
*
|
|
|
|
* In either case, NR_CPUS is fixed at compile time, as the static
|
|
|
|
* size of these bitmaps. The cpu_possible_map is fixed at boot
|
|
|
|
* time, as the set of CPU id's that it is possible might ever
|
|
|
|
* be plugged in at anytime during the life of that system boot.
|
|
|
|
* The cpu_present_map is dynamic(*), representing which CPUs
|
|
|
|
* are currently plugged in. And cpu_online_map is the dynamic
|
|
|
|
* subset of cpu_present_map, indicating those CPUs available
|
|
|
|
* for scheduling.
|
|
|
|
*
|
|
|
|
* If HOTPLUG is enabled, then cpu_possible_map is forced to have
|
|
|
|
* all NR_CPUS bits set, otherwise it is just the set of CPUs that
|
|
|
|
* ACPI reports present at boot.
|
|
|
|
*
|
|
|
|
* If HOTPLUG is enabled, then cpu_present_map varies dynamically,
|
|
|
|
* depending on what ACPI reports as currently plugged in, otherwise
|
|
|
|
* cpu_present_map is just a copy of cpu_possible_map.
|
|
|
|
*
|
|
|
|
* (*) Well, cpu_present_map is dynamic in the hotplug case. If not
|
|
|
|
* hotplug, it's a copy of cpu_possible_map, hence fixed at boot.
|
|
|
|
*
|
|
|
|
* Subtleties:
|
|
|
|
* 1) UP arch's (NR_CPUS == 1, CONFIG_SMP not defined) hardcode
|
|
|
|
* assumption that their single CPU is online. The UP
|
|
|
|
* cpu_{online,possible,present}_maps are placebos. Changing them
|
|
|
|
* will have no useful affect on the following num_*_cpus()
|
|
|
|
* and cpu_*() macros in the UP case. This ugliness is a UP
|
|
|
|
* optimization - don't waste any instructions or memory references
|
|
|
|
* asking if you're online or how many CPUs there are if there is
|
|
|
|
* only one CPU.
|
|
|
|
* 2) Most SMP arch's #define some of these maps to be some
|
|
|
|
* other map specific to that arch. Therefore, the following
|
|
|
|
* must be #define macros, not inlines. To see why, examine
|
|
|
|
* the assembly code produced by the following. Note that
|
|
|
|
* set1() writes phys_x_map, but set2() writes x_map:
|
|
|
|
* int x_map, phys_x_map;
|
|
|
|
* #define set1(a) x_map = a
|
|
|
|
* inline void set2(int a) { x_map = a; }
|
|
|
|
* #define x_map phys_x_map
|
|
|
|
* main(){ set1(3); set2(5); }
|
|
|
|
*/
|
|
|
|
|
|
|
|
extern cpumask_t cpu_possible_map;
|
|
|
|
extern cpumask_t cpu_online_map;
|
|
|
|
extern cpumask_t cpu_present_map;
|
2008-07-15 15:43:49 +04:00
|
|
|
extern cpumask_t cpu_active_map;
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
#if NR_CPUS > 1
|
2008-05-12 23:21:13 +04:00
|
|
|
#define num_online_cpus() cpus_weight_nr(cpu_online_map)
|
|
|
|
#define num_possible_cpus() cpus_weight_nr(cpu_possible_map)
|
|
|
|
#define num_present_cpus() cpus_weight_nr(cpu_present_map)
|
2005-04-17 02:20:36 +04:00
|
|
|
#define cpu_online(cpu) cpu_isset((cpu), cpu_online_map)
|
|
|
|
#define cpu_possible(cpu) cpu_isset((cpu), cpu_possible_map)
|
|
|
|
#define cpu_present(cpu) cpu_isset((cpu), cpu_present_map)
|
2008-07-15 15:43:49 +04:00
|
|
|
#define cpu_active(cpu) cpu_isset((cpu), cpu_active_map)
|
2005-04-17 02:20:36 +04:00
|
|
|
#else
|
|
|
|
#define num_online_cpus() 1
|
|
|
|
#define num_possible_cpus() 1
|
|
|
|
#define num_present_cpus() 1
|
|
|
|
#define cpu_online(cpu) ((cpu) == 0)
|
|
|
|
#define cpu_possible(cpu) ((cpu) == 0)
|
|
|
|
#define cpu_present(cpu) ((cpu) == 0)
|
2008-07-15 15:43:49 +04:00
|
|
|
#define cpu_active(cpu) ((cpu) == 0)
|
2005-04-17 02:20:36 +04:00
|
|
|
#endif
|
|
|
|
|
2007-12-30 13:58:17 +03:00
|
|
|
#define cpu_is_offline(cpu) unlikely(!cpu_online(cpu))
|
|
|
|
|
2008-05-12 23:21:13 +04:00
|
|
|
#define for_each_possible_cpu(cpu) for_each_cpu_mask_nr((cpu), cpu_possible_map)
|
|
|
|
#define for_each_online_cpu(cpu) for_each_cpu_mask_nr((cpu), cpu_online_map)
|
|
|
|
#define for_each_present_cpu(cpu) for_each_cpu_mask_nr((cpu), cpu_present_map)
|
2005-04-17 02:20:36 +04:00
|
|
|
|
|
|
|
#endif /* __LINUX_CPUMASK_H */
|