Linus Torvalds 3510ca20ec Minor page waitqueue cleanups ... Tim Chen and Kan Liang have been battling a customer load that shows extremely long page wakeup lists. The cause seems to be constant NUMA migration of a hot page that is shared across a lot of threads, but the actual root cause for the exact behavior has not been found. Tim has a patch that batches the wait list traversal at wakeup time, so that we at least don't get long uninterruptible cases where we traverse and wake up thousands of processes and get nasty latency spikes. That is likely 4.14 material, but we're still discussing the page waitqueue specific parts of it. In the meantime, I've tried to look at making the page wait queues less expensive, and failing miserably. If you have thousands of threads waiting for the same page, it will be painful. We'll need to try to figure out the NUMA balancing issue some day, in addition to avoiding the excessive spinlock hold times. That said, having tried to rewrite the page wait queues, I can at least fix up some of the braindamage in the current situation. In particular: (a) we don't want to continue walking the page wait list if the bit we're waiting for already got set again (which seems to be one of the patterns of the bad load). That makes no progress and just causes pointless cache pollution chasing the pointers. (b) we don't want to put the non-locking waiters always on the front of the queue, and the locking waiters always on the back. Not only is that unfair, it means that we wake up thousands of reading threads that will just end up being blocked by the writer later anyway. Also add a comment about the layout of 'struct wait_page_key' - there is an external user of it in the cachefiles code that means that it has to match the layout of 'struct wait_bit_key' in the two first members. It so happens to match, because 'struct page *' and 'unsigned long *' end up having the same values simply because the page flags are the first member in struct page. Cc: Tim Chen <tim.c.chen@linux.intel.com> Cc: Kan Liang <kan.liang@intel.com> Cc: Mel Gorman <mgorman@techsingularity.net> Cc: Christopher Lameter <cl@linux.com> Cc: Andi Kleen <ak@linux.intel.com> Cc: Davidlohr Bueso <dave@stgolabs.net> Cc: Peter Zijlstra <peterz@infradead.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>		2017-08-27 13:55:12 -07:00
..
bpf	bpf: fix bpf_prog_get_info_by_fd to dump correct xlated_prog_len	2017-07-29 23:29:41 -07:00
cgroup	cpuset: fix a deadlock due to incomplete patching of cpusets_enabled()	2017-08-02 17:16:12 -07:00
configs	config: android-base: disable CONFIG_NFSD and CONFIG_NFS_FS	2017-06-09 11:47:38 +02:00
debug	sched/headers: Prepare for new header dependencies before moving code to <linux/sched/debug.h>	2017-03-02 08:42:34 +01:00
events	perf/core: Fix group {cpu,task} validation	2017-08-25 11:00:34 +02:00
gcov	gcov: support GCC 7.1	2017-05-12 15:57:15 -07:00
irq	genirq/ipi: Fixup checks against nr_cpu_ids	2017-08-20 10:49:05 +02:00
livepatch	livepatch: Fix stacking of patches with respect to RCU	2017-06-20 10:42:19 +02:00
locking	Merge branch 'locking-urgent-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip	2017-07-21 11:11:23 -07:00
power	mm: fix global NR_SLAB_.*CLAIMABLE counter reads	2017-08-10 15:54:06 -07:00
printk	Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/pmladek/printk	2017-07-05 11:11:26 -07:00
rcu	rcu: Remove RCU CPU stall warnings from Tiny RCU	2017-06-08 18:52:45 -07:00
sched	Minor page waitqueue cleanups	2017-08-27 13:55:12 -07:00
time	timers: Fix excessive granularity of new timers after a nohz idle	2017-08-24 11:40:18 +02:00
trace	Various bug fixes:	2017-08-24 14:08:22 -07:00
.gitignore	…
Kconfig.freezer	…
Kconfig.hz	…
Kconfig.locks	…
Kconfig.preempt	…
Makefile	kernel/watchdog: split up config options	2017-07-12 16:26:02 -07:00
acct.c	sched/headers: Prepare to move cputime functionality from <linux/sched.h> into <linux/sched/cputime.h>	2017-03-02 08:42:39 +01:00
async.c	async: Adjust system_state checks	2017-05-23 10:01:37 +02:00
audit.c	Merge branch 'stable-4.13' of git://git.infradead.org/users/pcmoore/audit	2017-07-20 10:22:26 -07:00
audit.h	audit: style fix	2017-06-12 18:07:43 -04:00
audit_fsnotify.c	Merge branch 'fsnotify' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs	2017-05-03 11:05:15 -07:00
audit_tree.c	Merge branch 'fsnotify' of git://git.kernel.org/pub/scm/linux/kernel/git/jack/linux-fs	2017-05-03 11:05:15 -07:00
audit_watch.c	audit/stable-4.13 PR 20170816	2017-08-16 16:48:34 -07:00
auditfilter.c	audit: kernel generated netlink traffic should have a portid of 0	2017-05-02 10:16:05 -04:00
auditsc.c	Merge branch 'stable-4.13' of git://git.infradead.org/users/pcmoore/audit	2017-07-05 11:24:05 -07:00
backtracetest.c	…
bounds.c	…
capability.c	capability: export has_capability	2017-01-12 07:01:56 -07:00
compat.c	Merge branch 'misc.compat' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs	2017-07-06 20:57:13 -07:00
configs.c	Replace <asm/uaccess.h> with <linux/uaccess.h> globally	2016-12-24 11:46:01 -08:00
context_tracking.c	…
cpu.c	smp/hotplug: Replace BUG_ON and react useful	2017-07-11 22:25:44 +02:00
cpu_pm.c	…
crash_core.c	kdump: protect vmcoreinfo data under the crash memory	2017-07-12 16:26:00 -07:00
crash_dump.c	…
cred.c	doc: ReSTify credentials.txt	2017-05-18 10:30:19 -06:00
delayacct.c	sched/headers: Prepare to move cputime functionality from <linux/sched.h> into <linux/sched/cputime.h>	2017-03-02 08:42:39 +01:00
dma.c	…
elfcore.c	…
exec_domain.c	…
exit.c	kernel/exit.c: avoid undefined behaviour when calling wait4()	2017-07-10 16:32:36 -07:00
extable.c	lib/extable.c: use bsearch() library function in search_extable()	2017-07-10 16:32:35 -07:00
fork.c	fork: fix incorrect fput of ->exe_file causing use-after-free	2017-08-25 16:12:46 -07:00
freezer.c	…
futex.c	futex: Remove unnecessary warning from get_futex_key	2017-08-09 14:00:54 -07:00
futex_compat.c	Replace <asm/uaccess.h> with <linux/uaccess.h> globally	2016-12-24 11:46:01 -08:00
groups.c	kernel/groups.c: use sort library function	2017-07-10 16:32:34 -07:00
hung_task.c	kernel/hung_task.c: defer showing held locks	2017-05-08 17:15:10 -07:00
irq_work.c	…
jump_label.c	jump_label: Reorder hotplug lock and jump_label_lock	2017-05-26 10:10:45 +02:00
kallsyms.c	kernel/kallsyms.c: replace all_var with IS_ENABLED(CONFIG_KALLSYMS_ALL)	2017-07-10 16:32:34 -07:00
kcmp.c	kcmp: add KCMP_EPOLL_TFD mode to compare epoll target files	2017-07-12 16:26:01 -07:00
kcov.c	kcov: simplify interrupt check	2017-05-08 17:15:12 -07:00
kexec.c	kdump: protect vmcoreinfo data under the crash memory	2017-07-12 16:26:00 -07:00
kexec_core.c	kdump: protect vmcoreinfo data under the crash memory	2017-07-12 16:26:00 -07:00
kexec_file.c	kexec_file: adjust declaration of kexec_purgatory	2017-07-12 16:26:02 -07:00
kexec_internal.h	kexec_file: adjust declaration of kexec_purgatory	2017-07-12 16:26:02 -07:00
kmod.c	kmod: fix wait on recursive loop	2017-08-18 15:32:01 -07:00
kprobes.c	kprobes: Ensure that jprobe probepoints are at function entry	2017-07-08 11:05:35 +02:00
ksysfs.c	kexec: move vmcoreinfo out of the kernel's .bss section	2017-07-12 16:25:59 -07:00
kthread.c	cgroup, kthread: close race window where new kthreads can be migrated to non-root cgroups	2017-03-17 10:18:47 -04:00
latencytop.c	sched/headers: Prepare to move sched_info_on() and force_schedstat_enabled() from <linux/sched.h> to <linux/sched/stat.h>	2017-03-02 08:42:39 +01:00
membarrier.c	Fix: Disable sys_membarrier when nohz_full is enabled	2017-01-23 11:32:16 -08:00
memremap.c	mm, memory_hotplug: replace for_device by want_memblock in arch_add_memory	2017-07-06 16:24:32 -07:00
module-internal.h	…
module.c	Modules updates for v4.13	2017-07-12 17:22:01 -07:00
module_signing.c	…
notifier.c	kernel/notifier.c: simplify expression	2017-02-24 17:46:56 -08:00
nsproxy.c	perf: Add PERF_RECORD_NAMESPACES to include namespaces related info	2017-03-13 15:57:41 -03:00
padata.c	padata: Avoid nested calls to cpus_read_lock() in pcrypt_init_padata()	2017-05-26 10:10:37 +02:00
panic.c	sched/headers: Prepare for new header dependencies before moving code to <linux/sched/debug.h>	2017-03-02 08:42:34 +01:00
params.c	boot/param: Move next_arg() function to lib/cmdline.c for later reuse	2017-04-18 10:37:13 +02:00
pid.c	pids: make task_tgid_nr_ns() safe	2017-08-21 12:47:31 -07:00
pid_namespace.c	pid_ns: Sleep in TASK_INTERRUPTIBLE in zap_pid_ns_processes	2017-05-13 17:26:01 -05:00
profile.c	sched/headers: Prepare to move sched_info_on() and force_schedstat_enabled() from <linux/sched.h> to <linux/sched/stat.h>	2017-03-02 08:42:39 +01:00
ptrace.c	ptrace: Properly initialize ptracer_cred on fork	2017-05-23 07:40:44 -05:00
range.c	…
reboot.c	…
relay.c	Merge branch 'work.splice' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs	2017-05-02 11:38:06 -07:00
resource.c	…
seccomp.c	seccomp: Switch from atomic_t to recount_t	2017-06-26 09:24:00 -07:00
signal.c	signal: don't remove SIGNAL_UNKILLABLE for traced tasks.	2017-08-18 15:32:02 -07:00
smp.c	smp, cpumask: Use non-atomic cpumask_{set,clear}_cpu()	2017-05-23 10:01:32 +02:00
smpboot.c	sched/headers: Prepare for new header dependencies before moving code to <linux/sched/task.h>	2017-03-02 08:42:35 +01:00
smpboot.h	…
softirq.c	sched/core: Remove 'task' parameter and rename tsk_restore_flags() to current_restore_flags()	2017-04-11 09:06:32 +02:00
stacktrace.c	stacktrace/x86: add function for detecting reliable stack traces	2017-03-08 09:18:02 +01:00
stop_machine.c	stop_machine: Provide stop_machine_cpuslocked()	2017-05-26 10:10:36 +02:00
sys.c	fix a braino in compat_sys_getrlimit()	2017-07-12 09:15:00 -07:00
sys_ni.c	move aio compat to fs/aio.c	2016-12-22 22:58:37 -05:00
sysctl.c	kernel/watchdog: split up config options	2017-07-12 16:26:02 -07:00
sysctl_binary.c	kernel/sysctl_binary.c: check name array length in deprecated_sysctl_warning()	2017-07-12 16:26:00 -07:00
task_work.c	…
taskstats.c	taskstats: add e/u/stime for TGID command	2017-05-08 17:15:12 -07:00
test_kprobes.c	…
torture.c	sched/headers: Prepare for new header dependencies before moving code to <linux/sched/clock.h>	2017-03-02 08:42:27 +01:00
tracepoint.c	sched/headers: Prepare for new header dependencies before moving code to <linux/sched/task.h>	2017-03-02 08:42:35 +01:00
tsacct.c	sched/headers: Prepare to move cputime functionality from <linux/sched.h> into <linux/sched/cputime.h>	2017-03-02 08:42:39 +01:00
ucount.c	ucount: Remove the atomicity from ucount->count	2017-03-06 15:26:37 -06:00
uid16.c	sched/headers: Prepare to remove <linux/cred.h> inclusion from <linux/sched.h>	2017-03-02 08:42:31 +01:00
up.c	…
user-return-notifier.c	…
user.c	sched/headers: Prepare for new header dependencies before moving code to <linux/sched/user.h>	2017-03-02 08:42:29 +01:00
user_namespace.c	sched/headers: Prepare for new header dependencies before moving code to <linux/sched/signal.h>	2017-03-02 08:42:29 +01:00
utsname.c	sched/headers: Prepare to move the task_lock()/unlock() APIs to <linux/sched/task.h>	2017-03-02 08:42:38 +01:00
utsname_sysctl.c	sched/headers: Remove <linux/rwsem.h> from <linux/sched.h>	2017-03-03 01:45:36 +01:00
watchdog.c	kernel/watchdog: Prevent false positives with turbo modes	2017-08-18 12:35:02 +02:00
watchdog_hld.c	kernel/watchdog: Prevent false positives with turbo modes	2017-08-18 12:35:02 +02:00
workqueue.c	workqueue: Work around edge cases for calc of pool's cpumask	2017-07-28 11:05:52 -04:00
workqueue_internal.h	…