Mel Gorman d9ed0d08b6 mm: only drain per-cpu pagevecs once per pagevec usage ... When a pagevec is initialised on the stack, it is generally used multiple times over a range of pages, looking up entries and then releasing them. On each pagevec_release, the per-cpu deferred LRU pagevecs are drained on the grounds the page being released may be on those queues and the pages may be cache hot. In many cases only the first drain is necessary as it's unlikely that the range of pages being walked is racing against LRU addition. Even if there is such a race, the impact is marginal where as constantly redraining the lru pagevecs costs. This patch ensures that pagevec is only drained once in a given lifecycle without increasing the cache footprint of the pagevec structure. Only sparsetruncate tiny is shown here as large files have many exceptional entries and calls pagecache_release less frequently. sparsetruncate (tiny) 4.14.0-rc4 4.14.0-rc4 batchshadow-v1r1 onedrain-v1r1 Min Time 141.00 ( 0.00%) 141.00 ( 0.00%) 1st-qrtle Time 142.00 ( 0.00%) 142.00 ( 0.00%) 2nd-qrtle Time 142.00 ( 0.00%) 142.00 ( 0.00%) 3rd-qrtle Time 143.00 ( 0.00%) 143.00 ( 0.00%) Max-90% Time 144.00 ( 0.00%) 144.00 ( 0.00%) Max-95% Time 146.00 ( 0.00%) 145.00 ( 0.68%) Max-99% Time 198.00 ( 0.00%) 194.00 ( 2.02%) Max Time 254.00 ( 0.00%) 208.00 ( 18.11%) Amean Time 145.12 ( 0.00%) 144.30 ( 0.56%) Stddev Time 12.74 ( 0.00%) 9.62 ( 24.49%) Coeff Time 8.78 ( 0.00%) 6.67 ( 24.06%) Best99%Amean Time 144.29 ( 0.00%) 143.82 ( 0.32%) Best95%Amean Time 142.68 ( 0.00%) 142.31 ( 0.26%) Best90%Amean Time 142.52 ( 0.00%) 142.19 ( 0.24%) Best75%Amean Time 142.26 ( 0.00%) 141.98 ( 0.20%) Best50%Amean Time 141.90 ( 0.00%) 141.71 ( 0.13%) Best25%Amean Time 141.80 ( 0.00%) 141.43 ( 0.26%) The impact on bonnie is marginal and within the noise because a significant percentage of the file being truncated has been reclaimed and consists of shadow entries which reduce the hotness of the pagevec_release path. Link: http://lkml.kernel.org/r/20171018075952.10627-5-mgorman@techsingularity.net Signed-off-by: Mel Gorman <mgorman@techsingularity.net> Cc: Andi Kleen <ak@linux.intel.com> Cc: Dave Chinner <david@fromorbit.com> Cc: Dave Hansen <dave.hansen@intel.com> Cc: Jan Kara <jack@suse.cz> Cc: Johannes Weiner <hannes@cmpxchg.org> Cc: Vlastimil Babka <vbabka@suse.cz> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>		2017-11-15 18:21:06 -08:00
..
kasan	slab, slub, slob: add slab_flags_t	2017-11-15 18:21:01 -08:00
Kconfig	mm/hmm: avoid bloating arch that do not make use of HMM	2017-09-08 18:26:46 -07:00
Kconfig.debug	kmemcheck: rip it out	2017-11-15 18:21:05 -08:00
Makefile	kmemcheck: rip it out	2017-11-15 18:21:05 -08:00
backing-dev.c	backing-dev: kill unused pdflush_proc_obsolete()	2017-10-06 08:15:15 -06:00
balloon_compaction.c	mm/migrate: new migrate mode MIGRATE_SYNC_NO_COPY	2017-09-08 18:26:46 -07:00
bootmem.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
cleancache.c	fs: switch ->s_uuid to uuid_t	2017-06-05 16:59:12 +02:00
cma.c	mm/cma.c: change pr_info to pr_err for cma_alloc fail log	2017-11-15 18:21:03 -08:00
cma.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
cma_debug.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
compaction.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
debug.c	mm: consolidate page table accounting	2017-11-15 18:21:04 -08:00
debug_page_ref.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
dmapool.c	lib/vsprintf.c: remove %Z support	2017-02-27 18:43:47 -08:00
early_ioremap.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
fadvise.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
failslab.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
filemap.c	mm, truncate: do not check mapping for every page being truncated	2017-11-15 18:21:06 -08:00
frame_vector.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
frontswap.c	mm, frontswap: convert frontswap_enabled to static key	2016-07-26 16:19:19 -07:00
gup.c	Merge branch 'x86/urgent' into x86/mm, to pick up fixes	2017-10-20 13:06:52 +02:00
highmem.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
hmm.c	mm/hmm: avoid bloating arch that do not make use of HMM	2017-09-08 18:26:46 -07:00
huge_memory.c	mm: consolidate page table accounting	2017-11-15 18:21:04 -08:00
hugetlb.c	mm/mmu_notifier: avoid double notification when it is useless	2017-11-15 18:21:03 -08:00
hugetlb_cgroup.c	mm, hugetlb_cgroup: round limit_in_bytes down to hugepage size	2016-05-20 17:58:30 -07:00
hwpoison-inject.c	mm: hwpoison: call shake_page() unconditionally	2017-05-03 15:52:12 -07:00
init-mm.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
internal.h	mm, oom: do not rely on TIF_MEMDIE for memory reserves access	2017-09-06 17:27:30 -07:00
interval_tree.c	lib/interval_tree: fast overlap detection	2017-09-08 18:26:49 -07:00
khugepaged.c	mm: introduce wrappers to access mm->nr_ptes	2017-11-15 18:21:04 -08:00
kmemcheck.c	kmemcheck: rip it out	2017-11-15 18:21:05 -08:00
kmemleak-test.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
kmemleak.c	kmemcheck: remove annotations	2017-11-15 18:21:04 -08:00
ksm.c	mm/mmu_notifier: avoid double notification when it is useless	2017-11-15 18:21:03 -08:00
list_lru.c	mm: memcontrol: use vmalloc fallback for large kmem memcg arrays	2017-10-03 17:54:25 -07:00
maccess.c	x86: remove more uaccess_32.h complexity	2016-05-22 17:21:27 -07:00
madvise.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
memblock.c	mm: define memblock_virt_alloc_try_nid_raw	2017-11-15 18:21:05 -08:00
memcontrol.c	mm: slabinfo: remove CONFIG_SLABINFO	2017-11-15 18:21:01 -08:00
memory-failure.c	mm, soft_offline: improve hugepage soft offlining error log	2017-11-15 18:21:05 -08:00
memory.c	mm: introduce wrappers to access mm->nr_ptes	2017-11-15 18:21:04 -08:00
memory_hotplug.c	mm, memory_hotplug: remove timeout from __offline_memory	2017-11-15 18:21:02 -08:00
mempolicy.c	mm/mempolicy: fix NUMA_INTERLEAVE_HIT counter	2017-10-13 16:18:32 -07:00
mempool.c	mm/mempool.c: use kmalloc_array_node()	2017-11-15 18:21:02 -08:00
memtest.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
migrate.c	mm/mmu_notifier: avoid call to invalidate_range() in range_end()	2017-11-15 18:21:03 -08:00
mincore.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
mlock.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
mm_init.c	mm: convert printk(KERN_<LEVEL> to pr_<level>	2016-03-17 15:09:34 -07:00
mmap.c	lib/interval_tree: fast overlap detection	2017-09-08 18:26:49 -07:00
mmu_context.c	sched/headers: Prepare to move the task_lock()/unlock() APIs to <linux/sched/task.h>	2017-03-02 08:42:38 +01:00
mmu_notifier.c	mm/mmu_notifier: avoid call to invalidate_range() in range_end()	2017-11-15 18:21:03 -08:00
mmzone.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
mprotect.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
mremap.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
msync.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
nobootmem.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
nommu.c	Merge branch 'work.set_fs' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs	2017-09-14 18:13:32 -07:00
oom_kill.c	mm: consolidate page table accounting	2017-11-15 18:21:04 -08:00
page-writeback.c	mm: speed up cancel_dirty_page() for clean pages	2017-11-15 18:21:06 -08:00
page_alloc.c	mm, page_alloc: enable/disable IRQs once when freeing a list of pages	2017-11-15 18:21:06 -08:00
page_counter.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
page_ext.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
page_idle.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
page_io.c	mm, swap: skip swapcache for swapin of synchronous device	2017-11-15 18:21:02 -08:00
page_isolation.c	mm: distinguish CMA and MOVABLE isolation in has_unmovable_pages()	2017-11-15 18:21:02 -08:00
page_owner.c	mm/page_owner.c: reduce page_owner structure size	2017-11-15 18:21:03 -08:00
page_poison.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
page_vma_mapped.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
pagewalk.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
percpu-internal.h	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
percpu-km.c	percpu: replace area map allocator with bitmap	2017-07-26 17:41:05 -04:00
percpu-stats.c	percpu: fix starting offset for chunk statistics traversal	2017-09-27 14:45:57 -07:00
percpu-vm.c	percpu: fix static checker warnings in pcpu_destroy_chunk	2017-06-29 11:23:38 -04:00
percpu.c	mm, percpu: add support for __GFP_NOWARN flag	2017-10-19 13:13:49 +01:00
pgtable-generic.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
process_vm_access.c	sched/headers: Prepare for new header dependencies before moving code to <linux/sched/mm.h>	2017-03-02 08:42:28 +01:00
quicklist.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
readahead.c	mm: don't cap request size based on read-ahead setting	2016-12-12 18:55:08 -08:00
rmap.c	mm/rmap.c: remove redundant variable cend	2017-11-15 18:21:04 -08:00
rodata_test.c	mm: fix RODATA_TEST failure "rodata_test: test data was not read only"	2017-10-03 17:54:24 -07:00
shmem.c	mm, truncate: do not check mapping for every page being truncated	2017-11-15 18:21:06 -08:00
slab.c	kmemcheck: stop using GFP_NOTRACK and SLAB_NOTRACK	2017-11-15 18:21:04 -08:00
slab.h	kmemcheck: stop using GFP_NOTRACK and SLAB_NOTRACK	2017-11-15 18:21:04 -08:00
slab_common.c	kmemcheck: stop using GFP_NOTRACK and SLAB_NOTRACK	2017-11-15 18:21:04 -08:00
slob.c	slab, slub, slob: add slab_flags_t	2017-11-15 18:21:01 -08:00
slub.c	kmemcheck: rip it out	2017-11-15 18:21:05 -08:00
sparse-vmemmap.c	mm: stop zeroing memory during allocation in vmemmap	2017-11-15 18:21:05 -08:00
sparse.c	mm: stop zeroing memory during allocation in vmemmap	2017-11-15 18:21:05 -08:00
swap.c	mm: only drain per-cpu pagevecs once per pagevec usage	2017-11-15 18:21:06 -08:00
swap_cgroup.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
swap_slots.c	mm/swap_slots.c: fix race conditions in swap_slots cache init	2017-11-15 18:21:03 -08:00
swap_state.c	mm/swap_state.c: declare a few variables as __read_mostly	2017-11-15 18:21:05 -08:00
swapfile.c	mm: swap: SWP_SYNCHRONOUS_IO: skip swapcache only if swapped page has no other reference	2017-11-15 18:21:02 -08:00
truncate.c	mm, truncate: remove all exceptional entries from pagevec under one lock	2017-11-15 18:21:06 -08:00
usercopy.c	mm/usercopy: Drop extra is_vmalloc_or_module() check	2017-04-05 12:30:18 -07:00
userfaultfd.c	userfaultfd: shmem: wire up shmem_mfill_zeropage_pte	2017-09-06 17:27:28 -07:00
util.c	mm: rename global_page_state to global_zone_page_state	2017-09-06 17:27:29 -07:00
vmacache.c	License cleanup: add SPDX GPL-2.0 license identifier to files with no license	2017-11-02 11:10:55 +01:00
vmalloc.c	Revert "vmalloc: back off when the current task is killed"	2017-10-13 16:18:32 -07:00
vmpressure.c	mm, vmpressure: pass-through notification support	2017-07-10 16:32:31 -07:00
vmscan.c	mm: remove unused pgdat->inactive_ratio	2017-11-15 18:21:03 -08:00
vmstat.c	mm: remove unused pgdat->inactive_ratio	2017-11-15 18:21:03 -08:00
workingset.c	mm, truncate: do not check mapping for every page being truncated	2017-11-15 18:21:06 -08:00
z3fold.c	z3fold: fix stale list handling	2017-10-03 17:54:24 -07:00
zbud.c	mm/zbud.c: use list_last_entry() instead of list_tail_entry()	2016-01-15 11:40:52 -08:00
zpool.c	mm: zsmalloc: constify struct zs_pool name	2015-11-06 17:50:42 -08:00
zsmalloc.c	zsmalloc: calling zs_map_object() from irq is a bug	2017-11-15 18:21:03 -08:00
zswap.c	mm/zswap.c: delete an error message for a failed memory allocation in zswap_dstmem_prepare()	2017-07-06 16:24:35 -07:00