d065bd810b
This change reduces mmap_sem hold times that are caused by waiting for disk transfers when accessing file mapped VMAs. It introduces the VM_FAULT_ALLOW_RETRY flag, which indicates that the call site wants mmap_sem to be released if blocking on a pending disk transfer. In that case, filemap_fault() returns the VM_FAULT_RETRY status bit and do_page_fault() will then re-acquire mmap_sem and retry the page fault. It is expected that the retry will hit the same page which will now be cached, and thus it will complete with a low mmap_sem hold time. Tests: - microbenchmark: thread A mmaps a large file and does random read accesses to the mmaped area - achieves about 55 iterations/s. Thread B does mmap/munmap in a loop at a separate location - achieves 55 iterations/s before, 15000 iterations/s after. - We are seeing related effects in some applications in house, which show significant performance regressions when running without this change. [akpm@linux-foundation.org: fix warning & crash] Signed-off-by: Michel Lespinasse <walken@google.com> Acked-by: Rik van Riel <riel@redhat.com> Acked-by: Linus Torvalds <torvalds@linux-foundation.org> Cc: Nick Piggin <nickpiggin@yahoo.com.au> Reviewed-by: Wu Fengguang <fengguang.wu@intel.com> Cc: Ying Han <yinghan@google.com> Cc: Peter Zijlstra <a.p.zijlstra@chello.nl> Cc: Ingo Molnar <mingo@elte.hu> Cc: Thomas Gleixner <tglx@linutronix.de> Acked-by: "H. Peter Anvin" <hpa@zytor.com> Cc: <linux-arch@vger.kernel.org> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> |
||
|---|---|---|
| .. | ||
| kmemcheck | ||
| Makefile | ||
| dump_pagetables.c | ||
| extable.c | ||
| fault.c | ||
| gup.c | ||
| highmem_32.c | ||
| hugetlbpage.c | ||
| init.c | ||
| init_32.c | ||
| init_64.c | ||
| iomap_32.c | ||
| ioremap.c | ||
| k8topology_64.c | ||
| kmmio.c | ||
| memblock.c | ||
| memtest.c | ||
| mmap.c | ||
| mmio-mod.c | ||
| numa.c | ||
| numa_32.c | ||
| numa_64.c | ||
| pageattr-test.c | ||
| pageattr.c | ||
| pat.c | ||
| pat_internal.h | ||
| pat_rbtree.c | ||
| pf_in.c | ||
| pf_in.h | ||
| pgtable.c | ||
| pgtable_32.c | ||
| physaddr.c | ||
| physaddr.h | ||
| setup_nx.c | ||
| srat_32.c | ||
| srat_64.c | ||
| testmmiotrace.c | ||
| tlb.c | ||