WSL2-Linux-Kernel/mm
Yang Shi 70cbc3cc78 mm: gup: fix the fast GUP race against THP collapse
Since general RCU GUP fast was introduced in commit 2667f50e8b ("mm:
introduce a general RCU get_user_pages_fast()"), a TLB flush is no longer
sufficient to handle concurrent GUP-fast in all cases, it only handles
traditional IPI-based GUP-fast correctly.  On architectures that send an
IPI broadcast on TLB flush, it works as expected.  But on the
architectures that do not use IPI to broadcast TLB flush, it may have the
below race:

   CPU A                                          CPU B
THP collapse                                     fast GUP
                                              gup_pmd_range() <-- see valid pmd
                                                  gup_pte_range() <-- work on pte
pmdp_collapse_flush() <-- clear pmd and flush
__collapse_huge_page_isolate()
    check page pinned <-- before GUP bump refcount
                                                      pin the page
                                                      check PTE <-- no change
__collapse_huge_page_copy()
    copy data to huge page
    ptep_clear()
install huge pmd for the huge page
                                                      return the stale page
discard the stale page

The race can be fixed by checking whether PMD is changed or not after
taking the page pin in fast GUP, just like what it does for PTE.  If the
PMD is changed it means there may be parallel THP collapse, so GUP should
back off.

Also update the stale comment about serializing against fast GUP in
khugepaged.

Link: https://lkml.kernel.org/r/20220907180144.555485-1-shy828301@gmail.com
Fixes: 2667f50e8b ("mm: introduce a general RCU get_user_pages_fast()")
Acked-by: David Hildenbrand <david@redhat.com>
Acked-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Yang Shi <shy828301@gmail.com>
Reviewed-by: John Hubbard <jhubbard@nvidia.com>
Cc: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
Cc: Hugh Dickins <hughd@google.com>
Cc: Jason Gunthorpe <jgg@nvidia.com>
Cc: "Kirill A. Shutemov" <kirill.shutemov@linux.intel.com>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Nicholas Piggin <npiggin@gmail.com>
Cc: Christophe Leroy <christophe.leroy@csgroup.eu>
Cc: <stable@vger.kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
2022-09-26 12:14:33 -07:00
..
damon mm/damon/dbgfs: fix memory leak when using debugfs_lookup() 2022-09-11 16:22:31 -07:00
kasan
kfence
Kconfig cxl for 6.0 2022-08-10 11:07:26 -07:00
Kconfig.debug
Makefile
backing-dev.c writeback: avoid use-after-free after removing device 2022-08-28 14:02:43 -07:00
balloon_compaction.c
bootmem_info.c bootmem: remove the vmemmap pages from kmemleak in put_page_bootmem 2022-08-28 14:02:45 -07:00
cma.c
cma.h
cma_debug.c
cma_sysfs.c
compaction.c
debug.c
debug_page_ref.c
debug_vm_pgtable.c
dmapool.c
early_ioremap.c
fadvise.c
failslab.c
filemap.c
folio-compat.c
frontswap.c
gup.c mm: gup: fix the fast GUP race against THP collapse 2022-09-26 12:14:33 -07:00
gup_test.c
gup_test.h
highmem.c
hmm.c
huge_memory.c mm/gup: fix FOLL_FORCE COW security issue and remove FOLL_COW 2022-08-20 15:17:44 -07:00
hugetlb.c mm/hugetlb: avoid corrupting page->mapping in hugetlb_mcopy_atomic_pte 2022-08-28 14:02:43 -07:00
hugetlb_cgroup.c
hugetlb_vmemmap.c
hugetlb_vmemmap.h
hwpoison-inject.c
init-mm.c
internal.h
interval_tree.c
io-mapping.c
ioremap.c
khugepaged.c mm: gup: fix the fast GUP race against THP collapse 2022-09-26 12:14:33 -07:00
kmemleak.c
ksm.c
list_lru.c
maccess.c
madvise.c
mapping_dirty_helpers.c
memblock.c memblock updates for v5.20 2022-08-09 09:48:30 -07:00
memcontrol.c
memfd.c
memory-failure.c mm/memory-failure: fall back to vma_address() when ->notify_failure() fails 2022-09-11 16:22:30 -07:00
memory.c
memory_hotplug.c
mempolicy.c
mempool.c
memremap.c
memtest.c
migrate.c
migrate_device.c mm/migrate_device.c: copy pte dirty bit to page 2022-09-11 16:22:30 -07:00
mincore.c
mlock.c
mm_init.c
mmap.c mm/hugetlb: fix hugetlb not supporting softdirty tracking 2022-08-20 15:17:45 -07:00
mmap_lock.c
mmu_gather.c
mmu_notifier.c
mmzone.c
mprotect.c mm/mprotect: only reference swap pfn page if type match 2022-08-28 14:02:46 -07:00
mremap.c
msync.c
nommu.c
oom_kill.c
page-writeback.c writeback: avoid use-after-free after removing device 2022-08-28 14:02:43 -07:00
page_alloc.c mm/page_alloc: fix race condition between build_all_zonelists and page allocation 2022-09-11 16:22:29 -07:00
page_counter.c
page_ext.c
page_idle.c
page_io.c
page_isolation.c
page_owner.c
page_poison.c
page_reporting.c
page_reporting.h
page_table_check.c
page_vma_mapped.c
pagewalk.c
percpu-internal.h
percpu-km.c
percpu-stats.c
percpu-vm.c
percpu.c
pgalloc-track.h
pgtable-generic.c
process_vm_access.c
ptdump.c
readahead.c
rmap.c
rodata_test.c
secretmem.c mm: fix dereferencing possible ERR_PTR 2022-09-11 16:22:31 -07:00
shmem.c shmem: update folio if shmem_replace_page() updates the page 2022-08-28 14:02:43 -07:00
shrinker_debug.c
shuffle.c
shuffle.h
slab.c
slab.h
slab_common.c
slob.c
slub.c
sparse-vmemmap.c
sparse.c
swap.c
swap.h
swap_cgroup.c
swap_slots.c
swap_state.c mm: fix VM_BUG_ON in __delete_from_swap_cache() 2022-09-11 16:22:31 -07:00
swapfile.c
truncate.c
usercopy.c
userfaultfd.c mm/uffd: reset write protection when unregister with wp-mode 2022-08-20 15:17:45 -07:00
util.c
vmacache.c
vmalloc.c
vmpressure.c
vmscan.c vmscan: check folio_test_private(), not folio_get_private() 2022-09-11 16:22:31 -07:00
vmstat.c mm: add DEVICE_ZONE to FOR_ALL_ZONES 2022-08-20 15:17:45 -07:00
workingset.c
z3fold.c
zbud.c
zpool.c
zsmalloc.c mm/zsmalloc: do not attempt to free IS_ERR handle 2022-08-28 14:02:44 -07:00
zswap.c