WSL2-Linux-Kernel/lib
Jens Axboe 4e5dff41be blk-mq: improve heavily contended tag case
Even with a number of waitqueues, we can get into a situation where we
are heavily contended on the waitqueue lock. I got a report on spc1
where we're spending seconds doing this. Arguably the use case is nasty,
I reproduce it with one device and 1000 threads banging on the device.
But that doesn't mean we shouldn't be handling it better.

What ends up happening is that a thread will fail to get a tag, add
itself to the waitqueue, and subsequently get woken up when a tag is
freed - only to find itself going back to sleep on the waitqueue.

Instead of waking all threads, use an exclusive wait and wake up our
sbitmap batch count instead. This seems to work well for me (massive
improvement for this use case), and it survives basic testing. But I
haven't fully verified it yet.

An additional improvement is running the queue and checking for a new
tag BEFORE needing to add ourselves to the waitqueue.

Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-12-22 11:09:37 -07:00
..
842 License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
fonts License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
lz4 lib/lz4: make arrays static const, reduces object code size 2017-10-03 17:54:25 -07:00
lzo License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
mpi Merge branch 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6 2017-11-14 10:52:09 -08:00
raid6 License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
reed_solomon
xz lib/xz: Add fall-through comments to a switch statement 2017-10-12 15:10:12 +02:00
zlib_deflate
zlib_inflate
zstd lib: Add zstd modules 2017-08-15 09:02:08 -07:00
.gitignore
Kconfig This pull request contains the following core changes: 2017-11-22 20:46:06 -10:00
Kconfig.debug locking/lockdep: Remove the cross-release locking checks 2017-12-12 12:38:51 +01:00
Kconfig.kasan
Kconfig.kgdb
Kconfig.ubsan
Makefile lib: test module for find_*_bit() functions 2017-11-17 16:10:02 -08:00
argv_split.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ashldi3.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
ashrdi3.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
asn1_decoder.c ASN.1: check for error from ASN1_OP_END__ACT actions 2017-12-08 15:13:27 +00:00
assoc_array.c Merge branch 'linus' into locking/core, to resolve conflicts 2017-11-07 10:32:44 +01:00
atomic64.c
atomic64_test.c lib/atomic64_test.c: add a test that atomic64_inc_not_zero() returns an int 2017-07-14 15:05:13 -07:00
audit.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
bcd.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
bch.c
bitmap.c documentation: kernel-api: add more info on bitmap functions 2017-10-19 13:01:40 -06:00
bitrev.c
bsearch.c lib/bsearch.c: micro-optimize pivot position calculation 2017-07-10 16:32:35 -07:00
btree.c
bug.c bug: define the "cut here" string in a single place 2017-11-17 16:10:01 -08:00
build_OID_registry
bust_spinlocks.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
chacha20.c
check_signature.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
checksum.c
clz_ctz.c
clz_tab.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
cmdline.c lib/cmdline.c: remove meaningless comment 2017-09-08 18:26:49 -07:00
cmpdi2.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
compat_audit.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
cordic.c
cpu_rmap.c
cpumask.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
crc-ccitt.c
crc-itu-t.c
crc-t10dif.c
crc4.c docs: clean up and add rest of CRC functions to kernel-api.rst 2017-09-26 15:01:20 -06:00
crc7.c
crc8.c docs: clean up and add rest of CRC functions to kernel-api.rst 2017-09-26 15:01:20 -06:00
crc16.c
crc32.c docs: clean up and add rest of CRC functions to kernel-api.rst 2017-09-26 15:01:20 -06:00
crc32defs.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
crc32test.c
ctype.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
debug_info.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
debug_locks.c
debugobjects.c debugobjects: Make kmemleak ignore debug objects 2017-08-14 16:51:01 +02:00
dec_and_lock.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
decompress.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
decompress_bunzip2.c
decompress_inflate.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
decompress_unlz4.c
decompress_unlzma.c
decompress_unlzo.c
decompress_unxz.c
devres.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
digsig.c lib/digsig: fix dereference of NULL user_key_payload 2017-10-12 17:16:40 +01:00
div64.c A relatively calm cycle for the docs tree again. 2017-11-13 08:25:06 -08:00
dma-debug.c lib/dma-debug.c: fix incorrect pfn calculation 2017-11-17 16:10:00 -08:00
dma-noop.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dma-virt.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dump_stack.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
dynamic_debug.c dynamic-debug-howto: fix optional/omitted ending line number to be LARGE instead of 0 2017-11-17 16:10:01 -08:00
dynamic_queue_limits.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next 2017-11-15 11:56:19 -08:00
earlycpio.c
errseq.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
extable.c lib/extable.c: use bsearch() library function in search_extable() 2017-07-10 16:32:35 -07:00
fault-inject.c fault-inject: fix wrong should_fail() decision in task context 2017-08-10 15:54:06 -07:00
fdt.c
fdt_empty_tree.c
fdt_ro.c
fdt_rw.c
fdt_strerror.c
fdt_sw.c
fdt_wip.c
find_bit.c
flex_array.c
flex_proportions.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
gcd.c lib/gcd: add kernel-doc notation 2017-10-07 10:45:14 -06:00
gen_crc32table.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
genalloc.c lib/genalloc.c: make the avail variable an atomic_long_t 2017-11-17 16:10:02 -08:00
glob.c
globtest.c
hexdump.c lib/hexdump.c: return -EINVAL in case of error in hex2bin() 2017-09-08 18:26:49 -07:00
hweight.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
idr.c mm, truncate: do not check mapping for every page being truncated 2017-11-15 18:21:06 -08:00
inflate.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
int_sqrt.c lib/int_sqrt: adjust comments 2017-11-17 16:10:01 -08:00
interval_tree.c
interval_tree_test.c lib/rbtree-test: lower default params 2017-11-17 16:10:02 -08:00
iomap.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
iomap_copy.c
iommu-common.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
iommu-helper.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ioremap.c License cleanup: add SPDX license identifiers to some files 2017-11-02 10:04:46 -07:00
iov_iter.c new primitive: iov_iter_for_each_range() 2017-10-11 22:36:54 -04:00
irq_poll.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
irq_regs.c
is_single_threaded.c
jedec_ddr_data.c
kasprintf.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
kfifo.c
klist.c
kobject.c
kobject_uevent.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2017-10-05 18:19:22 -07:00
kstrtox.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
kstrtox.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
lcm.c
libcrc32c.c
list_debug.c
list_sort.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
llist.c locking/atomics: COCCINELLE/treewide: Convert trivial ACCESS_ONCE() patterns to READ_ONCE()/WRITE_ONCE() 2017-10-25 11:01:08 +02:00
locking-selftest-hardirq.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-mutex.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-rlock-hardirq.h
locking-selftest-rlock-softirq.h
locking-selftest-rlock.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-rsem.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-rtmutex.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-softirq.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-spin-hardirq.h
locking-selftest-spin-softirq.h
locking-selftest-spin.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-wlock-hardirq.h
locking-selftest-wlock-softirq.h
locking-selftest-wlock.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest-wsem.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
locking-selftest.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
lockref.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
lru_cache.c
lshrdi3.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
memory-notifier-error-inject.c
memweight.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
muldi3.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
net_utils.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
netdev-notifier-error-inject.c
nlattr.c netlink: Relax attr validation for fixed length types 2017-12-07 14:00:57 -05:00
nmi_backtrace.c lib/nmi_backtrace.c: fix kernel text address leak 2017-11-17 16:10:02 -08:00
nodemask.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
notifier-error-inject.c
notifier-error-inject.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
of-reconfig-notifier-error-inject.c
oid_registry.c 509: fix printing uninitialized stack memory when OID is empty 2017-12-08 15:13:28 +00:00
once.c Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2017-11-04 09:26:51 +09:00
parman.c
parser.c
pci_iomap.c
percpu-refcount.c
percpu_counter.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
percpu_ida.c
percpu_test.c
plist.c
pm-notifier-error-inject.c
prime_numbers.c
radix-tree.c mm, truncate: do not check mapping for every page being truncated 2017-11-15 18:21:06 -08:00
random32.c treewide: Switch DEFINE_TIMER callbacks to struct timer_list * 2017-11-21 15:57:05 -08:00
ratelimit.c lib/ratelimit.c: use deferred printk() version 2017-10-03 17:54:26 -07:00
rational.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
rbtree.c lib/rbtree,drm/mm: add rbtree_replace_node_cached() 2017-12-14 16:00:48 -08:00
rbtree_test.c lib/rbtree-test: lower default params 2017-11-17 16:10:02 -08:00
reciprocal_div.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
refcount.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
rhashtable.c rhashtable: Documentation tweak 2017-09-19 15:18:33 -07:00
sbitmap.c blk-mq: improve heavily contended tag case 2017-12-22 11:09:37 -07:00
scatterlist.c lib/scatterlist: Introduce and export __sg_alloc_table_from_pages 2017-09-07 10:48:29 +01:00
seq_buf.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
sg_pool.c
sg_split.c
sha1.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
show_mem.c
siphash.c
smp_processor_id.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
sort.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
stackdepot.c
stmp_device.c
string.c lib: add module support to string tests 2017-11-17 16:10:01 -08:00
string_helpers.c mm: treewide: remove GFP_TEMPORARY allocation flag 2017-09-13 18:53:16 -07:00
strncpy_from_user.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
strnlen_user.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
swiotlb.c x86/mm: Add DMA support for SEV memory encryption 2017-11-07 15:35:58 +01:00
syscall.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
test-kstrtox.c
test-string_helpers.c
test_bitmap.c lib/test_bitmap.c: use ULL suffix for 64-bit constants 2017-09-13 18:53:15 -07:00
test_bpf.c bpf: add BPF_J{LT,LE,SLT,SLE} instructions 2017-08-09 16:53:56 -07:00
test_debug_virtual.c lib: add test module for CONFIG_DEBUG_VIRTUAL 2017-09-08 18:26:49 -07:00
test_find_bit.c lib: test module for find_*_bit() functions 2017-11-17 16:10:02 -08:00
test_firmware.c test_firmware: add batched firmware tests 2017-08-10 13:58:41 -07:00
test_hash.c
test_hexdump.c
test_kasan.c lib/test: delete five error messages for failed memory allocations 2017-11-17 16:10:01 -08:00
test_kmod.c lib/test: delete five error messages for failed memory allocations 2017-11-17 16:10:01 -08:00
test_list_sort.c lib/test: delete five error messages for failed memory allocations 2017-11-17 16:10:01 -08:00
test_module.c
test_parman.c
test_printf.c printk: hash addresses printed with %p 2017-11-29 12:09:02 +11:00
test_rhashtable.c test_rhashtable: remove initdata annotation 2017-09-21 20:40:55 -07:00
test_siphash.c
test_sort.c
test_static_key_base.c
test_static_keys.c
test_string.c lib: add module support to string tests 2017-11-17 16:10:01 -08:00
test_sysctl.c test_sysctl: test against int proc_dointvec() array support 2017-07-12 16:26:00 -07:00
test_user_copy.c
test_uuid.c uuid: fix incorrect uuid_equal conversion in test_uuid_test 2017-07-21 09:38:30 +02:00
textsearch.c
timerqueue.c
ts_bm.c
ts_fsm.c textsearch: fix typos in library helpers 2017-10-22 03:14:07 +01:00
ts_kmp.c textsearch: fix typos in library helpers 2017-10-22 03:14:07 +01:00
ubsan.c
ubsan.h License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
ucmpdi2.c move libgcc.h to include/linux 2017-12-01 13:09:40 -08:00
ucs2_string.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
usercopy.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
uuid.c
vsprintf.c vsprintf: don't use 'restricted_pointer()' when not restricting 2017-11-29 11:28:09 -08:00
win_minmax.c License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
xxhash.c lib: Add xxhash module 2017-08-15 09:02:07 -07:00