2019-04-30 21:42:43 +03:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2017-01-17 16:03:22 +03:00
|
|
|
/*
|
|
|
|
* blk-mq scheduling framework
|
|
|
|
*
|
|
|
|
* Copyright (C) 2016 Jens Axboe
|
|
|
|
*/
|
|
|
|
#include <linux/kernel.h>
|
|
|
|
#include <linux/module.h>
|
|
|
|
#include <linux/blk-mq.h>
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
#include <linux/list_sort.h>
|
2017-01-17 16:03:22 +03:00
|
|
|
|
|
|
|
#include <trace/events/block.h>
|
|
|
|
|
|
|
|
#include "blk.h"
|
|
|
|
#include "blk-mq.h"
|
2017-05-04 17:24:40 +03:00
|
|
|
#include "blk-mq-debugfs.h"
|
2017-01-17 16:03:22 +03:00
|
|
|
#include "blk-mq-sched.h"
|
|
|
|
#include "blk-mq-tag.h"
|
|
|
|
#include "blk-wbt.h"
|
|
|
|
|
2017-06-21 02:56:13 +03:00
|
|
|
/*
|
|
|
|
* Mark a hardware queue as needing a restart. For shared queues, maintain
|
|
|
|
* a count of how many hardware queues are marked for restart.
|
|
|
|
*/
|
2018-12-17 09:14:05 +03:00
|
|
|
void blk_mq_sched_mark_restart_hctx(struct blk_mq_hw_ctx *hctx)
|
2017-06-21 02:56:13 +03:00
|
|
|
{
|
|
|
|
if (test_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state))
|
|
|
|
return;
|
|
|
|
|
blk-mq: remove synchronize_rcu() from blk_mq_del_queue_tag_set()
We have to remove synchronize_rcu() from blk_queue_cleanup(),
otherwise long delay can be caused during lun probe. For removing
it, we have to avoid to iterate the set->tag_list in IO path, eg,
blk_mq_sched_restart().
This patch reverts 5b79413946d (Revert "blk-mq: don't handle
TAG_SHARED in restart"). Given we have fixed enough IO hang issue,
and there isn't any reason to restart all queues in one tags any more,
see the following reasons:
1) blk-mq core can deal with shared-tags case well via blk_mq_get_driver_tag(),
which can wake up queues waiting for driver tag.
2) SCSI is a bit special because it may return BLK_STS_RESOURCE if queue,
target or host is ready, but SCSI built-in restart can cover all these well,
see scsi_end_request(), queue will be rerun after any request initiated from
this host/target is completed.
In my test on scsi_debug(8 luns), this patch may improve IOPS by 20% ~ 30%
when running I/O on these 8 luns concurrently.
Fixes: 705cda97ee3a ("blk-mq: Make it safe to use RCU to iterate over blk_mq_tag_set.tag_list")
Cc: Omar Sandoval <osandov@fb.com>
Cc: Bart Van Assche <bart.vanassche@wdc.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Martin K. Petersen <martin.petersen@oracle.com>
Cc: linux-scsi@vger.kernel.org
Reported-by: Andrew Jones <drjones@redhat.com>
Tested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-06-25 14:31:48 +03:00
|
|
|
set_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
|
2017-06-21 02:56:13 +03:00
|
|
|
}
|
2018-12-17 09:14:05 +03:00
|
|
|
EXPORT_SYMBOL_GPL(blk_mq_sched_mark_restart_hctx);
|
2017-06-21 02:56:13 +03:00
|
|
|
|
2021-10-09 15:25:42 +03:00
|
|
|
void __blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx)
|
2017-06-21 02:56:13 +03:00
|
|
|
{
|
blk-mq: remove synchronize_rcu() from blk_mq_del_queue_tag_set()
We have to remove synchronize_rcu() from blk_queue_cleanup(),
otherwise long delay can be caused during lun probe. For removing
it, we have to avoid to iterate the set->tag_list in IO path, eg,
blk_mq_sched_restart().
This patch reverts 5b79413946d (Revert "blk-mq: don't handle
TAG_SHARED in restart"). Given we have fixed enough IO hang issue,
and there isn't any reason to restart all queues in one tags any more,
see the following reasons:
1) blk-mq core can deal with shared-tags case well via blk_mq_get_driver_tag(),
which can wake up queues waiting for driver tag.
2) SCSI is a bit special because it may return BLK_STS_RESOURCE if queue,
target or host is ready, but SCSI built-in restart can cover all these well,
see scsi_end_request(), queue will be rerun after any request initiated from
this host/target is completed.
In my test on scsi_debug(8 luns), this patch may improve IOPS by 20% ~ 30%
when running I/O on these 8 luns concurrently.
Fixes: 705cda97ee3a ("blk-mq: Make it safe to use RCU to iterate over blk_mq_tag_set.tag_list")
Cc: Omar Sandoval <osandov@fb.com>
Cc: Bart Van Assche <bart.vanassche@wdc.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Martin K. Petersen <martin.petersen@oracle.com>
Cc: linux-scsi@vger.kernel.org
Reported-by: Andrew Jones <drjones@redhat.com>
Tested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-06-25 14:31:48 +03:00
|
|
|
clear_bit(BLK_MQ_S_SCHED_RESTART, &hctx->state);
|
2017-06-21 02:56:13 +03:00
|
|
|
|
2020-08-17 13:01:15 +03:00
|
|
|
/*
|
|
|
|
* Order clearing SCHED_RESTART and list_empty_careful(&hctx->dispatch)
|
|
|
|
* in blk_mq_run_hw_queue(). Its pair is the barrier in
|
|
|
|
* blk_mq_dispatch_rq_list(). So dispatch code won't see SCHED_RESTART,
|
|
|
|
* meantime new request added to hctx->dispatch is missed to check in
|
|
|
|
* blk_mq_run_hw_queue().
|
|
|
|
*/
|
|
|
|
smp_mb();
|
|
|
|
|
blk-mq: remove synchronize_rcu() from blk_mq_del_queue_tag_set()
We have to remove synchronize_rcu() from blk_queue_cleanup(),
otherwise long delay can be caused during lun probe. For removing
it, we have to avoid to iterate the set->tag_list in IO path, eg,
blk_mq_sched_restart().
This patch reverts 5b79413946d (Revert "blk-mq: don't handle
TAG_SHARED in restart"). Given we have fixed enough IO hang issue,
and there isn't any reason to restart all queues in one tags any more,
see the following reasons:
1) blk-mq core can deal with shared-tags case well via blk_mq_get_driver_tag(),
which can wake up queues waiting for driver tag.
2) SCSI is a bit special because it may return BLK_STS_RESOURCE if queue,
target or host is ready, but SCSI built-in restart can cover all these well,
see scsi_end_request(), queue will be rerun after any request initiated from
this host/target is completed.
In my test on scsi_debug(8 luns), this patch may improve IOPS by 20% ~ 30%
when running I/O on these 8 luns concurrently.
Fixes: 705cda97ee3a ("blk-mq: Make it safe to use RCU to iterate over blk_mq_tag_set.tag_list")
Cc: Omar Sandoval <osandov@fb.com>
Cc: Bart Van Assche <bart.vanassche@wdc.com>
Cc: Christoph Hellwig <hch@lst.de>
Cc: Martin K. Petersen <martin.petersen@oracle.com>
Cc: linux-scsi@vger.kernel.org
Reported-by: Andrew Jones <drjones@redhat.com>
Tested-by: Andrew Jones <drjones@redhat.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2018-06-25 14:31:48 +03:00
|
|
|
blk_mq_run_hw_queue(hctx, true);
|
2017-06-21 02:56:13 +03:00
|
|
|
}
|
|
|
|
|
2021-04-08 21:28:34 +03:00
|
|
|
static int sched_rq_cmp(void *priv, const struct list_head *a,
|
|
|
|
const struct list_head *b)
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
{
|
|
|
|
struct request *rqa = container_of(a, struct request, queuelist);
|
|
|
|
struct request *rqb = container_of(b, struct request, queuelist);
|
|
|
|
|
|
|
|
return rqa->mq_hctx > rqb->mq_hctx;
|
|
|
|
}
|
|
|
|
|
|
|
|
static bool blk_mq_dispatch_hctx_list(struct list_head *rq_list)
|
|
|
|
{
|
|
|
|
struct blk_mq_hw_ctx *hctx =
|
|
|
|
list_first_entry(rq_list, struct request, queuelist)->mq_hctx;
|
|
|
|
struct request *rq;
|
|
|
|
LIST_HEAD(hctx_list);
|
|
|
|
unsigned int count = 0;
|
|
|
|
|
|
|
|
list_for_each_entry(rq, rq_list, queuelist) {
|
|
|
|
if (rq->mq_hctx != hctx) {
|
|
|
|
list_cut_before(&hctx_list, rq_list, &rq->queuelist);
|
|
|
|
goto dispatch;
|
|
|
|
}
|
|
|
|
count++;
|
|
|
|
}
|
|
|
|
list_splice_tail_init(rq_list, &hctx_list);
|
|
|
|
|
|
|
|
dispatch:
|
2020-07-04 10:26:14 +03:00
|
|
|
return blk_mq_dispatch_rq_list(hctx, &hctx_list, count);
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
}
|
|
|
|
|
blk-mq: Rerun dispatching in the case of budget contention
If ever a thread running blk-mq code tries to get budget and fails it
immediately stops doing work and assumes that whenever budget is freed
up that queues will be kicked and whatever work the thread was trying
to do will be tried again.
One path where budget is freed and queues are kicked in the normal
case can be seen in scsi_finish_command(). Specifically:
- scsi_finish_command()
- scsi_device_unbusy()
- # Decrement "device_busy", AKA release budget
- scsi_io_completion()
- scsi_end_request()
- blk_mq_run_hw_queues()
The above is all well and good. The problem comes up when a thread
claims the budget but then releases it without actually dispatching
any work. Since we didn't schedule any work we'll never run the path
of finishing work / kicking the queues.
This isn't often actually a problem which is why this issue has
existed for a while and nobody noticed. Specifically we only get into
this situation when we unexpectedly found that we weren't going to do
any work. Code that later receives new work kicks the queues. All
good, right?
The problem shows up, however, if timing is just wrong and we hit a
race. To see this race let's think about the case where we only have
a budget of 1 (only one thread can hold budget). Now imagine that a
thread got budget and then decided not to dispatch work. It's about
to call put_budget() but then the thread gets context switched out for
a long, long time. While in this state, any and all kicks of the
queue (like the when we received new work) will be no-ops because
nobody can get budget. Finally the thread holding budget gets to run
again and returns. All the normal kicks will have been no-ops and we
have an I/O stall.
As you can see from the above, you need just the right timing to see
the race. To start with, the only case it happens if we thought we
had work, actually managed to get the budget, but then actually didn't
have work. That's pretty rare to start with. Even then, there's
usually a very small amount of time between realizing that there's no
work and putting the budget. During this small amount of time new
work has to come in and the queue kick has to make it all the way to
trying to get the budget and fail. It's pretty unlikely.
One case where this could have failed is illustrated by an example of
threads running blk_mq_do_dispatch_sched():
* Threads A and B both run has_work() at the same time with the same
"hctx". Imagine has_work() is exact. There's no lock, so it's OK
if Thread A and B both get back true.
* Thread B gets interrupted for a long time right after it decides
that there is work. Maybe its CPU gets an interrupt and the
interrupt handler is slow.
* Thread A runs, get budget, dispatches work.
* Thread A's work finishes and budget is released.
* Thread B finally runs again and gets budget.
* Since Thread A already took care of the work and no new work has
come in, Thread B will get NULL from dispatch_request(). I believe
this is specifically why dispatch_request() is allowed to return
NULL in the first place if has_work() must be exact.
* Thread B will now be holding the budget and is about to call
put_budget(), but hasn't called it yet.
* Thread B gets interrupted for a long time (again). Dang interrupts.
* Now Thread C (maybe with a different "hctx" but the same queue)
comes along and runs blk_mq_do_dispatch_sched().
* Thread C won't do anything because it can't get budget.
* Finally Thread B will run again and put the budget without kicking
any queues.
Even though the example above is with blk_mq_do_dispatch_sched() I
believe the race is possible any time someone is holding budget but
doesn't do work.
Unfortunately, the unlikely has become more likely if you happen to be
using the BFQ I/O scheduler. BFQ, by design, sometimes returns "true"
for has_work() but then NULL for dispatch_request() and stays in this
state for a while (currently up to 9 ms). Suddenly you only need one
race to hit, not two races in a row. With my current setup this is
easy to reproduce in reboot tests and traces have actually shown that
we hit a race similar to the one described above.
Note that we only need to fix blk_mq_do_dispatch_sched() and
blk_mq_do_dispatch_ctx() and not the other places that put budget. In
other cases we know that we have work to do on at least one "hctx" and
code already exists to kick that "hctx"'s queue. When that work
finally finishes all the queues will be kicked using the normal flow.
One last note is that (at least in the SCSI case) budget is shared by
all "hctx"s that have the same queue. Thus we need to make sure to
kick the whole queue, not just re-run dispatching on a single "hctx".
Signed-off-by: Douglas Anderson <dianders@chromium.org>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-04-20 19:24:53 +03:00
|
|
|
#define BLK_MQ_BUDGET_DELAY 3 /* ms units */
|
|
|
|
|
2017-10-27 07:43:30 +03:00
|
|
|
/*
|
|
|
|
* Only SCSI implements .get_budget and .put_budget, and SCSI restarts
|
|
|
|
* its queue by itself in its completion handler, so we don't need to
|
|
|
|
* restart queue if .get_budget() returns BLK_STS_NO_RESOURCE.
|
2020-04-24 18:03:21 +03:00
|
|
|
*
|
|
|
|
* Returns -EAGAIN if hctx->dispatch was found non-empty and run_work has to
|
|
|
|
* be run again. This is necessary to avoid starving flushes.
|
2017-10-27 07:43:30 +03:00
|
|
|
*/
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
static int __blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)
|
2017-10-14 12:22:26 +03:00
|
|
|
{
|
|
|
|
struct request_queue *q = hctx->queue;
|
|
|
|
struct elevator_queue *e = q->elevator;
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
bool multi_hctxs = false, run_queue = false;
|
|
|
|
bool dispatched = false, busy = false;
|
|
|
|
unsigned int max_dispatch;
|
2017-10-14 12:22:26 +03:00
|
|
|
LIST_HEAD(rq_list);
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
int count = 0;
|
|
|
|
|
|
|
|
if (hctx->dispatch_busy)
|
|
|
|
max_dispatch = 1;
|
|
|
|
else
|
|
|
|
max_dispatch = hctx->queue->nr_requests;
|
2017-10-14 12:22:26 +03:00
|
|
|
|
|
|
|
do {
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
struct request *rq;
|
2021-01-22 05:33:12 +03:00
|
|
|
int budget_token;
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
|
2018-11-02 01:41:41 +03:00
|
|
|
if (e->type->ops.has_work && !e->type->ops.has_work(hctx))
|
2017-10-14 12:22:26 +03:00
|
|
|
break;
|
2017-10-14 12:22:29 +03:00
|
|
|
|
2020-04-24 18:03:21 +03:00
|
|
|
if (!list_empty_careful(&hctx->dispatch)) {
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
busy = true;
|
2020-04-24 18:03:21 +03:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2021-01-22 05:33:12 +03:00
|
|
|
budget_token = blk_mq_get_dispatch_budget(q);
|
|
|
|
if (budget_token < 0)
|
2017-10-27 07:43:30 +03:00
|
|
|
break;
|
2017-10-14 12:22:29 +03:00
|
|
|
|
2018-11-02 01:41:41 +03:00
|
|
|
rq = e->type->ops.dispatch_request(hctx);
|
2017-10-14 12:22:29 +03:00
|
|
|
if (!rq) {
|
2021-01-22 05:33:12 +03:00
|
|
|
blk_mq_put_dispatch_budget(q, budget_token);
|
blk-mq: Rerun dispatching in the case of budget contention
If ever a thread running blk-mq code tries to get budget and fails it
immediately stops doing work and assumes that whenever budget is freed
up that queues will be kicked and whatever work the thread was trying
to do will be tried again.
One path where budget is freed and queues are kicked in the normal
case can be seen in scsi_finish_command(). Specifically:
- scsi_finish_command()
- scsi_device_unbusy()
- # Decrement "device_busy", AKA release budget
- scsi_io_completion()
- scsi_end_request()
- blk_mq_run_hw_queues()
The above is all well and good. The problem comes up when a thread
claims the budget but then releases it without actually dispatching
any work. Since we didn't schedule any work we'll never run the path
of finishing work / kicking the queues.
This isn't often actually a problem which is why this issue has
existed for a while and nobody noticed. Specifically we only get into
this situation when we unexpectedly found that we weren't going to do
any work. Code that later receives new work kicks the queues. All
good, right?
The problem shows up, however, if timing is just wrong and we hit a
race. To see this race let's think about the case where we only have
a budget of 1 (only one thread can hold budget). Now imagine that a
thread got budget and then decided not to dispatch work. It's about
to call put_budget() but then the thread gets context switched out for
a long, long time. While in this state, any and all kicks of the
queue (like the when we received new work) will be no-ops because
nobody can get budget. Finally the thread holding budget gets to run
again and returns. All the normal kicks will have been no-ops and we
have an I/O stall.
As you can see from the above, you need just the right timing to see
the race. To start with, the only case it happens if we thought we
had work, actually managed to get the budget, but then actually didn't
have work. That's pretty rare to start with. Even then, there's
usually a very small amount of time between realizing that there's no
work and putting the budget. During this small amount of time new
work has to come in and the queue kick has to make it all the way to
trying to get the budget and fail. It's pretty unlikely.
One case where this could have failed is illustrated by an example of
threads running blk_mq_do_dispatch_sched():
* Threads A and B both run has_work() at the same time with the same
"hctx". Imagine has_work() is exact. There's no lock, so it's OK
if Thread A and B both get back true.
* Thread B gets interrupted for a long time right after it decides
that there is work. Maybe its CPU gets an interrupt and the
interrupt handler is slow.
* Thread A runs, get budget, dispatches work.
* Thread A's work finishes and budget is released.
* Thread B finally runs again and gets budget.
* Since Thread A already took care of the work and no new work has
come in, Thread B will get NULL from dispatch_request(). I believe
this is specifically why dispatch_request() is allowed to return
NULL in the first place if has_work() must be exact.
* Thread B will now be holding the budget and is about to call
put_budget(), but hasn't called it yet.
* Thread B gets interrupted for a long time (again). Dang interrupts.
* Now Thread C (maybe with a different "hctx" but the same queue)
comes along and runs blk_mq_do_dispatch_sched().
* Thread C won't do anything because it can't get budget.
* Finally Thread B will run again and put the budget without kicking
any queues.
Even though the example above is with blk_mq_do_dispatch_sched() I
believe the race is possible any time someone is holding budget but
doesn't do work.
Unfortunately, the unlikely has become more likely if you happen to be
using the BFQ I/O scheduler. BFQ, by design, sometimes returns "true"
for has_work() but then NULL for dispatch_request() and stays in this
state for a while (currently up to 9 ms). Suddenly you only need one
race to hit, not two races in a row. With my current setup this is
easy to reproduce in reboot tests and traces have actually shown that
we hit a race similar to the one described above.
Note that we only need to fix blk_mq_do_dispatch_sched() and
blk_mq_do_dispatch_ctx() and not the other places that put budget. In
other cases we know that we have work to do on at least one "hctx" and
code already exists to kick that "hctx"'s queue. When that work
finally finishes all the queues will be kicked using the normal flow.
One last note is that (at least in the SCSI case) budget is shared by
all "hctx"s that have the same queue. Thus we need to make sure to
kick the whole queue, not just re-run dispatching on a single "hctx".
Signed-off-by: Douglas Anderson <dianders@chromium.org>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-04-20 19:24:53 +03:00
|
|
|
/*
|
|
|
|
* We're releasing without dispatching. Holding the
|
|
|
|
* budget could have blocked any "hctx"s with the
|
|
|
|
* same queue and if we didn't dispatch then there's
|
|
|
|
* no guarantee anyone will kick the queue. Kick it
|
|
|
|
* ourselves.
|
|
|
|
*/
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
run_queue = true;
|
2017-10-14 12:22:29 +03:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2021-01-22 05:33:12 +03:00
|
|
|
blk_mq_set_rq_budget_token(rq, budget_token);
|
|
|
|
|
2017-10-14 12:22:29 +03:00
|
|
|
/*
|
|
|
|
* Now this rq owns the budget which has to be released
|
|
|
|
* if this rq won't be queued to driver via .queue_rq()
|
|
|
|
* in blk_mq_dispatch_rq_list().
|
|
|
|
*/
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
list_add_tail(&rq->queuelist, &rq_list);
|
2021-06-03 13:47:21 +03:00
|
|
|
count++;
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
if (rq->mq_hctx != hctx)
|
|
|
|
multi_hctxs = true;
|
2021-06-03 13:47:21 +03:00
|
|
|
|
|
|
|
/*
|
|
|
|
* If we cannot get tag for the request, stop dequeueing
|
|
|
|
* requests from the IO scheduler. We are unlikely to be able
|
|
|
|
* to submit them anyway and it creates false impression for
|
|
|
|
* scheduling heuristics that the device can take more IO.
|
|
|
|
*/
|
|
|
|
if (!blk_mq_get_driver_tag(rq))
|
|
|
|
break;
|
|
|
|
} while (count < max_dispatch);
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
|
|
|
|
if (!count) {
|
|
|
|
if (run_queue)
|
|
|
|
blk_mq_delay_run_hw_queues(q, BLK_MQ_BUDGET_DELAY);
|
|
|
|
} else if (multi_hctxs) {
|
|
|
|
/*
|
|
|
|
* Requests from different hctx may be dequeued from some
|
|
|
|
* schedulers, such as bfq and deadline.
|
|
|
|
*
|
|
|
|
* Sort the requests in the list according to their hctx,
|
|
|
|
* dispatch batching requests from same hctx at a time.
|
|
|
|
*/
|
|
|
|
list_sort(NULL, &rq_list, sched_rq_cmp);
|
|
|
|
do {
|
|
|
|
dispatched |= blk_mq_dispatch_hctx_list(&rq_list);
|
|
|
|
} while (!list_empty(&rq_list));
|
|
|
|
} else {
|
|
|
|
dispatched = blk_mq_dispatch_rq_list(hctx, &rq_list, count);
|
|
|
|
}
|
|
|
|
|
|
|
|
if (busy)
|
|
|
|
return -EAGAIN;
|
|
|
|
return !!dispatched;
|
|
|
|
}
|
|
|
|
|
|
|
|
static int blk_mq_do_dispatch_sched(struct blk_mq_hw_ctx *hctx)
|
|
|
|
{
|
block: limit request dispatch loop duration
When IO requests are made continuously and the target block device
handles requests faster than request arrival, the request dispatch loop
keeps on repeating to dispatch the arriving requests very long time,
more than a minute. Since the loop runs as a workqueue worker task, the
very long loop duration triggers workqueue watchdog timeout and BUG [1].
To avoid the very long loop duration, break the loop periodically. When
opportunity to dispatch requests still exists, check need_resched(). If
need_resched() returns true, the dispatch loop already consumed its time
slice, then reschedule the dispatch work and break the loop. With heavy
IO load, need_resched() does not return true for 20~30 seconds. To cover
such case, check time spent in the dispatch loop with jiffies. If more
than 1 second is spent, reschedule the dispatch work and break the loop.
[1]
[ 609.691437] BUG: workqueue lockup - pool cpus=10 node=1 flags=0x0 nice=-20 stuck for 35s!
[ 609.701820] Showing busy workqueues and worker pools:
[ 609.707915] workqueue events: flags=0x0
[ 609.712615] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 609.712626] pending: drm_fb_helper_damage_work [drm_kms_helper]
[ 609.712687] workqueue events_freezable: flags=0x4
[ 609.732943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 609.732952] pending: pci_pme_list_scan
[ 609.732968] workqueue events_power_efficient: flags=0x80
[ 609.751947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 609.751955] pending: neigh_managed_work
[ 609.752018] workqueue kblockd: flags=0x18
[ 609.769480] pwq 21: cpus=10 node=1 flags=0x0 nice=-20 active=3/256 refcnt=4
[ 609.769488] in-flight: 1020:blk_mq_run_work_fn
[ 609.769498] pending: blk_mq_timeout_work, blk_mq_run_work_fn
[ 609.769744] pool 21: cpus=10 node=1 flags=0x0 nice=-20 hung=35s workers=2 idle: 67
[ 639.899730] BUG: workqueue lockup - pool cpus=10 node=1 flags=0x0 nice=-20 stuck for 66s!
[ 639.909513] Showing busy workqueues and worker pools:
[ 639.915404] workqueue events: flags=0x0
[ 639.920197] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 639.920215] pending: drm_fb_helper_damage_work [drm_kms_helper]
[ 639.920365] workqueue kblockd: flags=0x18
[ 639.939932] pwq 21: cpus=10 node=1 flags=0x0 nice=-20 active=3/256 refcnt=4
[ 639.939942] in-flight: 1020:blk_mq_run_work_fn
[ 639.939955] pending: blk_mq_timeout_work, blk_mq_run_work_fn
[ 639.940212] pool 21: cpus=10 node=1 flags=0x0 nice=-20 hung=66s workers=2 idle: 67
Fixes: 6e6fcbc27e778 ("blk-mq: support batching dispatch in case of io")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Cc: stable@vger.kernel.org # v5.10+
Link: https://lore.kernel.org/linux-block/20220310091649.zypaem5lkyfadymg@shindev/
Link: https://lore.kernel.org/r/20220318022641.133484-1-shinichiro.kawasaki@wdc.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-03-18 05:26:41 +03:00
|
|
|
unsigned long end = jiffies + HZ;
|
blk-mq: support batching dispatch in case of io
More and more drivers want to get batching requests queued from
block layer, such as mmc, and tcp based storage drivers. Also
current in-tree users have virtio-scsi, virtio-blk and nvme.
For none, we already support batching dispatch.
But for io scheduler, every time we just take one request from scheduler
and pass the single request to blk_mq_dispatch_rq_list(). This way makes
batching dispatch not possible when io scheduler is applied. One reason
is that we don't want to hurt sequential IO performance, becasue IO
merge chance is reduced if more requests are dequeued from scheduler
queue.
Try to support batching dispatch for io scheduler by starting with the
following simple approach:
1) still make sure we can get budget before dequeueing request
2) use hctx->dispatch_busy to evaluate if queue is busy, if it is busy
we fackback to non-batching dispatch, otherwise dequeue as many as
possible requests from scheduler, and pass them to blk_mq_dispatch_rq_list().
Wrt. 2), we use similar policy for none, and turns out that SCSI SSD
performance got improved much.
In future, maybe we can develop more intelligent algorithem for batching
dispatch.
Baolin has tested this patch and found that MMC performance is improved[3].
[1] https://lore.kernel.org/linux-block/20200512075501.GF1531898@T590/#r
[2] https://lore.kernel.org/linux-block/fe6bd8b9-6ed9-b225-f80c-314746133722@grimberg.me/
[3] https://lore.kernel.org/linux-block/CADBw62o9eTQDJ9RvNgEqSpXmg6Xcq=2TxH0Hfxhp29uF2W=TXA@mail.gmail.com/
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Tested-by: Baolin Wang <baolin.wang7@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Baolin Wang <baolin.wang7@gmail.com>
Cc: Christoph Hellwig <hch@infradead.org>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-06-30 13:25:01 +03:00
|
|
|
int ret;
|
|
|
|
|
|
|
|
do {
|
|
|
|
ret = __blk_mq_do_dispatch_sched(hctx);
|
block: limit request dispatch loop duration
When IO requests are made continuously and the target block device
handles requests faster than request arrival, the request dispatch loop
keeps on repeating to dispatch the arriving requests very long time,
more than a minute. Since the loop runs as a workqueue worker task, the
very long loop duration triggers workqueue watchdog timeout and BUG [1].
To avoid the very long loop duration, break the loop periodically. When
opportunity to dispatch requests still exists, check need_resched(). If
need_resched() returns true, the dispatch loop already consumed its time
slice, then reschedule the dispatch work and break the loop. With heavy
IO load, need_resched() does not return true for 20~30 seconds. To cover
such case, check time spent in the dispatch loop with jiffies. If more
than 1 second is spent, reschedule the dispatch work and break the loop.
[1]
[ 609.691437] BUG: workqueue lockup - pool cpus=10 node=1 flags=0x0 nice=-20 stuck for 35s!
[ 609.701820] Showing busy workqueues and worker pools:
[ 609.707915] workqueue events: flags=0x0
[ 609.712615] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 609.712626] pending: drm_fb_helper_damage_work [drm_kms_helper]
[ 609.712687] workqueue events_freezable: flags=0x4
[ 609.732943] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 609.732952] pending: pci_pme_list_scan
[ 609.732968] workqueue events_power_efficient: flags=0x80
[ 609.751947] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 609.751955] pending: neigh_managed_work
[ 609.752018] workqueue kblockd: flags=0x18
[ 609.769480] pwq 21: cpus=10 node=1 flags=0x0 nice=-20 active=3/256 refcnt=4
[ 609.769488] in-flight: 1020:blk_mq_run_work_fn
[ 609.769498] pending: blk_mq_timeout_work, blk_mq_run_work_fn
[ 609.769744] pool 21: cpus=10 node=1 flags=0x0 nice=-20 hung=35s workers=2 idle: 67
[ 639.899730] BUG: workqueue lockup - pool cpus=10 node=1 flags=0x0 nice=-20 stuck for 66s!
[ 639.909513] Showing busy workqueues and worker pools:
[ 639.915404] workqueue events: flags=0x0
[ 639.920197] pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[ 639.920215] pending: drm_fb_helper_damage_work [drm_kms_helper]
[ 639.920365] workqueue kblockd: flags=0x18
[ 639.939932] pwq 21: cpus=10 node=1 flags=0x0 nice=-20 active=3/256 refcnt=4
[ 639.939942] in-flight: 1020:blk_mq_run_work_fn
[ 639.939955] pending: blk_mq_timeout_work, blk_mq_run_work_fn
[ 639.940212] pool 21: cpus=10 node=1 flags=0x0 nice=-20 hung=66s workers=2 idle: 67
Fixes: 6e6fcbc27e778 ("blk-mq: support batching dispatch in case of io")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Cc: stable@vger.kernel.org # v5.10+
Link: https://lore.kernel.org/linux-block/20220310091649.zypaem5lkyfadymg@shindev/
Link: https://lore.kernel.org/r/20220318022641.133484-1-shinichiro.kawasaki@wdc.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-03-18 05:26:41 +03:00
|
|
|
if (ret != 1)
|
|
|
|
break;
|
|
|
|
if (need_resched() || time_is_before_jiffies(end)) {
|
|
|
|
blk_mq_delay_run_hw_queue(hctx, 0);
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
} while (1);
|
2020-04-24 18:03:21 +03:00
|
|
|
|
|
|
|
return ret;
|
2017-10-14 12:22:26 +03:00
|
|
|
}
|
|
|
|
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
static struct blk_mq_ctx *blk_mq_next_ctx(struct blk_mq_hw_ctx *hctx,
|
|
|
|
struct blk_mq_ctx *ctx)
|
|
|
|
{
|
2018-10-29 22:13:29 +03:00
|
|
|
unsigned short idx = ctx->index_hw[hctx->type];
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
|
|
|
|
if (++idx == hctx->nr_ctx)
|
|
|
|
idx = 0;
|
|
|
|
|
|
|
|
return hctx->ctxs[idx];
|
|
|
|
}
|
|
|
|
|
2017-10-27 07:43:30 +03:00
|
|
|
/*
|
|
|
|
* Only SCSI implements .get_budget and .put_budget, and SCSI restarts
|
|
|
|
* its queue by itself in its completion handler, so we don't need to
|
|
|
|
* restart queue if .get_budget() returns BLK_STS_NO_RESOURCE.
|
2020-04-24 18:03:21 +03:00
|
|
|
*
|
|
|
|
* Returns -EAGAIN if hctx->dispatch was found non-empty and run_work has to
|
2020-07-31 04:42:32 +03:00
|
|
|
* be run again. This is necessary to avoid starving flushes.
|
2017-10-27 07:43:30 +03:00
|
|
|
*/
|
2020-04-24 18:03:21 +03:00
|
|
|
static int blk_mq_do_dispatch_ctx(struct blk_mq_hw_ctx *hctx)
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
{
|
|
|
|
struct request_queue *q = hctx->queue;
|
|
|
|
LIST_HEAD(rq_list);
|
|
|
|
struct blk_mq_ctx *ctx = READ_ONCE(hctx->dispatch_from);
|
2020-04-24 18:03:21 +03:00
|
|
|
int ret = 0;
|
2020-06-30 13:24:57 +03:00
|
|
|
struct request *rq;
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
|
|
|
|
do {
|
2021-01-22 05:33:12 +03:00
|
|
|
int budget_token;
|
|
|
|
|
2020-04-24 18:03:21 +03:00
|
|
|
if (!list_empty_careful(&hctx->dispatch)) {
|
|
|
|
ret = -EAGAIN;
|
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
if (!sbitmap_any_bit_set(&hctx->ctx_map))
|
|
|
|
break;
|
|
|
|
|
2021-01-22 05:33:12 +03:00
|
|
|
budget_token = blk_mq_get_dispatch_budget(q);
|
|
|
|
if (budget_token < 0)
|
2017-10-27 07:43:30 +03:00
|
|
|
break;
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
|
|
|
|
rq = blk_mq_dequeue_from_ctx(hctx, ctx);
|
|
|
|
if (!rq) {
|
2021-01-22 05:33:12 +03:00
|
|
|
blk_mq_put_dispatch_budget(q, budget_token);
|
blk-mq: Rerun dispatching in the case of budget contention
If ever a thread running blk-mq code tries to get budget and fails it
immediately stops doing work and assumes that whenever budget is freed
up that queues will be kicked and whatever work the thread was trying
to do will be tried again.
One path where budget is freed and queues are kicked in the normal
case can be seen in scsi_finish_command(). Specifically:
- scsi_finish_command()
- scsi_device_unbusy()
- # Decrement "device_busy", AKA release budget
- scsi_io_completion()
- scsi_end_request()
- blk_mq_run_hw_queues()
The above is all well and good. The problem comes up when a thread
claims the budget but then releases it without actually dispatching
any work. Since we didn't schedule any work we'll never run the path
of finishing work / kicking the queues.
This isn't often actually a problem which is why this issue has
existed for a while and nobody noticed. Specifically we only get into
this situation when we unexpectedly found that we weren't going to do
any work. Code that later receives new work kicks the queues. All
good, right?
The problem shows up, however, if timing is just wrong and we hit a
race. To see this race let's think about the case where we only have
a budget of 1 (only one thread can hold budget). Now imagine that a
thread got budget and then decided not to dispatch work. It's about
to call put_budget() but then the thread gets context switched out for
a long, long time. While in this state, any and all kicks of the
queue (like the when we received new work) will be no-ops because
nobody can get budget. Finally the thread holding budget gets to run
again and returns. All the normal kicks will have been no-ops and we
have an I/O stall.
As you can see from the above, you need just the right timing to see
the race. To start with, the only case it happens if we thought we
had work, actually managed to get the budget, but then actually didn't
have work. That's pretty rare to start with. Even then, there's
usually a very small amount of time between realizing that there's no
work and putting the budget. During this small amount of time new
work has to come in and the queue kick has to make it all the way to
trying to get the budget and fail. It's pretty unlikely.
One case where this could have failed is illustrated by an example of
threads running blk_mq_do_dispatch_sched():
* Threads A and B both run has_work() at the same time with the same
"hctx". Imagine has_work() is exact. There's no lock, so it's OK
if Thread A and B both get back true.
* Thread B gets interrupted for a long time right after it decides
that there is work. Maybe its CPU gets an interrupt and the
interrupt handler is slow.
* Thread A runs, get budget, dispatches work.
* Thread A's work finishes and budget is released.
* Thread B finally runs again and gets budget.
* Since Thread A already took care of the work and no new work has
come in, Thread B will get NULL from dispatch_request(). I believe
this is specifically why dispatch_request() is allowed to return
NULL in the first place if has_work() must be exact.
* Thread B will now be holding the budget and is about to call
put_budget(), but hasn't called it yet.
* Thread B gets interrupted for a long time (again). Dang interrupts.
* Now Thread C (maybe with a different "hctx" but the same queue)
comes along and runs blk_mq_do_dispatch_sched().
* Thread C won't do anything because it can't get budget.
* Finally Thread B will run again and put the budget without kicking
any queues.
Even though the example above is with blk_mq_do_dispatch_sched() I
believe the race is possible any time someone is holding budget but
doesn't do work.
Unfortunately, the unlikely has become more likely if you happen to be
using the BFQ I/O scheduler. BFQ, by design, sometimes returns "true"
for has_work() but then NULL for dispatch_request() and stays in this
state for a while (currently up to 9 ms). Suddenly you only need one
race to hit, not two races in a row. With my current setup this is
easy to reproduce in reboot tests and traces have actually shown that
we hit a race similar to the one described above.
Note that we only need to fix blk_mq_do_dispatch_sched() and
blk_mq_do_dispatch_ctx() and not the other places that put budget. In
other cases we know that we have work to do on at least one "hctx" and
code already exists to kick that "hctx"'s queue. When that work
finally finishes all the queues will be kicked using the normal flow.
One last note is that (at least in the SCSI case) budget is shared by
all "hctx"s that have the same queue. Thus we need to make sure to
kick the whole queue, not just re-run dispatching on a single "hctx".
Signed-off-by: Douglas Anderson <dianders@chromium.org>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2020-04-20 19:24:53 +03:00
|
|
|
/*
|
|
|
|
* We're releasing without dispatching. Holding the
|
|
|
|
* budget could have blocked any "hctx"s with the
|
|
|
|
* same queue and if we didn't dispatch then there's
|
|
|
|
* no guarantee anyone will kick the queue. Kick it
|
|
|
|
* ourselves.
|
|
|
|
*/
|
|
|
|
blk_mq_delay_run_hw_queues(q, BLK_MQ_BUDGET_DELAY);
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
break;
|
|
|
|
}
|
|
|
|
|
2021-01-22 05:33:12 +03:00
|
|
|
blk_mq_set_rq_budget_token(rq, budget_token);
|
|
|
|
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
/*
|
|
|
|
* Now this rq owns the budget which has to be released
|
|
|
|
* if this rq won't be queued to driver via .queue_rq()
|
|
|
|
* in blk_mq_dispatch_rq_list().
|
|
|
|
*/
|
|
|
|
list_add(&rq->queuelist, &rq_list);
|
|
|
|
|
|
|
|
/* round robin for fair dispatch */
|
|
|
|
ctx = blk_mq_next_ctx(hctx, rq->mq_ctx);
|
|
|
|
|
2020-06-30 13:25:00 +03:00
|
|
|
} while (blk_mq_dispatch_rq_list(rq->mq_hctx, &rq_list, 1));
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
|
|
|
|
WRITE_ONCE(hctx->dispatch_from, ctx);
|
2020-04-24 18:03:21 +03:00
|
|
|
return ret;
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
}
|
|
|
|
|
2020-04-29 04:36:32 +03:00
|
|
|
static int __blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
|
2017-01-17 16:03:22 +03:00
|
|
|
{
|
blk-mq: use the right hctx when getting a driver tag fails
While dispatching requests, if we fail to get a driver tag, we mark the
hardware queue as waiting for a tag and put the requests on a
hctx->dispatch list to be run later when a driver tag is freed. However,
blk_mq_dispatch_rq_list() may dispatch requests from multiple hardware
queues if using a single-queue scheduler with a multiqueue device. If
blk_mq_get_driver_tag() fails, it doesn't update the hardware queue we
are processing. This means we end up using the hardware queue of the
previous request, which may or may not be the same as that of the
current request. If it isn't, the wrong hardware queue will end up
waiting for a tag, and the requests will be on the wrong dispatch list,
leading to a hang.
The fix is twofold:
1. Make sure we save which hardware queue we were trying to get a
request for in blk_mq_get_driver_tag() regardless of whether it
succeeds or not.
2. Make blk_mq_dispatch_rq_list() take a request_queue instead of a
blk_mq_hw_queue to make it clear that it must handle multiple
hardware queues, since I've already messed this up on a couple of
occasions.
This didn't appear in testing with nvme and mq-deadline because nvme has
more driver tags than the default number of scheduler tags. However,
with the blk_mq_update_nr_hw_queues() fix, it showed up with nbd.
Signed-off-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Jens Axboe <axboe@fb.com>
2017-04-07 17:56:26 +03:00
|
|
|
struct request_queue *q = hctx->queue;
|
2021-06-18 04:59:22 +03:00
|
|
|
const bool has_sched = q->elevator;
|
2020-04-24 18:03:21 +03:00
|
|
|
int ret = 0;
|
2017-01-17 16:03:22 +03:00
|
|
|
LIST_HEAD(rq_list);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* If we have previous entries on our dispatch list, grab them first for
|
|
|
|
* more fair dispatch.
|
|
|
|
*/
|
|
|
|
if (!list_empty_careful(&hctx->dispatch)) {
|
|
|
|
spin_lock(&hctx->lock);
|
|
|
|
if (!list_empty(&hctx->dispatch))
|
|
|
|
list_splice_init(&hctx->dispatch, &rq_list);
|
|
|
|
spin_unlock(&hctx->lock);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Only ask the scheduler for requests, if we didn't have residual
|
|
|
|
* requests from the dispatch list. This is to avoid the case where
|
|
|
|
* we only ever dispatch a fraction of the requests available because
|
|
|
|
* of low device queue depth. Once we pull requests out of the IO
|
|
|
|
* scheduler, we can no longer merge or sort them. So it's best to
|
|
|
|
* leave them there for as long as we can. Mark the hw queue as
|
|
|
|
* needing a restart in that case.
|
2017-10-14 12:22:26 +03:00
|
|
|
*
|
|
|
|
* We want to dispatch from the scheduler if there was nothing
|
|
|
|
* on the dispatch list or we were able to dispatch from the
|
|
|
|
* dispatch list.
|
2017-01-17 16:03:22 +03:00
|
|
|
*/
|
2017-01-26 22:40:07 +03:00
|
|
|
if (!list_empty(&rq_list)) {
|
2017-02-22 21:58:30 +03:00
|
|
|
blk_mq_sched_mark_restart_hctx(hctx);
|
2020-06-30 13:25:00 +03:00
|
|
|
if (blk_mq_dispatch_rq_list(hctx, &rq_list, 0)) {
|
2021-06-18 04:59:22 +03:00
|
|
|
if (has_sched)
|
2020-04-24 18:03:21 +03:00
|
|
|
ret = blk_mq_do_dispatch_sched(hctx);
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
else
|
2020-04-24 18:03:21 +03:00
|
|
|
ret = blk_mq_do_dispatch_ctx(hctx);
|
blk-mq-sched: improve dispatching from sw queue
SCSI devices use host-wide tagset, and the shared driver tag space is
often quite big. However, there is also a queue depth for each lun(
.cmd_per_lun), which is often small, for example, on both lpfc and
qla2xxx, .cmd_per_lun is just 3.
So lots of requests may stay in sw queue, and we always flush all
belonging to same hw queue and dispatch them all to driver.
Unfortunately it is easy to cause queue busy because of the small
.cmd_per_lun. Once these requests are flushed out, they have to stay in
hctx->dispatch, and no bio merge can happen on these requests, and
sequential IO performance is harmed.
This patch introduces blk_mq_dequeue_from_ctx for dequeuing a request
from a sw queue, so that we can dispatch them in scheduler's way. We can
then avoid dequeueing too many requests from sw queue, since we don't
flush ->dispatch completely.
This patch improves dispatching from sw queue by using the .get_budget
and .put_budget callbacks.
Reviewed-by: Omar Sandoval <osandov@fb.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2017-10-14 12:22:30 +03:00
|
|
|
}
|
2021-06-18 04:59:22 +03:00
|
|
|
} else if (has_sched) {
|
2020-04-24 18:03:21 +03:00
|
|
|
ret = blk_mq_do_dispatch_sched(hctx);
|
2018-07-03 18:03:16 +03:00
|
|
|
} else if (hctx->dispatch_busy) {
|
|
|
|
/* dequeue request one by one from sw queue if queue is busy */
|
2020-04-24 18:03:21 +03:00
|
|
|
ret = blk_mq_do_dispatch_ctx(hctx);
|
2017-10-14 12:22:26 +03:00
|
|
|
} else {
|
2017-01-26 22:40:07 +03:00
|
|
|
blk_mq_flush_busy_ctxs(hctx, &rq_list);
|
2020-06-30 13:25:00 +03:00
|
|
|
blk_mq_dispatch_rq_list(hctx, &rq_list, 0);
|
2017-02-17 21:39:26 +03:00
|
|
|
}
|
2020-04-24 18:03:21 +03:00
|
|
|
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
|
|
|
|
void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx)
|
|
|
|
{
|
|
|
|
struct request_queue *q = hctx->queue;
|
|
|
|
|
|
|
|
/* RCU or SRCU read lock is needed before checking quiesced flag */
|
|
|
|
if (unlikely(blk_mq_hctx_stopped(hctx) || blk_queue_quiesced(q)))
|
|
|
|
return;
|
|
|
|
|
|
|
|
hctx->run++;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* A return of -EAGAIN is an indication that hctx->dispatch is not
|
|
|
|
* empty and we must run again in order to avoid starving flushes.
|
|
|
|
*/
|
|
|
|
if (__blk_mq_sched_dispatch_requests(hctx) == -EAGAIN) {
|
|
|
|
if (__blk_mq_sched_dispatch_requests(hctx) == -EAGAIN)
|
|
|
|
blk_mq_run_hw_queue(hctx, true);
|
|
|
|
}
|
2017-01-17 16:03:22 +03:00
|
|
|
}
|
|
|
|
|
2021-10-20 22:00:49 +03:00
|
|
|
bool blk_mq_sched_bio_merge(struct request_queue *q, struct bio *bio,
|
2019-06-06 13:29:01 +03:00
|
|
|
unsigned int nr_segs)
|
2017-01-17 16:03:22 +03:00
|
|
|
{
|
|
|
|
struct elevator_queue *e = q->elevator;
|
2021-05-11 03:05:35 +03:00
|
|
|
struct blk_mq_ctx *ctx;
|
|
|
|
struct blk_mq_hw_ctx *hctx;
|
2017-05-26 14:53:20 +03:00
|
|
|
bool ret = false;
|
2018-12-17 18:44:05 +03:00
|
|
|
enum hctx_type type;
|
2017-01-17 16:03:22 +03:00
|
|
|
|
2021-11-03 14:47:09 +03:00
|
|
|
if (e && e->type->ops.bio_merge) {
|
|
|
|
ret = e->type->ops.bio_merge(q, bio, nr_segs);
|
|
|
|
goto out_put;
|
|
|
|
}
|
2017-01-17 16:03:22 +03:00
|
|
|
|
2021-05-11 03:05:35 +03:00
|
|
|
ctx = blk_mq_get_ctx(q);
|
|
|
|
hctx = blk_mq_map_queue(q, bio->bi_opf, ctx);
|
2018-12-17 18:44:05 +03:00
|
|
|
type = hctx->type;
|
2020-08-28 05:52:57 +03:00
|
|
|
if (!(hctx->flags & BLK_MQ_F_SHOULD_MERGE) ||
|
|
|
|
list_empty_careful(&ctx->rq_lists[type]))
|
2021-11-03 14:47:09 +03:00
|
|
|
goto out_put;
|
2020-08-28 05:52:57 +03:00
|
|
|
|
|
|
|
/* default per sw-queue merge */
|
|
|
|
spin_lock(&ctx->lock);
|
|
|
|
/*
|
|
|
|
* Reverse check our software queue for entries that we could
|
|
|
|
* potentially merge with. Currently includes a hand-wavy stop
|
|
|
|
* count of 8, to not spend too much time checking for merges.
|
|
|
|
*/
|
2021-10-17 02:27:20 +03:00
|
|
|
if (blk_bio_list_merge(q, &ctx->rq_lists[type], bio, nr_segs))
|
2020-08-28 05:52:57 +03:00
|
|
|
ret = true;
|
2017-05-26 14:53:20 +03:00
|
|
|
|
2020-08-28 05:52:57 +03:00
|
|
|
spin_unlock(&ctx->lock);
|
2021-11-03 14:47:09 +03:00
|
|
|
out_put:
|
2017-05-26 14:53:20 +03:00
|
|
|
return ret;
|
2017-01-17 16:03:22 +03:00
|
|
|
}
|
|
|
|
|
2021-06-23 12:36:34 +03:00
|
|
|
bool blk_mq_sched_try_insert_merge(struct request_queue *q, struct request *rq,
|
|
|
|
struct list_head *free)
|
2017-01-17 16:03:22 +03:00
|
|
|
{
|
2021-06-23 12:36:34 +03:00
|
|
|
return rq_mergeable(rq) && elv_attempt_insert_merge(q, rq, free);
|
2017-01-17 16:03:22 +03:00
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(blk_mq_sched_try_insert_merge);
|
|
|
|
|
2017-02-03 02:42:39 +03:00
|
|
|
static bool blk_mq_sched_bypass_insert(struct blk_mq_hw_ctx *hctx,
|
|
|
|
struct request *rq)
|
2017-01-17 16:03:22 +03:00
|
|
|
{
|
2020-02-25 04:04:32 +03:00
|
|
|
/*
|
|
|
|
* dispatch flush and passthrough rq directly
|
|
|
|
*
|
|
|
|
* passthrough request has to be added to hctx->dispatch directly.
|
|
|
|
* For some reason, device may be in one situation which can't
|
|
|
|
* handle FS request, so STS_RESOURCE is always returned and the
|
|
|
|
* FS request will be added to hctx->dispatch. However passthrough
|
|
|
|
* request may be required at that time for fixing the problem. If
|
|
|
|
* passthrough request is added to scheduler queue, there isn't any
|
|
|
|
* chance to dispatch it given we prioritize requests in hctx->dispatch.
|
|
|
|
*/
|
|
|
|
if ((rq->rq_flags & RQF_FLUSH_SEQ) || blk_rq_is_passthrough(rq))
|
2017-11-02 18:24:36 +03:00
|
|
|
return true;
|
|
|
|
|
|
|
|
return false;
|
2017-01-17 16:03:22 +03:00
|
|
|
}
|
|
|
|
|
2017-01-27 11:00:47 +03:00
|
|
|
void blk_mq_sched_insert_request(struct request *rq, bool at_head,
|
2018-01-17 19:25:58 +03:00
|
|
|
bool run_queue, bool async)
|
2017-01-27 11:00:47 +03:00
|
|
|
{
|
|
|
|
struct request_queue *q = rq->q;
|
|
|
|
struct elevator_queue *e = q->elevator;
|
|
|
|
struct blk_mq_ctx *ctx = rq->mq_ctx;
|
2018-10-30 00:06:13 +03:00
|
|
|
struct blk_mq_hw_ctx *hctx = rq->mq_hctx;
|
2017-01-27 11:00:47 +03:00
|
|
|
|
2020-08-27 09:34:17 +03:00
|
|
|
WARN_ON(e && (rq->tag != BLK_MQ_NO_TAG));
|
2017-11-02 18:24:38 +03:00
|
|
|
|
2021-03-01 19:23:25 +03:00
|
|
|
if (blk_mq_sched_bypass_insert(hctx, rq)) {
|
2020-03-12 12:15:48 +03:00
|
|
|
/*
|
|
|
|
* Firstly normal IO request is inserted to scheduler queue or
|
|
|
|
* sw queue, meantime we add flush request to dispatch queue(
|
|
|
|
* hctx->dispatch) directly and there is at most one in-flight
|
|
|
|
* flush request for each hw queue, so it doesn't matter to add
|
|
|
|
* flush request to tail or front of the dispatch queue.
|
|
|
|
*
|
|
|
|
* Secondly in case of NCQ, flush request belongs to non-NCQ
|
|
|
|
* command, and queueing it will fail when there is any
|
|
|
|
* in-flight normal IO request(NCQ command). When adding flush
|
|
|
|
* rq to the front of hctx->dispatch, it is easier to introduce
|
|
|
|
* extra time to flush rq's latency because of S_SCHED_RESTART
|
|
|
|
* compared with adding to the tail of dispatch queue, then
|
|
|
|
* chance of flush merge is increased, and less flush requests
|
|
|
|
* will be issued to controller. It is observed that ~10% time
|
|
|
|
* is saved in blktests block/004 on disk attached to AHCI/NCQ
|
|
|
|
* drive when adding flush rq to the front of hctx->dispatch.
|
|
|
|
*
|
|
|
|
* Simply queue flush rq to the front of hctx->dispatch so that
|
|
|
|
* intensive flush workloads can benefit in case of NCQ HW.
|
|
|
|
*/
|
|
|
|
at_head = (rq->rq_flags & RQF_FLUSH_SEQ) ? true : at_head;
|
2020-02-25 04:04:32 +03:00
|
|
|
blk_mq_request_bypass_insert(rq, at_head, false);
|
2017-02-03 02:42:39 +03:00
|
|
|
goto run;
|
2020-02-25 04:04:32 +03:00
|
|
|
}
|
2017-02-03 02:42:39 +03:00
|
|
|
|
2021-06-18 04:59:22 +03:00
|
|
|
if (e) {
|
2017-01-27 11:00:47 +03:00
|
|
|
LIST_HEAD(list);
|
|
|
|
|
|
|
|
list_add(&rq->queuelist, &list);
|
2018-11-02 01:41:41 +03:00
|
|
|
e->type->ops.insert_requests(hctx, &list, at_head);
|
2017-01-27 11:00:47 +03:00
|
|
|
} else {
|
|
|
|
spin_lock(&ctx->lock);
|
|
|
|
__blk_mq_insert_request(hctx, rq, at_head);
|
|
|
|
spin_unlock(&ctx->lock);
|
|
|
|
}
|
|
|
|
|
2017-02-03 02:42:39 +03:00
|
|
|
run:
|
2017-01-27 11:00:47 +03:00
|
|
|
if (run_queue)
|
|
|
|
blk_mq_run_hw_queue(hctx, async);
|
|
|
|
}
|
|
|
|
|
2018-10-30 20:31:51 +03:00
|
|
|
void blk_mq_sched_insert_requests(struct blk_mq_hw_ctx *hctx,
|
2017-01-27 11:00:47 +03:00
|
|
|
struct blk_mq_ctx *ctx,
|
|
|
|
struct list_head *list, bool run_queue_async)
|
|
|
|
{
|
2018-10-29 22:11:38 +03:00
|
|
|
struct elevator_queue *e;
|
blk-mq: grab .q_usage_counter when queuing request from plug code path
Just like aio/io_uring, we need to grab 2 refcount for queuing one
request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed
in generic_make_request() serves for submission. In theroy, this
refcount should have been released after the sumission(async run queue)
is done. blk_freeze_queue() works with blk_sync_queue() together
for avoiding race between cleanup queue and IO submission, given async
run queue activities are canceled because hctx->run_work is scheduled with
the refcount held, so it is fine to not hold the refcount when
running the run queue work function for dispatch IO.
However, if request is staggered into plug list, and finally queued
from plug code path, the refcount in submission side is actually missed.
And we may start to run queue after queue is removed because the queue's
kobject refcount isn't guaranteed to be grabbed in flushing plug list
context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queue
blk_mq_run_hw_queue
Because of concurrent run queue, all requests inserted above may be
completed before calling the above blk_mq_run_hw_queue. Then queue can
be freed during the above blk_mq_run_hw_queue().
Fixes the issue by grab .q_usage_counter before calling
blk_mq_sched_insert_requests() in blk_mq_flush_plug_list(). This way is
safe because the queue is absolutely alive before inserting request.
Cc: Dongli Zhang <dongli.zhang@oracle.com>
Cc: James Smart <james.smart@broadcom.com>
Cc: linux-scsi@vger.kernel.org,
Cc: Martin K . Petersen <martin.petersen@oracle.com>,
Cc: Christoph Hellwig <hch@lst.de>,
Cc: James E . J . Bottomley <jejb@linux.vnet.ibm.com>,
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Tested-by: James Smart <james.smart@broadcom.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-30 04:52:23 +03:00
|
|
|
struct request_queue *q = hctx->queue;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* blk_mq_sched_insert_requests() is called from flush plug
|
|
|
|
* context only, and hold one usage counter to prevent queue
|
|
|
|
* from being released.
|
|
|
|
*/
|
|
|
|
percpu_ref_get(&q->q_usage_counter);
|
2017-01-27 11:00:47 +03:00
|
|
|
|
2018-10-29 22:11:38 +03:00
|
|
|
e = hctx->queue->elevator;
|
2021-06-18 04:59:22 +03:00
|
|
|
if (e) {
|
2018-11-02 01:41:41 +03:00
|
|
|
e->type->ops.insert_requests(hctx, list, false);
|
2021-06-18 04:59:22 +03:00
|
|
|
} else {
|
2018-07-10 04:03:31 +03:00
|
|
|
/*
|
|
|
|
* try to issue requests directly if the hw queue isn't
|
|
|
|
* busy in case of 'none' scheduler, and this way may save
|
|
|
|
* us one extra enqueue & dequeue to sw queue.
|
|
|
|
*/
|
2021-10-29 23:29:45 +03:00
|
|
|
if (!hctx->dispatch_busy && !run_queue_async) {
|
2021-12-03 16:15:34 +03:00
|
|
|
blk_mq_run_dispatch_ops(hctx->queue,
|
|
|
|
blk_mq_try_issue_list_directly(hctx, list));
|
2019-04-04 20:08:43 +03:00
|
|
|
if (list_empty(list))
|
blk-mq: grab .q_usage_counter when queuing request from plug code path
Just like aio/io_uring, we need to grab 2 refcount for queuing one
request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed
in generic_make_request() serves for submission. In theroy, this
refcount should have been released after the sumission(async run queue)
is done. blk_freeze_queue() works with blk_sync_queue() together
for avoiding race between cleanup queue and IO submission, given async
run queue activities are canceled because hctx->run_work is scheduled with
the refcount held, so it is fine to not hold the refcount when
running the run queue work function for dispatch IO.
However, if request is staggered into plug list, and finally queued
from plug code path, the refcount in submission side is actually missed.
And we may start to run queue after queue is removed because the queue's
kobject refcount isn't guaranteed to be grabbed in flushing plug list
context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queue
blk_mq_run_hw_queue
Because of concurrent run queue, all requests inserted above may be
completed before calling the above blk_mq_run_hw_queue. Then queue can
be freed during the above blk_mq_run_hw_queue().
Fixes the issue by grab .q_usage_counter before calling
blk_mq_sched_insert_requests() in blk_mq_flush_plug_list(). This way is
safe because the queue is absolutely alive before inserting request.
Cc: Dongli Zhang <dongli.zhang@oracle.com>
Cc: James Smart <james.smart@broadcom.com>
Cc: linux-scsi@vger.kernel.org,
Cc: Martin K . Petersen <martin.petersen@oracle.com>,
Cc: Christoph Hellwig <hch@lst.de>,
Cc: James E . J . Bottomley <jejb@linux.vnet.ibm.com>,
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Tested-by: James Smart <james.smart@broadcom.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-30 04:52:23 +03:00
|
|
|
goto out;
|
2019-04-04 20:08:43 +03:00
|
|
|
}
|
|
|
|
blk_mq_insert_requests(hctx, ctx, list);
|
2018-07-10 04:03:31 +03:00
|
|
|
}
|
2017-01-27 11:00:47 +03:00
|
|
|
|
|
|
|
blk_mq_run_hw_queue(hctx, run_queue_async);
|
blk-mq: grab .q_usage_counter when queuing request from plug code path
Just like aio/io_uring, we need to grab 2 refcount for queuing one
request, one is for submission, another is for completion.
If the request isn't queued from plug code path, the refcount grabbed
in generic_make_request() serves for submission. In theroy, this
refcount should have been released after the sumission(async run queue)
is done. blk_freeze_queue() works with blk_sync_queue() together
for avoiding race between cleanup queue and IO submission, given async
run queue activities are canceled because hctx->run_work is scheduled with
the refcount held, so it is fine to not hold the refcount when
running the run queue work function for dispatch IO.
However, if request is staggered into plug list, and finally queued
from plug code path, the refcount in submission side is actually missed.
And we may start to run queue after queue is removed because the queue's
kobject refcount isn't guaranteed to be grabbed in flushing plug list
context, then kernel oops is triggered, see the following race:
blk_mq_flush_plug_list():
blk_mq_sched_insert_requests()
insert requests to sw queue or scheduler queue
blk_mq_run_hw_queue
Because of concurrent run queue, all requests inserted above may be
completed before calling the above blk_mq_run_hw_queue. Then queue can
be freed during the above blk_mq_run_hw_queue().
Fixes the issue by grab .q_usage_counter before calling
blk_mq_sched_insert_requests() in blk_mq_flush_plug_list(). This way is
safe because the queue is absolutely alive before inserting request.
Cc: Dongli Zhang <dongli.zhang@oracle.com>
Cc: James Smart <james.smart@broadcom.com>
Cc: linux-scsi@vger.kernel.org,
Cc: Martin K . Petersen <martin.petersen@oracle.com>,
Cc: Christoph Hellwig <hch@lst.de>,
Cc: James E . J . Bottomley <jejb@linux.vnet.ibm.com>,
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
Tested-by: James Smart <james.smart@broadcom.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-04-30 04:52:23 +03:00
|
|
|
out:
|
|
|
|
percpu_ref_put(&q->q_usage_counter);
|
2017-01-27 11:00:47 +03:00
|
|
|
}
|
|
|
|
|
2021-10-05 13:23:30 +03:00
|
|
|
static int blk_mq_sched_alloc_map_and_rqs(struct request_queue *q,
|
|
|
|
struct blk_mq_hw_ctx *hctx,
|
|
|
|
unsigned int hctx_idx)
|
2017-04-05 22:01:30 +03:00
|
|
|
{
|
2021-10-05 13:23:39 +03:00
|
|
|
if (blk_mq_is_shared_tags(q->tag_set->flags)) {
|
|
|
|
hctx->sched_tags = q->sched_shared_tags;
|
2021-10-05 13:23:37 +03:00
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
2021-10-05 13:23:35 +03:00
|
|
|
hctx->sched_tags = blk_mq_alloc_map_and_rqs(q->tag_set, hctx_idx,
|
|
|
|
q->nr_requests);
|
2017-04-05 22:01:30 +03:00
|
|
|
|
|
|
|
if (!hctx->sched_tags)
|
|
|
|
return -ENOMEM;
|
2021-10-05 13:23:35 +03:00
|
|
|
return 0;
|
2017-04-05 22:01:30 +03:00
|
|
|
}
|
|
|
|
|
2021-10-05 13:23:39 +03:00
|
|
|
static void blk_mq_exit_sched_shared_tags(struct request_queue *queue)
|
2021-10-05 13:23:37 +03:00
|
|
|
{
|
2021-10-05 13:23:39 +03:00
|
|
|
blk_mq_free_rq_map(queue->sched_shared_tags);
|
|
|
|
queue->sched_shared_tags = NULL;
|
2021-10-05 13:23:37 +03:00
|
|
|
}
|
|
|
|
|
block: free sched's request pool in blk_cleanup_queue
In theory, IO scheduler belongs to request queue, and the request pool
of sched tags belongs to the request queue too.
However, the current tags allocation interfaces are re-used for both
driver tags and sched tags, and driver tags is definitely host wide,
and doesn't belong to any request queue, same with its request pool.
So we need tagset instance for freeing request of sched tags.
Meantime, blk_mq_free_tag_set() often follows blk_cleanup_queue() in case
of non-BLK_MQ_F_TAG_SHARED, this way requires that request pool of sched
tags to be freed before calling blk_mq_free_tag_set().
Commit 47cdee29ef9d94e ("block: move blk_exit_queue into __blk_release_queue")
moves blk_exit_queue into __blk_release_queue for simplying the fast
path in generic_make_request(), then causes oops during freeing requests
of sched tags in __blk_release_queue().
Fix the above issue by move freeing request pool of sched tags into
blk_cleanup_queue(), this way is safe becasue queue has been frozen and no any
in-queue requests at that time. Freeing sched tags has to be kept in queue's
release handler becasue there might be un-completed dispatch activity
which might refer to sched tags.
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Christoph Hellwig <hch@lst.de>
Fixes: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue into __blk_release_queue")
Tested-by: Yi Zhang <yi.zhang@redhat.com>
Reported-by: kernel test robot <rong.a.chen@intel.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-04 16:08:02 +03:00
|
|
|
/* called in queue's release handler, tagset has gone away */
|
2021-10-05 13:23:37 +03:00
|
|
|
static void blk_mq_sched_tags_teardown(struct request_queue *q, unsigned int flags)
|
2017-01-17 16:03:22 +03:00
|
|
|
{
|
|
|
|
struct blk_mq_hw_ctx *hctx;
|
2022-03-08 10:32:18 +03:00
|
|
|
unsigned long i;
|
2017-04-05 22:01:30 +03:00
|
|
|
|
block: free sched's request pool in blk_cleanup_queue
In theory, IO scheduler belongs to request queue, and the request pool
of sched tags belongs to the request queue too.
However, the current tags allocation interfaces are re-used for both
driver tags and sched tags, and driver tags is definitely host wide,
and doesn't belong to any request queue, same with its request pool.
So we need tagset instance for freeing request of sched tags.
Meantime, blk_mq_free_tag_set() often follows blk_cleanup_queue() in case
of non-BLK_MQ_F_TAG_SHARED, this way requires that request pool of sched
tags to be freed before calling blk_mq_free_tag_set().
Commit 47cdee29ef9d94e ("block: move blk_exit_queue into __blk_release_queue")
moves blk_exit_queue into __blk_release_queue for simplying the fast
path in generic_make_request(), then causes oops during freeing requests
of sched tags in __blk_release_queue().
Fix the above issue by move freeing request pool of sched tags into
blk_cleanup_queue(), this way is safe becasue queue has been frozen and no any
in-queue requests at that time. Freeing sched tags has to be kept in queue's
release handler becasue there might be un-completed dispatch activity
which might refer to sched tags.
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Christoph Hellwig <hch@lst.de>
Fixes: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue into __blk_release_queue")
Tested-by: Yi Zhang <yi.zhang@redhat.com>
Reported-by: kernel test robot <rong.a.chen@intel.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-04 16:08:02 +03:00
|
|
|
queue_for_each_hw_ctx(q, hctx, i) {
|
|
|
|
if (hctx->sched_tags) {
|
2021-10-22 11:12:20 +03:00
|
|
|
if (!blk_mq_is_shared_tags(flags))
|
2021-10-05 13:23:37 +03:00
|
|
|
blk_mq_free_rq_map(hctx->sched_tags);
|
block: free sched's request pool in blk_cleanup_queue
In theory, IO scheduler belongs to request queue, and the request pool
of sched tags belongs to the request queue too.
However, the current tags allocation interfaces are re-used for both
driver tags and sched tags, and driver tags is definitely host wide,
and doesn't belong to any request queue, same with its request pool.
So we need tagset instance for freeing request of sched tags.
Meantime, blk_mq_free_tag_set() often follows blk_cleanup_queue() in case
of non-BLK_MQ_F_TAG_SHARED, this way requires that request pool of sched
tags to be freed before calling blk_mq_free_tag_set().
Commit 47cdee29ef9d94e ("block: move blk_exit_queue into __blk_release_queue")
moves blk_exit_queue into __blk_release_queue for simplying the fast
path in generic_make_request(), then causes oops during freeing requests
of sched tags in __blk_release_queue().
Fix the above issue by move freeing request pool of sched tags into
blk_cleanup_queue(), this way is safe becasue queue has been frozen and no any
in-queue requests at that time. Freeing sched tags has to be kept in queue's
release handler becasue there might be un-completed dispatch activity
which might refer to sched tags.
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Christoph Hellwig <hch@lst.de>
Fixes: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue into __blk_release_queue")
Tested-by: Yi Zhang <yi.zhang@redhat.com>
Reported-by: kernel test robot <rong.a.chen@intel.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-04 16:08:02 +03:00
|
|
|
hctx->sched_tags = NULL;
|
|
|
|
}
|
|
|
|
}
|
2021-10-05 13:23:37 +03:00
|
|
|
|
2021-10-05 13:23:39 +03:00
|
|
|
if (blk_mq_is_shared_tags(flags))
|
|
|
|
blk_mq_exit_sched_shared_tags(q);
|
2017-04-05 22:01:30 +03:00
|
|
|
}
|
|
|
|
|
2021-10-05 13:23:39 +03:00
|
|
|
static int blk_mq_init_sched_shared_tags(struct request_queue *queue)
|
blk-mq: Use request queue-wide tags for tagset-wide sbitmap
The tags used for an IO scheduler are currently per hctx.
As such, when q->nr_hw_queues grows, so does the request queue total IO
scheduler tag depth.
This may cause problems for SCSI MQ HBAs whose total driver depth is
fixed.
Ming and Yanhui report higher CPU usage and lower throughput in scenarios
where the fixed total driver tag depth is appreciably lower than the total
scheduler tag depth:
https://lore.kernel.org/linux-block/440dfcfc-1a2c-bd98-1161-cec4d78c6dfc@huawei.com/T/#mc0d6d4f95275a2743d1c8c3e4dc9ff6c9aa3a76b
In that scenario, since the scheduler tag is got first, much contention
is introduced since a driver tag may not be available after we have got
the sched tag.
Improve this scenario by introducing request queue-wide tags for when
a tagset-wide sbitmap is used. The static sched requests are still
allocated per hctx, as requests are initialised per hctx, as in
blk_mq_init_request(..., hctx_idx, ...) ->
set->ops->init_request(.., hctx_idx, ...).
For simplicity of resizing the request queue sbitmap when updating the
request queue depth, just init at the max possible size, so we don't need
to deal with the possibly with swapping out a new sbitmap for old if
we need to grow.
Signed-off-by: John Garry <john.garry@huawei.com>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/1620907258-30910-3-git-send-email-john.garry@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-05-13 15:00:58 +03:00
|
|
|
{
|
|
|
|
struct blk_mq_tag_set *set = queue->tag_set;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Set initial depth at max so that we don't need to reallocate for
|
|
|
|
* updating nr_requests.
|
|
|
|
*/
|
2021-10-05 13:23:39 +03:00
|
|
|
queue->sched_shared_tags = blk_mq_alloc_map_and_rqs(set,
|
2021-10-05 13:23:37 +03:00
|
|
|
BLK_MQ_NO_HCTX_IDX,
|
|
|
|
MAX_SCHED_RQ);
|
2021-10-05 13:23:39 +03:00
|
|
|
if (!queue->sched_shared_tags)
|
2021-10-05 13:23:37 +03:00
|
|
|
return -ENOMEM;
|
blk-mq: Use request queue-wide tags for tagset-wide sbitmap
The tags used for an IO scheduler are currently per hctx.
As such, when q->nr_hw_queues grows, so does the request queue total IO
scheduler tag depth.
This may cause problems for SCSI MQ HBAs whose total driver depth is
fixed.
Ming and Yanhui report higher CPU usage and lower throughput in scenarios
where the fixed total driver tag depth is appreciably lower than the total
scheduler tag depth:
https://lore.kernel.org/linux-block/440dfcfc-1a2c-bd98-1161-cec4d78c6dfc@huawei.com/T/#mc0d6d4f95275a2743d1c8c3e4dc9ff6c9aa3a76b
In that scenario, since the scheduler tag is got first, much contention
is introduced since a driver tag may not be available after we have got
the sched tag.
Improve this scenario by introducing request queue-wide tags for when
a tagset-wide sbitmap is used. The static sched requests are still
allocated per hctx, as requests are initialised per hctx, as in
blk_mq_init_request(..., hctx_idx, ...) ->
set->ops->init_request(.., hctx_idx, ...).
For simplicity of resizing the request queue sbitmap when updating the
request queue depth, just init at the max possible size, so we don't need
to deal with the possibly with swapping out a new sbitmap for old if
we need to grow.
Signed-off-by: John Garry <john.garry@huawei.com>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/1620907258-30910-3-git-send-email-john.garry@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-05-13 15:00:58 +03:00
|
|
|
|
2021-10-05 13:23:39 +03:00
|
|
|
blk_mq_tag_update_sched_shared_tags(queue);
|
blk-mq: Use request queue-wide tags for tagset-wide sbitmap
The tags used for an IO scheduler are currently per hctx.
As such, when q->nr_hw_queues grows, so does the request queue total IO
scheduler tag depth.
This may cause problems for SCSI MQ HBAs whose total driver depth is
fixed.
Ming and Yanhui report higher CPU usage and lower throughput in scenarios
where the fixed total driver tag depth is appreciably lower than the total
scheduler tag depth:
https://lore.kernel.org/linux-block/440dfcfc-1a2c-bd98-1161-cec4d78c6dfc@huawei.com/T/#mc0d6d4f95275a2743d1c8c3e4dc9ff6c9aa3a76b
In that scenario, since the scheduler tag is got first, much contention
is introduced since a driver tag may not be available after we have got
the sched tag.
Improve this scenario by introducing request queue-wide tags for when
a tagset-wide sbitmap is used. The static sched requests are still
allocated per hctx, as requests are initialised per hctx, as in
blk_mq_init_request(..., hctx_idx, ...) ->
set->ops->init_request(.., hctx_idx, ...).
For simplicity of resizing the request queue sbitmap when updating the
request queue depth, just init at the max possible size, so we don't need
to deal with the possibly with swapping out a new sbitmap for old if
we need to grow.
Signed-off-by: John Garry <john.garry@huawei.com>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/1620907258-30910-3-git-send-email-john.garry@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-05-13 15:00:58 +03:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|
|
|
|
|
block: fix up elevator_type refcounting
The current reference management logic of io scheduler modules contains
refcnt problems. For example, blk_mq_init_sched may fail before or after
the calling of e->ops.init_sched. If it fails before the calling, it does
nothing to the reference to the io scheduler module. But if it fails after
the calling, it releases the reference by calling kobject_put(&eq->kobj).
As the callers of blk_mq_init_sched can't know exactly where the failure
happens, they can't handle the reference to the io scheduler module
properly: releasing the reference on failure results in double-release if
blk_mq_init_sched has released it, and not releasing the reference results
in ghost reference if blk_mq_init_sched did not release it either.
The same problem also exists in io schedulers' init_sched implementations.
We can address the problem by adding releasing statements to the error
handling procedures of blk_mq_init_sched and init_sched implementations.
But that is counterintuitive and requires modifications to existing io
schedulers.
Instead, We make elevator_alloc get the io scheduler module references
that will be released by elevator_release. And then, we match each
elevator_get with an elevator_put. Therefore, each reference to an io
scheduler module explicitly has its own getter and releaser, and we no
longer need to worry about the refcnt problems.
The bugs and the patch can be validated with tools here:
https://github.com/nickyc975/linux_elv_refcnt_bug.git
[hch: split out a few bits into separate patches, use a non-try
module_get in elevator_alloc]
Signed-off-by: Jinlong Chen <nickyc975@zju.edu.cn>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/r/20221020064819.1469928-5-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2022-10-20 09:48:19 +03:00
|
|
|
/* caller must have a reference to @e, will grab another one if successful */
|
2017-04-05 22:01:30 +03:00
|
|
|
int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e)
|
|
|
|
{
|
2022-03-08 10:32:18 +03:00
|
|
|
unsigned int flags = q->tag_set->flags;
|
2017-04-05 22:01:30 +03:00
|
|
|
struct blk_mq_hw_ctx *hctx;
|
2017-04-05 22:01:34 +03:00
|
|
|
struct elevator_queue *eq;
|
2022-03-08 10:32:18 +03:00
|
|
|
unsigned long i;
|
2017-04-05 22:01:30 +03:00
|
|
|
int ret;
|
|
|
|
|
2017-01-17 16:03:22 +03:00
|
|
|
/*
|
2017-07-03 15:37:14 +03:00
|
|
|
* Default to double of smaller one between hw queue_depth and 128,
|
|
|
|
* since we don't split into sync/async like the old code did.
|
|
|
|
* Additionally, this is a per-hw queue depth.
|
2017-01-17 16:03:22 +03:00
|
|
|
*/
|
2017-07-03 15:37:14 +03:00
|
|
|
q->nr_requests = 2 * min_t(unsigned int, q->tag_set->queue_depth,
|
2021-10-05 13:23:27 +03:00
|
|
|
BLKDEV_DEFAULT_RQ);
|
2017-01-17 16:03:22 +03:00
|
|
|
|
2021-10-05 13:23:39 +03:00
|
|
|
if (blk_mq_is_shared_tags(flags)) {
|
|
|
|
ret = blk_mq_init_sched_shared_tags(q);
|
2017-01-17 16:03:22 +03:00
|
|
|
if (ret)
|
2021-10-05 13:23:37 +03:00
|
|
|
return ret;
|
blk-mq: Use request queue-wide tags for tagset-wide sbitmap
The tags used for an IO scheduler are currently per hctx.
As such, when q->nr_hw_queues grows, so does the request queue total IO
scheduler tag depth.
This may cause problems for SCSI MQ HBAs whose total driver depth is
fixed.
Ming and Yanhui report higher CPU usage and lower throughput in scenarios
where the fixed total driver tag depth is appreciably lower than the total
scheduler tag depth:
https://lore.kernel.org/linux-block/440dfcfc-1a2c-bd98-1161-cec4d78c6dfc@huawei.com/T/#mc0d6d4f95275a2743d1c8c3e4dc9ff6c9aa3a76b
In that scenario, since the scheduler tag is got first, much contention
is introduced since a driver tag may not be available after we have got
the sched tag.
Improve this scenario by introducing request queue-wide tags for when
a tagset-wide sbitmap is used. The static sched requests are still
allocated per hctx, as requests are initialised per hctx, as in
blk_mq_init_request(..., hctx_idx, ...) ->
set->ops->init_request(.., hctx_idx, ...).
For simplicity of resizing the request queue sbitmap when updating the
request queue depth, just init at the max possible size, so we don't need
to deal with the possibly with swapping out a new sbitmap for old if
we need to grow.
Signed-off-by: John Garry <john.garry@huawei.com>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/1620907258-30910-3-git-send-email-john.garry@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-05-13 15:00:58 +03:00
|
|
|
}
|
|
|
|
|
2021-10-05 13:23:37 +03:00
|
|
|
queue_for_each_hw_ctx(q, hctx, i) {
|
|
|
|
ret = blk_mq_sched_alloc_map_and_rqs(q, hctx, i);
|
blk-mq: Use request queue-wide tags for tagset-wide sbitmap
The tags used for an IO scheduler are currently per hctx.
As such, when q->nr_hw_queues grows, so does the request queue total IO
scheduler tag depth.
This may cause problems for SCSI MQ HBAs whose total driver depth is
fixed.
Ming and Yanhui report higher CPU usage and lower throughput in scenarios
where the fixed total driver tag depth is appreciably lower than the total
scheduler tag depth:
https://lore.kernel.org/linux-block/440dfcfc-1a2c-bd98-1161-cec4d78c6dfc@huawei.com/T/#mc0d6d4f95275a2743d1c8c3e4dc9ff6c9aa3a76b
In that scenario, since the scheduler tag is got first, much contention
is introduced since a driver tag may not be available after we have got
the sched tag.
Improve this scenario by introducing request queue-wide tags for when
a tagset-wide sbitmap is used. The static sched requests are still
allocated per hctx, as requests are initialised per hctx, as in
blk_mq_init_request(..., hctx_idx, ...) ->
set->ops->init_request(.., hctx_idx, ...).
For simplicity of resizing the request queue sbitmap when updating the
request queue depth, just init at the max possible size, so we don't need
to deal with the possibly with swapping out a new sbitmap for old if
we need to grow.
Signed-off-by: John Garry <john.garry@huawei.com>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/1620907258-30910-3-git-send-email-john.garry@huawei.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2021-05-13 15:00:58 +03:00
|
|
|
if (ret)
|
2021-10-05 13:23:30 +03:00
|
|
|
goto err_free_map_and_rqs;
|
2017-01-17 16:03:22 +03:00
|
|
|
}
|
|
|
|
|
2018-11-02 01:41:41 +03:00
|
|
|
ret = e->ops.init_sched(q, e);
|
2017-04-05 22:01:30 +03:00
|
|
|
if (ret)
|
2021-10-05 13:23:37 +03:00
|
|
|
goto err_free_map_and_rqs;
|
2017-01-17 16:03:22 +03:00
|
|
|
|
2022-06-14 10:48:25 +03:00
|
|
|
mutex_lock(&q->debugfs_mutex);
|
2017-05-04 17:24:40 +03:00
|
|
|
blk_mq_debugfs_register_sched(q);
|
2022-06-14 10:48:25 +03:00
|
|
|
mutex_unlock(&q->debugfs_mutex);
|
2017-05-04 17:24:40 +03:00
|
|
|
|
|
|
|
queue_for_each_hw_ctx(q, hctx, i) {
|
2018-11-02 01:41:41 +03:00
|
|
|
if (e->ops.init_hctx) {
|
|
|
|
ret = e->ops.init_hctx(hctx, i);
|
2017-04-05 22:01:34 +03:00
|
|
|
if (ret) {
|
|
|
|
eq = q->elevator;
|
2021-10-05 13:23:31 +03:00
|
|
|
blk_mq_sched_free_rqs(q);
|
2017-04-05 22:01:34 +03:00
|
|
|
blk_mq_exit_sched(q, eq);
|
|
|
|
kobject_put(&eq->kobj);
|
|
|
|
return ret;
|
|
|
|
}
|
|
|
|
}
|
2022-06-14 10:48:25 +03:00
|
|
|
mutex_lock(&q->debugfs_mutex);
|
2017-05-04 17:24:40 +03:00
|
|
|
blk_mq_debugfs_register_sched_hctx(q, hctx);
|
2022-06-14 10:48:25 +03:00
|
|
|
mutex_unlock(&q->debugfs_mutex);
|
2017-04-05 22:01:34 +03:00
|
|
|
}
|
|
|
|
|
2017-01-17 16:03:22 +03:00
|
|
|
return 0;
|
|
|
|
|
2021-10-05 13:23:30 +03:00
|
|
|
err_free_map_and_rqs:
|
2021-10-05 13:23:31 +03:00
|
|
|
blk_mq_sched_free_rqs(q);
|
2021-10-05 13:23:37 +03:00
|
|
|
blk_mq_sched_tags_teardown(q, flags);
|
|
|
|
|
2017-04-07 17:52:27 +03:00
|
|
|
q->elevator = NULL;
|
2017-04-05 22:01:30 +03:00
|
|
|
return ret;
|
2017-01-17 16:03:22 +03:00
|
|
|
}
|
2017-01-14 00:43:58 +03:00
|
|
|
|
block: free sched's request pool in blk_cleanup_queue
In theory, IO scheduler belongs to request queue, and the request pool
of sched tags belongs to the request queue too.
However, the current tags allocation interfaces are re-used for both
driver tags and sched tags, and driver tags is definitely host wide,
and doesn't belong to any request queue, same with its request pool.
So we need tagset instance for freeing request of sched tags.
Meantime, blk_mq_free_tag_set() often follows blk_cleanup_queue() in case
of non-BLK_MQ_F_TAG_SHARED, this way requires that request pool of sched
tags to be freed before calling blk_mq_free_tag_set().
Commit 47cdee29ef9d94e ("block: move blk_exit_queue into __blk_release_queue")
moves blk_exit_queue into __blk_release_queue for simplying the fast
path in generic_make_request(), then causes oops during freeing requests
of sched tags in __blk_release_queue().
Fix the above issue by move freeing request pool of sched tags into
blk_cleanup_queue(), this way is safe becasue queue has been frozen and no any
in-queue requests at that time. Freeing sched tags has to be kept in queue's
release handler becasue there might be un-completed dispatch activity
which might refer to sched tags.
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Christoph Hellwig <hch@lst.de>
Fixes: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue into __blk_release_queue")
Tested-by: Yi Zhang <yi.zhang@redhat.com>
Reported-by: kernel test robot <rong.a.chen@intel.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-04 16:08:02 +03:00
|
|
|
/*
|
|
|
|
* called in either blk_queue_cleanup or elevator_switch, tagset
|
|
|
|
* is required for freeing requests
|
|
|
|
*/
|
2021-10-05 13:23:31 +03:00
|
|
|
void blk_mq_sched_free_rqs(struct request_queue *q)
|
block: free sched's request pool in blk_cleanup_queue
In theory, IO scheduler belongs to request queue, and the request pool
of sched tags belongs to the request queue too.
However, the current tags allocation interfaces are re-used for both
driver tags and sched tags, and driver tags is definitely host wide,
and doesn't belong to any request queue, same with its request pool.
So we need tagset instance for freeing request of sched tags.
Meantime, blk_mq_free_tag_set() often follows blk_cleanup_queue() in case
of non-BLK_MQ_F_TAG_SHARED, this way requires that request pool of sched
tags to be freed before calling blk_mq_free_tag_set().
Commit 47cdee29ef9d94e ("block: move blk_exit_queue into __blk_release_queue")
moves blk_exit_queue into __blk_release_queue for simplying the fast
path in generic_make_request(), then causes oops during freeing requests
of sched tags in __blk_release_queue().
Fix the above issue by move freeing request pool of sched tags into
blk_cleanup_queue(), this way is safe becasue queue has been frozen and no any
in-queue requests at that time. Freeing sched tags has to be kept in queue's
release handler becasue there might be un-completed dispatch activity
which might refer to sched tags.
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Christoph Hellwig <hch@lst.de>
Fixes: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue into __blk_release_queue")
Tested-by: Yi Zhang <yi.zhang@redhat.com>
Reported-by: kernel test robot <rong.a.chen@intel.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-04 16:08:02 +03:00
|
|
|
{
|
|
|
|
struct blk_mq_hw_ctx *hctx;
|
2022-03-08 10:32:18 +03:00
|
|
|
unsigned long i;
|
block: free sched's request pool in blk_cleanup_queue
In theory, IO scheduler belongs to request queue, and the request pool
of sched tags belongs to the request queue too.
However, the current tags allocation interfaces are re-used for both
driver tags and sched tags, and driver tags is definitely host wide,
and doesn't belong to any request queue, same with its request pool.
So we need tagset instance for freeing request of sched tags.
Meantime, blk_mq_free_tag_set() often follows blk_cleanup_queue() in case
of non-BLK_MQ_F_TAG_SHARED, this way requires that request pool of sched
tags to be freed before calling blk_mq_free_tag_set().
Commit 47cdee29ef9d94e ("block: move blk_exit_queue into __blk_release_queue")
moves blk_exit_queue into __blk_release_queue for simplying the fast
path in generic_make_request(), then causes oops during freeing requests
of sched tags in __blk_release_queue().
Fix the above issue by move freeing request pool of sched tags into
blk_cleanup_queue(), this way is safe becasue queue has been frozen and no any
in-queue requests at that time. Freeing sched tags has to be kept in queue's
release handler becasue there might be un-completed dispatch activity
which might refer to sched tags.
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Christoph Hellwig <hch@lst.de>
Fixes: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue into __blk_release_queue")
Tested-by: Yi Zhang <yi.zhang@redhat.com>
Reported-by: kernel test robot <rong.a.chen@intel.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-04 16:08:02 +03:00
|
|
|
|
2021-10-05 13:23:39 +03:00
|
|
|
if (blk_mq_is_shared_tags(q->tag_set->flags)) {
|
|
|
|
blk_mq_free_rqs(q->tag_set, q->sched_shared_tags,
|
2021-10-05 13:23:37 +03:00
|
|
|
BLK_MQ_NO_HCTX_IDX);
|
|
|
|
} else {
|
|
|
|
queue_for_each_hw_ctx(q, hctx, i) {
|
|
|
|
if (hctx->sched_tags)
|
|
|
|
blk_mq_free_rqs(q->tag_set,
|
|
|
|
hctx->sched_tags, i);
|
|
|
|
}
|
block: free sched's request pool in blk_cleanup_queue
In theory, IO scheduler belongs to request queue, and the request pool
of sched tags belongs to the request queue too.
However, the current tags allocation interfaces are re-used for both
driver tags and sched tags, and driver tags is definitely host wide,
and doesn't belong to any request queue, same with its request pool.
So we need tagset instance for freeing request of sched tags.
Meantime, blk_mq_free_tag_set() often follows blk_cleanup_queue() in case
of non-BLK_MQ_F_TAG_SHARED, this way requires that request pool of sched
tags to be freed before calling blk_mq_free_tag_set().
Commit 47cdee29ef9d94e ("block: move blk_exit_queue into __blk_release_queue")
moves blk_exit_queue into __blk_release_queue for simplying the fast
path in generic_make_request(), then causes oops during freeing requests
of sched tags in __blk_release_queue().
Fix the above issue by move freeing request pool of sched tags into
blk_cleanup_queue(), this way is safe becasue queue has been frozen and no any
in-queue requests at that time. Freeing sched tags has to be kept in queue's
release handler becasue there might be un-completed dispatch activity
which might refer to sched tags.
Cc: Bart Van Assche <bvanassche@acm.org>
Cc: Christoph Hellwig <hch@lst.de>
Fixes: 47cdee29ef9d94e485eb08f962c74943023a5271 ("block: move blk_exit_queue into __blk_release_queue")
Tested-by: Yi Zhang <yi.zhang@redhat.com>
Reported-by: kernel test robot <rong.a.chen@intel.com>
Signed-off-by: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
2019-06-04 16:08:02 +03:00
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2017-04-07 17:52:27 +03:00
|
|
|
void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e)
|
|
|
|
{
|
2017-04-05 22:01:34 +03:00
|
|
|
struct blk_mq_hw_ctx *hctx;
|
2022-03-08 10:32:18 +03:00
|
|
|
unsigned long i;
|
2021-06-09 09:30:46 +03:00
|
|
|
unsigned int flags = 0;
|
2017-04-05 22:01:34 +03:00
|
|
|
|
2017-05-04 17:24:40 +03:00
|
|
|
queue_for_each_hw_ctx(q, hctx, i) {
|
2022-06-14 10:48:25 +03:00
|
|
|
mutex_lock(&q->debugfs_mutex);
|
2017-05-04 17:24:40 +03:00
|
|
|
blk_mq_debugfs_unregister_sched_hctx(hctx);
|
2022-06-14 10:48:25 +03:00
|
|
|
mutex_unlock(&q->debugfs_mutex);
|
|
|
|
|
2018-11-02 01:41:41 +03:00
|
|
|
if (e->type->ops.exit_hctx && hctx->sched_data) {
|
|
|
|
e->type->ops.exit_hctx(hctx, i);
|
2017-05-04 17:24:40 +03:00
|
|
|
hctx->sched_data = NULL;
|
2017-04-05 22:01:34 +03:00
|
|
|
}
|
2021-06-09 09:30:46 +03:00
|
|
|
flags = hctx->flags;
|
2017-04-05 22:01:34 +03:00
|
|
|
}
|
2022-06-14 10:48:25 +03:00
|
|
|
|
|
|
|
mutex_lock(&q->debugfs_mutex);
|
2017-05-04 17:24:40 +03:00
|
|
|
blk_mq_debugfs_unregister_sched(q);
|
2022-06-14 10:48:25 +03:00
|
|
|
mutex_unlock(&q->debugfs_mutex);
|
|
|
|
|
2018-11-02 01:41:41 +03:00
|
|
|
if (e->type->ops.exit_sched)
|
|
|
|
e->type->ops.exit_sched(e);
|
2021-10-05 13:23:37 +03:00
|
|
|
blk_mq_sched_tags_teardown(q, flags);
|
2017-04-07 17:52:27 +03:00
|
|
|
q->elevator = NULL;
|
|
|
|
}
|