WSL2-Linux-Kernel/drivers/target/target_core_user.c

3306 строки
83 KiB
C
Исходник Обычный вид История

// SPDX-License-Identifier: GPL-2.0-only
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
/*
* Copyright (C) 2013 Shaohua Li <shli@kernel.org>
* Copyright (C) 2014 Red Hat, Inc.
* Copyright (C) 2015 Arrikto, Inc.
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
* Copyright (C) 2017 Chinamobile, Inc.
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
*/
#include <linux/spinlock.h>
#include <linux/module.h>
#include <linux/kernel.h>
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
#include <linux/timer.h>
#include <linux/parser.h>
#include <linux/vmalloc.h>
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
#include <linux/uio_driver.h>
#include <linux/xarray.h>
#include <linux/stringify.h>
#include <linux/bitops.h>
#include <linux/highmem.h>
#include <linux/configfs.h>
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
#include <linux/mutex.h>
#include <linux/workqueue.h>
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
#include <net/genetlink.h>
#include <scsi/scsi_common.h>
#include <scsi/scsi_proto.h>
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
#include <target/target_core_base.h>
#include <target/target_core_fabric.h>
#include <target/target_core_backend.h>
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
#include <linux/target_core_user.h>
/**
* DOC: Userspace I/O
* Userspace I/O
* -------------
*
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
* Define a shared-memory interface for LIO to pass SCSI commands and
* data to userspace for processing. This is to allow backends that
* are too complex for in-kernel support to be possible.
*
* It uses the UIO framework to do a lot of the device-creation and
* introspection work for us.
*
* See the .h file for how the ring is laid out. Note that while the
* command ring is defined, the particulars of the data area are
* not. Offset values in the command entry point to other locations
* internal to the mmap-ed area. There is separate space outside the
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
* command ring for data buffers. This leaves maximum flexibility for
* moving buffer allocations, or even page flipping or other
* allocation techniques, without altering the command ring layout.
*
* SECURITY:
* The user process must be assumed to be malicious. There's no way to
* prevent it breaking the command ring protocol if it wants, but in
* order to prevent other issues we must only ever read *data* from
* the shared memory area, not offsets or sizes. This applies to
* command ring entries as well as the mailbox. Extra code needed for
* this may have a 'UAM' comment.
*/
#define TCMU_TIME_OUT (30 * MSEC_PER_SEC)
/* For mailbox plus cmd ring, the size is fixed 8MB */
#define MB_CMDR_SIZE (8 * 1024 * 1024)
/* Offset of cmd ring is size of mailbox */
#define CMDR_OFF sizeof(struct tcmu_mailbox)
#define CMDR_SIZE (MB_CMDR_SIZE - CMDR_OFF)
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
/*
* For data area, the default block size is PAGE_SIZE and
* the default total size is 256K * PAGE_SIZE.
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
*/
#define DATA_PAGES_PER_BLK_DEF 1
#define DATA_AREA_PAGES_DEF (256 * 1024)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
#define TCMU_MBS_TO_PAGES(_mbs) ((size_t)_mbs << (20 - PAGE_SHIFT))
#define TCMU_PAGES_TO_MBS(_pages) (_pages >> (20 - PAGE_SHIFT))
/*
* Default number of global data blocks(512K * PAGE_SIZE)
* when the unmap thread will be started.
*/
#define TCMU_GLOBAL_MAX_PAGES_DEF (512 * 1024)
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
static u8 tcmu_kern_cmd_reply_supported;
static u8 tcmu_netlink_blocked;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
static struct device *tcmu_root_device;
struct tcmu_hba {
u32 host_id;
};
#define TCMU_CONFIG_LEN 256
static DEFINE_MUTEX(tcmu_nl_cmd_mutex);
static LIST_HEAD(tcmu_nl_cmd_list);
struct tcmu_dev;
struct tcmu_nl_cmd {
/* wake up thread waiting for reply */
struct completion complete;
struct list_head nl_list;
struct tcmu_dev *udev;
int cmd;
int status;
};
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
struct tcmu_dev {
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
struct list_head node;
struct kref kref;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
struct se_device se_dev;
struct se_dev_plug se_plug;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
char *name;
struct se_hba *hba;
#define TCMU_DEV_BIT_OPEN 0
#define TCMU_DEV_BIT_BROKEN 1
#define TCMU_DEV_BIT_BLOCKED 2
#define TCMU_DEV_BIT_TMR_NOTIFY 3
#define TCMU_DEV_BIT_PLUGGED 4
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
unsigned long flags;
struct uio_info uio_info;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
struct inode *inode;
uint64_t dev_size;
struct tcmu_mailbox *mb_addr;
void *cmdr;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
u32 cmdr_size;
u32 cmdr_last_cleaned;
/* Offset of data area from start of mb */
/* Must add data_off and mb_addr to get the address */
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
size_t data_off;
int data_area_mb;
uint32_t max_blocks;
size_t mmap_pages;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
struct mutex cmdr_lock;
struct list_head qfull_queue;
2020-07-26 18:35:09 +03:00
struct list_head tmr_queue;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
uint32_t dbi_max;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
uint32_t dbi_thresh;
unsigned long *data_bitmap;
struct xarray data_pages;
uint32_t data_pages_per_blk;
uint32_t data_blk_size;
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
struct xarray commands;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
struct timer_list cmd_timer;
unsigned int cmd_time_out;
struct list_head inflight_queue;
struct timer_list qfull_timer;
int qfull_time_out;
struct list_head timedout_entry;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
struct tcmu_nl_cmd curr_nl_cmd;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
char dev_config[TCMU_CONFIG_LEN];
int nl_reply_supported;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
};
#define TCMU_DEV(_se_dev) container_of(_se_dev, struct tcmu_dev, se_dev)
struct tcmu_cmd {
struct se_cmd *se_cmd;
struct tcmu_dev *tcmu_dev;
struct list_head queue_entry;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
uint16_t cmd_id;
/* Can't use se_cmd when cleaning up expired cmds, because if
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
cmd has been completed then accessing se_cmd is off limits */
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
uint32_t dbi_cnt;
uint32_t dbi_bidi_cnt;
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
uint32_t dbi_cur;
uint32_t *dbi;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
uint32_t data_len_bidi;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
unsigned long deadline;
#define TCMU_CMD_BIT_EXPIRED 0
#define TCMU_CMD_BIT_KEEP_BUF 1
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
unsigned long flags;
};
2020-07-26 18:35:09 +03:00
struct tcmu_tmr {
struct list_head queue_entry;
uint8_t tmr_type;
uint32_t tmr_cmd_cnt;
int16_t tmr_cmd_ids[];
2020-07-26 18:35:09 +03:00
};
/*
* To avoid dead lock the mutex lock order should always be:
*
* mutex_lock(&root_udev_mutex);
* ...
* mutex_lock(&tcmu_dev->cmdr_lock);
* mutex_unlock(&tcmu_dev->cmdr_lock);
* ...
* mutex_unlock(&root_udev_mutex);
*/
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
static DEFINE_MUTEX(root_udev_mutex);
static LIST_HEAD(root_udev);
static DEFINE_SPINLOCK(timed_out_udevs_lock);
static LIST_HEAD(timed_out_udevs);
static struct kmem_cache *tcmu_cmd_cache;
static atomic_t global_page_count = ATOMIC_INIT(0);
static struct delayed_work tcmu_unmap_work;
static int tcmu_global_max_pages = TCMU_GLOBAL_MAX_PAGES_DEF;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
static int tcmu_set_global_max_data_area(const char *str,
const struct kernel_param *kp)
{
int ret, max_area_mb;
ret = kstrtoint(str, 10, &max_area_mb);
if (ret)
return -EINVAL;
if (max_area_mb <= 0) {
pr_err("global_max_data_area must be larger than 0.\n");
return -EINVAL;
}
tcmu_global_max_pages = TCMU_MBS_TO_PAGES(max_area_mb);
if (atomic_read(&global_page_count) > tcmu_global_max_pages)
schedule_delayed_work(&tcmu_unmap_work, 0);
else
cancel_delayed_work_sync(&tcmu_unmap_work);
return 0;
}
static int tcmu_get_global_max_data_area(char *buffer,
const struct kernel_param *kp)
{
return sprintf(buffer, "%d\n", TCMU_PAGES_TO_MBS(tcmu_global_max_pages));
}
static const struct kernel_param_ops tcmu_global_max_data_area_op = {
.set = tcmu_set_global_max_data_area,
.get = tcmu_get_global_max_data_area,
};
module_param_cb(global_max_data_area_mb, &tcmu_global_max_data_area_op, NULL,
S_IWUSR | S_IRUGO);
MODULE_PARM_DESC(global_max_data_area_mb,
"Max MBs allowed to be allocated to all the tcmu device's "
"data areas.");
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
static int tcmu_get_block_netlink(char *buffer,
const struct kernel_param *kp)
{
return sprintf(buffer, "%s\n", tcmu_netlink_blocked ?
"blocked" : "unblocked");
}
static int tcmu_set_block_netlink(const char *str,
const struct kernel_param *kp)
{
int ret;
u8 val;
ret = kstrtou8(str, 0, &val);
if (ret < 0)
return ret;
if (val > 1) {
pr_err("Invalid block netlink value %u\n", val);
return -EINVAL;
}
tcmu_netlink_blocked = val;
return 0;
}
static const struct kernel_param_ops tcmu_block_netlink_op = {
.set = tcmu_set_block_netlink,
.get = tcmu_get_block_netlink,
};
module_param_cb(block_netlink, &tcmu_block_netlink_op, NULL, S_IWUSR | S_IRUGO);
MODULE_PARM_DESC(block_netlink, "Block new netlink commands.");
static int tcmu_fail_netlink_cmd(struct tcmu_nl_cmd *nl_cmd)
{
struct tcmu_dev *udev = nl_cmd->udev;
if (!tcmu_netlink_blocked) {
pr_err("Could not reset device's netlink interface. Netlink is not blocked.\n");
return -EBUSY;
}
if (nl_cmd->cmd != TCMU_CMD_UNSPEC) {
pr_debug("Aborting nl cmd %d on %s\n", nl_cmd->cmd, udev->name);
nl_cmd->status = -EINTR;
list_del(&nl_cmd->nl_list);
complete(&nl_cmd->complete);
}
return 0;
}
static int tcmu_set_reset_netlink(const char *str,
const struct kernel_param *kp)
{
struct tcmu_nl_cmd *nl_cmd, *tmp_cmd;
int ret;
u8 val;
ret = kstrtou8(str, 0, &val);
if (ret < 0)
return ret;
if (val != 1) {
pr_err("Invalid reset netlink value %u\n", val);
return -EINVAL;
}
mutex_lock(&tcmu_nl_cmd_mutex);
list_for_each_entry_safe(nl_cmd, tmp_cmd, &tcmu_nl_cmd_list, nl_list) {
ret = tcmu_fail_netlink_cmd(nl_cmd);
if (ret)
break;
}
mutex_unlock(&tcmu_nl_cmd_mutex);
return ret;
}
static const struct kernel_param_ops tcmu_reset_netlink_op = {
.set = tcmu_set_reset_netlink,
};
module_param_cb(reset_netlink, &tcmu_reset_netlink_op, NULL, S_IWUSR);
MODULE_PARM_DESC(reset_netlink, "Reset netlink commands.");
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
/* multicast group */
enum tcmu_multicast_groups {
TCMU_MCGRP_CONFIG,
};
static const struct genl_multicast_group tcmu_mcgrps[] = {
[TCMU_MCGRP_CONFIG] = { .name = "config", },
};
static struct nla_policy tcmu_attr_policy[TCMU_ATTR_MAX+1] = {
[TCMU_ATTR_DEVICE] = { .type = NLA_STRING },
[TCMU_ATTR_MINOR] = { .type = NLA_U32 },
[TCMU_ATTR_CMD_STATUS] = { .type = NLA_S32 },
[TCMU_ATTR_DEVICE_ID] = { .type = NLA_U32 },
[TCMU_ATTR_SUPP_KERN_CMD_REPLY] = { .type = NLA_U8 },
};
static int tcmu_genl_cmd_done(struct genl_info *info, int completed_cmd)
{
struct tcmu_dev *udev = NULL;
struct tcmu_nl_cmd *nl_cmd;
int dev_id, rc, ret = 0;
if (!info->attrs[TCMU_ATTR_CMD_STATUS] ||
!info->attrs[TCMU_ATTR_DEVICE_ID]) {
printk(KERN_ERR "TCMU_ATTR_CMD_STATUS or TCMU_ATTR_DEVICE_ID not set, doing nothing\n");
return -EINVAL;
}
dev_id = nla_get_u32(info->attrs[TCMU_ATTR_DEVICE_ID]);
rc = nla_get_s32(info->attrs[TCMU_ATTR_CMD_STATUS]);
mutex_lock(&tcmu_nl_cmd_mutex);
list_for_each_entry(nl_cmd, &tcmu_nl_cmd_list, nl_list) {
if (nl_cmd->udev->se_dev.dev_index == dev_id) {
udev = nl_cmd->udev;
break;
}
}
if (!udev) {
pr_err("tcmu nl cmd %u/%d completion could not find device with dev id %u.\n",
completed_cmd, rc, dev_id);
ret = -ENODEV;
goto unlock;
}
list_del(&nl_cmd->nl_list);
pr_debug("%s genl cmd done got id %d curr %d done %d rc %d stat %d\n",
udev->name, dev_id, nl_cmd->cmd, completed_cmd, rc,
nl_cmd->status);
if (nl_cmd->cmd != completed_cmd) {
pr_err("Mismatched commands on %s (Expecting reply for %d. Current %d).\n",
udev->name, completed_cmd, nl_cmd->cmd);
ret = -EINVAL;
goto unlock;
}
nl_cmd->status = rc;
complete(&nl_cmd->complete);
unlock:
mutex_unlock(&tcmu_nl_cmd_mutex);
return ret;
}
static int tcmu_genl_rm_dev_done(struct sk_buff *skb, struct genl_info *info)
{
return tcmu_genl_cmd_done(info, TCMU_CMD_REMOVED_DEVICE);
}
static int tcmu_genl_add_dev_done(struct sk_buff *skb, struct genl_info *info)
{
return tcmu_genl_cmd_done(info, TCMU_CMD_ADDED_DEVICE);
}
static int tcmu_genl_reconfig_dev_done(struct sk_buff *skb,
struct genl_info *info)
{
return tcmu_genl_cmd_done(info, TCMU_CMD_RECONFIG_DEVICE);
}
static int tcmu_genl_set_features(struct sk_buff *skb, struct genl_info *info)
{
if (info->attrs[TCMU_ATTR_SUPP_KERN_CMD_REPLY]) {
tcmu_kern_cmd_reply_supported =
nla_get_u8(info->attrs[TCMU_ATTR_SUPP_KERN_CMD_REPLY]);
printk(KERN_INFO "tcmu daemon: command reply support %u.\n",
tcmu_kern_cmd_reply_supported);
}
return 0;
}
static const struct genl_small_ops tcmu_genl_ops[] = {
{
.cmd = TCMU_CMD_SET_FEATURES,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.flags = GENL_ADMIN_PERM,
.doit = tcmu_genl_set_features,
},
{
.cmd = TCMU_CMD_ADDED_DEVICE_DONE,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.flags = GENL_ADMIN_PERM,
.doit = tcmu_genl_add_dev_done,
},
{
.cmd = TCMU_CMD_REMOVED_DEVICE_DONE,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.flags = GENL_ADMIN_PERM,
.doit = tcmu_genl_rm_dev_done,
},
{
.cmd = TCMU_CMD_RECONFIG_DEVICE_DONE,
.validate = GENL_DONT_VALIDATE_STRICT | GENL_DONT_VALIDATE_DUMP,
.flags = GENL_ADMIN_PERM,
.doit = tcmu_genl_reconfig_dev_done,
},
};
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
/* Our generic netlink family */
static struct genl_family tcmu_genl_family __ro_after_init = {
.module = THIS_MODULE,
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
.hdrsize = 0,
.name = "TCM-USER",
.version = 2,
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
.maxattr = TCMU_ATTR_MAX,
.policy = tcmu_attr_policy,
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
.mcgrps = tcmu_mcgrps,
.n_mcgrps = ARRAY_SIZE(tcmu_mcgrps),
.netnsok = true,
.small_ops = tcmu_genl_ops,
.n_small_ops = ARRAY_SIZE(tcmu_genl_ops),
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
};
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
#define tcmu_cmd_set_dbi_cur(cmd, index) ((cmd)->dbi_cur = (index))
#define tcmu_cmd_reset_dbi_cur(cmd) tcmu_cmd_set_dbi_cur(cmd, 0)
#define tcmu_cmd_set_dbi(cmd, index) ((cmd)->dbi[(cmd)->dbi_cur++] = (index))
#define tcmu_cmd_get_dbi(cmd) ((cmd)->dbi[(cmd)->dbi_cur++])
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
static void tcmu_cmd_free_data(struct tcmu_cmd *tcmu_cmd, uint32_t len)
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
{
struct tcmu_dev *udev = tcmu_cmd->tcmu_dev;
uint32_t i;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
for (i = 0; i < len; i++)
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
clear_bit(tcmu_cmd->dbi[i], udev->data_bitmap);
}
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
static inline int tcmu_get_empty_block(struct tcmu_dev *udev,
struct tcmu_cmd *tcmu_cmd,
int prev_dbi, int length, int *iov_cnt)
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
{
XA_STATE(xas, &udev->data_pages, 0);
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
struct page *page;
int i, cnt, dbi, dpi;
int page_cnt = DIV_ROUND_UP(length, PAGE_SIZE);
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
dbi = find_first_zero_bit(udev->data_bitmap, udev->dbi_thresh);
if (dbi == udev->dbi_thresh)
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
return -1;
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
dpi = dbi * udev->data_pages_per_blk;
/* Count the number of already allocated pages */
xas_set(&xas, dpi);
scsi: target: tcmu: Fix xarray RCU warning Commit f5ce815f34bc ("scsi: target: tcmu: Support DATA_BLOCK_SIZE = N * PAGE_SIZE") introduced xas_next() calls to iterate xarray elements. These calls triggered the WARNING "suspicious RCU usage" at tcmu device set up [1]. In the call stack of xas_next(), xas_load() was called. According to its comment, this function requires "the xa_lock or the RCU lock". To avoid the warning: - Guard the small loop calling xas_next() in tcmu_get_empty_block with RCU lock. - In the large loop in tcmu_copy_data using RCU lock would possibly disable preemtion for a long time (copy multi MBs). Therefore replace XA_STATE, xas_set and xas_next with a single xa_load. [1] [ 1899.867091] ============================= [ 1899.871199] WARNING: suspicious RCU usage [ 1899.875310] 5.13.0-rc1+ #41 Not tainted [ 1899.879222] ----------------------------- [ 1899.883299] include/linux/xarray.h:1182 suspicious rcu_dereference_check() usage! [ 1899.890940] other info that might help us debug this: [ 1899.899082] rcu_scheduler_active = 2, debug_locks = 1 [ 1899.905719] 3 locks held by kworker/0:1/1368: [ 1899.910161] #0: ffffa1f8c8b98738 ((wq_completion)target_submission){+.+.}-{0:0}, at: process_one_work+0x1ee/0x580 [ 1899.920732] #1: ffffbd7040cd7e78 ((work_completion)(&q->sq.work)){+.+.}-{0:0}, at: process_one_work+0x1ee/0x580 [ 1899.931146] #2: ffffa1f8d1c99768 (&udev->cmdr_lock){+.+.}-{3:3}, at: tcmu_queue_cmd+0xea/0x160 [target_core_user] [ 1899.941678] stack backtrace: [ 1899.946093] CPU: 0 PID: 1368 Comm: kworker/0:1 Not tainted 5.13.0-rc1+ #41 [ 1899.953070] Hardware name: System manufacturer System Product Name/PRIME Z270-A, BIOS 1302 03/15/2018 [ 1899.962459] Workqueue: target_submission target_queued_submit_work [target_core_mod] [ 1899.970337] Call Trace: [ 1899.972839] dump_stack+0x6d/0x89 [ 1899.976222] xas_descend+0x10e/0x120 [ 1899.979875] xas_load+0x39/0x50 [ 1899.983077] tcmu_get_empty_blocks+0x115/0x1c0 [target_core_user] [ 1899.989318] queue_cmd_ring+0x1da/0x630 [target_core_user] [ 1899.994897] ? rcu_read_lock_sched_held+0x3f/0x70 [ 1899.999695] ? trace_kmalloc+0xa6/0xd0 [ 1900.003501] ? __kmalloc+0x205/0x380 [ 1900.007167] tcmu_queue_cmd+0x12f/0x160 [target_core_user] [ 1900.012746] __target_execute_cmd+0x23/0xa0 [target_core_mod] [ 1900.018589] transport_generic_new_cmd+0x1f3/0x370 [target_core_mod] [ 1900.025046] transport_handle_cdb_direct+0x34/0x50 [target_core_mod] [ 1900.031517] target_queued_submit_work+0x43/0xe0 [target_core_mod] [ 1900.037837] process_one_work+0x268/0x580 [ 1900.041952] ? process_one_work+0x580/0x580 [ 1900.046195] worker_thread+0x55/0x3b0 [ 1900.049921] ? process_one_work+0x580/0x580 [ 1900.054192] kthread+0x143/0x160 [ 1900.057499] ? kthread_create_worker_on_cpu+0x40/0x40 [ 1900.062661] ret_from_fork+0x1f/0x30 Link: https://lore.kernel.org/r/20210519135440.26773-1-bostroesser@gmail.com Fixes: f5ce815f34bc ("scsi: target: tcmu: Support DATA_BLOCK_SIZE = N * PAGE_SIZE") Reported-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Tested-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-05-19 16:54:40 +03:00
rcu_read_lock();
for (cnt = 0; xas_next(&xas) && cnt < page_cnt;)
cnt++;
scsi: target: tcmu: Fix xarray RCU warning Commit f5ce815f34bc ("scsi: target: tcmu: Support DATA_BLOCK_SIZE = N * PAGE_SIZE") introduced xas_next() calls to iterate xarray elements. These calls triggered the WARNING "suspicious RCU usage" at tcmu device set up [1]. In the call stack of xas_next(), xas_load() was called. According to its comment, this function requires "the xa_lock or the RCU lock". To avoid the warning: - Guard the small loop calling xas_next() in tcmu_get_empty_block with RCU lock. - In the large loop in tcmu_copy_data using RCU lock would possibly disable preemtion for a long time (copy multi MBs). Therefore replace XA_STATE, xas_set and xas_next with a single xa_load. [1] [ 1899.867091] ============================= [ 1899.871199] WARNING: suspicious RCU usage [ 1899.875310] 5.13.0-rc1+ #41 Not tainted [ 1899.879222] ----------------------------- [ 1899.883299] include/linux/xarray.h:1182 suspicious rcu_dereference_check() usage! [ 1899.890940] other info that might help us debug this: [ 1899.899082] rcu_scheduler_active = 2, debug_locks = 1 [ 1899.905719] 3 locks held by kworker/0:1/1368: [ 1899.910161] #0: ffffa1f8c8b98738 ((wq_completion)target_submission){+.+.}-{0:0}, at: process_one_work+0x1ee/0x580 [ 1899.920732] #1: ffffbd7040cd7e78 ((work_completion)(&q->sq.work)){+.+.}-{0:0}, at: process_one_work+0x1ee/0x580 [ 1899.931146] #2: ffffa1f8d1c99768 (&udev->cmdr_lock){+.+.}-{3:3}, at: tcmu_queue_cmd+0xea/0x160 [target_core_user] [ 1899.941678] stack backtrace: [ 1899.946093] CPU: 0 PID: 1368 Comm: kworker/0:1 Not tainted 5.13.0-rc1+ #41 [ 1899.953070] Hardware name: System manufacturer System Product Name/PRIME Z270-A, BIOS 1302 03/15/2018 [ 1899.962459] Workqueue: target_submission target_queued_submit_work [target_core_mod] [ 1899.970337] Call Trace: [ 1899.972839] dump_stack+0x6d/0x89 [ 1899.976222] xas_descend+0x10e/0x120 [ 1899.979875] xas_load+0x39/0x50 [ 1899.983077] tcmu_get_empty_blocks+0x115/0x1c0 [target_core_user] [ 1899.989318] queue_cmd_ring+0x1da/0x630 [target_core_user] [ 1899.994897] ? rcu_read_lock_sched_held+0x3f/0x70 [ 1899.999695] ? trace_kmalloc+0xa6/0xd0 [ 1900.003501] ? __kmalloc+0x205/0x380 [ 1900.007167] tcmu_queue_cmd+0x12f/0x160 [target_core_user] [ 1900.012746] __target_execute_cmd+0x23/0xa0 [target_core_mod] [ 1900.018589] transport_generic_new_cmd+0x1f3/0x370 [target_core_mod] [ 1900.025046] transport_handle_cdb_direct+0x34/0x50 [target_core_mod] [ 1900.031517] target_queued_submit_work+0x43/0xe0 [target_core_mod] [ 1900.037837] process_one_work+0x268/0x580 [ 1900.041952] ? process_one_work+0x580/0x580 [ 1900.046195] worker_thread+0x55/0x3b0 [ 1900.049921] ? process_one_work+0x580/0x580 [ 1900.054192] kthread+0x143/0x160 [ 1900.057499] ? kthread_create_worker_on_cpu+0x40/0x40 [ 1900.062661] ret_from_fork+0x1f/0x30 Link: https://lore.kernel.org/r/20210519135440.26773-1-bostroesser@gmail.com Fixes: f5ce815f34bc ("scsi: target: tcmu: Support DATA_BLOCK_SIZE = N * PAGE_SIZE") Reported-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Tested-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-05-19 16:54:40 +03:00
rcu_read_unlock();
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
for (i = cnt; i < page_cnt; i++) {
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
/* try to get new page from the mm */
page = alloc_page(GFP_NOIO);
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
if (!page)
break;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
if (xa_store(&udev->data_pages, dpi + i, page, GFP_NOIO)) {
__free_page(page);
break;
}
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
}
if (atomic_add_return(i - cnt, &global_page_count) >
tcmu_global_max_pages)
schedule_delayed_work(&tcmu_unmap_work, 0);
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
if (i && dbi > udev->dbi_max)
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
udev->dbi_max = dbi;
set_bit(dbi, udev->data_bitmap);
tcmu_cmd_set_dbi(tcmu_cmd, dbi);
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
if (dbi != prev_dbi + 1)
*iov_cnt += 1;
return i == page_cnt ? dbi : -1;
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
}
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
static int tcmu_get_empty_blocks(struct tcmu_dev *udev,
struct tcmu_cmd *tcmu_cmd, int length)
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
{
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
/* start value of dbi + 1 must not be a valid dbi */
int dbi = -2;
int blk_data_len, iov_cnt = 0;
uint32_t blk_size = udev->data_blk_size;
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
for (; length > 0; length -= blk_size) {
blk_data_len = min_t(uint32_t, length, blk_size);
dbi = tcmu_get_empty_block(udev, tcmu_cmd, dbi, blk_data_len,
&iov_cnt);
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
if (dbi < 0)
return -1;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
}
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
return iov_cnt;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
}
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
static inline void tcmu_free_cmd(struct tcmu_cmd *tcmu_cmd)
{
kfree(tcmu_cmd->dbi);
kmem_cache_free(tcmu_cmd_cache, tcmu_cmd);
}
static inline void tcmu_cmd_set_block_cnts(struct tcmu_cmd *cmd)
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
{
int i, len;
struct se_cmd *se_cmd = cmd->se_cmd;
uint32_t blk_size = cmd->tcmu_dev->data_blk_size;
cmd->dbi_cnt = DIV_ROUND_UP(se_cmd->data_length, blk_size);
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
if (se_cmd->se_cmd_flags & SCF_BIDI) {
BUG_ON(!(se_cmd->t_bidi_data_sg && se_cmd->t_bidi_data_nents));
for (i = 0, len = 0; i < se_cmd->t_bidi_data_nents; i++)
len += se_cmd->t_bidi_data_sg[i].length;
cmd->dbi_bidi_cnt = DIV_ROUND_UP(len, blk_size);
cmd->dbi_cnt += cmd->dbi_bidi_cnt;
cmd->data_len_bidi = len;
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
}
}
static int new_block_to_iov(struct tcmu_dev *udev, struct tcmu_cmd *cmd,
struct iovec **iov, int prev_dbi, int len)
{
/* Get the next dbi */
int dbi = tcmu_cmd_get_dbi(cmd);
/* Do not add more than udev->data_blk_size to iov */
len = min_t(int, len, udev->data_blk_size);
/*
* The following code will gather and map the blocks to the same iovec
* when the blocks are all next to each other.
*/
if (dbi != prev_dbi + 1) {
/* dbi is not next to previous dbi, so start new iov */
if (prev_dbi >= 0)
(*iov)++;
/* write offset relative to mb_addr */
(*iov)->iov_base = (void __user *)
(udev->data_off + dbi * udev->data_blk_size);
}
(*iov)->iov_len += len;
return dbi;
}
static void tcmu_setup_iovs(struct tcmu_dev *udev, struct tcmu_cmd *cmd,
struct iovec **iov, int data_length)
{
/* start value of dbi + 1 must not be a valid dbi */
int dbi = -2;
/* We prepare the IOVs for DMA_FROM_DEVICE transfer direction */
for (; data_length > 0; data_length -= udev->data_blk_size)
dbi = new_block_to_iov(udev, cmd, iov, dbi, data_length);
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
static struct tcmu_cmd *tcmu_alloc_cmd(struct se_cmd *se_cmd)
{
struct se_device *se_dev = se_cmd->se_dev;
struct tcmu_dev *udev = TCMU_DEV(se_dev);
struct tcmu_cmd *tcmu_cmd;
tcmu_cmd = kmem_cache_zalloc(tcmu_cmd_cache, GFP_NOIO);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
if (!tcmu_cmd)
return NULL;
INIT_LIST_HEAD(&tcmu_cmd->queue_entry);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
tcmu_cmd->se_cmd = se_cmd;
tcmu_cmd->tcmu_dev = udev;
tcmu_cmd_set_block_cnts(tcmu_cmd);
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
tcmu_cmd->dbi = kcalloc(tcmu_cmd->dbi_cnt, sizeof(uint32_t),
GFP_NOIO);
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
if (!tcmu_cmd->dbi) {
kmem_cache_free(tcmu_cmd_cache, tcmu_cmd);
return NULL;
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return tcmu_cmd;
}
static inline void tcmu_flush_dcache_range(void *vaddr, size_t size)
{
unsigned long offset = offset_in_page(vaddr);
void *start = vaddr - offset;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
size = round_up(size+offset, PAGE_SIZE);
while (size) {
scsi: target: tcmu: Fix crash in tcmu_flush_dcache_range on ARM This patch fixes the following crash (see https://bugzilla.kernel.org/show_bug.cgi?id=208045) Process iscsi_trx (pid: 7496, stack limit = 0x0000000010dd111a) CPU: 0 PID: 7496 Comm: iscsi_trx Not tainted 4.19.118-0419118-generic #202004230533 Hardware name: Greatwall QingTian DF720/F601, BIOS 601FBE20 Sep 26 2019 pstate: 80400005 (Nzcv daif +PAN -UAO) pc : flush_dcache_page+0x18/0x40 lr : is_ring_space_avail+0x68/0x2f8 [target_core_user] sp : ffff000015123a80 x29: ffff000015123a80 x28: 0000000000000000 x27: 0000000000001000 x26: ffff000023ea5000 x25: ffffcfa25bbe08b8 x24: 0000000000000078 x23: ffff7e0000000000 x22: ffff000023ea5001 x21: ffffcfa24b79c000 x20: 0000000000000fff x19: ffff7e00008fa940 x18: 0000000000000000 x17: 0000000000000000 x16: ffff2d047e709138 x15: 0000000000000000 x14: 0000000000000000 x13: 0000000000000000 x12: ffff2d047fbd0a40 x11: 0000000000000000 x10: 0000000000000030 x9 : 0000000000000000 x8 : ffffc9a254820a00 x7 : 00000000000013b0 x6 : 000000000000003f x5 : 0000000000000040 x4 : ffffcfa25bbe08e8 x3 : 0000000000001000 x2 : 0000000000000078 x1 : ffffcfa25bbe08b8 x0 : ffff2d040bc88a18 Call trace: flush_dcache_page+0x18/0x40 is_ring_space_avail+0x68/0x2f8 [target_core_user] queue_cmd_ring+0x1f8/0x680 [target_core_user] tcmu_queue_cmd+0xe4/0x158 [target_core_user] __target_execute_cmd+0x30/0xf0 [target_core_mod] target_execute_cmd+0x294/0x390 [target_core_mod] transport_generic_new_cmd+0x1e8/0x358 [target_core_mod] transport_handle_cdb_direct+0x50/0xb0 [target_core_mod] iscsit_execute_cmd+0x2b4/0x350 [iscsi_target_mod] iscsit_sequence_cmd+0xd8/0x1d8 [iscsi_target_mod] iscsit_process_scsi_cmd+0xac/0xf8 [iscsi_target_mod] iscsit_get_rx_pdu+0x404/0xd00 [iscsi_target_mod] iscsi_target_rx_thread+0xb8/0x130 [iscsi_target_mod] kthread+0x130/0x138 ret_from_fork+0x10/0x18 Code: f9000bf3 aa0003f3 aa1e03e0 d503201f (f9400260) ---[ end trace 1e451c73f4266776 ]--- The solution is based on patch: "scsi: target: tcmu: Optimize use of flush_dcache_page" which restricts the use of tcmu_flush_dcache_range() to addresses from vmalloc'ed areas only. This patch now replaces the virt_to_page() call in tcmu_flush_dcache_range() - which is wrong for vmalloced addrs - by vmalloc_to_page(). The patch was tested on ARM with kernel 4.19.118 and 5.7.2 Link: https://lore.kernel.org/r/20200618131632.32748-3-bstroesser@ts.fujitsu.com Tested-by: JiangYu <lnsyyj@hotmail.com> Tested-by: Daniel Meyerholt <dxm523@gmail.com> Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-06-18 16:16:32 +03:00
flush_dcache_page(vmalloc_to_page(start));
start += PAGE_SIZE;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
size -= PAGE_SIZE;
}
}
/*
* Some ring helper functions. We don't assume size is a power of 2 so
* we can't use circ_buf.h.
*/
static inline size_t spc_used(size_t head, size_t tail, size_t size)
{
int diff = head - tail;
if (diff >= 0)
return diff;
else
return size + diff;
}
static inline size_t spc_free(size_t head, size_t tail, size_t size)
{
/* Keep 1 byte unused or we can't tell full from empty */
return (size - spc_used(head, tail, size) - 1);
}
static inline size_t head_to_end(size_t head, size_t size)
{
return size - head;
}
#define UPDATE_HEAD(head, used, size) smp_store_release(&head, ((head % size) + used) % size)
#define TCMU_SG_TO_DATA_AREA 1
#define TCMU_DATA_AREA_TO_SG 2
static inline void tcmu_copy_data(struct tcmu_dev *udev,
struct tcmu_cmd *tcmu_cmd, uint32_t direction,
struct scatterlist *sg, unsigned int sg_nents,
struct iovec **iov, size_t data_len)
{
/* start value of dbi + 1 must not be a valid dbi */
int dbi = -2;
size_t page_remaining, cp_len;
scsi: target: tcmu: Fix xarray RCU warning Commit f5ce815f34bc ("scsi: target: tcmu: Support DATA_BLOCK_SIZE = N * PAGE_SIZE") introduced xas_next() calls to iterate xarray elements. These calls triggered the WARNING "suspicious RCU usage" at tcmu device set up [1]. In the call stack of xas_next(), xas_load() was called. According to its comment, this function requires "the xa_lock or the RCU lock". To avoid the warning: - Guard the small loop calling xas_next() in tcmu_get_empty_block with RCU lock. - In the large loop in tcmu_copy_data using RCU lock would possibly disable preemtion for a long time (copy multi MBs). Therefore replace XA_STATE, xas_set and xas_next with a single xa_load. [1] [ 1899.867091] ============================= [ 1899.871199] WARNING: suspicious RCU usage [ 1899.875310] 5.13.0-rc1+ #41 Not tainted [ 1899.879222] ----------------------------- [ 1899.883299] include/linux/xarray.h:1182 suspicious rcu_dereference_check() usage! [ 1899.890940] other info that might help us debug this: [ 1899.899082] rcu_scheduler_active = 2, debug_locks = 1 [ 1899.905719] 3 locks held by kworker/0:1/1368: [ 1899.910161] #0: ffffa1f8c8b98738 ((wq_completion)target_submission){+.+.}-{0:0}, at: process_one_work+0x1ee/0x580 [ 1899.920732] #1: ffffbd7040cd7e78 ((work_completion)(&q->sq.work)){+.+.}-{0:0}, at: process_one_work+0x1ee/0x580 [ 1899.931146] #2: ffffa1f8d1c99768 (&udev->cmdr_lock){+.+.}-{3:3}, at: tcmu_queue_cmd+0xea/0x160 [target_core_user] [ 1899.941678] stack backtrace: [ 1899.946093] CPU: 0 PID: 1368 Comm: kworker/0:1 Not tainted 5.13.0-rc1+ #41 [ 1899.953070] Hardware name: System manufacturer System Product Name/PRIME Z270-A, BIOS 1302 03/15/2018 [ 1899.962459] Workqueue: target_submission target_queued_submit_work [target_core_mod] [ 1899.970337] Call Trace: [ 1899.972839] dump_stack+0x6d/0x89 [ 1899.976222] xas_descend+0x10e/0x120 [ 1899.979875] xas_load+0x39/0x50 [ 1899.983077] tcmu_get_empty_blocks+0x115/0x1c0 [target_core_user] [ 1899.989318] queue_cmd_ring+0x1da/0x630 [target_core_user] [ 1899.994897] ? rcu_read_lock_sched_held+0x3f/0x70 [ 1899.999695] ? trace_kmalloc+0xa6/0xd0 [ 1900.003501] ? __kmalloc+0x205/0x380 [ 1900.007167] tcmu_queue_cmd+0x12f/0x160 [target_core_user] [ 1900.012746] __target_execute_cmd+0x23/0xa0 [target_core_mod] [ 1900.018589] transport_generic_new_cmd+0x1f3/0x370 [target_core_mod] [ 1900.025046] transport_handle_cdb_direct+0x34/0x50 [target_core_mod] [ 1900.031517] target_queued_submit_work+0x43/0xe0 [target_core_mod] [ 1900.037837] process_one_work+0x268/0x580 [ 1900.041952] ? process_one_work+0x580/0x580 [ 1900.046195] worker_thread+0x55/0x3b0 [ 1900.049921] ? process_one_work+0x580/0x580 [ 1900.054192] kthread+0x143/0x160 [ 1900.057499] ? kthread_create_worker_on_cpu+0x40/0x40 [ 1900.062661] ret_from_fork+0x1f/0x30 Link: https://lore.kernel.org/r/20210519135440.26773-1-bostroesser@gmail.com Fixes: f5ce815f34bc ("scsi: target: tcmu: Support DATA_BLOCK_SIZE = N * PAGE_SIZE") Reported-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Tested-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-05-19 16:54:40 +03:00
int page_cnt, page_inx, dpi;
struct sg_mapping_iter sg_iter;
unsigned int sg_flags;
struct page *page;
void *data_page_start, *data_addr;
if (direction == TCMU_SG_TO_DATA_AREA)
sg_flags = SG_MITER_ATOMIC | SG_MITER_FROM_SG;
else
sg_flags = SG_MITER_ATOMIC | SG_MITER_TO_SG;
sg_miter_start(&sg_iter, sg, sg_nents, sg_flags);
while (data_len) {
if (direction == TCMU_SG_TO_DATA_AREA)
dbi = new_block_to_iov(udev, tcmu_cmd, iov, dbi,
data_len);
else
dbi = tcmu_cmd_get_dbi(tcmu_cmd);
page_cnt = DIV_ROUND_UP(data_len, PAGE_SIZE);
if (page_cnt > udev->data_pages_per_blk)
page_cnt = udev->data_pages_per_blk;
scsi: target: tcmu: Fix xarray RCU warning Commit f5ce815f34bc ("scsi: target: tcmu: Support DATA_BLOCK_SIZE = N * PAGE_SIZE") introduced xas_next() calls to iterate xarray elements. These calls triggered the WARNING "suspicious RCU usage" at tcmu device set up [1]. In the call stack of xas_next(), xas_load() was called. According to its comment, this function requires "the xa_lock or the RCU lock". To avoid the warning: - Guard the small loop calling xas_next() in tcmu_get_empty_block with RCU lock. - In the large loop in tcmu_copy_data using RCU lock would possibly disable preemtion for a long time (copy multi MBs). Therefore replace XA_STATE, xas_set and xas_next with a single xa_load. [1] [ 1899.867091] ============================= [ 1899.871199] WARNING: suspicious RCU usage [ 1899.875310] 5.13.0-rc1+ #41 Not tainted [ 1899.879222] ----------------------------- [ 1899.883299] include/linux/xarray.h:1182 suspicious rcu_dereference_check() usage! [ 1899.890940] other info that might help us debug this: [ 1899.899082] rcu_scheduler_active = 2, debug_locks = 1 [ 1899.905719] 3 locks held by kworker/0:1/1368: [ 1899.910161] #0: ffffa1f8c8b98738 ((wq_completion)target_submission){+.+.}-{0:0}, at: process_one_work+0x1ee/0x580 [ 1899.920732] #1: ffffbd7040cd7e78 ((work_completion)(&q->sq.work)){+.+.}-{0:0}, at: process_one_work+0x1ee/0x580 [ 1899.931146] #2: ffffa1f8d1c99768 (&udev->cmdr_lock){+.+.}-{3:3}, at: tcmu_queue_cmd+0xea/0x160 [target_core_user] [ 1899.941678] stack backtrace: [ 1899.946093] CPU: 0 PID: 1368 Comm: kworker/0:1 Not tainted 5.13.0-rc1+ #41 [ 1899.953070] Hardware name: System manufacturer System Product Name/PRIME Z270-A, BIOS 1302 03/15/2018 [ 1899.962459] Workqueue: target_submission target_queued_submit_work [target_core_mod] [ 1899.970337] Call Trace: [ 1899.972839] dump_stack+0x6d/0x89 [ 1899.976222] xas_descend+0x10e/0x120 [ 1899.979875] xas_load+0x39/0x50 [ 1899.983077] tcmu_get_empty_blocks+0x115/0x1c0 [target_core_user] [ 1899.989318] queue_cmd_ring+0x1da/0x630 [target_core_user] [ 1899.994897] ? rcu_read_lock_sched_held+0x3f/0x70 [ 1899.999695] ? trace_kmalloc+0xa6/0xd0 [ 1900.003501] ? __kmalloc+0x205/0x380 [ 1900.007167] tcmu_queue_cmd+0x12f/0x160 [target_core_user] [ 1900.012746] __target_execute_cmd+0x23/0xa0 [target_core_mod] [ 1900.018589] transport_generic_new_cmd+0x1f3/0x370 [target_core_mod] [ 1900.025046] transport_handle_cdb_direct+0x34/0x50 [target_core_mod] [ 1900.031517] target_queued_submit_work+0x43/0xe0 [target_core_mod] [ 1900.037837] process_one_work+0x268/0x580 [ 1900.041952] ? process_one_work+0x580/0x580 [ 1900.046195] worker_thread+0x55/0x3b0 [ 1900.049921] ? process_one_work+0x580/0x580 [ 1900.054192] kthread+0x143/0x160 [ 1900.057499] ? kthread_create_worker_on_cpu+0x40/0x40 [ 1900.062661] ret_from_fork+0x1f/0x30 Link: https://lore.kernel.org/r/20210519135440.26773-1-bostroesser@gmail.com Fixes: f5ce815f34bc ("scsi: target: tcmu: Support DATA_BLOCK_SIZE = N * PAGE_SIZE") Reported-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Tested-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-05-19 16:54:40 +03:00
dpi = dbi * udev->data_pages_per_blk;
for (page_inx = 0; page_inx < page_cnt && data_len;
page_inx++, dpi++) {
page = xa_load(&udev->data_pages, dpi);
if (direction == TCMU_DATA_AREA_TO_SG)
flush_dcache_page(page);
data_page_start = kmap_atomic(page);
page_remaining = PAGE_SIZE;
while (page_remaining && data_len) {
if (!sg_miter_next(&sg_iter)) {
/* set length to 0 to abort outer loop */
data_len = 0;
pr_debug("%s: aborting data copy due to exhausted sg_list\n",
__func__);
break;
}
cp_len = min3(sg_iter.length, page_remaining,
data_len);
data_addr = data_page_start +
PAGE_SIZE - page_remaining;
if (direction == TCMU_SG_TO_DATA_AREA)
memcpy(data_addr, sg_iter.addr, cp_len);
else
memcpy(sg_iter.addr, data_addr, cp_len);
data_len -= cp_len;
page_remaining -= cp_len;
sg_iter.consumed = cp_len;
}
sg_miter_stop(&sg_iter);
kunmap_atomic(data_page_start);
if (direction == TCMU_SG_TO_DATA_AREA)
flush_dcache_page(page);
}
}
}
static void scatter_data_area(struct tcmu_dev *udev, struct tcmu_cmd *tcmu_cmd,
struct iovec **iov)
{
struct se_cmd *se_cmd = tcmu_cmd->se_cmd;
tcmu_copy_data(udev, tcmu_cmd, TCMU_SG_TO_DATA_AREA, se_cmd->t_data_sg,
se_cmd->t_data_nents, iov, se_cmd->data_length);
}
static void gather_data_area(struct tcmu_dev *udev, struct tcmu_cmd *tcmu_cmd,
scsi: target: tcmu: add read length support Generally target core and TCMUser seem to work fine for tape devices and media changers. But there is at least one situation where TCMUser is not able to support sequential access device emulation correctly. The situation is when an initiator sends a SCSI READ CDB with a length that is greater than the length of the tape block to read. We can distinguish two subcases: A) The initiator sent the READ CDB with the SILI bit being set. In this case the sequential access device has to transfer the data from the tape block (only the length of the tape block) and transmit a good status. The current interface between TCMUser and the userspace does not support reduction of the read data size by the userspace program. The patch below fixes this subcase by allowing the userspace program to specify a reduced data size in read direction. B) The initiator sent the READ CDB with the SILI bit not being set. In this case the sequential access device has to transfer the data from the tape block as in A), but additionally has to transmit CHECK CONDITION with the ILI bit set and NO SENSE in the sensebytes. The information field in the sensebytes must contain the residual count. With the below patch a user space program can specify the real read data length and appropriate sensebytes. TCMUser then uses the se_cmd flag SCF_TREAT_READ_AS_NORMAL, to force target core to transmit the real data size and the sensebytes. Note: the flag SCF_TREAT_READ_AS_NORMAL is introduced by Lee Duncan's patch "[PATCH v4] target: transport should handle st FM/EOM/ILI reads" from Tue, 15 May 2018 18:25:24 -0700. Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Acked-by: Mike Christie <mchristi@redhat.com> Reviewed-by: Lee Duncan <lduncan@suse.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2018-05-24 19:49:41 +03:00
bool bidi, uint32_t read_len)
{
struct se_cmd *se_cmd = tcmu_cmd->se_cmd;
struct scatterlist *data_sg;
unsigned int data_nents;
if (!bidi) {
data_sg = se_cmd->t_data_sg;
data_nents = se_cmd->t_data_nents;
} else {
/*
* For bidi case, the first count blocks are for Data-Out
* buffer blocks, and before gathering the Data-In buffer
* the Data-Out buffer blocks should be skipped.
*/
tcmu_cmd_set_dbi_cur(tcmu_cmd,
tcmu_cmd->dbi_cnt - tcmu_cmd->dbi_bidi_cnt);
data_sg = se_cmd->t_bidi_data_sg;
data_nents = se_cmd->t_bidi_data_nents;
}
tcmu_copy_data(udev, tcmu_cmd, TCMU_DATA_AREA_TO_SG, data_sg,
data_nents, NULL, read_len);
}
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
static inline size_t spc_bitmap_free(unsigned long *bitmap, uint32_t thresh)
{
return thresh - bitmap_weight(bitmap, thresh);
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
/*
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
* We can't queue a command until we have space available on the cmd ring.
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
*
* Called with ring lock held.
*/
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
static bool is_ring_space_avail(struct tcmu_dev *udev, size_t cmd_size)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
struct tcmu_mailbox *mb = udev->mb_addr;
size_t space, cmd_needed;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
u32 cmd_head;
tcmu_flush_dcache_range(mb, sizeof(*mb));
cmd_head = mb->cmd_head % udev->cmdr_size; /* UAM */
/*
* If cmd end-of-ring space is too small then we need space for a NOP plus
* original cmd - cmds are internally contiguous.
*/
if (head_to_end(cmd_head, udev->cmdr_size) >= cmd_size)
cmd_needed = cmd_size;
else
cmd_needed = cmd_size + head_to_end(cmd_head, udev->cmdr_size);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
space = spc_free(cmd_head, udev->cmdr_last_cleaned, udev->cmdr_size);
if (space < cmd_needed) {
pr_debug("no cmd space: %u %u %u\n", cmd_head,
udev->cmdr_last_cleaned, udev->cmdr_size);
return false;
}
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
return true;
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
/*
* We have to allocate data buffers before we can queue a command.
* Returns -1 on error (not enough space) or number of needed iovs on success
*
* Called with ring lock held.
*/
static int tcmu_alloc_data_space(struct tcmu_dev *udev, struct tcmu_cmd *cmd,
int *iov_bidi_cnt)
{
int space, iov_cnt = 0, ret = 0;
if (!cmd->dbi_cnt)
goto wr_iov_cnts;
2020-07-26 18:35:09 +03:00
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
/* try to check and get the data blocks as needed */
space = spc_bitmap_free(udev->data_bitmap, udev->dbi_thresh);
if (space < cmd->dbi_cnt) {
unsigned long blocks_left =
(udev->max_blocks - udev->dbi_thresh) + space;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
if (blocks_left < cmd->dbi_cnt) {
pr_debug("no data space: only %lu available, but ask for %u\n",
blocks_left * udev->data_blk_size,
cmd->dbi_cnt * udev->data_blk_size);
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
return -1;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
}
udev->dbi_thresh += cmd->dbi_cnt;
if (udev->dbi_thresh > udev->max_blocks)
udev->dbi_thresh = udev->max_blocks;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
iov_cnt = tcmu_get_empty_blocks(udev, cmd, cmd->se_cmd->data_length);
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
if (iov_cnt < 0)
return -1;
if (cmd->dbi_bidi_cnt) {
ret = tcmu_get_empty_blocks(udev, cmd, cmd->data_len_bidi);
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
if (ret < 0)
return -1;
}
wr_iov_cnts:
*iov_bidi_cnt = ret;
return iov_cnt + ret;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
static inline size_t tcmu_cmd_get_base_cmd_size(size_t iov_cnt)
{
return max(offsetof(struct tcmu_cmd_entry, req.iov[iov_cnt]),
sizeof(struct tcmu_cmd_entry));
}
static inline size_t tcmu_cmd_get_cmd_size(struct tcmu_cmd *tcmu_cmd,
size_t base_command_size)
{
struct se_cmd *se_cmd = tcmu_cmd->se_cmd;
size_t command_size;
command_size = base_command_size +
round_up(scsi_command_size(se_cmd->t_task_cdb),
TCMU_OP_ALIGN_SIZE);
WARN_ON(command_size & (TCMU_OP_ALIGN_SIZE-1));
return command_size;
}
static void tcmu_setup_cmd_timer(struct tcmu_cmd *tcmu_cmd, unsigned int tmo,
struct timer_list *timer)
{
if (!tmo)
return;
tcmu_cmd->deadline = round_jiffies_up(jiffies + msecs_to_jiffies(tmo));
if (!timer_pending(timer))
mod_timer(timer, tcmu_cmd->deadline);
pr_debug("Timeout set up for cmd %p, dev = %s, tmo = %lu\n", tcmu_cmd,
tcmu_cmd->tcmu_dev->name, tmo / MSEC_PER_SEC);
}
static int add_to_qfull_queue(struct tcmu_cmd *tcmu_cmd)
{
struct tcmu_dev *udev = tcmu_cmd->tcmu_dev;
unsigned int tmo;
/*
* For backwards compat if qfull_time_out is not set use
* cmd_time_out and if that's not set use the default time out.
*/
if (!udev->qfull_time_out)
return -ETIMEDOUT;
else if (udev->qfull_time_out > 0)
tmo = udev->qfull_time_out;
else if (udev->cmd_time_out)
tmo = udev->cmd_time_out;
else
tmo = TCMU_TIME_OUT;
tcmu_setup_cmd_timer(tcmu_cmd, tmo, &udev->qfull_timer);
list_add_tail(&tcmu_cmd->queue_entry, &udev->qfull_queue);
pr_debug("adding cmd %p on dev %s to ring space wait queue\n",
tcmu_cmd, udev->name);
return 0;
}
static uint32_t ring_insert_padding(struct tcmu_dev *udev, size_t cmd_size)
{
struct tcmu_cmd_entry_hdr *hdr;
struct tcmu_mailbox *mb = udev->mb_addr;
uint32_t cmd_head = mb->cmd_head % udev->cmdr_size; /* UAM */
/* Insert a PAD if end-of-ring space is too small */
if (head_to_end(cmd_head, udev->cmdr_size) < cmd_size) {
size_t pad_size = head_to_end(cmd_head, udev->cmdr_size);
hdr = udev->cmdr + cmd_head;
tcmu_hdr_set_op(&hdr->len_op, TCMU_OP_PAD);
tcmu_hdr_set_len(&hdr->len_op, pad_size);
hdr->cmd_id = 0; /* not used for PAD */
hdr->kflags = 0;
hdr->uflags = 0;
tcmu_flush_dcache_range(hdr, sizeof(*hdr));
UPDATE_HEAD(mb->cmd_head, pad_size, udev->cmdr_size);
tcmu_flush_dcache_range(mb, sizeof(*mb));
cmd_head = mb->cmd_head % udev->cmdr_size; /* UAM */
WARN_ON(cmd_head != 0);
}
return cmd_head;
}
static void tcmu_unplug_device(struct se_dev_plug *se_plug)
{
struct se_device *se_dev = se_plug->se_dev;
struct tcmu_dev *udev = TCMU_DEV(se_dev);
clear_bit(TCMU_DEV_BIT_PLUGGED, &udev->flags);
uio_event_notify(&udev->uio_info);
}
static struct se_dev_plug *tcmu_plug_device(struct se_device *se_dev)
{
struct tcmu_dev *udev = TCMU_DEV(se_dev);
if (!test_and_set_bit(TCMU_DEV_BIT_PLUGGED, &udev->flags))
return &udev->se_plug;
return NULL;
}
/**
* queue_cmd_ring - queue cmd to ring or internally
* @tcmu_cmd: cmd to queue
* @scsi_err: TCM error code if failure (-1) returned.
*
* Returns:
* -1 we cannot queue internally or to the ring.
* 0 success
* 1 internally queued to wait for ring memory to free.
*/
static int queue_cmd_ring(struct tcmu_cmd *tcmu_cmd, sense_reason_t *scsi_err)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
struct tcmu_dev *udev = tcmu_cmd->tcmu_dev;
struct se_cmd *se_cmd = tcmu_cmd->se_cmd;
size_t base_command_size, command_size;
struct tcmu_mailbox *mb = udev->mb_addr;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
struct tcmu_cmd_entry *entry;
struct iovec *iov;
int iov_cnt, iov_bidi_cnt;
uint32_t cmd_id, cmd_head;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
uint64_t cdb_off;
uint32_t blk_size = udev->data_blk_size;
/* size of data buffer needed */
size_t data_length = (size_t)tcmu_cmd->dbi_cnt * blk_size;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
*scsi_err = TCM_NO_SENSE;
if (test_bit(TCMU_DEV_BIT_BLOCKED, &udev->flags)) {
*scsi_err = TCM_LUN_BUSY;
return -1;
}
if (test_bit(TCMU_DEV_BIT_BROKEN, &udev->flags)) {
*scsi_err = TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
return -1;
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
if (!list_empty(&udev->qfull_queue))
goto queue;
if (data_length > (size_t)udev->max_blocks * blk_size) {
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
pr_warn("TCMU: Request of size %zu is too big for %zu data area\n",
data_length, (size_t)udev->max_blocks * blk_size);
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
*scsi_err = TCM_INVALID_CDB_FIELD;
return -1;
}
iov_cnt = tcmu_alloc_data_space(udev, tcmu_cmd, &iov_bidi_cnt);
if (iov_cnt < 0)
goto free_and_queue;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
/*
* Must be a certain minimum size for response sense info, but
* also may be larger if the iov array is large.
*/
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
base_command_size = tcmu_cmd_get_base_cmd_size(iov_cnt);
command_size = tcmu_cmd_get_cmd_size(tcmu_cmd, base_command_size);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
if (command_size > (udev->cmdr_size / 2)) {
pr_warn("TCMU: Request of size %zu is too big for %u cmd ring\n",
command_size, udev->cmdr_size);
tcmu_cmd_free_data(tcmu_cmd, tcmu_cmd->dbi_cur);
*scsi_err = TCM_INVALID_CDB_FIELD;
return -1;
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
if (!is_ring_space_avail(udev, command_size))
/*
* Don't leave commands partially setup because the unmap
* thread might need the blocks to make forward progress.
*/
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
goto free_and_queue;
if (xa_alloc(&udev->commands, &cmd_id, tcmu_cmd, XA_LIMIT(1, 0xffff),
GFP_NOWAIT) < 0) {
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
pr_err("tcmu: Could not allocate cmd id.\n");
tcmu_cmd_free_data(tcmu_cmd, tcmu_cmd->dbi_cnt);
*scsi_err = TCM_OUT_OF_RESOURCES;
return -1;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
tcmu_cmd->cmd_id = cmd_id;
pr_debug("allocated cmd id %u for cmd %p dev %s\n", tcmu_cmd->cmd_id,
tcmu_cmd, udev->name);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
cmd_head = ring_insert_padding(udev, command_size);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
entry = udev->cmdr + cmd_head;
tcmu: Fix possbile memory leak / OOPs when recalculating cmd base size For all the entries allocated from the ring cmd area, the memory is something like the stack memory, which will always reserve the old data, so the entry->req.iov_bidi_cnt maybe none zero. On some environments, the crash could be reproduce very easy and some not. The following is the crash core trace as reported by Damien: [ 240.143969] CPU: 0 PID: 1285 Comm: iscsi_trx Not tainted 4.12.0-rc1+ #3 [ 240.150607] Hardware name: ASUS All Series/H87-PRO, BIOS 2104 10/28/2014 [ 240.157331] task: ffff8807de4f5800 task.stack: ffffc900047dc000 [ 240.163270] RIP: 0010:memcpy_erms+0x6/0x10 [ 240.167377] RSP: 0018:ffffc900047dfc68 EFLAGS: 00010202 [ 240.172621] RAX: ffffc9065db85540 RBX: ffff8807f7980000 RCX: 0000000000000010 [ 240.179771] RDX: 0000000000000010 RSI: ffff8807de574fe0 RDI: ffffc9065db85540 [ 240.186930] RBP: ffffc900047dfd30 R08: ffff8807de41b000 R09: 0000000000000000 [ 240.194088] R10: 0000000000000040 R11: ffff8807e9b726f0 R12: 00000006565726b0 [ 240.201246] R13: ffffc90007612ea0 R14: 000000065657d540 R15: 0000000000000000 [ 240.208397] FS: 0000000000000000(0000) GS:ffff88081fa00000(0000) knlGS:0000000000000000 [ 240.216510] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 240.222280] CR2: ffffc9065db85540 CR3: 0000000001c0f000 CR4: 00000000001406f0 [ 240.229430] Call Trace: [ 240.231887] ? tcmu_queue_cmd+0x83c/0xa80 [ 240.235916] ? target_check_reservation+0xcd/0x6f0 [ 240.240725] __target_execute_cmd+0x27/0xa0 [ 240.244918] target_execute_cmd+0x232/0x2c0 [ 240.249124] ? __local_bh_enable_ip+0x64/0xa0 [ 240.253499] iscsit_execute_cmd+0x20d/0x270 [ 240.257693] iscsit_sequence_cmd+0x110/0x190 [ 240.261985] iscsit_get_rx_pdu+0x360/0xc80 [ 240.267565] ? iscsi_target_rx_thread+0x54/0xd0 [ 240.273571] iscsi_target_rx_thread+0x9a/0xd0 [ 240.279413] kthread+0x113/0x150 [ 240.284120] ? iscsi_target_tx_thread+0x1e0/0x1e0 [ 240.290297] ? kthread_create_on_node+0x40/0x40 [ 240.296297] ret_from_fork+0x2e/0x40 [ 240.301332] Code: 90 90 90 90 90 eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 [ 240.321751] RIP: memcpy_erms+0x6/0x10 RSP: ffffc900047dfc68 [ 240.328838] CR2: ffffc9065db85540 [ 240.333667] ---[ end trace b7e5354cfb54d08b ]--- To fix this, just memset all the entry memory before using it, and also to be more readable we adjust the bidi code. Fixed: fe25cc34795(tcmu: Recalculate the tcmu_cmd size to save cmd area memories) Reported-by: Bryant G. Ly <bryantly@linux.vnet.ibm.com> Tested-by: Bryant G. Ly <bryantly@linux.vnet.ibm.com> Reported-by: Damien Le Moal <damien.lemoal@wdc.com> Tested-by: Damien Le Moal <damien.lemoal@wdc.com> Reviewed-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Cc: <stable@vger.kernel.org> # 4.12+ Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-07-11 12:59:43 +03:00
memset(entry, 0, command_size);
target: Version 2 of TCMU ABI The initial version of TCMU (in 3.18) does not properly handle bidirectional SCSI commands -- those with both an in and out buffer. In looking to fix this it also became clear that TCMU's support for adding new types of entries (opcodes) to the command ring was broken. We need to fix this now, so that future issues can be handled properly by adding new opcodes. We make the most of this ABI break by enabling bidi cmd handling within TCMP_OP_CMD opcode. Add an iov_bidi_cnt field to tcmu_cmd_entry.req. This enables TCMU to describe bidi commands, but further kernel work is needed for full bidi support. Enlarge tcmu_cmd_entry_hdr by 32 bits by pulling in cmd_id and __pad1. Turn __pad1 into two 8 bit flags fields, for kernel-set and userspace-set flags, "kflags" and "uflags" respectively. Update version fields so userspace can tell the interface is changed. Update tcmu-design.txt with details of how new stuff works: - Specify an additional requirement for userspace to set UNKNOWN_OP (bit 0) in hdr.uflags for unknown/unhandled opcodes. - Define how Data-In and Data-Out fields are described in req.iov[] Changed in v2: - Change name of SKIPPED bit to UNKNOWN bit - PAD op does not set the bit any more - Change len_op helper functions to take just len_op, not the whole struct - Change version to 2 in missed spots, and use defines - Add 16 unused bytes to cmd_entry.req, in case additional SAM cmd parameters need to be included - Add iov_dif_cnt field to specify buffers used for DIF info in iov[] - Rearrange fields to naturally align cdb_off - Handle if userspace sets UNKNOWN_OP by indicating failure of the cmd - Wrap some overly long UPDATE_HEAD lines (Add missing req.iov_bidi_cnt + req.iov_dif_cnt zeroing - Ilias) Signed-off-by: Andy Grover <agrover@redhat.com> Reviewed-by: Ilias Tsitsimpis <iliastsi@arrikto.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2015-04-15 03:30:04 +03:00
tcmu_hdr_set_op(&entry->hdr.len_op, TCMU_OP_CMD);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
/* prepare iov list and copy data to data area if necessary */
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
tcmu_cmd_reset_dbi_cur(tcmu_cmd);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
iov = &entry->req.iov[0];
if (se_cmd->data_direction == DMA_TO_DEVICE ||
se_cmd->se_cmd_flags & SCF_BIDI)
scatter_data_area(udev, tcmu_cmd, &iov);
else
tcmu_setup_iovs(udev, tcmu_cmd, &iov, se_cmd->data_length);
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
entry->req.iov_cnt = iov_cnt - iov_bidi_cnt;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
/* Handle BIDI commands */
if (se_cmd->se_cmd_flags & SCF_BIDI) {
iov++;
tcmu_setup_iovs(udev, tcmu_cmd, &iov, tcmu_cmd->data_len_bidi);
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
entry->req.iov_bidi_cnt = iov_bidi_cnt;
}
tcmu_setup_cmd_timer(tcmu_cmd, udev->cmd_time_out, &udev->cmd_timer);
entry->hdr.cmd_id = tcmu_cmd->cmd_id;
tcmu_hdr_set_len(&entry->hdr.len_op, command_size);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
/* All offsets relative to mb_addr, not start of entry! */
cdb_off = CMDR_OFF + cmd_head + base_command_size;
memcpy((void *) mb + cdb_off, se_cmd->t_task_cdb, scsi_command_size(se_cmd->t_task_cdb));
entry->req.cdb_off = cdb_off;
scsi: target: tcmu: Fix size in calls to tcmu_flush_dcache_range 1) If remaining ring space before the end of the ring is smaller then the next cmd to write, tcmu writes a padding entry which fills the remaining space at the end of the ring. Then tcmu calls tcmu_flush_dcache_range() with the size of struct tcmu_cmd_entry as data length to flush. If the space filled by the padding was smaller then tcmu_cmd_entry, tcmu_flush_dcache_range() is called for an address range reaching behind the end of the vmalloc'ed ring. tcmu_flush_dcache_range() in a loop calls flush_dcache_page(virt_to_page(start)); for every page being part of the range. On x86 the line is optimized out by the compiler, as flush_dcache_page() is empty on x86. But I assume the above can cause trouble on other architectures that really have a flush_dcache_page(). For paddings only the header part of an entry is relevant due to alignment rules the header always fits in the remaining space, if padding is needed. So tcmu_flush_dcache_range() can safely be called with sizeof(entry->hdr) as the length here. 2) After it has written a command to cmd ring, tcmu calls tcmu_flush_dcache_range() using the size of a struct tcmu_cmd_entry as data length to flush. But if a command needs many iovecs, the real size of the command may be bigger then tcmu_cmd_entry, so a part of the written command is not flushed then. Link: https://lore.kernel.org/r/20200528193108.9085-1-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-05-28 22:31:08 +03:00
tcmu_flush_dcache_range(entry, command_size);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
UPDATE_HEAD(mb->cmd_head, command_size, udev->cmdr_size);
tcmu_flush_dcache_range(mb, sizeof(*mb));
list_add_tail(&tcmu_cmd->queue_entry, &udev->inflight_queue);
if (!test_bit(TCMU_DEV_BIT_PLUGGED, &udev->flags))
uio_event_notify(&udev->uio_info);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return 0;
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
free_and_queue:
tcmu_cmd_free_data(tcmu_cmd, tcmu_cmd->dbi_cur);
tcmu_cmd_reset_dbi_cur(tcmu_cmd);
queue:
if (add_to_qfull_queue(tcmu_cmd)) {
*scsi_err = TCM_OUT_OF_RESOURCES;
return -1;
}
return 1;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
2020-07-26 18:35:09 +03:00
/**
* queue_tmr_ring - queue tmr info to ring or internally
* @udev: related tcmu_dev
* @tmr: tcmu_tmr containing tmr info to queue
*
* Returns:
* 0 success
* 1 internally queued to wait for ring memory to free.
*/
static int
queue_tmr_ring(struct tcmu_dev *udev, struct tcmu_tmr *tmr)
{
struct tcmu_tmr_entry *entry;
int cmd_size;
int id_list_sz;
struct tcmu_mailbox *mb = udev->mb_addr;
uint32_t cmd_head;
if (test_bit(TCMU_DEV_BIT_BROKEN, &udev->flags))
goto out_free;
id_list_sz = sizeof(tmr->tmr_cmd_ids[0]) * tmr->tmr_cmd_cnt;
cmd_size = round_up(sizeof(*entry) + id_list_sz, TCMU_OP_ALIGN_SIZE);
if (!list_empty(&udev->tmr_queue) ||
scsi: target: tcmu: Optimize queue_cmd_ring() queue_cmd_ring() needs to check whether there is enough space in cmd ring and data area for the cmd to queue. Currently the sequence is: 1) Calculate size the cmd will occupy on the ring based on estimation of needed iovs. 2) Check whether there is enough space on the ring based on size from 1) 3) Allocate buffers in data area. 4) Calculate number of iovs the command really needs while copying incoming data (if any) to data area. 5) Re-calculate real size of cmd on ring based on real number of iovs. 6) Set up possible padding and cmd on the ring. Step 1) must not underestimate the cmd size so use max possible number of iovs for the given I/O data size. The resulting overestimation can be really high so this sequence is not ideal. The earliest the real number of iovs can be calculated is after data buffer allocation. Therefore rework the code to implement the following sequence: A) Allocate buffers on data area and calculate number of necessary iovs during this. B) Calculate real size of cmd on ring based on number of iovs. C) Check whether there is enough space on the ring. D) Set up possible padding and cmd on the ring. The new sequence enforces the split of new function tcmu_alloc_data_space() from is_ring_space_avail(). Using this function, change queue_cmd_ring() according to the new sequence. Change routines called by tcmu_alloc_data_space() to allow calculating and returning the iov count. Remove counting of iovs in scatter_data_area(). Link: https://lore.kernel.org/r/20200910155041.17654-3-bstroesser@ts.fujitsu.com Acked-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-09-10 18:50:40 +03:00
!is_ring_space_avail(udev, cmd_size)) {
2020-07-26 18:35:09 +03:00
list_add_tail(&tmr->queue_entry, &udev->tmr_queue);
pr_debug("adding tmr %p on dev %s to TMR ring space wait queue\n",
tmr, udev->name);
return 1;
}
cmd_head = ring_insert_padding(udev, cmd_size);
entry = udev->cmdr + cmd_head;
2020-07-26 18:35:09 +03:00
memset(entry, 0, cmd_size);
tcmu_hdr_set_op(&entry->hdr.len_op, TCMU_OP_TMR);
tcmu_hdr_set_len(&entry->hdr.len_op, cmd_size);
entry->tmr_type = tmr->tmr_type;
entry->cmd_cnt = tmr->tmr_cmd_cnt;
memcpy(&entry->cmd_ids[0], &tmr->tmr_cmd_ids[0], id_list_sz);
tcmu_flush_dcache_range(entry, cmd_size);
UPDATE_HEAD(mb->cmd_head, cmd_size, udev->cmdr_size);
tcmu_flush_dcache_range(mb, sizeof(*mb));
uio_event_notify(&udev->uio_info);
out_free:
kfree(tmr);
return 0;
}
static sense_reason_t
tcmu_queue_cmd(struct se_cmd *se_cmd)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
struct se_device *se_dev = se_cmd->se_dev;
struct tcmu_dev *udev = TCMU_DEV(se_dev);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
struct tcmu_cmd *tcmu_cmd;
sense_reason_t scsi_ret = TCM_CHECK_CONDITION_ABORT_CMD;
int ret = -1;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
tcmu_cmd = tcmu_alloc_cmd(se_cmd);
if (!tcmu_cmd)
return TCM_LOGICAL_UNIT_COMMUNICATION_FAILURE;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
mutex_lock(&udev->cmdr_lock);
if (!(se_cmd->transport_state & CMD_T_ABORTED))
ret = queue_cmd_ring(tcmu_cmd, &scsi_ret);
if (ret < 0)
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
tcmu_free_cmd(tcmu_cmd);
scsi: target: tcmu: Fix use-after-free of se_cmd->priv Commit a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") modified tcmu_free_cmd() to set NULL to priv pointer in se_cmd. However, se_cmd can be already freed by work queue triggered in target_complete_cmd(). This caused BUG KASAN use-after-free [1]. To fix the bug, do not touch priv pointer in tcmu_free_cmd(). Instead, set NULL to priv pointer before target_complete_cmd() calls. Also, to avoid unnecessary priv pointer change in tcmu_queue_cmd(), modify priv pointer in the function only when tcmu_free_cmd() is not called. [1] BUG: KASAN: use-after-free in tcmu_handle_completions+0x1172/0x1770 [target_core_user] Write of size 8 at addr ffff88814cf79a40 by task cmdproc-uio0/14842 CPU: 2 PID: 14842 Comm: cmdproc-uio0 Not tainted 5.11.0-rc2 #1 Hardware name: Supermicro Super Server/X10SRL-F, BIOS 3.2 11/22/2019 Call Trace: dump_stack+0x9a/0xcc ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] print_address_description.constprop.0+0x18/0x130 ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] kasan_report.cold+0x7f/0x10e ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? queue_tmr_ring+0x5d0/0x5d0 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 ? uio_vma_fault+0x460/0x460 ? security_file_permission+0x4f/0x440 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 ? __ia32_sys_read+0xb0/0xb0 ? syscall_enter_from_user_mode+0x27/0x70 ? trace_hardirqs_on+0x1c/0x110 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fcf8b61905f Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 0c fd ff ff 48 RSP: 002b:00007fcf7b3e6c30 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcf8b61905f RDX: 0000000000000004 RSI: 00007fcf7b3e6c78 RDI: 000000000000000c RBP: 00007fcf7b3e6c80 R08: 0000000000000000 R09: 00007fcf7b3e6aa8 R10: 000000000b01c000 R11: 0000000000000293 R12: 00007ffe0c32a52e R13: 00007ffe0c32a52f R14: 0000000000000000 R15: 00007fcf7b3e7640 Allocated by task 383: kasan_save_stack+0x1b/0x40 ____kasan_kmalloc.constprop.0+0x84/0xa0 kmem_cache_alloc+0x142/0x330 tcm_loop_queuecommand+0x2a/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Freed by task 11655: kasan_save_stack+0x1b/0x40 kasan_set_track+0x1c/0x30 kasan_set_free_info+0x20/0x30 ____kasan_slab_free+0xec/0x120 slab_free_freelist_hook+0x53/0x160 kmem_cache_free+0xf4/0x5c0 target_release_cmd_kref+0x3ea/0x9e0 [target_core_mod] transport_generic_free_cmd+0x28b/0x2f0 [target_core_mod] target_complete_ok_work+0x250/0xac0 [target_core_mod] process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcmu_handle_completions+0xad0/0x1770 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Second to last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcm_loop_queuecommand+0x1c3/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 The buggy address belongs to the object at ffff88814cf79800 which belongs to the cache tcm_loop_cmd_cache of size 896. Link: https://lore.kernel.org/r/20210113024508.1264992-1-shinichiro.kawasaki@wdc.com Fixes: a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") Cc: stable@vger.kernel.org # v5.9+ Acked-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-01-13 05:45:08 +03:00
else
se_cmd->priv = tcmu_cmd;
mutex_unlock(&udev->cmdr_lock);
return scsi_ret;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
scsi: target: tcmu: Fix and simplify timeout handling During cmd timeout handling in check_timedout_devices(), due to a race, it can happen that tcmu_set_next_deadline() does not start a timer as expected: 1) Either tcmu_check_expired_ring_cmd() checks the inflight_queue or tcmu_check_expired_queue_cmd() checks the qfull_queue while jiffies has the value X 2) At the end of the check the queue contains one remaining command with deadline X (time_after(X, X) is false and thus the command is not handled as being timed out). 3) After tcmu_check_expired_xxxxx_cmd() a timer interrupt happens and jiffies is incremented to X+1. 4) Now tcmu_set_next_deadline() is called, but it skips the command, since time_after(X+1, X) is true. Therefore tcmu_set_next_deadline() finds no new deadline and stops the timer, which it shouldn't. Since commands that time out are removed from inflight_queue or qfull_queue, we don't need the check with time_after() in tcmu_set_next_deadline() but can use the deadline from the first cmd in the queue. Additionally, replace the remaining time_after() calls in tcmu_check_expired_xxxxx_cmd() with time_after_eq(), because it is not useful to set the timeout to deadline but then check for jiffies being greater than deadline. Simplify the end of tcmu_handle_completions() and change the check for no more pending commands from mb->cmd_tail == mb->cmd_head to idr_is_empty(&udev->commands) because the old check doesn't work correctly if paddings or in the future TMRs are in the ring. Finally tcmu_set_next_deadline() was shifted in the source as preparation for later implementation of tmr_notify callback. Link: https://lore.kernel.org/r/20200726153510.13077-7-bstroesser@ts.fujitsu.com Reviewed-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-07-26 18:35:08 +03:00
static void tcmu_set_next_deadline(struct list_head *queue,
struct timer_list *timer)
{
struct tcmu_cmd *cmd;
if (!list_empty(queue)) {
cmd = list_first_entry(queue, struct tcmu_cmd, queue_entry);
mod_timer(timer, cmd->deadline);
} else
del_timer(timer);
}
2020-07-26 18:35:09 +03:00
static int
tcmu_tmr_type(enum tcm_tmreq_table tmf)
{
switch (tmf) {
case TMR_ABORT_TASK: return TCMU_TMR_ABORT_TASK;
case TMR_ABORT_TASK_SET: return TCMU_TMR_ABORT_TASK_SET;
case TMR_CLEAR_ACA: return TCMU_TMR_CLEAR_ACA;
case TMR_CLEAR_TASK_SET: return TCMU_TMR_CLEAR_TASK_SET;
case TMR_LUN_RESET: return TCMU_TMR_LUN_RESET;
case TMR_TARGET_WARM_RESET: return TCMU_TMR_TARGET_WARM_RESET;
case TMR_TARGET_COLD_RESET: return TCMU_TMR_TARGET_COLD_RESET;
case TMR_LUN_RESET_PRO: return TCMU_TMR_LUN_RESET_PRO;
default: return TCMU_TMR_UNKNOWN;
}
}
static void
tcmu_tmr_notify(struct se_device *se_dev, enum tcm_tmreq_table tmf,
struct list_head *cmd_list)
{
int i = 0, cmd_cnt = 0;
bool unqueued = false;
uint16_t *cmd_ids = NULL;
struct tcmu_cmd *cmd;
struct se_cmd *se_cmd;
struct tcmu_tmr *tmr;
struct tcmu_dev *udev = TCMU_DEV(se_dev);
mutex_lock(&udev->cmdr_lock);
/* First we check for aborted commands in qfull_queue */
list_for_each_entry(se_cmd, cmd_list, state_list) {
i++;
if (!se_cmd->priv)
continue;
cmd = se_cmd->priv;
/* Commands on qfull queue have no id yet */
if (cmd->cmd_id) {
cmd_cnt++;
continue;
}
pr_debug("Removing aborted command %p from queue on dev %s.\n",
cmd, udev->name);
list_del_init(&cmd->queue_entry);
tcmu_free_cmd(cmd);
scsi: target: tcmu: Fix use-after-free of se_cmd->priv Commit a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") modified tcmu_free_cmd() to set NULL to priv pointer in se_cmd. However, se_cmd can be already freed by work queue triggered in target_complete_cmd(). This caused BUG KASAN use-after-free [1]. To fix the bug, do not touch priv pointer in tcmu_free_cmd(). Instead, set NULL to priv pointer before target_complete_cmd() calls. Also, to avoid unnecessary priv pointer change in tcmu_queue_cmd(), modify priv pointer in the function only when tcmu_free_cmd() is not called. [1] BUG: KASAN: use-after-free in tcmu_handle_completions+0x1172/0x1770 [target_core_user] Write of size 8 at addr ffff88814cf79a40 by task cmdproc-uio0/14842 CPU: 2 PID: 14842 Comm: cmdproc-uio0 Not tainted 5.11.0-rc2 #1 Hardware name: Supermicro Super Server/X10SRL-F, BIOS 3.2 11/22/2019 Call Trace: dump_stack+0x9a/0xcc ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] print_address_description.constprop.0+0x18/0x130 ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] kasan_report.cold+0x7f/0x10e ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? queue_tmr_ring+0x5d0/0x5d0 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 ? uio_vma_fault+0x460/0x460 ? security_file_permission+0x4f/0x440 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 ? __ia32_sys_read+0xb0/0xb0 ? syscall_enter_from_user_mode+0x27/0x70 ? trace_hardirqs_on+0x1c/0x110 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fcf8b61905f Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 0c fd ff ff 48 RSP: 002b:00007fcf7b3e6c30 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcf8b61905f RDX: 0000000000000004 RSI: 00007fcf7b3e6c78 RDI: 000000000000000c RBP: 00007fcf7b3e6c80 R08: 0000000000000000 R09: 00007fcf7b3e6aa8 R10: 000000000b01c000 R11: 0000000000000293 R12: 00007ffe0c32a52e R13: 00007ffe0c32a52f R14: 0000000000000000 R15: 00007fcf7b3e7640 Allocated by task 383: kasan_save_stack+0x1b/0x40 ____kasan_kmalloc.constprop.0+0x84/0xa0 kmem_cache_alloc+0x142/0x330 tcm_loop_queuecommand+0x2a/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Freed by task 11655: kasan_save_stack+0x1b/0x40 kasan_set_track+0x1c/0x30 kasan_set_free_info+0x20/0x30 ____kasan_slab_free+0xec/0x120 slab_free_freelist_hook+0x53/0x160 kmem_cache_free+0xf4/0x5c0 target_release_cmd_kref+0x3ea/0x9e0 [target_core_mod] transport_generic_free_cmd+0x28b/0x2f0 [target_core_mod] target_complete_ok_work+0x250/0xac0 [target_core_mod] process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcmu_handle_completions+0xad0/0x1770 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Second to last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcm_loop_queuecommand+0x1c3/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 The buggy address belongs to the object at ffff88814cf79800 which belongs to the cache tcm_loop_cmd_cache of size 896. Link: https://lore.kernel.org/r/20210113024508.1264992-1-shinichiro.kawasaki@wdc.com Fixes: a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") Cc: stable@vger.kernel.org # v5.9+ Acked-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-01-13 05:45:08 +03:00
se_cmd->priv = NULL;
2020-07-26 18:35:09 +03:00
target_complete_cmd(se_cmd, SAM_STAT_TASK_ABORTED);
unqueued = true;
}
if (unqueued)
tcmu_set_next_deadline(&udev->qfull_queue, &udev->qfull_timer);
if (!test_bit(TCMU_DEV_BIT_TMR_NOTIFY, &udev->flags))
goto unlock;
2020-07-26 18:35:09 +03:00
pr_debug("TMR event %d on dev %s, aborted cmds %d, afflicted cmd_ids %d\n",
tcmu_tmr_type(tmf), udev->name, i, cmd_cnt);
tmr = kmalloc(sizeof(*tmr) + cmd_cnt * sizeof(*cmd_ids), GFP_NOIO);
2020-07-26 18:35:09 +03:00
if (!tmr)
goto unlock;
tmr->tmr_type = tcmu_tmr_type(tmf);
tmr->tmr_cmd_cnt = cmd_cnt;
if (cmd_cnt != 0) {
cmd_cnt = 0;
list_for_each_entry(se_cmd, cmd_list, state_list) {
if (!se_cmd->priv)
continue;
cmd = se_cmd->priv;
if (cmd->cmd_id)
tmr->tmr_cmd_ids[cmd_cnt++] = cmd->cmd_id;
}
}
queue_tmr_ring(udev, tmr);
unlock:
mutex_unlock(&udev->cmdr_lock);
}
static bool tcmu_handle_completion(struct tcmu_cmd *cmd,
struct tcmu_cmd_entry *entry, bool keep_buf)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
struct se_cmd *se_cmd = cmd->se_cmd;
struct tcmu_dev *udev = cmd->tcmu_dev;
scsi: target: tcmu: add read length support Generally target core and TCMUser seem to work fine for tape devices and media changers. But there is at least one situation where TCMUser is not able to support sequential access device emulation correctly. The situation is when an initiator sends a SCSI READ CDB with a length that is greater than the length of the tape block to read. We can distinguish two subcases: A) The initiator sent the READ CDB with the SILI bit being set. In this case the sequential access device has to transfer the data from the tape block (only the length of the tape block) and transmit a good status. The current interface between TCMUser and the userspace does not support reduction of the read data size by the userspace program. The patch below fixes this subcase by allowing the userspace program to specify a reduced data size in read direction. B) The initiator sent the READ CDB with the SILI bit not being set. In this case the sequential access device has to transfer the data from the tape block as in A), but additionally has to transmit CHECK CONDITION with the ILI bit set and NO SENSE in the sensebytes. The information field in the sensebytes must contain the residual count. With the below patch a user space program can specify the real read data length and appropriate sensebytes. TCMUser then uses the se_cmd flag SCF_TREAT_READ_AS_NORMAL, to force target core to transmit the real data size and the sensebytes. Note: the flag SCF_TREAT_READ_AS_NORMAL is introduced by Lee Duncan's patch "[PATCH v4] target: transport should handle st FM/EOM/ILI reads" from Tue, 15 May 2018 18:25:24 -0700. Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Acked-by: Mike Christie <mchristi@redhat.com> Reviewed-by: Lee Duncan <lduncan@suse.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2018-05-24 19:49:41 +03:00
bool read_len_valid = false;
bool ret = true;
uint32_t read_len;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
/*
* cmd has been completed already from timeout, just reclaim
* data area space and free cmd
*/
if (test_bit(TCMU_CMD_BIT_EXPIRED, &cmd->flags)) {
WARN_ON_ONCE(se_cmd);
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
goto out;
}
if (test_bit(TCMU_CMD_BIT_KEEP_BUF, &cmd->flags)) {
pr_err("cmd_id %u already completed with KEEP_BUF, ring is broken\n",
entry->hdr.cmd_id);
set_bit(TCMU_DEV_BIT_BROKEN, &udev->flags);
ret = false;
goto out;
}
list_del_init(&cmd->queue_entry);
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
tcmu_cmd_reset_dbi_cur(cmd);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
target: Version 2 of TCMU ABI The initial version of TCMU (in 3.18) does not properly handle bidirectional SCSI commands -- those with both an in and out buffer. In looking to fix this it also became clear that TCMU's support for adding new types of entries (opcodes) to the command ring was broken. We need to fix this now, so that future issues can be handled properly by adding new opcodes. We make the most of this ABI break by enabling bidi cmd handling within TCMP_OP_CMD opcode. Add an iov_bidi_cnt field to tcmu_cmd_entry.req. This enables TCMU to describe bidi commands, but further kernel work is needed for full bidi support. Enlarge tcmu_cmd_entry_hdr by 32 bits by pulling in cmd_id and __pad1. Turn __pad1 into two 8 bit flags fields, for kernel-set and userspace-set flags, "kflags" and "uflags" respectively. Update version fields so userspace can tell the interface is changed. Update tcmu-design.txt with details of how new stuff works: - Specify an additional requirement for userspace to set UNKNOWN_OP (bit 0) in hdr.uflags for unknown/unhandled opcodes. - Define how Data-In and Data-Out fields are described in req.iov[] Changed in v2: - Change name of SKIPPED bit to UNKNOWN bit - PAD op does not set the bit any more - Change len_op helper functions to take just len_op, not the whole struct - Change version to 2 in missed spots, and use defines - Add 16 unused bytes to cmd_entry.req, in case additional SAM cmd parameters need to be included - Add iov_dif_cnt field to specify buffers used for DIF info in iov[] - Rearrange fields to naturally align cdb_off - Handle if userspace sets UNKNOWN_OP by indicating failure of the cmd - Wrap some overly long UPDATE_HEAD lines (Add missing req.iov_bidi_cnt + req.iov_dif_cnt zeroing - Ilias) Signed-off-by: Andy Grover <agrover@redhat.com> Reviewed-by: Ilias Tsitsimpis <iliastsi@arrikto.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2015-04-15 03:30:04 +03:00
if (entry->hdr.uflags & TCMU_UFLAG_UNKNOWN_OP) {
pr_warn("TCMU: Userspace set UNKNOWN_OP flag on se_cmd %p\n",
cmd->se_cmd);
entry->rsp.scsi_status = SAM_STAT_CHECK_CONDITION;
scsi: target: tcmu: add read length support Generally target core and TCMUser seem to work fine for tape devices and media changers. But there is at least one situation where TCMUser is not able to support sequential access device emulation correctly. The situation is when an initiator sends a SCSI READ CDB with a length that is greater than the length of the tape block to read. We can distinguish two subcases: A) The initiator sent the READ CDB with the SILI bit being set. In this case the sequential access device has to transfer the data from the tape block (only the length of the tape block) and transmit a good status. The current interface between TCMUser and the userspace does not support reduction of the read data size by the userspace program. The patch below fixes this subcase by allowing the userspace program to specify a reduced data size in read direction. B) The initiator sent the READ CDB with the SILI bit not being set. In this case the sequential access device has to transfer the data from the tape block as in A), but additionally has to transmit CHECK CONDITION with the ILI bit set and NO SENSE in the sensebytes. The information field in the sensebytes must contain the residual count. With the below patch a user space program can specify the real read data length and appropriate sensebytes. TCMUser then uses the se_cmd flag SCF_TREAT_READ_AS_NORMAL, to force target core to transmit the real data size and the sensebytes. Note: the flag SCF_TREAT_READ_AS_NORMAL is introduced by Lee Duncan's patch "[PATCH v4] target: transport should handle st FM/EOM/ILI reads" from Tue, 15 May 2018 18:25:24 -0700. Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Acked-by: Mike Christie <mchristi@redhat.com> Reviewed-by: Lee Duncan <lduncan@suse.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2018-05-24 19:49:41 +03:00
goto done;
}
read_len = se_cmd->data_length;
scsi: target: tcmu: add read length support Generally target core and TCMUser seem to work fine for tape devices and media changers. But there is at least one situation where TCMUser is not able to support sequential access device emulation correctly. The situation is when an initiator sends a SCSI READ CDB with a length that is greater than the length of the tape block to read. We can distinguish two subcases: A) The initiator sent the READ CDB with the SILI bit being set. In this case the sequential access device has to transfer the data from the tape block (only the length of the tape block) and transmit a good status. The current interface between TCMUser and the userspace does not support reduction of the read data size by the userspace program. The patch below fixes this subcase by allowing the userspace program to specify a reduced data size in read direction. B) The initiator sent the READ CDB with the SILI bit not being set. In this case the sequential access device has to transfer the data from the tape block as in A), but additionally has to transmit CHECK CONDITION with the ILI bit set and NO SENSE in the sensebytes. The information field in the sensebytes must contain the residual count. With the below patch a user space program can specify the real read data length and appropriate sensebytes. TCMUser then uses the se_cmd flag SCF_TREAT_READ_AS_NORMAL, to force target core to transmit the real data size and the sensebytes. Note: the flag SCF_TREAT_READ_AS_NORMAL is introduced by Lee Duncan's patch "[PATCH v4] target: transport should handle st FM/EOM/ILI reads" from Tue, 15 May 2018 18:25:24 -0700. Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Acked-by: Mike Christie <mchristi@redhat.com> Reviewed-by: Lee Duncan <lduncan@suse.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2018-05-24 19:49:41 +03:00
if (se_cmd->data_direction == DMA_FROM_DEVICE &&
(entry->hdr.uflags & TCMU_UFLAG_READ_LEN) && entry->rsp.read_len) {
read_len_valid = true;
if (entry->rsp.read_len < read_len)
read_len = entry->rsp.read_len;
}
if (entry->rsp.scsi_status == SAM_STAT_CHECK_CONDITION) {
transport_copy_sense_to_cmd(se_cmd, entry->rsp.sense_buffer);
scsi: target: tcmu: add read length support Generally target core and TCMUser seem to work fine for tape devices and media changers. But there is at least one situation where TCMUser is not able to support sequential access device emulation correctly. The situation is when an initiator sends a SCSI READ CDB with a length that is greater than the length of the tape block to read. We can distinguish two subcases: A) The initiator sent the READ CDB with the SILI bit being set. In this case the sequential access device has to transfer the data from the tape block (only the length of the tape block) and transmit a good status. The current interface between TCMUser and the userspace does not support reduction of the read data size by the userspace program. The patch below fixes this subcase by allowing the userspace program to specify a reduced data size in read direction. B) The initiator sent the READ CDB with the SILI bit not being set. In this case the sequential access device has to transfer the data from the tape block as in A), but additionally has to transmit CHECK CONDITION with the ILI bit set and NO SENSE in the sensebytes. The information field in the sensebytes must contain the residual count. With the below patch a user space program can specify the real read data length and appropriate sensebytes. TCMUser then uses the se_cmd flag SCF_TREAT_READ_AS_NORMAL, to force target core to transmit the real data size and the sensebytes. Note: the flag SCF_TREAT_READ_AS_NORMAL is introduced by Lee Duncan's patch "[PATCH v4] target: transport should handle st FM/EOM/ILI reads" from Tue, 15 May 2018 18:25:24 -0700. Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Acked-by: Mike Christie <mchristi@redhat.com> Reviewed-by: Lee Duncan <lduncan@suse.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2018-05-24 19:49:41 +03:00
if (!read_len_valid )
goto done;
else
se_cmd->se_cmd_flags |= SCF_TREAT_READ_AS_NORMAL;
}
if (se_cmd->se_cmd_flags & SCF_BIDI) {
/* Get Data-In buffer before clean up */
scsi: target: tcmu: add read length support Generally target core and TCMUser seem to work fine for tape devices and media changers. But there is at least one situation where TCMUser is not able to support sequential access device emulation correctly. The situation is when an initiator sends a SCSI READ CDB with a length that is greater than the length of the tape block to read. We can distinguish two subcases: A) The initiator sent the READ CDB with the SILI bit being set. In this case the sequential access device has to transfer the data from the tape block (only the length of the tape block) and transmit a good status. The current interface between TCMUser and the userspace does not support reduction of the read data size by the userspace program. The patch below fixes this subcase by allowing the userspace program to specify a reduced data size in read direction. B) The initiator sent the READ CDB with the SILI bit not being set. In this case the sequential access device has to transfer the data from the tape block as in A), but additionally has to transmit CHECK CONDITION with the ILI bit set and NO SENSE in the sensebytes. The information field in the sensebytes must contain the residual count. With the below patch a user space program can specify the real read data length and appropriate sensebytes. TCMUser then uses the se_cmd flag SCF_TREAT_READ_AS_NORMAL, to force target core to transmit the real data size and the sensebytes. Note: the flag SCF_TREAT_READ_AS_NORMAL is introduced by Lee Duncan's patch "[PATCH v4] target: transport should handle st FM/EOM/ILI reads" from Tue, 15 May 2018 18:25:24 -0700. Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Acked-by: Mike Christie <mchristi@redhat.com> Reviewed-by: Lee Duncan <lduncan@suse.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2018-05-24 19:49:41 +03:00
gather_data_area(udev, cmd, true, read_len);
} else if (se_cmd->data_direction == DMA_FROM_DEVICE) {
scsi: target: tcmu: add read length support Generally target core and TCMUser seem to work fine for tape devices and media changers. But there is at least one situation where TCMUser is not able to support sequential access device emulation correctly. The situation is when an initiator sends a SCSI READ CDB with a length that is greater than the length of the tape block to read. We can distinguish two subcases: A) The initiator sent the READ CDB with the SILI bit being set. In this case the sequential access device has to transfer the data from the tape block (only the length of the tape block) and transmit a good status. The current interface between TCMUser and the userspace does not support reduction of the read data size by the userspace program. The patch below fixes this subcase by allowing the userspace program to specify a reduced data size in read direction. B) The initiator sent the READ CDB with the SILI bit not being set. In this case the sequential access device has to transfer the data from the tape block as in A), but additionally has to transmit CHECK CONDITION with the ILI bit set and NO SENSE in the sensebytes. The information field in the sensebytes must contain the residual count. With the below patch a user space program can specify the real read data length and appropriate sensebytes. TCMUser then uses the se_cmd flag SCF_TREAT_READ_AS_NORMAL, to force target core to transmit the real data size and the sensebytes. Note: the flag SCF_TREAT_READ_AS_NORMAL is introduced by Lee Duncan's patch "[PATCH v4] target: transport should handle st FM/EOM/ILI reads" from Tue, 15 May 2018 18:25:24 -0700. Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Acked-by: Mike Christie <mchristi@redhat.com> Reviewed-by: Lee Duncan <lduncan@suse.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2018-05-24 19:49:41 +03:00
gather_data_area(udev, cmd, false, read_len);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
} else if (se_cmd->data_direction == DMA_TO_DEVICE) {
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
/* TODO: */
} else if (se_cmd->data_direction != DMA_NONE) {
pr_warn("TCMU: data direction was %d!\n",
se_cmd->data_direction);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
scsi: target: tcmu: add read length support Generally target core and TCMUser seem to work fine for tape devices and media changers. But there is at least one situation where TCMUser is not able to support sequential access device emulation correctly. The situation is when an initiator sends a SCSI READ CDB with a length that is greater than the length of the tape block to read. We can distinguish two subcases: A) The initiator sent the READ CDB with the SILI bit being set. In this case the sequential access device has to transfer the data from the tape block (only the length of the tape block) and transmit a good status. The current interface between TCMUser and the userspace does not support reduction of the read data size by the userspace program. The patch below fixes this subcase by allowing the userspace program to specify a reduced data size in read direction. B) The initiator sent the READ CDB with the SILI bit not being set. In this case the sequential access device has to transfer the data from the tape block as in A), but additionally has to transmit CHECK CONDITION with the ILI bit set and NO SENSE in the sensebytes. The information field in the sensebytes must contain the residual count. With the below patch a user space program can specify the real read data length and appropriate sensebytes. TCMUser then uses the se_cmd flag SCF_TREAT_READ_AS_NORMAL, to force target core to transmit the real data size and the sensebytes. Note: the flag SCF_TREAT_READ_AS_NORMAL is introduced by Lee Duncan's patch "[PATCH v4] target: transport should handle st FM/EOM/ILI reads" from Tue, 15 May 2018 18:25:24 -0700. Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Acked-by: Mike Christie <mchristi@redhat.com> Reviewed-by: Lee Duncan <lduncan@suse.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2018-05-24 19:49:41 +03:00
done:
scsi: target: tcmu: Fix use-after-free of se_cmd->priv Commit a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") modified tcmu_free_cmd() to set NULL to priv pointer in se_cmd. However, se_cmd can be already freed by work queue triggered in target_complete_cmd(). This caused BUG KASAN use-after-free [1]. To fix the bug, do not touch priv pointer in tcmu_free_cmd(). Instead, set NULL to priv pointer before target_complete_cmd() calls. Also, to avoid unnecessary priv pointer change in tcmu_queue_cmd(), modify priv pointer in the function only when tcmu_free_cmd() is not called. [1] BUG: KASAN: use-after-free in tcmu_handle_completions+0x1172/0x1770 [target_core_user] Write of size 8 at addr ffff88814cf79a40 by task cmdproc-uio0/14842 CPU: 2 PID: 14842 Comm: cmdproc-uio0 Not tainted 5.11.0-rc2 #1 Hardware name: Supermicro Super Server/X10SRL-F, BIOS 3.2 11/22/2019 Call Trace: dump_stack+0x9a/0xcc ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] print_address_description.constprop.0+0x18/0x130 ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] kasan_report.cold+0x7f/0x10e ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? queue_tmr_ring+0x5d0/0x5d0 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 ? uio_vma_fault+0x460/0x460 ? security_file_permission+0x4f/0x440 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 ? __ia32_sys_read+0xb0/0xb0 ? syscall_enter_from_user_mode+0x27/0x70 ? trace_hardirqs_on+0x1c/0x110 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fcf8b61905f Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 0c fd ff ff 48 RSP: 002b:00007fcf7b3e6c30 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcf8b61905f RDX: 0000000000000004 RSI: 00007fcf7b3e6c78 RDI: 000000000000000c RBP: 00007fcf7b3e6c80 R08: 0000000000000000 R09: 00007fcf7b3e6aa8 R10: 000000000b01c000 R11: 0000000000000293 R12: 00007ffe0c32a52e R13: 00007ffe0c32a52f R14: 0000000000000000 R15: 00007fcf7b3e7640 Allocated by task 383: kasan_save_stack+0x1b/0x40 ____kasan_kmalloc.constprop.0+0x84/0xa0 kmem_cache_alloc+0x142/0x330 tcm_loop_queuecommand+0x2a/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Freed by task 11655: kasan_save_stack+0x1b/0x40 kasan_set_track+0x1c/0x30 kasan_set_free_info+0x20/0x30 ____kasan_slab_free+0xec/0x120 slab_free_freelist_hook+0x53/0x160 kmem_cache_free+0xf4/0x5c0 target_release_cmd_kref+0x3ea/0x9e0 [target_core_mod] transport_generic_free_cmd+0x28b/0x2f0 [target_core_mod] target_complete_ok_work+0x250/0xac0 [target_core_mod] process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcmu_handle_completions+0xad0/0x1770 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Second to last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcm_loop_queuecommand+0x1c3/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 The buggy address belongs to the object at ffff88814cf79800 which belongs to the cache tcm_loop_cmd_cache of size 896. Link: https://lore.kernel.org/r/20210113024508.1264992-1-shinichiro.kawasaki@wdc.com Fixes: a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") Cc: stable@vger.kernel.org # v5.9+ Acked-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-01-13 05:45:08 +03:00
se_cmd->priv = NULL;
scsi: target: tcmu: add read length support Generally target core and TCMUser seem to work fine for tape devices and media changers. But there is at least one situation where TCMUser is not able to support sequential access device emulation correctly. The situation is when an initiator sends a SCSI READ CDB with a length that is greater than the length of the tape block to read. We can distinguish two subcases: A) The initiator sent the READ CDB with the SILI bit being set. In this case the sequential access device has to transfer the data from the tape block (only the length of the tape block) and transmit a good status. The current interface between TCMUser and the userspace does not support reduction of the read data size by the userspace program. The patch below fixes this subcase by allowing the userspace program to specify a reduced data size in read direction. B) The initiator sent the READ CDB with the SILI bit not being set. In this case the sequential access device has to transfer the data from the tape block as in A), but additionally has to transmit CHECK CONDITION with the ILI bit set and NO SENSE in the sensebytes. The information field in the sensebytes must contain the residual count. With the below patch a user space program can specify the real read data length and appropriate sensebytes. TCMUser then uses the se_cmd flag SCF_TREAT_READ_AS_NORMAL, to force target core to transmit the real data size and the sensebytes. Note: the flag SCF_TREAT_READ_AS_NORMAL is introduced by Lee Duncan's patch "[PATCH v4] target: transport should handle st FM/EOM/ILI reads" from Tue, 15 May 2018 18:25:24 -0700. Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Acked-by: Mike Christie <mchristi@redhat.com> Reviewed-by: Lee Duncan <lduncan@suse.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2018-05-24 19:49:41 +03:00
if (read_len_valid) {
pr_debug("read_len = %d\n", read_len);
target_complete_cmd_with_length(cmd->se_cmd,
entry->rsp.scsi_status, read_len);
} else
target_complete_cmd(cmd->se_cmd, entry->rsp.scsi_status);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
out:
if (!keep_buf) {
tcmu_cmd_free_data(cmd, cmd->dbi_cnt);
tcmu_free_cmd(cmd);
} else {
/*
* Keep this command after completion, since userspace still
* needs the data buffer. Mark it with TCMU_CMD_BIT_KEEP_BUF
* and reset potential TCMU_CMD_BIT_EXPIRED, so we don't accept
* a second completion later.
* Userspace can free the buffer later by writing the cmd_id
* to new action attribute free_kept_buf.
*/
clear_bit(TCMU_CMD_BIT_EXPIRED, &cmd->flags);
set_bit(TCMU_CMD_BIT_KEEP_BUF, &cmd->flags);
}
return ret;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
2020-07-26 18:35:09 +03:00
static int tcmu_run_tmr_queue(struct tcmu_dev *udev)
{
struct tcmu_tmr *tmr, *tmp;
LIST_HEAD(tmrs);
if (list_empty(&udev->tmr_queue))
return 1;
pr_debug("running %s's tmr queue\n", udev->name);
list_splice_init(&udev->tmr_queue, &tmrs);
list_for_each_entry_safe(tmr, tmp, &tmrs, queue_entry) {
list_del_init(&tmr->queue_entry);
pr_debug("removing tmr %p on dev %s from queue\n",
tmr, udev->name);
if (queue_tmr_ring(udev, tmr)) {
pr_debug("ran out of space during tmr queue run\n");
/*
* tmr was requeued, so just put all tmrs back in
* the queue
*/
list_splice_tail(&tmrs, &udev->tmr_queue);
return 0;
}
}
return 1;
}
static bool tcmu_handle_completions(struct tcmu_dev *udev)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
struct tcmu_mailbox *mb;
struct tcmu_cmd *cmd;
2020-07-26 18:35:09 +03:00
bool free_space = false;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
if (test_bit(TCMU_DEV_BIT_BROKEN, &udev->flags)) {
pr_err("ring broken, not handling completions\n");
return false;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
mb = udev->mb_addr;
tcmu_flush_dcache_range(mb, sizeof(*mb));
locking/atomics: COCCINELLE/treewide: Convert trivial ACCESS_ONCE() patterns to READ_ONCE()/WRITE_ONCE() Please do not apply this to mainline directly, instead please re-run the coccinelle script shown below and apply its output. For several reasons, it is desirable to use {READ,WRITE}_ONCE() in preference to ACCESS_ONCE(), and new code is expected to use one of the former. So far, there's been no reason to change most existing uses of ACCESS_ONCE(), as these aren't harmful, and changing them results in churn. However, for some features, the read/write distinction is critical to correct operation. To distinguish these cases, separate read/write accessors must be used. This patch migrates (most) remaining ACCESS_ONCE() instances to {READ,WRITE}_ONCE(), using the following coccinelle script: ---- // Convert trivial ACCESS_ONCE() uses to equivalent READ_ONCE() and // WRITE_ONCE() // $ make coccicheck COCCI=/home/mark/once.cocci SPFLAGS="--include-headers" MODE=patch virtual patch @ depends on patch @ expression E1, E2; @@ - ACCESS_ONCE(E1) = E2 + WRITE_ONCE(E1, E2) @ depends on patch @ expression E; @@ - ACCESS_ONCE(E) + READ_ONCE(E) ---- Signed-off-by: Mark Rutland <mark.rutland@arm.com> Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com> Cc: Linus Torvalds <torvalds@linux-foundation.org> Cc: Peter Zijlstra <peterz@infradead.org> Cc: Thomas Gleixner <tglx@linutronix.de> Cc: davem@davemloft.net Cc: linux-arch@vger.kernel.org Cc: mpe@ellerman.id.au Cc: shuah@kernel.org Cc: snitzer@redhat.com Cc: thor.thayer@linux.intel.com Cc: tj@kernel.org Cc: viro@zeniv.linux.org.uk Cc: will.deacon@arm.com Link: http://lkml.kernel.org/r/1508792849-3115-19-git-send-email-paulmck@linux.vnet.ibm.com Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-24 00:07:29 +03:00
while (udev->cmdr_last_cleaned != READ_ONCE(mb->cmd_tail)) {
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
struct tcmu_cmd_entry *entry = udev->cmdr + udev->cmdr_last_cleaned;
bool keep_buf;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
/*
* Flush max. up to end of cmd ring since current entry might
* be a padding that is shorter than sizeof(*entry)
*/
size_t ring_left = head_to_end(udev->cmdr_last_cleaned,
udev->cmdr_size);
tcmu_flush_dcache_range(entry, ring_left < sizeof(*entry) ?
ring_left : sizeof(*entry));
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
2020-07-26 18:35:09 +03:00
free_space = true;
if (tcmu_hdr_get_op(entry->hdr.len_op) == TCMU_OP_PAD ||
tcmu_hdr_get_op(entry->hdr.len_op) == TCMU_OP_TMR) {
target: Version 2 of TCMU ABI The initial version of TCMU (in 3.18) does not properly handle bidirectional SCSI commands -- those with both an in and out buffer. In looking to fix this it also became clear that TCMU's support for adding new types of entries (opcodes) to the command ring was broken. We need to fix this now, so that future issues can be handled properly by adding new opcodes. We make the most of this ABI break by enabling bidi cmd handling within TCMP_OP_CMD opcode. Add an iov_bidi_cnt field to tcmu_cmd_entry.req. This enables TCMU to describe bidi commands, but further kernel work is needed for full bidi support. Enlarge tcmu_cmd_entry_hdr by 32 bits by pulling in cmd_id and __pad1. Turn __pad1 into two 8 bit flags fields, for kernel-set and userspace-set flags, "kflags" and "uflags" respectively. Update version fields so userspace can tell the interface is changed. Update tcmu-design.txt with details of how new stuff works: - Specify an additional requirement for userspace to set UNKNOWN_OP (bit 0) in hdr.uflags for unknown/unhandled opcodes. - Define how Data-In and Data-Out fields are described in req.iov[] Changed in v2: - Change name of SKIPPED bit to UNKNOWN bit - PAD op does not set the bit any more - Change len_op helper functions to take just len_op, not the whole struct - Change version to 2 in missed spots, and use defines - Add 16 unused bytes to cmd_entry.req, in case additional SAM cmd parameters need to be included - Add iov_dif_cnt field to specify buffers used for DIF info in iov[] - Rearrange fields to naturally align cdb_off - Handle if userspace sets UNKNOWN_OP by indicating failure of the cmd - Wrap some overly long UPDATE_HEAD lines (Add missing req.iov_bidi_cnt + req.iov_dif_cnt zeroing - Ilias) Signed-off-by: Andy Grover <agrover@redhat.com> Reviewed-by: Ilias Tsitsimpis <iliastsi@arrikto.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2015-04-15 03:30:04 +03:00
UPDATE_HEAD(udev->cmdr_last_cleaned,
tcmu_hdr_get_len(entry->hdr.len_op),
udev->cmdr_size);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
continue;
}
target: Version 2 of TCMU ABI The initial version of TCMU (in 3.18) does not properly handle bidirectional SCSI commands -- those with both an in and out buffer. In looking to fix this it also became clear that TCMU's support for adding new types of entries (opcodes) to the command ring was broken. We need to fix this now, so that future issues can be handled properly by adding new opcodes. We make the most of this ABI break by enabling bidi cmd handling within TCMP_OP_CMD opcode. Add an iov_bidi_cnt field to tcmu_cmd_entry.req. This enables TCMU to describe bidi commands, but further kernel work is needed for full bidi support. Enlarge tcmu_cmd_entry_hdr by 32 bits by pulling in cmd_id and __pad1. Turn __pad1 into two 8 bit flags fields, for kernel-set and userspace-set flags, "kflags" and "uflags" respectively. Update version fields so userspace can tell the interface is changed. Update tcmu-design.txt with details of how new stuff works: - Specify an additional requirement for userspace to set UNKNOWN_OP (bit 0) in hdr.uflags for unknown/unhandled opcodes. - Define how Data-In and Data-Out fields are described in req.iov[] Changed in v2: - Change name of SKIPPED bit to UNKNOWN bit - PAD op does not set the bit any more - Change len_op helper functions to take just len_op, not the whole struct - Change version to 2 in missed spots, and use defines - Add 16 unused bytes to cmd_entry.req, in case additional SAM cmd parameters need to be included - Add iov_dif_cnt field to specify buffers used for DIF info in iov[] - Rearrange fields to naturally align cdb_off - Handle if userspace sets UNKNOWN_OP by indicating failure of the cmd - Wrap some overly long UPDATE_HEAD lines (Add missing req.iov_bidi_cnt + req.iov_dif_cnt zeroing - Ilias) Signed-off-by: Andy Grover <agrover@redhat.com> Reviewed-by: Ilias Tsitsimpis <iliastsi@arrikto.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2015-04-15 03:30:04 +03:00
WARN_ON(tcmu_hdr_get_op(entry->hdr.len_op) != TCMU_OP_CMD);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
keep_buf = !!(entry->hdr.uflags & TCMU_UFLAG_KEEP_BUF);
if (keep_buf)
cmd = xa_load(&udev->commands, entry->hdr.cmd_id);
else
cmd = xa_erase(&udev->commands, entry->hdr.cmd_id);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
if (!cmd) {
pr_err("cmd_id %u not found, ring is broken\n",
entry->hdr.cmd_id);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
set_bit(TCMU_DEV_BIT_BROKEN, &udev->flags);
return false;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
if (!tcmu_handle_completion(cmd, entry, keep_buf))
break;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
target: Version 2 of TCMU ABI The initial version of TCMU (in 3.18) does not properly handle bidirectional SCSI commands -- those with both an in and out buffer. In looking to fix this it also became clear that TCMU's support for adding new types of entries (opcodes) to the command ring was broken. We need to fix this now, so that future issues can be handled properly by adding new opcodes. We make the most of this ABI break by enabling bidi cmd handling within TCMP_OP_CMD opcode. Add an iov_bidi_cnt field to tcmu_cmd_entry.req. This enables TCMU to describe bidi commands, but further kernel work is needed for full bidi support. Enlarge tcmu_cmd_entry_hdr by 32 bits by pulling in cmd_id and __pad1. Turn __pad1 into two 8 bit flags fields, for kernel-set and userspace-set flags, "kflags" and "uflags" respectively. Update version fields so userspace can tell the interface is changed. Update tcmu-design.txt with details of how new stuff works: - Specify an additional requirement for userspace to set UNKNOWN_OP (bit 0) in hdr.uflags for unknown/unhandled opcodes. - Define how Data-In and Data-Out fields are described in req.iov[] Changed in v2: - Change name of SKIPPED bit to UNKNOWN bit - PAD op does not set the bit any more - Change len_op helper functions to take just len_op, not the whole struct - Change version to 2 in missed spots, and use defines - Add 16 unused bytes to cmd_entry.req, in case additional SAM cmd parameters need to be included - Add iov_dif_cnt field to specify buffers used for DIF info in iov[] - Rearrange fields to naturally align cdb_off - Handle if userspace sets UNKNOWN_OP by indicating failure of the cmd - Wrap some overly long UPDATE_HEAD lines (Add missing req.iov_bidi_cnt + req.iov_dif_cnt zeroing - Ilias) Signed-off-by: Andy Grover <agrover@redhat.com> Reviewed-by: Ilias Tsitsimpis <iliastsi@arrikto.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2015-04-15 03:30:04 +03:00
UPDATE_HEAD(udev->cmdr_last_cleaned,
tcmu_hdr_get_len(entry->hdr.len_op),
udev->cmdr_size);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
2020-07-26 18:35:09 +03:00
if (free_space)
free_space = tcmu_run_tmr_queue(udev);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
if (atomic_read(&global_page_count) > tcmu_global_max_pages &&
xa_empty(&udev->commands) && list_empty(&udev->qfull_queue)) {
scsi: target: tcmu: Fix and simplify timeout handling During cmd timeout handling in check_timedout_devices(), due to a race, it can happen that tcmu_set_next_deadline() does not start a timer as expected: 1) Either tcmu_check_expired_ring_cmd() checks the inflight_queue or tcmu_check_expired_queue_cmd() checks the qfull_queue while jiffies has the value X 2) At the end of the check the queue contains one remaining command with deadline X (time_after(X, X) is false and thus the command is not handled as being timed out). 3) After tcmu_check_expired_xxxxx_cmd() a timer interrupt happens and jiffies is incremented to X+1. 4) Now tcmu_set_next_deadline() is called, but it skips the command, since time_after(X+1, X) is true. Therefore tcmu_set_next_deadline() finds no new deadline and stops the timer, which it shouldn't. Since commands that time out are removed from inflight_queue or qfull_queue, we don't need the check with time_after() in tcmu_set_next_deadline() but can use the deadline from the first cmd in the queue. Additionally, replace the remaining time_after() calls in tcmu_check_expired_xxxxx_cmd() with time_after_eq(), because it is not useful to set the timeout to deadline but then check for jiffies being greater than deadline. Simplify the end of tcmu_handle_completions() and change the check for no more pending commands from mb->cmd_tail == mb->cmd_head to idr_is_empty(&udev->commands) because the old check doesn't work correctly if paddings or in the future TMRs are in the ring. Finally tcmu_set_next_deadline() was shifted in the source as preparation for later implementation of tmr_notify callback. Link: https://lore.kernel.org/r/20200726153510.13077-7-bstroesser@ts.fujitsu.com Reviewed-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-07-26 18:35:08 +03:00
/*
* Allocated blocks exceeded global block limit, currently no
* more pending or waiting commands so try to reclaim blocks.
*/
schedule_delayed_work(&tcmu_unmap_work, 0);
}
scsi: target: tcmu: Fix and simplify timeout handling During cmd timeout handling in check_timedout_devices(), due to a race, it can happen that tcmu_set_next_deadline() does not start a timer as expected: 1) Either tcmu_check_expired_ring_cmd() checks the inflight_queue or tcmu_check_expired_queue_cmd() checks the qfull_queue while jiffies has the value X 2) At the end of the check the queue contains one remaining command with deadline X (time_after(X, X) is false and thus the command is not handled as being timed out). 3) After tcmu_check_expired_xxxxx_cmd() a timer interrupt happens and jiffies is incremented to X+1. 4) Now tcmu_set_next_deadline() is called, but it skips the command, since time_after(X+1, X) is true. Therefore tcmu_set_next_deadline() finds no new deadline and stops the timer, which it shouldn't. Since commands that time out are removed from inflight_queue or qfull_queue, we don't need the check with time_after() in tcmu_set_next_deadline() but can use the deadline from the first cmd in the queue. Additionally, replace the remaining time_after() calls in tcmu_check_expired_xxxxx_cmd() with time_after_eq(), because it is not useful to set the timeout to deadline but then check for jiffies being greater than deadline. Simplify the end of tcmu_handle_completions() and change the check for no more pending commands from mb->cmd_tail == mb->cmd_head to idr_is_empty(&udev->commands) because the old check doesn't work correctly if paddings or in the future TMRs are in the ring. Finally tcmu_set_next_deadline() was shifted in the source as preparation for later implementation of tmr_notify callback. Link: https://lore.kernel.org/r/20200726153510.13077-7-bstroesser@ts.fujitsu.com Reviewed-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-07-26 18:35:08 +03:00
if (udev->cmd_time_out)
tcmu_set_next_deadline(&udev->inflight_queue, &udev->cmd_timer);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
2020-07-26 18:35:09 +03:00
return free_space;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
static void tcmu_check_expired_ring_cmd(struct tcmu_cmd *cmd)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
struct se_cmd *se_cmd;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
scsi: target: tcmu: Fix and simplify timeout handling During cmd timeout handling in check_timedout_devices(), due to a race, it can happen that tcmu_set_next_deadline() does not start a timer as expected: 1) Either tcmu_check_expired_ring_cmd() checks the inflight_queue or tcmu_check_expired_queue_cmd() checks the qfull_queue while jiffies has the value X 2) At the end of the check the queue contains one remaining command with deadline X (time_after(X, X) is false and thus the command is not handled as being timed out). 3) After tcmu_check_expired_xxxxx_cmd() a timer interrupt happens and jiffies is incremented to X+1. 4) Now tcmu_set_next_deadline() is called, but it skips the command, since time_after(X+1, X) is true. Therefore tcmu_set_next_deadline() finds no new deadline and stops the timer, which it shouldn't. Since commands that time out are removed from inflight_queue or qfull_queue, we don't need the check with time_after() in tcmu_set_next_deadline() but can use the deadline from the first cmd in the queue. Additionally, replace the remaining time_after() calls in tcmu_check_expired_xxxxx_cmd() with time_after_eq(), because it is not useful to set the timeout to deadline but then check for jiffies being greater than deadline. Simplify the end of tcmu_handle_completions() and change the check for no more pending commands from mb->cmd_tail == mb->cmd_head to idr_is_empty(&udev->commands) because the old check doesn't work correctly if paddings or in the future TMRs are in the ring. Finally tcmu_set_next_deadline() was shifted in the source as preparation for later implementation of tmr_notify callback. Link: https://lore.kernel.org/r/20200726153510.13077-7-bstroesser@ts.fujitsu.com Reviewed-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-07-26 18:35:08 +03:00
if (!time_after_eq(jiffies, cmd->deadline))
return;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
set_bit(TCMU_CMD_BIT_EXPIRED, &cmd->flags);
list_del_init(&cmd->queue_entry);
se_cmd = cmd->se_cmd;
se_cmd->priv = NULL;
cmd->se_cmd = NULL;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
pr_debug("Timing out inflight cmd %u on dev %s.\n",
cmd->cmd_id, cmd->tcmu_dev->name);
target_complete_cmd(se_cmd, SAM_STAT_CHECK_CONDITION);
}
static void tcmu_check_expired_queue_cmd(struct tcmu_cmd *cmd)
{
struct se_cmd *se_cmd;
scsi: target: tcmu: Fix and simplify timeout handling During cmd timeout handling in check_timedout_devices(), due to a race, it can happen that tcmu_set_next_deadline() does not start a timer as expected: 1) Either tcmu_check_expired_ring_cmd() checks the inflight_queue or tcmu_check_expired_queue_cmd() checks the qfull_queue while jiffies has the value X 2) At the end of the check the queue contains one remaining command with deadline X (time_after(X, X) is false and thus the command is not handled as being timed out). 3) After tcmu_check_expired_xxxxx_cmd() a timer interrupt happens and jiffies is incremented to X+1. 4) Now tcmu_set_next_deadline() is called, but it skips the command, since time_after(X+1, X) is true. Therefore tcmu_set_next_deadline() finds no new deadline and stops the timer, which it shouldn't. Since commands that time out are removed from inflight_queue or qfull_queue, we don't need the check with time_after() in tcmu_set_next_deadline() but can use the deadline from the first cmd in the queue. Additionally, replace the remaining time_after() calls in tcmu_check_expired_xxxxx_cmd() with time_after_eq(), because it is not useful to set the timeout to deadline but then check for jiffies being greater than deadline. Simplify the end of tcmu_handle_completions() and change the check for no more pending commands from mb->cmd_tail == mb->cmd_head to idr_is_empty(&udev->commands) because the old check doesn't work correctly if paddings or in the future TMRs are in the ring. Finally tcmu_set_next_deadline() was shifted in the source as preparation for later implementation of tmr_notify callback. Link: https://lore.kernel.org/r/20200726153510.13077-7-bstroesser@ts.fujitsu.com Reviewed-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bstroesser@ts.fujitsu.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2020-07-26 18:35:08 +03:00
if (!time_after_eq(jiffies, cmd->deadline))
return;
pr_debug("Timing out queued cmd %p on dev %s.\n",
cmd, cmd->tcmu_dev->name);
list_del_init(&cmd->queue_entry);
se_cmd = cmd->se_cmd;
tcmu_free_cmd(cmd);
scsi: target: tcmu: Fix use-after-free of se_cmd->priv Commit a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") modified tcmu_free_cmd() to set NULL to priv pointer in se_cmd. However, se_cmd can be already freed by work queue triggered in target_complete_cmd(). This caused BUG KASAN use-after-free [1]. To fix the bug, do not touch priv pointer in tcmu_free_cmd(). Instead, set NULL to priv pointer before target_complete_cmd() calls. Also, to avoid unnecessary priv pointer change in tcmu_queue_cmd(), modify priv pointer in the function only when tcmu_free_cmd() is not called. [1] BUG: KASAN: use-after-free in tcmu_handle_completions+0x1172/0x1770 [target_core_user] Write of size 8 at addr ffff88814cf79a40 by task cmdproc-uio0/14842 CPU: 2 PID: 14842 Comm: cmdproc-uio0 Not tainted 5.11.0-rc2 #1 Hardware name: Supermicro Super Server/X10SRL-F, BIOS 3.2 11/22/2019 Call Trace: dump_stack+0x9a/0xcc ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] print_address_description.constprop.0+0x18/0x130 ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] kasan_report.cold+0x7f/0x10e ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? queue_tmr_ring+0x5d0/0x5d0 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 ? uio_vma_fault+0x460/0x460 ? security_file_permission+0x4f/0x440 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 ? __ia32_sys_read+0xb0/0xb0 ? syscall_enter_from_user_mode+0x27/0x70 ? trace_hardirqs_on+0x1c/0x110 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fcf8b61905f Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 0c fd ff ff 48 RSP: 002b:00007fcf7b3e6c30 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcf8b61905f RDX: 0000000000000004 RSI: 00007fcf7b3e6c78 RDI: 000000000000000c RBP: 00007fcf7b3e6c80 R08: 0000000000000000 R09: 00007fcf7b3e6aa8 R10: 000000000b01c000 R11: 0000000000000293 R12: 00007ffe0c32a52e R13: 00007ffe0c32a52f R14: 0000000000000000 R15: 00007fcf7b3e7640 Allocated by task 383: kasan_save_stack+0x1b/0x40 ____kasan_kmalloc.constprop.0+0x84/0xa0 kmem_cache_alloc+0x142/0x330 tcm_loop_queuecommand+0x2a/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Freed by task 11655: kasan_save_stack+0x1b/0x40 kasan_set_track+0x1c/0x30 kasan_set_free_info+0x20/0x30 ____kasan_slab_free+0xec/0x120 slab_free_freelist_hook+0x53/0x160 kmem_cache_free+0xf4/0x5c0 target_release_cmd_kref+0x3ea/0x9e0 [target_core_mod] transport_generic_free_cmd+0x28b/0x2f0 [target_core_mod] target_complete_ok_work+0x250/0xac0 [target_core_mod] process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcmu_handle_completions+0xad0/0x1770 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Second to last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcm_loop_queuecommand+0x1c3/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 The buggy address belongs to the object at ffff88814cf79800 which belongs to the cache tcm_loop_cmd_cache of size 896. Link: https://lore.kernel.org/r/20210113024508.1264992-1-shinichiro.kawasaki@wdc.com Fixes: a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") Cc: stable@vger.kernel.org # v5.9+ Acked-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-01-13 05:45:08 +03:00
se_cmd->priv = NULL;
target_complete_cmd(se_cmd, SAM_STAT_TASK_SET_FULL);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
static void tcmu_device_timedout(struct tcmu_dev *udev)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
spin_lock(&timed_out_udevs_lock);
if (list_empty(&udev->timedout_entry))
list_add_tail(&udev->timedout_entry, &timed_out_udevs);
spin_unlock(&timed_out_udevs_lock);
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
schedule_delayed_work(&tcmu_unmap_work, 0);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
static void tcmu_cmd_timedout(struct timer_list *t)
{
struct tcmu_dev *udev = from_timer(udev, t, cmd_timer);
pr_debug("%s cmd timeout has expired\n", udev->name);
tcmu_device_timedout(udev);
}
static void tcmu_qfull_timedout(struct timer_list *t)
{
struct tcmu_dev *udev = from_timer(udev, t, qfull_timer);
pr_debug("%s qfull timeout has expired\n", udev->name);
tcmu_device_timedout(udev);
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
static int tcmu_attach_hba(struct se_hba *hba, u32 host_id)
{
struct tcmu_hba *tcmu_hba;
tcmu_hba = kzalloc(sizeof(struct tcmu_hba), GFP_KERNEL);
if (!tcmu_hba)
return -ENOMEM;
tcmu_hba->host_id = host_id;
hba->hba_ptr = tcmu_hba;
return 0;
}
static void tcmu_detach_hba(struct se_hba *hba)
{
kfree(hba->hba_ptr);
hba->hba_ptr = NULL;
}
static struct se_device *tcmu_alloc_device(struct se_hba *hba, const char *name)
{
struct tcmu_dev *udev;
udev = kzalloc(sizeof(struct tcmu_dev), GFP_KERNEL);
if (!udev)
return NULL;
kref_init(&udev->kref);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
udev->name = kstrdup(name, GFP_KERNEL);
if (!udev->name) {
kfree(udev);
return NULL;
}
udev->hba = hba;
udev->cmd_time_out = TCMU_TIME_OUT;
udev->qfull_time_out = -1;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
udev->data_pages_per_blk = DATA_PAGES_PER_BLK_DEF;
udev->max_blocks = DATA_AREA_PAGES_DEF / udev->data_pages_per_blk;
udev->data_area_mb = TCMU_PAGES_TO_MBS(DATA_AREA_PAGES_DEF);
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
mutex_init(&udev->cmdr_lock);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
INIT_LIST_HEAD(&udev->node);
INIT_LIST_HEAD(&udev->timedout_entry);
INIT_LIST_HEAD(&udev->qfull_queue);
2020-07-26 18:35:09 +03:00
INIT_LIST_HEAD(&udev->tmr_queue);
INIT_LIST_HEAD(&udev->inflight_queue);
xa_init_flags(&udev->commands, XA_FLAGS_ALLOC1);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
timer_setup(&udev->qfull_timer, tcmu_qfull_timedout, 0);
timer_setup(&udev->cmd_timer, tcmu_cmd_timedout, 0);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
xa_init(&udev->data_pages);
tcmu: fix crash when removing the tcmu device Before the nl REMOVE msg has been sent to the userspace, the ring's and other resources have been released, but the userspace maybe still using them. And then we can see the crash messages like: ring broken, not handling completions BUG: unable to handle kernel paging request at ffffffffffffffd0 IP: tcmu_handle_completions+0x134/0x2f0 [target_core_user] PGD 11bdc0c067 P4D 11bdc0c067 PUD 11bdc0e067 PMD 0 Oops: 0000 [#1] SMP cmd_id not found, ring is broken RIP: 0010:tcmu_handle_completions+0x134/0x2f0 [target_core_user] RSP: 0018:ffffb8a2d8983d88 EFLAGS: 00010296 RAX: 0000000000000000 RBX: ffffb8a2aaa4e000 RCX: 00000000ffffffff RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000220 R10: 0000000076c71401 R11: ffff8d2e76c713f0 R12: ffffb8a2aad56bc0 R13: 000000000000001c R14: ffff8d2e32c90000 R15: ffff8d2e76c713f0 FS: 00007f411ffff700(0000) GS:ffff8d1e7fdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffffffffd0 CR3: 0000001027070000 CR4: 00000000001406e0 Call Trace: ? tcmu_irqcontrol+0x2a/0x40 [target_core_user] ? uio_write+0x7b/0xc0 [uio] ? __vfs_write+0x37/0x150 ? __getnstimeofday64+0x3b/0xd0 ? vfs_write+0xb2/0x1b0 ? syscall_trace_enter+0x1d0/0x2b0 ? SyS_write+0x55/0xc0 ? do_syscall_64+0x67/0x150 ? entry_SYSCALL64_slow_path+0x25/0x25 Code: 41 5d 41 5e 41 5f 5d c3 83 f8 01 0f 85 cf 01 00 00 48 8b 7d d0 e8 dd 5c 1d f3 41 0f b7 74 24 04 48 8b 7d c8 31 d2 e8 5c c7 1b f3 <48> 8b 7d d0 49 89 c7 c6 07 00 0f 1f 40 00 4d 85 ff 0f 84 82 01 RIP: tcmu_handle_completions+0x134/0x2f0 [target_core_user] RSP: ffffb8a2d8983d88 CR2: ffffffffffffffd0 And the crash also could happen in tcmu_page_fault and other places. Signed-off-by: Zhang Zhuoyu <zhangzhuoyu@cmss.chinamobile.com> Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Reviewed-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-09-14 04:30:05 +03:00
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return &udev->se_dev;
}
static void tcmu_dev_call_rcu(struct rcu_head *p)
{
struct se_device *dev = container_of(p, struct se_device, rcu_head);
struct tcmu_dev *udev = TCMU_DEV(dev);
kfree(udev->uio_info.name);
kfree(udev->name);
kfree(udev);
}
static int tcmu_check_and_free_pending_cmd(struct tcmu_cmd *cmd)
{
if (test_bit(TCMU_CMD_BIT_EXPIRED, &cmd->flags) ||
test_bit(TCMU_CMD_BIT_KEEP_BUF, &cmd->flags)) {
kmem_cache_free(tcmu_cmd_cache, cmd);
return 0;
}
return -EINVAL;
}
static u32 tcmu_blocks_release(struct tcmu_dev *udev, unsigned long first,
unsigned long last)
{
XA_STATE(xas, &udev->data_pages, first * udev->data_pages_per_blk);
struct page *page;
u32 pages_freed = 0;
xas_lock(&xas);
xas_for_each(&xas, page, (last + 1) * udev->data_pages_per_blk - 1) {
xas_store(&xas, NULL);
__free_page(page);
pages_freed++;
}
xas_unlock(&xas);
atomic_sub(pages_freed, &global_page_count);
return pages_freed;
}
static void tcmu_remove_all_queued_tmr(struct tcmu_dev *udev)
{
struct tcmu_tmr *tmr, *tmp;
list_for_each_entry_safe(tmr, tmp, &udev->tmr_queue, queue_entry) {
list_del_init(&tmr->queue_entry);
kfree(tmr);
}
}
static void tcmu_dev_kref_release(struct kref *kref)
{
struct tcmu_dev *udev = container_of(kref, struct tcmu_dev, kref);
struct se_device *dev = &udev->se_dev;
struct tcmu_cmd *cmd;
bool all_expired = true;
unsigned long i;
vfree(udev->mb_addr);
udev->mb_addr = NULL;
spin_lock_bh(&timed_out_udevs_lock);
if (!list_empty(&udev->timedout_entry))
list_del(&udev->timedout_entry);
spin_unlock_bh(&timed_out_udevs_lock);
/* Upper layer should drain all requests before calling this */
mutex_lock(&udev->cmdr_lock);
xa_for_each(&udev->commands, i, cmd) {
if (tcmu_check_and_free_pending_cmd(cmd) != 0)
all_expired = false;
}
/* There can be left over TMR cmds. Remove them. */
tcmu_remove_all_queued_tmr(udev);
if (!list_empty(&udev->qfull_queue))
all_expired = false;
xa_destroy(&udev->commands);
WARN_ON(!all_expired);
tcmu_blocks_release(udev, 0, udev->dbi_max);
bitmap_free(udev->data_bitmap);
mutex_unlock(&udev->cmdr_lock);
scsi: target: tcmu: Fix memory leak caused by wrong uio usage When user deletes a tcmu device via configFS, tcmu calls uio_unregister_device(). During that call uio resets its pointer to struct uio_info provided by tcmu. That means, after uio_unregister_device() uio will no longer execute any of the callbacks tcmu had set in uio_info. Especially, if userspace daemon still holds the corresponding uio device open or mmap'ed while tcmu calls uio_unregister_device(), uio will not call tcmu_release() when userspace finally closes and munmaps the uio device. Since tcmu does refcounting for the tcmu device in tcmu_open() and tcmu_release(), in the decribed case refcount does not drop to 0 and tcmu does not free tcmu device's resources. In extreme cases this can cause memory leaking of up to 1 GB for a single tcmu device. After uio_unregister_device(), uio will reject every open, read, write, mmap from userspace with -EOI. But userspace daemon can still access the mmap'ed command ring and data area. Therefore tcmu should wait until userspace munmaps the uio device before it frees the resources, as we don't want to cause SIGSEGV or SIGBUS to user space. That said, current refcounting during tcmu_open and tcmu_release does not work correctly, and refcounting better should be done in the open and close callouts of the vm_operations_struct, which tcmu assigns to each mmap of the uio device (because it wants its own page fault handler). This patch fixes the memory leak by removing refcounting from tcmu_open and tcmu_close, and instead adding new tcmu_vma_open() and tcmu_vma_close() handlers that only do refcounting. Link: https://lore.kernel.org/r/20210218175039.7829-3-bostroesser@gmail.com Reviewed-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-02-18 20:50:39 +03:00
pr_debug("dev_kref_release\n");
call_rcu(&dev->rcu_head, tcmu_dev_call_rcu);
}
static void run_qfull_queue(struct tcmu_dev *udev, bool fail)
{
struct tcmu_cmd *tcmu_cmd, *tmp_cmd;
LIST_HEAD(cmds);
sense_reason_t scsi_ret;
int ret;
if (list_empty(&udev->qfull_queue))
return;
pr_debug("running %s's cmdr queue forcefail %d\n", udev->name, fail);
list_splice_init(&udev->qfull_queue, &cmds);
list_for_each_entry_safe(tcmu_cmd, tmp_cmd, &cmds, queue_entry) {
list_del_init(&tcmu_cmd->queue_entry);
pr_debug("removing cmd %p on dev %s from queue\n",
tcmu_cmd, udev->name);
if (fail) {
/*
* We were not able to even start the command, so
* fail with busy to allow a retry in case runner
* was only temporarily down. If the device is being
* removed then LIO core will do the right thing and
* fail the retry.
*/
scsi: target: tcmu: Fix use-after-free of se_cmd->priv Commit a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") modified tcmu_free_cmd() to set NULL to priv pointer in se_cmd. However, se_cmd can be already freed by work queue triggered in target_complete_cmd(). This caused BUG KASAN use-after-free [1]. To fix the bug, do not touch priv pointer in tcmu_free_cmd(). Instead, set NULL to priv pointer before target_complete_cmd() calls. Also, to avoid unnecessary priv pointer change in tcmu_queue_cmd(), modify priv pointer in the function only when tcmu_free_cmd() is not called. [1] BUG: KASAN: use-after-free in tcmu_handle_completions+0x1172/0x1770 [target_core_user] Write of size 8 at addr ffff88814cf79a40 by task cmdproc-uio0/14842 CPU: 2 PID: 14842 Comm: cmdproc-uio0 Not tainted 5.11.0-rc2 #1 Hardware name: Supermicro Super Server/X10SRL-F, BIOS 3.2 11/22/2019 Call Trace: dump_stack+0x9a/0xcc ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] print_address_description.constprop.0+0x18/0x130 ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] kasan_report.cold+0x7f/0x10e ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? queue_tmr_ring+0x5d0/0x5d0 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 ? uio_vma_fault+0x460/0x460 ? security_file_permission+0x4f/0x440 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 ? __ia32_sys_read+0xb0/0xb0 ? syscall_enter_from_user_mode+0x27/0x70 ? trace_hardirqs_on+0x1c/0x110 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fcf8b61905f Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 0c fd ff ff 48 RSP: 002b:00007fcf7b3e6c30 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcf8b61905f RDX: 0000000000000004 RSI: 00007fcf7b3e6c78 RDI: 000000000000000c RBP: 00007fcf7b3e6c80 R08: 0000000000000000 R09: 00007fcf7b3e6aa8 R10: 000000000b01c000 R11: 0000000000000293 R12: 00007ffe0c32a52e R13: 00007ffe0c32a52f R14: 0000000000000000 R15: 00007fcf7b3e7640 Allocated by task 383: kasan_save_stack+0x1b/0x40 ____kasan_kmalloc.constprop.0+0x84/0xa0 kmem_cache_alloc+0x142/0x330 tcm_loop_queuecommand+0x2a/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Freed by task 11655: kasan_save_stack+0x1b/0x40 kasan_set_track+0x1c/0x30 kasan_set_free_info+0x20/0x30 ____kasan_slab_free+0xec/0x120 slab_free_freelist_hook+0x53/0x160 kmem_cache_free+0xf4/0x5c0 target_release_cmd_kref+0x3ea/0x9e0 [target_core_mod] transport_generic_free_cmd+0x28b/0x2f0 [target_core_mod] target_complete_ok_work+0x250/0xac0 [target_core_mod] process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcmu_handle_completions+0xad0/0x1770 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Second to last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcm_loop_queuecommand+0x1c3/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 The buggy address belongs to the object at ffff88814cf79800 which belongs to the cache tcm_loop_cmd_cache of size 896. Link: https://lore.kernel.org/r/20210113024508.1264992-1-shinichiro.kawasaki@wdc.com Fixes: a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") Cc: stable@vger.kernel.org # v5.9+ Acked-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-01-13 05:45:08 +03:00
tcmu_cmd->se_cmd->priv = NULL;
target_complete_cmd(tcmu_cmd->se_cmd, SAM_STAT_BUSY);
tcmu_free_cmd(tcmu_cmd);
continue;
}
ret = queue_cmd_ring(tcmu_cmd, &scsi_ret);
if (ret < 0) {
pr_debug("cmd %p on dev %s failed with %u\n",
tcmu_cmd, udev->name, scsi_ret);
/*
* Ignore scsi_ret for now. target_complete_cmd
* drops it.
*/
scsi: target: tcmu: Fix use-after-free of se_cmd->priv Commit a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") modified tcmu_free_cmd() to set NULL to priv pointer in se_cmd. However, se_cmd can be already freed by work queue triggered in target_complete_cmd(). This caused BUG KASAN use-after-free [1]. To fix the bug, do not touch priv pointer in tcmu_free_cmd(). Instead, set NULL to priv pointer before target_complete_cmd() calls. Also, to avoid unnecessary priv pointer change in tcmu_queue_cmd(), modify priv pointer in the function only when tcmu_free_cmd() is not called. [1] BUG: KASAN: use-after-free in tcmu_handle_completions+0x1172/0x1770 [target_core_user] Write of size 8 at addr ffff88814cf79a40 by task cmdproc-uio0/14842 CPU: 2 PID: 14842 Comm: cmdproc-uio0 Not tainted 5.11.0-rc2 #1 Hardware name: Supermicro Super Server/X10SRL-F, BIOS 3.2 11/22/2019 Call Trace: dump_stack+0x9a/0xcc ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] print_address_description.constprop.0+0x18/0x130 ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] kasan_report.cold+0x7f/0x10e ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? queue_tmr_ring+0x5d0/0x5d0 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 ? uio_vma_fault+0x460/0x460 ? security_file_permission+0x4f/0x440 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 ? __ia32_sys_read+0xb0/0xb0 ? syscall_enter_from_user_mode+0x27/0x70 ? trace_hardirqs_on+0x1c/0x110 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fcf8b61905f Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 0c fd ff ff 48 RSP: 002b:00007fcf7b3e6c30 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcf8b61905f RDX: 0000000000000004 RSI: 00007fcf7b3e6c78 RDI: 000000000000000c RBP: 00007fcf7b3e6c80 R08: 0000000000000000 R09: 00007fcf7b3e6aa8 R10: 000000000b01c000 R11: 0000000000000293 R12: 00007ffe0c32a52e R13: 00007ffe0c32a52f R14: 0000000000000000 R15: 00007fcf7b3e7640 Allocated by task 383: kasan_save_stack+0x1b/0x40 ____kasan_kmalloc.constprop.0+0x84/0xa0 kmem_cache_alloc+0x142/0x330 tcm_loop_queuecommand+0x2a/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Freed by task 11655: kasan_save_stack+0x1b/0x40 kasan_set_track+0x1c/0x30 kasan_set_free_info+0x20/0x30 ____kasan_slab_free+0xec/0x120 slab_free_freelist_hook+0x53/0x160 kmem_cache_free+0xf4/0x5c0 target_release_cmd_kref+0x3ea/0x9e0 [target_core_mod] transport_generic_free_cmd+0x28b/0x2f0 [target_core_mod] target_complete_ok_work+0x250/0xac0 [target_core_mod] process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcmu_handle_completions+0xad0/0x1770 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Second to last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcm_loop_queuecommand+0x1c3/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 The buggy address belongs to the object at ffff88814cf79800 which belongs to the cache tcm_loop_cmd_cache of size 896. Link: https://lore.kernel.org/r/20210113024508.1264992-1-shinichiro.kawasaki@wdc.com Fixes: a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") Cc: stable@vger.kernel.org # v5.9+ Acked-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-01-13 05:45:08 +03:00
tcmu_cmd->se_cmd->priv = NULL;
target_complete_cmd(tcmu_cmd->se_cmd,
SAM_STAT_CHECK_CONDITION);
tcmu_free_cmd(tcmu_cmd);
} else if (ret > 0) {
pr_debug("ran out of space during cmdr queue run\n");
/*
* cmd was requeued, so just put all cmds back in
* the queue
*/
list_splice_tail(&cmds, &udev->qfull_queue);
break;
}
}
tcmu_set_next_deadline(&udev->qfull_queue, &udev->qfull_timer);
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
static int tcmu_irqcontrol(struct uio_info *info, s32 irq_on)
{
struct tcmu_dev *udev = container_of(info, struct tcmu_dev, uio_info);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
mutex_lock(&udev->cmdr_lock);
2020-07-26 18:35:09 +03:00
if (tcmu_handle_completions(udev))
run_qfull_queue(udev, false);
mutex_unlock(&udev->cmdr_lock);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return 0;
}
/*
* mmap code from uio.c. Copied here because we want to hook mmap()
* and this stuff must come along.
*/
static int tcmu_find_mem_index(struct vm_area_struct *vma)
{
struct tcmu_dev *udev = vma->vm_private_data;
struct uio_info *info = &udev->uio_info;
if (vma->vm_pgoff < MAX_UIO_MAPS) {
if (info->mem[vma->vm_pgoff].size == 0)
return -1;
return (int)vma->vm_pgoff;
}
return -1;
}
static struct page *tcmu_try_get_data_page(struct tcmu_dev *udev, uint32_t dpi)
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
{
struct page *page;
mutex_lock(&udev->cmdr_lock);
page = xa_load(&udev->data_pages, dpi);
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
if (likely(page)) {
mutex_unlock(&udev->cmdr_lock);
return page;
}
/*
* Userspace messed up and passed in a address not in the
* data iov passed to it.
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
*/
pr_err("Invalid addr to data page mapping (dpi %u) on device %s\n",
dpi, udev->name);
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
mutex_unlock(&udev->cmdr_lock);
return NULL;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
}
scsi: target: tcmu: Fix memory leak caused by wrong uio usage When user deletes a tcmu device via configFS, tcmu calls uio_unregister_device(). During that call uio resets its pointer to struct uio_info provided by tcmu. That means, after uio_unregister_device() uio will no longer execute any of the callbacks tcmu had set in uio_info. Especially, if userspace daemon still holds the corresponding uio device open or mmap'ed while tcmu calls uio_unregister_device(), uio will not call tcmu_release() when userspace finally closes and munmaps the uio device. Since tcmu does refcounting for the tcmu device in tcmu_open() and tcmu_release(), in the decribed case refcount does not drop to 0 and tcmu does not free tcmu device's resources. In extreme cases this can cause memory leaking of up to 1 GB for a single tcmu device. After uio_unregister_device(), uio will reject every open, read, write, mmap from userspace with -EOI. But userspace daemon can still access the mmap'ed command ring and data area. Therefore tcmu should wait until userspace munmaps the uio device before it frees the resources, as we don't want to cause SIGSEGV or SIGBUS to user space. That said, current refcounting during tcmu_open and tcmu_release does not work correctly, and refcounting better should be done in the open and close callouts of the vm_operations_struct, which tcmu assigns to each mmap of the uio device (because it wants its own page fault handler). This patch fixes the memory leak by removing refcounting from tcmu_open and tcmu_close, and instead adding new tcmu_vma_open() and tcmu_vma_close() handlers that only do refcounting. Link: https://lore.kernel.org/r/20210218175039.7829-3-bostroesser@gmail.com Reviewed-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-02-18 20:50:39 +03:00
static void tcmu_vma_open(struct vm_area_struct *vma)
{
struct tcmu_dev *udev = vma->vm_private_data;
pr_debug("vma_open\n");
kref_get(&udev->kref);
}
static void tcmu_vma_close(struct vm_area_struct *vma)
{
struct tcmu_dev *udev = vma->vm_private_data;
pr_debug("vma_close\n");
/* release ref from tcmu_vma_open */
kref_put(&udev->kref, tcmu_dev_kref_release);
}
static vm_fault_t tcmu_vma_fault(struct vm_fault *vmf)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
struct tcmu_dev *udev = vmf->vma->vm_private_data;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
struct uio_info *info = &udev->uio_info;
struct page *page;
unsigned long offset;
void *addr;
int mi = tcmu_find_mem_index(vmf->vma);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
if (mi < 0)
return VM_FAULT_SIGBUS;
/*
* We need to subtract mi because userspace uses offset = N*PAGE_SIZE
* to use mem[N].
*/
offset = (vmf->pgoff - mi) << PAGE_SHIFT;
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
if (offset < udev->data_off) {
/* For the vmalloc()ed cmd area pages */
addr = (void *)(unsigned long)info->mem[mi].addr + offset;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
page = vmalloc_to_page(addr);
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
} else {
uint32_t dpi;
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
/* For the dynamically growing data area pages */
dpi = (offset - udev->data_off) / PAGE_SIZE;
page = tcmu_try_get_data_page(udev, dpi);
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
if (!page)
return VM_FAULT_SIGBUS;
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
get_page(page);
vmf->page = page;
return 0;
}
static const struct vm_operations_struct tcmu_vm_ops = {
scsi: target: tcmu: Fix memory leak caused by wrong uio usage When user deletes a tcmu device via configFS, tcmu calls uio_unregister_device(). During that call uio resets its pointer to struct uio_info provided by tcmu. That means, after uio_unregister_device() uio will no longer execute any of the callbacks tcmu had set in uio_info. Especially, if userspace daemon still holds the corresponding uio device open or mmap'ed while tcmu calls uio_unregister_device(), uio will not call tcmu_release() when userspace finally closes and munmaps the uio device. Since tcmu does refcounting for the tcmu device in tcmu_open() and tcmu_release(), in the decribed case refcount does not drop to 0 and tcmu does not free tcmu device's resources. In extreme cases this can cause memory leaking of up to 1 GB for a single tcmu device. After uio_unregister_device(), uio will reject every open, read, write, mmap from userspace with -EOI. But userspace daemon can still access the mmap'ed command ring and data area. Therefore tcmu should wait until userspace munmaps the uio device before it frees the resources, as we don't want to cause SIGSEGV or SIGBUS to user space. That said, current refcounting during tcmu_open and tcmu_release does not work correctly, and refcounting better should be done in the open and close callouts of the vm_operations_struct, which tcmu assigns to each mmap of the uio device (because it wants its own page fault handler). This patch fixes the memory leak by removing refcounting from tcmu_open and tcmu_close, and instead adding new tcmu_vma_open() and tcmu_vma_close() handlers that only do refcounting. Link: https://lore.kernel.org/r/20210218175039.7829-3-bostroesser@gmail.com Reviewed-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-02-18 20:50:39 +03:00
.open = tcmu_vma_open,
.close = tcmu_vma_close,
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
.fault = tcmu_vma_fault,
};
static int tcmu_mmap(struct uio_info *info, struct vm_area_struct *vma)
{
struct tcmu_dev *udev = container_of(info, struct tcmu_dev, uio_info);
vma->vm_flags |= VM_DONTEXPAND | VM_DONTDUMP;
vma->vm_ops = &tcmu_vm_ops;
vma->vm_private_data = udev;
/* Ensure the mmap is exactly the right size */
if (vma_pages(vma) != udev->mmap_pages)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return -EINVAL;
scsi: target: tcmu: Fix memory leak caused by wrong uio usage When user deletes a tcmu device via configFS, tcmu calls uio_unregister_device(). During that call uio resets its pointer to struct uio_info provided by tcmu. That means, after uio_unregister_device() uio will no longer execute any of the callbacks tcmu had set in uio_info. Especially, if userspace daemon still holds the corresponding uio device open or mmap'ed while tcmu calls uio_unregister_device(), uio will not call tcmu_release() when userspace finally closes and munmaps the uio device. Since tcmu does refcounting for the tcmu device in tcmu_open() and tcmu_release(), in the decribed case refcount does not drop to 0 and tcmu does not free tcmu device's resources. In extreme cases this can cause memory leaking of up to 1 GB for a single tcmu device. After uio_unregister_device(), uio will reject every open, read, write, mmap from userspace with -EOI. But userspace daemon can still access the mmap'ed command ring and data area. Therefore tcmu should wait until userspace munmaps the uio device before it frees the resources, as we don't want to cause SIGSEGV or SIGBUS to user space. That said, current refcounting during tcmu_open and tcmu_release does not work correctly, and refcounting better should be done in the open and close callouts of the vm_operations_struct, which tcmu assigns to each mmap of the uio device (because it wants its own page fault handler). This patch fixes the memory leak by removing refcounting from tcmu_open and tcmu_close, and instead adding new tcmu_vma_open() and tcmu_vma_close() handlers that only do refcounting. Link: https://lore.kernel.org/r/20210218175039.7829-3-bostroesser@gmail.com Reviewed-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-02-18 20:50:39 +03:00
tcmu_vma_open(vma);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return 0;
}
static int tcmu_open(struct uio_info *info, struct inode *inode)
{
struct tcmu_dev *udev = container_of(info, struct tcmu_dev, uio_info);
/* O_EXCL not supported for char devs, so fake it? */
if (test_and_set_bit(TCMU_DEV_BIT_OPEN, &udev->flags))
return -EBUSY;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
udev->inode = inode;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
pr_debug("open\n");
return 0;
}
static int tcmu_release(struct uio_info *info, struct inode *inode)
{
struct tcmu_dev *udev = container_of(info, struct tcmu_dev, uio_info);
struct tcmu_cmd *cmd;
unsigned long i;
bool freed = false;
mutex_lock(&udev->cmdr_lock);
xa_for_each(&udev->commands, i, cmd) {
/* Cmds with KEEP_BUF set are no longer on the ring, but
* userspace still holds the data buffer. If userspace closes
* we implicitly free these cmds and buffers, since after new
* open the (new ?) userspace cannot find the cmd in the ring
* and thus never will release the buffer by writing cmd_id to
* free_kept_buf action attribute.
*/
if (!test_bit(TCMU_CMD_BIT_KEEP_BUF, &cmd->flags))
continue;
pr_debug("removing KEEP_BUF cmd %u on dev %s from ring\n",
cmd->cmd_id, udev->name);
freed = true;
xa_erase(&udev->commands, i);
tcmu_cmd_free_data(cmd, cmd->dbi_cnt);
tcmu_free_cmd(cmd);
}
/*
* We only freed data space, not ring space. Therefore we dont call
* run_tmr_queue, but call run_qfull_queue if tmr_list is empty.
*/
if (freed && list_empty(&udev->tmr_queue))
run_qfull_queue(udev, false);
mutex_unlock(&udev->cmdr_lock);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
clear_bit(TCMU_DEV_BIT_OPEN, &udev->flags);
pr_debug("close\n");
scsi: target: tcmu: Fix memory leak caused by wrong uio usage When user deletes a tcmu device via configFS, tcmu calls uio_unregister_device(). During that call uio resets its pointer to struct uio_info provided by tcmu. That means, after uio_unregister_device() uio will no longer execute any of the callbacks tcmu had set in uio_info. Especially, if userspace daemon still holds the corresponding uio device open or mmap'ed while tcmu calls uio_unregister_device(), uio will not call tcmu_release() when userspace finally closes and munmaps the uio device. Since tcmu does refcounting for the tcmu device in tcmu_open() and tcmu_release(), in the decribed case refcount does not drop to 0 and tcmu does not free tcmu device's resources. In extreme cases this can cause memory leaking of up to 1 GB for a single tcmu device. After uio_unregister_device(), uio will reject every open, read, write, mmap from userspace with -EOI. But userspace daemon can still access the mmap'ed command ring and data area. Therefore tcmu should wait until userspace munmaps the uio device before it frees the resources, as we don't want to cause SIGSEGV or SIGBUS to user space. That said, current refcounting during tcmu_open and tcmu_release does not work correctly, and refcounting better should be done in the open and close callouts of the vm_operations_struct, which tcmu assigns to each mmap of the uio device (because it wants its own page fault handler). This patch fixes the memory leak by removing refcounting from tcmu_open and tcmu_close, and instead adding new tcmu_vma_open() and tcmu_vma_close() handlers that only do refcounting. Link: https://lore.kernel.org/r/20210218175039.7829-3-bostroesser@gmail.com Reviewed-by: Mike Christie <michael.christie@oracle.com> Signed-off-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-02-18 20:50:39 +03:00
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return 0;
}
static int tcmu_init_genl_cmd_reply(struct tcmu_dev *udev, int cmd)
{
struct tcmu_nl_cmd *nl_cmd = &udev->curr_nl_cmd;
if (!tcmu_kern_cmd_reply_supported)
return 0;
if (udev->nl_reply_supported <= 0)
return 0;
mutex_lock(&tcmu_nl_cmd_mutex);
if (tcmu_netlink_blocked) {
mutex_unlock(&tcmu_nl_cmd_mutex);
pr_warn("Failing nl cmd %d on %s. Interface is blocked.\n", cmd,
udev->name);
return -EAGAIN;
}
if (nl_cmd->cmd != TCMU_CMD_UNSPEC) {
mutex_unlock(&tcmu_nl_cmd_mutex);
pr_warn("netlink cmd %d already executing on %s\n",
nl_cmd->cmd, udev->name);
return -EBUSY;
}
memset(nl_cmd, 0, sizeof(*nl_cmd));
nl_cmd->cmd = cmd;
nl_cmd->udev = udev;
init_completion(&nl_cmd->complete);
INIT_LIST_HEAD(&nl_cmd->nl_list);
list_add_tail(&nl_cmd->nl_list, &tcmu_nl_cmd_list);
mutex_unlock(&tcmu_nl_cmd_mutex);
return 0;
}
static void tcmu_destroy_genl_cmd_reply(struct tcmu_dev *udev)
{
struct tcmu_nl_cmd *nl_cmd = &udev->curr_nl_cmd;
if (!tcmu_kern_cmd_reply_supported)
return;
if (udev->nl_reply_supported <= 0)
return;
mutex_lock(&tcmu_nl_cmd_mutex);
list_del(&nl_cmd->nl_list);
memset(nl_cmd, 0, sizeof(*nl_cmd));
mutex_unlock(&tcmu_nl_cmd_mutex);
}
static int tcmu_wait_genl_cmd_reply(struct tcmu_dev *udev)
{
struct tcmu_nl_cmd *nl_cmd = &udev->curr_nl_cmd;
int ret;
if (!tcmu_kern_cmd_reply_supported)
return 0;
if (udev->nl_reply_supported <= 0)
return 0;
pr_debug("sleeping for nl reply\n");
wait_for_completion(&nl_cmd->complete);
mutex_lock(&tcmu_nl_cmd_mutex);
nl_cmd->cmd = TCMU_CMD_UNSPEC;
ret = nl_cmd->status;
mutex_unlock(&tcmu_nl_cmd_mutex);
return ret;
}
static int tcmu_netlink_event_init(struct tcmu_dev *udev,
enum tcmu_genl_cmd cmd,
struct sk_buff **buf, void **hdr)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
struct sk_buff *skb;
void *msg_header;
int ret = -ENOMEM;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
skb = genlmsg_new(NLMSG_GOODSIZE, GFP_KERNEL);
if (!skb)
return ret;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
msg_header = genlmsg_put(skb, 0, 0, &tcmu_genl_family, 0, cmd);
if (!msg_header)
goto free_skb;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
ret = nla_put_string(skb, TCMU_ATTR_DEVICE, udev->uio_info.name);
if (ret < 0)
goto free_skb;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
ret = nla_put_u32(skb, TCMU_ATTR_MINOR, udev->uio_info.uio_dev->minor);
if (ret < 0)
goto free_skb;
ret = nla_put_u32(skb, TCMU_ATTR_DEVICE_ID, udev->se_dev.dev_index);
if (ret < 0)
goto free_skb;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
*buf = skb;
*hdr = msg_header;
return ret;
free_skb:
nlmsg_free(skb);
return ret;
}
static int tcmu_netlink_event_send(struct tcmu_dev *udev,
enum tcmu_genl_cmd cmd,
struct sk_buff *skb, void *msg_header)
{
int ret;
netlink: make nlmsg_end() and genlmsg_end() void Contrary to common expectations for an "int" return, these functions return only a positive value -- if used correctly they cannot even return 0 because the message header will necessarily be in the skb. This makes the very common pattern of if (genlmsg_end(...) < 0) { ... } be a whole bunch of dead code. Many places also simply do return nlmsg_end(...); and the caller is expected to deal with it. This also commonly (at least for me) causes errors, because it is very common to write if (my_function(...)) /* error condition */ and if my_function() does "return nlmsg_end()" this is of course wrong. Additionally, there's not a single place in the kernel that actually needs the message length returned, and if anyone needs it later then it'll be very easy to just use skb->len there. Remove this, and make the functions void. This removes a bunch of dead code as described above. The patch adds lines because I did - return nlmsg_end(...); + nlmsg_end(...); + return 0; I could have preserved all the function's return values by returning skb->len, but instead I've audited all the places calling the affected functions and found that none cared. A few places actually compared the return value with <= 0 in dump functionality, but that could just be changed to < 0 with no change in behaviour, so I opted for the more efficient version. One instance of the error I've made numerous times now is also present in net/phonet/pn_netlink.c in the route_dumpit() function - it didn't check for <0 or <=0 and thus broke out of the loop every single time. I've preserved this since it will (I think) have caused the messages to userspace to be formatted differently with just a single message for every SKB returned to userspace. It's possible that this isn't needed for the tools that actually use this, but I don't even know what they are so couldn't test that changing this behaviour would be acceptable. Signed-off-by: Johannes Berg <johannes.berg@intel.com> Signed-off-by: David S. Miller <davem@davemloft.net>
2015-01-17 00:09:00 +03:00
genlmsg_end(skb, msg_header);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
ret = tcmu_init_genl_cmd_reply(udev, cmd);
if (ret) {
nlmsg_free(skb);
return ret;
}
ret = genlmsg_multicast_allns(&tcmu_genl_family, skb, 0,
TCMU_MCGRP_CONFIG, GFP_KERNEL);
/* Wait during an add as the listener may not be up yet */
if (ret == 0 ||
(ret == -ESRCH && cmd == TCMU_CMD_ADDED_DEVICE))
return tcmu_wait_genl_cmd_reply(udev);
else
tcmu_destroy_genl_cmd_reply(udev);
return ret;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
static int tcmu_send_dev_add_event(struct tcmu_dev *udev)
{
struct sk_buff *skb = NULL;
void *msg_header = NULL;
int ret = 0;
ret = tcmu_netlink_event_init(udev, TCMU_CMD_ADDED_DEVICE, &skb,
&msg_header);
if (ret < 0)
return ret;
return tcmu_netlink_event_send(udev, TCMU_CMD_ADDED_DEVICE, skb,
msg_header);
}
static int tcmu_send_dev_remove_event(struct tcmu_dev *udev)
{
struct sk_buff *skb = NULL;
void *msg_header = NULL;
int ret = 0;
ret = tcmu_netlink_event_init(udev, TCMU_CMD_REMOVED_DEVICE,
&skb, &msg_header);
if (ret < 0)
return ret;
return tcmu_netlink_event_send(udev, TCMU_CMD_REMOVED_DEVICE,
skb, msg_header);
}
static int tcmu_update_uio_info(struct tcmu_dev *udev)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
struct tcmu_hba *hba = udev->hba->hba_ptr;
struct uio_info *info;
char *str;
info = &udev->uio_info;
if (udev->dev_config[0])
str = kasprintf(GFP_KERNEL, "tcm-user/%u/%s/%s", hba->host_id,
udev->name, udev->dev_config);
else
str = kasprintf(GFP_KERNEL, "tcm-user/%u/%s", hba->host_id,
udev->name);
if (!str)
return -ENOMEM;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
/* If the old string exists, free it */
kfree(info->name);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
info->name = str;
return 0;
}
static int tcmu_configure_device(struct se_device *dev)
{
struct tcmu_dev *udev = TCMU_DEV(dev);
struct uio_info *info;
struct tcmu_mailbox *mb;
size_t data_size;
int ret = 0;
ret = tcmu_update_uio_info(udev);
if (ret)
return ret;
info = &udev->uio_info;
mutex_lock(&udev->cmdr_lock);
udev->data_bitmap = bitmap_zalloc(udev->max_blocks, GFP_KERNEL);
mutex_unlock(&udev->cmdr_lock);
if (!udev->data_bitmap) {
ret = -ENOMEM;
goto err_bitmap_alloc;
}
mb = vzalloc(MB_CMDR_SIZE);
if (!mb) {
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
ret = -ENOMEM;
goto err_vzalloc;
}
/* mailbox fits in first part of CMDR space */
udev->mb_addr = mb;
udev->cmdr = (void *)mb + CMDR_OFF;
udev->cmdr_size = CMDR_SIZE;
udev->data_off = MB_CMDR_SIZE;
data_size = TCMU_MBS_TO_PAGES(udev->data_area_mb) << PAGE_SHIFT;
udev->mmap_pages = (data_size + MB_CMDR_SIZE) >> PAGE_SHIFT;
udev->data_blk_size = udev->data_pages_per_blk * PAGE_SIZE;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
udev->dbi_thresh = 0; /* Default in Idle state */
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
/* Initialise the mailbox of the ring buffer */
target: Version 2 of TCMU ABI The initial version of TCMU (in 3.18) does not properly handle bidirectional SCSI commands -- those with both an in and out buffer. In looking to fix this it also became clear that TCMU's support for adding new types of entries (opcodes) to the command ring was broken. We need to fix this now, so that future issues can be handled properly by adding new opcodes. We make the most of this ABI break by enabling bidi cmd handling within TCMP_OP_CMD opcode. Add an iov_bidi_cnt field to tcmu_cmd_entry.req. This enables TCMU to describe bidi commands, but further kernel work is needed for full bidi support. Enlarge tcmu_cmd_entry_hdr by 32 bits by pulling in cmd_id and __pad1. Turn __pad1 into two 8 bit flags fields, for kernel-set and userspace-set flags, "kflags" and "uflags" respectively. Update version fields so userspace can tell the interface is changed. Update tcmu-design.txt with details of how new stuff works: - Specify an additional requirement for userspace to set UNKNOWN_OP (bit 0) in hdr.uflags for unknown/unhandled opcodes. - Define how Data-In and Data-Out fields are described in req.iov[] Changed in v2: - Change name of SKIPPED bit to UNKNOWN bit - PAD op does not set the bit any more - Change len_op helper functions to take just len_op, not the whole struct - Change version to 2 in missed spots, and use defines - Add 16 unused bytes to cmd_entry.req, in case additional SAM cmd parameters need to be included - Add iov_dif_cnt field to specify buffers used for DIF info in iov[] - Rearrange fields to naturally align cdb_off - Handle if userspace sets UNKNOWN_OP by indicating failure of the cmd - Wrap some overly long UPDATE_HEAD lines (Add missing req.iov_bidi_cnt + req.iov_dif_cnt zeroing - Ilias) Signed-off-by: Andy Grover <agrover@redhat.com> Reviewed-by: Ilias Tsitsimpis <iliastsi@arrikto.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2015-04-15 03:30:04 +03:00
mb->version = TCMU_MAILBOX_VERSION;
2020-07-26 18:35:09 +03:00
mb->flags = TCMU_MAILBOX_FLAG_CAP_OOOC |
TCMU_MAILBOX_FLAG_CAP_READ_LEN |
TCMU_MAILBOX_FLAG_CAP_TMR |
TCMU_MAILBOX_FLAG_CAP_KEEP_BUF;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
mb->cmdr_off = CMDR_OFF;
mb->cmdr_size = udev->cmdr_size;
WARN_ON(!PAGE_ALIGNED(udev->data_off));
WARN_ON(data_size % PAGE_SIZE);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
info->version = __stringify(TCMU_MAILBOX_VERSION);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
info->mem[0].name = "tcm-user command & data buffer";
info->mem[0].addr = (phys_addr_t)(uintptr_t)udev->mb_addr;
info->mem[0].size = data_size + MB_CMDR_SIZE;
tcmu: Add dynamic growing data area feature support Currently for the TCMU, the ring buffer size is fixed to 64K cmd area + 1M data area, and this will be bottlenecks for high iops. The struct tcmu_cmd_entry {} size is fixed about 112 bytes with iovec[N] & N <= 4, and the size of struct iovec is about 16 bytes. If N == 0, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : 0, no data area is need. If 0 < N <=4, the ratio will be sizeof(cmd entry) : sizeof(datas) == 112Bytes : (N * 4096)Bytes = 28 : (N * 1024), so the max will be 28 : 1024. If N > 4, the sizeof(cmd entry) will be [(N - 4) *16 + 112] bytes, and its corresponding data size will be [N * 4096], so the ratio of sizeof(cmd entry) : sizeof(datas) == [(N - 4) * 16 + 112)Bytes : (N * 4096)Bytes == 4/1024 - 12/(N * 1024), so the max is about 4 : 1024. When N is bigger, the ratio will be smaller. As the initial patch, we will set the cmd area size to 2M, and the cmd area size to 32M. The TCMU will dynamically grows the data area from 0 to max 32M size as needed. The cmd area memory will be allocated through vmalloc(), and the data area's blocks will be allocated individually later when needed. The allocated data area block memory will be managed via radix tree. For now the bitmap still be the most efficient way to search and manage the block index, this could be update later. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:05 +03:00
info->mem[0].memtype = UIO_MEM_NONE;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
info->irqcontrol = tcmu_irqcontrol;
info->irq = UIO_IRQ_CUSTOM;
info->mmap = tcmu_mmap;
info->open = tcmu_open;
info->release = tcmu_release;
ret = uio_register_device(tcmu_root_device, info);
if (ret)
goto err_register;
/* User can set hw_block_size before enable the device */
if (dev->dev_attrib.hw_block_size == 0)
dev->dev_attrib.hw_block_size = 512;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
/* Other attributes can be configured in userspace */
if (!dev->dev_attrib.hw_max_sectors)
dev->dev_attrib.hw_max_sectors = 128;
if (!dev->dev_attrib.emulate_write_cache)
dev->dev_attrib.emulate_write_cache = 0;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
dev->dev_attrib.hw_queue_depth = 128;
/* If user didn't explicitly disable netlink reply support, use
* module scope setting.
*/
if (udev->nl_reply_supported >= 0)
udev->nl_reply_supported = tcmu_kern_cmd_reply_supported;
/*
* Get a ref incase userspace does a close on the uio device before
* LIO has initiated tcmu_free_device.
*/
kref_get(&udev->kref);
ret = tcmu_send_dev_add_event(udev);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
if (ret)
goto err_netlink;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
mutex_lock(&root_udev_mutex);
list_add(&udev->node, &root_udev);
mutex_unlock(&root_udev_mutex);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return 0;
err_netlink:
kref_put(&udev->kref, tcmu_dev_kref_release);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
uio_unregister_device(&udev->uio_info);
err_register:
vfree(udev->mb_addr);
tcmu: fix crash when removing the tcmu device Before the nl REMOVE msg has been sent to the userspace, the ring's and other resources have been released, but the userspace maybe still using them. And then we can see the crash messages like: ring broken, not handling completions BUG: unable to handle kernel paging request at ffffffffffffffd0 IP: tcmu_handle_completions+0x134/0x2f0 [target_core_user] PGD 11bdc0c067 P4D 11bdc0c067 PUD 11bdc0e067 PMD 0 Oops: 0000 [#1] SMP cmd_id not found, ring is broken RIP: 0010:tcmu_handle_completions+0x134/0x2f0 [target_core_user] RSP: 0018:ffffb8a2d8983d88 EFLAGS: 00010296 RAX: 0000000000000000 RBX: ffffb8a2aaa4e000 RCX: 00000000ffffffff RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000 RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000220 R10: 0000000076c71401 R11: ffff8d2e76c713f0 R12: ffffb8a2aad56bc0 R13: 000000000000001c R14: ffff8d2e32c90000 R15: ffff8d2e76c713f0 FS: 00007f411ffff700(0000) GS:ffff8d1e7fdc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffffffffffffd0 CR3: 0000001027070000 CR4: 00000000001406e0 Call Trace: ? tcmu_irqcontrol+0x2a/0x40 [target_core_user] ? uio_write+0x7b/0xc0 [uio] ? __vfs_write+0x37/0x150 ? __getnstimeofday64+0x3b/0xd0 ? vfs_write+0xb2/0x1b0 ? syscall_trace_enter+0x1d0/0x2b0 ? SyS_write+0x55/0xc0 ? do_syscall_64+0x67/0x150 ? entry_SYSCALL64_slow_path+0x25/0x25 Code: 41 5d 41 5e 41 5f 5d c3 83 f8 01 0f 85 cf 01 00 00 48 8b 7d d0 e8 dd 5c 1d f3 41 0f b7 74 24 04 48 8b 7d c8 31 d2 e8 5c c7 1b f3 <48> 8b 7d d0 49 89 c7 c6 07 00 0f 1f 40 00 4d 85 ff 0f 84 82 01 RIP: tcmu_handle_completions+0x134/0x2f0 [target_core_user] RSP: ffffb8a2d8983d88 CR2: ffffffffffffffd0 And the crash also could happen in tcmu_page_fault and other places. Signed-off-by: Zhang Zhuoyu <zhangzhuoyu@cmss.chinamobile.com> Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Reviewed-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-09-14 04:30:05 +03:00
udev->mb_addr = NULL;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
err_vzalloc:
bitmap_free(udev->data_bitmap);
udev->data_bitmap = NULL;
err_bitmap_alloc:
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
kfree(info->name);
info->name = NULL;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return ret;
}
static void tcmu_free_device(struct se_device *dev)
{
struct tcmu_dev *udev = TCMU_DEV(dev);
/* release ref from init */
kref_put(&udev->kref, tcmu_dev_kref_release);
}
static void tcmu_destroy_device(struct se_device *dev)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
struct tcmu_dev *udev = TCMU_DEV(dev);
del_timer_sync(&udev->cmd_timer);
del_timer_sync(&udev->qfull_timer);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
mutex_lock(&root_udev_mutex);
list_del(&udev->node);
mutex_unlock(&root_udev_mutex);
tcmu_send_dev_remove_event(udev);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
uio_unregister_device(&udev->uio_info);
/* release ref from configure */
kref_put(&udev->kref, tcmu_dev_kref_release);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
static void tcmu_unblock_dev(struct tcmu_dev *udev)
{
mutex_lock(&udev->cmdr_lock);
clear_bit(TCMU_DEV_BIT_BLOCKED, &udev->flags);
mutex_unlock(&udev->cmdr_lock);
}
static void tcmu_block_dev(struct tcmu_dev *udev)
{
mutex_lock(&udev->cmdr_lock);
if (test_and_set_bit(TCMU_DEV_BIT_BLOCKED, &udev->flags))
goto unlock;
/* complete IO that has executed successfully */
tcmu_handle_completions(udev);
/* fail IO waiting to be queued */
run_qfull_queue(udev, true);
unlock:
mutex_unlock(&udev->cmdr_lock);
}
static void tcmu_reset_ring(struct tcmu_dev *udev, u8 err_level)
{
struct tcmu_mailbox *mb;
struct tcmu_cmd *cmd;
unsigned long i;
mutex_lock(&udev->cmdr_lock);
xa_for_each(&udev->commands, i, cmd) {
pr_debug("removing cmd %u on dev %s from ring %s\n",
cmd->cmd_id, udev->name,
test_bit(TCMU_CMD_BIT_EXPIRED, &cmd->flags) ?
"(is expired)" :
(test_bit(TCMU_CMD_BIT_KEEP_BUF, &cmd->flags) ?
"(is keep buffer)" : ""));
xa_erase(&udev->commands, i);
if (!test_bit(TCMU_CMD_BIT_EXPIRED, &cmd->flags) &&
!test_bit(TCMU_CMD_BIT_KEEP_BUF, &cmd->flags)) {
WARN_ON(!cmd->se_cmd);
list_del_init(&cmd->queue_entry);
scsi: target: tcmu: Fix use-after-free of se_cmd->priv Commit a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") modified tcmu_free_cmd() to set NULL to priv pointer in se_cmd. However, se_cmd can be already freed by work queue triggered in target_complete_cmd(). This caused BUG KASAN use-after-free [1]. To fix the bug, do not touch priv pointer in tcmu_free_cmd(). Instead, set NULL to priv pointer before target_complete_cmd() calls. Also, to avoid unnecessary priv pointer change in tcmu_queue_cmd(), modify priv pointer in the function only when tcmu_free_cmd() is not called. [1] BUG: KASAN: use-after-free in tcmu_handle_completions+0x1172/0x1770 [target_core_user] Write of size 8 at addr ffff88814cf79a40 by task cmdproc-uio0/14842 CPU: 2 PID: 14842 Comm: cmdproc-uio0 Not tainted 5.11.0-rc2 #1 Hardware name: Supermicro Super Server/X10SRL-F, BIOS 3.2 11/22/2019 Call Trace: dump_stack+0x9a/0xcc ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] print_address_description.constprop.0+0x18/0x130 ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] kasan_report.cold+0x7f/0x10e ? tcmu_handle_completions+0x1172/0x1770 [target_core_user] tcmu_handle_completions+0x1172/0x1770 [target_core_user] ? queue_tmr_ring+0x5d0/0x5d0 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 ? uio_vma_fault+0x460/0x460 ? security_file_permission+0x4f/0x440 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 ? __ia32_sys_read+0xb0/0xb0 ? syscall_enter_from_user_mode+0x27/0x70 ? trace_hardirqs_on+0x1c/0x110 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 RIP: 0033:0x7fcf8b61905f Code: 89 54 24 18 48 89 74 24 10 89 7c 24 08 e8 b9 fc ff ff 48 8b 54 24 18 48 8b 74 24 10 41 89 c0 8b 7c 24 08 b8 01 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 44 24 08 e8 0c fd ff ff 48 RSP: 002b:00007fcf7b3e6c30 EFLAGS: 00000293 ORIG_RAX: 0000000000000001 RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcf8b61905f RDX: 0000000000000004 RSI: 00007fcf7b3e6c78 RDI: 000000000000000c RBP: 00007fcf7b3e6c80 R08: 0000000000000000 R09: 00007fcf7b3e6aa8 R10: 000000000b01c000 R11: 0000000000000293 R12: 00007ffe0c32a52e R13: 00007ffe0c32a52f R14: 0000000000000000 R15: 00007fcf7b3e7640 Allocated by task 383: kasan_save_stack+0x1b/0x40 ____kasan_kmalloc.constprop.0+0x84/0xa0 kmem_cache_alloc+0x142/0x330 tcm_loop_queuecommand+0x2a/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Freed by task 11655: kasan_save_stack+0x1b/0x40 kasan_set_track+0x1c/0x30 kasan_set_free_info+0x20/0x30 ____kasan_slab_free+0xec/0x120 slab_free_freelist_hook+0x53/0x160 kmem_cache_free+0xf4/0x5c0 target_release_cmd_kref+0x3ea/0x9e0 [target_core_mod] transport_generic_free_cmd+0x28b/0x2f0 [target_core_mod] target_complete_ok_work+0x250/0xac0 [target_core_mod] process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 Last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcmu_handle_completions+0xad0/0x1770 [target_core_user] tcmu_irqcontrol+0x28/0x60 [target_core_user] uio_write+0x155/0x230 vfs_write+0x1ce/0x860 ksys_write+0xe9/0x1b0 do_syscall_64+0x33/0x40 entry_SYSCALL_64_after_hwframe+0x44/0xa9 Second to last potentially related work creation: kasan_save_stack+0x1b/0x40 kasan_record_aux_stack+0xa3/0xb0 insert_work+0x48/0x2e0 __queue_work+0x4e8/0xdf0 queue_work_on+0x78/0x80 tcm_loop_queuecommand+0x1c3/0x4e0 [tcm_loop] scsi_queue_rq+0x12ec/0x2d20 blk_mq_dispatch_rq_list+0x30a/0x1db0 __blk_mq_do_dispatch_sched+0x326/0x830 __blk_mq_sched_dispatch_requests+0x2c8/0x3f0 blk_mq_sched_dispatch_requests+0xca/0x120 __blk_mq_run_hw_queue+0x93/0xe0 process_one_work+0x7b6/0x1290 worker_thread+0x590/0xf80 kthread+0x362/0x430 ret_from_fork+0x22/0x30 The buggy address belongs to the object at ffff88814cf79800 which belongs to the cache tcm_loop_cmd_cache of size 896. Link: https://lore.kernel.org/r/20210113024508.1264992-1-shinichiro.kawasaki@wdc.com Fixes: a35129024e88 ("scsi: target: tcmu: Use priv pointer in se_cmd") Cc: stable@vger.kernel.org # v5.9+ Acked-by: Bodo Stroesser <bostroesser@gmail.com> Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com> Signed-off-by: Martin K. Petersen <martin.petersen@oracle.com>
2021-01-13 05:45:08 +03:00
cmd->se_cmd->priv = NULL;
if (err_level == 1) {
/*
* Userspace was not able to start the
* command or it is retryable.
*/
target_complete_cmd(cmd->se_cmd, SAM_STAT_BUSY);
} else {
/* hard failure */
target_complete_cmd(cmd->se_cmd,
SAM_STAT_CHECK_CONDITION);
}
}
tcmu_cmd_free_data(cmd, cmd->dbi_cnt);
tcmu_free_cmd(cmd);
}
mb = udev->mb_addr;
tcmu_flush_dcache_range(mb, sizeof(*mb));
pr_debug("mb last %u head %u tail %u\n", udev->cmdr_last_cleaned,
mb->cmd_tail, mb->cmd_head);
udev->cmdr_last_cleaned = 0;
mb->cmd_tail = 0;
mb->cmd_head = 0;
tcmu_flush_dcache_range(mb, sizeof(*mb));
clear_bit(TCMU_DEV_BIT_BROKEN, &udev->flags);
del_timer(&udev->cmd_timer);
2020-07-26 18:35:09 +03:00
/*
* ring is empty and qfull queue never contains aborted commands.
* So TMRs in tmr queue do not contain relevant cmd_ids.
* After a ring reset userspace should do a fresh start, so
* even LUN RESET message is no longer relevant.
* Therefore remove all TMRs from qfull queue
*/
tcmu_remove_all_queued_tmr(udev);
run_qfull_queue(udev, false);
mutex_unlock(&udev->cmdr_lock);
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
enum {
Opt_dev_config, Opt_dev_size, Opt_hw_block_size, Opt_hw_max_sectors,
Opt_nl_reply_supported, Opt_max_data_area_mb, Opt_data_pages_per_blk,
Opt_err,
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
};
static match_table_t tokens = {
{Opt_dev_config, "dev_config=%s"},
{Opt_dev_size, "dev_size=%s"},
{Opt_hw_block_size, "hw_block_size=%d"},
{Opt_hw_max_sectors, "hw_max_sectors=%d"},
{Opt_nl_reply_supported, "nl_reply_supported=%d"},
{Opt_max_data_area_mb, "max_data_area_mb=%d"},
{Opt_data_pages_per_blk, "data_pages_per_blk=%d"},
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{Opt_err, NULL}
};
static int tcmu_set_dev_attrib(substring_t *arg, u32 *dev_attrib)
{
int val, ret;
ret = match_int(arg, &val);
if (ret < 0) {
pr_err("match_int() failed for dev attrib. Error %d.\n",
ret);
return ret;
}
if (val <= 0) {
pr_err("Invalid dev attrib value %d. Must be greater than zero.\n",
val);
return -EINVAL;
}
*dev_attrib = val;
return 0;
}
static int tcmu_set_max_blocks_param(struct tcmu_dev *udev, substring_t *arg)
{
int val, ret;
uint32_t pages_per_blk = udev->data_pages_per_blk;
ret = match_int(arg, &val);
if (ret < 0) {
pr_err("match_int() failed for max_data_area_mb=. Error %d.\n",
ret);
return ret;
}
if (val <= 0) {
pr_err("Invalid max_data_area %d.\n", val);
return -EINVAL;
}
if (val > TCMU_PAGES_TO_MBS(tcmu_global_max_pages)) {
pr_err("%d is too large. Adjusting max_data_area_mb to global limit of %u\n",
val, TCMU_PAGES_TO_MBS(tcmu_global_max_pages));
val = TCMU_PAGES_TO_MBS(tcmu_global_max_pages);
}
if (TCMU_MBS_TO_PAGES(val) < pages_per_blk) {
pr_err("Invalid max_data_area %d (%zu pages): smaller than data_pages_per_blk (%u pages).\n",
val, TCMU_MBS_TO_PAGES(val), pages_per_blk);
return -EINVAL;
}
mutex_lock(&udev->cmdr_lock);
if (udev->data_bitmap) {
pr_err("Cannot set max_data_area_mb after it has been enabled.\n");
ret = -EINVAL;
goto unlock;
}
udev->data_area_mb = val;
udev->max_blocks = TCMU_MBS_TO_PAGES(val) / pages_per_blk;
unlock:
mutex_unlock(&udev->cmdr_lock);
return ret;
}
static int tcmu_set_data_pages_per_blk(struct tcmu_dev *udev, substring_t *arg)
{
int val, ret;
ret = match_int(arg, &val);
if (ret < 0) {
pr_err("match_int() failed for data_pages_per_blk=. Error %d.\n",
ret);
return ret;
}
if (val > TCMU_MBS_TO_PAGES(udev->data_area_mb)) {
pr_err("Invalid data_pages_per_blk %d: greater than max_data_area_mb %d -> %zd pages).\n",
val, udev->data_area_mb,
TCMU_MBS_TO_PAGES(udev->data_area_mb));
return -EINVAL;
}
mutex_lock(&udev->cmdr_lock);
if (udev->data_bitmap) {
pr_err("Cannot set data_pages_per_blk after it has been enabled.\n");
ret = -EINVAL;
goto unlock;
}
udev->data_pages_per_blk = val;
udev->max_blocks = TCMU_MBS_TO_PAGES(udev->data_area_mb) / val;
unlock:
mutex_unlock(&udev->cmdr_lock);
return ret;
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
static ssize_t tcmu_set_configfs_dev_params(struct se_device *dev,
const char *page, ssize_t count)
{
struct tcmu_dev *udev = TCMU_DEV(dev);
char *orig, *ptr, *opts;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
substring_t args[MAX_OPT_ARGS];
int ret = 0, token;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
opts = kstrdup(page, GFP_KERNEL);
if (!opts)
return -ENOMEM;
orig = opts;
while ((ptr = strsep(&opts, ",\n")) != NULL) {
if (!*ptr)
continue;
token = match_token(ptr, tokens, args);
switch (token) {
case Opt_dev_config:
if (match_strlcpy(udev->dev_config, &args[0],
TCMU_CONFIG_LEN) == 0) {
ret = -EINVAL;
break;
}
pr_debug("TCMU: Referencing Path: %s\n", udev->dev_config);
break;
case Opt_dev_size:
ret = match_u64(&args[0], &udev->dev_size);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
if (ret < 0)
pr_err("match_u64() failed for dev_size=. Error %d.\n",
ret);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
break;
case Opt_hw_block_size:
ret = tcmu_set_dev_attrib(&args[0],
&(dev->dev_attrib.hw_block_size));
break;
case Opt_hw_max_sectors:
ret = tcmu_set_dev_attrib(&args[0],
&(dev->dev_attrib.hw_max_sectors));
break;
case Opt_nl_reply_supported:
ret = match_int(&args[0], &udev->nl_reply_supported);
if (ret < 0)
pr_err("match_int() failed for nl_reply_supported=. Error %d.\n",
ret);
break;
case Opt_max_data_area_mb:
ret = tcmu_set_max_blocks_param(udev, &args[0]);
break;
case Opt_data_pages_per_blk:
ret = tcmu_set_data_pages_per_blk(udev, &args[0]);
break;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
default:
break;
}
if (ret)
break;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
kfree(orig);
return (!ret) ? count : ret;
}
static ssize_t tcmu_show_configfs_dev_params(struct se_device *dev, char *b)
{
struct tcmu_dev *udev = TCMU_DEV(dev);
ssize_t bl = 0;
bl = sprintf(b + bl, "Config: %s ",
udev->dev_config[0] ? udev->dev_config : "NULL");
bl += sprintf(b + bl, "Size: %llu ", udev->dev_size);
bl += sprintf(b + bl, "MaxDataAreaMB: %u ", udev->data_area_mb);
bl += sprintf(b + bl, "DataPagesPerBlk: %u\n", udev->data_pages_per_blk);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return bl;
}
static sector_t tcmu_get_blocks(struct se_device *dev)
{
struct tcmu_dev *udev = TCMU_DEV(dev);
return div_u64(udev->dev_size - dev->dev_attrib.block_size,
dev->dev_attrib.block_size);
}
static sense_reason_t
tcmu_parse_cdb(struct se_cmd *cmd)
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
{
return passthrough_parse_cdb(cmd, tcmu_queue_cmd);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
}
static ssize_t tcmu_cmd_time_out_show(struct config_item *item, char *page)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
return snprintf(page, PAGE_SIZE, "%lu\n", udev->cmd_time_out / MSEC_PER_SEC);
}
static ssize_t tcmu_cmd_time_out_store(struct config_item *item, const char *page,
size_t count)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = container_of(da->da_dev,
struct tcmu_dev, se_dev);
u32 val;
int ret;
if (da->da_dev->export_count) {
pr_err("Unable to set tcmu cmd_time_out while exports exist\n");
return -EINVAL;
}
ret = kstrtou32(page, 0, &val);
if (ret < 0)
return ret;
udev->cmd_time_out = val * MSEC_PER_SEC;
return count;
}
CONFIGFS_ATTR(tcmu_, cmd_time_out);
static ssize_t tcmu_qfull_time_out_show(struct config_item *item, char *page)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
return snprintf(page, PAGE_SIZE, "%ld\n", udev->qfull_time_out <= 0 ?
udev->qfull_time_out :
udev->qfull_time_out / MSEC_PER_SEC);
}
static ssize_t tcmu_qfull_time_out_store(struct config_item *item,
const char *page, size_t count)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
s32 val;
int ret;
ret = kstrtos32(page, 0, &val);
if (ret < 0)
return ret;
if (val >= 0) {
udev->qfull_time_out = val * MSEC_PER_SEC;
} else if (val == -1) {
udev->qfull_time_out = val;
} else {
printk(KERN_ERR "Invalid qfull timeout value %d\n", val);
return -EINVAL;
}
return count;
}
CONFIGFS_ATTR(tcmu_, qfull_time_out);
static ssize_t tcmu_max_data_area_mb_show(struct config_item *item, char *page)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
return snprintf(page, PAGE_SIZE, "%u\n", udev->data_area_mb);
}
CONFIGFS_ATTR_RO(tcmu_, max_data_area_mb);
static ssize_t tcmu_data_pages_per_blk_show(struct config_item *item,
char *page)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
return snprintf(page, PAGE_SIZE, "%u\n", udev->data_pages_per_blk);
}
CONFIGFS_ATTR_RO(tcmu_, data_pages_per_blk);
static ssize_t tcmu_dev_config_show(struct config_item *item, char *page)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
return snprintf(page, PAGE_SIZE, "%s\n", udev->dev_config);
}
static int tcmu_send_dev_config_event(struct tcmu_dev *udev,
const char *reconfig_data)
{
struct sk_buff *skb = NULL;
void *msg_header = NULL;
int ret = 0;
ret = tcmu_netlink_event_init(udev, TCMU_CMD_RECONFIG_DEVICE,
&skb, &msg_header);
if (ret < 0)
return ret;
ret = nla_put_string(skb, TCMU_ATTR_DEV_CFG, reconfig_data);
if (ret < 0) {
nlmsg_free(skb);
return ret;
}
return tcmu_netlink_event_send(udev, TCMU_CMD_RECONFIG_DEVICE,
skb, msg_header);
}
static ssize_t tcmu_dev_config_store(struct config_item *item, const char *page,
size_t count)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
int ret, len;
len = strlen(page);
if (!len || len > TCMU_CONFIG_LEN - 1)
return -EINVAL;
/* Check if device has been configured before */
if (target_dev_configured(&udev->se_dev)) {
ret = tcmu_send_dev_config_event(udev, page);
if (ret) {
pr_err("Unable to reconfigure device\n");
return ret;
}
strlcpy(udev->dev_config, page, TCMU_CONFIG_LEN);
ret = tcmu_update_uio_info(udev);
if (ret)
return ret;
return count;
}
strlcpy(udev->dev_config, page, TCMU_CONFIG_LEN);
return count;
}
CONFIGFS_ATTR(tcmu_, dev_config);
static ssize_t tcmu_dev_size_show(struct config_item *item, char *page)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
return snprintf(page, PAGE_SIZE, "%llu\n", udev->dev_size);
}
static int tcmu_send_dev_size_event(struct tcmu_dev *udev, u64 size)
{
struct sk_buff *skb = NULL;
void *msg_header = NULL;
int ret = 0;
ret = tcmu_netlink_event_init(udev, TCMU_CMD_RECONFIG_DEVICE,
&skb, &msg_header);
if (ret < 0)
return ret;
ret = nla_put_u64_64bit(skb, TCMU_ATTR_DEV_SIZE,
size, TCMU_ATTR_PAD);
if (ret < 0) {
nlmsg_free(skb);
return ret;
}
return tcmu_netlink_event_send(udev, TCMU_CMD_RECONFIG_DEVICE,
skb, msg_header);
}
static ssize_t tcmu_dev_size_store(struct config_item *item, const char *page,
size_t count)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
u64 val;
int ret;
ret = kstrtou64(page, 0, &val);
if (ret < 0)
return ret;
/* Check if device has been configured before */
if (target_dev_configured(&udev->se_dev)) {
ret = tcmu_send_dev_size_event(udev, val);
if (ret) {
pr_err("Unable to reconfigure device\n");
return ret;
}
}
udev->dev_size = val;
return count;
}
CONFIGFS_ATTR(tcmu_, dev_size);
static ssize_t tcmu_nl_reply_supported_show(struct config_item *item,
char *page)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
return snprintf(page, PAGE_SIZE, "%d\n", udev->nl_reply_supported);
}
static ssize_t tcmu_nl_reply_supported_store(struct config_item *item,
const char *page, size_t count)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
s8 val;
int ret;
ret = kstrtos8(page, 0, &val);
if (ret < 0)
return ret;
udev->nl_reply_supported = val;
return count;
}
CONFIGFS_ATTR(tcmu_, nl_reply_supported);
static ssize_t tcmu_emulate_write_cache_show(struct config_item *item,
char *page)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
return snprintf(page, PAGE_SIZE, "%i\n", da->emulate_write_cache);
}
static int tcmu_send_emulate_write_cache(struct tcmu_dev *udev, u8 val)
{
struct sk_buff *skb = NULL;
void *msg_header = NULL;
int ret = 0;
ret = tcmu_netlink_event_init(udev, TCMU_CMD_RECONFIG_DEVICE,
&skb, &msg_header);
if (ret < 0)
return ret;
ret = nla_put_u8(skb, TCMU_ATTR_WRITECACHE, val);
if (ret < 0) {
nlmsg_free(skb);
return ret;
}
return tcmu_netlink_event_send(udev, TCMU_CMD_RECONFIG_DEVICE,
skb, msg_header);
}
static ssize_t tcmu_emulate_write_cache_store(struct config_item *item,
const char *page, size_t count)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
u8 val;
int ret;
ret = kstrtou8(page, 0, &val);
if (ret < 0)
return ret;
/* Check if device has been configured before */
if (target_dev_configured(&udev->se_dev)) {
ret = tcmu_send_emulate_write_cache(udev, val);
if (ret) {
pr_err("Unable to reconfigure device\n");
return ret;
}
}
da->emulate_write_cache = val;
return count;
}
CONFIGFS_ATTR(tcmu_, emulate_write_cache);
static ssize_t tcmu_tmr_notification_show(struct config_item *item, char *page)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
return snprintf(page, PAGE_SIZE, "%i\n",
test_bit(TCMU_DEV_BIT_TMR_NOTIFY, &udev->flags));
}
static ssize_t tcmu_tmr_notification_store(struct config_item *item,
const char *page, size_t count)
{
struct se_dev_attrib *da = container_of(to_config_group(item),
struct se_dev_attrib, da_group);
struct tcmu_dev *udev = TCMU_DEV(da->da_dev);
u8 val;
int ret;
ret = kstrtou8(page, 0, &val);
if (ret < 0)
return ret;
if (val > 1)
return -EINVAL;
if (val)
set_bit(TCMU_DEV_BIT_TMR_NOTIFY, &udev->flags);
else
clear_bit(TCMU_DEV_BIT_TMR_NOTIFY, &udev->flags);
return count;
}
CONFIGFS_ATTR(tcmu_, tmr_notification);
static ssize_t tcmu_block_dev_show(struct config_item *item, char *page)
{
struct se_device *se_dev = container_of(to_config_group(item),
struct se_device,
dev_action_group);
struct tcmu_dev *udev = TCMU_DEV(se_dev);
if (test_bit(TCMU_DEV_BIT_BLOCKED, &udev->flags))
return snprintf(page, PAGE_SIZE, "%s\n", "blocked");
else
return snprintf(page, PAGE_SIZE, "%s\n", "unblocked");
}
static ssize_t tcmu_block_dev_store(struct config_item *item, const char *page,
size_t count)
{
struct se_device *se_dev = container_of(to_config_group(item),
struct se_device,
dev_action_group);
struct tcmu_dev *udev = TCMU_DEV(se_dev);
u8 val;
int ret;
if (!target_dev_configured(&udev->se_dev)) {
pr_err("Device is not configured.\n");
return -EINVAL;
}
ret = kstrtou8(page, 0, &val);
if (ret < 0)
return ret;
if (val > 1) {
pr_err("Invalid block value %d\n", val);
return -EINVAL;
}
if (!val)
tcmu_unblock_dev(udev);
else
tcmu_block_dev(udev);
return count;
}
CONFIGFS_ATTR(tcmu_, block_dev);
static ssize_t tcmu_reset_ring_store(struct config_item *item, const char *page,
size_t count)
{
struct se_device *se_dev = container_of(to_config_group(item),
struct se_device,
dev_action_group);
struct tcmu_dev *udev = TCMU_DEV(se_dev);
u8 val;
int ret;
if (!target_dev_configured(&udev->se_dev)) {
pr_err("Device is not configured.\n");
return -EINVAL;
}
ret = kstrtou8(page, 0, &val);
if (ret < 0)
return ret;
if (val != 1 && val != 2) {
pr_err("Invalid reset ring value %d\n", val);
return -EINVAL;
}
tcmu_reset_ring(udev, val);
return count;
}
CONFIGFS_ATTR_WO(tcmu_, reset_ring);
static ssize_t tcmu_free_kept_buf_store(struct config_item *item, const char *page,
size_t count)
{
struct se_device *se_dev = container_of(to_config_group(item),
struct se_device,
dev_action_group);
struct tcmu_dev *udev = TCMU_DEV(se_dev);
struct tcmu_cmd *cmd;
u16 cmd_id;
int ret;
if (!target_dev_configured(&udev->se_dev)) {
pr_err("Device is not configured.\n");
return -EINVAL;
}
ret = kstrtou16(page, 0, &cmd_id);
if (ret < 0)
return ret;
mutex_lock(&udev->cmdr_lock);
{
XA_STATE(xas, &udev->commands, cmd_id);
xas_lock(&xas);
cmd = xas_load(&xas);
if (!cmd) {
pr_err("free_kept_buf: cmd_id %d not found\n", cmd_id);
count = -EINVAL;
xas_unlock(&xas);
goto out_unlock;
}
if (!test_bit(TCMU_CMD_BIT_KEEP_BUF, &cmd->flags)) {
pr_err("free_kept_buf: cmd_id %d was not completed with KEEP_BUF\n",
cmd_id);
count = -EINVAL;
xas_unlock(&xas);
goto out_unlock;
}
xas_store(&xas, NULL);
xas_unlock(&xas);
}
tcmu_cmd_free_data(cmd, cmd->dbi_cnt);
tcmu_free_cmd(cmd);
/*
* We only freed data space, not ring space. Therefore we dont call
* run_tmr_queue, but call run_qfull_queue if tmr_list is empty.
*/
if (list_empty(&udev->tmr_queue))
run_qfull_queue(udev, false);
out_unlock:
mutex_unlock(&udev->cmdr_lock);
return count;
}
CONFIGFS_ATTR_WO(tcmu_, free_kept_buf);
static struct configfs_attribute *tcmu_attrib_attrs[] = {
&tcmu_attr_cmd_time_out,
&tcmu_attr_qfull_time_out,
&tcmu_attr_max_data_area_mb,
&tcmu_attr_data_pages_per_blk,
&tcmu_attr_dev_config,
&tcmu_attr_dev_size,
&tcmu_attr_emulate_write_cache,
&tcmu_attr_tmr_notification,
&tcmu_attr_nl_reply_supported,
NULL,
};
static struct configfs_attribute **tcmu_attrs;
static struct configfs_attribute *tcmu_action_attrs[] = {
&tcmu_attr_block_dev,
&tcmu_attr_reset_ring,
&tcmu_attr_free_kept_buf,
NULL,
};
static struct target_backend_ops tcmu_ops = {
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
.name = "user",
.owner = THIS_MODULE,
.transport_flags_default = TRANSPORT_FLAG_PASSTHROUGH,
.transport_flags_changeable = TRANSPORT_FLAG_PASSTHROUGH_PGR |
TRANSPORT_FLAG_PASSTHROUGH_ALUA,
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
.attach_hba = tcmu_attach_hba,
.detach_hba = tcmu_detach_hba,
.alloc_device = tcmu_alloc_device,
.configure_device = tcmu_configure_device,
.destroy_device = tcmu_destroy_device,
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
.free_device = tcmu_free_device,
.unplug_device = tcmu_unplug_device,
.plug_device = tcmu_plug_device,
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
.parse_cdb = tcmu_parse_cdb,
2020-07-26 18:35:09 +03:00
.tmr_notify = tcmu_tmr_notify,
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
.set_configfs_dev_params = tcmu_set_configfs_dev_params,
.show_configfs_dev_params = tcmu_show_configfs_dev_params,
.get_device_type = sbc_get_device_type,
.get_blocks = tcmu_get_blocks,
.tb_dev_action_attrs = tcmu_action_attrs,
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
};
static void find_free_blocks(void)
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
{
struct tcmu_dev *udev;
loff_t off;
u32 pages_freed, total_pages_freed = 0;
u32 start, end, block, total_blocks_freed = 0;
if (atomic_read(&global_page_count) <= tcmu_global_max_pages)
return;
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
mutex_lock(&root_udev_mutex);
list_for_each_entry(udev, &root_udev, node) {
mutex_lock(&udev->cmdr_lock);
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
if (!target_dev_configured(&udev->se_dev)) {
mutex_unlock(&udev->cmdr_lock);
continue;
}
/* Try to complete the finished commands first */
2020-07-26 18:35:09 +03:00
if (tcmu_handle_completions(udev))
run_qfull_queue(udev, false);
/* Skip the udevs in idle */
if (!udev->dbi_thresh) {
mutex_unlock(&udev->cmdr_lock);
continue;
}
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
end = udev->dbi_max + 1;
block = find_last_bit(udev->data_bitmap, end);
if (block == udev->dbi_max) {
/*
* The last bit is dbi_max, so it is not possible
* reclaim any blocks.
*/
mutex_unlock(&udev->cmdr_lock);
continue;
} else if (block == end) {
/* The current udev will goto idle state */
udev->dbi_thresh = start = 0;
udev->dbi_max = 0;
} else {
udev->dbi_thresh = start = block + 1;
udev->dbi_max = block;
}
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
/* Here will truncate the data area from off */
off = udev->data_off + (loff_t)start * udev->data_blk_size;
unmap_mapping_range(udev->inode->i_mapping, off, 0, 1);
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
/* Release the block pages */
pages_freed = tcmu_blocks_release(udev, start, end - 1);
mutex_unlock(&udev->cmdr_lock);
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
total_pages_freed += pages_freed;
total_blocks_freed += end - start;
pr_debug("Freed %u pages (total %u) from %u blocks (total %u) from %s.\n",
pages_freed, total_pages_freed, end - start,
total_blocks_freed, udev->name);
}
mutex_unlock(&root_udev_mutex);
if (atomic_read(&global_page_count) > tcmu_global_max_pages)
schedule_delayed_work(&tcmu_unmap_work, msecs_to_jiffies(5000));
}
static void check_timedout_devices(void)
{
struct tcmu_dev *udev, *tmp_dev;
struct tcmu_cmd *cmd, *tmp_cmd;
LIST_HEAD(devs);
spin_lock_bh(&timed_out_udevs_lock);
list_splice_init(&timed_out_udevs, &devs);
list_for_each_entry_safe(udev, tmp_dev, &devs, timedout_entry) {
list_del_init(&udev->timedout_entry);
spin_unlock_bh(&timed_out_udevs_lock);
mutex_lock(&udev->cmdr_lock);
/*
* If cmd_time_out is disabled but qfull is set deadline
* will only reflect the qfull timeout. Ignore it.
*/
if (udev->cmd_time_out) {
list_for_each_entry_safe(cmd, tmp_cmd,
&udev->inflight_queue,
queue_entry) {
tcmu_check_expired_ring_cmd(cmd);
}
tcmu_set_next_deadline(&udev->inflight_queue,
&udev->cmd_timer);
}
list_for_each_entry_safe(cmd, tmp_cmd, &udev->qfull_queue,
queue_entry) {
tcmu_check_expired_queue_cmd(cmd);
}
tcmu_set_next_deadline(&udev->qfull_queue, &udev->qfull_timer);
mutex_unlock(&udev->cmdr_lock);
spin_lock_bh(&timed_out_udevs_lock);
}
spin_unlock_bh(&timed_out_udevs_lock);
}
static void tcmu_unmap_work_fn(struct work_struct *work)
{
check_timedout_devices();
find_free_blocks();
tcmu: Add global data block pool support For each target there will be one ring, when the target number grows larger and larger, it could eventually runs out of the system memories. In this patch for each target ring, currently for the cmd area the size will be fixed to 8MB and for the data area the size will grow from 0 to max 256K * PAGE_SIZE(1G for 4K page size). For all the targets' data areas, they will get empty blocks from the "global data block pool", which has limited to 512K * PAGE_SIZE(2G for 4K page size) for now. When the "global data block pool" has been used up, then any target could wake up the unmap thread routine to shrink other targets' data area memories. And the unmap thread routine will always try to truncate the ring vma from the last using block offset. When user space has touched the data blocks out of tcmu_cmd iov[], the tcmu_page_fault() will try to return one zeroed blocks. Here we move the timeout's tcmu_handle_completions() into unmap thread routine, that's to say when the timeout fired, it will only do the tcmu_check_expired_cmd() and then wake up the unmap thread to do the completions() and then try to shrink its idle memories. Then the cmdr_lock could be a mutex and could simplify this patch because the unmap_mapping_range() or zap_* may go to sleep. Signed-off-by: Xiubo Li <lixiubo@cmss.chinamobile.com> Signed-off-by: Jianfei Hu <hujianfei@cmss.chinamobile.com> Acked-by: Mike Christie <mchristi@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2017-05-02 06:38:06 +03:00
}
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
static int __init tcmu_module_init(void)
{
int ret, i, k, len = 0;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
BUILD_BUG_ON((sizeof(struct tcmu_cmd_entry) % TCMU_OP_ALIGN_SIZE) != 0);
INIT_DELAYED_WORK(&tcmu_unmap_work, tcmu_unmap_work_fn);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
tcmu_cmd_cache = kmem_cache_create("tcmu_cmd_cache",
sizeof(struct tcmu_cmd),
__alignof__(struct tcmu_cmd),
0, NULL);
if (!tcmu_cmd_cache)
return -ENOMEM;
tcmu_root_device = root_device_register("tcm_user");
if (IS_ERR(tcmu_root_device)) {
ret = PTR_ERR(tcmu_root_device);
goto out_free_cache;
}
ret = genl_register_family(&tcmu_genl_family);
if (ret < 0) {
goto out_unreg_device;
}
for (i = 0; passthrough_attrib_attrs[i] != NULL; i++)
len += sizeof(struct configfs_attribute *);
for (i = 0; passthrough_pr_attrib_attrs[i] != NULL; i++)
len += sizeof(struct configfs_attribute *);
for (i = 0; tcmu_attrib_attrs[i] != NULL; i++)
len += sizeof(struct configfs_attribute *);
len += sizeof(struct configfs_attribute *);
tcmu_attrs = kzalloc(len, GFP_KERNEL);
if (!tcmu_attrs) {
ret = -ENOMEM;
goto out_unreg_genl;
}
for (i = 0; passthrough_attrib_attrs[i] != NULL; i++)
tcmu_attrs[i] = passthrough_attrib_attrs[i];
for (k = 0; passthrough_pr_attrib_attrs[k] != NULL; k++)
tcmu_attrs[i++] = passthrough_pr_attrib_attrs[k];
for (k = 0; tcmu_attrib_attrs[k] != NULL; k++)
tcmu_attrs[i++] = tcmu_attrib_attrs[k];
tcmu_ops.tb_dev_attrib_attrs = tcmu_attrs;
ret = transport_backend_register(&tcmu_ops);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
if (ret)
goto out_attrs;
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
return 0;
out_attrs:
kfree(tcmu_attrs);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
out_unreg_genl:
genl_unregister_family(&tcmu_genl_family);
out_unreg_device:
root_device_unregister(tcmu_root_device);
out_free_cache:
kmem_cache_destroy(tcmu_cmd_cache);
return ret;
}
static void __exit tcmu_module_exit(void)
{
cancel_delayed_work_sync(&tcmu_unmap_work);
target_backend_unregister(&tcmu_ops);
kfree(tcmu_attrs);
target: Add a user-passthrough backstore Add a LIO storage engine that presents commands to userspace for execution. This would allow more complex backstores to be implemented out-of-kernel, and also make experimentation a-la FUSE (but at the SCSI level -- "SUSE"?) possible. It uses a mmap()able UIO device per LUN to share a command ring and data area. The commands are raw SCSI CDBs and iovs for in/out data. The command ring is also reused for returning scsi command status and optional sense data. This implementation is based on Shaohua Li's earlier version but heavily modified. Differences include: * Shared memory allocated by kernel, not locked-down user pages * Single ring for command request and response * Offsets instead of embedded pointers * Generic SCSI CDB passthrough instead of per-cmd specialization in ring format. * Uses UIO device instead of anon_file passed in mailbox. * Optional in-kernel handling of some commands. The main reason for these differences is to permit greater resiliency if the user process dies or hangs. Things not yet implemented (on purpose): * Zero copy. The data area is flexible enough to allow page flipping or backend-allocated pages to be used by fabrics, but it's not clear these are performance wins. Can come later. * Out-of-order command completion by userspace. Possible to add by just allowing userspace to change cmd_id in rsp cmd entries, but currently not supported. * No locks between kernel cmd submission and completion routines. Sounds like it's possible, but this can come later. * Sparse allocation of mmaped area. Current code vmallocs the whole thing. If the mapped area was larger and not fully mapped then the driver would have more freedom to change cmd and data area sizes based on demand. Current code open issues: * The use of idrs may be overkill -- we maybe can replace them with a simple counter to generate cmd_ids, and a hash table to get a cmd_id's associated pointer. * Use of a free-running counter for cmd ring instead of explicit modulo math. This would require power-of-2 cmd ring size. (Add kconfig depends NET - Randy) Signed-off-by: Andy Grover <agrover@redhat.com> Signed-off-by: Nicholas Bellinger <nab@linux-iscsi.org>
2014-10-02 03:07:05 +04:00
genl_unregister_family(&tcmu_genl_family);
root_device_unregister(tcmu_root_device);
kmem_cache_destroy(tcmu_cmd_cache);
}
MODULE_DESCRIPTION("TCM USER subsystem plugin");
MODULE_AUTHOR("Shaohua Li <shli@kernel.org>");
MODULE_AUTHOR("Andy Grover <agrover@redhat.com>");
MODULE_LICENSE("GPL");
module_init(tcmu_module_init);
module_exit(tcmu_module_exit);