There is an extra semicolon in z_erofs_vle_unzip_all, remove it.
Reported-by: Julia Lawall <julia.lawall@lip6.fr>
Signed-off-by: zhong jiang <zhongjiang@huawei.com>
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
As Stephen Rothwell reported:
"After merging the staging tree, today's linux-next build (x86_64
allmodconfig) failed like this:
drivers/staging/erofs/super.c: In function 'erofs_read_super':
drivers/staging/erofs/super.c:343:17: error: 'MS_RDONLY' undeclared (first use in this function); did you mean 'IS_RDONLY'?
sb->s_flags |= MS_RDONLY | MS_NOATIME;
^~~~~~~~~
IS_RDONLY
drivers/staging/erofs/super.c:343:17: note: each undeclared identifier is reported only once for each function it appears in
drivers/staging/erofs/super.c:343:29: error: 'MS_NOATIME' undeclared (first use in this function); did you mean 'S_NOATIME'?
sb->s_flags |= MS_RDONLY | MS_NOATIME;
^~~~~~~~~~
S_NOATIME
drivers/staging/erofs/super.c: In function 'erofs_mount':
drivers/staging/erofs/super.c:501:10: warning: passing argument 5 of 'mount_bdev' makes integer from pointer without a cast [-Wint-conversion]
&priv, erofs_fill_super);
^~~~~~~~~~~~~~~~
In file included from include/linux/buffer_head.h:12:0,
from drivers/staging/erofs/super.c:14:
include/linux/fs.h:2151:23: note: expected 'size_t {aka long unsigned int}' but argument is of type 'int (*)(struct super_block *, void *, int)'
extern struct dentry *mount_bdev(struct file_system_type *fs_type,
^~~~~~~~~~
drivers/staging/erofs/super.c:500:9: error: too few arguments to function 'mount_bdev'
return mount_bdev(fs_type, flags, dev_name,
^~~~~~~~~~
In file included from include/linux/buffer_head.h:12:0,
from drivers/staging/erofs/super.c:14:
include/linux/fs.h:2151:23: note: declared here
extern struct dentry *mount_bdev(struct file_system_type *fs_type,
^~~~~~~~~~
drivers/staging/erofs/super.c: At top level:
drivers/staging/erofs/super.c:518:20: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
.mount = erofs_mount,
^~~~~~~~~~~
drivers/staging/erofs/super.c:518:20: note: (near initialization for 'erofs_fs_type.mount')
drivers/staging/erofs/super.c: In function 'erofs_remount':
drivers/staging/erofs/super.c:630:12: error: 'MS_RDONLY' undeclared (first use in this function); did you mean 'IS_RDONLY'?
*flags |= MS_RDONLY;
^~~~~~~~~
IS_RDONLY
drivers/staging/erofs/super.c: At top level:
drivers/staging/erofs/super.c:640:16: error: initialization from incompatible pointer type [-Werror=incompatible-pointer-types]
.remount_fs = erofs_remount,
^~~~~~~~~~~~~
Caused by various commits creating erofs in the staging tree interacting
with various commits redoing the mount infrastructure in the vfs tree.
I have disabed CONFIG_EROFS_FS for now:"
The reason of compiling error is:
Since -next collects and merges developing patches including common vfs
stuff from multi-trees, but those patches didn't cover erofs, such as:
('vfs: Suppress MS_* flag defs within the kernel unless explicitly enabled")
https://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git/commit/?h=for-next&id=109b45090d7d3ce2797bb1ef7f70eead5bfe0ff3
("vfs: Require specification of size of mount data for internal mounts")
https://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs.git/commit/?h=for-next&id=0a191e4505a4f255e6513b49426213da69bf0e80
Above vfs related patches has not been merged in staging tree, if we
submit those erofs patches to staging mailing list and after including
them in staging-{test,nexts} tree, it can easily cause compiling error.
We worked out some patches to adjust those vfs change, but now we just
submit them to -next tree temporarily to avoid compiling error.
For potentail conflict in between erofs and vfs changes in incoming
merge window, Stephen suggested that we can disable CONFIG_EROFS_FS
temporarily to pass merge window, and after that we can do restore by
reenabling CONFIG_EROFS_FS and applying those fixing patches. Also
Greg confirmed this solution.
So, let's disable compiling erofs for a while.
Suggested-by: Stephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Reviewed-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
There is no need to '#if CONFIG_EROFS_FS_XATTR' in xattr.c,
let's remove it.
Reviewed-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
There is a type mismatch in the definition of
Z_EROFS_VLE_VMAP_ONSTACK_PAGES, let's fix it.
Link: https://lists.01.org/pipermail/kbuild-all/2018-July/050707.html
Reported-by: kbuild test robot <lkp@intel.com>
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This patch adds a TODO to list the things to be done, and
the relevant info to MAINTAINERS so we can take all the blame :)
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This patch adds an optional choice which can be
enabled by users in order to cache both incomplete
ends of compressed clusters as a complement to
the in-place decompression in order to boost random
read, but it costs more memory than the in-place
decompression only.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This patch introduces the basic in-place VLE decompression
implementation for the erofs file system.
Compared with fixed-sized input compression, it implements
what we call 'the variable-length extent compression' which
specifies the same output size for each compression block
to make the full use of IO bandwidth (which means almost
all data from block device can be directly used for decomp-
ression), improve the real (rather than just via data caching,
which costs more memory) random read and keep the relatively
lower compression ratios (it saves more storage space than
fixed-sized input compression which is also configured with
the same input block size), as illustrated below:
|--- variable-length extent ---|------ VLE ------|--- VLE ---|
/> clusterofs /> clusterofs /> clusterofs /> clusterofs
++---|-------++-----------++---------|-++-----------++-|---------++-|
...|| | || || | || || | || | ... original data
++---|-------++-----------++---------|-++-----------++-|---------++-|
++->cluster<-++->cluster<-++->cluster<-++->cluster<-++->cluster<-++
size size size size size
\ / / /
\ / / /
\ / / /
++-----------++-----------++-----------++
... || || || || ... compressed clusters
++-----------++-----------++-----------++
++->cluster<-++->cluster<-++->cluster<-++
size size size
The main point of 'in-place' refers to the decompression mode:
Instead of allocating independent compressed pages and data
structures, it reuses the allocated file cache pages at most
to store its compressed data and the corresponding pagevec in
a time-sharing approach by default, which will be useful for
low memory scenario.
In the end, unlike the other filesystems with (de)compression
support using a relatively large compression block size, which
reads and decompresses >= 128KB at once, and gains a more
good-looking random read (In fact it collects small random reads
into large sequential reads and caches all decompressed data
in memory, but it is unacceptable especially for embedded devices
with limited memory, and it is not the real random read), we
select a universal small-sized 4KB compressed cluster, which is
the smallest page size for most architectures, and all compressed
clusters can be read and decompressed independently, which ensures
random read number for all use cases.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This patch introduces another concept used by the unzip
subsystem called 'workstation'. It can be seen as a sparse
array that stores pointers pointed to data structures
related to the corresponding physical blocks.
All lookup cases are protected by RCU read lock. Besides,
reference count and spin_lock are also introduced to
manage its lifetime and serialize all update operations.
'workstation' is currently implemented on the in-kernel
radix tree approach for backward compatibility.
With the evolution of linux kernel, it could be migrated
into XArray implementation in the future.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This patch adds a dedicated shrinker targeting to free unneeded
memory consumed by a number of erofs in-memory data structures.
Like F2FS and UBIFS, it also adds:
- sbi->umount_mutex to avoid races on shrinker and put_super
- sbi->shrinker_run_no to not revisit recently scaned objects
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
In order to introducing shrinker solution for erofs,
let's manage all mounted erofs instances at first.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Currently, this patch only simply implements LZ4
decompressor due to its development priority.
In the future, erofs will support more compression
algorithm and format other than LZ4, thus a generic
decompressor interface will be needed.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
We have to reduce the memory cost as much as possible,
so we don't want to decompress more data beyond
the output buffer size, however "LZ4_decompress_safe_partial"
doesn't guarantee to stop at the arbitary end position,
but stop just after its current LZ4 "sequence" is completed.
Link: https://groups.google.com/forum/#!topic/lz4c/_3kkz5N6n00
Therefore, I hacked the LZ4 decompression logic by hand,
probably NOT the fastest approach, and hope for better
implementation.
Signed-off-by: Miao Xie <miaoxie@huawei.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
The unzip subsystem also uses these functions,
let's export them to internal.h.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This patch introduces an temporary _on-stack_ page
pool to reuse the freed page directly as much as
it can for better performance and release all pages
at a time, it also slightly reduces the possibility of
the potential memory allocation failure.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This patch introduces an iterable L2P mapping
operation 'erofs_map_blocks_iter'.
Compared with 'erofs_map_blocks', it avoids
a number of redundant 'release and regrab'
processes if they request the same meta page.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
For each compressed cluster, there is a straight-forward
way of allocating a fixed or variable-sized (for VLE) array
to record the corresponding file pages for its decompression
if we decide to decompress these pages asynchronously (eg.
read-ahead case), however it could take much extra on-heap
memory compared with traditional uncompressed filesystems.
This patch introduces a pagevec solution to reuse some
allocated file page in the time-sharing approach storing
parts of the array itself in order to minimize the extra
memory overhead, thus only a constant and small-sized array
used for booting the whole array itself up will be needed.
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Currently kernel has scattered tagged pointer usages hacked
by hand in plain code, without a unique and portable functionset
to highlight the tagged pointer itself and wrap these hacked code
in order to clean up all over meaningless magic masks.
Therefore, this patch introduces simple generic methods to fold
tags into a pointer integer. It currently supports the last n bits
of the pointer for tags, which can be selected by users.
In addition, it will also be used for the upcoming EROFS filesystem,
which heavily uses tagged pointer approach for high performance
and reducing extra memory allocation.
Link: https://en.wikipedia.org/wiki/Tagged_pointer
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This patch introduces error injection infrastructure, with it, we can
inject error in any kernel exported common functions which erofs used,
so that it can force erofs running into error paths, it turns out that
tests can cover real rare paths more easily to find bugs.
Reviewed-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This patch adds to support special inode, such as block dev, char,
socket, pipe inode.
Reviewed-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This implements xattr and acl functionalities.
Inline and shared xattrs are introduced for flexibility.
Specifically, if the same xattr occurs for many times
in a large number of inodes or the value of a xattr is so large
that it isn't suitable to be inlined, a shared xattr
kept in the xattr meta will be used instead.
Signed-off-by: Miao Xie <miaoxie@huawei.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This commit adds Makefile and Kconfig for erofs, and
updates Makefile and Kconfig files in the fs directory.
Signed-off-by: Miao Xie <miaoxie@huawei.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This commit adds functions for meta and raw data, and also
provides address_space_operations for raw data access.
Signed-off-by: Miao Xie <miaoxie@huawei.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
- erofs_sb_info:
contains erofs-specific in-memory information.
- erofs_vnode:
contains vfs_inode and other fs-specific information.
same as super block, the only one in-memory definition exists.
- erofs_map_blocks
plays a role in the file L2P mapping
Signed-off-by: Miao Xie <miaoxie@huawei.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This commit adds the on-disk layout header file of erofs.
Note that the on-disk layout is still WIP, and some fields are
reserved for the future use by design.
Any comments are welcome.
Thanks-to: Li Guifu <liguifu2@huawei.com>
Thanks-to: Sun Qiuyang <sunqiuyang@huawei.com>
Signed-off-by: Miao Xie <miaoxie@huawei.com>
Signed-off-by: Chao Yu <yuchao0@huawei.com>
Signed-off-by: Gao Xiang <gaoxiang25@huawei.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Gao Xiang
Kristaps Čivkulis
Chao Yu
Gao Xiang