libprojfs

A Linux projected filesystem library, similar in concept to the Windows Projected File System and developed in conjunction with the VFSForGit project.

While the libprojfs C library may also be used independently of VFSForGit, our primary goal is to enable users to run the VFSForGit client on a Linux system.

PLEASE NOTE: At present libprojfs is undergoing rapid development and its design and APIs may change significantly; some features are also incomplete, and so we do not recommend libprojfs for any production environments.

However, we wanted to share our progress to date, and we hope others are as excited as we are at the prospect of running the VFSForGit client on Linux in the not-too-distant future!

Current Status

The library is under active development and supports basic directory projection, with additional features added regularly.

We will only make a first, official versioned release once initial development is complete.

A libprojfs filesystem can currently be used to mount a VFSForGit MirrorProvider test client; we expect to continue our pre-release development until the VFSForGit client proper is also functional with libprojfs, at which point we may make an initial tagged release.

Design

The libprojfs library is designed to function as a stackable Linux filesystem supporting a "provider" process which implements custom callbacks to populate files and directories on demand. This is illustrated in the context of a VFSForGit provider below:

Actual file storage is delegated to a "lower" storage filesystem, which may be any Linux filesystem, e.g., ext4. At the time when a libprojfs filesystem mount is created, there may be no files or directories in the lower filesystem yet.

As normal filesystem requests are made within the libprojfs mount (e.g., by ls or cat), libprojfs intercepts the requests and queries the provider process for the actual contents of the file or directory. The provider's response is then written to the lower filesystem so future accesses may be satisfied without querying the provider again.

See our design document for more details.

We anticipate that whether libprojfs remains a FUSE-based library, or becomes a libfuse-like interface to a Linux kernel module, it may be useful for purposes other than running a VFSForGit client.

For this reason, we have tried to ensure that our native event notification API is aligned closely with the Linux kernel's fanotify/inotify/fsnotify APIs.

Getting Started

Please see our detailed Build and Installation page for step-by-step instructions on how to build and install libprojfs and its depedencies on Linux, and (optionally) also build and run the VFSForGit MirrorProvider test utility with libprojfs.

So long as libprojfs remains based on FUSE, the primary depedency for libprojfs is the user-space libfuse library, as well as having the Linux fuse kernel module installed.

Support for user.* extended attributes must also be enabled on the "lower" storage filesystem. Your Linux kernel should have been compiled with the necessary configuration options (e.g., CONFIG_EXT2_FS_XATTR for ext2/3/4) and the underlying filesystem you use for libprojfs should be mounted with appropriate settings (e.g., user_xattr for ext2/3/4).

If you are using a Docker container, you may need to ensure the fuse kernel module is installed on your host OS, and user.* extended attributes are supported on your host's filesystem. See the Using Docker containers section for more details.

At present we require some custom modifications to the libfuse version 3.x library, and therefore building libprojfs against a default installation of libfuse will not work. We hope to work with the libfuse maintainers to develop patches will can eventually be accepted upstream; see PR #346 in the libfuse project for the current status of this work.

To test libprojfs with the Microsoft VFSForGit MirrorProvider (as we do not support the primary VFSForGit GVFS provider yet), .NET Core must be installed and parts of the VFSForGit project built. See the Building and Running MirrorProvider section for details.

Contributing

Thank you for your interest in libprojfs!

We welcome contributions; please see our CONTRIBUTING guidelines and our contributor CODE_OF_CONDUCT.

Licensing

The libprojfs library is licensed under the LGPL v2.1.

See the NOTICE file for a list of other licenses used in the project, and the comments in each file for the licenses applicable to them.

Development Roadmap

We are developing the libprojfs library first, using FUSE to prototype and test its performance, before migrating functionality into a Linux kernel module (assuming that proves to be necessary to meet our performance criteria).

For more details on the planned development phases, see our design document.

The VFSForGit API, which is currently supported through the use of the --enable-vfs-api configuration option to libprojfs, may at some point refactored out of this library entirely and handled exclusively within the ProjFS.Linux code in the VFSForGit project.

However, for the moment it has proven efficient to keep the VFS API within this library while libprojfs undergoes rapid early development.

Authors

The libprojfs library is currently maintained and developed by several members of GitHub's Engineering organization, including:

• @chrisd8088
• @kivikakk
• @wrighty

You can also contact the GitHub project team at opensource+libprojfs@github.com.