1.6 KiB
1.6 KiB
hdfs-mount
Allows to mount remote HDFS as a local Linux filesystem and allow arbitrary applications / shell scripts to access HDFS as normal files and directories in efficient and secure way.
Features (Planned)
- High performance
- directly interfacing Linux kernel for FUSE and HDFS using protocol buffers (requires no JavaVM)
- designed and optimized for throughput-intensive workloads (throughput is traded for latency whenever possible)
- full streaming and automatic read-ahead support
- concurrent operations
- In-memory metadata caching (very fast ls!)
- High stability and robust failure-handling behavior
- automatic retries and failover, all configurable
- optional lazy mounting, before HDFS becomes available
- Support for both reads and writes
- support for random writes [slow, but functionally correct]
- support for file truncations
- CoreOS and Docker-friendly
- optionally packagable as a statically-linked self-contained executable
Current state
"Alpha", under active development. Basic R/O scenarios and key R/O throughout optimizations are implemented and outperform existing HDFS/FUSE solutions. If you want to use the component - come back in few weeks If you want to help - contact authors
Other Platforms
It should be relatively easy to enable this working on MacOS and FreeBSD, since all underlying dependencies are MacOS and FreeBSD-ready. Very few changes are needed to the code to get it working on those platforms, but it is currently not a priority for authors. Contact authors if you want to help.