This cache module uses a Memcached cluster.
To make sccache use this, set SCCACHE_MEMCACHED
to a list of tcp://<hostname>:<port> addresses,
separated by whitespace.
This commit alters the main sccache server to operate and orchestrate its own
GNU make style jobserver. This is primarily intended for interoperation with
rustc itself.
The Rust compiler currently has a multithreaded mode where it will execute code
generation and optimization on the LLVM side of things in parallel. This
parallelism, however, can overload a machine quickly if not properly accounted
for (e.g. if 10 rustcs all spawn 10 threads...). The usage of a GNU make style
jobserver is intended to arbitrate and rate limit all these rustc instances to
ensure that one build's maximal parallelism never exceeds a particular amount.
Currently for Rust Cargo is the primary driver for setting up a jobserver. Cargo
will create this and manage this per compilation, ensuring that any one `cargo
build` invocation never exceeds a maximal parallelism. When sccache enters the
picture, however, the story gets slightly more odd.
The jobserver implementation on Unix relies on inheritance of file descriptors
in spawned processes. With sccache, however, there's no inheritance as the
actual rustc invocation is spawned by the server, not the client. In this case
the env vars used to configure the jobsever are usually incorrect.
To handle this problem this commit bakes a jobserver directly into sccache
itself. The jobserver then overrides whatever jobserver the client has
configured in its own env vars to ensure correct operation. The settings of each
jobserver may be misconfigured (there's no way to configure sccache's jobserver
right now), but hopefully that's not too much of a problem for the forseeable
future.
The implementation here was to provide a thin wrapper around the `jobserver`
crate with a futures-based interface. This interface was then hooked into the
mock command infrastructure to automatically acquire a jobserver token when
spawning a process and automatically drop the token when the process exits.
Additionally, all spawned processes will now automatically receive a configured
jobserver.
cc rust-lang/rust#42867, the original motivation for this commit
Local benchmarking showed that the implementation of SHA-512 in the *ring* crate
is 3x faster than the implementation of SHA-1 in the `sha1` crate. I've also
noticed that 80%+ of sccache's runtime on a fully cached build is spent hashing.
With this change I noticed a decrease from 108s to 92s when building a fully
cached LLVM from the network. Not a huge win but if the network were faster
could perhaps add up!
Closes#108
The S3 and Redis caches are optional during build time.
Enable feature "s3" and/or "redis" to build sccache with
support for those backends. Only the local disk cache
is available by default. The "all" feature enables both.
This patch is changing the current default behavior that
always has S3 support.
This cache module uses a Redis instance. To make sccache use this,
set SCCACHE_REDIS to redis://[:<passwd>@]<hostname>[:port][/<db>].
The maximum and current cache size is retrieved from the INFO and
CONFIG GET redis command.
I was sporadically receiving a segfault locally when trying to debug issues on
Windows and in tracking this down I discovered blackbeam/named_pipe#3 which
leads to segfaults locally on startup.
This switches the one use case to the relevant functionality in
`mio-named-pipes` (already pulled in as part of `tokio-process`) and then
otherwise mirrors the same logic as the Unix version, just waiting for a byte
with a timeout.
If none of the logging environment variables are supplied
sccache creates a sccache2.log file on every sccache
invocation. Means, in every directory where a compilation takes
place a logfile is left afterwards. In the happy case this file
is empty. This can be kind of messy for larger projects where
the compiler invocation is done from multiple directories.
This patch changes the logging initialization:
* SCCACHE2_LOG_LEVEL is renamed to SCCACHE_LOG_LEVEL
* sccache2.log is renamed to sccache.log
* If none of RUST_LOG or SCCACHE_LOG_LEVEL is set, no logging is initialized.
* If RUST_LOG is set, only env_logger is used - no logfile is created.
Julian Gehring
Felix Bruns
Felix Obenhuber
Ted Mielczarek
Rondinelly
Ben Bader
David Michael Barr
Alex Crichton
tiagomoraismorgado
Aaron Meese
Jeff Muizelaar
Felix Obenhuber
Taylor Cramer
Andre Parodi
Alex Newman
Yixi Zhang
Mike Hommey