This commit lifts out some logic of creation of `Command` structures to simplify
a few code paths and reduce clones, making it a little more ergonomic to
read/write. We'll just throw away the `Command` if we end up not using it!
This commit pulls in Hyper from its master branch which contains the
asynchronous/Tokio integration. All instances of HTTP, namely when
interacting with S3, are now all powered by async Hyper and work with
futures rather than synchronous requests on thread pools.
This commit replaces all process spawning with the futures-powered
versions in the `tokio-process` crate. Most functions were already ready
to return futures, but a few minor updates were made to ensure that
functions could return futures.
Additionally, some `&CpuPool` arguments have shown up to allow executing
work on a thread pool, such as filesystem operations. A number of
compilers write data to tempdirs and such before running a compiler, and
these are all executed on thread pools rather than the main thread.
My main takeaway from this commit is that translating synchronous to
asynchronous code isn't always easy. The "easy" version ends up being
relatively wasteful in terms of clones for low-level performance, but
probably shouldn't matter much in the context of spawning processes.
Other than that though I found that there was a translation for all
patterns found didn't actually hit any bugs (yet at least) from the
translation.
This commit just takes the synchronous version and reimplements it with
futures, using necessary combinators and such to prepare for running
commands with futures as well.
All process spawning/waiting still happens in a thread pool, this
commit just refactors methods to push futures all the way to down to the
lowest levels so we can integrate `tokio-process`.
This commit rewrites the `server` module of sccache to be backed with Tokio. The
previous version was written with `mio`, which Tokio is built on, but is
unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous
programming in Rust and sccache serves as a great testing ground for ergonomics!
It's intended that the support added here will eventually extend to many other
operations that sccache does as well. For example thread spawning has all been
replaced with `CpuPool` to have a shared pool for I/O operations and such
(namely the filesystem). Eventually the HTTP requests made by the S3 backend can
be integrated with the Tokio branch of Hyper as well to run that on the event
loop instead of in a worker thread. I'd also like to eventually extend this with
`tokio-process` as well to move process spawning off helper threads as well, but
I'm leaving that to a future commit as well.
Overall I found the transition was quite smooth, with the high level
architecture look like:
* The `tokio-proto` crate is used in streaming mode. The streaming part is used
for the one RPC sccache gets which requires a second response to be sent later
on. This second response is the "response body" in tokio-proto terms.
* All of sccache's logic is manifested in an implementation of the `Service`
trait.
* The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and
simple deserialization/serialization is performed with protobuf.
Some differences in design are:
* The `SccacheService` for now is just a bunch of reference-counted pointers,
making it cheap to clone. As the futures it returns progress they will each
retain a reference to a cloned copy of the `SccacheService`. Before all this
data was just stored and manipulated in a struct directly, but it's now
directly managed through shared memory.
* The storage backends share a thread pool with the main server instead of
spawning threads.
And finally, some things I've learned along the way:
* Sharing data between futures isn't a trivial operation. It took an explicit
decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics
played out.
* Shutdown is pretty tricky here. I've tried to carry over all the previous
logic but it definitely required not using `TcpServer` in tokio-proto at the
very least, and otherwise required a few custom futures and such to track the
various states. I have a hunch that tokio-proto could provide more options out
of the box for something like this.
It looks like the logic in #27 changed the default host name from
$bucket.s3.amazonaws.com to s3.amazonaws.com, but didn't accompany with changes
to upload to /$bucket in the path name. This restores the original behavior by
uploading to $bucket.s3.amazonaws.com without requiring changes to how paths are
constructed.
This commit adds support for the `@file` option that gcc/clang supports. This
option means that an `file` should be read and `@file` should be replaced with
all the options specified in `file`.
The online documentation indicates that gcc supports arguments with spaces
through quoting, but this seemed like it may be nontrivial to implement, so I
figured that for now those cases could continue to be un-cacheable.
Closes#43
If none of the logging environment variables are supplied
sccache creates a sccache2.log file on every sccache
invocation. Means, in every directory where a compilation takes
place a logfile is left afterwards. In the happy case this file
is empty. This can be kind of messy for larger projects where
the compiler invocation is done from multiple directories.
This patch changes the logging initialization:
* SCCACHE2_LOG_LEVEL is renamed to SCCACHE_LOG_LEVEL
* sccache2.log is renamed to sccache.log
* If none of RUST_LOG or SCCACHE_LOG_LEVEL is set, no logging is initialized.
* If RUST_LOG is set, only env_logger is used - no logfile is created.
Straight forward implementation of clearing statistics
when -z or --zero-stats is passed on command line.
This command also outputs the stats upon successful
execution in order to provide feedback.
* Fix avg write duration and extend server stats
The average duration that is taken by a miss or hit is interesting
when testing different cache types. This patch adds the average time
taken by a cache miss and hit to the server stats. This measurements
can be entirely done in the server and do not need any changes to the
cache implementations.
Furthermore the calculation of the average duration for cache writes is
fixed (wrong factor used, that produced micros instead of millis).
If running the preprocessor fails (because a #included file was not found, say)
sccache was returning the preprocessor stdout+stderr to the calling process,
which is wrong. This fixes it to just return the stderr, and drop the stdout.
This fixes the extra console windows that show up when the sccache server
creates compiler processes on Windows, but it only works on nightly Rust
currently, and requires building with `--features=unstable`, because it relies
on the not-yet-stable `windows_process_extensions` feature (https://github.com/rust-lang/rust/issues/37827).
Also I added a bunch of extra documentation to the mock_command traits.
As part of 030ed02ab4 I made `DiskCache::finish_put` write the cache entry to disk asynchronously, where previously it had been synchronous. This caused the `get_cached_or_compile` tests to be racy, specifically `test_compiler_get_cached_or_compile_cached`, which would store a cache entry and then immediately try to retrieve it. I've fixed this by making all the tests wait on the cache write future.
Specify a new endpoint with `SCCACHE_ENDPOINT`
such as Minio https://github.com/minio/minio.
If not defaults to 's3.amazonaws.com'. Current
code for some reason was not working with custom
regions. Introduced a new ENV called `SCCACHE_REGION`
to handle cross region s3 URLs.
Alex Crichton
Ted Mielczarek
Jędrzej Nowacki
Mike Hommey
Felix Obenhuber
Felix Obenhuber
Alex Newman
Alex Newman
Harshavardhana
Yixi Zhang
Ms2ger
Christian Legnitto