This cache module uses a Redis instance. To make sccache use this,
set SCCACHE_REDIS to redis://[:<passwd>@]<hostname>[:port][/<db>].
The maximum and current cache size is retrieved from the INFO and
CONFIG GET redis command.
This commit migrates away from the `protobuf` crate to instead just working with
bincode on the wire as a serialization format. This is done by leveraging a few
different crates:
* The `bincode` and `serde_derive` crates are used to define serialization
for Rust structures as well as provide a bincode implementation.
* The `tokio_io::codec::length_delimited` module implements framing via length
prefixes to transform an asynchronous stream of bytes into a literal `Stream`
of `BytesMut`.
* The `tokio_serde_bincode` crate is then used to tie it all together, parsing
these `BytesMut` as the request/response types of sccache.
Most of the changes here are related to moving away from the protobuf API
throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more
rustic-ish API that just uses enums/structs. Overall it felt quite natural (as
one would expect) to just use the raw enum/struct values.
This may not be quite as performant as before but that doesn't really apply to
sccache's use case where perf is hugely dominated by actually compiling and
hashing, so I'm not too too worried about that.
My personal motivation for this is twofold:
1. Using `protobuf` was a little clunky throughout the codebase and definitely
had some sharp edges that felt good to smooth out.
2. There's currently what I believe some mysterious segfault and/or stray write
happening in sccache and I'm not sure where. The `protobuf` crate had a lot
of `unsafe` code and in lieu of actually auditing it I figured it'd be good
to kill two birds with one stone. I have no idea if this fixes my segfault
problem (I never could reproduce it) but I figured it's worth a shot.
This is an abstraction of the Tokio stack now and doesn't have much impact on
sccache itself but I figured it'd be good to update a few deps and pick up the
recent version of things!
Plus that and I'd like to start using the length_delimited module soon...
Previous versions of futures-cpupool depended on crossbeam which unfortunately
has known segfaults (spurious) so the updated version no longer uses
futures-cpupool and instead relies on channels in the standard library.
I was sporadically receiving a segfault locally when trying to debug issues on
Windows and in tracking this down I discovered blackbeam/named_pipe#3 which
leads to segfaults locally on startup.
This switches the one use case to the relevant functionality in
`mio-named-pipes` (already pulled in as part of `tokio-process`) and then
otherwise mirrors the same logic as the Unix version, just waiting for a byte
with a timeout.
This commit pulls in Hyper from its master branch which contains the
asynchronous/Tokio integration. All instances of HTTP, namely when
interacting with S3, are now all powered by async Hyper and work with
futures rather than synchronous requests on thread pools.
This commit replaces all process spawning with the futures-powered
versions in the `tokio-process` crate. Most functions were already ready
to return futures, but a few minor updates were made to ensure that
functions could return futures.
Additionally, some `&CpuPool` arguments have shown up to allow executing
work on a thread pool, such as filesystem operations. A number of
compilers write data to tempdirs and such before running a compiler, and
these are all executed on thread pools rather than the main thread.
My main takeaway from this commit is that translating synchronous to
asynchronous code isn't always easy. The "easy" version ends up being
relatively wasteful in terms of clones for low-level performance, but
probably shouldn't matter much in the context of spawning processes.
Other than that though I found that there was a translation for all
patterns found didn't actually hit any bugs (yet at least) from the
translation.
This commit rewrites the `server` module of sccache to be backed with Tokio. The
previous version was written with `mio`, which Tokio is built on, but is
unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous
programming in Rust and sccache serves as a great testing ground for ergonomics!
It's intended that the support added here will eventually extend to many other
operations that sccache does as well. For example thread spawning has all been
replaced with `CpuPool` to have a shared pool for I/O operations and such
(namely the filesystem). Eventually the HTTP requests made by the S3 backend can
be integrated with the Tokio branch of Hyper as well to run that on the event
loop instead of in a worker thread. I'd also like to eventually extend this with
`tokio-process` as well to move process spawning off helper threads as well, but
I'm leaving that to a future commit as well.
Overall I found the transition was quite smooth, with the high level
architecture look like:
* The `tokio-proto` crate is used in streaming mode. The streaming part is used
for the one RPC sccache gets which requires a second response to be sent later
on. This second response is the "response body" in tokio-proto terms.
* All of sccache's logic is manifested in an implementation of the `Service`
trait.
* The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and
simple deserialization/serialization is performed with protobuf.
Some differences in design are:
* The `SccacheService` for now is just a bunch of reference-counted pointers,
making it cheap to clone. As the futures it returns progress they will each
retain a reference to a cloned copy of the `SccacheService`. Before all this
data was just stored and manipulated in a struct directly, but it's now
directly managed through shared memory.
* The storage backends share a thread pool with the main server instead of
spawning threads.
And finally, some things I've learned along the way:
* Sharing data between futures isn't a trivial operation. It took an explicit
decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics
played out.
* Shutdown is pretty tricky here. I've tried to carry over all the previous
logic but it definitely required not using `TcpServer` in tokio-proto at the
very least, and otherwise required a few custom futures and such to track the
various states. I have a hunch that tokio-proto could provide more options out
of the box for something like this.
This change makes `get_cached_or_compile` return a `Future` for the
cache write instead of blocking on it. The server then waits for the
result on a thread. It also changes the server to record cache write
timings, and display the average in the stats.
Unlike sccache1, you need to specify both bucket and region, like:
SCCACHE_BUCKET=sccache
SCCACHE_REGION=us-east-1
Pretty limited testing, no automated tests currently.
Ted Mielczarek
Felix Obenhuber
Alex Crichton
Stefan Ilic