mozilla
/
sccache
зеркало из https://github.com/mozilla/sccache.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
sccache/Cargo.lock

1317 строки
60 KiB
Plaintext
Исходник Обычный вид История

Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[root]
name = "sccache"
Add an `SCCACHE_ERROR_LOG` variable to redirect the server's stderr, so we can get backtraces for unhandled panics.
2017-01-09 20:20:56 +03:00
version = "0.1.1-pre"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
dependencies = [
Pull in app_dirs to get a better default local disk cache location
2016-11-17 18:59:15 +03:00
"app_dirs 1.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"bincode 1.0.0-alpha6 (git+https://github.com/TyOverby/bincode)",
"byteorder 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
drop rusoto, import some simple s3 code and use that
2016-08-12 21:03:37 +03:00
"chrono 0.2.25 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"clap 2.9.2 (registry+https://github.com/rust-lang/crates.io-index)",
daemonize the server process on unix
2016-10-26 23:22:52 +03:00
"daemonize 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"env_logger 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
Use error-chain to define Result+Error types, refactor cmdline::parse to use it.
2017-01-28 00:01:29 +03:00
"error-chain 0.7.2 (registry+https://github.com/rust-lang/crates.io-index)",
Start using fern for logging (#9)
2016-06-28 14:12:54 +03:00
"fern 0.3.5 (registry+https://github.com/rust-lang/crates.io-index)",
turn Compiler into CompilerKind, make a Compiler struct to hold some more info about compilers
2016-05-18 21:30:11 +03:00
"filetime 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)",
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Update the futures-cpupool dependency Previous versions of futures-cpupool depended on crossbeam which unfortunately has known segfaults (spurious) so the updated version no longer uses futures-cpupool and instead relies on channels in the standard library.
2017-03-22 02:09:12 +03:00
"futures-cpupool 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"hyper 0.11.0-a.0 (git+https://github.com/hyperium/hyper)",
"hyper-tls 0.0.0 (git+https://github.com/hyperium/hyper-tls)",
When hashing Rust compile arguments, sort -L and --cfg args as well as --extern
2017-04-03 17:21:13 +03:00
"itertools 0.5.10 (registry+https://github.com/rust-lang/crates.io-index)",
Detect MSVC -showIncludes prefix (not actually wired up to anything yet)
2016-06-21 14:42:22 +03:00
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
use local-encoding crate for stdout conversion
2016-09-21 22:57:28 +03:00
"local-encoding 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Make DiskCache limit the size of the on-disk cache by using LruDiskCache. Fixes #29
2016-11-29 03:59:42 +03:00
"lru-disk-cache 0.1.0",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"mio-named-pipes 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)",
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
"number_prefix 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)",
Add a Redis cache variant This cache module uses a Redis instance. To make sccache use this, set SCCACHE_REDIS to redis://[:<passwd>@]<hostname>[:port][/<db>]. The maximum and current cache size is retrieved from the INFO and CONFIG GET redis command.
2017-02-16 22:56:41 +03:00
"redis 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)",
drop rusoto, import some simple s3 code and use that
2016-08-12 21:03:37 +03:00
"regex 0.1.73 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"retry 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
drop rusoto, import some simple s3 code and use that
2016-08-12 21:03:37 +03:00
"rust-crypto 0.2.36 (registry+https://github.com/rust-lang/crates.io-index)",
"rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)",
update serde to 0.9.13 to pick up a fix for OsString deserialization on Windows (fixes Windows breakage)
2017-04-07 13:55:26 +03:00
"serde 0.9.13 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"serde_derive 0.9.11 (registry+https://github.com/rust-lang/crates.io-index)",
"serde_json 0.9.9 (registry+https://github.com/rust-lang/crates.io-index)",
bump sha1 crate to 0.2.0
2016-07-15 17:42:08 +03:00
"sha1 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"tempdir 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
Start using fern for logging (#9)
2016-06-28 14:12:54 +03:00
"time 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"tokio-core 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"tokio-io 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"tokio-process 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-proto 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"tokio-serde-bincode 0.1.0 (git+https://github.com/alexcrichton/tokio-serde-bincode)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"tokio-service 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"tokio-tls 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-uds 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
Use a named unix socket or named pipe to let the background server notify the client that it started successfully
2017-01-27 22:39:47 +03:00
"uuid 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)",
switch back to released version of which, switch to a rusoto branch with a bug fix
2016-08-08 22:06:06 +03:00
"which 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
Store file permissions for files in cache
2017-03-07 15:52:53 +03:00
"zip 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "advapi32-sys"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "aho-corasick"
Update deps, fix build with newer rust-protobuf
2016-05-12 22:14:13 +03:00
version = "0.5.2"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "ansi_term"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "0.7.5"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Pull in app_dirs to get a better default local disk cache location
2016-11-17 18:59:15 +03:00
[[package]]
name = "app_dirs"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"ole32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
"shell32-sys 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"xdg 2.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "base64"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.4.0"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"byteorder 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
[[package]]
name = "bincode"
version = "1.0.0-alpha6"
source = "git+https://github.com/TyOverby/bincode#c971c4956d3719c62fa07e385b349428b94dddf7"
dependencies = [
"byteorder 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
"num-traits 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
update serde to 0.9.13 to pick up a fix for OsString deserialization on Windows (fixes Windows breakage)
2017-04-07 13:55:26 +03:00
"serde 0.9.13 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "bitflags"
version = "0.5.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "bitflags"
version = "0.7.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "byteorder"
version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
[[package]]
name = "bytes"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"byteorder 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
"iovec 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "cfg-if"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "chrono"
use rusoto master YOLO
2016-08-09 04:15:03 +03:00
version = "0.2.25"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num 0.1.34 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
"time 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)",
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "clap"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "2.9.2"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"ansi_term 0.7.5 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"bitflags 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"strsim 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"term_size 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"unicode-width 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)",
"vec_map 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
name = "core-foundation"
version = "0.2.3"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"core-foundation-sys 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
]
[[package]]
name = "core-foundation-sys"
version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "crypt32-sys"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
daemonize the server process on unix
2016-10-26 23:22:52 +03:00
[[package]]
name = "daemonize"
version = "0.2.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
daemonize the server process on unix
2016-10-26 23:22:52 +03:00
]
switch back to released version of which, switch to a rusoto branch with a bug fix
2016-08-08 22:06:06 +03:00
[[package]]
name = "dtoa"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
version = "0.4.1"
switch back to released version of which, switch to a rusoto branch with a bug fix
2016-08-08 22:06:06 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
When hashing Rust compile arguments, sort -L and --cfg args as well as --extern
2017-04-03 17:21:13 +03:00
[[package]]
name = "either"
version = "1.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "env_logger"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "0.3.4"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"regex 0.1.73 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
]
Use error-chain to define Result+Error types, refactor cmdline::parse to use it.
2017-01-28 00:01:29 +03:00
[[package]]
name = "error-chain"
version = "0.7.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
Start using fern for logging (#9)
2016-06-28 14:12:54 +03:00
[[package]]
name = "fern"
version = "0.3.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Start using fern for logging (#9)
2016-06-28 14:12:54 +03:00
]
turn Compiler into CompilerKind, make a Compiler struct to hold some more info about compilers
2016-05-18 21:30:11 +03:00
[[package]]
name = "filetime"
version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
turn Compiler into CompilerKind, make a Compiler struct to hold some more info about compilers
2016-05-18 21:30:11 +03:00
]
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
[[package]]
name = "flate2"
version = "0.2.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
"miniz-sys 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)",
]
Don't block sending compile result to client on cache write. Fixes #19. This change makes `get_cached_or_compile` return a `Future` for the cache write instead of blocking on it. The server then waits for the result on a thread. It also changes the server to record cache write timings, and display the average in the stats.
2016-11-03 00:15:14 +03:00
[[package]]
name = "futures"
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
version = "0.1.11"
Don't block sending compile result to client on cache write. Fixes #19. This change makes `get_cached_or_compile` return a `Future` for the cache write instead of blocking on it. The server then waits for the result on a thread. It also changes the server to record cache write timings, and display the average in the stats.
2016-11-03 00:15:14 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
[[package]]
name = "futures-cpupool"
Update the futures-cpupool dependency Previous versions of futures-cpupool depended on crossbeam which unfortunately has known segfaults (spurious) so the updated version no longer uses futures-cpupool and instead relies on channels in the standard library.
2017-03-22 02:09:12 +03:00
version = "0.1.5"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"num_cpus 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
[[package]]
name = "gcc"
Update gcc crate in Cargo.lock Picks up support for `-C target-feature=+crt-static` on MSVC and such
2017-02-25 12:48:02 +03:00
version = "0.3.43"
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
use local-encoding crate for stdout conversion
2016-09-21 22:57:28 +03:00
[[package]]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
name = "gdi32-sys"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "getopts"
version = "0.2.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "httparse"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
version = "1.2.1"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "hyper"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
version = "0.11.0-a.0"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
source = "git+https://github.com/hyperium/hyper#43cf9aefe8f9147a0f08aa0f9d609d275549fd7c"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"base64 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
"bytes 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Update the futures-cpupool dependency Previous versions of futures-cpupool depended on crossbeam which unfortunately has known segfaults (spurious) so the updated version no longer uses futures-cpupool and instead relies on channels in the standard library.
2017-03-22 02:09:12 +03:00
"futures-cpupool 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"httparse 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
"language-tags 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"mime 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"rustc_version 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
"time 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"tokio-core 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"tokio-io 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"tokio-proto 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"tokio-service 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
"unicase 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"url 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "hyper-tls"
version = "0.0.0"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
source = "git+https://github.com/hyperium/hyper-tls#74aefbe500df5bb2d9a87844580b4ac21d49f134"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
dependencies = [
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"hyper 0.11.0-a.0 (git+https://github.com/hyperium/hyper)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"native-tls 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"tokio-core 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"tokio-io 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"tokio-service 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"tokio-tls 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "idna"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"matches 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"unicode-bidi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)",
"unicode-normalization 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "iovec"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
]
When hashing Rust compile arguments, sort -L and --cfg args as well as --extern
2017-04-03 17:21:13 +03:00
[[package]]
name = "itertools"
version = "0.5.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"either 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
[[package]]
name = "itoa"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
version = "0.3.1"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "kernel32-sys"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "language-tags"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "lazy_static"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
[[package]]
name = "lazycell"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "libc"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.2.21"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Make DiskCache limit the size of the on-disk cache by using LruDiskCache. Fixes #29
2016-11-29 03:59:42 +03:00
[[package]]
name = "linked-hash-map"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
use local-encoding crate for stdout conversion
2016-09-21 22:57:28 +03:00
[[package]]
name = "local-encoding"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"skeptic 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "log"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.3.7"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Make DiskCache limit the size of the on-disk cache by using LruDiskCache. Fixes #29
2016-11-29 03:59:42 +03:00
[[package]]
name = "lru-cache"
version = "0.1.0"
Use my lru-cache branch until the changes land upstream
2016-12-07 23:58:42 +03:00
source = "git+https://github.com/luser/lru-cache?branch=non-mut-get#94ac965a95fae83359ee23e78beeca32cc6bb5b2"
Make DiskCache limit the size of the on-disk cache by using LruDiskCache. Fixes #29
2016-11-29 03:59:42 +03:00
dependencies = [
"linked-hash-map 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "lru-disk-cache"
version = "0.1.0"
dependencies = [
"env_logger 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
use filetime
2016-12-07 04:47:02 +03:00
"filetime 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Use my lru-cache branch until the changes land upstream
2016-12-07 23:58:42 +03:00
"lru-cache 0.1.0 (git+https://github.com/luser/lru-cache?branch=non-mut-get)",
Make DiskCache limit the size of the on-disk cache by using LruDiskCache. Fixes #29
2016-11-29 03:59:42 +03:00
"tempdir 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
"walkdir 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)",
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "matches"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
version = "0.1.4"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "memchr"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "metadeps"
version = "1.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"error-chain 0.7.2 (registry+https://github.com/rust-lang/crates.io-index)",
"pkg-config 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)",
"toml 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "mime"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
version = "0.2.2"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
]
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
[[package]]
name = "miniz-sys"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update gcc crate in Cargo.lock Picks up support for `-C target-feature=+crt-static` on MSVC and such
2017-02-25 12:48:02 +03:00
"gcc 0.3.43 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "mio"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
version = "0.6.6"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"iovec 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"lazycell 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
"miow 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"net2 0.2.25 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"slab 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
]
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
[[package]]
name = "mio-named-pipes"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.1.5"
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"mio 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"miow 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "mio-uds"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.6.3"
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"mio 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "miow"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.2.1"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"net2 0.2.25 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"ws2_32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
[[package]]
name = "msdos_time"
version = "0.1.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"time 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "native-tls"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"openssl 0.9.6 (registry+https://github.com/rust-lang/crates.io-index)",
"schannel 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"security-framework 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)",
"security-framework-sys 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)",
"tempdir 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "net2"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "0.2.25"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"cfg-if 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"ws2_32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
[[package]]
name = "num"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "0.1.34"
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num-bigint 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
"num-complex 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
"num-integer 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)",
"num-iter 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)",
"num-rational 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num-traits 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
]
[[package]]
name = "num-bigint"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "0.1.33"
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"num-integer 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num-traits 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
"rand 0.3.14 (registry+https://github.com/rust-lang/crates.io-index)",
"rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-complex"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "0.1.33"
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num-traits 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
"rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-integer"
version = "0.1.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num-traits 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
]
[[package]]
name = "num-iter"
version = "0.1.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"num-integer 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num-traits 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
]
[[package]]
name = "num-rational"
version = "0.1.32"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num-bigint 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
"num-integer 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num-traits 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
"rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "num-traits"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "0.1.33"
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "num_cpus"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
version = "1.2.1"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
]
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
[[package]]
name = "number_prefix"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num 0.1.34 (registry+https://github.com/rust-lang/crates.io-index)",
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
]
Pull in app_dirs to get a better default local disk cache location
2016-11-17 18:59:15 +03:00
[[package]]
name = "ole32-sys"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "openssl"
version = "0.9.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"bitflags 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)",
"lazy_static 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"openssl-sys 0.9.6 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "openssl-sys"
version = "0.9.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"gdi32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"metadeps 1.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
"user32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "pkg-config"
version = "0.3.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
[[package]]
name = "podio"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
use local-encoding crate for stdout conversion
2016-09-21 22:57:28 +03:00
[[package]]
name = "pulldown-cmark"
version = "0.0.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"getopts 0.2.14 (registry+https://github.com/rust-lang/crates.io-index)",
]
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
[[package]]
name = "quote"
version = "0.3.15"
source = "registry+https://github.com/rust-lang/crates.io-index"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "rand"
version = "0.3.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
]
Add a Redis cache variant This cache module uses a Redis instance. To make sccache use this, set SCCACHE_REDIS to redis://[:<passwd>@]<hostname>[:port][/<db>]. The maximum and current cache size is retrieved from the INFO and CONFIG GET redis command.
2017-02-16 22:56:41 +03:00
[[package]]
name = "redis"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"sha1 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
"url 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "regex"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "0.1.73"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update deps, fix build with newer rust-protobuf
2016-05-12 22:14:13 +03:00
"aho-corasick 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"regex-syntax 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
"thread_local 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"utf8-ranges 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "regex-syntax"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "0.3.4"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "retry"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"rand 0.3.14 (registry+https://github.com/rust-lang/crates.io-index)",
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
drop rusoto, import some simple s3 code and use that
2016-08-12 21:03:37 +03:00
name = "rust-crypto"
version = "0.2.36"
source = "registry+https://github.com/rust-lang/crates.io-index"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
dependencies = [
Update gcc crate in Cargo.lock Picks up support for `-C target-feature=+crt-static` on MSVC and such
2017-02-25 12:48:02 +03:00
"gcc 0.3.43 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
drop rusoto, import some simple s3 code and use that
2016-08-12 21:03:37 +03:00
"rand 0.3.14 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
"rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)",
"time 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)",
]
drop format_size code in favor of number_prefix crate
2016-05-12 22:46:22 +03:00
[[package]]
name = "rustc-serialize"
version = "0.3.19"
source = "registry+https://github.com/rust-lang/crates.io-index"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "rustc_version"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"semver 0.1.20 (registry+https://github.com/rust-lang/crates.io-index)",
]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "schannel"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"advapi32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
"crypt32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"lazy_static 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"secur32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
[[package]]
name = "scoped-tls"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "secur32-sys"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "security-framework"
version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"core-foundation 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)",
"core-foundation-sys 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"security-framework-sys 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "security-framework-sys"
version = "0.1.10"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"core-foundation-sys 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "semver"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "serde"
update serde to 0.9.13 to pick up a fix for OsString deserialization on Windows (fixes Windows breakage)
2017-04-07 13:55:26 +03:00
version = "0.9.13"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "serde_codegen_internals"
version = "0.14.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"syn 0.11.9 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "serde_derive"
version = "0.9.11"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
dependencies = [
"quote 0.3.15 (registry+https://github.com/rust-lang/crates.io-index)",
"serde_codegen_internals 0.14.1 (registry+https://github.com/rust-lang/crates.io-index)",
"syn 0.11.9 (registry+https://github.com/rust-lang/crates.io-index)",
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "serde_json"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
version = "0.9.9"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"dtoa 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
"itoa 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"num-traits 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)",
update serde to 0.9.13 to pick up a fix for OsString deserialization on Windows (fixes Windows breakage)
2017-04-07 13:55:26 +03:00
"serde 0.9.13 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
]
turn Compiler into CompilerKind, make a Compiler struct to hold some more info about compilers
2016-05-18 21:30:11 +03:00
[[package]]
name = "sha1"
bump sha1 crate to 0.2.0
2016-07-15 17:42:08 +03:00
version = "0.2.0"
turn Compiler into CompilerKind, make a Compiler struct to hold some more info about compilers
2016-05-18 21:30:11 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Pull in app_dirs to get a better default local disk cache location
2016-11-17 18:59:15 +03:00
[[package]]
name = "shell32-sys"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
use local-encoding crate for stdout conversion
2016-09-21 22:57:28 +03:00
[[package]]
name = "skeptic"
version = "0.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"pulldown-cmark 0.0.3 (registry+https://github.com/rust-lang/crates.io-index)",
"tempdir 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)",
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "slab"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "smallvec"
version = "0.2.1"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "strsim"
version = "0.4.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
[[package]]
name = "syn"
version = "0.11.9"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"quote 0.3.15 (registry+https://github.com/rust-lang/crates.io-index)",
"synom 0.11.3 (registry+https://github.com/rust-lang/crates.io-index)",
"unicode-xid 0.0.4 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "synom"
version = "0.11.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"unicode-xid 0.0.4 (registry+https://github.com/rust-lang/crates.io-index)",
]
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
[[package]]
name = "take"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "tempdir"
version = "0.3.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"rand 0.3.14 (registry+https://github.com/rust-lang/crates.io-index)",
]
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
[[package]]
name = "term_size"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "thread-id"
version = "2.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
]
[[package]]
name = "thread_local"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
version = "0.2.6"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"thread-id 2.0.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "time"
version = "0.1.35"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
]
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
[[package]]
name = "tokio-core"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.1.6"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"bytes 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"iovec 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"mio 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"scoped-tls 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"slab 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"tokio-io 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
]
[[package]]
name = "tokio-io"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
version = "0.1.1"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"bytes 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
]
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
[[package]]
name = "tokio-process"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.1.3"
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"mio 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"mio-named-pipes 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-core 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"tokio-io 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
"tokio-signal 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
[[package]]
name = "tokio-proto"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.1.1"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"net2 0.2.25 (registry+https://github.com/rust-lang/crates.io-index)",
"rand 0.3.14 (registry+https://github.com/rust-lang/crates.io-index)",
"slab 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)",
"smallvec 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)",
"take 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"tokio-core 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"tokio-io 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"tokio-service 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
]
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
[[package]]
name = "tokio-serde"
version = "0.1.0"
source = "git+https://github.com/carllerche/tokio-serde#fbafd2386d525d615332941d4e62c71af90dc40c"
dependencies = [
"bytes 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "tokio-serde-bincode"
version = "0.1.0"
source = "git+https://github.com/alexcrichton/tokio-serde-bincode#31e64455a3e8ccedd08d240633dedf94a7eeb557"
dependencies = [
"bincode 1.0.0-alpha6 (git+https://github.com/TyOverby/bincode)",
"bytes 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
update serde to 0.9.13 to pick up a fix for OsString deserialization on Windows (fixes Windows breakage)
2017-04-07 13:55:26 +03:00
"serde 0.9.13 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"tokio-serde 0.1.0 (git+https://github.com/carllerche/tokio-serde)",
]
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
[[package]]
name = "tokio-service"
version = "0.1.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
]
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
[[package]]
name = "tokio-signal"
version = "0.1.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"mio 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"mio-uds 0.6.3 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-core 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
name = "tokio-tls"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.1.2"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
dependencies = [
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"native-tls 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"tokio-core 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"tokio-io 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
Remove raw `poll` in favor of `tokio-uds` Now that we've got tokio at our fingertips it's much easier to time out simple operations!
2017-02-17 19:43:43 +03:00
[[package]]
name = "tokio-uds"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.1.4"
Remove raw `poll` in favor of `tokio-uds` Now that we've got tokio at our fingertips it's much easier to time out simple operations!
2017-02-17 19:43:43 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"bytes 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)",
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"iovec 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
"log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"mio 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)",
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"mio-uds 0.6.3 (registry+https://github.com/rust-lang/crates.io-index)",
"tokio-core 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)",
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"tokio-io 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
Remove raw `poll` in favor of `tokio-uds` Now that we've got tokio at our fingertips it's much easier to time out simple operations!
2017-02-17 19:43:43 +03:00
]
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
name = "toml"
version = "0.2.1"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "unicase"
version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"rustc_version 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)",
]
[[package]]
name = "unicode-bidi"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.2.5"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"matches 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
]
[[package]]
name = "unicode-normalization"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
version = "0.1.4"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "unicode-width"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
[[package]]
name = "unicode-xid"
version = "0.0.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
[[package]]
name = "url"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
version = "1.4.0"
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"idna 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)",
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"matches 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
Implement S3Cache Unlike sccache1, you need to specify both bucket and region, like: SCCACHE_BUCKET=sccache SCCACHE_REGION=us-east-1 Pretty limited testing, no automated tests currently.
2016-06-03 22:09:38 +03:00
]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
[[package]]
name = "user32-sys"
version = "0.2.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "utf8-ranges"
version = "0.1.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
Use a named unix socket or named pipe to let the background server notify the client that it started successfully
2017-01-27 22:39:47 +03:00
[[package]]
name = "uuid"
version = "0.3.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"rand 0.3.14 (registry+https://github.com/rust-lang/crates.io-index)",
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "vec_map"
version = "0.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
Make DiskCache limit the size of the on-disk cache by using LruDiskCache. Fixes #29
2016-11-29 03:59:42 +03:00
[[package]]
name = "walkdir"
version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)",
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
]
Started using which crate (#14)
2016-07-14 13:38:26 +03:00
[[package]]
name = "which"
switch back to released version of which, switch to a rusoto branch with a bug fix
2016-08-08 22:06:06 +03:00
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
Started using which crate (#14)
2016-07-14 13:38:26 +03:00
dependencies = [
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)",
Started using which crate (#14)
2016-07-14 13:38:26 +03:00
]
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
[[package]]
name = "winapi"
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
version = "0.2.8"
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "winapi-build"
version = "0.1.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
[[package]]
name = "ws2_32-sys"
version = "0.2.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
Update to a released rusoto, and 'cargo update' while we're at it
2016-07-15 18:06:16 +03:00
"winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)",
Commit Cargo.lock
2016-05-12 22:03:37 +03:00
"winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)",
]
Pull in app_dirs to get a better default local disk cache location
2016-11-17 18:59:15 +03:00
[[package]]
name = "xdg"
version = "2.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
[[package]]
name = "zip"
Store file permissions for files in cache
2017-03-07 15:52:53 +03:00
version = "0.2.2"
Implement fetching/storing to cache, interfaces for doing so, implement a disk cache
2016-05-21 21:14:26 +03:00
source = "registry+https://github.com/rust-lang/crates.io-index"
dependencies = [
"flate2 0.2.14 (registry+https://github.com/rust-lang/crates.io-index)",
"msdos_time 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)",
"podio 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)",
"time 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)",
]
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
[metadata]
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum advapi32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "e06588080cb19d0acb6739808aafa5f26bfb2ca015b2b6370028b44cf7cb8a9a"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum aho-corasick 0.5.2 (registry+https://github.com/rust-lang/crates.io-index)" = "2b3fb52b09c1710b961acb35390d514be82e4ac96a9969a8e38565a29b878dc9"
"checksum ansi_term 0.7.5 (registry+https://github.com/rust-lang/crates.io-index)" = "30275ad0ad84ec1c06dde3b3f7d23c6006b7d76d61a85e7060b426b747eff70d"
Pull in app_dirs to get a better default local disk cache location
2016-11-17 18:59:15 +03:00
"checksum app_dirs 1.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "b7d1c0d48a81bbb13043847f957971f4d87c81542d80ece5e84ba3cba4058fd4"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"checksum base64 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "065a0ce220ab84d0b6d5ae3e7bb77232209519c366f51f946fe28c19e84989d0"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum bincode 1.0.0-alpha6 (git+https://github.com/TyOverby/bincode)" = "<none>"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum bitflags 0.5.0 (registry+https://github.com/rust-lang/crates.io-index)" = "4f67931368edf3a9a51d29886d245f1c3db2f1ef0dcc9e35ff70341b78c10d23"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum bitflags 0.7.0 (registry+https://github.com/rust-lang/crates.io-index)" = "aad18937a628ec6abcd26d1489012cc0e18c21798210f491af69ded9b881106d"
"checksum byteorder 1.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "c40977b0ee6b9885c9013cd41d9feffdd22deb3bb4dc3a71d901cc7a77de18c8"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"checksum bytes 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)" = "46112a0060ae15e3a3f9a445428a53e082b91215b744fa27a1948842f4a64b96"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum cfg-if 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "de1e760d7b6535af4241fca8bd8adf68e2e7edacc6b29f5d399050c5e48cf88c"
"checksum chrono 0.2.25 (registry+https://github.com/rust-lang/crates.io-index)" = "9213f7cd7c27e95c2b57c49f0e69b1ea65b27138da84a170133fd21b07659c00"
"checksum clap 2.9.2 (registry+https://github.com/rust-lang/crates.io-index)" = "4cbebe3ce784f9c63d83684d07cf2da470b88bb149ac17dc262b3062e6fe8d93"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum core-foundation 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "25bfd746d203017f7d5cbd31ee5d8e17f94b6521c7af77ece6c9e4b2d4b16c67"
"checksum core-foundation-sys 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "065a5d7ffdcbc8fa145d6f0746f3555025b9097a9e9cda59f7467abae670c78d"
"checksum crypt32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "e34988f7e069e0b2f3bfc064295161e489b2d4e04a2e4248fb94360cdf00b4ec"
daemonize the server process on unix
2016-10-26 23:22:52 +03:00
"checksum daemonize 0.2.3 (registry+https://github.com/rust-lang/crates.io-index)" = "0239832c1b4ca406d5ec73728cf4c7336d25cf85dd32db9e047e9e706ee0e935"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum dtoa 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)" = "80c8b71fd71146990a9742fc06dcbbde19161a267e0ad4e572c35162f4578c90"
When hashing Rust compile arguments, sort -L and --cfg args as well as --extern
2017-04-03 17:21:13 +03:00
"checksum either 1.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "18785c1ba806c258137c937e44ada9ee7e69a37e3c72077542cd2f069d78562a"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum env_logger 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "82dcb9ceed3868a03b335657b85a159736c961900f7e7747d3b0b97b9ccb5ccb"
Use error-chain to define Result+Error types, refactor cmdline::parse to use it.
2017-01-28 00:01:29 +03:00
"checksum error-chain 0.7.2 (registry+https://github.com/rust-lang/crates.io-index)" = "318cb3c71ee4cdea69fdc9e15c173b245ed6063e1709029e8fd32525a881120f"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum fern 0.3.5 (registry+https://github.com/rust-lang/crates.io-index)" = "4d2f58d053ad7791bfaad58a3f3541fe2d2aecc564dd82aee7f92fa402c054b2"
"checksum filetime 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)" = "5363ab8e4139b8568a6237db5248646e5a8a2f89bd5ccb02092182b11fd3e922"
"checksum flate2 0.2.14 (registry+https://github.com/rust-lang/crates.io-index)" = "3eeb481e957304178d2e782f2da1257f1434dfecbae883bafb61ada2a9fea3bb"
Update futures dependency required by futures-cpupool Shoulda run the last commit through CI!
2017-03-22 02:15:56 +03:00
"checksum futures 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)" = "8e51e7f9c150ba7fd4cee9df8bf6ea3dea5b63b68955ddad19ccd35b71dcfb4d"
Update the futures-cpupool dependency Previous versions of futures-cpupool depended on crossbeam which unfortunately has known segfaults (spurious) so the updated version no longer uses futures-cpupool and instead relies on channels in the standard library.
2017-03-22 02:09:12 +03:00
"checksum futures-cpupool 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "a283c84501e92cade5ea673a2a7ca44f71f209ccdd302a3e0896f50083d2c5ff"
Update gcc crate in Cargo.lock Picks up support for `-C target-feature=+crt-static` on MSVC and such
2017-02-25 12:48:02 +03:00
"checksum gcc 0.3.43 (registry+https://github.com/rust-lang/crates.io-index)" = "c07c758b972368e703a562686adb39125707cc1ef3399da8c019fc6c2498a75d"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum gdi32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "0912515a8ff24ba900422ecda800b52f4016a56251922d397c576bf92c690518"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum getopts 0.2.14 (registry+https://github.com/rust-lang/crates.io-index)" = "d9047cfbd08a437050b363d35ef160452c5fe8ea5187ae0a624708c91581d685"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"checksum httparse 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "a6e7a63e511f9edffbab707141fbb8707d1a3098615fb2adbd5769cdfcc9b17d"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum hyper 0.11.0-a.0 (git+https://github.com/hyperium/hyper)" = "<none>"
"checksum hyper-tls 0.0.0 (git+https://github.com/hyperium/hyper-tls)" = "<none>"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum idna 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "1053236e00ce4f668aeca4a769a09b3bf5a682d802abd6f3cb39374f6b162c11"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"checksum iovec 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "29d062ee61fccdf25be172e70f34c9f6efc597e1fb8f6526e8437b2046ab26be"
When hashing Rust compile arguments, sort -L and --cfg args as well as --extern
2017-04-03 17:21:13 +03:00
"checksum itertools 0.5.10 (registry+https://github.com/rust-lang/crates.io-index)" = "4833d6978da405305126af4ac88569b5d71ff758581ce5a987dbfa3755f694fc"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum itoa 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)" = "eb2f404fbc66fd9aac13e998248505e7ecb2ad8e44ab6388684c5fb11c6c251c"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum kernel32-sys 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "7507624b29483431c0ba2d82aece8ca6cdba9382bff4ddd0f7490560c056098d"
"checksum language-tags 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "a91d884b6667cd606bb5a69aa0c99ba811a115fc68915e7056ec08a46e93199a"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum lazy_static 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "6abe0ee2e758cd6bc8a2cd56726359007748fbf4128da998b65d0b70f881e19b"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"checksum lazycell 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "ce12306c4739d86ee97c23139f3a34ddf0387bbf181bc7929d287025a8c3ef6b"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"checksum libc 0.2.21 (registry+https://github.com/rust-lang/crates.io-index)" = "88ee81885f9f04bff991e306fea7c1c60a5f0f9e409e99f6b40e3311a3363135"
Make DiskCache limit the size of the on-disk cache by using LruDiskCache. Fixes #29
2016-11-29 03:59:42 +03:00
"checksum linked-hash-map 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "bda158e0dabeb97ee8a401f4d17e479d6b891a14de0bba79d5cc2d4d325b5e48"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum local-encoding 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "e1ceb20f39ff7ae42f3ff9795f3986b1daad821caaa1e1732a0944103a5a1a66"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"checksum log 0.3.7 (registry+https://github.com/rust-lang/crates.io-index)" = "5141eca02775a762cc6cd564d8d2c50f67c0ea3a372cbf1c51592b3e029e10ad"
Use my lru-cache branch until the changes land upstream
2016-12-07 23:58:42 +03:00
"checksum lru-cache 0.1.0 (git+https://github.com/luser/lru-cache?branch=non-mut-get)" = "<none>"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"checksum matches 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)" = "efd7622e3022e1a6eaa602c4cea8912254e5582c9c692e9167714182244801b1"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum memchr 0.1.11 (registry+https://github.com/rust-lang/crates.io-index)" = "d8b629fb514376c675b98c1421e80b151d3817ac42d7c667717d282761418d20"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum metadeps 1.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "829fffe7ea1d747e23f64be972991bc516b2f1ac2ae4a3b33d8bea150c410151"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"checksum mime 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "b5c93a4bd787ddc6e7833c519b73a50883deb5863d76d9b71eb8216fb7f94e66"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum miniz-sys 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "9d1f4d337a01c32e1f2122510fed46393d53ca35a7f429cb0450abaedfa3ed54"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum mio 0.6.6 (registry+https://github.com/rust-lang/crates.io-index)" = "f27d38f824a0d267d55b29b171e9e99269a53812e385fa75c1fe700ae254a6a4"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"checksum mio-named-pipes 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "82f43a815b57d2d652550f3d20cec88a495bb2d0956aa873dc43040278455677"
"checksum mio-uds 0.6.3 (registry+https://github.com/rust-lang/crates.io-index)" = "c6a39bf21f6bf5cb2f07513c1e3519291db579386c34f8d1dd3b4f8565b53db8"
"checksum miow 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "8c1f2f3b1cf331de6896aabf6e9d55dca90356cc9960cca7eaaf408a355ae919"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum msdos_time 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)" = "c04b68cc63a8480fb2550343695f7be72effdec953a9d4508161c3e69041c7d8"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum native-tls 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "b805ee0e8fa268f67a4e5c7f4f80adb8af1fc4428ea0ce5b0ecab1430ef17ec0"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum net2 0.2.25 (registry+https://github.com/rust-lang/crates.io-index)" = "1dd775c6de972a1f57a34016f3b2bdc9e086e948f870b38675d1db410a21566b"
"checksum num 0.1.34 (registry+https://github.com/rust-lang/crates.io-index)" = "d2ee34a0338c16ae67afb55824aaf8852700eb0f77ccd977807ccb7606b295f6"
"checksum num-bigint 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)" = "fbc450723a2fe91d332a29edd8660e099b937d29e1a3ebe914e0da3f77ac1ad3"
"checksum num-complex 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)" = "8aabbc079e1855ce8415141fee0ebebf171f56505373b3a966e2716ad7c0e555"
"checksum num-integer 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)" = "fb24d9bfb3f222010df27995441ded1e954f8f69cd35021f6bef02ca9552fb92"
"checksum num-iter 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)" = "287a1c9969a847055e1122ec0ea7a5c5d6f72aad97934e131c83d5c08ab4e45c"
"checksum num-rational 0.1.32 (registry+https://github.com/rust-lang/crates.io-index)" = "48cdcc9ff4ae2a8296805ac15af88b3d88ce62128ded0cb74ffb63a587502a84"
"checksum num-traits 0.1.33 (registry+https://github.com/rust-lang/crates.io-index)" = "9d06e4a6e3873968e97c0f9e6abbe4220d53793f63cdd87dbf0a90fe3b140d87"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"checksum num_cpus 1.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "a225d1e2717567599c24f88e49f00856c6e825a12125181ee42c4257e3688d39"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum number_prefix 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)" = "084d05f4bf60621a9ac9bde941a410df548f4de9545f06e5ee9d3aef4b97cd77"
Pull in app_dirs to get a better default local disk cache location
2016-11-17 18:59:15 +03:00
"checksum ole32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "5d2c49021782e5233cd243168edfa8037574afed4eba4bbaf538b3d8d1789d8c"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum openssl 0.9.6 (registry+https://github.com/rust-lang/crates.io-index)" = "0c00da69323449142e00a5410f0e022b39e8bbb7dc569cee8fc6af279279483c"
"checksum openssl-sys 0.9.6 (registry+https://github.com/rust-lang/crates.io-index)" = "b1482f9a06f56c906007e17ea14d73d102210b5d27bc948bf5e175f493f3f7c3"
"checksum pkg-config 0.3.9 (registry+https://github.com/rust-lang/crates.io-index)" = "3a8b4c6b8165cd1a1cd4b9b120978131389f64bdaf456435caa41e630edba903"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum podio 0.1.5 (registry+https://github.com/rust-lang/crates.io-index)" = "e5422a1ee1bc57cc47ae717b0137314258138f38fd5f3cea083f43a9725383a0"
"checksum pulldown-cmark 0.0.3 (registry+https://github.com/rust-lang/crates.io-index)" = "8361e81576d2e02643b04950e487ec172b687180da65c731c03cf336784e6c07"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum quote 0.3.15 (registry+https://github.com/rust-lang/crates.io-index)" = "7a6e920b65c65f10b2ae65c831a81a073a89edd28c7cce89475bff467ab4167a"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum rand 0.3.14 (registry+https://github.com/rust-lang/crates.io-index)" = "2791d88c6defac799c3f20d74f094ca33b9332612d9aef9078519c82e4fe04a5"
Add a Redis cache variant This cache module uses a Redis instance. To make sccache use this, set SCCACHE_REDIS to redis://[:<passwd>@]<hostname>[:port][/<db>]. The maximum and current cache size is retrieved from the INFO and CONFIG GET redis command.
2017-02-16 22:56:41 +03:00
"checksum redis 0.8.0 (registry+https://github.com/rust-lang/crates.io-index)" = "02a92e223490cc63d9230c4cdf132a48ce154ab1e063558e3841e219c2ea3f91"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum regex 0.1.73 (registry+https://github.com/rust-lang/crates.io-index)" = "56b7ee9f764ecf412c6e2fff779bca4b22980517ae335a21aeaf4e32625a5df2"
"checksum regex-syntax 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "31040aad7470ad9d8c46302dcffba337bb4289ca5da2e3cd6e37b64109a85199"
"checksum retry 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "29460f6011a25fc70b22010e796bd98330baccaa0005cba6f90b858a510dec0d"
"checksum rust-crypto 0.2.36 (registry+https://github.com/rust-lang/crates.io-index)" = "f76d05d3993fd5f4af9434e8e436db163a12a9d40e1a58a726f27a01dfd12a2a"
"checksum rustc-serialize 0.3.19 (registry+https://github.com/rust-lang/crates.io-index)" = "6159e4e6e559c81bd706afe9c8fd68f547d3e851ce12e76b1de7914bab61691b"
"checksum rustc_version 0.1.7 (registry+https://github.com/rust-lang/crates.io-index)" = "c5f5376ea5e30ce23c03eb77cbe4962b988deead10910c372b226388b594c084"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum schannel 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "0168331892e26bcd763535c1edd4b850708d0288b0e73942c116bbbf8e903c7f"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"checksum scoped-tls 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f417c22df063e9450888a7561788e9bd46d3bb3c1466435b4eccb903807f147d"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum secur32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "3f412dfa83308d893101dd59c10d6fda8283465976c28c287c5c855bf8d216bc"
"checksum security-framework 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)" = "d7c1ff1c71e4e4474b46ded6687f0c28c721de2f5a05577e7f533d36330e4e3a"
"checksum security-framework-sys 0.1.10 (registry+https://github.com/rust-lang/crates.io-index)" = "5103c988054803538fe4d85333abf4c633f069510ab687dc71a50572104216d0"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum semver 0.1.20 (registry+https://github.com/rust-lang/crates.io-index)" = "d4f410fedcf71af0345d7607d246e7ad15faaadd49d240ee3b24e5dc21a820ac"
update serde to 0.9.13 to pick up a fix for OsString deserialization on Windows (fixes Windows breakage)
2017-04-07 13:55:26 +03:00
"checksum serde 0.9.13 (registry+https://github.com/rust-lang/crates.io-index)" = "231dfd55909400769e437326cfb4af8bec97c3dd56ab3d02df8ef5c7e00f179b"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum serde_codegen_internals 0.14.1 (registry+https://github.com/rust-lang/crates.io-index)" = "4d52006899f910528a10631e5b727973fe668f3228109d1707ccf5bad5490b6e"
"checksum serde_derive 0.9.11 (registry+https://github.com/rust-lang/crates.io-index)" = "f15ea24bd037b2d64646b4d934fa99c649be66e3f7b29fb595a5543b212b1452"
"checksum serde_json 0.9.9 (registry+https://github.com/rust-lang/crates.io-index)" = "dbc45439552eb8fb86907a2c41c1fd0ef97458efb87ff7f878db466eb581824e"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum sha1 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "cc30b1e1e8c40c121ca33b86c23308a090d19974ef001b4bf6e61fd1a0fb095c"
Pull in app_dirs to get a better default local disk cache location
2016-11-17 18:59:15 +03:00
"checksum shell32-sys 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "72f20b8f3c060374edb8046591ba28f62448c369ccbdc7b02075103fb3a9e38d"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum skeptic 0.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "24ebf8a06f5f8bae61ae5bbc7af7aac4ef6907ae975130faba1199e5fe82256a"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"checksum slab 0.3.0 (registry+https://github.com/rust-lang/crates.io-index)" = "17b4fcaed89ab08ef143da37bc52adbcc04d4a69014f4c1208d6b51f0c47bc23"
"checksum smallvec 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "4c8cbcd6df1e117c2210e13ab5109635ad68a929fcbb8964dc965b76cb5ee013"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum strsim 0.4.1 (registry+https://github.com/rust-lang/crates.io-index)" = "0d5f575d5ced6634a5c4cb842163dab907dc7e9148b28dc482d81b8855cbe985"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum syn 0.11.9 (registry+https://github.com/rust-lang/crates.io-index)" = "480c834701caba3548aa991e54677281be3a5414a9d09ddbdf4ed74a569a9d19"
"checksum synom 0.11.3 (registry+https://github.com/rust-lang/crates.io-index)" = "a393066ed9010ebaed60b9eafa373d4b1baac186dd7e008555b0f702b51945b6"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"checksum take 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "b157868d8ac1f56b64604539990685fa7611d8fa9e5476cf0c02cf34d32917c5"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum tempdir 0.3.4 (registry+https://github.com/rust-lang/crates.io-index)" = "0b62933a3f96cd559700662c34f8bab881d9e3540289fb4f368419c7f13a5aa9"
"checksum term_size 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "a6a7c9a4de31e5622ec38533988a9e965aab09b26ee8bd7b8b0f56d488c3784d"
"checksum thread-id 2.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "a9539db560102d1cef46b8b78ce737ff0bb64e7e18d35b2a5688f7d097d0ff03"
"checksum thread_local 0.2.6 (registry+https://github.com/rust-lang/crates.io-index)" = "55dd963dbaeadc08aa7266bf7f91c3154a7805e32bb94b820b769d2ef3b4744d"
"checksum time 0.1.35 (registry+https://github.com/rust-lang/crates.io-index)" = "3c7ec6d62a20df54e07ab3b78b9a3932972f4b7981de295563686849eb3989af"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"checksum tokio-core 0.1.6 (registry+https://github.com/rust-lang/crates.io-index)" = "99e958104a67877907c1454386d5482fe8e965a55d60be834a15a44328e7dc76"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum tokio-io 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "48f55df1341bb92281f229a6030bc2abffde2c7a44c6d6b802b7687dd8be0775"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"checksum tokio-process 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "5414643999022a6efe54bc9ce03bae9db12694900208b4d3c56a18af71b8a4cb"
"checksum tokio-proto 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "8fbb47ae81353c63c487030659494b295f6cb6576242f907f203473b191b0389"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum tokio-serde 0.1.0 (git+https://github.com/carllerche/tokio-serde)" = "<none>"
"checksum tokio-serde-bincode 0.1.0 (git+https://github.com/alexcrichton/tokio-serde-bincode)" = "<none>"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"checksum tokio-service 0.1.0 (registry+https://github.com/rust-lang/crates.io-index)" = "24da22d077e0f15f55162bdbdc661228c1581892f52074fb242678d015b45162"
Integrate process spawning with tokio-process This commit replaces all process spawning with the futures-powered versions in the `tokio-process` crate. Most functions were already ready to return futures, but a few minor updates were made to ensure that functions could return futures. Additionally, some `&CpuPool` arguments have shown up to allow executing work on a thread pool, such as filesystem operations. A number of compilers write data to tempdirs and such before running a compiler, and these are all executed on thread pools rather than the main thread. My main takeaway from this commit is that translating synchronous to asynchronous code isn't always easy. The "easy" version ends up being relatively wasteful in terms of clones for low-level performance, but probably shouldn't matter much in the context of spawning processes. Other than that though I found that there was a translation for all patterns found didn't actually hit any bugs (yet at least) from the translation.
2017-02-01 03:19:35 +03:00
"checksum tokio-signal 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "3d121715f6917878a0df69f39365d01dd66c4463e4ba19efdcddcdfeb1bcb2bc"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"checksum tokio-tls 0.1.2 (registry+https://github.com/rust-lang/crates.io-index)" = "666266622d9a4d1974a0beda33d505999515b0c60edc0c3fda09784e56609a97"
"checksum tokio-uds 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)" = "bd209039933255ea77c6d7a1d18abc20b997d161acb900acca6eb74cdd049f31"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum toml 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "736b60249cb25337bc196faa43ee12c705e426f3d55c214d73a4e7be06f92cb4"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum unicase 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "13a5906ca2b98c799f4b1ab4557b76367ebd6ae5ef14930ec841c74aed5f3764"
Update to using the new tokio-io crate This is an abstraction of the Tokio stack now and doesn't have much impact on sccache itself but I figured it'd be good to update a few deps and pick up the recent version of things! Plus that and I'd like to start using the length_delimited module soon...
2017-03-09 21:00:18 +03:00
"checksum unicode-bidi 0.2.5 (registry+https://github.com/rust-lang/crates.io-index)" = "d3a078ebdd62c0e71a709c3d53d2af693fe09fe93fbff8344aebe289b78f9032"
"checksum unicode-normalization 0.1.4 (registry+https://github.com/rust-lang/crates.io-index)" = "e28fa37426fceeb5cf8f41ee273faa7c82c47dc8fba5853402841e665fcd86ff"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum unicode-width 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "2d6722facc10989f63ee0e20a83cd4e1714a9ae11529403ac7e0afd069abc39e"
Move from protobuf to bincode for a wire format This commit migrates away from the `protobuf` crate to instead just working with bincode on the wire as a serialization format. This is done by leveraging a few different crates: * The `bincode` and `serde_derive` crates are used to define serialization for Rust structures as well as provide a bincode implementation. * The `tokio_io::codec::length_delimited` module implements framing via length prefixes to transform an asynchronous stream of bytes into a literal `Stream` of `BytesMut`. * The `tokio_serde_bincode` crate is then used to tie it all together, parsing these `BytesMut` as the request/response types of sccache. Most of the changes here are related to moving away from the protobuf API throughout the codebase (e.g. `has_foo` and `take_foo`) towards a more rustic-ish API that just uses enums/structs. Overall it felt quite natural (as one would expect) to just use the raw enum/struct values. This may not be quite as performant as before but that doesn't really apply to sccache's use case where perf is hugely dominated by actually compiling and hashing, so I'm not too too worried about that. My personal motivation for this is twofold: 1. Using `protobuf` was a little clunky throughout the codebase and definitely had some sharp edges that felt good to smooth out. 2. There's currently what I believe some mysterious segfault and/or stray write happening in sccache and I'm not sure where. The `protobuf` crate had a lot of `unsafe` code and in lieu of actually auditing it I figured it'd be good to kill two birds with one stone. I have no idea if this fixes my segfault problem (I never could reproduce it) but I figured it's worth a shot.
2017-03-22 20:21:52 +03:00
"checksum unicode-xid 0.0.4 (registry+https://github.com/rust-lang/crates.io-index)" = "8c1f860d7d29cf02cb2f3f359fd35991af3d30bac52c57d265a3c461074cb4dc"
Rewrite the server module with Tokio This commit rewrites the `server` module of sccache to be backed with Tokio. The previous version was written with `mio`, which Tokio is built on, but is unfortunately less ergonomic. Tokio is the state-of-the-art for asynchronous programming in Rust and sccache serves as a great testing ground for ergonomics! It's intended that the support added here will eventually extend to many other operations that sccache does as well. For example thread spawning has all been replaced with `CpuPool` to have a shared pool for I/O operations and such (namely the filesystem). Eventually the HTTP requests made by the S3 backend can be integrated with the Tokio branch of Hyper as well to run that on the event loop instead of in a worker thread. I'd also like to eventually extend this with `tokio-process` as well to move process spawning off helper threads as well, but I'm leaving that to a future commit as well. Overall I found the transition was quite smooth, with the high level architecture look like: * The `tokio-proto` crate is used in streaming mode. The streaming part is used for the one RPC sccache gets which requires a second response to be sent later on. This second response is the "response body" in tokio-proto terms. * All of sccache's logic is manifested in an implementation of the `Service` trait. * The transport layer is provided with `tokio_core::io::{Framed, Codec}`, and simple deserialization/serialization is performed with protobuf. Some differences in design are: * The `SccacheService` for now is just a bunch of reference-counted pointers, making it cheap to clone. As the futures it returns progress they will each retain a reference to a cloned copy of the `SccacheService`. Before all this data was just stored and manipulated in a struct directly, but it's now directly managed through shared memory. * The storage backends share a thread pool with the main server instead of spawning threads. And finally, some things I've learned along the way: * Sharing data between futures isn't a trivial operation. It took an explicit decision to use `Rc` and I'm not sure I'm 100% happy with how the ergonomics played out. * Shutdown is pretty tricky here. I've tried to carry over all the previous logic but it definitely required not using `TcpServer` in tokio-proto at the very least, and otherwise required a few custom futures and such to track the various states. I have a hunch that tokio-proto could provide more options out of the box for something like this.
2017-01-31 04:04:03 +03:00
"checksum url 1.4.0 (registry+https://github.com/rust-lang/crates.io-index)" = "f5ba8a749fb4479b043733416c244fa9d1d3af3d7c23804944651c8a448cb87e"
Integrate the Tokio branch of Hyper This commit pulls in Hyper from its master branch which contains the asynchronous/Tokio integration. All instances of HTTP, namely when interacting with S3, are now all powered by async Hyper and work with futures rather than synchronous requests on thread pools.
2017-02-02 03:30:26 +03:00
"checksum user32-sys 0.2.0 (registry+https://github.com/rust-lang/crates.io-index)" = "4ef4711d107b21b410a3a974b1204d9accc8b10dad75d8324b5d755de1617d47"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum utf8-ranges 0.1.3 (registry+https://github.com/rust-lang/crates.io-index)" = "a1ca13c08c41c9c3e04224ed9ff80461d97e121589ff27c753a16cb10830ae0f"
Use a named unix socket or named pipe to let the background server notify the client that it started successfully
2017-01-27 22:39:47 +03:00
"checksum uuid 0.3.1 (registry+https://github.com/rust-lang/crates.io-index)" = "1a9ff57156caf7e22f37baf3c9d8f6ce8194842c23419dafcb0716024514d162"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum vec_map 0.6.0 (registry+https://github.com/rust-lang/crates.io-index)" = "cac5efe5cb0fa14ec2f84f83c701c562ee63f6dcc680861b21d65c682adfb05f"
Make DiskCache limit the size of the on-disk cache by using LruDiskCache. Fixes #29
2016-11-29 03:59:42 +03:00
"checksum walkdir 1.0.2 (registry+https://github.com/rust-lang/crates.io-index)" = "412de40d080c201baaed542b43a9e4c24f5df30d1d6c3362689ed8d0c583e74a"
make connecting to server a little more lenient when starting the server
2016-10-26 01:14:21 +03:00
"checksum which 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "9d9cef3b81ef66a3d575f226f1ce4ba48d1670c4529627b9496b6eb95fec48b0"
"checksum winapi 0.2.8 (registry+https://github.com/rust-lang/crates.io-index)" = "167dc9d6949a9b857f3451275e911c3f44255842c1f7a76f33c55103a909087a"
"checksum winapi-build 0.1.1 (registry+https://github.com/rust-lang/crates.io-index)" = "2d315eee3b34aca4797b2da6b13ed88266e6d612562a0c46390af8299fc699bc"
"checksum ws2_32-sys 0.2.1 (registry+https://github.com/rust-lang/crates.io-index)" = "d59cefebd0c892fa2dd6de581e937301d8552cb44489cdff035c6187cb63fa5e"
Pull in app_dirs to get a better default local disk cache location
2016-11-17 18:59:15 +03:00
"checksum xdg 2.0.0 (registry+https://github.com/rust-lang/crates.io-index)" = "77b831a5ba77110f438f0ac5583aafeb087f70432998ba6b7dcb1d32185db453"
Store file permissions for files in cache
2017-03-07 15:52:53 +03:00
"checksum zip 0.2.2 (registry+https://github.com/rust-lang/crates.io-index)" = "f360ab8d0a447363447d49847f029fc1f6ed0c2a334b5d24291e80fb381f0555"