zstd is faster and gives slightly smaller (~5%) compressed blobs than
deflate does, as measured on a Firefox build. Rather than inventing our
own compressed archive format, we piggyback on top of zip's "stored"
files to stuff zstd-compressed blobs in the zip archive.
The current trimming of rlibs in the dist-sccache case looks like:
1. Find the `rust.metadata.bin` file in the rlib (archive).
2. Read its data.
3. Create a new archive that will only contain said file.
4. Copy the data into the new archive.
We can do better than this, by doing the following:
1. Find the `rust.metadata.bin` file in the rlib (archive).
2. Create a new archive that will only contain said file.
3. Copy the data from the original rlib directly into the new archive.
We thus save a copy.
All that being said, it looks like recent versions of Rust don't
actually output `rust.metadata.bin` files into rlibs...unless we ought
to be only taking the `lib.rmeta` file that lives inside the archive.
But that would be a separate fix. (It's also great that we're seemingly
copying the rmeta file twice, once because it lives in the rlib and once
because it lives on disk as a separate file...)
Because `anyhow` is what seems to be the most common error library for
applications these days.
- The global `Result` type is now `anyhow::Result`.
- In errors.rs, there's no need for any boilerplate to wrap all the foreign
errors seen: `hyper::Error`, `io:Error`, etc.
- The internal errors that we care about are now separate types, rather
than within an enum, because that works better when we need to check for them
by downcasting an `anyhow::Error`. And it's nice to write
`Err(ProcessError(output))` rather than
`Err(ErrorKind::ProcessError(output))`.
- The `Which` error was unused and is removed.
- The most common change is that `.chain_err()` is changed to
`.context`/`.with_context()`.
- `anyhow!` is used where necessary, mostly to promote a string to an
`anyhow::Error`.
- Errors within futures: `FutureChainErr`/`.chain_err()` is changed to
`FutureContext`/`fcontext`/`fwith_context`. The `f` prefix is because I found
it helpful to distinguish these cases from the simple error cases.
- `BuilderIncoming`, `SchedulerIncoming`, `ServerIncoming` no longer have an
`Error` associated type, we just use `anyhow::Error` uniformly.
- `e.display_chain()` changes to `format!("{:?}")`, because they both print the
full cause chain, and the backtrace (if present).
- A few places where the old code was doing something weird or more
complicated than seemed necessary, I generally tried to replace it with
something simpler and more typical. Two places used `with_boxed_chain()`,
which doesn't have an equivalent in `anyhow`, so I did my best to do
something reasonable.
- In `src/server.rs` we now import `std::task::Context` as `TaskContext` to
avoid overshadowing the `anyhow::Context` trait :(
I was looking through old issues, and #160 mentioned that crc32
checksumming for zip files was showing up in profiles. zip-rs 0.4 uses
a byte-at-a-time version of crc32; zip-rs 0.5 relies on the `crc32fast`
crate, which features faster implementations.
This patch looks like a routine crate version update, but it fixes
something very fundamentally wrong with the tests: `escargot`, to find
the `sccache` binary during tests, would actually run `cargo build` and
parse the output. There were all manner of hacks to try and build the
binary with the correct set of features, none of which were reliable,
and the hacks were not reliably present in all places, either.
When the hacks were not present, `escargot` would rebuild the entire
sccache crate, which was responsible for the `sccache_cargo` test taking
entirely too long. When the hacks failed utterly, perfectly fine
patches would mysteriously fail, as seen in #774.
We can get rid of `escargot` because `assert-cmd` has been updated to a)
not use `escargot` under the hood; and b) do something smarter to locate
the target binary. The upshot is fewer mysterious test failures and
significantly faster running tests.
This gets us one step into tokio 0.2, with its new scheduler. This makes
things significantly faster on my 32-core 64-threads machine, with
Firefox builds with 100% cache hits at -j64 going down from 3 minutes to
2:24. Also, with the old scheduler, with 100% cache hits, -j16 was
actually faster than -j64, which is not the case anymore.
BLAKE3 is designed to be a very high performance cryptographic hash. The
BLAKE3 team has shown 8.5x higher single-thread performance than SHA-512
on modern server hardware (AWS `c5.metal`). This change did not result
in a significant improvement to my observed local build times, but
newer hardware may see a meaningful improvement.
Signed-off-by: George Hahn <george.hahn.vhs@gmail.com>
Omid Rad
Markus Westerlind
Arne Luenser
Bernhard Schuster
Markus Westerlind
Mike Hommey
Martin Matous
Jacob Hughes
Daniel Holbert
Philippe Normand
Nicholas Nethercote
Nathan Froyd
Emilio Cobos Álvarez
Ryan Dahl
George Hahn