After reading all the docs, I came away with the impression that the local storage may be safely used concurrently. I tried that, and it almost worked, until it filled it. Clarifying this should fix#407 and help future users avoid this problem.
This also simplifies `azure::credentials::EnvironmentProvider` (which only has 1 method) into simply exporting a `credentials_from_environment()` function.
Fixes https://github.com/mozilla/sccache/issues/514#issuecomment-1061945431 (comment 2022-03-08 by @mitchhentges)
Any console-based subprocess spawned with CREATE_NO_WINDOW actually has a hidden console, and thus an associated conhost process. Since the sccache server is already started with CREATE_NO_WINDOW, it's unnecessary to spawn further subprocesses with CREATE_NO_WINDOW. Removing this flag allows subprocesses to share the sccache server's hidden console, thus avoiding each subprocess from getting its own conhost.
For an extended explanation see this comment and its follow-ups:
https://github.com/mozilla/sccache/issues/514#issuecomment-1068961965
* Bump the MSRV to 1.58
Bumps the official MSRV to 1.58.
This is likely more than required the upcoming version bumps.
* Upgrade cargo zstd to 0.10
* Bump serial_test to v0.6
Changelog does not mention any breaking changes.
* Bump blake to v1
* Bump rouille to 3.5
Required to bump syslog to v6, since otherwise no common version
can be selected for `time`
* Bump syslog to v6
No changelog available. Depends on rouille bump to 3.5, to get a
common version of ´time´.
* Upgrade JWT to v8
The explicit insecure decode method was removed and replaced by
an option for the Validation struct.
The `exp` field had to be added to JobJwt, since otherwise the validation would fail at runtime.
Apparently the `exp` field is required now, even if the valdiation of `exp` is turned off.
* Add -fminimize-whitespace preprocessor flag when Clang >= 14 to increase cache hits
This proof of concept tries to take advantage of the new `-fminimize-whitespace`
preprocessor option in Clang. `-fminimize-whitespace` was added to Clang with the
intention to help increase the chance of a cache hit in tools like `sccache`
when only whitespace changes.
I also added a `version()` method to CCompilerImpl trait to expose the compiler
version to the preprocessor. This could be useful if version-dependent
features are added to `sccache` in the future.
Feedback on this is definitely welcome!
* Add tests for clang whitespace normalization
* Use clang -fminimize-whitespace to incread cache hits.
* Use ubuntu-22.04 for testing Clang 14
* Move compiler version from CCompiler to each CCompilerImpl.
* Use semver
Co-authored-by: Max Fan <root@max.fan>
Co-authored-by: Michael Kruse <sccache@meinersbur.de>
* Replace `dist-client` usages of blocking `reqwest` to be async instead
The blocking implementation of `reqwest` now uses an internal `tokio`
runtime as of `0.10`. `tokio` doesn't like having nested runtimes, so it
raises an error when `request>=0.10` is used from a tokio-`async`
function.
The primary rationale for continuing to use "blocking" `reqwest` was
because its async implementation was "extremely limited" and didn't
support streamed data from a file.
However, now, we're able to leverage
`tokio::fs::File -> AsyncRead -> ReaderStream -> Body::wrap_stream()`,
which obsoletes this one use case for blocking `reqwest`.
Convert all blocking usages of `reqwest` to `async` accordingly.
Fixes#1161.
* Avoid nested `tokio` runtime in `create_state()`
`tokio` doesn't allow for nested runtimes, yet `create_state` is a
non-async function that needs to run nested async code.
I'm not sure _exactly_ how this ran successfully before, but I'm
guessing that the updated `tokio` was stricter about nested runtimes.
Either way, work around this by manually reusing the current thread's
runtime with `Handle::try_current().enter()`. If no existing handle
exists (such as when `create_state()` is called from
entirely-non-async-code), then manually use `config.pool` again.
`-std` and `-pedantic` were handled specially to resolve gcc-specific
behaviour. However, for MSVC, these flags can continue to be handled the
way they were before: as pass-through options.
Fixes#1150.
Adding /experimental:external or /experimental:W... causes caching to fail:
[2022-03-14T09:34:13Z DEBUG sccache::server] check_compiler: Supported compiler
[2022-03-14T09:34:13Z DEBUG sccache::server] parse_arguments: CannotCache(multiple input files): ["/nologo", "/TP", "-Dlibrary_EXPORTS", ... "/experimental:external", "/external:I", "C:\\src", "/external:W0", "/O2", "/Ob2", "/DNDEBUG", "-MD", "/showIncludes", "/Fosrc\\library\\CMakeFiles\\library.dir\\speak.cpp.obj", "/Fdsrc\\library\\CMakeFiles\\library.dir\\", "/FS", "-c", "C:\\src\\library\\speak.cpp"]
This change adds the additional flags to the msvc_args macro and simply
passes them on to the compiler.
Tests have been added that use forward slashes instead of dashes
Previously, there would occur a mismatch on the GCC 9 version,
preventing us from installing it properly. Instead of that, try just
pulling GCC 10 rather than both 9 and 10.
* More complete unit tests for Azure Blob Storage backend
Pull request #1109 introduced key prefix support for the Azure backend, with
unit tests for `normalize_key()`, but those tests didn't test whether the
Storage impl *actually* used `normalize_key()` when constructing an API
request to Azure.
This change makes the unit tests more complete, by testing the Storage impl
directly through the use of a mock HTTP server as Azure API endpoint.
* Split up Azure storage tests
* GCC: suppress rewrite_includes_only with pedantic flags
CoinMP/CoinUtils configure checks the <cmath> header:
configure:9903: sccache g++ -c -O3 -pipe -DNDEBUG -pedantic-errors -Wparentheses -Wreturn-type -Wcast-qual -Wall -Wpointer-arith -Wwrite-strings -Wconversion -Wno-unknown-pragmas -Wno-long-long -DCOINUTILS_BUILD conftest.cc >&5
<built-in>: error: ISO C++ does not support '__int128' for '__x' [-Wpedantic]
<built-in>: error: ISO C++ does not support '__int128' for 'abs' [-Wpedantic]
In file included from /usr/include/c++/10/cmath:42,
from conftest.cc:64:
<built-in>: error: ISO C++ does not support '__int128' for 'type name' [-Wpedantic]
The problem is that in GCC 10, the bits/std_abs.h declares abs() for
GCC extension __int128, which -pedantic then complains about.
Implement the same solution that icecream has for this problem: for GCC,
if any -pedantic etc. flag is given and GNU extensions are enabled, don't
use -fdirectives-only.
* GCC: add test for suppress rewrite_includes_only with pedantic flags
Requires refactoring the preprocess() a bit so that the generated
arguments can be inspected by a test.
- travis-ci and appveyor don't seem to be used anymore
- The cargo reference now recommends putting badges into
the Readme file, where we already a github CI badge.
* Use different hashing functions for different file types
This adds a new special hasher for static libraries, which contain timestamps
and other info that prevent them from being cachable in some cases.
* Update src/util.rs
Co-authored-by: Mitchell Hentges <mitch9654@gmail.com>
Co-authored-by: Jack Moffitt <jack@metajack.im>
Co-authored-by: Mitchell Hentges <mitch9654@gmail.com>
* Refactor sccache cargo test
- Improve cleanup by using Result instead of panicking, to ensure that
the temporary folder gets cleaned up.
Previously, when a test failed, no clean up would happen. Now the cleanup
happens, unless there is an unexpected panic somewhere in the intergration test code.
- this commit aims to make it easier to add new tests
Split sccache cargo tests
To make it clearer what is failing, split the cargo test into multiple
tests. Since Sccache can't be invoked in parallel, we use the
`serial_test` to serialize testing,otherwise cargo by default would
start them in parallel.
The logger is now also lazily initialized, so the first test to run will
initialize it. It now writes a linebreak, because otherwise the output
is hard to read with
`RUST_LOG=debug cargo test --test sccache_cargo -- --nocapture`.
We catch panics (which are not intended by the tests ) to ensure that the
temporary directory for the test gets cleaned up.
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
Remove panic=abort from CI coverage test
I suspect the commandline was just taken like that from the example
here: https://doc.rust-lang.org/stable/unstable-book/compiler-flags/profile.html
However, there seems to be no reason to actually abort, since we don't have
a custom test harness or anything.
Since we want to catch panics, so we can correctly clean up, we need
panic=unwind.
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
* Parse env_deps from dep-info
This will trigger rebuilds if environment variables changed that the rust code
depended on with env!.
On the Rust side this requires at least Rust 1.46, otherwise there will be
no env-dep info in the dep-info file. In that case we cannot detect
the dependency on the env-value, and behaviour is unchanged compared to
the current sccache behaviour.
With recent Rust versions however, we can trigger a rebuild if a variable
that is referenced via env! or option_env! is changed.
Other env variables like `CARGO_*` or `RUSTFLAGS` which may affect
the compilation are not listed in the dep-info file, so they have to
be blanket added (as is currently already the case).
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
* Add test for cargo env_deps
The previous commit added support for parsing "env_dep"
information from dep_info files. This commit adds a test
for changing an environment variable, that is referenced
in rust code via env!, and asserts that sccache rebuilds
and the rust code uses the new value of the changed
environment variable.
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
This is similar to CCACHE_EXTRAFILES and ICECC_EXTRAFILES: the extra
files are hashed for caching and packaged for sccache-dist.
Possibly the same variable CCACHE_EXTRAFILES could be used instead of
a distinct one?
This can be implemented in one place for the C/C++ compilers, but not
also for Rust, which currently doesn't have extra_hash_files, so that
remains TODO.
The crypto crates are interdependent so bump them in one go.
It looks like `new_varkey()` was renamed to `new_from_slice()`, at least
that is what I gather from the changed example in the documentation of
the hmac crate between version 0.11 and 0.10.
See https://docs.rs/hmac/0.11.0/hmac/index.html
The changelog doesn't mention anything of the sort.
Also I'm not sure if the S3 version even works anymore, since I think
AWS requires the v4 authentication scheme now and not the older v2 version
used in simples3. It would probably make sense to use the new aws-sdk or
one of its subcrates.
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
With the dist-server feature enabled, additional clippy
warnings are emitted. Those are fixed by this commit.
Lints were:
- unneccessary borrow
- string != "" should be !string.is_empty()
- unwrap_or followed by a function call -> unwrap_or_else
Signed-off-by: Jonathan Schwender <schwenderjonathan@gmail.com>
dependabot[bot]
Sylvestre Ledru
Brian Silverman
Hongli Lai
aeiouaeiouaeiouaeiouaeiouaeiou
Byoungchan Lee
Igor Matuszewski
Jonathan Schwender
Mitchell Hentges
Michael Kruse
lissyx
Vlad Ivanov
Mitchell Hentges
Martyn Gigg
Michael Stahl
Emily Mabrey
Michael Woerister
Ben Boeckel
Adrien Guinet