Now that all GCC and related source tarballs extract to paths
independent of their version number, the scripts are all very
look-alike, so they can be consolidated.
Differential Revision: https://phabricator.services.mozilla.com/D40749
Bug 1479533 was proposing to add a similar functionality, but this
iteration avoids actually unpacking anything, and ensures
reproducibility by relying on the reproducible bits from the original
archives: file ordering, flags, etc. (since they are checksummed, those
are never going to change for a given archive).
Another notable difference is that this applies the repack on the fetch
task itself, rather than create a separate task to apply the repack. The
latter has advantages, in that it allows to change the repacking without
redownloading the original file from a third-party server, but in
practice, most changes to the repacking would trigger the download tasks
anyways.
This patch only takes care of changing the archive type (zip->tar), and
the compression type (anything->zstandard).
Differential Revision: https://phabricator.services.mozilla.com/D40740
What this means is that the sources for clang/llvm are downloaded
separately from the toolchain build (which also means we finally only
download a given version of clang once for all platforms).
In turn, this means the build-clang.py script needs to start with an
existing llvm-project tree, and we choose to make build-clang.py expect
that it's run from the llvm-project root directory.
This also means we don't need to download git for the windows toolchain
task.
Differential Revision: https://phabricator.services.mozilla.com/D40402
They are now all very similar, and there are only a few variations.
Differential Revision: https://phabricator.services.mozilla.com/D40365
MANUAL PUSH: avoid closing autoland while toolchains are rebuilt.
This removes the need for the hacks in wr-macos-cross-build-setup.sh and
wrench-windows-tests.sh, while keeping things working in other scripts.
Differential Revision: https://phabricator.services.mozilla.com/D40184
Plenty of places use `nproc`, and only a couple use `getconf
_NPROCESSORS_ONLN`. Use the former instead of the latter.
Differential Revision: https://phabricator.services.mozilla.com/D39999
--HG--
extra : moz-landing-system : lando
This is loosely based on what was in bug 1467359, but simplified to
handle git only, and simply using git-archive because, at least now,
it's deterministic (it uses the commit date as timestamp in tar
archives).
This also adds 4 tasks for some of the things we use for toolchains, but
doesn't hook them up yet.
This also upgrades the fetch docker image to Debian buster, and installs
the required packages in it.
Differential Revision: https://phabricator.services.mozilla.com/D39480
For some tasks, the workdir known to the decision task doesn't actually match
the workdir used in the task, which makes MOZ_FETCHES_DIR wrong when the
decision task derives it from the workdir.
On other tasks, MOZ_FETCHES_DIR is set to a relative directory, which
may work in some places where MOZ_FETCHES_DIR is used, but not in
others, that happen to be executed from a different directory.
To solve both problems, we set MOZ_FETCHES_DIR as a relative directory
everywhere, and we make run-task normalize it to an absolute path
before executing the task.
Differential Revision: https://phabricator.services.mozilla.com/D39152
--HG--
extra : moz-landing-system : lando
For some tasks, the workdir known to the decision task doesn't actually match
the workdir used in the task, which makes MOZ_FETCHES_DIR wrong when the
decision task derives it from the workdir.
On other tasks, MOZ_FETCHES_DIR is set to a relative directory, which
may work in some places where MOZ_FETCHES_DIR is used, but not in
others, that happen to be executed from a different directory.
To solve both problems, we set MOZ_FETCHES_DIR as a relative directory
everywhere, and we make run-task normalize it to an absolute path
before executing the task.
Differential Revision: https://phabricator.services.mozilla.com/D39152
--HG--
extra : moz-landing-system : lando
In browsertime.zip we should have:
browsertime/
package.json
package-lock.json
node_modules/
.bin/
browsertime -> ../browsertime/bin/browsertime.js
browsertime/
...
The idea is that we'll fetch browsertime.zip in a generic-worker
environment and be able to run Node.js from within the top level
browsertime/ directory.
Differential Revision: https://phabricator.services.mozilla.com/D38773
--HG--
extra : moz-landing-system : lando
This gets the wrench android builds working again after
07168db23c565e4506690612a7be50738844ddb2.
Differential Revision: https://phabricator.services.mozilla.com/D38882
--HG--
extra : moz-landing-system : lando
We need a fix from `cctools-port` master for cross-language LTO builds
to work properly on the Mac. Rather than cherry-picking yet another
commit, which would have to deal with a updated `ld64` upstream, we've
opted to go ahead and update directly to upstream.
This choice brings about some significant build changes, as TAPI support
has moved to a different library that is not easily buildable directly.
Differential Revision: https://phabricator.services.mozilla.com/D36636
--HG--
extra : moz-landing-system : lando
We need a fix from `cctools-port` master for cross-language LTO builds
to work properly on the Mac. Rather than cherry-picking yet another
commit, which would have to deal with a updated `ld64` upstream, we've
opted to go ahead and update directly to upstream.
This choice brings about some significant build changes, as TAPI support
has moved to a different library that is not easily buildable directly.
Differential Revision: https://phabricator.services.mozilla.com/D36636
--HG--
extra : moz-landing-system : lando
Bug 1451104 removed the GCC 4.9 toolchain, but left the build script and
the patch that only that toolchain build was using.
Differential Revision: https://phabricator.services.mozilla.com/D36886
--HG--
extra : moz-landing-system : lando
When adding Windows compatibility in bug 1557785, we stopped exported
UPLOAD_PATH by default. UPLOAD_PATH is used by profileserver.py to
determine where to store the logs; when it is unset, the logs go to
stdout and the artifacts aren't created. We should always export
UPLOAD_PATH to get the same logging behavior on all platforms.
Differential Revision: https://phabricator.services.mozilla.com/D36521
--HG--
extra : moz-landing-system : lando
This version includes several fixes for sccache-dist in addition
to changes that make sccache compatible with cargo pipelining.
Differential Revision: https://phabricator.services.mozilla.com/D36075
--HG--
extra : moz-landing-system : lando
The run-profileserver.sh script is a bridge between the Taskcluster task
and profileserver.py. It was originally written as a Linux-only script,
but with a few modifications it can support Windows as well. The xvfb
support needs to be optional, and the UPLOAD_PATH and PGO_RUNDIR
variables must not assume a Linux filesystem.
Differential Revision: https://phabricator.services.mozilla.com/D34796
--HG--
extra : moz-landing-system : lando
The current setup for bindgen relies on either finding clang/libclang
from the output of llvm-config, or from the paths given via the
configure flags --with-clang-path/--with-libclang-path.
One _very_ common problem is that the llvm-config we end up using does
not correspond to the clang used for compilation, which has some
undesirable side effect, like failing to build.
So instead of relying on llvm-config, we do the following:
- when the compiler is clang, we just use that
- when the compiler is clang-cl, we use clang from the same directory
- otherwise, we either try to find clang in PATH, or rely on
--with-clang-path.
Once clang is found, we try to deduce the location of the corresponding
libclang via the output of `clang -print-search-dirs`, or rely on
--with-libclang-path.
Differential Revision: https://phabricator.services.mozilla.com/D33241
--HG--
extra : moz-landing-system : lando
The last use of scm level in mozharness is in `mozharness.mozilla.secrets` which
uses the `MOZ_SCM_LEVEL` environment variable directy.
Differential Revision: https://phabricator.services.mozilla.com/D20897
--HG--
extra : moz-landing-system : lando
This change is necessary to make either e10s profiling or LLVM IR-based
PGO instrumentation work properly, as both will generate multiple
`.profraw` files.
Differential Revision: https://phabricator.services.mozilla.com/D32390
--HG--
extra : moz-landing-system : lando
Changes:
- removed UBUNTU_1604 detection mechanism at top of `test-linux.sh` file, since all tests are run on Ubuntu 16.04 anyway
- added new environment value `NEED_COMPIZ`, defaulting to `true`, which will inform the test if compiz is required for tests
- from `test-linux.sh` remove unconditional invocation of compiz, and replace it with detection of `NEED_COMPIZ` environment variable
Differential Revision: https://phabricator.services.mozilla.com/D31724
--HG--
extra : moz-landing-system : lando
Our current OS X builds use `--target=x86_64-darwin11` (which
corresponds to OS X 10.7). This target is problematic for two reasons:
* We're actually targeting for OS X 10.9 (`MACOSX_DEPLOYMENT_TARGET`);
* It's slightly different from the default Rust target.
Let's address these problems in reverse order: differences from the Rust
target are bad, because the `--target` we provide to `clang` and the
Rust target find their way into LLVM bitcode files and the linker will
refuse to link together bitcode files that have incompatible targets.
Why are the two incompatible? The current `--target` doesn't have a
"vendor" in triple-speak, whereas the Rust one has "apple" as the
vendor (`x86_64-apple-darwin`) We therefore need to change the
`--target` we pass to `clang` to have a vendor of "apple".
This need is behind the {init,toolchain}.configure changes,
but it has ramifications elsewhere, because `clang` looks for
`--target`-prefixed build tools. So we have to change the `--target`
for cctools to get the right tool prefixes and we have to change the
`--target` for building clang ourselves so that *those* builds can find
the newly renamed cctools.
Once we've done, that's really enough; we don't *need to address the
first problem: While the `--target` might be `x86_64-apple-darwin11`,
both `clang` and `rustc` will dynamically choose the target triple that
eventually lands in LLVM bitcode files based on
`MACOSX_DEPLOYMENT_TARGET`, which we set in all builds. But the current
target is slightly misleading, and the cctools don't need to be prefixed
with a particular Darwin version, since they work for all Darwin
targets. Let's just drop the "11" from the `--target` and eliminate a
little bit of confusion.
Differential Revision: https://phabricator.services.mozilla.com/D31128
--HG--
extra : moz-landing-system : lando
Our current OS X builds use `--target=x86_64-darwin11` (which
corresponds to OS X 10.7). This target is problematic for two reasons:
* We're actually targeting for OS X 10.9 (`MACOSX_DEPLOYMENT_TARGET`);
* It's slightly different from the default Rust target.
Let's address these problems in reverse order: differences from the Rust
target are bad, because the `--target` we provide to `clang` and the
Rust target find their way into LLVM bitcode files and the linker will
refuse to link together bitcode files that have incompatible targets.
Why are the two incompatible? The current `--target` doesn't have a
"vendor" in triple-speak, whereas the Rust one has "apple" as the
vendor (`x86_64-apple-darwin`) We therefore need to change the
`--target` we pass to `clang` to have a vendor of "apple".
This need is behind the {init,toolchain}.configure changes,
but it has ramifications elsewhere, because `clang` looks for
`--target`-prefixed build tools. So we have to change the `--target`
for cctools to get the right tool prefixes and we have to change the
`--target` for building clang ourselves so that *those* builds can find
the newly renamed cctools.
Once we've done, that's really enough; we don't *need to address the
first problem: While the `--target` might be `x86_64-apple-darwin11`,
both `clang` and `rustc` will dynamically choose the target triple that
eventually lands in LLVM bitcode files based on
`MACOSX_DEPLOYMENT_TARGET`, which we set in all builds. But the current
target is slightly misleading, and the cctools don't need to be prefixed
with a particular Darwin version, since they work for all Darwin
targets. Let's just drop the "11" from the `--target` and eliminate a
little bit of confusion.
Differential Revision: https://phabricator.services.mozilla.com/D31128
--HG--
extra : moz-landing-system : lando
Our current OS X builds use `--target=x86_64-darwin11` (which
corresponds to OS X 10.7). This target is problematic for two reasons:
* We're actually targeting for OS X 10.9 (`MACOSX_DEPLOYMENT_TARGET`);
* It's slightly different from the default Rust target.
Let's address these problems in reverse order: differences from the Rust
target are bad, because the `--target` we provide to `clang` and the
Rust target find their way into LLVM bitcode files and the linker will
refuse to link together bitcode files that have incompatible targets.
Why are the two incompatible? The current `--target` doesn't have a
"vendor" in triple-speak, whereas the Rust one has "apple" as the
vendor (`x86_64-apple-darwin`) We therefore need to change the
`--target` we pass to `clang` to have a vendor of "apple".
This need is behind the {init,toolchain}.configure changes,
but it has ramifications elsewhere, because `clang` looks for
`--target`-prefixed build tools. So we have to change the `--target`
for cctools to get the right tool prefixes and we have to change the
`--target` for building clang ourselves so that *those* builds can find
the newly renamed cctools.
Once we've done, that's really enough; we don't *need to address the
first problem: While the `--target` might be `x86_64-apple-darwin11`,
both `clang` and `rustc` will dynamically choose the target triple that
eventually lands in LLVM bitcode files based on
`MACOSX_DEPLOYMENT_TARGET`, which we set in all builds. But the current
target is slightly misleading, and the cctools don't need to be prefixed
with a particular Darwin version, since they work for all Darwin
targets. Let's just drop the "11" from the `--target` and eliminate a
little bit of confusion.
Differential Revision: https://phabricator.services.mozilla.com/D31128
--HG--
extra : moz-landing-system : lando
This will match the compiler version Tor would like. We backport several
llvm-objcopy patches that landed right after the 8 branch though. We
also grab some upstream changes from mingw-clang in the build script
Differential Revision: https://phabricator.services.mozilla.com/D31347
--HG--
extra : moz-landing-system : lando
We need this to auto-generate the copy-constructor for TransformOperation,
without which the patch wouldn't build.
Differential Revision: https://phabricator.services.mozilla.com/D30799
--HG--
extra : moz-landing-system : lando
This makes the rust toolchain artifacts contain the rust stdlib as well,
for use by searchfox. It does bring up the size of the toolchain
artifact slightly - rustc.tar.xz file for the Linux/rust 1.34 job for
example goes from 270483672 bytes to 273803148 bytes (1.23% larger) and
the equivalent android tarball goes from 230503888 to 235698736 bytes
(2.25% larger).
Differential Revision: https://phabricator.services.mozilla.com/D28282
--HG--
extra : moz-landing-system : lando
Mike Hommey
Tom Prince
Razvan Maries
Sylvestre Ledru
Bogdan Tara
Andreea Pavel
shindli
Oana Pop Rus
Eric Rahm
Nick Alexander
Agi Sferro
Jeff Muizelaar
Nicolas Silva
Chris Manchester
Nathan Froyd
Csoregi Natalia
Tom Ritter
Mike Shal
Emilio Cobos Álvarez
Rok Garbas
Andi-Bogdan Postelnicu
Edwin Gao
Noemi Erli
Coroiu Cristina
Ciure Andrei
Joel Maher
Kartikaya Gupta
Rob Thijssen
Mihai Alexandru Michis