For a long time two copies of the 'taskgraph' module have existed in parallel.
We've attempted to keep them in sync, but over time they have diverged and the
maintenance burden has increased.
In order to reduce this burden, we'd like to re-join the two code bases. The
canonical repo will be the one that lives outside of mozilla-central, and this
module will depend on it. Since they both have the same module name (taskgraph)
we need to rename the version in mozilla-central to avoid collisions.
Other consumers of 'taskgraph' (like mobile repos) have standardized on
'<project>_taskgraph' as their module names. So replicating that here as well.
Differential Revision: https://phabricator.services.mozilla.com/D127118
The wasi-sysroot toolchain contains both a sysroot for wasi and a
compiler-rt for clang. That makes it impractical to use as a
bootstrapped sysroot for wasm32-wasi builds of Spidermonkey.
We thus split the toolchain in two, one for the compiler-rt and one
for the sysroot. Ideally, the compiler-rt one would avoid building
clang/llvm the same way the sysroot one does, but that leads to
a case of chicken-and-egg, because the compiler-rt is needed to build
the clang toolchain. Eventually, the clang build would be split from
the addition of the compiler-rt, but we're not there yet.
Differential Revision: https://phabricator.services.mozilla.com/D122402
In cross-compilation setups (x86_64 host, i686 or aarch64 target), we're
going to need two sysroots. Obviously, we need the sysroot paths to be
different in that case, so the sysroot path themselves need to contain
some distinctive name, and we'll use the `target.toolchain` name for
that (the target triplet with the vendor/machine stripped out).
Because the path name needs to be reflected in the artifact name as well
as the toolchain name, we also change them.
And because the current prefix in the toolchain name is now redundant
with the suffix, we remove the prefix, and allow the bootstrapping
mechanism to try toolchains without the prefix.
Differential Revision: https://phabricator.services.mozilla.com/D119846
Bug 1715132 ended up switching the Valgrind jobs to run with Software
WebRender. Let's explicitly mark the task as running with WebRender and
set the prefs to make this clear.
Differential Revision: https://phabricator.services.mozilla.com/D117188
This removes the last uses of the 'push-interval-10' and 'push-interval-20' strategies.
They are being removed because they are dangerous in that its easy to accidentally not run
tasks when they should.
Instead, task authors should decide whether they want their tasks to run on
"backstop" pushes (run everything) or "expanded" pushes (run more than usual,
but still not as much as a backstop). Note that using "expanded" means the task
will *also* run on backstop pushes. It'll just additionally run on "expanded"
pushes.
In practice 'backstop' pushes will be every 20th push and 'expanded' pushes
will be every 10th push. Though this may vary due to the time component in
backstops.
Differential Revision: https://phabricator.services.mozilla.com/D89503
CLOSED TREE
Backed out changeset 93b955e5c048 (bug 1637544)
Backed out changeset be0717d76643 (bug 1637544)
Backed out changeset 447fea64b68d (bug 1637544)
And add suppressions for the new errors that this adds to the valgrind
run. It's not clear if it's a legitimate thing that LLVM does that
valgrind doesn't cope with, like many others, but the fact is running
valgrind on a PGO/LTO build doesn't show these errors, so we're not
actually shipping them anyways (but does show _different_ errors that we
don't see on the build the valgrind jobs do)
Differential Revision: https://phabricator.services.mozilla.com/D81016
The Backstop optimization doesn't take any arguments, yet the schema for 'push-optimization-*'
requires a 'schedules-component' that goes ignored. Fix this.
Differential Revision: https://phabricator.services.mozilla.com/D77058
The Backstop optimization doesn't take any arguments, yet the schema for 'push-optimization-*'
requires a 'schedules-component' that goes ignored. Fix this.
Differential Revision: https://phabricator.services.mozilla.com/D77058
The Backstop optimization doesn't take any arguments, yet the schema for 'push-optimization-*'
requires a 'schedules-component' that goes ignored. Fix this.
Differential Revision: https://phabricator.services.mozilla.com/D77058
This patch uses the new push-interval-10 to schedule the linux, windows plain and aarch64
builds on autoland every 10th push.
Tested locally with a local checkout whose pushlog_id was not divisible
by 10 using parameters.yml downloaded from the Gecko Decision Task using
./mach taskgraph optimized --verbose --parameters /tmp/parameters.yml
parameters.yml from autoland showed the following optimizations.
0:56.13 PushIntervalStrategy: Removing task build-linux64-aarch64/opt interval 10
0:56.13 PushIntervalStrategy: Removing task build-linux64-plain/debug interval 10
0:56.13 PushIntervalStrategy: Removing task build-signing-win64-aarch64/opt interval 10
0:56.13 PushIntervalStrategy: Removing task build-win64-aarch64/debug interval 10
0:56.13 PushIntervalStrategy: Removing task build-win64-plain/debug interval 10
0:56.18 PushIntervalStrategy: Removing task valgrind-linux64-valgrind/opt interval 10
while parameters.yml from mozilla-central did not show any PushIntervalStrategy
optimizations.
Differential Revision: https://phabricator.services.mozilla.com/D70182
GCE instances use Ubuntu 18.04, which is known to cause issues in
valgrind builds (Bug 1545094), while EC2 uses Ubuntu 14.04.
Move valgrind builds back to GCE to avoid perma fail of these jobs.
Differential Revision: https://phabricator.services.mozilla.com/D62552
--HG--
extra : moz-landing-system : lando
We're going to build `lucetc` for other platforms; let's prefix this
task with the platform name like we do for other toolchains.
Differential Revision: https://phabricator.services.mozilla.com/D59498
--HG--
extra : moz-landing-system : lando
Some of these were busted in the initial push for wasm sandboxing; some
of these didn't break in that push, but seem likely to if/when we have
more complete coverage of turning on wasm sandboxing.
Differential Revision: https://phabricator.services.mozilla.com/D58200
--HG--
extra : moz-landing-system : lando
This part removes the use_toolchains transform, and adjusts all task
definitions accordingly.
Differential Revision: https://phabricator.services.mozilla.com/D41891
MANUAL PUSH: avoid closing autoland while docker images and toolchains
are rebuilt.
Many tasks (release tasks and cached tasks, in particular) should be re-run rather
than retriggered. Disable retrigger action for those tasks by default.
Differential Revision: https://phabricator.services.mozilla.com/D27206
--HG--
extra : moz-landing-system : lando
This change switches most CI builds to clang, with a few exceptions:
- valgrind builds, until bug 1481670 is figured out.
- PGO and nightly builds, until that's fully tested.
- coverage builds, per bug 1471339 comment 17.
- base toolchain builds, to keep some builds on GCC even when we're
fully switched to clang.
- any build that doesn't use build/unix/mozconfig.linux (e.g. probably
all those driven by autospider.py, maybe others).
This change switches most CI builds to clang, with a few exceptions:
- valgrind builds, until bug 1481670 is figured out.
- PGO and nightly builds, until that's fully tested.
- coverage builds, per bug 1471339 comment 17.
- base toolchain builds, to keep some builds on GCC even when we're
fully switched to clang.
- any build that doesn't use build/unix/mozconfig.linux (e.g. probably
all those driven by autospider.py, maybe others).
While some builds have a PERFHERDER_EXTRA_OPTIONS environment set on the
taskcluster side, many others have the equivalent set at the mozharness
level. But only the former are actually linted against, which,
unsurprisingly, translates to conflicting values between some of the
mozharness configs.
So we move those configurations to taskcluster, enable the lint on all
the kinds that look like builds (based on them using the build_attrs
transform), and adjust the values to stop conflicting. Notably, for
searchfox and static-analysis-autotest.
--HG--
extra : rebase_source : 097333608e61e1df66e5d8f914e15784f35e58f2
There are now only a handful of buildbot jobs remaining and the concern over
outdated treeherder exclusion profiles has largely been resolved.
This does remove the tc() group from a substantial number of tasks which will
now show up as top level tasks, potentially adding clutter. In some cases, we
might want to re-add a new group (e.g group builds or compiled tests together).
However rather than try to predict the best group names for tasks I'm unfamiliar
with, I think it's best to land this as is. Then if things are looking too
cluttered at the root namespace, file follow-up bugs as needed.
MozReview-Commit-ID: 8SMwjDwAOzV
--HG--
extra : rebase_source : 2f6d89d11c139bdcd404e7537db799d0e36ee4c3
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : d8447b5422e63e88444008fddb76d658829694de
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : 0aa9ef8151c2fccf62507dfecc0bc57b157772e1
Andrew Halberstadt
Mike Hommey
Alexandru Michis
Andrew Osmond
Andrew Halberstadt
Cosmin Sabou
Tom Prince
Andrew Halberstadt
Bogdan Tara
Sebastian Hengst
Bob Clary
Wander Lairson Costa
Nathan Froyd
Thomas Daede
Ciure Andrei
Emilio Cobos Álvarez
Daniel Varga
Dan Mosedale
Coroiu Cristina
Steve Armand