While some builds have a PERFHERDER_EXTRA_OPTIONS environment set on the
taskcluster side, many others have the equivalent set at the mozharness
level. But only the former are actually linted against, which,
unsurprisingly, translates to conflicting values between some of the
mozharness configs.
So we move those configurations to taskcluster, enable the lint on all
the kinds that look like builds (based on them using the build_attrs
transform), and adjust the values to stop conflicting. Notably, for
searchfox and static-analysis-autotest.
--HG--
extra : rebase_source : 097333608e61e1df66e5d8f914e15784f35e58f2
Mainly so searching "toolchain" + "gcc" yields something useful in the
taskgraph.
MozReview-Commit-ID: HWiT3AwwYQ2
--HG--
extra : rebase_source : b1ba0dfb4f99b6f8abe42506e8b37db68ed03590
Mainly so searching "toolchain" + "gcc" yields something useful in the
taskgraph.
MozReview-Commit-ID: HWiT3AwwYQ2
--HG--
extra : rebase_source : b1ba0dfb4f99b6f8abe42506e8b37db68ed03590
Switch almost all builds currently using the desktop-build image to use
the right debian7-*-build image instead. The only exception is the rust
bindgen spidermonkey builds, that require cmake being installed, but I
don't want to add it to the base images because that involves risking a
cmake dependency unwantedly slipping in Firefox (rust-bindgen ironically
requires cmake to build a single C++ file...)
There are now only a handful of buildbot jobs remaining and the concern over
outdated treeherder exclusion profiles has largely been resolved.
This does remove the tc() group from a substantial number of tasks which will
now show up as top level tasks, potentially adding clutter. In some cases, we
might want to re-add a new group (e.g group builds or compiled tests together).
However rather than try to predict the best group names for tasks I'm unfamiliar
with, I think it's best to land this as is. Then if things are looking too
cluttered at the root namespace, file follow-up bugs as needed.
MozReview-Commit-ID: 8SMwjDwAOzV
--HG--
extra : rebase_source : 2f6d89d11c139bdcd404e7537db799d0e36ee4c3
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : d8447b5422e63e88444008fddb76d658829694de
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : 0aa9ef8151c2fccf62507dfecc0bc57b157772e1
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
The GCC plugin used for hazard builds was built for GCC 4.9. Until a new
plugin is built by a toolchain job for the "default" version of GCC we
build Firefox with, the builds using the plugin need to stay with the
version the plugin was built for.
There are essentially four categories of jobs that have dependencies on
sccache currently and that shouldn't:
- jobs that don't compile anything. They just inherited the dependency
because they were using the same tooltool manifests as compiling
builds.
- jobs that don't use sccache. Ideally, we'd make them use sccache, but
things are not currently setup to make that easy, so we'll keep that for
later.
- jobs that explicitly disable sccache through needs-sccache: false.
Like above, ideally, they would use sccache.
- jobs that can't use sccache. Those are hazard jobs, that rely on a GCC
plugin and AIUI on a global knowledge of the code, which the plugin needs
to see. Caching would break that.
--HG--
extra : rebase_source : 77455b9f0a58919838c8c64c36aa1db99baf8c7e
This leaves out fuzzing and linux static analysis builds, which are
using, respectively, clang 4.0.1 and clang 3.8, while linux64-clang
produces a 3.9 and win*-clang a 5.0
--HG--
extra : rebase_source : 45128ac74bf4fe7e6a2ace57043c34ecdf0fe929
Such a definition automatically sets up the corresponding dependencies
in the taskgraph, and adds the necessary artifact definitions for use in
the corresponding jobs. The jobs end up with a MOZ_TOOLCHAINS
environment variable with a list of path@task-id strings, where task-id
is corresponding to the (possibly optimized) toolchain job, and path
corresponding to the toolchain-artifact defined for that toolchain job.
--HG--
extra : rebase_source : b2d297bd75d9c416b30d2a6c6d61efcb64681727
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: zehcGJrUQX
--HG--
extra : source : feedcde68c2a54da210f03eb287ab5c862fc982b
extra : intermediate-source : 485d1af7805ad9fa0e701c3571fc1291fbfc6850
Hazard jobs use a specific tooltool-manifest field in their definition.
Since there is no post-processing happening on those definitions, and
since generalizing it would require adding the field to a bunch of
validation schemas, and the same code to various transforms, it's just
simpler to move to use environment variable definitions here too.
Eventually, tooltool manifests won't be necessary anyways, and those
environment variables will go away.
--HG--
extra : rebase_source : bd619024bde051424a2d811a1996caaf0f853247
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
--HG--
extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
As part of this, we eliminate tc-vcs since it is redundant with run-task.
MozReview-Commit-ID: 8HAFSM73BOU
--HG--
extra : rebase_source : 92947b92ce23a7ca4aa885d4dfd073d72410ae16
This definitely isn't all of it. But I got the obvious chunks. Still
need to clean up mozharness.
MozReview-Commit-ID: JTZBydP3i2r
--HG--
extra : rebase_source : 782401359036751896ef6c153a81a06cb14031bb
Secrets are accessed via http://taskcluster/secrets, and the hazard scripts
show no evidence of such calls.
MozReview-Commit-ID: 6xp6D2ybwap
--HG--
extra : rebase_source : 5bfe02773f26768c7e79621a43515d6bf8b921ca
Emilio Cobos Álvarez
Dan Mosedale
Mike Hommey
Gregory Szorc
Dorel Luca
Steve Fink
Andrew Halberstadt
Coroiu Cristina
Steve Armand
Dustin J. Mitchell
Wander Lairson Costa
Wes Kocher
Sebastian Hengst
Mike Shal