Artifact mozconfigs are not necessarily up-to-date wrt changes to the
nightly mozconfigs, and all in all, shouldn't be much different from
them.
It's just better to use the nightly mozconfigs (or beta on beta, etc.)
and make the mozconfigs themselves handle the few things that need to be
different when the USE_ARTIFACT environment is set (which is now
consistently set by taskcluster)
This does have the side effect of turning builds that actually don't
support artifact builds red when using --artifact on try, instead of
having them silently not be artifact builds as currently happens.
Depends on D21314
Differential Revision: https://phabricator.services.mozilla.com/D21315
--HG--
extra : moz-landing-system : lando
The artifact builds that are automatically derived using the artifact
template set the USE_ARTIFACT environment variable from taskcluster.
After the previous change, --artifact builds from try syntax do that
too.
That leaves us with only the artifact-build build not doing it, so for
consistency, do it there. That makes it not necessary to set
USE_ARTIFACT from mozconfig.artifact.automation anymore.
Depends on D22056
Differential Revision: https://phabricator.services.mozilla.com/D21313
--HG--
extra : moz-landing-system : lando
Currently, artifact builds pull from some task they determine from the
job type, and finding a pushhead.
Sometimes that latter fails, most notably on automation. But automation
can already know the right task to use without guesswork.
This change allows artifact builds to pull from a specific taskid given
through an environment variable, and make tasks from the artifact-build
kind (not the --artifact builds from try) use that.
Remove the workaround from bug 1382982 because it's now dead code.
Differential Revision: https://phabricator.services.mozilla.com/D19881
The max-run-time for tasks appears to have been mostly cargo
culted. In particular, a value of 36000 (10 hours) is quite absurd:
no single task takes that long to execute.
This commit reduces the max-run-time of several tasks to
more reasonable values. The goal here is to prevent
excessively long-running tasks. There is definitely room to
further tweak values to further mitigate long-running tasks. But
let's bite off the biggest chunk first, since that doesn't
require much mental effort.
There is a possibility I overshot on some of these tasks. If we
get timeouts, we can always increase the timeout again.
Differential Revision: https://phabricator.services.mozilla.com/D1716
While some builds have a PERFHERDER_EXTRA_OPTIONS environment set on the
taskcluster side, many others have the equivalent set at the mozharness
level. But only the former are actually linted against, which,
unsurprisingly, translates to conflicting values between some of the
mozharness configs.
So we move those configurations to taskcluster, enable the lint on all
the kinds that look like builds (based on them using the build_attrs
transform), and adjust the values to stop conflicting. Notably, for
searchfox and static-analysis-autotest.
--HG--
extra : rebase_source : 097333608e61e1df66e5d8f914e15784f35e58f2
The mozharness transform is supposed to set the docker image to
desktop-build when not already set, but was not doing it properly.
I guess this is why some jobs were setting the image themselves, despite
using the mozharness transform.
Consequently, don't manually set the image to desktop-build when it's
the default.
--HG--
extra : rebase_source : 024bd10960bedaee3416785348a5c12498c5286f
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : d8447b5422e63e88444008fddb76d658829694de
It turns out that in all cases it was the last tooltool manifest entry,
so we can remove the tooltool manifests entirely, and remove all
references to them.
--HG--
extra : rebase_source : 0aa9ef8151c2fccf62507dfecc0bc57b157772e1
They just inherited the dependencies because they were using the same
tooltool manifests as compiling builds.
--HG--
extra : rebase_source : 03ee8180e455e01b72016b3132001745b87f78e7
There are essentially four categories of jobs that have dependencies on
sccache currently and that shouldn't:
- jobs that don't compile anything. They just inherited the dependency
because they were using the same tooltool manifests as compiling
builds.
- jobs that don't use sccache. Ideally, we'd make them use sccache, but
things are not currently setup to make that easy, so we'll keep that for
later.
- jobs that explicitly disable sccache through needs-sccache: false.
Like above, ideally, they would use sccache.
- jobs that can't use sccache. Those are hazard jobs, that rely on a GCC
plugin and AIUI on a global knowledge of the code, which the plugin needs
to see. Caching would break that.
--HG--
extra : rebase_source : 77455b9f0a58919838c8c64c36aa1db99baf8c7e
This leaves out fuzzing and linux static analysis builds, which are
using, respectively, clang 4.0.1 and clang 3.8, while linux64-clang
produces a 3.9 and win*-clang a 5.0
--HG--
extra : rebase_source : 45128ac74bf4fe7e6a2ace57043c34ecdf0fe929
Such a definition automatically sets up the corresponding dependencies
in the taskgraph, and adds the necessary artifact definitions for use in
the corresponding jobs. The jobs end up with a MOZ_TOOLCHAINS
environment variable with a list of path@task-id strings, where task-id
is corresponding to the (possibly optimized) toolchain job, and path
corresponding to the toolchain-artifact defined for that toolchain job.
--HG--
extra : rebase_source : b2d297bd75d9c416b30d2a6c6d61efcb64681727
Since bug 1321847, mozharness tooltool manifests can be overriden from
the environment. We use that possibility to now define tooltool
manifests from taskcluster job definitions. Ideally, we'd also remove
the definitions from the mozharness configs, but with things still
running on buildbot, it's not clear what things might break because of
that. We'll do it in a separately back-out-able followup.
--HG--
extra : rebase_source : 860b8f1d4fdc4a557770a3749055f19b1ec45e93
Previously, mozharness defined a separate action to collect build
metrics. This required the script and/or config to define that
action.
Metrics collection for CI is important. So it should be enabled by
default.
This commit changes the "build" action/method to always call the
metrics collection function after successful build. References to
the "generate-build-stats" action have been removed because it is
redundant.
A side-effect of this change is we may generate build metrics where
we weren't before. This could lead to e.g. duplicate entries in some
Perfherder series. Let's see what breaks ;)
MozReview-Commit-ID: 42UQI5YQTMC
--HG--
extra : rebase_source : c57dc9ec6ac46003384edff098a0ad81c75539b7
extra : source : c9812dd7d27a174c0ee46d44ec595fbe29c9e1db
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
--HG--
extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
Tom Prince
Mike Hommey
Dan Mosedale
Gregory Szorc
Coroiu Cristina
Steve Armand
Dustin J. Mitchell
Ryan VanderMeulen