Hazard jobs use a specific tooltool-manifest field in their definition.
Since there is no post-processing happening on those definitions, and
since generalizing it would require adding the field to a bunch of
validation schemas, and the same code to various transforms, it's just
simpler to move to use environment variable definitions here too.
Eventually, tooltool manifests won't be necessary anyways, and those
environment variables will go away.
--HG--
extra : rebase_source : bd619024bde051424a2d811a1996caaf0f853247
It is really easy to copy and paste taskgraph or mozharness configs
and cause Perfherder data to be written to the same bucket, resulting
in non-useful metrics collection.
This commit adds a taskgraph transform for the "build" kind that
attempts to look for multiple build jobs writing to the same
Perfherder bucket.
It isn't perfect. But it has already flushed out some jobs writing
to the same bucket and therefore producing bimodal Perfherder data.
MozReview-Commit-ID: COyvXwMiM32
--HG--
extra : rebase_source : 05dd227ceddcd4822e5689217873ec76912e7d30
This copies the behavior of mozilla-taskcluster when it
submits try pushes. This will allow us to eventually stop
using mozilla-taskcluster.
MozReview-Commit-ID: J9zC92AE7HZ
--HG--
extra : rebase_source : 36575ef1fa5c43b6b6d5c0243c323af69e4ebd23
A number of developers find it convenient to build with
--disable-optimize --enable-debug for an improved debugging experience.
We don't currently have a configuration in CI that ensures this
combination of options works, so various changes break builds with this
configuration every so often. We should test such configurations to
ensure they build to provide a smooth experience for developers.
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
--HG--
extra : rebase_source : 100092c2f2ff609362a724fff60f46dd6e49c94e
extra : intermediate-source : d10f5ccd882b965fcad39914f7c3c930d1301a41
extra : source : a0e257e346ccf3c1db332ec5903241f4eeb9a7ee
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
--HG--
extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
While this was allowed by the schema, it would never be resolved, and was
unused.
MozReview-Commit-ID: LIxJmr9ZSdK
--HG--
extra : rebase_source : af4c1e3311035880a4d7e4515fa119cb0628dd7d
The logic here is a bit tricky to grok, but essentially there are two kinds of "job" tasks, those that
should always be considered (and possibly be optimized away later due to "skip-unless-changed"), and
those that should only be considered if their associated build-platform is also going to be scheduled.
If -j is specified, that should supercede both cases.
This patch uses the prescence of the 'build_platform' attribute to draw the distinction.
MozReview-Commit-ID: H9SjeYuZ8F0
--HG--
extra : rebase_source : 4c33061e882533df74159f3299278619a06d5945
As of bug 1342503 being fixed, all of our desktop firefox builds have
webrender compiled in by default. Webrender can therefore be enabled at
runtime either by a pref or environment variable on any desktop firefox
build. The old builds that we originally used to stand up webrender are
no longer needed, as the *only* difference between them and the regular
builds are that they build with the pref turned on instead of turned
off. This doesn't warrant keeping around these extra builds, and this
patch removes them along with all the associated goop that was needed to
configure them.
MozReview-Commit-ID: 5wlOWo11fEk
--HG--
extra : rebase_source : 696afdd2d9fb5f7932d0737a7d71c3aa6af0bd64
Add configurations for building and uploading AArch64 Nightly builds, in
tier 1 and without artifact support for now.
As for not denoting AArch64 builds as "api-21", I don't really think we
will split AArch64 the way we split ARMv7 before. Originally, we split
into API 9 and API 11+ because of lots of "constrained" devices that
were stuck with API 9. We made an API 9 APK in order to lower our
footprint on those devices. That probably will not be a problem for
AArch64, because devices with API 21+ and AArch64 support are usually
more than capable for running Fennec. Secondly, it was a big change for
Android going from API 9 to API 11+, so we saved quite a bit of
code/resources when we stripped out API 11+. I don't see such drastic
changes going from API 21 to upcoming versions, so even if we did split,
I don't think it'll get us much benefit.
MozReview-Commit-ID: 7N7Slv1pPgb
Several years ago there was a single zip file for all test files. Clients
would only extract the files they needed. Thus, zip was a reasonable
archive format because it allowed direct access to members without
having to decompress the entirety of the stream.
We have since split up that monolithic archive into separate,
domain-specific archives. e.g. 1 archive for mochitests and one
for xpcshell tests. This drastically cut down on network I/O
required on testers because they only fetched archives/data that
was relevant. It also enabled parallel generation of test archives,
we shaved dozens of seconds off builds due to compression being
a long pole.
Despite the architectural changes to test archive management, we
still used zip files. This is not ideal because we no longer access
specific files in test archives and thus don't care about single/partial
member access performance.
This commit implements support for generating tar.gz test archives.
And it switches the web-platform archive to a tar.gz file.
The performance implications for archive generation are significant:
before: 48,321,250 bytes; 6.05s
after: 31,844,267 bytes; 4.57s
The size is reduced because we have a single compression context
so data from 1 file can benefit compression in a subsequent file.
CPU usage is reduced because the compressor has to work less with
1 context than it does with N. While I didn't measure it, decompression
performance should also be improved for the same reasons. And of course
network I/O will be reduced.
mozharness consumers use a generic method for handling unarchiving.
This method automagically handles multiple file extensions. So as long
as downstream consumers aren't hard coding ".zip" this change should
"just work."
MozReview-Commit-ID: LQa5MIHLsms
--HG--
extra : rebase_source : cd029cdbbcccc1d16f03d63a5f1fdf60be5db4fd
extra : source : a0e257e346ccf3c1db332ec5903241f4eeb9a7ee
The current indexed task has a big rank number, we blocks a new indexed
task update.
Add an index rank to uploader task big enough to replace the old indexed
task.
MozReview-Commit-ID: Hg5Ya4D0qYt
--HG--
extra : rebase_source : c08386e5fe62ba4ee4e2832539b6aebf015d6799
This patch will allow jittest, jsreftest, and any other test suite to run on linux64-ccov regardless of whether or not the files in 'files-changed' have been matched with changes in a patch.
MozReview-Commit-ID: 78CWUTWBhV
--HG--
extra : rebase_source : 4e8bc8a43a6be2b15e2b51c986da9e5ba5adc039
Mike Hommey
Gregory Szorc
Kim Moir
Brian Stack
Sebastian Hengst
Wes Kocher
Nathan Froyd
Wander Lairson Costa
Ryan VanderMeulen
Ben Hearsum
Alin Selagea
Dustin J. Mitchell
Andrew Halberstadt
Kartikaya Gupta
rforbes
Johan Lorenzo
Phil Ringnalda
Aki Sasaki
Joel Maher
Jim Chen
Justin Wood
Greg Mierzwinski