The optimization doesn't actually do what it claims. For the optimization to be
considered, a task needs to be in the target graph, but then all the
dependencies will be too, which inhibits this optimization.
Differential Revision: https://phabricator.services.mozilla.com/D15151
--HG--
extra : moz-landing-system : lando
Address Sanitizer builds use --disable-crashreporter so they don't have
symbols zip files to upload. Don't run symbol upload tasks for these builds.
MozReview-Commit-ID: GeQgRZF3m8t
--HG--
extra : rebase_source : 6ac59c5c96b5fb5ddbbe8c60af3a203d02ea9883
Previously we would only generate upload-symbols tasks for nightly builds,
since we only want to upload symbols for builds we ship to users.
On try, developers may want to use the symbol server but very rarely want
to do nightly builds, so allow uploading symbols from any build type there.
MozReview-Commit-ID: IYs9mZii3DN
--HG--
extra : rebase_source : 15acb889510bb07ed3d29ea802997f11796dad9f
filter_upload_symbols is a relic of task configurations that were written
before we had a better handle on taskgraph generation. We should only be
uploading symbols for nightly builds anyway, so this is better served using
newer filtering methods.
upload-symbols tasks were specified to run on non-nightly build types in the
kind.yml, but those were filtered out in filter_upload_symbols. I believe
these were simply an artifact of the initial upload-symbols implementation
happening before nightly builds were running in Taskcluster.
MozReview-Commit-ID: Je1NytrVPT8
--HG--
extra : rebase_source : a961c17d329af848fa7bb64c5186135d37dd412f
If tasks still exist before the optimization phase, then any tasks these depend
on will also be scheduled. In particular for devedition on m-r, that means that
although the builds were being excluded from the target tasks due to the
build's run-on-projects settings, the upload-symbols and
upload-generated-sources tasks did exist in the target tasks, and so the builds
got re-added to the target tasks during graph optimization.
MozReview-Commit-ID: 1AWJuafULEE
--HG--
extra : rebase_source : 153653733e01433ae894a0ba4fd99228e0936024
The symbol-upload task currently downloads the symbols-full.zip artifact
from the build task and then uploads it to the symbol server. These zip
files can be very large (>1GB) so we spend a lot of time doing that.
Now that we're uploading to Tecken instead of Socorro, we can instead
just send the URL of the artifact to Tecken's upload API and ask it to
fetch that directly:
https://tecken.readthedocs.io/en/latest/upload.html#upload-by-download-url
This should make the symbol upload task a fair bit faster.
MozReview-Commit-ID: 8HcbgrWYT1O
--HG--
extra : rebase_source : 4e8f7a28c956befb3e291e8be4d41a2b6728e5cd
With upload-symbols tasks running in the lint image now there's no reason
for them to have a special worker type. Dustin suggested using the builder
worker because those are likely to have the lint image cached.
MozReview-Commit-ID: qLknmv1vaC
--HG--
extra : rebase_source : 4ecb68c3ada989445bb0d31503527a7780b3c128
The upload-symbols task simply runs an in-tree Python script with
`mach python` so using a sparse profile will make this a bit faster.
MozReview-Commit-ID: 5HzwMf1FZLU
--HG--
extra : rebase_source : 74046b7917ee2e2ea7be0ec24f777f5fe819ef58
This change makes upload-symbols tasks use run-task and the in-tree lint
image instead of the private upload-symbols image. A prior change changed
the script to get the token it needs from a Taskcluster secret, so it's no
longer necessary to use the private docker image containing the token.
MozReview-Commit-ID: 6QugVB4chE0
--HG--
extra : rebase_source : e13d29c2a88e055247da374cffa9ea153548749d
This change fixes symbol upload to use a token stored in the Taskcluster
secrets service instead of the token stored in the private Docker image.
Additionally, it changes the script to upload symbols to the Tecken staging
server when run on try so that the upload-symbols tasks can be tested on
try now. In the future there are plans to allow try tasks to upload symbols
to a separate storage area on the production Tecken instance.
MozReview-Commit-ID: BeZGiiwuGp8
--HG--
extra : rebase_source : ee4c680410822e94c3001d07242f69378703659f
Land date changes to support windows nightlies onto central
MozReview-Commit-ID: 4HgW0D2BR8B
--HG--
extra : rebase_source : b87b1159307205cebd10e1d8598386aac3976706
Land date changes to support windows nightlies onto central
MozReview-Commit-ID: HadAf7tPTQJ
--HG--
extra : rebase_source : ba4fc31de6bc28ac547ab06cd84b8dd18c2a1199
I like when my code search yields relevant data with 1 line of context.
MozReview-Commit-ID: 8DPyJeyFMwR
--HG--
extra : rebase_source : 0b4d690c8f175e11a1a0db5fda8fc9f8628441dc
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
--HG--
extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
Add configurations for building and uploading AArch64 Nightly builds, in
tier 1 and without artifact support for now.
As for not denoting AArch64 builds as "api-21", I don't really think we
will split AArch64 the way we split ARMv7 before. Originally, we split
into API 9 and API 11+ because of lots of "constrained" devices that
were stuck with API 9. We made an API 9 APK in order to lower our
footprint on those devices. That probably will not be a problem for
AArch64, because devices with API 21+ and AArch64 support are usually
more than capable for running Fennec. Secondly, it was a big change for
Android going from API 9 to API 11+, so we saved quite a bit of
code/resources when we stripped out API 11+. I don't see such drastic
changes going from API 21 to upcoming versions, so even if we did split,
I don't think it'll get us much benefit.
MozReview-Commit-ID: 7N7Slv1pPgb
This will fail until Bug 1265537 lands, which is what will switch windows to SIMPLE NAME package format.
MozReview-Commit-ID: J0zVLptvxNO
--HG--
extra : rebase_source : 6c388f24bce78ee6f6a8d7eb17cd6bce036b1d35
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
Mike Hommey
Dorel Luca
Justin Wood
Tom Prince
Dustin J. Mitchell
Sebastian Hengst
Margareta Eliza Balazs
Christian Holler
Ted Mielczarek
Chris AtLee
Steve Armand
Mike Shal
Phil Ringnalda
Ryan VanderMeulen
Johan Lorenzo
Gregory Szorc
Jim Chen
Ben Hearsum
Aki Sasaki