This adds the R.txt files produced by the build -- timestamped, so
they are kept in order -- to the Task Cluster artifacts, for later
comparison.
MozReview-Commit-ID: 3hj6XjFDIE4
--HG--
extra : rebase_source : 04c1bcd2bf62fd193541fa92dd8841db102f6b5a
extra : source : 5a00c3642c972385cd212fe8b15240bce6acef50
This also turns the tier 2 job B(n)g into tier 1, since moz.build is
still tier 1. It also pushes a lot of GeckoView related tasks into
the main builds, since they should run as part of Gradle builds.
This also removes unused tooltool manifests; the jobs that used these
manifests use only toolchain tasks now.
MozReview-Commit-ID: 2GmnJ7joCTT
--HG--
extra : rebase_source : 75cd2dfb51e0e1b510f5e618c2dc881cf5f22bf2
extra : source : 6b95b09d6afbb83ba89c47b237dfce6e15587bbe
- Building is nightly channel only. Beta and release for Fennec 58 don't build
stylo. It means that the package size for 58 beta/release isn't incremented
by this change.
- The preference for stylo is still turned off Nightly 58. It will be turned on
59 after fixing some bugs for crashtests and etc. Our target to enable stylo
for Android is 59.
- ./mach bootstrap already installs clang etc to build stylo and bindgen.
Developers for mobile won't require additional build options for this change.
MozReview-Commit-ID: CIpYl8I5d7x
--HG--
extra : rebase_source : 6387704e4a94db080d4add10298cf1cc254ddec0
The only tricky piece here is that the resulting toolchain archive is
private, and uses a newly allocated Task Cluster scope
(queue:get-artifact:project/gecko/android-sdk/*) to restrict access to
the archive. All SCM levels (1, 2, 3) have been given the new scope:
see https://tools.taskcluster.net/auth/roles/moz-tree:level:1 and
friends.
MozReview-Commit-ID: CcDqDOHODpe
--HG--
extra : rebase_source : 81dbb065f2a3c4e7733e964be66adb1733db52c6
The only tricky piece here is that the resulting toolchain archive is
private, and uses a newly allocated Task Cluster scope
(queue:get-artifact:project/gecko/android-sdk/*) to restrict access to
the archive. All SCM levels (1, 2, 3) have been given the new scope:
see https://tools.taskcluster.net/auth/roles/moz-tree:level:1 and
friends.
MozReview-Commit-ID: CcDqDOHODpe
--HG--
extra : rebase_source : 062bca8c65556f0f46e9c9cc6cd81eb04cf2b522
The goal of this approach is to tell Gradle to not connect (or allow
it to connect) to the network when fetching dependencies. No Android
automation tasks should fetch from the network, except the toolchain
tasks (which are specially intended to do so).
It's difficult to arrange this without including the `--offline` flag
everywhere. It _should_ be possible to set offline using an
environment variable -- which would allow us to get rid of these
dotgradle-* files -- but offline isn't an option in
https://docs.gradle.org/4.2.1/userguide/build_environment.html#sec:gradle_configuration_properties
(and certainly not in earlier versions either). Therefore,
environment variable that points to an init.gradle file in automation.
Before this patch, the files telling Gradle whether to start offline
were fetched from tooltool. That's just a layer that doesn't need to
be there.
None of this impacts local developers.
MozReview-Commit-ID: LAXktbBu1Az
--HG--
extra : rebase_source : d23801643d32135a87d410bf5e8508da556ef9be
Add a toolchain job description which calls the
repack_rust.py script to package the requested
upstream build of Rust and its standard libraries
for use in gecko builds.
Links are added to these new toolchains for various build
and analysis tasks as appropriate. The base-toolchain
tasks use an explicitly-versioned toolchain since those
can be different from the current release used for most builds.
The corresponding tooltool manifest entries are removed
now that taskcluster artifact versions are available.
This simplifies the update process since new toolchains
can be packaged and used automatically by just updating
the versions in the task descriptions.
A 'linux64-rust' toolchain can be added to other tasks
as a dependency and artifact. It supports linux64-
hosted builds of Rust code targeting linux64 or linux32.
A 'linux64-rust-macos' toolchain targets linux64-hosted
builds of Rust code targeting macOS on x86_64.
A 'linux64-rust-android' toolchain targets linux64-hosted
builds of Rust code targeting various Android architectures.
Two 'win64-rust' and 'win32-rust' toolchain tasks create
similar entries for Windows-hosted builds. All our automation
builds are hosted on win64, so we could use one artifact
with support for both targets, but currently this doesn't
work because of cross-compilation issues in some crates.
This patch maintains the previous separation between
win32 and win64 rust toolchains until that can be addressed.
MozReview-Commit-ID: GRiJml8CtzO
--HG--
extra : rebase_source : 09a3698ce7f9a8b5f2b5d9b5a1fde9c05dc6b540
Now that we have a Docker image with newer library versions on it, we
can move our builds over. The new images differ from the old
CentOS-based images in two important ways, though:
1) The system compilers in the new image are new enough to be used as
host compilers; additionally, our CentOS-built GCC compilers will not
work. We need to change the Android mozconfigs to reflect that. We
also need to change the Android tasks to not depend on the GCC
toolchain builds.
2) In a similar fashion, we can use the system JDK; we no longer need to
use the JDK from the Android NDK, which we had packaged up via the
Android dependencies task.
Both of these changes come with caveats: our l10n repack jobs continue
to run on the CentOS-based images; l10n repacks have not been completely
converted to Taskcluster. So we need to:
1) Retain the use of our custom GCC toolchain for HOST_CC/HOST_CXX on
the CentOS-based images.
2) Retain the JDK packages in the tooltool manifests, and referencing
them when we build on the CentOS-based images.
CLOSED TREE
--HG--
extra : amend_source : 84120d6bacb5d72a9fbe41e4c3b405d63825da7c
extra : histedit_source : 8320c2193761b745f10850055ee74a3c9ac73615%2Cfbc2a28d8c5004a53305ef858ca5aea4245691e0
Now that we have a Docker image with newer library versions on it, we
can move our builds over. The new images differ from the old
CentOS-based images in two important ways, though:
1) The system compilers in the new image are new enough to be used as
host compilers; additionally, our CentOS-built GCC compilers will not
work. We need to change the Android mozconfigs to reflect that. We
also need to change the Android tasks to not depend on the GCC
toolchain builds.
2) In a similar fashion, we can use the system JDK; we no longer need to
use the JDK from the Android NDK, which we had packaged up via the
Android dependencies task.
Both of these changes come with caveats: our l10n repack jobs continue
to run on the CentOS-based images; l10n repacks have not been completely
converted to Taskcluster. So we need to:
1) Retain the use of our custom GCC toolchain for HOST_CC/HOST_CXX on
the CentOS-based images.
2) Retain the JDK packages in the tooltool manifests, and referencing
them when we build on the CentOS-based images.
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: 554IPMRWgzK
--HG--
extra : rebase_source : 00827d3f6bd705419bc801eb05b543af1ddc274f
Using /home/worker is the build directory has a 30% talos performance
loss, because test machines has a /home mount directory.
MozReview-Commit-ID: zehcGJrUQX
--HG--
extra : source : feedcde68c2a54da210f03eb287ab5c862fc982b
extra : intermediate-source : 485d1af7805ad9fa0e701c3571fc1291fbfc6850
Since bug 1321847, mozharness tooltool manifests can be overriden from
the environment. We use that possibility to now define tooltool
manifests from taskcluster job definitions. Ideally, we'd also remove
the definitions from the mozharness configs, but with things still
running on buildbot, it's not clear what things might break because of
that. We'll do it in a separately back-out-able followup.
--HG--
extra : rebase_source : 860b8f1d4fdc4a557770a3749055f19b1ec45e93
Previously, mozharness defined a separate action to collect build
metrics. This required the script and/or config to define that
action.
Metrics collection for CI is important. So it should be enabled by
default.
This commit changes the "build" action/method to always call the
metrics collection function after successful build. References to
the "generate-build-stats" action have been removed because it is
redundant.
A side-effect of this change is we may generate build metrics where
we weren't before. This could lead to e.g. duplicate entries in some
Perfherder series. Let's see what breaks ;)
MozReview-Commit-ID: 42UQI5YQTMC
--HG--
extra : rebase_source : c57dc9ec6ac46003384edff098a0ad81c75539b7
extra : source : c9812dd7d27a174c0ee46d44ec595fbe29c9e1db
This was causing decision tsak failures; it snuck through while rebsaing to land.
MozReview-Commit-ID: 7xgnjkFzSYU
--HG--
extra : amend_source : 019af69e9f0d3f9db932a3494b8a24571de90f62
extra : histedit_source : 61e7cce868e648a9ca404e1841146f12a7659c94
To date we have variously specified both worker-type and worker-implementation,
often manually coordinated. We also embedded a few awkward assumptions such as
that the native engine only runs on OS X.
But a worker type has one and only one implementation, and that implementation
is stable over time (as changing it would require simultaneous landings on all
trees).
Instead, this change makes worker-type the primary configuration, and derives
both a worker implementation (defining the payload format) and worker OS
(determining what to include in the payload) from that value. The derivation
occurs when deciding how to implement a particular job, where the run_using
functions are distinguished by worker implementation.
The two-part logic to determine how and where to run a test task based on its
platform is combined into a single transform, `set_worker_type`.
This contains some other related changes:
- MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker
is used
- the URL to test packages is factored out into a shared function
- docker-worker test defaults are applied in `mozharness_test.py`
- the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is
split
- the 'invalid' workerType is assigned an 'invalid' implementation
- all tasks that do not use job descriptions but use docker-worker, etc. have
`worker.os` added
Tested to not produce a substantially different taskgraph for a regular push, a
try push, or a nightly cron.
MozReview-Commit-ID: LDHrmrpBo7I
--HG--
extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
Add configurations for building and uploading AArch64 Nightly builds, in
tier 1 and without artifact support for now.
As for not denoting AArch64 builds as "api-21", I don't really think we
will split AArch64 the way we split ARMv7 before. Originally, we split
into API 9 and API 11+ because of lots of "constrained" devices that
were stuck with API 9. We made an API 9 APK in order to lower our
footprint on those devices. That probably will not be a problem for
AArch64, because devices with API 21+ and AArch64 support are usually
more than capable for running Fennec. Secondly, it was a big change for
Android going from API 9 to API 11+, so we saved quite a bit of
code/resources when we stripped out API 11+. I don't see such drastic
changes going from API 21 to upcoming versions, so even if we did split,
I don't think it'll get us much benefit.
MozReview-Commit-ID: 7N7Slv1pPgb
This patch enables `run-on-projects` to work appropriately for
nightly builds and tests. Initially, we were setting an empty
`run-on-projects` for nightly `build_platform`s, then explicitly
targeting the platforms in nightly-specific `target_task_method`s.
Instead, this patch enables nightlies to `run-on-projects` everywhere,
but governs the use of nightlies by either the `include_nightly`
parameter, or the `--include-nightly` try option. This lets us filter
nightly-related `target_task_method`s against `run-on-projects` without
losing all nightly tasks.
Then, enable spidermonkey tests by removing optimization from beta and
release. This patch also enables everything then disables specific
tasks, rather than disabling everything and enabling specific tasks.
Since we're beginning with a `filter_for_project` call, we should be
able to reduce these if blocks to zero over time, if desired.
MozReview-Commit-ID: A9tolynaChF
--HG--
extra : rebase_source : 3465ee2c714de3e0359f14109096fc94de27aadf
Adds l10n and nightly indexing, matching (better) what Buildbot is currently doing with these types of tasks
(This patch is against `date`, will be grafted on review for real landing, using autoland)
MozReview-Commit-ID: K0BYwaCm6xL
--HG--
extra : source : b3254f6cc3f6f526ec3877add890d12a8947080c
extra : amend_source : 6f0467058cea9318d0a160ce6e2f76b6a9a5b119
We use a Maven repository and the (misleadingly named!) uploadArchives
task because this is the best way to make Android Studio download and
recognize the Javadoc and sources. With this, it's automatic; with a
single AAR file, it's a nightmare of point-and-click configuration.
This patch does a bunch of Gradle hacking to make -javadoc and
-sources JARs; there's nothing special or particularly likely to break
here.
This patch also adds Proguard declarations to the :geckoview library
project. That involves moving a good part of the Proguard
configuration into mobile/android/geckoview. (I also expand upon the
existing configuration.) This should be only a re-arrangement, and
the resulting file is included in the original, so nothing should be
changed.
MozReview-Commit-ID: BGNW1v92J0k
--HG--
extra : rebase_source : 94633d27e8ae6bafa3d6823996355c22d2e2e6eb
We also include the example APK, since it will be helpful to be able
to regression test the example without downloading the AAR and
manually building the example with that AAR.
MozReview-Commit-ID: CMtA1FRS0Rf
--HG--
extra : rebase_source : 42bab43c69093bd008242ec96e74f53bde150583
Steve Armand
Nick Alexander
Nathan Froyd
Makoto Kato
Sebastian Hengst
Ralph Giles
Gregory Szorc
Johan Lorenzo
Wander Lairson Costa
Mike Hommey
Wes Kocher
Dustin J. Mitchell
Jim Chen
Ben Hearsum
Carsten "Tomcat" Book
Simon Fraser
Geoff Brown
Aki Sasaki
Chris Manchester
Justin Wood
Mike Shal
Haard Panchal