for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1403
--HG--
extra : rebase_source : b7b7f34204bd06ba2baf92ae6d103f8c207214fb
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1403
--HG--
extra : rebase_source : fc669df941b70b720a6eb4118ad36d6679a28d48
Since it is run with `mach python`, it uses the environment defined by
`build/virtualenv_packages.txt`. Thus, we don't need to create a separate
virtualenv.
Differential Revision: https://phabricator.services.mozilla.com/D1015
--HG--
extra : rebase_source : 023095f82d7219a10dffb3579276c5285db37cfc
extra : source : a2999a5b9b7aa08a7c5c2ba6384724853d14b9c5
In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : 8c76bc91e1d5102f68c43e1050d61971fef32e9f
In many cases, building docker images starts on machines that don't have
a cached checkout, and it often takes forever to get a full clone. It
used to be worsened when 3 jobs could run at the same time because the
worker would start up clean, and 3 jobs would be doing a mercurial clone
at the same time, thrashing I/O, but that part is fortunately fixed.
It is still, however, appreciable not to waste time in the mercurial
clone part of image creation.
--HG--
extra : rebase_source : bbe8b001849e59bb655bb0e9766a6071ad38a52c
upload-generated-sources tasks simply use `mach python` to run an in-tree
script, so they can save time by using a sparse profile.
MozReview-Commit-ID: LbXlibOP34W
--HG--
extra : rebase_source : c6b580b15671edeb700d191f651c79c6d6423394
The upload-symbols task simply runs an in-tree Python script with
`mach python` so using a sparse profile will make this a bit faster.
MozReview-Commit-ID: 5HzwMf1FZLU
--HG--
extra : rebase_source : 74046b7917ee2e2ea7be0ec24f777f5fe819ef58
With Gradle integration centralized in gradle.configure, changing
these integration points will need to trigger the android-* tasks.
MozReview-Commit-ID: DuOuW1RIgCh
--HG--
extra : rebase_source : a3f0a76b1ae6b3cd952271782f9d0c2463b704e0
With Gradle integration centralized in gradle.configure, changing
these integration points will need to trigger the android-* tasks.
MozReview-Commit-ID: DuOuW1RIgCh
--HG--
extra : rebase_source : 9aecfab8a8ce41441fb4c25021b14263d9c115c2
This hooks up the `target_tasks_method`s and the
`release_promotion_flavor`s for these four release promotion flavors.
We also add some maple support and add `version_display.txt` to the
decision task sparse checkout so we can read the version number during
decision/action task time.
MozReview-Commit-ID: CdxUUXZtXO0
--HG--
extra : rebase_source : 99e4f060a964d34a377aea6a6e36fbcef66c6aec
At this point, we could tear out the `ignore-locales` attribute, since
l10n-bumper supplies that information. However, we may want to use flatfiles
for something; `ignore-locales` allows for that.
MozReview-Commit-ID: 8mD4iav3bKx
--HG--
extra : rebase_source : abe2075503838223a2c150676b9c72a1aa74df59
This is needed so we can import the WPT manifest parser.
Like for the reftest parser, this should only be temporary until we no
longer parse test manifests as part of Files() evaluation.
MozReview-Commit-ID: 2dmUfP8tLd2
--HG--
extra : rebase_source : 36624618f96fc89bae88172947b21108740d841d
Some moz.build files need to import in-tree Python modules in order to
parse test manifests. While bug 1402010 details a better solution
to remove this requirement, the easy fix is to expand the sparse
checkout to include the missing files.
MozReview-Commit-ID: FSx6u6r2XfE
--HG--
extra : rebase_source : 6e406dd76b4f78521cbbdc40110d0610c75e2b08
extra : amend_source : f68341a14f50c8ff6e7b62214fa455b75eabe908
Strictly speaking we don't need all files in the directories listed
in the profile. But the checkout is still small enough and it is far
less effort than cherry-picking every file needed by every toolchain
task.
This brings the checkout down to ~3700 files, which only takes 1-2s.
MozReview-Commit-ID: 2BpKdZ2Pvx9
--HG--
extra : rebase_source : e9589b29f7e1b60ac7ee4e8050689dc3d5a8f418
extra : source : 2c0d6e566cc719823515ec2403e1d6b2ace955aa
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : f208e4960cf76a9dfe77634b87f0058f676e9fa9
extra : source : 046d705929f7a41e977eec19c8503afccdec7592
Strictly speaking we don't need all files in the directories listed
in the profile. But the checkout is still small enough and it is far
less effort than cherry-picking every file needed by every toolchain
task.
This brings the checkout down to ~3700 files, which only takes 1-2s.
MozReview-Commit-ID: 2BpKdZ2Pvx9
--HG--
extra : rebase_source : 916c6722c6d495cdccea0076c673d75e4ee48f28
extra : source : 2c0d6e566cc719823515ec2403e1d6b2ace955aa
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : 4cf3efc9c57bb14d2f44147c8881d0a0a18703d6
extra : source : 046d705929f7a41e977eec19c8503afccdec7592
The Sphinx documentation only needs access to a relatively small number
of files in the repo in order to be generated. It is a good candidate
for using sparse profiles.
This commit defines and uses a "sphinx-docs" sparse profile containing
only the files relevant to Sphinx documentation generation.
There are some quirks with the profile:
* All moz.build files are included. This bloats the profile
by >1000 files. Worse, it realizes directories that have no business
being realized. This clutters the checkout and makes it harder to
find things. There is a moz.build reader that knows how to retrive
file data from Mercurial. We could use that. This feels like follow-up
fodder.
* All mach_commands.py files are included. `mach help` says you can do
things that you aren't able to do in the sparse checkout. There isn't
a good way to add all *.py files while excluding mach_commands.py
files. We /could/ do it with regular expressions. But those are slow.
Let's leave it as is for now and come up with a better solution later.
MozReview-Commit-ID: 7yiqGGE1nAh
--HG--
extra : rebase_source : c148040ea3618e8bfdd369b6f48fc60c6d179285
extra : source : b76e2f6204b20de137f2566dff8121ff3abe5760
This adds some new optimization strategies. For tests, we use Either(SETA,
SkipUnlessSchedules), thereby giving both mechanisms a chance to skip tasks. On
try, SETA is omitted.
MozReview-Commit-ID: GL4tlwyeBa6
--HG--
extra : rebase_source : 0c1ce762afc7a691788379d4f4206df106f6df63
Mercurial's sparse checkout support allows "profile" files defining
what's in sparse checkouts to be defined in-tree. This convenient
feature means you simply need to point a client at a path inside the
repository and it dynamically resolves what files to include in the
checkout. As you update revisions, the "profile" pulls in updates
to the underlying file.
We introduce 2 sparse profiles: 1 for mach and another for taskgraph.
The goal of the "mach" profile is to provide enough files to run
`mach`. If you activate this profile and run `mach`, it runs without
error. But there are practically no commands available. So it isn't
terribly useful.
The "taskgraph" profile allows us to run `mach taskgraph`. This
profile demonstrates a sparse profile feature: including other
profiles. The "taskgraph" profile is thus a union of "mach" and
its own entries.
There is definitely some fat in these profiles. I didn't feel like
chasing the long tail and getting overly granular with the profiles.
If we want to optimize later, we can do that.
For reference:
Full checkout: ~234,000 files
mach: ~2,000 files
taskgraph: ~3,600 files
MozReview-Commit-ID: 7pALt0MwHfE
--HG--
extra : rebase_source : 1a4ba4b8a63c522dab2841e2c0019501476da2fe
Johan Lorenzo
Justin Wood
Bogdan Tara
Sylvestre Ledru
Dorel Luca
Dave Hunt
Tom Prince
Mihai Tabara
Rail Aliiev
Ben Hearsum
Mike Hommey
shindli
Aki Sasaki
Ted Mielczarek
Nick Alexander
Robert Helmer
Maja Frydrychowicz
Brian Stack
Gregory Szorc
Dustin J. Mitchell
Phil Ringnalda
Ryan VanderMeulen