We recently changed how image compression works (bug 1350447) and
upgraded to Mercurial 4.1.1 (this bug). It's time to roll a new
image_builder Docker image.
This commit updates the version and hash of the image. The image
is already pushed to Docker Hub/Cloud. We just need to whitelist
it and deploy this change.
MozReview-Commit-ID: KQwJZ2OSGlt
--HG--
extra : rebase_source : 23f2686cd85256178b5d89c452aaeabe4e4198db
We've been running Mercurial 3.9 in automation for a while. Mercurial
4.1 is out. It has the usual performance improvements and bug fixes,
making it an attractive upgrade. But what really makes it enticing
is support for zstandard compression over the wire protocol. This will
reduce server-side CPU load and make transfers between 4.1+ servers
faster.
So, let's upgrade to Mercurial 4.1.1.
The produced packages are built from the current tip of the
stable Mercurial branch, not exactly 4.1.1. Specifically, they
correspond to revision ed5b25874d99. I did this because there is
a patch in the stable branch that drastically improves performance
on repos with many heads. When 4.1.2 is released in a few days, we
can upgrade to it and do away with the one-off.
MozReview-Commit-ID: 6BPhVheHQXI
--HG--
extra : rebase_source : d72452708fa0397272b79247d9d4844b35e5ee26
b2gtest uses *.xlarge instances, so it seems a reasonable substitution.
MozReview-Commit-ID: Ev6tnNpx13c
--HG--
extra : rebase_source : 87184b182d1428f041a5ae36e465d72dee6f7621
(For "Integrate and fully support OSX Signing in taskcluster")
Written as a mozharness script rather than using bare ./mach command because we need to download the upstream artifact
and because we need to download artifacts from tooltool to do the packing back into a .dmg. Future ideal would be to get
rid of the mozharness script and use JUST ./mach.
This is using the ./mach repackage code being created in Bug 1347576. Taking a signed tarball from a dmg supported with
Bug 1346015, and the taskgraph work to schedule this is in Bug 1318505.
MozReview-Commit-ID: rv9l285HKC
--HG--
extra : rebase_source : 054219511419b8bf44b1f57a8d834a12c13710e3
extra : intermediate-source : a52bc37e08efbf4d6c68cc0f4e2d4b76f79b192a
extra : source : 6ad7468a590f5a2779ffdc3713c1f6f74ce23731
In bug 1347269 I missed removal of a test set that should have been removed.
This follow up removes this set also.
MozReview-Commit-ID: Be4GjLm6OgB
--HG--
extra : rebase_source : ee29ebbe98a26994cc9ba87920f2191cbe3c9c6f
This fixes a regression introduced in Bug 1335873, which changes the mozbase
packages.txt to call mozlog's setup.py. Calling setup.py registers the
pytest_mozlog plugin for marionette-harness tests.
Instead, we can register the pytest-mozlog plugin via command-line arguments
to pytest, which are set in pytest.ini for the marionette-harness tests.
As a result, we can revert the mozbase packages.txt to not refer to mozlog's
setup.py
I'm leaving the pytest entry-point in mozlog's setup.py so that external
consumers don't have to register the pytest_mozlog plugin manually.
MozReview-Commit-ID: I5wNq5H1x3X
--HG--
extra : rebase_source : 614a47995bc1655f36053d2a05b08f94bfdbe476
Let's do this by modifying the mochitest retrigger action into a more
generic "test retrigger" action that supports both.
MozReview-Commit-ID: H3AbsJ58ASV
--HG--
extra : rebase_source : 048025a8665c6be54afa214355900a663fc84615
As part of the media teams ongoing sunsetting of the external media tests, we
are disabling tests once they become problematic. This disables the Linux
TaskCluster tests, which have been over represented in failures recently.
MozReview-Commit-ID: BOJ7TyL5LYC
--HG--
extra : rebase_source : 4c00ad9f91e92d88c3f5c5c6c8e8565c3aae42b9
Apply a 2-character indent to in-tree tooltool manifests to make
them easier to read, and to make the formatting more consistent
so automating updates is simpler.
Modern editors will maintain json indentation. The only long
lines we have are already over 80 characters, so the extra space
shouldn't create new long lines.
Also update mercurial installer script to generate json with
the same indentation, even though its output is temporary.
Tooltool itself was updated to generate manifests with this
indentation in Bug 1325225.
MozReview-Commit-ID: DKj6nL9OENv
--HG--
extra : rebase_source : fc3f8616ec689d74e06c0db84c2b261825f86453
`task.dependencies` is part of the internal taskgraph calculations, so it must
only refer to tasks that are being created, and not to those which already
exist. `task.task['dependencies']`, on the other hand, is what is given to
`queue.createTask` and expresses all dependencies.
MozReview-Commit-ID: GJ6JVj6JMBz
--HG--
extra : rebase_source : 553aec85134fe0e41c53f917327b13d1b66592aa
This patch enables `run-on-projects` to work appropriately for
nightly builds and tests. Initially, we were setting an empty
`run-on-projects` for nightly `build_platform`s, then explicitly
targeting the platforms in nightly-specific `target_task_method`s.
Instead, this patch enables nightlies to `run-on-projects` everywhere,
but governs the use of nightlies by either the `include_nightly`
parameter, or the `--include-nightly` try option. This lets us filter
nightly-related `target_task_method`s against `run-on-projects` without
losing all nightly tasks.
Then, enable spidermonkey tests by removing optimization from beta and
release. This patch also enables everything then disables specific
tasks, rather than disabling everything and enabling specific tasks.
Since we're beginning with a `filter_for_project` call, we should be
able to reduce these if blocks to zero over time, if desired.
MozReview-Commit-ID: A9tolynaChF
--HG--
extra : rebase_source : 3465ee2c714de3e0359f14109096fc94de27aadf
This attempts to address intermittent twitch test failures on Linux by having
the tests run on beefier hardware. Bug 1344961 shows a number of failures
where tests timeout before page loading is complete, the hope here is that
more powerful hardware speed up these load times.
MozReview-Commit-ID: 2yBQvKS8Xlh
--HG--
extra : rebase_source : 2b803204e90a736b4634279ca8199cb8910551ba
Graph morphs modify the graph after optimization, without changing its meaning.
In this case, that means adding index tasks that will insert paths into the
index beyond the relatively limited number afforded in task.routes.
MozReview-Commit-ID: AJy4exX7q2v
--HG--
extra : rebase_source : d61e7462defd41e7112739fb057edb493f495430
extra : source : c580568ed47c1ed2af40d98b47fbb0d136e63060
This introduces a new docker image: `index-task`,
which given a taskId and a set of namespaces will
index the given taskId under said namespaces.
Modified to include a script with a descriptive name that curious users can
find in the source code.
MozReview-Commit-ID: KPHVT0XPfsb
--HG--
extra : rebase_source : ed3abde6082f98b83bc5b13f65172562b8875263
extra : source : f8ca244c170f587ead852bb9fbdc99c9dec52f31
Note that the to_json method prefers the taskgraph's dependencies information
(edges) to that from the task.dependencies entries. At a few points in
task-graph generation, these values differ, although that is expected (for
example, the full task set contains no edges, but that information is still in
task.dependencies). Unifying that representation leads to some difficulty with
task transforms that reach into the dependency tree (beetmover), so the
different representations are left as-is.
MozReview-Commit-ID: GeW8HNwFA9Z
--HG--
extra : rebase_source : 549773e05e18371a399612d9bceccffc29be8cf2
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
The previous attempt at this didn't handle jobs that were keyed by platform,
which was most of them.
MozReview-Commit-ID: IC602td532T
--HG--
extra : rebase_source : 95cdf9ad37df8ef6665665f11e59f8ae8304dbd2
this patch:
- adds linux{32,64}-nightly/opt test platforms that mirror the non-nightly test platforms.
- adds an `include_nightly` per-project parameter; this is refered to in the default `target_task_method`. It's still possible to launch custom `target_task_method`s to trigger nightlies against, say, try.
- adds a `filter_for_project` method in `target_tasks.py` that allows for `include_nightly` and `run_on_projects` filtering in the various `target_task_method`s.
- adds nightly filtering into the `TryOptionSyntax` object. By default, this will be off. To trigger nightly tests on try, either submit a new decision task with a different `target_task_method` (e.g. `nightly_fennec`) or flip the `include_nightly` flag to True.
- adds the `nightly` attribute to tests if their builds have that attribute.
MozReview-Commit-ID: DttIZH0BHS2
--HG--
extra : rebase_source : d8acbe4c741f570b2e8d33a8e6a7f5c791b24ff6
Necessary for treeherder action retriggering code to recognize them as supporting
this action.
MozReview-Commit-ID: BY6OCUFsYlK
--HG--
extra : rebase_source : 6cc4bc8b1cfde29f793fd910bf99f8d3e36603da
* add run.using = 'run-task' for native-engine
* modify run-task to run on OS X
- not as root
- without assuming /home/worker (using ~ and os.expanduser instead)
- hg is in /usr/local/bin on OS X; trust the PATH
* add_build_dependency isn't docker-worker specific, so just rename
* support_vcs_checkout modified to omit caches on native-engine
* don't download fingerprints on OS X; these hosts are configured with
the proper fingerprint via puppet
MozReview-Commit-ID: C83XClXtcn4
--HG--
extra : rebase_source : 2ef1e8dced12ccc4acb7706d7f4587df19a379fc
This fixes the ability to run mozbase via `-j mozbase`, with the
added advantage that it will obey `-p` too.
MozReview-Commit-ID: 1zkitUephXk
--HG--
extra : rebase_source : 0ebb65363d5f5813bc7ccb379768df54310d39c1
This clears up some confusion and undocumented behavior around platforms in
source-tests (and available to any job).
With this change, the attributes come out like this:
"source-test-mozbase-linux64/opt": {
"attributes": {
"build_platform": "linux64",
"build_type": "opt",
"kind": "source-test",
"run_on_projects": [
"integration",
"release"
]
},
MozReview-Commit-ID: HN1Zi8YUf0
--HG--
extra : rebase_source : 552bffc4646a3eec46e7edb508d8eb4d2a8e2e03
* This adds a kind: 'task' property to actions in actions.json
* This is already specified in the schema for actions.json
MozReview-Commit-ID: JmlAZouVPIf
--HG--
extra : rebase_source : 5855b9ddbfff2faa9e1db12fc1f60bbef3cccabb
Bug 977849 added many new jsreftests on Android, resulting in more than
a 3x increase in some job run times. This significant addition of jobs
is required to return Android jsreftest job times to 'normal'.
Various modules under taskcluster are doing ad-hoc url formatting or
requests to taskcluster services. While we could use the taskcluster
client python module, it's kind of overkill for the simple requests done
here. So instead of vendoring that module, create a smaller one with
a limited set of functions we need.
This changes the behavior of the get_artifact function to return a
file-like object when the file is neither a json nor a yaml, but that
branch was never used (and was actually returning an unassigned
variable, so it was broken anyways).
At the same time, make the function that does HTTP requests more
error-resistant, using urllib3's Retry with a backoff factor.
Also add a function that retrieves the list of artifacts, that while
currently unused, will be used by `mach artifact` shortly.
--HG--
extra : rebase_source : d7ef633e8e5041dc8450f3ff2f3751c85f144cdc
Instead of every file trying to get the top source directory having an
ad-hoc definition that gets wrong if the files gets moved around for
some reason, define it in a more central location.
--HG--
extra : rebase_source : 1a0cbea267193f6b388b88b36166353e20ac8569
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 22382e3d65ce8454a1682cfced0d03477762e8fe
Disable non-tests for the following tests as the builder machines have
been turned off:
* web-platform-tests-reftests
* mochitest-webgl
* jsreftest
* reftest
MozReview-Commit-ID: 5KZ3KzT3dLo
--HG--
extra : rebase_source : 110ce9bee70adc2ad314b8927385b851f1ce67b7
Currently 'run_task' tasks have no easy way to depend on a build task. For example, some
python unittests need a Firefox binary for their tests, like the mozrunner tests and future
test harness selftests (like mochitest tests).
This patch allows kinds to add a new key to the kind config which maps test platforms to
build-labels. Then 'run_task' jobs can add a 'requires-build': true field to get a build
dependency automatically added. The build artifact url will also be stored in the
$GECKO_INSTALLER_URL environment variable on the test host.
MozReview-Commit-ID: Jqyhbj7nC6z
--HG--
extra : rebase_source : 2f44b6c94f35a0d2e11464cf773e821ae6fe8538
The name 'source-check' is a bit of a misnomer, because it already includes a bunch
of tasks that are proper unittests, not lints. Some of these unittests will soon
depend on a build task, which makes 'source-check' feel even more wrong.
They still have a lot in common with the lint tasks though, so it's probably not
worth splitting this into two separate kinds. Instead, let's just rename this kind
to 'source-test', which means, any task that tests stuff and is run from the source
directory (instead of a tests.zip). I think both lints and python-tests fall squarely
under this definition.
MozReview-Commit-ID: K0gZ5rVLyeD
--HG--
rename : taskcluster/ci/source-check/doc.yml => taskcluster/ci/source-test/doc.yml
rename : taskcluster/ci/source-check/kind.yml => taskcluster/ci/source-test/kind.yml
rename : taskcluster/ci/source-check/mozlint.yml => taskcluster/ci/source-test/mozlint.yml
rename : taskcluster/ci/source-check/python-tests.yml => taskcluster/ci/source-test/python-tests.yml
rename : taskcluster/ci/source-check/webidl.yml => taskcluster/ci/source-test/webidl.yml
extra : rebase_source : a683b5b6e243849de57f2681993046c776d6a8f2
This will be used to restrict mochitest actions to mochitest jobs only.
MozReview-Commit-ID: DbFb9V6s9Rb
--HG--
extra : rebase_source : 16ebd751bf7048fd46d71bd350119ca3f8a68302
Various modules under taskcluster are doing ad-hoc url formatting or
requests to taskcluster services. While we could use the taskcluster
client python module, it's kind of overkill for the simple requests done
here. So instead of vendoring that module, create a smaller one with
a limited set of functions we need.
This changes the behavior of the get_artifact function to return a
file-like object when the file is neither a json nor a yaml, but that
branch was never used (and was actually returning an unassigned
variable, so it was broken anyways).
At the same time, make the function that does HTTP requests more
error-resistant, using urllib3's Retry with a backoff factor.
Also add a function that retrieves the list of artifacts, that while
currently unused, will be used by `mach artifact` shortly.
--HG--
extra : rebase_source : 06777dea62e884f546a5b951baad80fd8aec1f1e
Instead of every file trying to get the top source directory having an
ad-hoc definition that gets wrong if the files gets moved around for
some reason, define it in a more central location.
--HG--
extra : rebase_source : 06fa06d47732223e19b0201f8791fdbffdc9ee03
When I refactored hash_paths to add caching, I mixed things up such that
for each file, we end up hashing:
(u'$sha256sum', u'$topsrcdir/$relpath') $relpath
when the intent was to hash:
$sha256sum $relpath
This change fixes it, such that now the index paths are independent of
the source path, as originally intended.
--HG--
extra : rebase_source : 8ff7b49927d2365ed87fa06d8e6fca157faddc7d
This patch allows the use of the flag '--jscov-dir-prefix' for mochitest plain tests to enable code coverage collection with the JS Debugger. It also enables the mochitest-plain tests for the linux64-jsdcov build platform.
MozReview-Commit-ID: 6RqMEZ1I0D7
--HG--
extra : rebase_source : 351754541801f69f7c54807f6bdd3a3d1baf9222
This allows to find them and optimize them out during the taskgraph
optimization phase, and will allow to get toolchain artifacts through a
mach command for developers.
The index path is generated similarly to git trees or mercurial
manifests, and allows to find the right task corresponding to the the
contents of the files in the task `extra.resources` along the toolchain
scripts.
`when.files-changed` is not used when a task has index paths because we
need tasks to happen independently of whether there were changes to
those files when the index or artifacts expire.
--HG--
extra : rebase_source : e9995cee0ee39d7b64090a243e380aeae336a69f
This does slightly change the behavior when artifacts expire, in that
if for some reason the artifact for the task that was found expired,
we don't try to get the artifact from a lower level task. In practice,
that shouldn't be a concern.
--HG--
extra : rebase_source : 8376c2cdec0b4608bce0b41a033d8ed74e7ee63f
The toolchain tasks are hard to spot on treeherder, in the ocean of
build and test jobs associated with the platforms they are currently
under.
Now that we have a significant number of toolchain tasks across
different platforms, it's even worse, especially combined with the fact
that they don't happen on every push.
To make them more easily visible, we move them to a new, separate,
"platform", with the name "toolchains", instead of having them in
different platforms. But since the distinction between Linux, OSX and
Windows 32/64 is still interesting to have, we create groups for each of
those platforms.
But because of bug 1215587, the jobs still end up associated to their
previous group, defeating the new grouping, so to work around that bug,
we also rename the jobs in subtle ways.
--HG--
extra : rebase_source : 6c093c070c18a64eba1c21bf2a2c97b2a9aaabc5
This is a minor refactor that aims to always attempt to set up 'mach' after running the 'run-mozharness' script.
We put it in a 'finally' block so we'll even do this if the user presses Ctrl-C or there was an exception
in the test harness. Importantly, this will be set up regardless of whether the user chooses "Option 1" or
"Option 2" at the wizard prompt.
The reason for this change is mostly 'might as well'. If it can save some users confusion, then it is worthwhile.
MozReview-Commit-ID: Dx3rV17FOoJ
--HG--
extra : rebase_source : 7ef4900409c4b5167bc7a18b27a87f77958e8937
Now that we use the real geckolib and have all dependencies vendored,
the dummy geckolib is no longer required, so we remove it.
Also, the taskgraph code for testing for Servo's presence always
passes and is no longer needed, so we remove it.
Pushed on a CLOSED TREE because ¯\_(ツ)_/¯
MozReview-Commit-ID: ITAqArK4Bks
--HG--
extra : rebase_source : 5eedb3994b679109246b89b0456dd2a59ef3212b
extra : amend_source : b0c97486ae2b72fd21c7968849735e4189e2e86f
We use buildbot-bridge to schedule macosx tests in buildbot, and disable
scheduling on buildbot. Also, schedule a subset of unittests in
taskcluster-worker Tier 3 machines.
MozReview-Commit-ID: 38I33BlUvmt
--HG--
extra : rebase_source : 36347b6fb976f8ec0a90e239ec05ebaedbdf2253
There's not a single well-maintained fork of libdmg-hfsplus, but there are
scattered forks with various fixes. The fork + branch I've chosen here
seems to have collected the most fixes, including a specific fix we need
for repacking DMG files on Linux:
5c92af354b
MozReview-Commit-ID: 3RB6gfgQmCA
--HG--
extra : rebase_source : 40d145852a3876a983f1de7cacbc5ce5e68062a8
Framework for defining actions in-tree that can be displayed
and triggered from Treeherder.
MozReview-Commit-ID: 3rvwgy2i4xu
--HG--
extra : rebase_source : beca394f4337aae4ab149e4db810352f57ec4988
Add specification for actions.json to be used as contract
between in-tree logic and Treeherder. Such that Treeherder
can provide actions that callback into the in-tree logic.
MozReview-Commit-ID: JM1ebU8zNK5
--HG--
extra : rebase_source : 289c7c800f214ccde99adfbdf58bba614b957fe6
The patch, taken from upstream libxcb version 1.11, and applied to the libxcb
1.8 version we use on the 12.04 testers, fixes the "xcb_conn.c:186: write_vec:
Assertion `!c->out.queue_len' failed" error that has been plaguing us in bug
1293474.
The Ubuntu 16.04 testers do not need to be patched becaused they are running
a newer version of libxcb that already has the upstream fix.
MozReview-Commit-ID: AoRNonpK0Dr
--HG--
extra : rebase_source : 6f92e7e8798f94c9fdb665525a2c930e9b652138
Revert the bump to beta for the android builders now that 1.15.0
is in stable release.
Generalize the cargo tarball name mangling to use a regular
expression to hand the tiny revision added for the stable
release, as suggested by mshal in bug 1332759.
MozReview-Commit-ID: 2xZWpvKxY77
--HG--
extra : rebase_source : 40121cae1868d4870a5aa05e3514a75891c3d77c
CLOSED TREE
Backed out changeset 158233bce738 (bug 1197325)
Backed out changeset b5ac3fa0bbe7 (bug 1197325)
Backed out changeset 55a8ad127517 (bug 1197325)
This patch enables the use of the transform 'enable_code_coverage' to set the 'run-on-projects' for all test suites used by linux64-jsdcov. It also removes the occurences of that flag from the test definition yaml.
MozReview-Commit-ID: 66zG9MrFn2i
--HG--
extra : rebase_source : 7d2b16c5c4c0c6d8c931f7f344db745432b92412
Bob Clary
Gregory Szorc
Aki Sasaki
Wes Kocher
Dustin J. Mitchell
Kim Moir
Paul Yang
Carsten "Tomcat" Book
Rob Thijssen
Justin Wood
Hiroyuki Ikezoe
Johan Lorenzo
Joel Maher
Kartikaya Gupta
Bryce Van Dyk
Ting-Yu Chou
Madeleine Chercover
Maja Frydrychowicz
Sebastian Hengst
Mihai Tabara
William Lachance
Steve Fink
Ralph Giles
Jonas Finnemann Jensen
Xidorn Quan
Mitchell Hentges
Geoff Brown
Henrik Skupin
Chris Manchester
Bobby Holley
Rail Aliiev
Mike Hommey
Andrew Halberstadt
Rob Wood
Wander Lairson Costa
Alin Selagea
Brian Stack
Greg Mierzwinski
Ryan VanderMeulen
Hammad Akhtar
Dave Hunt
Shu-yu Guo
Ted Mielczarek
Botond Ballo
Phil Ringnalda