Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
The previous attempt at this didn't handle jobs that were keyed by platform,
which was most of them.
MozReview-Commit-ID: IC602td532T
--HG--
extra : rebase_source : 95cdf9ad37df8ef6665665f11e59f8ae8304dbd2
this patch:
- adds linux{32,64}-nightly/opt test platforms that mirror the non-nightly test platforms.
- adds an `include_nightly` per-project parameter; this is refered to in the default `target_task_method`. It's still possible to launch custom `target_task_method`s to trigger nightlies against, say, try.
- adds a `filter_for_project` method in `target_tasks.py` that allows for `include_nightly` and `run_on_projects` filtering in the various `target_task_method`s.
- adds nightly filtering into the `TryOptionSyntax` object. By default, this will be off. To trigger nightly tests on try, either submit a new decision task with a different `target_task_method` (e.g. `nightly_fennec`) or flip the `include_nightly` flag to True.
- adds the `nightly` attribute to tests if their builds have that attribute.
MozReview-Commit-ID: DttIZH0BHS2
--HG--
extra : rebase_source : d8acbe4c741f570b2e8d33a8e6a7f5c791b24ff6
Necessary for treeherder action retriggering code to recognize them as supporting
this action.
MozReview-Commit-ID: BY6OCUFsYlK
--HG--
extra : rebase_source : 6cc4bc8b1cfde29f793fd910bf99f8d3e36603da
* add run.using = 'run-task' for native-engine
* modify run-task to run on OS X
- not as root
- without assuming /home/worker (using ~ and os.expanduser instead)
- hg is in /usr/local/bin on OS X; trust the PATH
* add_build_dependency isn't docker-worker specific, so just rename
* support_vcs_checkout modified to omit caches on native-engine
* don't download fingerprints on OS X; these hosts are configured with
the proper fingerprint via puppet
MozReview-Commit-ID: C83XClXtcn4
--HG--
extra : rebase_source : 2ef1e8dced12ccc4acb7706d7f4587df19a379fc
This fixes the ability to run mozbase via `-j mozbase`, with the
added advantage that it will obey `-p` too.
MozReview-Commit-ID: 1zkitUephXk
--HG--
extra : rebase_source : 0ebb65363d5f5813bc7ccb379768df54310d39c1
This clears up some confusion and undocumented behavior around platforms in
source-tests (and available to any job).
With this change, the attributes come out like this:
"source-test-mozbase-linux64/opt": {
"attributes": {
"build_platform": "linux64",
"build_type": "opt",
"kind": "source-test",
"run_on_projects": [
"integration",
"release"
]
},
MozReview-Commit-ID: HN1Zi8YUf0
--HG--
extra : rebase_source : 552bffc4646a3eec46e7edb508d8eb4d2a8e2e03
* This adds a kind: 'task' property to actions in actions.json
* This is already specified in the schema for actions.json
MozReview-Commit-ID: JmlAZouVPIf
--HG--
extra : rebase_source : 5855b9ddbfff2faa9e1db12fc1f60bbef3cccabb
Bug 977849 added many new jsreftests on Android, resulting in more than
a 3x increase in some job run times. This significant addition of jobs
is required to return Android jsreftest job times to 'normal'.
Various modules under taskcluster are doing ad-hoc url formatting or
requests to taskcluster services. While we could use the taskcluster
client python module, it's kind of overkill for the simple requests done
here. So instead of vendoring that module, create a smaller one with
a limited set of functions we need.
This changes the behavior of the get_artifact function to return a
file-like object when the file is neither a json nor a yaml, but that
branch was never used (and was actually returning an unassigned
variable, so it was broken anyways).
At the same time, make the function that does HTTP requests more
error-resistant, using urllib3's Retry with a backoff factor.
Also add a function that retrieves the list of artifacts, that while
currently unused, will be used by `mach artifact` shortly.
--HG--
extra : rebase_source : d7ef633e8e5041dc8450f3ff2f3751c85f144cdc
Instead of every file trying to get the top source directory having an
ad-hoc definition that gets wrong if the files gets moved around for
some reason, define it in a more central location.
--HG--
extra : rebase_source : 1a0cbea267193f6b388b88b36166353e20ac8569
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 22382e3d65ce8454a1682cfced0d03477762e8fe
Disable non-tests for the following tests as the builder machines have
been turned off:
* web-platform-tests-reftests
* mochitest-webgl
* jsreftest
* reftest
MozReview-Commit-ID: 5KZ3KzT3dLo
--HG--
extra : rebase_source : 110ce9bee70adc2ad314b8927385b851f1ce67b7
Currently 'run_task' tasks have no easy way to depend on a build task. For example, some
python unittests need a Firefox binary for their tests, like the mozrunner tests and future
test harness selftests (like mochitest tests).
This patch allows kinds to add a new key to the kind config which maps test platforms to
build-labels. Then 'run_task' jobs can add a 'requires-build': true field to get a build
dependency automatically added. The build artifact url will also be stored in the
$GECKO_INSTALLER_URL environment variable on the test host.
MozReview-Commit-ID: Jqyhbj7nC6z
--HG--
extra : rebase_source : 2f44b6c94f35a0d2e11464cf773e821ae6fe8538
The name 'source-check' is a bit of a misnomer, because it already includes a bunch
of tasks that are proper unittests, not lints. Some of these unittests will soon
depend on a build task, which makes 'source-check' feel even more wrong.
They still have a lot in common with the lint tasks though, so it's probably not
worth splitting this into two separate kinds. Instead, let's just rename this kind
to 'source-test', which means, any task that tests stuff and is run from the source
directory (instead of a tests.zip). I think both lints and python-tests fall squarely
under this definition.
MozReview-Commit-ID: K0gZ5rVLyeD
--HG--
rename : taskcluster/ci/source-check/doc.yml => taskcluster/ci/source-test/doc.yml
rename : taskcluster/ci/source-check/kind.yml => taskcluster/ci/source-test/kind.yml
rename : taskcluster/ci/source-check/mozlint.yml => taskcluster/ci/source-test/mozlint.yml
rename : taskcluster/ci/source-check/python-tests.yml => taskcluster/ci/source-test/python-tests.yml
rename : taskcluster/ci/source-check/webidl.yml => taskcluster/ci/source-test/webidl.yml
extra : rebase_source : a683b5b6e243849de57f2681993046c776d6a8f2
This will be used to restrict mochitest actions to mochitest jobs only.
MozReview-Commit-ID: DbFb9V6s9Rb
--HG--
extra : rebase_source : 16ebd751bf7048fd46d71bd350119ca3f8a68302
Various modules under taskcluster are doing ad-hoc url formatting or
requests to taskcluster services. While we could use the taskcluster
client python module, it's kind of overkill for the simple requests done
here. So instead of vendoring that module, create a smaller one with
a limited set of functions we need.
This changes the behavior of the get_artifact function to return a
file-like object when the file is neither a json nor a yaml, but that
branch was never used (and was actually returning an unassigned
variable, so it was broken anyways).
At the same time, make the function that does HTTP requests more
error-resistant, using urllib3's Retry with a backoff factor.
Also add a function that retrieves the list of artifacts, that while
currently unused, will be used by `mach artifact` shortly.
--HG--
extra : rebase_source : 06777dea62e884f546a5b951baad80fd8aec1f1e
Instead of every file trying to get the top source directory having an
ad-hoc definition that gets wrong if the files gets moved around for
some reason, define it in a more central location.
--HG--
extra : rebase_source : 06fa06d47732223e19b0201f8791fdbffdc9ee03
When I refactored hash_paths to add caching, I mixed things up such that
for each file, we end up hashing:
(u'$sha256sum', u'$topsrcdir/$relpath') $relpath
when the intent was to hash:
$sha256sum $relpath
This change fixes it, such that now the index paths are independent of
the source path, as originally intended.
--HG--
extra : rebase_source : 8ff7b49927d2365ed87fa06d8e6fca157faddc7d
This patch allows the use of the flag '--jscov-dir-prefix' for mochitest plain tests to enable code coverage collection with the JS Debugger. It also enables the mochitest-plain tests for the linux64-jsdcov build platform.
MozReview-Commit-ID: 6RqMEZ1I0D7
--HG--
extra : rebase_source : 351754541801f69f7c54807f6bdd3a3d1baf9222
This allows to find them and optimize them out during the taskgraph
optimization phase, and will allow to get toolchain artifacts through a
mach command for developers.
The index path is generated similarly to git trees or mercurial
manifests, and allows to find the right task corresponding to the the
contents of the files in the task `extra.resources` along the toolchain
scripts.
`when.files-changed` is not used when a task has index paths because we
need tasks to happen independently of whether there were changes to
those files when the index or artifacts expire.
--HG--
extra : rebase_source : e9995cee0ee39d7b64090a243e380aeae336a69f
This does slightly change the behavior when artifacts expire, in that
if for some reason the artifact for the task that was found expired,
we don't try to get the artifact from a lower level task. In practice,
that shouldn't be a concern.
--HG--
extra : rebase_source : 8376c2cdec0b4608bce0b41a033d8ed74e7ee63f
The toolchain tasks are hard to spot on treeherder, in the ocean of
build and test jobs associated with the platforms they are currently
under.
Now that we have a significant number of toolchain tasks across
different platforms, it's even worse, especially combined with the fact
that they don't happen on every push.
To make them more easily visible, we move them to a new, separate,
"platform", with the name "toolchains", instead of having them in
different platforms. But since the distinction between Linux, OSX and
Windows 32/64 is still interesting to have, we create groups for each of
those platforms.
But because of bug 1215587, the jobs still end up associated to their
previous group, defeating the new grouping, so to work around that bug,
we also rename the jobs in subtle ways.
--HG--
extra : rebase_source : 6c093c070c18a64eba1c21bf2a2c97b2a9aaabc5
This is a minor refactor that aims to always attempt to set up 'mach' after running the 'run-mozharness' script.
We put it in a 'finally' block so we'll even do this if the user presses Ctrl-C or there was an exception
in the test harness. Importantly, this will be set up regardless of whether the user chooses "Option 1" or
"Option 2" at the wizard prompt.
The reason for this change is mostly 'might as well'. If it can save some users confusion, then it is worthwhile.
MozReview-Commit-ID: Dx3rV17FOoJ
--HG--
extra : rebase_source : 7ef4900409c4b5167bc7a18b27a87f77958e8937
Now that we use the real geckolib and have all dependencies vendored,
the dummy geckolib is no longer required, so we remove it.
Also, the taskgraph code for testing for Servo's presence always
passes and is no longer needed, so we remove it.
Pushed on a CLOSED TREE because ¯\_(ツ)_/¯
MozReview-Commit-ID: ITAqArK4Bks
--HG--
extra : rebase_source : 5eedb3994b679109246b89b0456dd2a59ef3212b
extra : amend_source : b0c97486ae2b72fd21c7968849735e4189e2e86f
We use buildbot-bridge to schedule macosx tests in buildbot, and disable
scheduling on buildbot. Also, schedule a subset of unittests in
taskcluster-worker Tier 3 machines.
MozReview-Commit-ID: 38I33BlUvmt
--HG--
extra : rebase_source : 36347b6fb976f8ec0a90e239ec05ebaedbdf2253
There's not a single well-maintained fork of libdmg-hfsplus, but there are
scattered forks with various fixes. The fork + branch I've chosen here
seems to have collected the most fixes, including a specific fix we need
for repacking DMG files on Linux:
5c92af354b
MozReview-Commit-ID: 3RB6gfgQmCA
--HG--
extra : rebase_source : 40d145852a3876a983f1de7cacbc5ce5e68062a8
Framework for defining actions in-tree that can be displayed
and triggered from Treeherder.
MozReview-Commit-ID: 3rvwgy2i4xu
--HG--
extra : rebase_source : beca394f4337aae4ab149e4db810352f57ec4988
Add specification for actions.json to be used as contract
between in-tree logic and Treeherder. Such that Treeherder
can provide actions that callback into the in-tree logic.
MozReview-Commit-ID: JM1ebU8zNK5
--HG--
extra : rebase_source : 289c7c800f214ccde99adfbdf58bba614b957fe6
The patch, taken from upstream libxcb version 1.11, and applied to the libxcb
1.8 version we use on the 12.04 testers, fixes the "xcb_conn.c:186: write_vec:
Assertion `!c->out.queue_len' failed" error that has been plaguing us in bug
1293474.
The Ubuntu 16.04 testers do not need to be patched becaused they are running
a newer version of libxcb that already has the upstream fix.
MozReview-Commit-ID: AoRNonpK0Dr
--HG--
extra : rebase_source : 6f92e7e8798f94c9fdb665525a2c930e9b652138
Revert the bump to beta for the android builders now that 1.15.0
is in stable release.
Generalize the cargo tarball name mangling to use a regular
expression to hand the tiny revision added for the stable
release, as suggested by mshal in bug 1332759.
MozReview-Commit-ID: 2xZWpvKxY77
--HG--
extra : rebase_source : 40121cae1868d4870a5aa05e3514a75891c3d77c
CLOSED TREE
Backed out changeset 158233bce738 (bug 1197325)
Backed out changeset b5ac3fa0bbe7 (bug 1197325)
Backed out changeset 55a8ad127517 (bug 1197325)
This patch enables the use of the transform 'enable_code_coverage' to set the 'run-on-projects' for all test suites used by linux64-jsdcov. It also removes the occurences of that flag from the test definition yaml.
MozReview-Commit-ID: 66zG9MrFn2i
--HG--
extra : rebase_source : 7d2b16c5c4c0c6d8c931f7f344db745432b92412
This patch makes it possible to collect code coverage for xpcshell tests using the linux64-jsdcov build. It also enables the use of a 'coverage' flag to disable tests when they are instrumented with the js debugger for code coverage. Lastly, it uses the 'coverage' flag to disable certain tests.
MozReview-Commit-ID: 97VFkJmlwQn
--HG--
extra : rebase_source : 26c841f5a68f927889c0903e701bfde4b7ca84ac
I want to get Servo vendored into servo/. The previous plan was to
replace the dummy geckolib with the real deal when the vendoring is
done. Unfortunately, this will require a significant `cargo vendor`
change, which we want to punt on for a bit.
So, this commit moves our dummy geckolib outside of servo/ so we
don't need to `cargo update` or `cargo vendor` when the real servo/
is installed.
The change to toolkit/library/rust/shared/Cargo.toml can be reverted
in the stylo repo to allow it to use the real geckolib.
We also update the taskgraph code for detecting Servo. Previously,
it looked for a file in the possibly-vendored servo/ directory. Once
the vendoring happens, this check will always pass. But without the
real geckolib, the Servo builds will fail. So, we change the check
to look for the real geckolib. This is implemented a bit hackily.
But it will be short-lived until we run `cargo vendor`.
MozReview-Commit-ID: CxGTwy6bK9j
--HG--
rename : servo/ports/geckolib/Cargo.toml => toolkit/library/geckolib/Cargo.toml
rename : servo/ports/geckolib/lib.rs => toolkit/library/geckolib/lib.rs
extra : rebase_source : c0e9c867ae74c4eb124e72dc481fd8dc814e65e7
-w, --taskcluster-worker: schedule taskcluster-worker jobs.
This is necessary to relieve the load of the small pool of data center
machines.
MozReview-Commit-ID: IiOLHUs2ALi
--HG--
extra : rebase_source : 96acbbd40aef856f68c7ab94ae23bcee4f2f078d
This test suite involves a bunch of python tests which don't require
configure or any generated binaries to run. We can split them out into a
Taskcluster linter type task to run directly on the source tree in
parallel with builds.
MozReview-Commit-ID: 9ux3rAuFXAY
--HG--
extra : rebase_source : 95378cd686644e34ea017c682d7384906b17d13a
This patch has a few parts to make this work:
1) more tests pass on ubuntu 16, so remove old fail-if conditions
2) no support for GL_ARB_gpu_shader5, we cherry pick from updated ANGLE code
3) disable test_capture.html as it leaks on ASAN
MozReview-Commit-ID: BSSiTFvF9jN
Previously the run-wizard script would add a command to source the virtualenv in ~/.bashrc after
mozharness finished setting things up. This is fragile, assumes people are using bash, etc. Plus
it appeared to intermittently fail for some users.
Instead, this activates the virtualenv directly from individual mach commands that need it. This
guarantees we will always be using the virtualenv if required (and won't be using it if not). The
'activate_this.py' script is invoked the same way that we do it for in-tree mach commands:
https://dxr.mozilla.org/mozilla-central/rev/9c06e744b1befb3a2e2fdac7414ce18220774a1d/python/mozbuild/mozbuild/virtualenv.py#456
MozReview-Commit-ID: CfcoiVJXQTl
--HG--
extra : rebase_source : da01d1ce1bd9b41c89922e989f857c4de8c09341
Rename the base external media tests from "external-media-tests" to
"external-media-tests-base", to disambiguate them from the other tests in the
suite, as well as the set of all external media tests.
Adjust tests that are run per platform so that all external media tests are run
on linux and windows. Adjust the platform tests so that they select the
external-media-tests set, rather than adding individual external media tests to
each platform set.
MozReview-Commit-ID: DRiQQA1BB9n
--HG--
extra : rebase_source : 88b46930bc1fe09062ebcb445e75d1131a2f0116
Considering docker images contents depend very much on the moment they
were built, it is possible that two images with the same hash in the
taskcluster index (at different levels) have different contents. When
this happens, the build or test results could be significantly
different between e.g. try and mozilla-central, possibly leading to
misleading results at landing time.
So if for some reason multiple levels have images for the same hash, the
one used at the highest level should be prefered, such that try uses the
same as mozilla-central once mozilla-central generates the image for
that hash, even if there is an image previously generated for try.
--HG--
extra : rebase_source : 57f593a530da02f9f576872404915c26af544688
Considering docker images contents depend very much on the moment they
were built, it is possible that two images with the same hash in the
taskcluster index (at different levels) have different contents. When
this happens, the build or test results could be significantly
different between e.g. try and mozilla-central, possibly leading to
misleading results at landing time.
So if for some reason multiple levels have images for the same hash, the
one used at the highest level should be prefered, such that try uses the
same as mozilla-central once mozilla-central generates the image for
that hash, even if there is an image previously generated for try.
--HG--
extra : rebase_source : 3f92c1c97a8805cd2d4d6de791863936ed69e8d0
For correct platform name in TH, the buildername must contain the pgo
string in the buildername. We work a bit more generic by matching any
build variant.
Bump the Android builders to the latest beta release to reduce
the variance when we update Firefox 53 to 1.15.0 stable early
in the Aurora phase.
Android builds were moved to 1.15 early to address a code generation
issue with devices without neon.
Work around an issue with tarball naming in the cargo packages.
MozReview-Commit-ID: KQfkWmXV9hQ
--HG--
extra : rebase_source : 9448e0b948740fc3905ef70c8df316dc7342d52e
The 20 machines pool running taskcluster-worker is not enough to run
jobs from all branches. Let's limit these jobs to try branch while they
are tier 3.
MozReview-Commit-ID: KmkUjPp7NHL
--HG--
extra : rebase_source : 74303836d9b6550553f1b8a3a3bdbe05b893a573
This bug makes it simpler to add new test suites to linux64-ccov with the help of a new transform and also cleans up the test suite definitions. It also enables reftest, crashtest and jsreftest at the same time.
MozReview-Commit-ID: xCjKhUlfts
--HG--
extra : rebase_source : 6f1b34212913cf5d7f93390d7fd47c179577a604
This will ensure that if the taskgraph module, taskcluster configs related to a task or docker image related
to a task are modified, the task will run.
MozReview-Commit-ID: 1H6LnYsxxpc
--HG--
extra : rebase_source : 51aae8aa445be83f64d0a381cee01a0b5861b41a
This requires moving the schema utilities to their own util module.
MozReview-Commit-ID: KR5xSJ9ak5Y
--HG--
extra : rebase_source : 1c1f6bfb6a08deb8c0be4b2b58db02d85aafeb89
We parse_message needs to return an args object even if try commit
message is empty.
MozReview-Commit-ID: BPWvzjAyjHX
--HG--
extra : rebase_source : 57a699e95e55eabaad3e6a7596086c7f06248e91
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: GPFdi0FD6Vn
--HG--
extra : rebase_source : b992786158851f1099aedfce8669a163228edc51
This also adds a new 'py' group which should help make it obvious which tasks are python
unittests. The tg and MH tasks won't show up in this new py group right away due to a
caching bug in the treeherder database, but eventually they should move over.
MozReview-Commit-ID: JwL1NAR4bnY
--HG--
extra : rebase_source : 62cc240f490cf1a79021a5b0b0eae8121410ac8a
Dustin J. Mitchell
William Lachance
Xidorn Quan
Aki Sasaki
Jonas Finnemann Jensen
Mitchell Hentges
Kim Moir
Geoff Brown
Joel Maher
Wes Kocher
Henrik Skupin
Carsten "Tomcat" Book
Chris Manchester
Bryce Van Dyk
Kartikaya Gupta
Bobby Holley
Rail Aliiev
Mike Hommey
Andrew Halberstadt
Mihai Tabara
Rob Wood
Wander Lairson Costa
Alin Selagea
Brian Stack
Johan Lorenzo
Greg Mierzwinski
Ryan VanderMeulen
Hammad Akhtar
Dave Hunt
pyang
Gregory Szorc
Sebastian Hengst
Madeleine Chercover
Shu-yu Guo
Ted Mielczarek
Botond Ballo
Ralph Giles
Justin Wood
Phil Ringnalda
Ehsan Akhgari
Mike Shal
Benjamin Smedberg
Matthew Noorenberghe