Docker-worker's `command` field is actually not required, as it will run a
docker image's default command when command is not specified.
MozReview-Commit-ID: I3vBHeixlxW
--HG--
extra : rebase_source : a5d02c3131dd6ffb307c37e827d58aa8686ccaf8
A few commits ago, we bumped up the default zstandard compression level
from 3 to 10 when we switched to multi-threaded compression. Even with
multiple threads, this was a bit slower.
For images that will be built once and read multiple times, it is
worthwhile to burn extra CPU once and produce a small image. However,
for other tasks where the number of reads is limited, it isn't
worth it to use this extra CPU. This commit uses the SCM level as
a proxy for "optimize for speed." If the task is associated with level
1 (a try push), we lower the compression level and optimize for
speed. Otherwise, we keep the higher compression level and
optimize for image size.
Credit goes to Jonas for this terrific idea.
MozReview-Commit-ID: Hui97KsZpgw
--HG--
extra : rebase_source : 5a98e554166b51b8caa62b38d82e91c7b9fcb7ab
`task.dependencies` is part of the internal taskgraph calculations, so it must
only refer to tasks that are being created, and not to those which already
exist. `task.task['dependencies']`, on the other hand, is what is given to
`queue.createTask` and expresses all dependencies.
MozReview-Commit-ID: GJ6JVj6JMBz
--HG--
extra : rebase_source : 553aec85134fe0e41c53f917327b13d1b66592aa
This patch enables `run-on-projects` to work appropriately for
nightly builds and tests. Initially, we were setting an empty
`run-on-projects` for nightly `build_platform`s, then explicitly
targeting the platforms in nightly-specific `target_task_method`s.
Instead, this patch enables nightlies to `run-on-projects` everywhere,
but governs the use of nightlies by either the `include_nightly`
parameter, or the `--include-nightly` try option. This lets us filter
nightly-related `target_task_method`s against `run-on-projects` without
losing all nightly tasks.
Then, enable spidermonkey tests by removing optimization from beta and
release. This patch also enables everything then disables specific
tasks, rather than disabling everything and enabling specific tasks.
Since we're beginning with a `filter_for_project` call, we should be
able to reduce these if blocks to zero over time, if desired.
MozReview-Commit-ID: A9tolynaChF
--HG--
extra : rebase_source : 3465ee2c714de3e0359f14109096fc94de27aadf
Graph morphs modify the graph after optimization, without changing its meaning.
In this case, that means adding index tasks that will insert paths into the
index beyond the relatively limited number afforded in task.routes.
MozReview-Commit-ID: AJy4exX7q2v
--HG--
extra : rebase_source : d61e7462defd41e7112739fb057edb493f495430
extra : source : c580568ed47c1ed2af40d98b47fbb0d136e63060
Note that the to_json method prefers the taskgraph's dependencies information
(edges) to that from the task.dependencies entries. At a few points in
task-graph generation, these values differ, although that is expected (for
example, the full task set contains no edges, but that information is still in
task.dependencies). Unifying that representation leads to some difficulty with
task transforms that reach into the dependency tree (beetmover), so the
different representations are left as-is.
MozReview-Commit-ID: GeW8HNwFA9Z
--HG--
extra : rebase_source : 549773e05e18371a399612d9bceccffc29be8cf2
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
The previous attempt at this didn't handle jobs that were keyed by platform,
which was most of them.
MozReview-Commit-ID: IC602td532T
--HG--
extra : rebase_source : 95cdf9ad37df8ef6665665f11e59f8ae8304dbd2
this patch:
- adds linux{32,64}-nightly/opt test platforms that mirror the non-nightly test platforms.
- adds an `include_nightly` per-project parameter; this is refered to in the default `target_task_method`. It's still possible to launch custom `target_task_method`s to trigger nightlies against, say, try.
- adds a `filter_for_project` method in `target_tasks.py` that allows for `include_nightly` and `run_on_projects` filtering in the various `target_task_method`s.
- adds nightly filtering into the `TryOptionSyntax` object. By default, this will be off. To trigger nightly tests on try, either submit a new decision task with a different `target_task_method` (e.g. `nightly_fennec`) or flip the `include_nightly` flag to True.
- adds the `nightly` attribute to tests if their builds have that attribute.
MozReview-Commit-ID: DttIZH0BHS2
--HG--
extra : rebase_source : d8acbe4c741f570b2e8d33a8e6a7f5c791b24ff6
Necessary for treeherder action retriggering code to recognize them as supporting
this action.
MozReview-Commit-ID: BY6OCUFsYlK
--HG--
extra : rebase_source : 6cc4bc8b1cfde29f793fd910bf99f8d3e36603da
* add run.using = 'run-task' for native-engine
* modify run-task to run on OS X
- not as root
- without assuming /home/worker (using ~ and os.expanduser instead)
- hg is in /usr/local/bin on OS X; trust the PATH
* add_build_dependency isn't docker-worker specific, so just rename
* support_vcs_checkout modified to omit caches on native-engine
* don't download fingerprints on OS X; these hosts are configured with
the proper fingerprint via puppet
MozReview-Commit-ID: C83XClXtcn4
--HG--
extra : rebase_source : 2ef1e8dced12ccc4acb7706d7f4587df19a379fc
This fixes the ability to run mozbase via `-j mozbase`, with the
added advantage that it will obey `-p` too.
MozReview-Commit-ID: 1zkitUephXk
--HG--
extra : rebase_source : 0ebb65363d5f5813bc7ccb379768df54310d39c1
This clears up some confusion and undocumented behavior around platforms in
source-tests (and available to any job).
With this change, the attributes come out like this:
"source-test-mozbase-linux64/opt": {
"attributes": {
"build_platform": "linux64",
"build_type": "opt",
"kind": "source-test",
"run_on_projects": [
"integration",
"release"
]
},
MozReview-Commit-ID: HN1Zi8YUf0
--HG--
extra : rebase_source : 552bffc4646a3eec46e7edb508d8eb4d2a8e2e03
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 22382e3d65ce8454a1682cfced0d03477762e8fe
Various modules under taskcluster are doing ad-hoc url formatting or
requests to taskcluster services. While we could use the taskcluster
client python module, it's kind of overkill for the simple requests done
here. So instead of vendoring that module, create a smaller one with
a limited set of functions we need.
This changes the behavior of the get_artifact function to return a
file-like object when the file is neither a json nor a yaml, but that
branch was never used (and was actually returning an unassigned
variable, so it was broken anyways).
At the same time, make the function that does HTTP requests more
error-resistant, using urllib3's Retry with a backoff factor.
Also add a function that retrieves the list of artifacts, that while
currently unused, will be used by `mach artifact` shortly.
--HG--
extra : rebase_source : d7ef633e8e5041dc8450f3ff2f3751c85f144cdc
Instead of every file trying to get the top source directory having an
ad-hoc definition that gets wrong if the files gets moved around for
some reason, define it in a more central location.
--HG--
extra : rebase_source : 1a0cbea267193f6b388b88b36166353e20ac8569
We use buildbot-bridge to schedule macosx tests in buildbot, and disable
scheduling on buildbot. Also, schedule a subset of unittests in
taskcluster-worker Tier 3 machines.
MozReview-Commit-ID: Dbn6U4V2NLp
--HG--
extra : rebase_source : 4f2e5edb4da73975d6634e6313f76a1a9c7651a8
Currently 'run_task' tasks have no easy way to depend on a build task. For example, some
python unittests need a Firefox binary for their tests, like the mozrunner tests and future
test harness selftests (like mochitest tests).
This patch allows kinds to add a new key to the kind config which maps test platforms to
build-labels. Then 'run_task' jobs can add a 'requires-build': true field to get a build
dependency automatically added. The build artifact url will also be stored in the
$GECKO_INSTALLER_URL environment variable on the test host.
MozReview-Commit-ID: Jqyhbj7nC6z
--HG--
extra : rebase_source : 2f44b6c94f35a0d2e11464cf773e821ae6fe8538
The name 'source-check' is a bit of a misnomer, because it already includes a bunch
of tasks that are proper unittests, not lints. Some of these unittests will soon
depend on a build task, which makes 'source-check' feel even more wrong.
They still have a lot in common with the lint tasks though, so it's probably not
worth splitting this into two separate kinds. Instead, let's just rename this kind
to 'source-test', which means, any task that tests stuff and is run from the source
directory (instead of a tests.zip). I think both lints and python-tests fall squarely
under this definition.
MozReview-Commit-ID: K0gZ5rVLyeD
--HG--
rename : taskcluster/ci/source-check/doc.yml => taskcluster/ci/source-test/doc.yml
rename : taskcluster/ci/source-check/kind.yml => taskcluster/ci/source-test/kind.yml
rename : taskcluster/ci/source-check/mozlint.yml => taskcluster/ci/source-test/mozlint.yml
rename : taskcluster/ci/source-check/python-tests.yml => taskcluster/ci/source-test/python-tests.yml
rename : taskcluster/ci/source-check/webidl.yml => taskcluster/ci/source-test/webidl.yml
extra : rebase_source : a683b5b6e243849de57f2681993046c776d6a8f2
This will be used to restrict mochitest actions to mochitest jobs only.
MozReview-Commit-ID: DbFb9V6s9Rb
--HG--
extra : rebase_source : 16ebd751bf7048fd46d71bd350119ca3f8a68302
Various modules under taskcluster are doing ad-hoc url formatting or
requests to taskcluster services. While we could use the taskcluster
client python module, it's kind of overkill for the simple requests done
here. So instead of vendoring that module, create a smaller one with
a limited set of functions we need.
This changes the behavior of the get_artifact function to return a
file-like object when the file is neither a json nor a yaml, but that
branch was never used (and was actually returning an unassigned
variable, so it was broken anyways).
At the same time, make the function that does HTTP requests more
error-resistant, using urllib3's Retry with a backoff factor.
Also add a function that retrieves the list of artifacts, that while
currently unused, will be used by `mach artifact` shortly.
--HG--
extra : rebase_source : 06777dea62e884f546a5b951baad80fd8aec1f1e
Instead of every file trying to get the top source directory having an
ad-hoc definition that gets wrong if the files gets moved around for
some reason, define it in a more central location.
--HG--
extra : rebase_source : 06fa06d47732223e19b0201f8791fdbffdc9ee03
When I refactored hash_paths to add caching, I mixed things up such that
for each file, we end up hashing:
(u'$sha256sum', u'$topsrcdir/$relpath') $relpath
when the intent was to hash:
$sha256sum $relpath
This change fixes it, such that now the index paths are independent of
the source path, as originally intended.
--HG--
extra : rebase_source : 8ff7b49927d2365ed87fa06d8e6fca157faddc7d
This allows to find them and optimize them out during the taskgraph
optimization phase, and will allow to get toolchain artifacts through a
mach command for developers.
The index path is generated similarly to git trees or mercurial
manifests, and allows to find the right task corresponding to the the
contents of the files in the task `extra.resources` along the toolchain
scripts.
`when.files-changed` is not used when a task has index paths because we
need tasks to happen independently of whether there were changes to
those files when the index or artifacts expire.
--HG--
extra : rebase_source : e9995cee0ee39d7b64090a243e380aeae336a69f
This does slightly change the behavior when artifacts expire, in that
if for some reason the artifact for the task that was found expired,
we don't try to get the artifact from a lower level task. In practice,
that shouldn't be a concern.
--HG--
extra : rebase_source : 8376c2cdec0b4608bce0b41a033d8ed74e7ee63f
The toolchain tasks are hard to spot on treeherder, in the ocean of
build and test jobs associated with the platforms they are currently
under.
Now that we have a significant number of toolchain tasks across
different platforms, it's even worse, especially combined with the fact
that they don't happen on every push.
To make them more easily visible, we move them to a new, separate,
"platform", with the name "toolchains", instead of having them in
different platforms. But since the distinction between Linux, OSX and
Windows 32/64 is still interesting to have, we create groups for each of
those platforms.
But because of bug 1215587, the jobs still end up associated to their
previous group, defeating the new grouping, so to work around that bug,
we also rename the jobs in subtle ways.
--HG--
extra : rebase_source : 6c093c070c18a64eba1c21bf2a2c97b2a9aaabc5
Now that we use the real geckolib and have all dependencies vendored,
the dummy geckolib is no longer required, so we remove it.
Also, the taskgraph code for testing for Servo's presence always
passes and is no longer needed, so we remove it.
Pushed on a CLOSED TREE because ¯\_(ツ)_/¯
MozReview-Commit-ID: ITAqArK4Bks
--HG--
extra : rebase_source : 5eedb3994b679109246b89b0456dd2a59ef3212b
extra : amend_source : b0c97486ae2b72fd21c7968849735e4189e2e86f
We use buildbot-bridge to schedule macosx tests in buildbot, and disable
scheduling on buildbot. Also, schedule a subset of unittests in
taskcluster-worker Tier 3 machines.
MozReview-Commit-ID: 38I33BlUvmt
--HG--
extra : rebase_source : 36347b6fb976f8ec0a90e239ec05ebaedbdf2253
Framework for defining actions in-tree that can be displayed
and triggered from Treeherder.
MozReview-Commit-ID: 3rvwgy2i4xu
--HG--
extra : rebase_source : beca394f4337aae4ab149e4db810352f57ec4988
I want to get Servo vendored into servo/. The previous plan was to
replace the dummy geckolib with the real deal when the vendoring is
done. Unfortunately, this will require a significant `cargo vendor`
change, which we want to punt on for a bit.
So, this commit moves our dummy geckolib outside of servo/ so we
don't need to `cargo update` or `cargo vendor` when the real servo/
is installed.
The change to toolkit/library/rust/shared/Cargo.toml can be reverted
in the stylo repo to allow it to use the real geckolib.
We also update the taskgraph code for detecting Servo. Previously,
it looked for a file in the possibly-vendored servo/ directory. Once
the vendoring happens, this check will always pass. But without the
real geckolib, the Servo builds will fail. So, we change the check
to look for the real geckolib. This is implemented a bit hackily.
But it will be short-lived until we run `cargo vendor`.
MozReview-Commit-ID: CxGTwy6bK9j
--HG--
rename : servo/ports/geckolib/Cargo.toml => toolkit/library/geckolib/Cargo.toml
rename : servo/ports/geckolib/lib.rs => toolkit/library/geckolib/lib.rs
extra : rebase_source : c0e9c867ae74c4eb124e72dc481fd8dc814e65e7
-w, --taskcluster-worker: schedule taskcluster-worker jobs.
This is necessary to relieve the load of the small pool of data center
machines.
MozReview-Commit-ID: IiOLHUs2ALi
--HG--
extra : rebase_source : 96acbbd40aef856f68c7ab94ae23bcee4f2f078d
This patch enables the use of the transform 'enable_code_coverage' to set the 'run-on-projects' for all test suites used by linux64-jsdcov. It also removes the occurences of that flag from the test definition yaml.
MozReview-Commit-ID: 66zG9MrFn2i
--HG--
extra : rebase_source : 7d2b16c5c4c0c6d8c931f7f344db745432b92412
This requires moving the schema utilities to their own util module.
MozReview-Commit-ID: KR5xSJ9ak5Y
--HG--
extra : rebase_source : 1c1f6bfb6a08deb8c0be4b2b58db02d85aafeb89
We parse_message needs to return an args object even if try commit
message is empty.
MozReview-Commit-ID: BPWvzjAyjHX
--HG--
extra : rebase_source : 57a699e95e55eabaad3e6a7596086c7f06248e91
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: GPFdi0FD6Vn
--HG--
extra : rebase_source : b992786158851f1099aedfce8669a163228edc51
Considering docker images contents depend very much on the moment they
were built, it is possible that two images with the same hash in the
taskcluster index (at different levels) have different contents. When
this happens, the build or test results could be significantly
different between e.g. try and mozilla-central, possibly leading to
misleading results at landing time.
So if for some reason multiple levels have images for the same hash, the
one used at the highest level should be prefered, such that try uses the
same as mozilla-central once mozilla-central generates the image for
that hash, even if there is an image previously generated for try.
--HG--
extra : rebase_source : 57f593a530da02f9f576872404915c26af544688
Considering docker images contents depend very much on the moment they
were built, it is possible that two images with the same hash in the
taskcluster index (at different levels) have different contents. When
this happens, the build or test results could be significantly
different between e.g. try and mozilla-central, possibly leading to
misleading results at landing time.
So if for some reason multiple levels have images for the same hash, the
one used at the highest level should be prefered, such that try uses the
same as mozilla-central once mozilla-central generates the image for
that hash, even if there is an image previously generated for try.
--HG--
extra : rebase_source : 3f92c1c97a8805cd2d4d6de791863936ed69e8d0
The 20 machines pool running taskcluster-worker is not enough to run
jobs from all branches. Let's limit these jobs to try branch while they
are tier 3.
MozReview-Commit-ID: KmkUjPp7NHL
--HG--
extra : rebase_source : 74303836d9b6550553f1b8a3a3bdbe05b893a573
This bug makes it simpler to add new test suites to linux64-ccov with the help of a new transform and also cleans up the test suite definitions. It also enables reftest, crashtest and jsreftest at the same time.
MozReview-Commit-ID: xCjKhUlfts
--HG--
extra : rebase_source : 6f1b34212913cf5d7f93390d7fd47c179577a604
This will ensure that if the taskgraph module, taskcluster configs related to a task or docker image related
to a task are modified, the task will run.
MozReview-Commit-ID: 1H6LnYsxxpc
--HG--
extra : rebase_source : 51aae8aa445be83f64d0a381cee01a0b5861b41a
Peter Moore
Ting-Yu Chou
Johan Lorenzo
Carsten "Tomcat" Book
Wes Kocher
Aki Sasaki
Mihai Tabara
Kim Moir
Paul Bignier
Dustin J. Mitchell
Gregory Szorc
Rob Thijssen
Joel Maher
Madeleine Chercover
Steve Fink
Sebastian Hengst
William Lachance
Justin Wood
Paul Yang
Chris Manchester
Kartikaya Gupta
Andrew Halberstadt
Mike Hommey
Rob Wood
Wander Lairson Costa
Alin Selagea
Brian Stack
Hammad Akhtar
Ryan VanderMeulen
Jonas Finnemann Jensen
Greg Mierzwinski
Mitchell Hentges
Benjamin Smedberg
Phil Ringnalda