This provides an easy way to encode an artifact URL in static data such as
taskcluster/ci/nightly-l10n/kind.yml, without knowing in advance the format of
the URL.
--HG--
extra : rebase_source : 5b5229d96aad2916b1b3c8e72045c1d461fc1c02
extra : source : 9c35dd209c6b407bc3a45ce7b4c27272ef1bb486
This provides an easy way to encode an artifact URL in static data such as
taskcluster/ci/nightly-l10n/kind.yml, without knowing in advance the format of
the URL.
--HG--
extra : rebase_source : 25c99d392e9b71c514f236379a816fae971e161a
This provides an easy way to encode an artifact URL in static data such as
taskcluster/ci/nightly-l10n/kind.yml. This could be used in
mozharness_test.py, for example, as well -- but other code (such as to support
backfilling) expects `task-reference` there. To avoid breaking such subtle
bits, those can continue using `task-reference` with URLs generated based on
TASKCLUSTER_ROOT_URL.
Differential Revision: https://phabricator.services.mozilla.com/D14197
--HG--
extra : moz-landing-system : lando
Assorted fixes from trawling the sphinx logs - malformed formatting, broken references, leftovers from renaming action-task to action-callback and removing
yaml-templates, docstring fixes to make sphinx happier, and typos.
MozReview-Commit-ID: 6jUOljdLoE2
--HG--
extra : rebase_source : f2a9dbcde5180f760a80f47691a70eba76e58bad
The runnable jobs API on Treeherder is timing out quite frequently.
This is due that it downloads the full-task-graph.json artifact from the Gecko decision task.
This file has grown to be massive (over 30MB) and that takes too long to fetch. The API times out after 20 seconds.
The data we need from the artifact is minimal. We can generate a file with just the minimum amount of data needed.
This code change adds logic to generate a runnable-jobs.json.gz file after the full-task-graph.json artifact is generated.
MozReview-Commit-ID: 9u2H7HbUAcl
--HG--
extra : rebase_source : f450527e7dd565e29cfe2e04dc52761e77827cc0
This allows firefox and thunderbird builds to avoid using each others bits.
MozReview-Commit-ID: KYQYDd2tkGj
--HG--
extra : rebase_source : 42f1d13ec609f066cb3bd3050ed894296b72d982
Per IRC dicussion, there is no technical or policy restriction on dependencies
within a task kind. Update the documentation to remove the out-of-date mention
of this limitation. In particular, toolchain build tasks tend to depend on
each other.
MozReview-Commit-ID: K6p0mxyjcvY
--HG--
extra : rebase_source : 9439503e8f0a278b1bb41d8a71a2d31d5fc9c0e8
* eliminate heading for test kinds, of which there is now only one
* make the caches document have a single heading in the TOC
* break out mach commands into a separate document, add ./mach taskgraph morphed
* remove docs for YAML templates support (the .yml file wasn't actually
used -- I expect it was a merge leftover); these are still used for actions.yml,
but once that is gone the code should be removed, too.
* break try out into its own document, edit to distinguish "how to run try"
from "how to generate config"
MozReview-Commit-ID: 76ZopWA9TPL
--HG--
extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
Graph morphs modify the graph after optimization, without changing its meaning.
In this case, that means adding index tasks that will insert paths into the
index beyond the relatively limited number afforded in task.routes.
MozReview-Commit-ID: AJy4exX7q2v
--HG--
extra : rebase_source : d61e7462defd41e7112739fb057edb493f495430
extra : source : c580568ed47c1ed2af40d98b47fbb0d136e63060
Note that the to_json method prefers the taskgraph's dependencies information
(edges) to that from the task.dependencies entries. At a few points in
task-graph generation, these values differ, although that is expected (for
example, the full task set contains no edges, but that information is still in
task.dependencies). Unifying that representation leads to some difficulty with
task transforms that reach into the dependency tree (beetmover), so the
different representations are left as-is.
MozReview-Commit-ID: GeW8HNwFA9Z
--HG--
extra : rebase_source : 549773e05e18371a399612d9bceccffc29be8cf2
Previously, we ran a single "target task" function to mutate the full
task graph into a subset based on input parameters (try syntax,
repository being built for, etc). This concept is useful. But
the implementation was limiting because we could only have a single
"target tasks" function.
This commit introduces the concept of "filters." They conceptually
do the same thing as "target tasks methods" but you can run more than
1 of them.
Filters are simply functions that examine an input graph+parameters
and emit nodes that should be retained. Filters, like target tasks
methods, are defined via decorated functions in a module.
TaskGraphGenerator has been converted to use filters. The list of
defined filters can be defined in the parameters dict passed into
TaskGraphGenerator. A default filter list is provided in decision.py.
The intent is to eventually convert target tasks to filters. Until
that happens, we always run the registered target tasks method via
a filter proxy function.
No new tests have been added because we don't yet have any
functionality relying explicitly on filters. Tests will be added in
a subsequent commit once we add a new filter.
While I was here, I also snuck in some logging on the size of the
graphs.
MozReview-Commit-ID: ERn2hIYbMRp
--HG--
extra : rebase_source : 36b8e86aa64b2f52b03b31b5497759b0009fb921
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
This enables kinds that generate tasks based on those output by another kind.
For example, the test kind might generate a set of test tasks for each build
task.
MozReview-Commit-ID: K7ha9OmJ6gd
--HG--
extra : rebase_source : 36fc7e2d9c5987a4bb8b3779cf1a9308f5561828
extra : intermediate-source : 7898d1ab1afc08f78445165d0c94566b0682a2f7
extra : source : 0852b38cd86c42ebba0f9e74d7470a263969b784
This enables kinds that generate tasks based on those output by another kind.
For example, the test kind might generate a set of test tasks for each build
task.
MozReview-Commit-ID: K7ha9OmJ6gd
--HG--
extra : source : 0852b38cd86c42ebba0f9e74d7470a263969b784
extra : amend_source : f3e8c306afe29ae75bd1f93d8b76ff2b27ad8ed1
extra : histedit_source : aa1ae93aba51025a0e1bd2ecf473aaa33235e4c7%2C2c704328e983a3d75a834b069431e4f166389b02
The JSON output is suitable for processing with `jq` to extract features of
interest.
MozReview-Commit-ID: 5wpV7sXlOz3
--HG--
extra : rebase_source : 4ffb78ab7a85b32e64d10218a4a8841c22e689f8
* Implement & document optimization (although legacy kind doesn't do much of it)
* Introduce `optimize_target_tasks` parameter to control whether tasks in the
target set can be optimized (no for try, yes for most other branches)
* Refactor to include resolved taskIds in the optimized task graph
* Include a `label-to-taskid.json` artifact.
* Introduce {'task-reference': '... <dependency-name> ...'} for referring to
parent tasks' taskId.
MozReview-Commit-ID: LWvlWNz49U5
--HG--
extra : rebase_source : 780e0e23d24b268ade33ecdcbccb5081f32aac48
The `taskgraph` package generates TaskCluster task graphs based on collections
of task "kinds". Initially, there is only one kind, the "legacy" kind, which
reads the YAML files from `testing/taskcluster/tasks` to generate the task
graph.
Try syntax is implemented by filtering the tasks in the taskgraph after it has
been created, then extending the result to include any prerequisite tasks.
A collection of `mach taskgraph` subcommands are provided for developers to
extend or debug the task-graph generation process.
MozReview-Commit-ID: 1TJCns4XxZ8
--HG--
rename : testing/taskcluster/docs/index.rst => taskcluster/docs/index.rst
extra : rebase_source : 7b9125281d66044db9bd8e4a1fade16136f384b9
extra : histedit_source : 47640d27080acda0279270babbcf33f5badb0d1c
Bastien Abadie
Dustin J. Mitchell
Sebastian Hengst
Margareta Eliza Balazs
Nick Thomas
Sylvestre Ledru
Tom Prince
Armen Zambrano G.
Ralph Giles
Simon Fraser
Peter Moore
Paul Bignier
Justin Wood
Gregory Szorc
Kalpesh Krishna
Ryan VanderMeulen
Carsten "Tomcat" Book
Wes Kocher