This allows users to set TASKGRAPH_OPTIMIZE_STRATEGIES to a
python_path.find_object string. E.g:
TASKGRAPH_OPTIMIZE_STRATEGIES="module:strategies" ./mach taskgraph optimized
This opens the door to swap in external strategies at runtime and will be
used for back testing experimental strategies.
Differential Revision: https://phabricator.services.mozilla.com/D33203
--HG--
extra : moz-landing-system : lando
`mach artifact toolchain` gets task definitions from taskgraph, to get the
index path to find the artifacts at. Now that these index paths depend on the
digests of fetch tasks, those kinds need to be loaded as well. This adds a
supported API to get task definitions for a given kind, which loads all the
kind dependencies.
Differential Revision: https://phabricator.services.mozilla.com/D12161
--HG--
extra : moz-landing-system : lando
This makes it easier to see what failed, rather than have to guess based on
ordering.
Differential Revision: https://phabricator.services.mozilla.com/D9848
--HG--
extra : moz-landing-system : lando
yaml.load() can evaluate arbitrary Python code via syntax such as
`!!python/object/apply:os.system`. Seriously.
Let's switch taskgraph to yaml.safe_load(), which is reasonable
about limiting magic.
Differential Revision: https://phabricator.services.mozilla.com/D1736
This fixes the fallout from the mach command not being updated for 69c60c8fc528d8d038e22a2ae62f5ff7b1131231.
MozReview-Commit-ID: 7WZ8wW4gnCY
--HG--
extra : rebase_source : 36caffb4a8631ea478ec6f8d509a82c01016eeb5
This fixes the fallout from the mach command not being updated for 69c60c8fc528d8d038e22a2ae62f5ff7b1131231.
MozReview-Commit-ID: 7WZ8wW4gnCY
--HG--
extra : rebase_source : 887b386480b9cb69c8b7e0196851862fceec2dfe
Tasks that have the 'always_target' attribute set will be always be included
in the target_task_graph, regardless of target task filtering.
Furthermore, if they were only added because of this attribute (i.e, the
filters would have excluded the task), then the task will be a candidate for
optimization even if the 'optimize_target_tasks' parameter is False.
MozReview-Commit-ID: 9eoVJ5qpAMO
--HG--
extra : rebase_source : 9635002720d088ca9870649f3143d6293c666610
This is needed to prevent parameter mismatch errors when using |mach try fuzzy|
from an older revision. This can happen if the parameters.yml is being
downloaded from a commit with a recently added parameter.
MozReview-Commit-ID: 4NxCM7i8B4W
--HG--
extra : rebase_source : c47de38ad295e14c80c99806ea430fa641ae2be6
This is needed to prevent parameter mismatch errors when using |mach try fuzzy|
from an older revision. This can happen if the parameters.yml is being
downloaded from a commit with a recently added parameter.
MozReview-Commit-ID: 4NxCM7i8B4W
--HG--
extra : rebase_source : 4d2052aae33292fbd7928a79bfedba76426206b9
Add a release promotion custom action for releng's TC release promotion migration work.
This action generates a graph dependent on previously built tasks. To track these, we add the `do_not_optimize` and `existing_tasks` parameters. The `do_not_optimize` parameter specifies tasks that we want to explicitly exclude from taskgraph optimization. The `existing_tasks` parameter specifies a label-to-taskid map for tasks from previous graphs.
MozReview-Commit-ID: 1vKrNUavM4V
--HG--
extra : rebase_source : b8ba95d270aafe1464c2b3bfc318b9568500a7a1
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : 25e9966696d78d899783d9f38533d5ae66f9ccb9
extra : source : b53ff084c2d7968a1d9864d1343f2d9381fb652b
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
--HG--
extra : rebase_source : ba0aa536e8c474b36c63d1447c83ed9885f1e3e6
extra : source : a3b7bdfdb116300daa3f49e0dfc96177e1369440
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : 03a10610aa3337269fe76a1196bb9b1665e1ab20
extra : source : b53ff084c2d7968a1d9864d1343f2d9381fb652b
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
--HG--
extra : rebase_source : d5bce42fc0ea24616d885eed62e5e5a42b4fce24
extra : source : a3b7bdfdb116300daa3f49e0dfc96177e1369440
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : fdd8c3cfb9abf759a3c43c5713e62e4772c5bd06
It is not at *all* clear how multiple optimizations for a single task should
interact. No simple logical operation is right in all cases, and in fact in
most imaginable cases the desired behavior turns out to be independent of all
but one of the optimizations. For example, given both `seta` and
`skip-unless-files-changed` optimizations, if SETA says to skip a test, it is
low value and should be skipped regardless of what files have changed. But if
SETA says to run a test, then it has likely been skipped in previous pushes, so
it should be run regardless of what has changed in this push.
This also adds a bit more output about optimization, that may be useful for
anyone wondering why a particular job didn't run.
MozReview-Commit-ID: 3OsvRnWjai4
--HG--
extra : rebase_source : 4ac6a5cc592f4210918c73e667f3b5dd50230894
This provides a mechanism to modify the behaviour of tasks from a try push. The try_task_config.json
looks something like:
{
"tasks": ["build-linux64/opt", "test-linux64/opt-mochitest-e10s-1"],
"templates": {
"artifact": {"enabled": 1}
}
}
This tells taskgraph to apply the 'artifact' template to all tasks. Templates are JSONe based
.yml files that live under taskcluster/taskgraph/templates. Taskgraph will render every template
against every task definition. The templates themselves can then use JSONe condition statements to
filter out which tasks they should or shouldn't apply to.
MozReview-Commit-ID: J8HVZzOt4mX
--HG--
extra : rebase_source : 95a78bc56d3f90ff1b34aabd84ed92aff1e3d954
Graph morphs modify the graph after optimization, without changing its meaning.
In this case, that means adding index tasks that will insert paths into the
index beyond the relatively limited number afforded in task.routes.
MozReview-Commit-ID: AJy4exX7q2v
--HG--
extra : rebase_source : d61e7462defd41e7112739fb057edb493f495430
extra : source : c580568ed47c1ed2af40d98b47fbb0d136e63060
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: GPFdi0FD6Vn
--HG--
extra : rebase_source : b992786158851f1099aedfce8669a163228edc51
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: DMwRjuV2vpf
--HG--
extra : rebase_source : 211ecf52694078986caf290c5b0cca35c775da61
Andrew Halberstadt
Bastien Abadie
Tom Prince
Gurzau Raul
Gregory Szorc
Chris AtLee
Coroiu Cristina
Sebastian Hengst
Aki Sasaki
Dustin J. Mitchell
Phil Ringnalda
Ryan VanderMeulen
Mike Hommey
Wander Lairson Costa