We only use `include_nigthly` where we are also using
`filter_beta_release_tasks`, so just change the later to include nightly.
Differential Revision: https://phabricator.services.mozilla.com/D9687
--HG--
extra : moz-landing-system : lando
This adds a transform which will evaluate the 'serviceworker-e10s' key. If
False, the task is returned as is. If True, the task is run with the
'dom.serviceWorkers.parent_intercept' pref enabled. If 'both' the task is split
into two tasks, one with the pref, and one without.
Differential Revision: https://phabricator.services.mozilla.com/D7480
--HG--
extra : moz-landing-system : lando
There are a few external repos that hold copies of robustcheckout
in various places, mainly on release engineering infrastructure
that has been moved to GitHub. We should document those locations
in the same place we document the upgrade steps for Mercurial in
CI.
Differential Revision: https://phabricator.services.mozilla.com/D6989
--HG--
extra : moz-landing-system : lando
..and use this to allow create-interactive on level 3, but only for testers.
Differential Revision: https://phabricator.services.mozilla.com/D5573
--HG--
extra : moz-landing-system : lando
This additionally reconsiders the order of all of the actions, spacing them 50
"units" apart and putting the more common actions first.
MozReview-Commit-ID: 98IOYKVMcGU
--HG--
extra : rebase_source : 1273a8b86625bd8e4dc3bddab80c6912241f88c8
extra : histedit_source : 16314284a2b4e0368da843b036e22aaedf485307
Currently, `mach try fuzzy` generates a taskgraph that is configured exactly
like the most recent push to mozilla-central. This isn't always desirabe, so
pass some configuration down, to allow the taskgraph to behave differently.
Differential Revision: https://phabricator.services.mozilla.com/D2729
--HG--
extra : rebase_source : 99d6958b33211697227e65df17edc1eb337f63a4
extra : histedit_source : 69b5ff6805bc8409340eb71323a1f6fc637259d7
This is needed to not have a circular kind dependency when we actually spell out all dependencies (in a following patch)
Differential Revision: https://phabricator.services.mozilla.com/D1695
--HG--
rename : taskcluster/ci/repackage-signing/kind.yml => taskcluster/ci/repackage-signing-l10n/kind.yml
extra : rebase_source : c2998ba23f213090d27495eb44c3bde3a1628dff
Assorted fixes from trawling the sphinx logs - malformed formatting, broken references, leftovers from renaming action-task to action-callback and removing
yaml-templates, docstring fixes to make sphinx happier, and typos.
MozReview-Commit-ID: 6jUOljdLoE2
--HG--
extra : rebase_source : f2a9dbcde5180f760a80f47691a70eba76e58bad
Various fixes for the docs to make docs better and sphinx happier
* remove unused targets which were duplicated in balrog, partials
* fix up malformed targets and links
* convert docstrings to comments so sphinx ignores example response in utils/partials.py
* section underlines should match titles
MozReview-Commit-ID: GSYqsocBC4I
--HG--
extra : source : e464e9bbb42fb85170f1ce35a01f25811d753871
We shouldn't run this on central, as it falls back to the dev configs, and fails.
It should be fine on beta/release/esr60. I had to move this version of the check to its own
kind to avoid the dependency tree bringing in the entire build process. Perhaps we can
refactor later to avoid duplication
Differential Revision: https://phabricator.services.mozilla.com/D1765
--HG--
rename : taskcluster/ci/release-bouncer-check/kind.yml => taskcluster/ci/bouncer-check/kind.yml
This is just enough to make these repos discoverable and suggest their usage.
Anything more would duplicate documentation in those repositories.
MozReview-Commit-ID: 1WNEsoQBB9U
--HG--
extra : rebase_source : 19c25ee4b05fe72d2fed37ca68f75db3918d1c2f
Chains a release-eme-free-repack-beetmover-checksums kind after release-eme-free-repack-beetmover, to move
the target.checksums generated by the latter into the beetmover-checksums/ in candidates directory. Those
are then consumed by release-generate-checksum kind.
A lot of details like scopes, worker & provisioner, attributes, as well as data like repack_id and
partner_path, are inherited directly from the parent beetmover task. Mainly to avoid recalculating them.
In contrast to nightly builds, GPG signing of target.checksums has not been implemented. I don't believe
that adds any value in our current system because the sigs are not verified.
MozReview-Commit-ID: 38iz3J2PAXh
--HG--
extra : rebase_source : 8f2bebe747d97437780f1bc0d9e2f42159aee8a6
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : source : 0b941cbdca76fb2fbb98dc5bbc1a0237c69954d0
extra : histedit_source : a3e43bdd8a9a58550bef02fec3be832ca304ea93
Currently, many tasks fetch content from the Internets. A problem with
that is fetching from the Internets is unreliable: servers may have
outages or be slow; content may disappear or change out from under us.
The unreliability of 3rd party services poses a risk to Firefox CI.
If services aren't available, we could potentially not run some CI tasks.
In the worst case, we might not be able to release Firefox. That would
be bad. In fact, as I write this, gmplib.org has been unavailable for
~24 hours and Firefox CI is unable to retrieve the GMP source code.
As a result, building GCC toolchains is failing.
A solution to this is to make tasks more hermetic by depending on
fewer network services (which by definition aren't reliable over time
and therefore introduce instability).
This commit attempts to mitigate some external service dependencies
by introducing the *fetch* task kind.
The primary goal of the *fetch* kind is to obtain remote content and
re-expose it as a task artifact. By making external content available
as a cached task artifact, we allow dependent tasks to consume this
content without touching the service originally providing that
content, thus eliminating a run-time dependency and making tasks more
hermetic and reproducible over time.
We introduce a single "fetch-url" "using" flavor to define tasks that
fetch single URLs and then re-expose that URL as an artifact. Powering
this is a new, minimal "fetch" Docker image that contains a
"fetch-content" Python script that does the work for us.
We have added tasks to fetch source archives used to build the GCC
toolchains.
Fetching remote content and re-exposing it as an artifact is not
very useful by itself: the value is in having tasks use those
artifacts.
We introduce a taskgraph transform that allows tasks to define an
array of "fetches." Each entry corresponds to the name of a "fetch"
task kind. When present, the corresponding "fetch" task is added as a
dependency. And the task ID and artifact path from that "fetch" task
is added to the MOZ_FETCHES environment variable of the task depending
on it. Our "fetch-content" script has a "task-artifacts"
sub-command that tasks can execute to perform retrieval of all
artifacts listed in MOZ_FETCHES.
To prove all of this works, the code for fetching dependencies when
building GCC toolchains has been updated to use `fetch-content`. The
now-unused legacy code has been deleted.
This commit improves the reliability and efficiency of GCC toolchain
tasks. Dependencies now all come from task artifacts and should always
be available in the common case. In addition, `fetch-content` downloads
and extracts files concurrently. This makes it faster than the serial
application which we were previously using.
There are some things I don't like about this commit.
First, a new Docker image and Python script for downloading URLs feels
a bit heavyweight. The Docker image is definitely overkill as things
stand. I can eventually justify it because I want to implement support
for fetching and repackaging VCS repositories and for caching Debian
packages. These will require more packages than what I'm comfortable
installing on the base Debian image, therefore justifying a dedicated
image.
The `fetch-content static-url` sub-command could definitely be
implemented as a shell script. But Python is readily available and
is more pleasant to maintain than shell, so I wrote it in Python.
`fetch-content task-artifacts` is more advanced and writing it in
Python is more justified, IMO. FWIW, the script is Python 3 only,
which conveniently gives us access to `concurrent.futures`, which
facilitates concurrent download.
`fetch-content` also duplicates functionality found elsewhere.
generic-worker's task payload supports a "mounts" feature which
facilitates downloading remote content, including from a task
artifact. However, this feature doesn't exist on docker-worker.
So we have to implement downloading inside the task rather than
at the worker level. I concede that if all workers had generic-worker's
"mounts" feature and supported concurrent download, `fetch-content`
wouldn't need to exist.
`fetch-content` also duplicates functionality of
`mach artifact toolchain`. I probably could have used
`mach artifact toolchain` instead of writing
`fetch-content task-artifacts`. However, I didn't want to introduce
the requirement of a VCS checkout. `mach artifact toolchain` has its
origins in providing a feature to the build system. And "fetching
artifacts from tasks" is a more generic feature than that. I think
it should be implemented as a generic feature and not something that is
"toolchain" specific.
I think the best place for a generic "fetch content" feature is in
the worker, where content can be defined in the task payload. But as
explained above, that feature isn't universally available. The next
best place is probably run-task. run-task already performs generic,
very-early task preparation steps, such as performing a VCS checkout.
I would like to fold `fetch-content` into run-task and make it all
driven by environment variables. But run-task is currently Python 2
and achieving concurrency would involve a bit of programming (or
adding package dependencies). I may very well port run-task to Python
3 and then fold fetch-content into it. Or maybe we leave
`fetch-content` as a standalone script.
MozReview-Commit-ID: AGuTcwNcNJR
--HG--
extra : rebase_source : 4918b8c3bac53d63665006802054038bfbca0314
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1406
--HG--
extra : rebase_source : 207d1c25e37ab2619a09fb209282ffe55025de26
for L10n jobs should run per-push based on the corresponding builds
Differential Revision: https://phabricator.services.mozilla.com/D1406
--HG--
extra : rebase_source : 902aaff764db6df06b96e7ba8c7d717afd6869e3
Summary: A space was needed to make the code block show up
Reviewers: bhearsum
Reviewed By: bhearsum
Bug #: 1355482
Differential Revision: https://phabricator.services.mozilla.com/D957
--HG--
extra : rebase_source : 58c163102ddfd9982a98c0bc2ef9967fbf883d15
Summary:
Disable just the buildbot version of the periodic file updates in the file_update action.
Removing the code will happen later.
Reviewers: Callek
Reviewed By: Callek
Bug #: 1453616
Differential Revision: https://phabricator.services.mozilla.com/D921
--HG--
extra : rebase_source : 929ac46305f72ef0355f040243270e2f1f032de3
We don't have docs for how to upgrade Mercurial in CI. A lot of it
is organic institutional knowledge that lives in various people's
heads. That's not good for continuity of business.
This commit starts to establish documentation for how to upgrade
Mercurial in Firefox CI.
The documentation for updating things in-tree is pretty robust
(because I've done it a handful of times and know what is involved).
But the docs for OpenCloudConfig and Puppet could certainly be
improved. I'll poke the respective maintainers of these systems
to help improve these docs as a follow-up.
MozReview-Commit-ID: Aruh3N8Llls
--HG--
extra : rebase_source : 5d82ffd1d2b808082d824274db33586f71899599
- add balrog submit-toplevel - this replaces the final portion of the updates builder.
- rename balrog transform to balrog_submit, because it's for balrog locale submission
- make this default to the 'promote' phase. balrog and beetmover currently take the current
phase, which isn't always the wanted behavior.
- rename balrog publish to balrog schedule
- add balrog secondary push and secondary scheduling, for RCs
- remove the release_updates transforms
- make the task.py balrog transforms smarter
- get rid of the release_balrog_publishing transforms; ad a generic worker_type transform
- add BALROG_ACTIONS to scriptworker.py
- add get_balrog_action_scope()
- remove the unused balrog channel scopes
MozReview-Commit-ID: 369ACiOAd5F
--HG--
rename : taskcluster/taskgraph/transforms/balrog.py => taskcluster/taskgraph/transforms/balrog_submit.py
extra : rebase_source : 9311522460ae6790af14a6b8b9600019702f8cbd
This used to only be relevant to Devedition and Firefox releases.
In bug 1433536 we're going to add RC Fennec releases. Let's rename
the parameter now, for less parameters churn.
MozReview-Commit-ID: 28e1Y5FG4On
--HG--
extra : rebase_source : 59d41887d856481ab85bb8d2221dfcebca4430b0
extra : source : 4c96e312d88a3f7037c1eb39a3031d4baf997015
also clean up and move more config to the promotion config.
MozReview-Commit-ID: FmTWNNPcEaZ
--HG--
extra : rebase_source : 40431217fafb6796dbd65c7dfeab0e891ac1bbd4
extra : source : 0f5418a83477c1b6b221e4d28515792410e504d0
There are e.g. some build infrastructure changes that we want to have a
controlled impact on the Firefox builds we produce. We have, in multiple
occasions, gone through manual work to compare Firefox builds, most of
the time using the diffoscope tool (https://diffoscope.org/).
This change introduces a new task kind that takes two Firefox builds as
input, either by name (reference to a build from the current task graph)
or by index (reference to a build from a previous push), and compares
them.
In order to get a Firefox build by index, we rely on dummy tasks with
an optimization we expect to always hit, so we add the necessary bits
to ensure those dummy tasks can go through up to the optimization phase
and be optimized out there.
--HG--
extra : rebase_source : 37482f67652dab2fcef2db4e6b8efe653999bae5
Add support for the three firefox and devedition relpro flavors (we
could probably reduce devedition to 2).
Also, instead of defining which kinds to use from the previous graph
in `previous_graph_kinds`, specify which kinds to rebuild (ignore)
from the previous graph in `rebuild_kinds`. This list will be much
smaller (currently empty).
MozReview-Commit-ID: 5rH1TW7GbAD
--HG--
extra : rebase_source : b4294a0d17a99b2ffd48f5d62821c724324b242c
extra : histedit_source : b3fe87101e4595f5fc5c7daaa2d4a0bd88418667
The runnable jobs API on Treeherder is timing out quite frequently.
This is due that it downloads the full-task-graph.json artifact from the Gecko decision task.
This file has grown to be massive (over 30MB) and that takes too long to fetch. The API times out after 20 seconds.
The data we need from the artifact is minimal. We can generate a file with just the minimum amount of data needed.
This code change adds logic to generate a runnable-jobs.json.gz file after the full-task-graph.json artifact is generated.
MozReview-Commit-ID: 9u2H7HbUAcl
--HG--
extra : rebase_source : f450527e7dd565e29cfe2e04dc52761e77827cc0
This allows firefox and thunderbird builds to avoid using each others bits.
MozReview-Commit-ID: KYQYDd2tkGj
--HG--
extra : rebase_source : 42f1d13ec609f066cb3bd3050ed894296b72d982
These two new attributes will help us determine which tasks belong in which release promotion graphs. In the future, we'll specify these for all shippable builds as well, and we can reduce the usage of the `product` keys. However, sometimes we need `product` to differ from `shipping-product` (e.g., `mobile` vs `fennec`; in this case we mean `stage_product` vs `shipping_product`), so I haven't yet touched those key/value pairs.
MozReview-Commit-ID: LEuf4CS277Q
--HG--
extra : rebase_source : 4971883500658ace209bec97a639a54c94292b50
Instead of relying on environment variables, pass these in as parameters.
MozReview-Commit-ID: An58Bu2kd1g
--HG--
extra : rebase_source : 9c7b6beb13b676a3376a897f2c8143cc042b8276
This template takes a single integer as input, and sets the 'task_duplicates'
attribute on any tasks that were explicitly specified by try_task_config.json.
This means dependent tasks or 'always_target' tasks will not be rebuilt.
To support this template, the apply_jsone_templates morph now requires the
entire try_task_config object instead of just the templates.
MozReview-Commit-ID: DwxUtlC5VD5
--HG--
extra : rebase_source : c57a80ada2fc4591a5ce4c94965b8c9f02bb53b1
The motivation for this change is to create a 'rebuild' template that sets
'attributes.task_duplicates'. A nice side effect is that if any of 'task'
or 'attributes' aren't explicitly set, we can ignore the result which means
we don't need to have a whole bunch of 'else' clauses in the templates.
MozReview-Commit-ID: DX3UyFGb4ff
--HG--
extra : rebase_source : b768cb9f9df3cb22827209519f1cd2a4944af74f
Tasks that have the 'always_target' attribute set will be always be included
in the target_task_graph, regardless of target task filtering.
Furthermore, if they were only added because of this attribute (i.e, the
filters would have excluded the task), then the task will be a candidate for
optimization even if the 'optimize_target_tasks' parameter is False.
MozReview-Commit-ID: 9eoVJ5qpAMO
--HG--
extra : rebase_source : 9635002720d088ca9870649f3143d6293c666610
This patch adds the `release-notify-promote` and `release-notify-publish` kinds. It also genericizes all the notifications, and updates the kinds that use those notifications.
MozReview-Commit-ID: 9ymXKzthVF4
--HG--
extra : rebase_source : 6d7db48afa14e4124834321784da65e8d1e84c98
This patch adds the beetmover-cdns kind, and adds it to
`publish_fennec`.
This was the first non-buildbot-bridge, non-dummy relpro task, so this
needed a new transform.
This patch also updates the `previous_graph_kinds` and updates the
beetmover scopes in scriptworker.py.
MozReview-Commit-ID: 3rpkjuLjjXz
--HG--
extra : rebase_source : d491f1ce5d10ee8f2602698236106007e203c358
This patch adds the initial `release-version-bump` kind, and adds it to
the `publish_fennec` `target_tasks_method`.
It also adds support for `next_version`.
MozReview-Commit-ID: 9YRswddeuZ3
--HG--
extra : rebase_source : 991cbf89b40c4ef980c42251001e81be5cedaf00
This patch adds the initial `release-uptake-monitoring` kind, and adds
it to the `publish_fennec` `target_tasks_method`.
MozReview-Commit-ID: 3RDMNGrbBwD
--HG--
extra : rebase_source : f504c8e173337d45bc9f374bd0349c6823b50ffb
This patch adds the initial `release-bouncer-aliases` kind, and adds it
to the `publish_fennec` `target_tasks_method`.
It also adds the ability to specify the `tuxedo_server_url`
`by-project`.
MozReview-Commit-ID: 9I4IaUlbCCD
--HG--
extra : rebase_source : d0ed88a687ef6fb9f97dc67c0f04742bbfeb201d
This patch adds the initial `release-mark-as-shipped` kind, and adds it to
the `publish_fennec` `target_tasks_method`.
MozReview-Commit-ID: F8AYscJQWlh
--HG--
extra : rebase_source : 9522b95f65b5b373a54bc0fc01a20c72adbee0cb
This patch adds the first releasetask as a new kind. To support this, we
added a new `release-promotion` flag in the buildbot job. If this is
set, we use the new `bb_release_worker` function; otherwise we fall back
to the `bb_ci_worker` function (this is the old behavior, factored out
into a separate function).
We also added `build_number` and `release_promotion` attributes in the
task definition. Finally, `build_number` now defaults to 1, allowing us
to create the task graph locally without forcing us to set
`BUILD_NUMBER` in the environment.
MozReview-Commit-ID: 8vNMHJemqAG
--HG--
extra : rebase_source : c8816d3e7b2f5358ea51ee8d5fb12585e67e9853
Add a release promotion custom action for releng's TC release promotion migration work.
This action generates a graph dependent on previously built tasks. To track these, we add the `do_not_optimize` and `existing_tasks` parameters. The `do_not_optimize` parameter specifies tasks that we want to explicitly exclude from taskgraph optimization. The `existing_tasks` parameter specifies a label-to-taskid map for tasks from previous graphs.
MozReview-Commit-ID: 1vKrNUavM4V
--HG--
extra : rebase_source : b8ba95d270aafe1464c2b3bfc318b9568500a7a1
With this in place, all `-j`obs will not run by default on try. This will omit
such jobs in most try pushes even if files-changed matches. This is
unfortunate, but better than running them unconditionally. Fuzzy selections,
and later `just try it` pushes, are the ultimate solution here.
With this change, a push with no try syntax or try_task_config.json will schedule
no tasks at all.
MozReview-Commit-ID: FGjqlDW1FT6
--HG--
extra : rebase_source : 727ceafb1b6d24f83c0c7382b6a877ecb65863ab
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : 25e9966696d78d899783d9f38533d5ae66f9ccb9
extra : source : b53ff084c2d7968a1d9864d1343f2d9381fb652b
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
--HG--
extra : rebase_source : cf1654036041a64398a2cd38e35e8de4f3596ff9
extra : source : c25af2c111a5be4e3381d0b002641691d15fe4e8
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : 03a10610aa3337269fe76a1196bb9b1665e1ab20
extra : source : b53ff084c2d7968a1d9864d1343f2d9381fb652b
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
--HG--
extra : rebase_source : c6e2ff90316d43cd93826de5c30a1936f19c01ca
extra : source : c25af2c111a5be4e3381d0b002641691d15fe4e8
Per IRC dicussion, there is no technical or policy restriction on dependencies
within a task kind. Update the documentation to remove the out-of-date mention
of this limitation. In particular, toolchain build tasks tend to depend on
each other.
MozReview-Commit-ID: K6p0mxyjcvY
--HG--
extra : rebase_source : 9439503e8f0a278b1bb41d8a71a2d31d5fc9c0e8
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : bb1d194b49caa4b69ce8faf9f324c80850d15d3f
extra : histedit_source : a933e0a6f8819b8d0e9220ecc10c34d096a0d477
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
--HG--
extra : rebase_source : 661c6b7134c03481c3aa7520d892cc3565b04d85
extra : histedit_source : f0d058df0c8fae075c72f57d8b764f485eb4f6ed
This sets the try_mode property, and parses the try message (if given), early
in the decision task and puts the results into the parameters.
The proximate need is to set optimze_target_tasks for some try modes and not
others. This also replaces the existing logic for parsing messages for certain
kinds, and makes the distinction between the different try modes a little
clearer.
MozReview-Commit-ID: AXJEGLh6pEV
--HG--
extra : rebase_source : fdd8c3cfb9abf759a3c43c5713e62e4772c5bd06
In preparation for much more thorough optimization of task-graphs, this
makes a few changes:
* optimization is split into thre phases, with task removal in one phase
(following dependency links) and task replacement in the next (in the
reverse order).
* optimization uses class instances instead of functions for optimizations;
this allows different functions for different phases, and also leaves open
the possibility of composing optimizations.
* the replacement phase can also support removal; this is when utility tasks
like symbol uploads can be optimized away iff their parent task is
optimized.
MozReview-Commit-ID: C5QznNpwqXn
--HG--
extra : rebase_source : f8e65ee3ebb9fb584b00ee0db518b790a9c1b233
`run-task` is taught a --sparse-profile argument to be passed down
to `hg robustcheckout` for the main source checkout. It does what
you expect: performs a sparse checkout using the named profile.
The Taskgraph YAML for run-task is taught a "sparse-profile"
property to define the sparse profile. When defined, --sparse-profile
will be passed down to `run-task` and the cache name will be updated
to reflect the use of sparse checkout.
Our cache checking transform is updated to audit for the use of
--sparse-profile without the corresponding "-sparse" cache name
variation.
The reason we need a distinct cache name for sparse is because
clients that aren't sparse aware will be unable to read checkouts
that are sparse. By forcing sparse and non-sparse into different
cache pools, we avoid compatibility issues.
In the ideal world, we probably support sparse profiles on all the
VCS checkouts that `run-task` supports (e.g. --tools-checkout).
Perfect is the enemy of done. All of this is defined in-tree and
it is easy enough to change atomically.
MozReview-Commit-ID: 79k7Vul0hHO
--HG--
extra : rebase_source : babe9b42e2796c2341bffc6ecfe829f4daff9e0f
* eliminate heading for test kinds, of which there is now only one
* make the caches document have a single heading in the TOC
* break out mach commands into a separate document, add ./mach taskgraph morphed
* remove docs for YAML templates support (the .yml file wasn't actually
used -- I expect it was a merge leftover); these are still used for actions.yml,
but once that is gone the code should be removed, too.
* break try out into its own document, edit to distinguish "how to run try"
from "how to generate config"
MozReview-Commit-ID: 76ZopWA9TPL
--HG--
extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
Caches shared across levels scare me, even if readers are purported to
perform content verification. We shouldn't take any risks with released
Firefox builds being contaminated by e.g. Try tasks.
Also, the old cache name interferes with my desire to make cache
names dynamic. This requires dynamic scopes. We already have
have level-{{level}}-* scopes for caches. So having all caches
prefixed with this makes things flexible.
MozReview-Commit-ID: LsrcxIYoEh1
--HG--
extra : rebase_source : dfe97f92a726059200ed79afe215ef2cf1fd7bf1
This change adds an upload-generated-sources task kind that runs after nightly
builds, fetches their `target.generated-files.tar.gz` artifact, and uploads
all the contained files to an S3 bucket. For actual nightly and release builds
on SCM level 3 trees, the S3 bucket is configured to be publicly accessible,
so that tools like Socorro will be able to fetch generated source files that
appear in crash reports, and debuggers will be able to fetch generated sources
when they show up while debugging Nightly or Release builds.
There are also level-2 and level-1 S3 buckets configured for builds happening
on trees of other levels such as try. They are not configured as publicly
accessible, but they exist so that these tasks can be tested in try.
MozReview-Commit-ID: Js1HRftbtep
--HG--
extra : rebase_source : b1172c9cc8b8be437d3b94a6bf0ff6b2f7d3508b
extra : source : 73bf88110b3821d62a3d393e85b56896a12f2930
This provides a mechanism to modify the behaviour of tasks from a try push. The try_task_config.json
looks something like:
{
"tasks": ["build-linux64/opt", "test-linux64/opt-mochitest-e10s-1"],
"templates": {
"artifact": {"enabled": 1}
}
}
This tells taskgraph to apply the 'artifact' template to all tasks. Templates are JSONe based
.yml files that live under taskcluster/taskgraph/templates. Taskgraph will render every template
against every task definition. The templates themselves can then use JSONe condition statements to
filter out which tasks they should or shouldn't apply to.
MozReview-Commit-ID: J8HVZzOt4mX
--HG--
extra : rebase_source : 95a78bc56d3f90ff1b34aabd84ed92aff1e3d954
It is desirable for e.g. smooth toolchain transitions, to be able to
refer them with generic name from toolchain jobs, while they have more
specific names, including version numbers. For example, in a near
future, there could be a linux64-gcc-4.9 toolchain and a linux64-gcc-6.
The default would be former, but at some point we'd want to switch to
the latter, without having to change all the toolchain definitions.
Moreover, when the switch happens, it would be desirable to have some
jobs stick with the old version, which is hard to keep track of when
all the toolchain definitions for build jobs use the same versioned
toolchain. With an alias, jobs that want the default use the alias, and
jobs that want to use a specific version use the versioned toolchain
name.
--HG--
extra : rebase_source : 467d713edd00dbe358483f5ee749fa56900714dd
This introduces a 'try_task_config' method of scheduling. En lieu of (or in addition to) try
syntax, you can now check in a file called 'try_task_config.json' to the root of the source
tree. The format is either a list of task labels, or dict where task labels are the keys.
Taskcluster will simply schedule any tasks that are listed there.
This file is primarily meant to be generated by tools (which don't exist yet), as the json
format is much easier for tools to generate or consume. These tools should use an in-memory
commit to add the file so it is automatically removed again after the push.
A server-side hook will be added in bug 1380357 to prevent this file from accidentally
landing on non-try trees.
MozReview-Commit-ID: 2zKfZXuuDhH
--HG--
extra : rebase_source : b5d5ff47c607288657418fd041603093f8c29e85
The toolchain jobs produce artifacts that are going to be used by other
jobs, but there is no reliable way for the decision task to know the
name of those artifacts. So we make their definition required in the
toolchain job definitions.
--HG--
extra : rebase_source : eb668cf2bf7793b44dc3d91e3ddb8d7c69e3de52
This name can be used by user interfaces to find the action corresponding to a
dedicaetd UI element.
MozReview-Commit-ID: HFBDlX30J31
--HG--
extra : rebase_source : a70c66106f71f46fb4678eb076f53db073fc83f3
Minor modificatinons based on conversations at the SF all-hands.
MozReview-Commit-ID: AnkwXppUPJd
--HG--
extra : rebase_source : b3bef018689a614dd6a3048e85db30bb4f5722e5
Support OSX Signed nightlies (in the complete.mar too)
MozReview-Commit-ID: HkGRRm7k2Ra
--HG--
extra : rebase_source : a8fc5c76bdc623e4217840b7b75c39d0aa0b9051
Support OSX Signed nightlies (in the complete.mar too)
MozReview-Commit-ID: HVJwMuXUixX
--HG--
extra : rebase_source : b05bca1a539e751e8a8a59c32a0a392752c95c73
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: BeXoChssNjF
--HG--
extra : rebase_source : 1f086e96411a7683b77cfecb6079b54ab9b0f643
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: 2zOkiBS294y
--HG--
extra : rebase_source : d380d4975f8480b65d44c07c099ef6f166228e01
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: BeXoChssNjF
--HG--
extra : rebase_source : 1f086e96411a7683b77cfecb6079b54ab9b0f643
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: 2zOkiBS294y
--HG--
extra : rebase_source : d380d4975f8480b65d44c07c099ef6f166228e01
{central, autoland, inbound} are logically treated as a single unit for
many tasks and policies. Let's formalize that collection via a "trunk"
alias.
MozReview-Commit-ID: H4JPTyu2J2F
--HG--
extra : rebase_source : f4cabfc48e24b9f55b833bd95bdbf81b036cee6d
The purpose of this parameter has been superseded by the `include_nightly`
property.
MozReview-Commit-ID: 4iXQsv9Drqg
--HG--
extra : rebase_source : c94142282909a88c29fe6809d87721bef1f198c2
This is a more robust approach than using substring matching on task labels.
As an optimization, this simply avoids creating balrog tasks for unsigned beets
using only-for-attributes, rather than omitting them in a transform.
MozReview-Commit-ID: 8MNOxu0WgXo
--HG--
extra : source : 1aeb99ce3e6c2576b7b9b71c1cdf97a1d1889a49
This is a more robust approach than using substring matching on task labels.
As an optimization, this simply avoids creating balrog tasks for unsigned beets
using only-for-attributes, rather than omitting them in a transform.
MozReview-Commit-ID: 8MNOxu0WgXo
--HG--
extra : rebase_source : 9e93a996241bcb0d345c18813919a41320e95651
This patch enables `run-on-projects` to work appropriately for
nightly builds and tests. Initially, we were setting an empty
`run-on-projects` for nightly `build_platform`s, then explicitly
targeting the platforms in nightly-specific `target_task_method`s.
Instead, this patch enables nightlies to `run-on-projects` everywhere,
but governs the use of nightlies by either the `include_nightly`
parameter, or the `--include-nightly` try option. This lets us filter
nightly-related `target_task_method`s against `run-on-projects` without
losing all nightly tasks.
Then, enable spidermonkey tests by removing optimization from beta and
release. This patch also enables everything then disables specific
tasks, rather than disabling everything and enabling specific tasks.
Since we're beginning with a `filter_for_project` call, we should be
able to reduce these if blocks to zero over time, if desired.
MozReview-Commit-ID: A9tolynaChF
--HG--
extra : rebase_source : 3465ee2c714de3e0359f14109096fc94de27aadf
Graph morphs modify the graph after optimization, without changing its meaning.
In this case, that means adding index tasks that will insert paths into the
index beyond the relatively limited number afforded in task.routes.
MozReview-Commit-ID: AJy4exX7q2v
--HG--
extra : rebase_source : d61e7462defd41e7112739fb057edb493f495430
extra : source : c580568ed47c1ed2af40d98b47fbb0d136e63060
Note that the to_json method prefers the taskgraph's dependencies information
(edges) to that from the task.dependencies entries. At a few points in
task-graph generation, these values differ, although that is expected (for
example, the full task set contains no edges, but that information is still in
task.dependencies). Unifying that representation leads to some difficulty with
task transforms that reach into the dependency tree (beetmover), so the
different representations are left as-is.
MozReview-Commit-ID: GeW8HNwFA9Z
--HG--
extra : rebase_source : 549773e05e18371a399612d9bceccffc29be8cf2
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
this patch:
- adds linux{32,64}-nightly/opt test platforms that mirror the non-nightly test platforms.
- adds an `include_nightly` per-project parameter; this is refered to in the default `target_task_method`. It's still possible to launch custom `target_task_method`s to trigger nightlies against, say, try.
- adds a `filter_for_project` method in `target_tasks.py` that allows for `include_nightly` and `run_on_projects` filtering in the various `target_task_method`s.
- adds nightly filtering into the `TryOptionSyntax` object. By default, this will be off. To trigger nightly tests on try, either submit a new decision task with a different `target_task_method` (e.g. `nightly_fennec`) or flip the `include_nightly` flag to True.
- adds the `nightly` attribute to tests if their builds have that attribute.
MozReview-Commit-ID: DttIZH0BHS2
--HG--
extra : rebase_source : d8acbe4c741f570b2e8d33a8e6a7f5c791b24ff6
This fixes the ability to run mozbase via `-j mozbase`, with the
added advantage that it will obey `-p` too.
MozReview-Commit-ID: 1zkitUephXk
--HG--
extra : rebase_source : 0ebb65363d5f5813bc7ccb379768df54310d39c1
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 22382e3d65ce8454a1682cfced0d03477762e8fe
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 16ed70edd38a53c3279d8632d7cba3df4d5216c3
The name 'source-check' is a bit of a misnomer, because it already includes a bunch
of tasks that are proper unittests, not lints. Some of these unittests will soon
depend on a build task, which makes 'source-check' feel even more wrong.
They still have a lot in common with the lint tasks though, so it's probably not
worth splitting this into two separate kinds. Instead, let's just rename this kind
to 'source-test', which means, any task that tests stuff and is run from the source
directory (instead of a tests.zip). I think both lints and python-tests fall squarely
under this definition.
MozReview-Commit-ID: K0gZ5rVLyeD
--HG--
rename : taskcluster/ci/source-check/doc.yml => taskcluster/ci/source-test/doc.yml
rename : taskcluster/ci/source-check/kind.yml => taskcluster/ci/source-test/kind.yml
rename : taskcluster/ci/source-check/mozlint.yml => taskcluster/ci/source-test/mozlint.yml
rename : taskcluster/ci/source-check/python-tests.yml => taskcluster/ci/source-test/python-tests.yml
rename : taskcluster/ci/source-check/webidl.yml => taskcluster/ci/source-test/webidl.yml
extra : rebase_source : a683b5b6e243849de57f2681993046c776d6a8f2
Framework for defining actions in-tree that can be displayed
and triggered from Treeherder.
MozReview-Commit-ID: 3rvwgy2i4xu
--HG--
extra : rebase_source : beca394f4337aae4ab149e4db810352f57ec4988
Add specification for actions.json to be used as contract
between in-tree logic and Treeherder. Such that Treeherder
can provide actions that callback into the in-tree logic.
MozReview-Commit-ID: JM1ebU8zNK5
--HG--
extra : rebase_source : 289c7c800f214ccde99adfbdf58bba614b957fe6
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: GPFdi0FD6Vn
--HG--
extra : rebase_source : b992786158851f1099aedfce8669a163228edc51
Tom Prince
Mihai Tabara
Mike Hommey
Andrew Halberstadt
Connor Sheehan
Simon Fraser
Dustin J. Mitchell
Johan Lorenzo
Justin Wood
Chris AtLee
Ben Hearsum
Aki Sasaki
shindli
Nick Thomas
Cosmin Sabou
Dave Hunt
Coroiu Cristina
Gregory Szorc
Gurzau Raul
Bogdan Tara
Sylvestre Ledru
Andi-Bogdan Postelnicu
Rob Wood
Brindusan Cristian
Andreea Pavel
Csoregi Natalia
Rail Aliiev
Noemi Erli
Armen Zambrano G.
Bill McCloskey
Rok Garbas
Rail Aliiev
Brian Stack
Phil Ringnalda
Sebastian Hengst
Wes Kocher
Ralph Giles
Nick Fitzgerald
Ryan VanderMeulen
Ted Mielczarek
Carsten "Tomcat" Book
Peter Moore
Paul Bignier
Iris Hsiao
Hammad Akhtar
Jonas Finnemann Jensen
Wander Lairson Costa