`run-task` is taught a --sparse-profile argument to be passed down
to `hg robustcheckout` for the main source checkout. It does what
you expect: performs a sparse checkout using the named profile.
The Taskgraph YAML for run-task is taught a "sparse-profile"
property to define the sparse profile. When defined, --sparse-profile
will be passed down to `run-task` and the cache name will be updated
to reflect the use of sparse checkout.
Our cache checking transform is updated to audit for the use of
--sparse-profile without the corresponding "-sparse" cache name
variation.
The reason we need a distinct cache name for sparse is because
clients that aren't sparse aware will be unable to read checkouts
that are sparse. By forcing sparse and non-sparse into different
cache pools, we avoid compatibility issues.
In the ideal world, we probably support sparse profiles on all the
VCS checkouts that `run-task` supports (e.g. --tools-checkout).
Perfect is the enemy of done. All of this is defined in-tree and
it is easy enough to change atomically.
MozReview-Commit-ID: 79k7Vul0hHO
--HG--
extra : rebase_source : babe9b42e2796c2341bffc6ecfe829f4daff9e0f
* eliminate heading for test kinds, of which there is now only one
* make the caches document have a single heading in the TOC
* break out mach commands into a separate document, add ./mach taskgraph morphed
* remove docs for YAML templates support (the .yml file wasn't actually
used -- I expect it was a merge leftover); these are still used for actions.yml,
but once that is gone the code should be removed, too.
* break try out into its own document, edit to distinguish "how to run try"
from "how to generate config"
MozReview-Commit-ID: 76ZopWA9TPL
--HG--
extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
Today, cache names are mostly static and are brittle as a result.
In theory, when a backwards incompatible change is performed on
something that touches a cache, the cache name needs to be changed
to ensure tasks running the old code don't see cached data from the
new task. (Alternatively, all code is forward compatible, but that is
hard to implement in practice.)
For many things, the process works as planned. However, not everyone
knows that cache names need changed. And, it isn't always obvious
that some things require fresh caches. When mistakes are made, tasks
break intermittently due to cache wonkiness.
One area where we get into trouble is with UID and GID mismatch.
Task A will use a Docker image where our standard "worker" user/group
is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is
common when mixing Debian and RedHel based distros.) If they use the
same cache, then Task B needs to chown/chmod all files in the cache
or there could be a permissions problem. This is exactly why
run-task recursively chowns certain paths before dropping root
privileges.
Permissions setting in run-task solves permissions problems. But
it doesn't solve content incompatibility problems. For that, you
need to change cache names, not use caches, or blow away content
when incompatibilities are detected.
This commit starts the process of adding a little bit more coherence
to our caching story.
There are two main features in this commit:
1) Cache names tied to run-task content
2) Cache validation in run-task
Taskgraph now detects when a task is using caches with run-task. When
caches and run-task are both being used, the cache name is adjusted to
contain a hash of run-task's content. When run-task changes, the cache
name changes. So, changing run-task ensures that all caches from that point
forward are "clean." This frees run-task and any functionality related
to run-task (such as maintaining version control checkouts) from
having to maintain backwards or forwards compatibility with any other
version of run-task. This does mean that any changes to run-task
effectively wipe out caches. But changes to run-task tend to be
seldom, so this should be acceptable.
The second part of this change is code in run-task to record per-cache
properties and validate whether a populated cache is appropriate for
use. To enable this, taskgraph passes a list of cache paths via an
environment variable. For each cache path, run-task looks for a
well-defined file containing a list of "requirements." Right now,
that list is simply a version string. But other features will be
worked into it. If the cache is empty, we simply write out a new
requirements file and are done. If the file exists, we compare
requirements and fail fast if there is a mismatch. If the cache
has content but not this special file, then we abort (because this
should never happen).
The "requirements" validation isn't very useful now because the only
entry comes from run-task's source code and modifying run-task will
change the hash and cause a new cache to be used. The implementation
at this point is more demonstrating the concept than doing anything
terribly useful with it.
MozReview-Commit-ID: HtpXIc7OD1k
--HG--
extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
Caches shared across levels scare me, even if readers are purported to
perform content verification. We shouldn't take any risks with released
Firefox builds being contaminated by e.g. Try tasks.
Also, the old cache name interferes with my desire to make cache
names dynamic. This requires dynamic scopes. We already have
have level-{{level}}-* scopes for caches. So having all caches
prefixed with this makes things flexible.
MozReview-Commit-ID: LsrcxIYoEh1
--HG--
extra : rebase_source : dfe97f92a726059200ed79afe215ef2cf1fd7bf1
This change adds an upload-generated-sources task kind that runs after nightly
builds, fetches their `target.generated-files.tar.gz` artifact, and uploads
all the contained files to an S3 bucket. For actual nightly and release builds
on SCM level 3 trees, the S3 bucket is configured to be publicly accessible,
so that tools like Socorro will be able to fetch generated source files that
appear in crash reports, and debuggers will be able to fetch generated sources
when they show up while debugging Nightly or Release builds.
There are also level-2 and level-1 S3 buckets configured for builds happening
on trees of other levels such as try. They are not configured as publicly
accessible, but they exist so that these tasks can be tested in try.
MozReview-Commit-ID: Js1HRftbtep
--HG--
extra : rebase_source : b1172c9cc8b8be437d3b94a6bf0ff6b2f7d3508b
extra : source : 73bf88110b3821d62a3d393e85b56896a12f2930
This provides a mechanism to modify the behaviour of tasks from a try push. The try_task_config.json
looks something like:
{
"tasks": ["build-linux64/opt", "test-linux64/opt-mochitest-e10s-1"],
"templates": {
"artifact": {"enabled": 1}
}
}
This tells taskgraph to apply the 'artifact' template to all tasks. Templates are JSONe based
.yml files that live under taskcluster/taskgraph/templates. Taskgraph will render every template
against every task definition. The templates themselves can then use JSONe condition statements to
filter out which tasks they should or shouldn't apply to.
MozReview-Commit-ID: J8HVZzOt4mX
--HG--
extra : rebase_source : 95a78bc56d3f90ff1b34aabd84ed92aff1e3d954
It is desirable for e.g. smooth toolchain transitions, to be able to
refer them with generic name from toolchain jobs, while they have more
specific names, including version numbers. For example, in a near
future, there could be a linux64-gcc-4.9 toolchain and a linux64-gcc-6.
The default would be former, but at some point we'd want to switch to
the latter, without having to change all the toolchain definitions.
Moreover, when the switch happens, it would be desirable to have some
jobs stick with the old version, which is hard to keep track of when
all the toolchain definitions for build jobs use the same versioned
toolchain. With an alias, jobs that want the default use the alias, and
jobs that want to use a specific version use the versioned toolchain
name.
--HG--
extra : rebase_source : 467d713edd00dbe358483f5ee749fa56900714dd
This introduces a 'try_task_config' method of scheduling. En lieu of (or in addition to) try
syntax, you can now check in a file called 'try_task_config.json' to the root of the source
tree. The format is either a list of task labels, or dict where task labels are the keys.
Taskcluster will simply schedule any tasks that are listed there.
This file is primarily meant to be generated by tools (which don't exist yet), as the json
format is much easier for tools to generate or consume. These tools should use an in-memory
commit to add the file so it is automatically removed again after the push.
A server-side hook will be added in bug 1380357 to prevent this file from accidentally
landing on non-try trees.
MozReview-Commit-ID: 2zKfZXuuDhH
--HG--
extra : rebase_source : b5d5ff47c607288657418fd041603093f8c29e85
The toolchain jobs produce artifacts that are going to be used by other
jobs, but there is no reliable way for the decision task to know the
name of those artifacts. So we make their definition required in the
toolchain job definitions.
--HG--
extra : rebase_source : eb668cf2bf7793b44dc3d91e3ddb8d7c69e3de52
This name can be used by user interfaces to find the action corresponding to a
dedicaetd UI element.
MozReview-Commit-ID: HFBDlX30J31
--HG--
extra : rebase_source : a70c66106f71f46fb4678eb076f53db073fc83f3
Minor modificatinons based on conversations at the SF all-hands.
MozReview-Commit-ID: AnkwXppUPJd
--HG--
extra : rebase_source : b3bef018689a614dd6a3048e85db30bb4f5722e5
Support OSX Signed nightlies (in the complete.mar too)
MozReview-Commit-ID: HkGRRm7k2Ra
--HG--
extra : rebase_source : a8fc5c76bdc623e4217840b7b75c39d0aa0b9051
Support OSX Signed nightlies (in the complete.mar too)
MozReview-Commit-ID: HVJwMuXUixX
--HG--
extra : rebase_source : b05bca1a539e751e8a8a59c32a0a392752c95c73
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: BeXoChssNjF
--HG--
extra : rebase_source : 1f086e96411a7683b77cfecb6079b54ab9b0f643
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: 2zOkiBS294y
--HG--
extra : rebase_source : d380d4975f8480b65d44c07c099ef6f166228e01
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: BeXoChssNjF
--HG--
extra : rebase_source : 1f086e96411a7683b77cfecb6079b54ab9b0f643
(For Landing more OSX Nightly Support from date to central)
MozReview-Commit-ID: 2zOkiBS294y
--HG--
extra : rebase_source : d380d4975f8480b65d44c07c099ef6f166228e01
{central, autoland, inbound} are logically treated as a single unit for
many tasks and policies. Let's formalize that collection via a "trunk"
alias.
MozReview-Commit-ID: H4JPTyu2J2F
--HG--
extra : rebase_source : f4cabfc48e24b9f55b833bd95bdbf81b036cee6d
The purpose of this parameter has been superseded by the `include_nightly`
property.
MozReview-Commit-ID: 4iXQsv9Drqg
--HG--
extra : rebase_source : c94142282909a88c29fe6809d87721bef1f198c2
This is a more robust approach than using substring matching on task labels.
As an optimization, this simply avoids creating balrog tasks for unsigned beets
using only-for-attributes, rather than omitting them in a transform.
MozReview-Commit-ID: 8MNOxu0WgXo
--HG--
extra : source : 1aeb99ce3e6c2576b7b9b71c1cdf97a1d1889a49
This is a more robust approach than using substring matching on task labels.
As an optimization, this simply avoids creating balrog tasks for unsigned beets
using only-for-attributes, rather than omitting them in a transform.
MozReview-Commit-ID: 8MNOxu0WgXo
--HG--
extra : rebase_source : 9e93a996241bcb0d345c18813919a41320e95651
This patch enables `run-on-projects` to work appropriately for
nightly builds and tests. Initially, we were setting an empty
`run-on-projects` for nightly `build_platform`s, then explicitly
targeting the platforms in nightly-specific `target_task_method`s.
Instead, this patch enables nightlies to `run-on-projects` everywhere,
but governs the use of nightlies by either the `include_nightly`
parameter, or the `--include-nightly` try option. This lets us filter
nightly-related `target_task_method`s against `run-on-projects` without
losing all nightly tasks.
Then, enable spidermonkey tests by removing optimization from beta and
release. This patch also enables everything then disables specific
tasks, rather than disabling everything and enabling specific tasks.
Since we're beginning with a `filter_for_project` call, we should be
able to reduce these if blocks to zero over time, if desired.
MozReview-Commit-ID: A9tolynaChF
--HG--
extra : rebase_source : 3465ee2c714de3e0359f14109096fc94de27aadf
Graph morphs modify the graph after optimization, without changing its meaning.
In this case, that means adding index tasks that will insert paths into the
index beyond the relatively limited number afforded in task.routes.
MozReview-Commit-ID: AJy4exX7q2v
--HG--
extra : rebase_source : d61e7462defd41e7112739fb057edb493f495430
extra : source : c580568ed47c1ed2af40d98b47fbb0d136e63060
Note that the to_json method prefers the taskgraph's dependencies information
(edges) to that from the task.dependencies entries. At a few points in
task-graph generation, these values differ, although that is expected (for
example, the full task set contains no edges, but that information is still in
task.dependencies). Unifying that representation leads to some difficulty with
task transforms that reach into the dependency tree (beetmover), so the
different representations are left as-is.
MozReview-Commit-ID: GeW8HNwFA9Z
--HG--
extra : rebase_source : 549773e05e18371a399612d9bceccffc29be8cf2
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
this patch:
- adds linux{32,64}-nightly/opt test platforms that mirror the non-nightly test platforms.
- adds an `include_nightly` per-project parameter; this is refered to in the default `target_task_method`. It's still possible to launch custom `target_task_method`s to trigger nightlies against, say, try.
- adds a `filter_for_project` method in `target_tasks.py` that allows for `include_nightly` and `run_on_projects` filtering in the various `target_task_method`s.
- adds nightly filtering into the `TryOptionSyntax` object. By default, this will be off. To trigger nightly tests on try, either submit a new decision task with a different `target_task_method` (e.g. `nightly_fennec`) or flip the `include_nightly` flag to True.
- adds the `nightly` attribute to tests if their builds have that attribute.
MozReview-Commit-ID: DttIZH0BHS2
--HG--
extra : rebase_source : d8acbe4c741f570b2e8d33a8e6a7f5c791b24ff6
This fixes the ability to run mozbase via `-j mozbase`, with the
added advantage that it will obey `-p` too.
MozReview-Commit-ID: 1zkitUephXk
--HG--
extra : rebase_source : 0ebb65363d5f5813bc7ccb379768df54310d39c1
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 22382e3d65ce8454a1682cfced0d03477762e8fe
This formats the marionette-harness python tests to be a regular |mach python-test| suite. Though
we add subsuite=marionette, this is just for automation purposes. The new preferred way to run the
marionette harness tests locally is:
./mach python-test testing/marionette
They will also run if running the full suite.
The mozbase packages.txt file modifies mozlog to use 'setup.py' instead of 'pth'. The reason for
this is that the marionette-harness tests use the pytest_mozlog pytest plugin for formatting
their results (converts pytest format into something resembling the standard tbpl logging format).
In order for this plugin to get picked up however, mozlog's setup.py file needs to be processed.
MozReview-Commit-ID: Ata99evHxbd
--HG--
extra : rebase_source : 16ed70edd38a53c3279d8632d7cba3df4d5216c3
The name 'source-check' is a bit of a misnomer, because it already includes a bunch
of tasks that are proper unittests, not lints. Some of these unittests will soon
depend on a build task, which makes 'source-check' feel even more wrong.
They still have a lot in common with the lint tasks though, so it's probably not
worth splitting this into two separate kinds. Instead, let's just rename this kind
to 'source-test', which means, any task that tests stuff and is run from the source
directory (instead of a tests.zip). I think both lints and python-tests fall squarely
under this definition.
MozReview-Commit-ID: K0gZ5rVLyeD
--HG--
rename : taskcluster/ci/source-check/doc.yml => taskcluster/ci/source-test/doc.yml
rename : taskcluster/ci/source-check/kind.yml => taskcluster/ci/source-test/kind.yml
rename : taskcluster/ci/source-check/mozlint.yml => taskcluster/ci/source-test/mozlint.yml
rename : taskcluster/ci/source-check/python-tests.yml => taskcluster/ci/source-test/python-tests.yml
rename : taskcluster/ci/source-check/webidl.yml => taskcluster/ci/source-test/webidl.yml
extra : rebase_source : a683b5b6e243849de57f2681993046c776d6a8f2
Framework for defining actions in-tree that can be displayed
and triggered from Treeherder.
MozReview-Commit-ID: 3rvwgy2i4xu
--HG--
extra : rebase_source : beca394f4337aae4ab149e4db810352f57ec4988
Add specification for actions.json to be used as contract
between in-tree logic and Treeherder. Such that Treeherder
can provide actions that callback into the in-tree logic.
MozReview-Commit-ID: JM1ebU8zNK5
--HG--
extra : rebase_source : 289c7c800f214ccde99adfbdf58bba614b957fe6
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: GPFdi0FD6Vn
--HG--
extra : rebase_source : b992786158851f1099aedfce8669a163228edc51
We add the following command line options to Taskcluster try syntax:
--spsProfile - enable profile mode.
--rebuild-talos <N> - retrigger talos tests N times.
--setenv <VAR>=<val> - add extra environments variables.
--tag <TAG> - run tests only the tag TAG.
--no-retry - doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: DMwRjuV2vpf
--HG--
extra : rebase_source : 211ecf52694078986caf290c5b0cca35c775da61
We add the following command line options to Taskcluster try syntax:
--spsProfile: enable profile mode.
--rebuild-talos <N>: retrigger talos tests N times.
--setenv <VAR>=<val>: add extra environments variables.
--tag <TAG>: run tests only the tag TAG.
--no-retry: doesn't retry failed jobs.
We have a chicken-egg problem, as we first generate the full task graph
and then parse the try message. But the graph generation step needs to
know the try message to process the aforementioned options. The
solution is to parse the message before graph generation and then
pass the command line options to the transforms. Then, each transform
can look at the option that interests it and process it accordingly.
The message parse function is configured in kind.yml, which gives some
flexibility for future implementations of alternative syntaxes.
MozReview-Commit-ID: EQlE6q5E8z7
--HG--
extra : rebase_source : 4b7323cd915e8ef9820816015b4b45524811eaf1
This adds `.cron.yml` and a new mach command to interpret it. While
functionality is limited to nightlies right now, there is room to expand to
more diverse periodic tasks. Let your imagination run wild!
MozReview-Commit-ID: KxQkaUbsjQs
--HG--
extra : rebase_source : ddf0a1eadae5a1169c0ead7bcb7b9ce61b255fbf
This makes heavy use of by-test-platform to distinguish the various platforms, since Android
mozharness invocations differ substantially from desktop.
MozReview-Commit-ID: GsZFij6VPTf
--HG--
rename : taskcluster/ci/desktop-test/test-platforms.yml => taskcluster/ci/test/test-platforms.yml
rename : taskcluster/ci/desktop-test/test-sets.yml => taskcluster/ci/test/test-sets.yml
rename : taskcluster/ci/desktop-test/tests.yml => taskcluster/ci/test/tests.yml
extra : rebase_source : f1e98bfa3cdf53db7a5348534875f84cccb24397
This adds a simple layer of indirection to avoid needless repetition of long
lists of tests.
MozReview-Commit-ID: 4hHyoNMwwGy
--HG--
extra : rebase_source : 7731216e9195068a5b000cc894e52d125b63049c
extra : histedit_source : dbe961a08d919284fad42e63c3730a234a3e679b
Phil Ringnalda
Dustin J. Mitchell
Brian Stack
Johan Lorenzo
Gregory Szorc
Wes Kocher
Ted Mielczarek
Andrew Halberstadt
Simon Fraser
Mike Hommey
Tom Prince
Rail Aliiev
Justin Wood
Carsten "Tomcat" Book
Peter Moore
Paul Bignier
Iris Hsiao
Aki Sasaki
Sebastian Hengst
Hammad Akhtar
Jonas Finnemann Jensen
Mihai Tabara
Wander Lairson Costa
CuriousLearner