This uses the run_on_projects attribute introduced earlier for most branches,
adjusts the `ash` method to handle that branch as the legacy implementation
did, and updates try syntax to match builds as well as tests.
In the process, this enables optimizing target tasks, meaning that tasks
specifically requested in the try syntax might be optimized. While this is
probably not ideal, it matches the existing behavior of try (where `-j all` is
the default but all jobs are set to run only when certain files have been
modified). This change can be reverted later, in a more advanced version of
try.
MozReview-Commit-ID: 5FYeUTAsafr
--HG--
extra : rebase_source : b358e0e7cd8a401c50009e63dd55c59489c9b75b
The task description now includes
* flexible specification of index routes (this will get simpler once buildbot
and gecko.v1 routes are removed)
* "run-on-projects", indicating the projects on which this task should run
* "{level}" is allowed in workerTypes
* For the docker-worker/docker-engine worker implementations, "docker-image"
can have the form {in-tree: in-tree-name} to use an in-tree image. This was
previously implemented in the test transforms, but it is useful for other
tasks too!
* Optimizations, currently limited to "only-if-files-changed", can be specified
for each task.
* TreeHerder groupSymbol is optional
* expires-after and and deadline-after have default values (with the former
differing for try and non-try)
* coalesce-name triggers creation of both a coalesce route and a superseder URL
MozReview-Commit-ID: 70vtYs5lz5P
--HG--
extra : rebase_source : 9c557d68239f42466d9724d48ed5bf77648f9aa0
Rename to taskgraph.transforms.task.
This also adds some Required and Optional declarations to the schema to be explicit,
and adjusts the transform to handle treeherder being optional.
MozReview-Commit-ID: FuKYayvlwB9
--HG--
rename : taskcluster/taskgraph/transforms/make_task.py => taskcluster/taskgraph/transforms/task.py
extra : rebase_source : 0913aa8cdf153cd086a7786de957535e9b3a4ee8
This is cosmetic, and makes the cache names easier to generate automatically.
MozReview-Commit-ID: SPwDDnUhx9
--HG--
extra : rebase_source : 2666c8e9073f9598db4146773059065b0d9ba76e
The automatically-generated gecko.v2 routes will remain until a later commit in
this patchset.
MozReview-Commit-ID: A194OvRVWJi
--HG--
extra : rebase_source : f93569efdd6c4978cc42d27d4bbe4f56129ac440
For the most part, this means including the full build_platform (which is not
always the same as build_name) and build_type in the cache name. This makes
the names consistent so they can be generated automatically rather than
specified explicitly.
MozReview-Commit-ID: 7VWm48F1Fzy
--HG--
extra : rebase_source : 3a346b7fd3ed24df75b21a9f1db7e75f8ddfcbbf
MikeLing initially did this in bug 1287018. The intent of this conditional was
to make optimization faster by not even checking most tasks, based on the
assumption that if the prerequisite to a task has changed (for example, a
docker image or a build), then naturally we will want to execute that task.
However, as we have developed actual optimization methods, this has proven not
to be the case: we might want to optimize a test out if its inputs have not
changed, even if a new installer has been built. Similarly, SETA may optimize
tasks out even if their inputs have changed.
MozReview-Commit-ID: LgHET3Z84GB
--HG--
extra : rebase_source : efd297d37bd49dbe655266380641abc258dda725
Version control tends to leave kind directories around containing .orig files
or other such detritus. It's best to just ignore such directories, rather than
failing to generate the taskgraph.
MozReview-Commit-ID: B9GIZndbfi1
--HG--
extra : rebase_source : 58f1debe5219fc52ead2971718da932b63bb06d7
This is currently preventing adding new TaskCluster jobs.
MozReview-Commit-ID: IelDf8g494r
--HG--
extra : rebase_source : 443f65aeaf9487599a619eb103b9671200a5c649
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
It's worth noting that creating the workspace directory is no longer
necessary because it is defined as a VOLUME in the Dockerfile or a
cache in some task configurations. However, it is default owned by
root:root, so we do need to instruct run-task to chown it.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : f4b8f320fafa2da1aa4b06c9cbef76c69acdae8f
extra : intermediate-source : 905fb4a53386c76d931e4a04a31d03739a00a1ff
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
This allows the "regular" build task to be distinguished visually from the gradle build.
MozReview-Commit-ID: 4FgA0QEkUkT
--HG--
extra : rebase_source : 1f8e4014f635d55196c2f32c44a2a56c309c76b6
v1 routes aren't used anymore, as far as I know,
MozReview-Commit-ID: 8t0j8DXUxik
--HG--
extra : rebase_source : ac90e21579c479c334906248958d5d4a2dc28a54
Setting `build_platform` in $inherits.variables and then using it in the body
of the task does not work; the resulting task.extra.build_product was an empty
string
MozReview-Commit-ID: IOxikZStVbq
--HG--
extra : rebase_source : 9381b1291e56161c75c3e7a27261c16fcc1e1ebd
This removes the use of inheritance for these tasks, incorporating some things
that used to be inherited and dropping others. In particular,
* drops index routes
* drops the unused DISABLE_MOZHARNESS
* sets tier explicitly to 1
* uses tooltool for both, even if gcc ignores it
* drops the unused secrets scope
MozReview-Commit-ID: GaMUpHa7PWN
--HG--
extra : rebase_source : a2bcef79cdda65630762d322e0c1897ca0f2cd3d
Also, don't include treeherder info for upload-symbols. This information is
not used, as the tasks have no routes that would cause them to report to
treeherder.
MozReview-Commit-ID: Ji7db0eUL2G
--HG--
extra : rebase_source : 1890bcf94059a3ffd6d8fd1613e3be48d502f27c
This commit does a few things. First, it introduces a property on the
"test_description" schema that, if defined, will cause run-task to
perform a gecko checkout. The presence of the property also configures
the needed scopes and caches.
Second, we introduce the property on web platform test tasks so a
Gecko checkout is present. We also add volumes for the Mercurial
paths to the Docker images. We strictly only need this for
desktop1604-test since WPT tests don't run on desktop-test. However,
desktop-test and desktop1604-test are nearly mirror images of each
other and I feel it is best to keep them in sync.
This commit will make WPT tasks slower on average because they will
need to create a checkout. To add salt to the wound, the checkout
isn't used. However, we need to prove that performing checkouts in
test tasks in automation works at scale. I'd prefer to have this running
for a few weeks and incurring a wall time execution penalty than to
have a giant series of commits backed out because source checkouts
aren't working.
MozReview-Commit-ID: 9UrSWSSmr3w
--HG--
extra : rebase_source : 7b3786f5c612d47dc3b0e165b4abe0c47e8af9ed
Tasks that aren't using run-task may want to configure a task for
optimal VCS setup. Also, having all the code in the same place makes
it easier to keep track of.
MozReview-Commit-ID: GkB2YEUIQiX
--HG--
extra : rebase_source : 8cd3b0164c1ae803530bcafe39075ffd5efa2694
Robust checkout is the preferred method to clone a mercurial repository. This should
speed up the cloning process a bit and reduce storage size.
MozReview-Commit-ID: 92rcwMwRLYN
--HG--
extra : rebase_source : 33a2349d636a07ad1643798bf798bdb175c4f1ca
Without this, we get a 403 when accessing the secret. IRC reviewed.
MozReview-Commit-ID: DlxqpUuW2p6
--HG--
extra : rebase_source : 7431c7a65dcb679803f81123ee59dbe3a04dbf4a
extra : source : 15403ef52183487c9f40c5bcb8a9ce2af0c888c4
extra : histedit_source : 9a5be188eac2687c286644a035dde651cc9170bc
Without this, the "taskcluster" hostname isn't registered. We
will need to talk to taskcluster in a subsequent commit in order
to obtain the hg.mozilla.org fingerprint.
MozReview-Commit-ID: KlyuI4Yk9Ce
--HG--
extra : rebase_source : a85fa4312169df8863d2b41f4f4a23b90758dcd5
Containing the Mercurial 3.9.1 version bump and the change to pin the
hg.mo fingerprint from a TC secret.
MozReview-Commit-ID: LVU7P0LqIvD
--HG--
extra : rebase_source : 8ad46e014ba9840c9972b51ea43e8ccf14492cf2
stylo inherits the defaults, basically.
stylo-try is a copy of try's config minus some builds we don't care about.
MozReview-Commit-ID: GalvxMwh8rv
--HG--
rename : taskcluster/ci/legacy/tasks/branches/try/job_flags.yml => taskcluster/ci/legacy/tasks/branches/stylo-try/job_flags.yml
extra : rebase_source : dbbf4b1e82b9a09eff5f0fdae7535f8317bc42bc
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
One thing run-task isn't doing is created the workspace. So this
functionality has been moved into test-ubuntu1204.sh and
test-ubuntu1604.sh.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : 264efc9b30e62927dece848b9b1eab7542cf2cba
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
extra : histedit_source : a5ae54a810718d446ded8bab510e6cd605562737
Previously, the code performed a "hidden" assignment to
worker['command'] when creating the "command" variable. This
tripped me up when reading the code because relying on variable
references isn't exactly obvious.
We now defer the assignment to worker['command'] until the end,
making the code a little easier to understand.
MozReview-Commit-ID: 23qH4Z3RKY4
--HG--
extra : rebase_source : 52605d7091c73fc47da753bbb298b242f7a3a2ae
extra : histedit_source : 78eb25f963c940956c65fefb2509669a59f78160
Sometimes xvfb will not start up with the current retry/delay settings. This will
attempt to retry more and delay for longer to ensure xvfb has started up. Common
pieces of this have been factored out into a recipe that all docker images can schare
that need this functionality.
MozReview-Commit-ID: BTXkJkBWLZX
--HG--
extra : rebase_source : 7554d7784768a65deb2c9ccb89586e5f77550e9f
Without this, "debug" and "opt" builds share the same cache. This
causes builds to randomly fail in automation.
MozReview-Commit-ID: Fend6GGmhFk
--HG--
extra : rebase_source : c3370938364b1e2cc94081d58ba9f35ce1060e5d
This fixes the use of the bogus pattern `taskcluster/**.py` which does not
actually match python files in subdirectories.
MozReview-Commit-ID: 3UqM5UQ0HOE
--HG--
extra : rebase_source : 73eec18034a11256273589b69b6d95aabceecd03
We've created a new worker type based on c4.4xlarge and m4.4xlarge
instances to replace the existing worker types based on c3.2xlarge,
m3.2xlarge, and r3.2xlarge. The new worker type has 16 vCPU instead
of 8, which makes build tasks complete ~10 minutes faster.
We've also consolidated down into a single pool for workers.
MozReview-Commit-ID: sOfKspFjCS
--HG--
extra : rebase_source : 8c7b030e0d9376dd9c07b71b8987be0c300fb691
This was recently refactored (incorrectly) in 147b245de122
(bug 1290282). Before that refactor, we only defined {{build_name}}
in base_linux64.yml. After, we hard-coded "linux32" and "linux64."
The change to linux64 resulted in ASAN builds sharing the same cache
name. Derp.
This commit restores {{build_name}} to the cache path. And for
consistency, we use {{build_name}} on both 32 and 64-bit variants.
MozReview-Commit-ID: KhqIjzARETO
--HG--
extra : rebase_source : 6b110302b66e052ec3d79bdc5b05a190cb0a7749
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
It's worth noting that creating the workspace directory is no longer
necessary because it is defined as a VOLUME in the Dockerfile or a
cache in some task configurations. However, it is default owned by
root:root, so we do need to instruct run-task to chown it.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : 4b16837ad9362844668fe4d6cde5ed07a59a3ccd
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
Before, we'd open files and feed bytes to yaml.load(). When a str
is fed to yaml.load(), it attempts to guess the encoding. It defaults
to UTF-8 unless somebody set us up the BOM. This is probably OK.
Except if the file isn't valid UTF-8, the exception will be raised
in the bowels of YAML parsing and it may not be obvious the failure
is due to invalid UTF-8 input versus say Python str/unicode
coercion foo.
We change all call sites that load YAML from a file to use
codecs.open() to open the file in UTF-8 and perform UTF-8
decoding/validation at file read time. This should make any UTF-8
failures more obvious. Furthermore, it reinforces that our YAML files
are UTF-8 and not some other encoding.
I discovered this issue as part of trying to get emoji symbols to
render on Treeherder. Unfortunately, it appears pyyaml detects
many emoji as unprintable characters and refuses to load them. This
makes me sad and makes me want to abandon pyyaml/YAML in favor of
something that supports emoji :P
MozReview-Commit-ID: AOvAruZFfnK
--HG--
extra : rebase_source : 4c5d42cb63da5c9ebfac55ab84b4f7f8bcc648fa
Sometimes xvfb will not start up with the current retry/delay settings. This will
attempt to retry more and delay for longer to ensure xvfb has started up. Common
pieces of this have been factored out into a recipe that all docker images can schare
that need this functionality.
MozReview-Commit-ID: 2ww0eT3cIt6
--HG--
extra : rebase_source : f343f8a2bf8370657b63afdd9d739988bbb6261c
We run only a small subset of the test suite to save resources. Over
time, we'll run more of the platform test suite.
--HG--
extra : rebase_source : 264fbe55be5b56857278ae8d731ef51b1b41cf3c
extra : histedit_source : a3550c7e2ea8a41cd6285e511b7268a9e7af11cd%2Cec6024dadcb7c071170aa452064776666283bb27
MozReview-Commit-ID: Lrxi8t53nwy
If a developer adds '--rebuild N' to their try syntax they will get test jobs scheduled N times.
This is useful to determine intermittency rate.
This fixes a regression due to the recent refactoring on how we schedule tasks.
--HG--
extra : rebase_source : 355ca631353015bf63461c194168d753efd6958e
Without this, current umask may influence test results. That was
causing differences between automation and local runs.
MozReview-Commit-ID: 1eu613aBpKB
--HG--
extra : rebase_source : 41c92b9ea795217e715dfa949d3444534aafb7c7
Before, we were returning None, which gets converted to 0.
Derp.
Also fix a flake8 failure introduced by 9f5fbb3066c9.
We'll also need to generate a new decision image. But that will
require someone with TC privileges to be around. That can be done
in a separate commit to unblock this from landing and fixing
consumers of run-task that aren't the decision image.
MozReview-Commit-ID: 6XuoIxjDozF
--HG--
extra : amend_source : 4a3047d09d9bdce5a173a71ec3c68b8c4579a910
Before, we were returning None, which gets converted to 0.
Derp.
Also fix a flake8 failure introduced by 9f5fbb3066c9.
We'll also need to generate a new decision image. But that will
require someone with TC privileges to be around. That can be done
in a separate commit to unblock this from landing and fixing
consumers of run-task that aren't the decision image.
MozReview-Commit-ID: 6XuoIxjDozF
--HG--
extra : rebase_source : 836f4996cb9be24e19504f73792b159f0ffbdbee
Until now Gecko decision task artifacts would expire after a week. Unfortunately, some of these artifacts are
needed in order for Treeherder to determine which tasks it can schedule on a given push.
In this change, we're changing the expiration of all artifacts except those that are related to docker image
optimizations, which, are very large files and we only want to keep for a week.
MozReview-Commit-ID: 96ahyWmoI7W
--HG--
extra : rebase_source : d0f4d65e3b5bfae832a5aaed60c0dfd58f0bdc62
I was looking at some decision task logs and noticed lines like:
Connection pool is full, discarding connection: taskcluster
I also noticed lines like:
Starting new HTTP connection (153): taskcluster
In this case, we had established 153 TCP connections to a server.
Looking at the requests source code, a requests.session by default
creates a connection pool with capacity of 10. There are actually 2
components to the capacity: idle connections and active connections.
What appeared to be happening was we could obtain an idle connection,
use it, and then it would be discarded when put back in the idle
pool because the idle pool was at capacity. Furthermore, it also
appears that connections were sitting around waiting for a TCP
connection.
This commit uses a custom "adapter" with an increased pool size that
matches the concurrency level of the code issuing the HTTP requests.
This should increase the number of concurrent TCP connections / requests,
decreease the number of TCP connections being used overall, and make
decision tasks complete faster.
MozReview-Commit-ID: 6NDbz78TM2y
--HG--
extra : rebase_source : a48e1dae31a5e2af2bc554b544a9019d50dc784f
This commit does a lot. But it's really not too difficult to comprehend
once you focus on the final state, which is basically the same as the
"lint" image and derived tasks.
Before, the "decision" image contained a "checkout-gecko" script and
"run-action" and "run-decision" scripts. The latter 2 invoked the first
script.
The "run-task" script basically does what the combination of these
scripts were doing before. So we switch to it.
While we're here, we also replaced the custom Mercurial installation in
this image with the shared install-mercurial.sh script. The
system-setup.sh script for the decision image is now short and sweet.
The YAML files for tasks using this image have been updated to use
"run-task." We no longer have to pass an environment variable to hold
command arguments. So we revert to putting these arguments inline in
the task's command. Dustin never liked passing the arguments as
environment variables, so it should make him happy ;) We add
--log-no-times because "run-task" prefixes its own timestamps on output
lines.
The path to the Gecko checkout has been changed from
/home/worker/workspace to /home/worker/checkouts to match changes made
in bug 1289643.
Finally, since "checkout-gecko-and-run" is no longer used, we delete it.
The Docker image version has been bumped accordingly.
MozReview-Commit-ID: HAGuCz6668Q
--HG--
extra : rebase_source : 9586873c21d67f8ce869b7ceee221e8fa8775539
Before, we simply executed scripts inside Docker containers. This
frequently resulted in a wall of text with command output. It was
difficult to discern things like the time spent performing certain
actions.
Before, individual tasks had to drop permissions from the default
root user themselves. Dropping permissions isn't exactly a trivial
thing to do and a number of tasks didn't do it or did it wrong.
Before, we had a "checkout-gecko-and-run" script that kinda/sorta
did common activities for us. But it was written as a shell script
and doing advanced things was difficult.
This commit can be treated as a rewrite of "checkout-gecko-and-run"
as a Python script. But it also does a bit more. It prefixes output
with timestamps so we know how long operations took. It features more
robust argument parsing, so we can add new features more easily.
To prove the new wrapper script works, the lint image and all tasks
using it have been converted to use it.
MozReview-Commit-ID: 5d95u5Xebtq
--HG--
extra : rebase_source : 3a1d84782b01d7743e846bd0c04d7867813dd8a3
Before, it ignored mach logging settings passed via command line
arguments.
MozReview-Commit-ID: CNckQHrOdX9
--HG--
extra : rebase_source : b4b190e8e2d073e8932799c8d8bedf49b5f3cfd8
Use the source RIDEALONG_BUILDS value in the module under test so that changes
to that variable do not cause the test to fail.
MozReview-Commit-ID: EfHQ7baBziB
--HG--
extra : rebase_source : 1c52bf62709236db14a3ce318495891a2eb274f4
The existing hash was for an empty tarfile. Oops!
MozReview-Commit-ID: 1KOZxnDmoOH
--HG--
extra : rebase_source : 5d4db299dba80f98ba0383e88a1f4cfcb1dbcc70
Now that Docker image building is called from Python, we can start to
do advanced stuff with it.
With this commit, we switch from building Docker images directly from
the source directory ("the Docker way") to using our custom Docker image
build contexts.
The main advantage of this is that locally-built Docker images can now
use our custom Dockerfile syntax to include extra files in the build
context!
The code for building a Docker image from a context has been extracted
to its own standalone function. I have nefarious plans for this in the
future, such as the ability to override the FROM syntax to specify
URLs of images. This would allow us to host base images on our own
server, which removes a dependency on Docker Hub and improves
determinism, since images on Docker Hub change all the time.
MozReview-Commit-ID: 5lTdV8yEHkc
--HG--
extra : rebase_source : c374558b82d0d0302351ffbf3c82878c6663f40c
build.sh had been reduced to invoking `docker`. We move that invocation
to Python and remove build.sh. Long live build.sh!
MozReview-Commit-ID: FQBDJv4HSaU
--HG--
extra : rebase_source : 82979f2526ab1ab6dd5c25032b8d81074c797d94
We already had code for resolving the image registry and tag. We
refactored it slightly to be more useful then changed build.sh to
accept the tag as an argument.
At this point, build.sh is basically a wrapper around `docker`. But
there's a special case for executing custom "build.sh" files we
need to eliminate first...
MozReview-Commit-ID: A9HVvxgCdG2
--HG--
extra : rebase_source : 30a408860aea619813f32723fe960d1224b5dbc7
Now that we have a mach command and Python code for doing Docker image
building, we can start moving code from build.sh to Python.
We start with searching for and validating the `docker` binary works.
MozReview-Commit-ID: 2DCc3b8UyZ3
--HG--
extra : rebase_source : ce5ad9dc39eea1fae97aec93eaefbc2c169ec313
extra : amend_source : 806411d3777c657b21d33bbb0e18e06de95716ce
Docker image building will soon need to use Python in order to produce
the image build context archive.
As the first step towards this, we introduce a Python function that
calls out to build.sh. We also implement a mach command that calls it
so we can test the functionality.
I'm not keen about introducing a new mach command. I'd prefer to have
a sub-command instead. I'm not sure what all uses
`mach taskcluster-load-image`. Perhaps we could make a `taskcluster`
top-level command. Or perhaps we could fold these image commands into
`mach taskgraph`? Either way, the mach side of this isn't terribly
important to the commit series: most of the code will live inside a
Python module outside of mach.
MozReview-Commit-ID: AI8p6H4psNh
--HG--
extra : rebase_source : a3a18cbeb05152ab65f2bf152fd2517bbcd8981e
This abstracts out the process of running transforms as specified in kind.yml,
with some useful default behavior that can be overridden by subclasses.
MozReview-Commit-ID: 8pY4IYaN7NU
--HG--
extra : rebase_source : 66711546879edacc5757197674b5bd3f643a0093
extra : source : 9930bede24bcda58c40ca0ee9085aedc00e8ec1b
Previously, the VCS checkout cache would be optimized away on Try. This
meant every task on Try had to create its own checkout from scratch. This
meant writing ~150,000 files. By allowing use of the cache, we minimize
the amount of work tasks need to perform on Try and therefore make tasks
complete faster, on average.
MozReview-Commit-ID: EWuic8lFVtv
--HG--
extra : rebase_source : fd1096da52b499064ab3c0962a4321e8d4f92bb9
Currently, TaskCluster tasks tend to use the "workspace" directory as
a cache that manages the source checkout *and* additional state.
Historically at Mozilla, we've lumped "source checkout" and "workspace"
(sometimes known as an "objdir") into the same directory. This is
not ideal. Ideally, there is an immutable, read-only source checkout
and all files produced from that source live in a separate directory.
In this commit, the "workspace" directory for the "lint" image has been
renamed to "checkouts" and all tasks using the image have been updated
accordingly. By having "checkout" in the name, we clearly identify this
cache as being relevant to source checkouts, which IMO can serve a
different role from "workspaces." This distinction is important, as the
next commit will prevent the "checkouts" cache from getting optimized
out in certain tasks.
To hammer this point home, documentation on common caches has been
introduced.
MozReview-Commit-ID: BSEc4dM5YCt
--HG--
extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
All the other expressions are anchored via ^ and $.
MozReview-Commit-ID: BkCbZZUnwK6
--HG--
extra : rebase_source : 0dc6c7577898e447946217494e5cd491b13e22fa
Currently, the gecko-decision and action tasks maintain a separate cache
for each "project" (Firefox repo) source checkout. In my opinion, this
is not necessary.
Separate caches/source checkouts mean a lower chance for a cache hit:
a lower chance that any given worker will have a source checkout readily
available for the "project" being built. They also mean workers store
more data and files in caches.
By using a shared cache for source checkouts, workers will ideally
only have a single source checkout cache and will be pretty much
guaranteed to have a checkout readily available (the main exception
being the first task on a freshly-provisioned worker).
A downside of a shared cache is that workers may spend more time doing
`hg update` operations. e.g. if the last task was from mozilla-release
and the current task is from mozilla-central, thousands of files will
be updated. However, I believe we will still be better off because
separate caches *increase* the probability for a full checkout, which
involves writing ~150,000 files! So I believe this change will lower
the overall amount of VCS work done by workers while possibly
sacrificing the lowest possible execution time (which is achieved by
minimizing the `hg update` work).
It's worth noting that `hg robustcheckout` (which is used by these
tasks) does the equivalent of an `hg purge --all`. This ensures that all
untracked and ignored files from previous tasks are deleted. So there
should be no contamination from previous tasks.
MozReview-Commit-ID: I5Ng3xFEfou
--HG--
extra : rebase_source : db423e5005d1e26d7dfebbbfa28893fc32caef6f
There are some packages like 'requests' that are bundled in the mozharness venv, but not in the test
package. It would be easy to manually add these to sys.path in the mach bootstrap script, but it's
much nicer to simply activate this virtualenv in the first place.
MozReview-Commit-ID: 8xeJEIgUbLj
--HG--
extra : rebase_source : b87e5ef46041d9d5a89d419e97fe3a316de6c8c2
DONTBUILD because it's not worth it.
--HG--
extra : rebase_source : 1cea5d6121fda41fc668bf24990734abf4a56677
extra : amend_source : 318ec9edcf8072ffe1840b5f7d8ce144be63424f
This removes the in-tree plugin from the tooltool archive and uses that code
directly from the Gecko checkout instead.
For automation, we now get ESLint and external plugins from tooltool and then
symbolic link to the in-tree plugin.
For local development, we install ESLint and external plugins following the
shrinkwrap file created from the last change to the tooltool archive. The local
plugin is then installed.
This change also removes the list of module versions from mach_commands.py, so
there is only one place to update module versions for the future.
MozReview-Commit-ID: AhbZ8lVPmN4
The first thing we do is switch from the desktop-test image to the
lint image: the mozharness tests don't need anything in the
desktop-test image. This does mean converting from Ubuntu 12.04
to 16.04. I'm fine upgrading to a newer operating system by
4 years :)
Once we're on the lint image, we switch to use checkout-gecko-and-run.
The mozharness-gecko task now does version control just like
eslint-gecko and mozlint-flake8.
MozReview-Commit-ID: 6mzAZ7LSBWl
--HG--
extra : rebase_source : a766e5db61998d1cfbcb792ad09a265f45027f54
MozReview-Commit-ID: KAqZlUjMXdQ
***
Bug 1289122 - bugfix to chain together the elements to be removed. r=dustin
MozReview-Commit-ID: 3XhHZgmq6e3
--HG--
extra : rebase_source : 2d0ac3710b4ce1376db17a50c6b3bc4354da3ea4
Normally we start Xvfb as a background task, then run the tests from the
same script. However, in interactive mode we were starting Xvfb, the script
would exit, and then we would potentially run tests later on from another
script. Unforunately this meant that Xvfb was dying with the first script
and tests would fail.
This patch runs Xvfb in a screen session so that it will still be available
later on when running an interactive shell.
MozReview-Commit-ID: EduVyglo2BG
--HG--
extra : rebase_source : 6e2c40ee16f80792f038fd581168e181a2c4bf51
Previously, every lint task would have to create its own checkout. This
was time consuming.
The robustcheckout extension purges the working copy of *all*
untracked and ignored files. It also restores modified files to
their original state. In other words, as long as you trust Mercurial
to go from revision X to revision Y, robustcheckout is as good as
a fresh checkout.
This commit adds a cache for the working directory checkout so lint
tasks only have to effectively perform incremental `hg update` between
task executions. This should make tasks spend a lot less time
doing version control foo.
On Try, time for flake8 tasks is currently hovering around 4 minutes.
After this change, I've seen tasks finish as quickly as 11s! But that
was with a hacked up legacy.py that added the workspace cache to the
whitelist for Try. While I would like to see workspace reuse on Try
eventually, this is not the right commit to roll that out in.
MozReview-Commit-ID: 66P2rt896qE
--HG--
extra : rebase_source : 2a7c8d396e85ba4eae84f8843256050a2288c9d0
extra : intermediate-source : c478ecb14bf6164ef0c955acee3a0a2f18e415c5
extra : source : 63d54efdc1f8effb0370644c11014a3f0404073b
extra : histedit_source : 560701ce41c870171443c78e3a9de3998fbf9306
The robustcheckout Mercurial extension does a clone+checkout optimally.
Read the bug for more on it.
robustcheckout is already used by mozharness automation. It has resulted
in a significant reduction in I/O usage and utilization in automation.
This commit replaces tc-vcs with the robustcheckout equivalent.
We replace the existing tc-vcs scope and cache with a new one.
Because Dustin and I are paranoid, we maintain separate caches per
SCM level - even though we could arguably share the same cache. Defense
in depth.
Robustcheckout (when used with --sharebase) pools storage for related
repos automatically. i.e. changesets from inbound and central will
be in the same store. This means you likely only have one copy of
each changeset per cache. This can result in significant space savings.
And, since there are fewer copies floating around, hg.mozilla.org
and various network appliances are working less too!
Since tc-vcs is no longer used, we stop it from being installed.
While we're here, we also change the images to execute as the
"worker" user. This happens automatically as a result of using
the "checkout-and-run" script.
MozReview-Commit-ID: EDeebuP7TkT
--HG--
extra : rebase_source : 2bec5dd9d6fe5565831bb35f195859aa12dd0bf2
extra : intermediate-source : 06481d97a485f6566554b087bc3880d76361e8ec
extra : source : d368700c93ef085325a081219d7aeb8512bc54a1
extra : histedit_source : c07505273fc8f10acf8e8d3ee01e327afd0aa63d
https://hg.mozilla.org/mozilla-unified contains heads from all the
major Firefox repos. In addition, it is encoded on the server in such a
way that it is several hundred megabytes smaller despite containing
30,000+ more changesets. This means faster clones and faster
operations.
Cloning this repo does require a new version of Mercurial. So if any
TC tasks are using an ancient Mercurial - one that is vulnerable to
known CVE issues in fact - this will flush them out.
MozReview-Commit-ID: 2VHDa6FEeeJ
--HG--
extra : rebase_source : db4990bcde0503fd14d82a5d16c71adbb4f92be3
extra : histedit_source : 6c555bd3df12536d1c48b45d8cc76611e3f7032f
Changes to the decision Docker image have been compelted. We're ready to
use the new image.
We tag the image, update version references, change the task caches
so the new Mercurial pooled storage from the robustcheckout extension is
used, and convert the decision tasks to run as the "worker" user.
MozReview-Commit-ID: 61v9Ivy59zG
--HG--
extra : rebase_source : 640318a87660950c5e0680867a1bfdd68e35f127
extra : histedit_source : ec53fc576c00e5f2053167b37544ac7afccaecb5
The decision task configures a /home/worker/workspace cache. However,
the command we run in the container references a "workspace" relative
path. From logs in automation, it appears that PWD during execution
is "/" because "workspace" is being resolved to "/workspace." The
net result of this is we appear to be performing a VCS clone+checkout
on every single task.
This commit fixes the paths so our workspace cache is actually used.
MozReview-Commit-ID: Kj6REep5bSs
--HG--
extra : rebase_source : 8cd8be43dfd34f2970b47721c3da8e3957a8bfed
extra : histedit_source : b81dce523a88e44eb3fa8b1a68840066edca382d
These scopes will be introduced in subsequent commits. We don't want
them to get removed, just like we don't want the tc-vcs caches to get
removed.
MozReview-Commit-ID: 790SzpAUxE4
--HG--
extra : rebase_source : d8148aa38bad09468f67ded1a64f432e2e30c63a
extra : source : 923b86a15d91cb22999a7c94a4a708845d23aa9d
extra : histedit_source : 6d85127fdcc2e795a54eeb836ce11f158e5e88cd
The function was only used once and was providing little to no value.
A test of this function has been removed. Tests for the lower-level
context creation function are sufficient.
MozReview-Commit-ID: D9EhmZQlqG5
--HG--
extra : rebase_source : 4b3faa0fc5f085c1c77fe5636744946a6d442b05
A limitation of traditional docker build context generation is it
only includes files from the same directory as the Dockerfile. When
repositories have multiple, related Dockerfiles, this limitation
results file duplication or putting all Dockerfiles in the same
directory (which isn't feasible for mozilla-central since they would
need to be in the root directory).
This commit enhances Dockerfiles to allow *any* file from the
repository checkout to be ADDed to the docker build context.
Using the syntax "# %include <path>" you are able to include paths
or directories (relative from the top source directory root) in the
generated context archive. Files add this way are available under the
"topsrcdir/" path and can be ADDed to Docker images.
Since context archive generation is deterministic and the hash of
the resulting archive is used to determine when images need to be
rebuilt, any extra included file that changes will change the hash
of the context archive and force image regeneration.
Basic tests for the new feature have been added.
MozReview-Commit-ID: 4hPZesJuGQV
--HG--
extra : rebase_source : 99fae9fe82102126fbee879c134981047bb4a601
This restores order to only having a single hash for a context
directory.
Using a tempfile here is a bit unfortunate. It can be optimized later,
if needed.
MozReview-Commit-ID: LMNsvt3fDYx
--HG--
extra : rebase_source : 8c2b70164aed6d744a71d170d0324797e755cbaf
Now that the context tar creation function is standalone and doesn't
rely on external state, we can start unit testing it easier.
We establish a basic unit test that verifies the function works as
advertised and that output is deterministic.
MozReview-Commit-ID: H4MY28PiHSN
--HG--
extra : rebase_source : 692a5e3d5af6edd14b3d4ceb7c90cd1e0344052f
Relying on global variables like GECKO is a bit dangerous. To facilitate
testing of archive generation in subsequent commits, let's pass an
path into this function.
The argument is currently unused.
MozReview-Commit-ID: Et1UYraflDP
--HG--
extra : rebase_source : 012095bc2450c72467f3f65f71bcb4cf6efcde66
Now that tar file generation is deterministic, we can use the hash
of the created archive rather than the hash of the files that are
(presumably) in the archive.
This temporarily breaks consistent hashing by using independent
hashing mechanisms. This will be cleaned up in a subsequent commit.
MozReview-Commit-ID: CWooVGfDKZO
--HG--
extra : rebase_source : 86923635f960e23f1483068bf809dec7aeacefd1
We recently implemented code in mozpack for performing deterministic
tar file creation. It normalizes things like uids, gids, and mtimes
that creep into archives.
MozReview-Commit-ID: 1tn5eXkqACQ
--HG--
extra : rebase_source : 6b069a3a50c9103ae4f6185b26d6a37658179f42
Upcoming commits will refactor how context tarballs are created. In
preparation for this, we establish a standalone function for creating
context tarballs and refactor docker_image.py to use it.
MozReview-Commit-ID: KEW6ppO1vCl
--HG--
extra : rebase_source : b81decf9ca14ff0216514f47419e96eb57d6f851
This removes the unnecessary setting of c-basic-offset from all
python-mode files.
This was automatically generated using
perl -pi -e 's/; *c-basic-offset: *[0-9]+//'
... on the affected files.
The bulk of these files are moz.build files but there a few others as
well.
MozReview-Commit-ID: 2pPf3DEiZqx
--HG--
extra : rebase_source : 0a7dcac80b924174a2c429b093791148ea6ac204
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
MOZ_NODE_PATH should not be set for Android tests. Rather than setting it to
the literal "None", this omits the environment variable on that platform.
Only `fx_docker_desktop_generic.yml` (and now `fx_docker_android_generic.yml`)
inherit from `fx_docker_test_base.yml`. And `fx_android-api-15_generic.yml` is
the only Android-related YAML file to inherit from
`fx_docker_desktop_generic.yml`.
MozReview-Commit-ID: 7BhGXk9c8ir
--HG--
extra : rebase_source : 8190d2f6f2368acd1c9c11a3e16062e4589a23ac
This makes the two copies of the suite consistent.
MozReview-Commit-ID: FhxzwQlERnD
--HG--
extra : rebase_source : e13291d9d852521cd9af9de79addc58ccfe62949
extra : source : 85f75a951f93fb7105cd9613ed98d041575c6a35
For all of ther suites, the -e10s is attached to the group symbol, not the job
symbol.
MozReview-Commit-ID: BZcRCnEMy5U
--HG--
extra : rebase_source : 6b0b294d8ece3b325fd3af43a60f994427b9ddd8
extra : source : 05714f5e19e6353aaf1f96540e70ba2385bbee89
This makes the instance size consistent for the whole suite
MozReview-Commit-ID: 8s6n6BwXSjB
--HG--
extra : rebase_source : 87744db0ff2122cde214e22dc356af3ee3b87503
extra : source : 46e29447f7f31124baae8008ec0033b1b3dddbe0
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
One thing run-task isn't doing is created the workspace. So this
functionality has been moved into test-ubuntu1204.sh and
test-ubuntu1604.sh.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : 75a937321d1850caebbb1eeaab42d04638ef75d9
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
Previously, the code performed a "hidden" assignment to
worker['command'] when creating the "command" variable. This
tripped me up when reading the code because relying on variable
references isn't exactly obvious.
We now defer the assignment to worker['command'] until the end,
making the code a little easier to understand.
MozReview-Commit-ID: 23qH4Z3RKY4
--HG--
extra : rebase_source : 44e8a2ddd7b229221d76710c4394618eb13ef348
In the `python-test` mach command and the mozharness script for
the Marionette harness tests, use the vendored-in Pytest
instead of installing from pip.
Add the Marionette harness test requirements file to the
file_patterns in the definition of the marionette-harness taskcluster
job, as changes to the requirements should trigger the job to run.
MozReview-Commit-ID: J5pln2WB4GY
--HG--
extra : rebase_source : 5144ccfabb84eb0da244b24f6d27b59ae183c174
looks like we have to be more explicit with each build and suite compared to
https://bugzilla.mozilla.org/show_bug.cgi?id=1243024
but the way we bump this to tier 1 will likely change as
https://bugzilla.mozilla.org/show_bug.cgi?id=1281004 lands
as per dev.planning announcement and coop's email, we are going to try making
tc tier 2 jobs more critical for less critical jobs, e.g. debug builds, by
making them tier 1.
MozReview-Commit-ID: 4IwNCjOHo7x
--HG--
extra : source : 0b6844851660c49267f74d589345ebdaeb304db6
extra : amend_source : cd1507f8d9c6be75c68929051bbf8ee30cfe9cee
note, as bug 1281004 is about to land, I might require a new patch as we move
away from legacy towards da futures!
this yielded:
current bbot opt routes:
"index.gecko.v2.try.revision.f40f15f50508b78e369c8ac5e6a8743bcd064193.mobile.android-api-15-opt",
"index.gecko.v2.try.pushdate.2016.07.11.20160711204636.mobile.android-api-15-opt",
"index.gecko.v2.try.latest.mobile.android-api-15-opt",
"index.buildbot.branches.try.android-api-15",
"index.buildbot.revisions.f40f15f50508b78e369c8ac5e6a8743bcd064193.try.android-api-15"
my patch tc opt routes:
"index.gecko.v1.try.revision.linux.f40f15f50508b78e369c8ac5e6a8743bcd064193.android-api-15.opt",
"index.gecko.v1.try.latest.linux.android-api-15.opt",
"index.buildbot.branches.try.android-api-15",
"index.buildbot.revisions.f40f15f50508b78e369c8ac5e6a8743bcd064193.try.android-api-15",
"tc-treeherder.v2.try.f40f15f50508b78e369c8ac5e6a8743bcd064193.133427",
"tc-treeherder-stage.v2.try.f40f15f50508b78e369c8ac5e6a8743bcd064193.133427",
"index.gecko.v2.try.revision.f40f15f50508b78e369c8ac5e6a8743bcd064193.mobile.android-api-15-opt",
"index.gecko.v2.try.pushdate.2016.07.11.20160711204636.mobile.android-api-15-opt",
"index.gecko.v2.try.latest.mobile.android-api-15-opt"
current bbot debug routes:
"index.gecko.v2.try.revision.f40f15f50508b78e369c8ac5e6a8743bcd064193.mobile.android-api-15-debug",
"index.gecko.v2.try.pushdate.2016.07.11.20160711204636.mobile.android-api-15-debug",
"index.gecko.v2.try.latest.mobile.android-api-15-debug",
"index.buildbot.branches.try.android-api-15-debug",
"index.buildbot.revisions.f40f15f50508b78e369c8ac5e6a8743bcd064193.try.android-api-15-debug"
my patch tc debug routes:
"index.gecko.v1.try.revision.linux.f40f15f50508b78e369c8ac5e6a8743bcd064193.android-api-15.debug",
"index.gecko.v1.try.latest.linux.android-api-15.debug",
"index.buildbot.branches.try.android-api-15-debug",
"index.buildbot.revisions.f40f15f50508b78e369c8ac5e6a8743bcd064193.try.android-api-15-debug",
"tc-treeherder.v2.try.f40f15f50508b78e369c8ac5e6a8743bcd064193.133427",
"tc-treeherder-stage.v2.try.f40f15f50508b78e369c8ac5e6a8743bcd064193.133427",
"index.gecko.v2.try.revision.f40f15f50508b78e369c8ac5e6a8743bcd064193.mobile.android-api-15-debug",
"index.gecko.v2.try.pushdate.2016.07.11.20160711204636.mobile.android-api-15-debug",
"index.gecko.v2.try.latest.mobile.android-api-15-debug"
so all looks well and as a bonus, the index.buildbot routes now match too.
MozReview-Commit-ID: 5HilJOpONst
--HG--
extra : rebase_source : 24a3c895681284a8dca16cbf3b2a47b66eaa1f08
extra : amend_source : b8b8b91c9787e26a6dd6d54fbe39cb9cb515056b
This fixes a race condition between the 'test-linux.sh' process and the 'taskcluster-interactive-shell'
process in interactive tasks.
MozReview-Commit-ID: GhqKpq5pAtj
--HG--
extra : rebase_source : 67b756d0373432404a4f7cc928bac09fc3f82e8a
This makes the two copies of the suite consistent.
MozReview-Commit-ID: FhxzwQlERnD
--HG--
extra : rebase_source : 5400f06ac4c75d13c2cd5f1c81beac92eadb63bb
extra : source : 85f75a951f93fb7105cd9613ed98d041575c6a35
For all of ther suites, the -e10s is attached to the group symbol, not the job
symbol.
MozReview-Commit-ID: BZcRCnEMy5U
--HG--
extra : rebase_source : 2c8d973d224c91ae844bbb99248a736437777d80
extra : source : 05714f5e19e6353aaf1f96540e70ba2385bbee89
This makes the instance size consistent for the whole suite
MozReview-Commit-ID: 8s6n6BwXSjB
--HG--
extra : rebase_source : b0f252bc975316db7f77c9a3893f85c2d18c6e06
extra : source : 46e29447f7f31124baae8008ec0033b1b3dddbe0
This makes the two copies of the suite consistent.
MozReview-Commit-ID: FhxzwQlERnD
--HG--
extra : rebase_source : 3183b94c946b324173f46ef2185323c3b00b86bc
For all of ther suites, the -e10s is attached to the group symbol, not the job
symbol.
MozReview-Commit-ID: BZcRCnEMy5U
--HG--
extra : rebase_source : 10b0c265f06047738aef053c9825399c121daea6
This makes the instance size consistent for the whole suite
MozReview-Commit-ID: 8s6n6BwXSjB
--HG--
extra : rebase_source : 8e7f857ec1ec583d68efe7fbdba263222ced4e0a
When running an interactive worker (aka one-click loaner), developers have the option of setting
up the mozharness environment without running tests. When they do this, we should automatically
symlink the mach binary found in the tests.zip to their path. This way developers don't need to
go searching for $HOME/workspace/build/tests/mach to run their tests.
MozReview-Commit-ID: 1JKPYSsYKlN
--HG--
extra : rebase_source : 1b9bd2a201641fee168778268c3612020d7ee868
Dustin J. Mitchell
Chris Manchester
Armen Zambrano Gasparnian
Gregory Szorc
Rob Thijssen
Andrew Halberstadt
Justin Wood
Ryan VanderMeulen
Nick Alexander
Jordan Lund
amiyaguchi@mozilla.com
Henrik Skupin
Wes Kocher
Maja Frydrychowicz
Kim Moir
Phil Ringnalda
Matthew Noorenberghe
Andrew Comminos
Geoff Brown
aselagea
Joel Maher
Gregory Arndt
Steve Fink
Carsten "Tomcat" Book
Ehsan Akhgari
James Graham
Iris Hsiao
Alexandre Lissy
J. Ryan Stinnett
Mihai Tabara
Tom Tromey
Kalpesh Krishna
Anjana Vakil
Peter Moore