This was regressed by bug 1296397 which stopped setting the GECKO_HEAD_REPOSITORY and
the like. It wasn't caught because the task doesn't actually depend on those environment
variables except when using the interactive loaner.
This patch supports vcs checkout regardless of whether the task runs from source or not.
MozReview-Commit-ID: CDxdG8G6JDd
--HG--
extra : rebase_source : 8a3d380c86bbfabe6d76854fd8af8e794e75970e
The lint jobs have been running for many weeks and have been stable and
useful throughout. It is time to make them tier 1 tasks.
MozReview-Commit-ID: 5i5KOPhMNKM
--HG--
extra : rebase_source : 556c11aec6ff152c2c8051d8f4a015c9e6fb22b7
In this case, the tasks must have the same schedulerId as the existing task,
so this is calculated using parameters['level'].
MozReview-Commit-ID: G8EE2kvFstT
--HG--
extra : rebase_source : 214885da9b58520727d5f80b9a31bb8a206f9279
These builds can be run on taskcluster to obtain per-test (JSDebugger) code coverage with the linux64-jsdcov build and overall (GCOV) code coverage with the linux64-ccov build. The linux64-jsdcov build also needed to have leak checking disabled for debug mode.
MozReview-Commit-ID: ASgrU2X7RQV
--HG--
extra : rebase_source : b2098e4d01039edd6cff37f3e6a26c2ed3d3d3ba
Currently, build tasks produce a zip file artifact containing the
content of testing/mozharness. This zip file is downloaded and applied
by any test using mozharness (which is most of them).
Now that some tests have a source checkout, we don't need to download
a mozharness zip because we can just use the source checkout.
This commit teaches test-ubuntu.sh to accept a custom path to
mozharness files. If that path is defined, we skip downloading the
mozharness zip.
The taskgraph has been updated to pass the path of mozharness in the
checkout to test-ubuntu.sh when a source checkout is available.
MozReview-Commit-ID: 2P17Wx1oytI
--HG--
extra : rebase_source : a631a87737b253aece3bbc550ce58168212ae87c
extra : source : 21fdf73bbb17e34cfe00e372695c4f21e4ba3e6a
Same deal as the last commit except for config paths.
MozReview-Commit-ID: 3AXwc3i3Iqu
--HG--
extra : rebase_source : 588020b22853f669d9b9ec1b368d82f4b6f14a97
extra : source : dda84d1fb12b3416c81aab675c58b95ecb9f37cd
Mozharness scripts all live in "mozharness/scripts." So, we don't need
to redundantly define this path. Let's define the script as the relative
path within "mozharness/scripts" instead.
IMO this cosmetic improvement can stand on its own. However, the real
reason for this change is to pave the way for running mozharness from
a source checkout instead of downloading a zip archive of it.
MozReview-Commit-ID: FUnMiLlCcGK
--HG--
extra : rebase_source : 8e11f98dc2f63fa7d7fbd893d68e3c6d475a5759
extra : source : 11ffdc56f5becfeb2ee03423f67749f4baf1f3a7
The only difference in these files was the order that pulseaudio is
started and whether compiz is started. We rename test-ubuntu1604.sh
to test-ubuntu.sh, add some distro release detection, and add
some conditional branches so it works on both Ubuntu 12.04 and 16.04.
MozReview-Commit-ID: CaSfuDxss3d
--HG--
rename : taskcluster/scripts/tester/test-ubuntu1604.sh => taskcluster/scripts/tester/test-ubuntu.sh
extra : rebase_source : 29040c1cf7baedda0aaeff4bd788d5d400c127f1
extra : source : f227ea4d52fed84e3e682de0aa4cb8869539d645
Currently, build tasks produce a zip file artifact containing the
content of testing/mozharness. This zip file is downloaded and applied
by any test using mozharness (which is most of them).
Now that some tests have a source checkout, we don't need to download
a mozharness zip because we can just use the source checkout.
This commit teaches test-ubuntu.sh to accept a custom path to
mozharness files. If that path is defined, we skip downloading the
mozharness zip.
The taskgraph has been updated to pass the path of mozharness in the
checkout to test-ubuntu.sh when a source checkout is available.
MozReview-Commit-ID: 2P17Wx1oytI
--HG--
extra : rebase_source : 5d97079f054f3cd236a1c2a3039c3894207377f9
Same deal as the last commit except for config paths.
MozReview-Commit-ID: 3AXwc3i3Iqu
--HG--
extra : rebase_source : c249f77289e24aceaae73dbca0c994b949215814
Mozharness scripts all live in "mozharness/scripts." So, we don't need
to redundantly define this path. Let's define the script as the relative
path within "mozharness/scripts" instead.
IMO this cosmetic improvement can stand on its own. However, the real
reason for this change is to pave the way for running mozharness from
a source checkout instead of downloading a zip archive of it.
MozReview-Commit-ID: FUnMiLlCcGK
--HG--
extra : rebase_source : 3a0093f1995120264fed89dc4af94eea3554bfe5
The only difference in these files was the order that pulseaudio is
started and whether compiz is started. We rename test-ubuntu1604.sh
to test-ubuntu.sh, add some distro release detection, and add
some conditional branches so it works on both Ubuntu 12.04 and 16.04.
MozReview-Commit-ID: CaSfuDxss3d
--HG--
rename : taskcluster/scripts/tester/test-ubuntu1604.sh => taskcluster/scripts/tester/test-ubuntu.sh
extra : rebase_source : 2153d24fbf8208851a6df8728b8a820166278751
These builds can be run on taskcluster to obtain per-test (JSDebugger) code coverage with the linux64-jsdcov build and overall (GCOV) code coverage with the linux64-ccov build. The linux64-jsdcov build also needed to have leak checking disabled for debug mode.
MozReview-Commit-ID: ASgrU2X7RQV
--HG--
extra : rebase_source : af40a6e582665ffcb575092586731f595a362ae4
Set up tooltool to actually use its cache, and remove the unused tc-vcs cache
scope
MozReview-Commit-ID: H1VLxryQPEM
--HG--
extra : rebase_source : b52de7f5a10e5fb4576872bf3169b2faf87ded32
Secrets are accessed via http://taskcluster/secrets, and the hazard scripts
show no evidence of such calls.
MozReview-Commit-ID: 6xp6D2ybwap
--HG--
extra : rebase_source : 5bfe02773f26768c7e79621a43515d6bf8b921ca
This leaves a lot of room for improvement, but gets the job done for now.
MozReview-Commit-ID: HmeTxZjOfOb
--HG--
extra : rebase_source : b6f55134c25cb670409e906da1acc7c0bd8d0dc0
This has some notes for future work on the task, but will work for now.
MozReview-Commit-ID: 7J4tQeKj3KJ
--HG--
extra : rebase_source : d171b5393fa340a47927e8b89ece93392ff23e56
This uses the run_on_projects attribute introduced earlier for most branches,
adjusts the `ash` method to handle that branch as the legacy implementation
did, and updates try syntax to match builds as well as tests.
In the process, this enables optimizing target tasks, meaning that tasks
specifically requested in the try syntax might be optimized. While this is
probably not ideal, it matches the existing behavior of try (where `-j all` is
the default but all jobs are set to run only when certain files have been
modified). This change can be reverted later, in a more advanced version of
try.
MozReview-Commit-ID: 5FYeUTAsafr
--HG--
extra : rebase_source : b358e0e7cd8a401c50009e63dd55c59489c9b75b
The task description now includes
* flexible specification of index routes (this will get simpler once buildbot
and gecko.v1 routes are removed)
* "run-on-projects", indicating the projects on which this task should run
* "{level}" is allowed in workerTypes
* For the docker-worker/docker-engine worker implementations, "docker-image"
can have the form {in-tree: in-tree-name} to use an in-tree image. This was
previously implemented in the test transforms, but it is useful for other
tasks too!
* Optimizations, currently limited to "only-if-files-changed", can be specified
for each task.
* TreeHerder groupSymbol is optional
* expires-after and and deadline-after have default values (with the former
differing for try and non-try)
* coalesce-name triggers creation of both a coalesce route and a superseder URL
MozReview-Commit-ID: 70vtYs5lz5P
--HG--
extra : rebase_source : 9c557d68239f42466d9724d48ed5bf77648f9aa0
Rename to taskgraph.transforms.task.
This also adds some Required and Optional declarations to the schema to be explicit,
and adjusts the transform to handle treeherder being optional.
MozReview-Commit-ID: FuKYayvlwB9
--HG--
rename : taskcluster/taskgraph/transforms/make_task.py => taskcluster/taskgraph/transforms/task.py
extra : rebase_source : 0913aa8cdf153cd086a7786de957535e9b3a4ee8
This is cosmetic, and makes the cache names easier to generate automatically.
MozReview-Commit-ID: SPwDDnUhx9
--HG--
extra : rebase_source : 2666c8e9073f9598db4146773059065b0d9ba76e
The automatically-generated gecko.v2 routes will remain until a later commit in
this patchset.
MozReview-Commit-ID: A194OvRVWJi
--HG--
extra : rebase_source : f93569efdd6c4978cc42d27d4bbe4f56129ac440
For the most part, this means including the full build_platform (which is not
always the same as build_name) and build_type in the cache name. This makes
the names consistent so they can be generated automatically rather than
specified explicitly.
MozReview-Commit-ID: 7VWm48F1Fzy
--HG--
extra : rebase_source : 3a346b7fd3ed24df75b21a9f1db7e75f8ddfcbbf
MikeLing initially did this in bug 1287018. The intent of this conditional was
to make optimization faster by not even checking most tasks, based on the
assumption that if the prerequisite to a task has changed (for example, a
docker image or a build), then naturally we will want to execute that task.
However, as we have developed actual optimization methods, this has proven not
to be the case: we might want to optimize a test out if its inputs have not
changed, even if a new installer has been built. Similarly, SETA may optimize
tasks out even if their inputs have changed.
MozReview-Commit-ID: LgHET3Z84GB
--HG--
extra : rebase_source : efd297d37bd49dbe655266380641abc258dda725
Version control tends to leave kind directories around containing .orig files
or other such detritus. It's best to just ignore such directories, rather than
failing to generate the taskgraph.
MozReview-Commit-ID: B9GIZndbfi1
--HG--
extra : rebase_source : 58f1debe5219fc52ead2971718da932b63bb06d7
This is currently preventing adding new TaskCluster jobs.
MozReview-Commit-ID: IelDf8g494r
--HG--
extra : rebase_source : 443f65aeaf9487599a619eb103b9671200a5c649
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
It's worth noting that creating the workspace directory is no longer
necessary because it is defined as a VOLUME in the Dockerfile or a
cache in some task configurations. However, it is default owned by
root:root, so we do need to instruct run-task to chown it.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : f4b8f320fafa2da1aa4b06c9cbef76c69acdae8f
extra : intermediate-source : 905fb4a53386c76d931e4a04a31d03739a00a1ff
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
This allows the "regular" build task to be distinguished visually from the gradle build.
MozReview-Commit-ID: 4FgA0QEkUkT
--HG--
extra : rebase_source : 1f8e4014f635d55196c2f32c44a2a56c309c76b6
v1 routes aren't used anymore, as far as I know,
MozReview-Commit-ID: 8t0j8DXUxik
--HG--
extra : rebase_source : ac90e21579c479c334906248958d5d4a2dc28a54
Setting `build_platform` in $inherits.variables and then using it in the body
of the task does not work; the resulting task.extra.build_product was an empty
string
MozReview-Commit-ID: IOxikZStVbq
--HG--
extra : rebase_source : 9381b1291e56161c75c3e7a27261c16fcc1e1ebd
This removes the use of inheritance for these tasks, incorporating some things
that used to be inherited and dropping others. In particular,
* drops index routes
* drops the unused DISABLE_MOZHARNESS
* sets tier explicitly to 1
* uses tooltool for both, even if gcc ignores it
* drops the unused secrets scope
MozReview-Commit-ID: GaMUpHa7PWN
--HG--
extra : rebase_source : a2bcef79cdda65630762d322e0c1897ca0f2cd3d
Also, don't include treeherder info for upload-symbols. This information is
not used, as the tasks have no routes that would cause them to report to
treeherder.
MozReview-Commit-ID: Ji7db0eUL2G
--HG--
extra : rebase_source : 1890bcf94059a3ffd6d8fd1613e3be48d502f27c
This commit does a few things. First, it introduces a property on the
"test_description" schema that, if defined, will cause run-task to
perform a gecko checkout. The presence of the property also configures
the needed scopes and caches.
Second, we introduce the property on web platform test tasks so a
Gecko checkout is present. We also add volumes for the Mercurial
paths to the Docker images. We strictly only need this for
desktop1604-test since WPT tests don't run on desktop-test. However,
desktop-test and desktop1604-test are nearly mirror images of each
other and I feel it is best to keep them in sync.
This commit will make WPT tasks slower on average because they will
need to create a checkout. To add salt to the wound, the checkout
isn't used. However, we need to prove that performing checkouts in
test tasks in automation works at scale. I'd prefer to have this running
for a few weeks and incurring a wall time execution penalty than to
have a giant series of commits backed out because source checkouts
aren't working.
MozReview-Commit-ID: 9UrSWSSmr3w
--HG--
extra : rebase_source : 7b3786f5c612d47dc3b0e165b4abe0c47e8af9ed
Tasks that aren't using run-task may want to configure a task for
optimal VCS setup. Also, having all the code in the same place makes
it easier to keep track of.
MozReview-Commit-ID: GkB2YEUIQiX
--HG--
extra : rebase_source : 8cd3b0164c1ae803530bcafe39075ffd5efa2694
Robust checkout is the preferred method to clone a mercurial repository. This should
speed up the cloning process a bit and reduce storage size.
MozReview-Commit-ID: 92rcwMwRLYN
--HG--
extra : rebase_source : 33a2349d636a07ad1643798bf798bdb175c4f1ca
Without this, we get a 403 when accessing the secret. IRC reviewed.
MozReview-Commit-ID: DlxqpUuW2p6
--HG--
extra : rebase_source : 7431c7a65dcb679803f81123ee59dbe3a04dbf4a
extra : source : 15403ef52183487c9f40c5bcb8a9ce2af0c888c4
extra : histedit_source : 9a5be188eac2687c286644a035dde651cc9170bc
Without this, the "taskcluster" hostname isn't registered. We
will need to talk to taskcluster in a subsequent commit in order
to obtain the hg.mozilla.org fingerprint.
MozReview-Commit-ID: KlyuI4Yk9Ce
--HG--
extra : rebase_source : a85fa4312169df8863d2b41f4f4a23b90758dcd5
Containing the Mercurial 3.9.1 version bump and the change to pin the
hg.mo fingerprint from a TC secret.
MozReview-Commit-ID: LVU7P0LqIvD
--HG--
extra : rebase_source : 8ad46e014ba9840c9972b51ea43e8ccf14492cf2
stylo inherits the defaults, basically.
stylo-try is a copy of try's config minus some builds we don't care about.
MozReview-Commit-ID: GalvxMwh8rv
--HG--
rename : taskcluster/ci/legacy/tasks/branches/try/job_flags.yml => taskcluster/ci/legacy/tasks/branches/stylo-try/job_flags.yml
extra : rebase_source : dbbf4b1e82b9a09eff5f0fdae7535f8317bc42bc
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
One thing run-task isn't doing is created the workspace. So this
functionality has been moved into test-ubuntu1204.sh and
test-ubuntu1604.sh.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : 264efc9b30e62927dece848b9b1eab7542cf2cba
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
extra : histedit_source : a5ae54a810718d446ded8bab510e6cd605562737
Previously, the code performed a "hidden" assignment to
worker['command'] when creating the "command" variable. This
tripped me up when reading the code because relying on variable
references isn't exactly obvious.
We now defer the assignment to worker['command'] until the end,
making the code a little easier to understand.
MozReview-Commit-ID: 23qH4Z3RKY4
--HG--
extra : rebase_source : 52605d7091c73fc47da753bbb298b242f7a3a2ae
extra : histedit_source : 78eb25f963c940956c65fefb2509669a59f78160
Sometimes xvfb will not start up with the current retry/delay settings. This will
attempt to retry more and delay for longer to ensure xvfb has started up. Common
pieces of this have been factored out into a recipe that all docker images can schare
that need this functionality.
MozReview-Commit-ID: BTXkJkBWLZX
--HG--
extra : rebase_source : 7554d7784768a65deb2c9ccb89586e5f77550e9f
Without this, "debug" and "opt" builds share the same cache. This
causes builds to randomly fail in automation.
MozReview-Commit-ID: Fend6GGmhFk
--HG--
extra : rebase_source : c3370938364b1e2cc94081d58ba9f35ce1060e5d
This fixes the use of the bogus pattern `taskcluster/**.py` which does not
actually match python files in subdirectories.
MozReview-Commit-ID: 3UqM5UQ0HOE
--HG--
extra : rebase_source : 73eec18034a11256273589b69b6d95aabceecd03
We've created a new worker type based on c4.4xlarge and m4.4xlarge
instances to replace the existing worker types based on c3.2xlarge,
m3.2xlarge, and r3.2xlarge. The new worker type has 16 vCPU instead
of 8, which makes build tasks complete ~10 minutes faster.
We've also consolidated down into a single pool for workers.
MozReview-Commit-ID: sOfKspFjCS
--HG--
extra : rebase_source : 8c7b030e0d9376dd9c07b71b8987be0c300fb691
This was recently refactored (incorrectly) in 147b245de122
(bug 1290282). Before that refactor, we only defined {{build_name}}
in base_linux64.yml. After, we hard-coded "linux32" and "linux64."
The change to linux64 resulted in ASAN builds sharing the same cache
name. Derp.
This commit restores {{build_name}} to the cache path. And for
consistency, we use {{build_name}} on both 32 and 64-bit variants.
MozReview-Commit-ID: KhqIjzARETO
--HG--
extra : rebase_source : 6b110302b66e052ec3d79bdc5b05a190cb0a7749
Before, test.sh (duplicated between the desktop-test and
desktop1604-test images) was dropping permissions, creating a workspace,
and executing test-linux.sh. This is functionality now provided by
run-task.
So, convert the test tasks to use run-task.
It's worth noting that creating the workspace directory is no longer
necessary because it is defined as a VOLUME in the Dockerfile or a
cache in some task configurations. However, it is default owned by
root:root, so we do need to instruct run-task to chown it.
Since the test.sh files are no longer used after this change, they have
been deleted. The desktop-test image no longer has any files in the
bin/ directory, so the Dockerfile entry to copy those files has been
removed.
MozReview-Commit-ID: 1BiskrMs6xW
--HG--
extra : rebase_source : 4b16837ad9362844668fe4d6cde5ed07a59a3ccd
extra : source : 8335aa40265b1d17421d06d9e9a180eb8419fe47
Before, we'd open files and feed bytes to yaml.load(). When a str
is fed to yaml.load(), it attempts to guess the encoding. It defaults
to UTF-8 unless somebody set us up the BOM. This is probably OK.
Except if the file isn't valid UTF-8, the exception will be raised
in the bowels of YAML parsing and it may not be obvious the failure
is due to invalid UTF-8 input versus say Python str/unicode
coercion foo.
We change all call sites that load YAML from a file to use
codecs.open() to open the file in UTF-8 and perform UTF-8
decoding/validation at file read time. This should make any UTF-8
failures more obvious. Furthermore, it reinforces that our YAML files
are UTF-8 and not some other encoding.
I discovered this issue as part of trying to get emoji symbols to
render on Treeherder. Unfortunately, it appears pyyaml detects
many emoji as unprintable characters and refuses to load them. This
makes me sad and makes me want to abandon pyyaml/YAML in favor of
something that supports emoji :P
MozReview-Commit-ID: AOvAruZFfnK
--HG--
extra : rebase_source : 4c5d42cb63da5c9ebfac55ab84b4f7f8bcc648fa
Sometimes xvfb will not start up with the current retry/delay settings. This will
attempt to retry more and delay for longer to ensure xvfb has started up. Common
pieces of this have been factored out into a recipe that all docker images can schare
that need this functionality.
MozReview-Commit-ID: 2ww0eT3cIt6
--HG--
extra : rebase_source : f343f8a2bf8370657b63afdd9d739988bbb6261c
We run only a small subset of the test suite to save resources. Over
time, we'll run more of the platform test suite.
--HG--
extra : rebase_source : 264fbe55be5b56857278ae8d731ef51b1b41cf3c
extra : histedit_source : a3550c7e2ea8a41cd6285e511b7268a9e7af11cd%2Cec6024dadcb7c071170aa452064776666283bb27
MozReview-Commit-ID: Lrxi8t53nwy
If a developer adds '--rebuild N' to their try syntax they will get test jobs scheduled N times.
This is useful to determine intermittency rate.
This fixes a regression due to the recent refactoring on how we schedule tasks.
--HG--
extra : rebase_source : 355ca631353015bf63461c194168d753efd6958e
Geoff Brown
Carsten "Tomcat" Book
Phil Ringnalda
Mihai Tabara
Andrew Halberstadt
Alastair Pharo
Brian Stack
Wes Kocher
Kalpesh Krishna
Iris Hsiao
Justin Wood
Dustin J. Mitchell
Mike Shal
Greg Mierzwinski
Jordan Lund
Ted Mielczarek
Gregory Szorc
Enes Goktas
Steve Fink
James Graham
Ryan VanderMeulen
Chris Manchester
Armen Zambrano Gasparnian
Rob Thijssen
Nick Alexander
amiyaguchi@mozilla.com
Henrik Skupin
Maja Frydrychowicz
Kim Moir
Matthew Noorenberghe
Andrew Comminos
aselagea
Joel Maher
Gregory Arndt
Ehsan Akhgari