Normally we start Xvfb as a background task, then run the tests from the
same script. However, in interactive mode we were starting Xvfb, the script
would exit, and then we would potentially run tests later on from another
script. Unforunately this meant that Xvfb was dying with the first script
and tests would fail.
This patch runs Xvfb in a screen session so that it will still be available
later on when running an interactive shell.
MozReview-Commit-ID: EduVyglo2BG
--HG--
extra : rebase_source : 6e2c40ee16f80792f038fd581168e181a2c4bf51
Previously, every lint task would have to create its own checkout. This
was time consuming.
The robustcheckout extension purges the working copy of *all*
untracked and ignored files. It also restores modified files to
their original state. In other words, as long as you trust Mercurial
to go from revision X to revision Y, robustcheckout is as good as
a fresh checkout.
This commit adds a cache for the working directory checkout so lint
tasks only have to effectively perform incremental `hg update` between
task executions. This should make tasks spend a lot less time
doing version control foo.
On Try, time for flake8 tasks is currently hovering around 4 minutes.
After this change, I've seen tasks finish as quickly as 11s! But that
was with a hacked up legacy.py that added the workspace cache to the
whitelist for Try. While I would like to see workspace reuse on Try
eventually, this is not the right commit to roll that out in.
MozReview-Commit-ID: 66P2rt896qE
--HG--
extra : rebase_source : 2a7c8d396e85ba4eae84f8843256050a2288c9d0
extra : intermediate-source : c478ecb14bf6164ef0c955acee3a0a2f18e415c5
extra : source : 63d54efdc1f8effb0370644c11014a3f0404073b
extra : histedit_source : 560701ce41c870171443c78e3a9de3998fbf9306
The robustcheckout Mercurial extension does a clone+checkout optimally.
Read the bug for more on it.
robustcheckout is already used by mozharness automation. It has resulted
in a significant reduction in I/O usage and utilization in automation.
This commit replaces tc-vcs with the robustcheckout equivalent.
We replace the existing tc-vcs scope and cache with a new one.
Because Dustin and I are paranoid, we maintain separate caches per
SCM level - even though we could arguably share the same cache. Defense
in depth.
Robustcheckout (when used with --sharebase) pools storage for related
repos automatically. i.e. changesets from inbound and central will
be in the same store. This means you likely only have one copy of
each changeset per cache. This can result in significant space savings.
And, since there are fewer copies floating around, hg.mozilla.org
and various network appliances are working less too!
Since tc-vcs is no longer used, we stop it from being installed.
While we're here, we also change the images to execute as the
"worker" user. This happens automatically as a result of using
the "checkout-and-run" script.
MozReview-Commit-ID: EDeebuP7TkT
--HG--
extra : rebase_source : 2bec5dd9d6fe5565831bb35f195859aa12dd0bf2
extra : intermediate-source : 06481d97a485f6566554b087bc3880d76361e8ec
extra : source : d368700c93ef085325a081219d7aeb8512bc54a1
extra : histedit_source : c07505273fc8f10acf8e8d3ee01e327afd0aa63d
https://hg.mozilla.org/mozilla-unified contains heads from all the
major Firefox repos. In addition, it is encoded on the server in such a
way that it is several hundred megabytes smaller despite containing
30,000+ more changesets. This means faster clones and faster
operations.
Cloning this repo does require a new version of Mercurial. So if any
TC tasks are using an ancient Mercurial - one that is vulnerable to
known CVE issues in fact - this will flush them out.
MozReview-Commit-ID: 2VHDa6FEeeJ
--HG--
extra : rebase_source : db4990bcde0503fd14d82a5d16c71adbb4f92be3
extra : histedit_source : 6c555bd3df12536d1c48b45d8cc76611e3f7032f
Changes to the decision Docker image have been compelted. We're ready to
use the new image.
We tag the image, update version references, change the task caches
so the new Mercurial pooled storage from the robustcheckout extension is
used, and convert the decision tasks to run as the "worker" user.
MozReview-Commit-ID: 61v9Ivy59zG
--HG--
extra : rebase_source : 640318a87660950c5e0680867a1bfdd68e35f127
extra : histedit_source : ec53fc576c00e5f2053167b37544ac7afccaecb5
The decision task configures a /home/worker/workspace cache. However,
the command we run in the container references a "workspace" relative
path. From logs in automation, it appears that PWD during execution
is "/" because "workspace" is being resolved to "/workspace." The
net result of this is we appear to be performing a VCS clone+checkout
on every single task.
This commit fixes the paths so our workspace cache is actually used.
MozReview-Commit-ID: Kj6REep5bSs
--HG--
extra : rebase_source : 8cd8be43dfd34f2970b47721c3da8e3957a8bfed
extra : histedit_source : b81dce523a88e44eb3fa8b1a68840066edca382d
These scopes will be introduced in subsequent commits. We don't want
them to get removed, just like we don't want the tc-vcs caches to get
removed.
MozReview-Commit-ID: 790SzpAUxE4
--HG--
extra : rebase_source : d8148aa38bad09468f67ded1a64f432e2e30c63a
extra : source : 923b86a15d91cb22999a7c94a4a708845d23aa9d
extra : histedit_source : 6d85127fdcc2e795a54eeb836ce11f158e5e88cd
The function was only used once and was providing little to no value.
A test of this function has been removed. Tests for the lower-level
context creation function are sufficient.
MozReview-Commit-ID: D9EhmZQlqG5
--HG--
extra : rebase_source : 4b3faa0fc5f085c1c77fe5636744946a6d442b05
A limitation of traditional docker build context generation is it
only includes files from the same directory as the Dockerfile. When
repositories have multiple, related Dockerfiles, this limitation
results file duplication or putting all Dockerfiles in the same
directory (which isn't feasible for mozilla-central since they would
need to be in the root directory).
This commit enhances Dockerfiles to allow *any* file from the
repository checkout to be ADDed to the docker build context.
Using the syntax "# %include <path>" you are able to include paths
or directories (relative from the top source directory root) in the
generated context archive. Files add this way are available under the
"topsrcdir/" path and can be ADDed to Docker images.
Since context archive generation is deterministic and the hash of
the resulting archive is used to determine when images need to be
rebuilt, any extra included file that changes will change the hash
of the context archive and force image regeneration.
Basic tests for the new feature have been added.
MozReview-Commit-ID: 4hPZesJuGQV
--HG--
extra : rebase_source : 99fae9fe82102126fbee879c134981047bb4a601
This restores order to only having a single hash for a context
directory.
Using a tempfile here is a bit unfortunate. It can be optimized later,
if needed.
MozReview-Commit-ID: LMNsvt3fDYx
--HG--
extra : rebase_source : 8c2b70164aed6d744a71d170d0324797e755cbaf
Now that the context tar creation function is standalone and doesn't
rely on external state, we can start unit testing it easier.
We establish a basic unit test that verifies the function works as
advertised and that output is deterministic.
MozReview-Commit-ID: H4MY28PiHSN
--HG--
extra : rebase_source : 692a5e3d5af6edd14b3d4ceb7c90cd1e0344052f
Relying on global variables like GECKO is a bit dangerous. To facilitate
testing of archive generation in subsequent commits, let's pass an
path into this function.
The argument is currently unused.
MozReview-Commit-ID: Et1UYraflDP
--HG--
extra : rebase_source : 012095bc2450c72467f3f65f71bcb4cf6efcde66
Now that tar file generation is deterministic, we can use the hash
of the created archive rather than the hash of the files that are
(presumably) in the archive.
This temporarily breaks consistent hashing by using independent
hashing mechanisms. This will be cleaned up in a subsequent commit.
MozReview-Commit-ID: CWooVGfDKZO
--HG--
extra : rebase_source : 86923635f960e23f1483068bf809dec7aeacefd1
We recently implemented code in mozpack for performing deterministic
tar file creation. It normalizes things like uids, gids, and mtimes
that creep into archives.
MozReview-Commit-ID: 1tn5eXkqACQ
--HG--
extra : rebase_source : 6b069a3a50c9103ae4f6185b26d6a37658179f42
Upcoming commits will refactor how context tarballs are created. In
preparation for this, we establish a standalone function for creating
context tarballs and refactor docker_image.py to use it.
MozReview-Commit-ID: KEW6ppO1vCl
--HG--
extra : rebase_source : b81decf9ca14ff0216514f47419e96eb57d6f851
This removes the unnecessary setting of c-basic-offset from all
python-mode files.
This was automatically generated using
perl -pi -e 's/; *c-basic-offset: *[0-9]+//'
... on the affected files.
The bulk of these files are moz.build files but there a few others as
well.
MozReview-Commit-ID: 2pPf3DEiZqx
--HG--
extra : rebase_source : 0a7dcac80b924174a2c429b093791148ea6ac204
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
MOZ_NODE_PATH should not be set for Android tests. Rather than setting it to
the literal "None", this omits the environment variable on that platform.
Only `fx_docker_desktop_generic.yml` (and now `fx_docker_android_generic.yml`)
inherit from `fx_docker_test_base.yml`. And `fx_android-api-15_generic.yml` is
the only Android-related YAML file to inherit from
`fx_docker_desktop_generic.yml`.
MozReview-Commit-ID: 7BhGXk9c8ir
--HG--
extra : rebase_source : 8190d2f6f2368acd1c9c11a3e16062e4589a23ac
This makes the two copies of the suite consistent.
MozReview-Commit-ID: FhxzwQlERnD
--HG--
extra : rebase_source : e13291d9d852521cd9af9de79addc58ccfe62949
extra : source : 85f75a951f93fb7105cd9613ed98d041575c6a35
For all of ther suites, the -e10s is attached to the group symbol, not the job
symbol.
MozReview-Commit-ID: BZcRCnEMy5U
--HG--
extra : rebase_source : 6b0b294d8ece3b325fd3af43a60f994427b9ddd8
extra : source : 05714f5e19e6353aaf1f96540e70ba2385bbee89
This makes the instance size consistent for the whole suite
MozReview-Commit-ID: 8s6n6BwXSjB
--HG--
extra : rebase_source : 87744db0ff2122cde214e22dc356af3ee3b87503
extra : source : 46e29447f7f31124baae8008ec0033b1b3dddbe0
looks like we have to be more explicit with each build and suite compared to
https://bugzilla.mozilla.org/show_bug.cgi?id=1243024
but the way we bump this to tier 1 will likely change as
https://bugzilla.mozilla.org/show_bug.cgi?id=1281004 lands
as per dev.planning announcement and coop's email, we are going to try making
tc tier 2 jobs more critical for less critical jobs, e.g. debug builds, by
making them tier 1.
MozReview-Commit-ID: 4IwNCjOHo7x
--HG--
extra : source : 0b6844851660c49267f74d589345ebdaeb304db6
extra : amend_source : cd1507f8d9c6be75c68929051bbf8ee30cfe9cee
note, as bug 1281004 is about to land, I might require a new patch as we move
away from legacy towards da futures!
this yielded:
current bbot opt routes:
"index.gecko.v2.try.revision.f40f15f50508b78e369c8ac5e6a8743bcd064193.mobile.android-api-15-opt",
"index.gecko.v2.try.pushdate.2016.07.11.20160711204636.mobile.android-api-15-opt",
"index.gecko.v2.try.latest.mobile.android-api-15-opt",
"index.buildbot.branches.try.android-api-15",
"index.buildbot.revisions.f40f15f50508b78e369c8ac5e6a8743bcd064193.try.android-api-15"
my patch tc opt routes:
"index.gecko.v1.try.revision.linux.f40f15f50508b78e369c8ac5e6a8743bcd064193.android-api-15.opt",
"index.gecko.v1.try.latest.linux.android-api-15.opt",
"index.buildbot.branches.try.android-api-15",
"index.buildbot.revisions.f40f15f50508b78e369c8ac5e6a8743bcd064193.try.android-api-15",
"tc-treeherder.v2.try.f40f15f50508b78e369c8ac5e6a8743bcd064193.133427",
"tc-treeherder-stage.v2.try.f40f15f50508b78e369c8ac5e6a8743bcd064193.133427",
"index.gecko.v2.try.revision.f40f15f50508b78e369c8ac5e6a8743bcd064193.mobile.android-api-15-opt",
"index.gecko.v2.try.pushdate.2016.07.11.20160711204636.mobile.android-api-15-opt",
"index.gecko.v2.try.latest.mobile.android-api-15-opt"
current bbot debug routes:
"index.gecko.v2.try.revision.f40f15f50508b78e369c8ac5e6a8743bcd064193.mobile.android-api-15-debug",
"index.gecko.v2.try.pushdate.2016.07.11.20160711204636.mobile.android-api-15-debug",
"index.gecko.v2.try.latest.mobile.android-api-15-debug",
"index.buildbot.branches.try.android-api-15-debug",
"index.buildbot.revisions.f40f15f50508b78e369c8ac5e6a8743bcd064193.try.android-api-15-debug"
my patch tc debug routes:
"index.gecko.v1.try.revision.linux.f40f15f50508b78e369c8ac5e6a8743bcd064193.android-api-15.debug",
"index.gecko.v1.try.latest.linux.android-api-15.debug",
"index.buildbot.branches.try.android-api-15-debug",
"index.buildbot.revisions.f40f15f50508b78e369c8ac5e6a8743bcd064193.try.android-api-15-debug",
"tc-treeherder.v2.try.f40f15f50508b78e369c8ac5e6a8743bcd064193.133427",
"tc-treeherder-stage.v2.try.f40f15f50508b78e369c8ac5e6a8743bcd064193.133427",
"index.gecko.v2.try.revision.f40f15f50508b78e369c8ac5e6a8743bcd064193.mobile.android-api-15-debug",
"index.gecko.v2.try.pushdate.2016.07.11.20160711204636.mobile.android-api-15-debug",
"index.gecko.v2.try.latest.mobile.android-api-15-debug"
so all looks well and as a bonus, the index.buildbot routes now match too.
MozReview-Commit-ID: 5HilJOpONst
--HG--
extra : rebase_source : 24a3c895681284a8dca16cbf3b2a47b66eaa1f08
extra : amend_source : b8b8b91c9787e26a6dd6d54fbe39cb9cb515056b
This fixes a race condition between the 'test-linux.sh' process and the 'taskcluster-interactive-shell'
process in interactive tasks.
MozReview-Commit-ID: GhqKpq5pAtj
--HG--
extra : rebase_source : 67b756d0373432404a4f7cc928bac09fc3f82e8a
Joel Maher
Dustin J. Mitchell
Andrew Halberstadt
Carsten "Tomcat" Book
Gregory Szorc
Geoff Brown
Rob Thijssen
Justin Wood
Tom Tromey
Kalpesh Krishna
Wes Kocher
Jordan Lund