654a2a08db
Bug 1333255: Task for indexing other tasks. r=jonasfj
...
This introduces a new docker image: `index-task`,
which given a taskId and a set of namespaces will
index the given taskId under said namespaces.
Modified to include a script with a descriptive name that curious users can
find in the source code.
MozReview-Commit-ID: KPHVT0XPfsb
--HG--
extra : rebase_source : ed3abde6082f98b83bc5b13f65172562b8875263
extra : source : f8ca244c170f587ead852bb9fbdc99c9dec52f31
2017-03-14 15:25:57 +00:00
449d528cf3
Bug 1333255: rename taskgraph.task to taskgraph.loader; r=jonasfj
...
MozReview-Commit-ID: 6aPdGbxn0D1
--HG--
rename : taskcluster/taskgraph/task/__init__.py => taskcluster/taskgraph/loader/__init__.py
rename : taskcluster/taskgraph/task/balrog.py => taskcluster/taskgraph/loader/balrog.py
rename : taskcluster/taskgraph/task/beetmover.py => taskcluster/taskgraph/loader/beetmover.py
rename : taskcluster/taskgraph/task/beetmover_checksums.py => taskcluster/taskgraph/loader/beetmover_checksums.py
rename : taskcluster/taskgraph/task/checksums_signing.py => taskcluster/taskgraph/loader/checksums_signing.py
rename : taskcluster/taskgraph/task/post_build.py => taskcluster/taskgraph/loader/post_build.py
rename : taskcluster/taskgraph/task/repacks.py => taskcluster/taskgraph/loader/repacks.py
rename : taskcluster/taskgraph/task/signing.py => taskcluster/taskgraph/loader/signing.py
rename : taskcluster/taskgraph/task/test.py => taskcluster/taskgraph/loader/test.py
rename : taskcluster/taskgraph/task/transform.py => taskcluster/taskgraph/loader/transform.py
rename : taskcluster/taskgraph/task/base.py => taskcluster/taskgraph/task.py
extra : rebase_source : 42a183bae9aedfa04876d99a59119fd08bbf7d73
2017-03-09 19:00:49 -05:00
7e3408a70e
Bug 1333255: always transform tasks; r=jonasfj
...
MozReview-Commit-ID: 3A5wzXKG0Yp
--HG--
extra : rebase_source : 10063aa354d18b08db182b41e9ae61f2f884becd
2017-03-09 16:40:33 -05:00
d46264d259
Bug 1333255: and then there was only one Task class; r=jonasfj
...
Note that the to_json method prefers the taskgraph's dependencies information
(edges) to that from the task.dependencies entries. At a few points in
task-graph generation, these values differ, although that is expected (for
example, the full task set contains no edges, but that information is still in
task.dependencies). Unifying that representation leads to some difficulty with
task transforms that reach into the dependency tree (beetmover), so the
different representations are left as-is.
MozReview-Commit-ID: GeW8HNwFA9Z
--HG--
extra : rebase_source : 549773e05e18371a399612d9bceccffc29be8cf2
2017-03-09 23:14:30 -05:00
a39d3b7ea3
Bug 1333255: use transforms to make docker image tasks, too; r=jonasfj
...
MozReview-Commit-ID: Eke3TjLbEfE
--HG--
extra : rebase_source : b91aa6733c26aed2f8f40de73dfef6ae863d1899
2017-03-08 20:52:13 +00:00
b0117f4f53
Bug 1333255: use normal old functions to load tasks; r=jonasfj
...
Instead of using a class's static method, use a simple function, specified by
the `loader` key.
MozReview-Commit-ID: IeOl9qiSCXf
--HG--
extra : rebase_source : 72e0a9dd8385b250a46c9f4adf8a8a0e5b01c156
2017-03-09 23:14:40 -05:00
e67707b149
Bug 1302763 - Move docker images out of testing/docker into taskcluster/docker; r=dustin r=CuriousLearner
...
MozReview-Commit-ID: 7v1uCDB5qoN
--HG--
rename : testing/docker/README.md => taskcluster/docker/README.md
rename : testing/docker/REGISTRY => taskcluster/docker/REGISTRY
rename : testing/docker/android-gradle-build/Dockerfile => taskcluster/docker/android-gradle-build/Dockerfile
rename : testing/docker/android-gradle-build/README.md => taskcluster/docker/android-gradle-build/README.md
rename : testing/docker/android-gradle-build/REGISTRY => taskcluster/docker/android-gradle-build/REGISTRY
rename : testing/docker/android-gradle-build/VERSION => taskcluster/docker/android-gradle-build/VERSION
rename : testing/docker/android-gradle-build/bin/after.sh => taskcluster/docker/android-gradle-build/bin/after.sh
rename : testing/docker/android-gradle-build/bin/before.sh => taskcluster/docker/android-gradle-build/bin/before.sh
rename : testing/docker/android-gradle-build/bin/build.sh => taskcluster/docker/android-gradle-build/bin/build.sh
rename : testing/docker/android-gradle-build/bin/checkout-script.sh => taskcluster/docker/android-gradle-build/bin/checkout-script.sh
rename : testing/docker/android-gradle-build/bin/checkout-sources.sh => taskcluster/docker/android-gradle-build/bin/checkout-sources.sh
rename : testing/docker/android-gradle-build/buildprops.json => taskcluster/docker/android-gradle-build/buildprops.json
rename : testing/docker/android-gradle-build/dot-config/pip/pip.conf => taskcluster/docker/android-gradle-build/dot-config/pip/pip.conf
rename : testing/docker/android-gradle-build/oauth.txt => taskcluster/docker/android-gradle-build/oauth.txt
rename : testing/docker/base-build/Dockerfile => taskcluster/docker/base-build/Dockerfile
rename : testing/docker/base-build/HASH => taskcluster/docker/base-build/HASH
rename : testing/docker/base-build/VERSION => taskcluster/docker/base-build/VERSION
rename : testing/docker/base-build/system-setup.sh => taskcluster/docker/base-build/system-setup.sh
rename : testing/docker/base-test/Dockerfile => taskcluster/docker/base-test/Dockerfile
rename : testing/docker/base-test/HASH => taskcluster/docker/base-test/HASH
rename : testing/docker/base-test/REGISTRY => taskcluster/docker/base-test/REGISTRY
rename : testing/docker/base-test/VERSION => taskcluster/docker/base-test/VERSION
rename : testing/docker/base-test/sources.list => taskcluster/docker/base-test/sources.list
rename : testing/docker/centos6-build-upd/Dockerfile => taskcluster/docker/centos6-build-upd/Dockerfile
rename : testing/docker/centos6-build-upd/HASH => taskcluster/docker/centos6-build-upd/HASH
rename : testing/docker/centos6-build-upd/REGISTRY => taskcluster/docker/centos6-build-upd/REGISTRY
rename : testing/docker/centos6-build-upd/VERSION => taskcluster/docker/centos6-build-upd/VERSION
rename : testing/docker/centos6-build/Dockerfile => taskcluster/docker/centos6-build/Dockerfile
rename : testing/docker/centos6-build/HASH => taskcluster/docker/centos6-build/HASH
rename : testing/docker/centos6-build/REGISTRY => taskcluster/docker/centos6-build/REGISTRY
rename : testing/docker/centos6-build/VERSION => taskcluster/docker/centos6-build/VERSION
rename : testing/docker/centos6-build/hgrc => taskcluster/docker/centos6-build/hgrc
rename : testing/docker/centos6-build/system-setup.sh => taskcluster/docker/centos6-build/system-setup.sh
rename : testing/docker/decision/Dockerfile => taskcluster/docker/decision/Dockerfile
rename : testing/docker/decision/HASH => taskcluster/docker/decision/HASH
rename : testing/docker/decision/README.md => taskcluster/docker/decision/README.md
rename : testing/docker/decision/REGISTRY => taskcluster/docker/decision/REGISTRY
rename : testing/docker/decision/VERSION => taskcluster/docker/decision/VERSION
rename : testing/docker/decision/system-setup.sh => taskcluster/docker/decision/system-setup.sh
rename : testing/docker/desktop-build/Dockerfile => taskcluster/docker/desktop-build/Dockerfile
rename : testing/docker/desktop-build/bin/build.sh => taskcluster/docker/desktop-build/bin/build.sh
rename : testing/docker/desktop-build/bin/checkout-script.sh => taskcluster/docker/desktop-build/bin/checkout-script.sh
rename : testing/docker/desktop-build/bin/checkout-sources.sh => taskcluster/docker/desktop-build/bin/checkout-sources.sh
rename : testing/docker/desktop-build/buildprops.json => taskcluster/docker/desktop-build/buildprops.json
rename : testing/docker/desktop-build/dot-config/pip/pip.conf => taskcluster/docker/desktop-build/dot-config/pip/pip.conf
rename : testing/docker/desktop-build/oauth.txt => taskcluster/docker/desktop-build/oauth.txt
rename : testing/docker/desktop-test/Dockerfile => taskcluster/docker/desktop-test/Dockerfile
rename : testing/docker/desktop-test/apport => taskcluster/docker/desktop-test/apport
rename : testing/docker/desktop-test/buildprops.json => taskcluster/docker/desktop-test/buildprops.json
rename : testing/docker/desktop-test/deja-dup-monitor.desktop => taskcluster/docker/desktop-test/deja-dup-monitor.desktop
rename : testing/docker/desktop-test/dot-files/config/pip/pip.conf => taskcluster/docker/desktop-test/dot-files/config/pip/pip.conf
rename : testing/docker/desktop-test/dot-files/config/user-dirs.dirs => taskcluster/docker/desktop-test/dot-files/config/user-dirs.dirs
rename : testing/docker/desktop-test/dot-files/config/user-dirs.locale => taskcluster/docker/desktop-test/dot-files/config/user-dirs.locale
rename : testing/docker/desktop-test/dot-files/pulse/default.pa => taskcluster/docker/desktop-test/dot-files/pulse/default.pa
rename : testing/docker/desktop-test/fonts.conf => taskcluster/docker/desktop-test/fonts.conf
rename : testing/docker/desktop-test/jockey-gtk.desktop => taskcluster/docker/desktop-test/jockey-gtk.desktop
rename : testing/docker/desktop-test/motd => taskcluster/docker/desktop-test/motd
rename : testing/docker/desktop-test/release-upgrades => taskcluster/docker/desktop-test/release-upgrades
rename : testing/docker/desktop-test/taskcluster-interactive-shell => taskcluster/docker/desktop-test/taskcluster-interactive-shell
rename : testing/docker/desktop-test/tc-vcs-config.yml => taskcluster/docker/desktop-test/tc-vcs-config.yml
rename : testing/docker/desktop-test/tester.env => taskcluster/docker/desktop-test/tester.env
rename : testing/docker/desktop1604-test/Dockerfile => taskcluster/docker/desktop1604-test/Dockerfile
rename : testing/docker/desktop1604-test/apport => taskcluster/docker/desktop1604-test/apport
rename : testing/docker/desktop1604-test/buildprops.json => taskcluster/docker/desktop1604-test/buildprops.json
rename : testing/docker/desktop1604-test/deja-dup-monitor.desktop => taskcluster/docker/desktop1604-test/deja-dup-monitor.desktop
rename : testing/docker/desktop1604-test/dot-files/config/pip/pip.conf => taskcluster/docker/desktop1604-test/dot-files/config/pip/pip.conf
rename : testing/docker/desktop1604-test/dot-files/config/user-dirs.dirs => taskcluster/docker/desktop1604-test/dot-files/config/user-dirs.dirs
rename : testing/docker/desktop1604-test/dot-files/config/user-dirs.locale => taskcluster/docker/desktop1604-test/dot-files/config/user-dirs.locale
rename : testing/docker/desktop1604-test/dot-files/pulse/default.pa => taskcluster/docker/desktop1604-test/dot-files/pulse/default.pa
rename : testing/docker/desktop1604-test/fonts.conf => taskcluster/docker/desktop1604-test/fonts.conf
rename : testing/docker/desktop1604-test/jockey-gtk.desktop => taskcluster/docker/desktop1604-test/jockey-gtk.desktop
rename : testing/docker/desktop1604-test/motd => taskcluster/docker/desktop1604-test/motd
rename : testing/docker/desktop1604-test/release-upgrades => taskcluster/docker/desktop1604-test/release-upgrades
rename : testing/docker/desktop1604-test/taskcluster-interactive-shell => taskcluster/docker/desktop1604-test/taskcluster-interactive-shell
rename : testing/docker/desktop1604-test/tc-vcs-config.yml => taskcluster/docker/desktop1604-test/tc-vcs-config.yml
rename : testing/docker/desktop1604-test/tester.env => taskcluster/docker/desktop1604-test/tester.env
rename : testing/docker/image_builder/Dockerfile => taskcluster/docker/image_builder/Dockerfile
rename : testing/docker/image_builder/HASH => taskcluster/docker/image_builder/HASH
rename : testing/docker/image_builder/REGISTRY => taskcluster/docker/image_builder/REGISTRY
rename : testing/docker/image_builder/VERSION => taskcluster/docker/image_builder/VERSION
rename : testing/docker/image_builder/build-image.sh => taskcluster/docker/image_builder/build-image.sh
rename : testing/docker/image_builder/setup.sh => taskcluster/docker/image_builder/setup.sh
rename : testing/docker/lint/Dockerfile => taskcluster/docker/lint/Dockerfile
rename : testing/docker/lint/system-setup.sh => taskcluster/docker/lint/system-setup.sh
rename : testing/docker/recipes/centos6-build-system-setup.sh => taskcluster/docker/recipes/centos6-build-system-setup.sh
rename : testing/docker/recipes/common.sh => taskcluster/docker/recipes/common.sh
rename : testing/docker/recipes/install-mercurial.sh => taskcluster/docker/recipes/install-mercurial.sh
rename : testing/docker/recipes/run-task => taskcluster/docker/recipes/run-task
rename : testing/docker/recipes/tooltool.py => taskcluster/docker/recipes/tooltool.py
rename : testing/docker/recipes/ubuntu1204-test-system-setup.sh => taskcluster/docker/recipes/ubuntu1204-test-system-setup.sh
rename : testing/docker/recipes/ubuntu1604-test-system-setup.sh => taskcluster/docker/recipes/ubuntu1604-test-system-setup.sh
rename : testing/docker/recipes/xvfb.sh => taskcluster/docker/recipes/xvfb.sh
rename : testing/docker/rust-build/Dockerfile => taskcluster/docker/rust-build/Dockerfile
rename : testing/docker/rust-build/README.md => taskcluster/docker/rust-build/README.md
rename : testing/docker/rust-build/REGISTRY => taskcluster/docker/rust-build/REGISTRY
rename : testing/docker/rust-build/VERSION => taskcluster/docker/rust-build/VERSION
rename : testing/docker/rust-build/build_cargo.sh => taskcluster/docker/rust-build/build_cargo.sh
rename : testing/docker/rust-build/build_rust.sh => taskcluster/docker/rust-build/build_rust.sh
rename : testing/docker/rust-build/build_rust_mac.sh => taskcluster/docker/rust-build/build_rust_mac.sh
rename : testing/docker/rust-build/fetch_cargo.sh => taskcluster/docker/rust-build/fetch_cargo.sh
rename : testing/docker/rust-build/fetch_rust.sh => taskcluster/docker/rust-build/fetch_rust.sh
rename : testing/docker/rust-build/package_rust.sh => taskcluster/docker/rust-build/package_rust.sh
rename : testing/docker/rust-build/repack_rust.py => taskcluster/docker/rust-build/repack_rust.py
rename : testing/docker/rust-build/splat_rust.py => taskcluster/docker/rust-build/splat_rust.py
rename : testing/docker/rust-build/task.json => taskcluster/docker/rust-build/task.json
rename : testing/docker/rust-build/tcbuild.py => taskcluster/docker/rust-build/tcbuild.py
rename : testing/docker/rust-build/upload_rust.sh => taskcluster/docker/rust-build/upload_rust.sh
rename : testing/docker/tester/Dockerfile => taskcluster/docker/tester/Dockerfile
rename : testing/docker/tester/HASH => taskcluster/docker/tester/HASH
rename : testing/docker/tester/REGISTRY => taskcluster/docker/tester/REGISTRY
rename : testing/docker/tester/VERSION => taskcluster/docker/tester/VERSION
rename : testing/docker/tester/bin/test.sh => taskcluster/docker/tester/bin/test.sh
rename : testing/docker/tester/dot-config/pip/pip.conf => taskcluster/docker/tester/dot-config/pip/pip.conf
rename : testing/docker/tester/dot-config/user-dirs.dirs => taskcluster/docker/tester/dot-config/user-dirs.dirs
rename : testing/docker/tester/dot-config/user-dirs.locale => taskcluster/docker/tester/dot-config/user-dirs.locale
rename : testing/docker/tester/dot-pulse/default.pa => taskcluster/docker/tester/dot-pulse/default.pa
rename : testing/docker/tester/tc-vcs-config.yml => taskcluster/docker/tester/tc-vcs-config.yml
rename : testing/docker/tester/tester.env => taskcluster/docker/tester/tester.env
rename : testing/docker/upload-symbols/Dockerfile => taskcluster/docker/upload-symbols/Dockerfile
rename : testing/docker/upload-symbols/README.md => taskcluster/docker/upload-symbols/README.md
rename : testing/docker/upload-symbols/bin/checkout-script.sh => taskcluster/docker/upload-symbols/bin/checkout-script.sh
rename : testing/docker/upload-symbols/bin/upload.sh => taskcluster/docker/upload-symbols/bin/upload.sh
rename : testing/docker/upload-symbols/test_exports.sh => taskcluster/docker/upload-symbols/test_exports.sh
extra : rebase_source : fd02b10c77de5b68476ce462b5f888475520a6fe
extra : source : 50adff295bf00ea0d9d7426e745acf3635e7dba5
2016-10-20 18:25:34 +05:30
4dd59586f3
Bug 1289779 - Make sure that different docker-image tasks use different job symbols; r=dustin
...
MozReview-Commit-ID: 2sn231bKJ1z
--HG--
extra : rebase_source : 8094c6677c6927dfd2e50203b8f918ec3c78b8cf
2016-12-13 13:39:47 +05:30
0363e0cffb
Bug 1306641 - Remove TaskCluster automation related to b2g; r=dustin
...
This definitely isn't all of it. But I got the obvious chunks. Still
need to clean up mozharness.
MozReview-Commit-ID: JTZBydP3i2r
--HG--
extra : rebase_source : 782401359036751896ef6c153a81a06cb14031bb
2016-09-30 09:21:09 -07:00
65771e9829
Bug 1281179 - Create a docker image for Ubuntu 16.04 for use in tests. r=dustin
...
MozReview-Commit-ID: ApHhev5J21x
--HG--
rename : taskcluster/scripts/tester/test-linux.sh => taskcluster/scripts/tester/test-ubuntu1204.sh
2016-07-25 10:44:37 -04:00
ffca84ff7d
Bug 1281004: Specify test tasks more flexibly; r=gps; r=gbrown
...
This introduces a completely new way of specifying test task in-tree,
completely replacing the old spider-web of YAML files.
The high-level view is this:
- some configuration files are used to determine which test suites to run
for each test platform, and against which build platforms
- each test suite is then represented by a dictionary, and modified by a
sequence of transforms, duplicating as necessary (e.g., chunks), until
it becomes a task definition
The transforms allow sufficient generality to support just about any desired
configuration, with the advantage that common configurations are "easy" while
unusual configurations are supported but notable for their oddness (they
require a custom transform).
As of this commit, this system produces the same set of test graphs as the
existing YAML, modulo:
- extra.treeherder.groupName -- this was not consistent in the YAML
- extra.treeherder.build -- this is ignored by taskcluster-treeherder anyway
- mozharness command argument order
- boolean True values for environment variables are now the string "true"
- metadata -- this is now much more consistent, with task name being the label
Testing of this commit demonstrates that it produces the same set of test tasks for
the following projects (those which had special cases defined in the YAML):
- autoland
- ash (*)
- willow
- mozilla-inbound
- mozilla-central
- try:
-b do -p all -t all -u all
-b d -p linux64,linux64-asan -u reftest -t none
-b d -p linux64,linux64-asan -u reftest[x64] -t none[x64]
(*) this patch omits the linux64/debug tc-M-e10s(dt) test, which is enabled on
ash; ash will require a small changeset to re-enable this test.
IGNORE BAD COMMIT MESSAGES (because the hook flags try syntax!)
MozReview-Commit-ID: G34dg9f17Hq
--HG--
rename : taskcluster/taskgraph/kind/base.py => taskcluster/taskgraph/task/base.py
rename : taskcluster/taskgraph/kind/docker_image.py => taskcluster/taskgraph/task/docker_image.py
rename : taskcluster/taskgraph/kind/legacy.py => taskcluster/taskgraph/task/legacy.py
extra : rebase_source : 03e70902c2d3a297eb9e3ce852f8737c2550d5a6
extra : histedit_source : d4d9f4b192605af21f41d83495fc3c923759c3cb
2016-07-11 23:27:14 +00:00
2eb2535e97
Bug 1282226 - Kill secrets from phone-builder image and fix aries/hammerhead r=wcosta
...
MozReview-Commit-ID: LOZHgih2TrM
--HG--
extra : rebase_source : e17d08683891ca5c74bb8b3930d0aa5e62121c4f
2016-06-28 19:46:40 +02:00
49f6131a63
Bug 1280231: refactor task kinds to task classes; r=jonasfj
...
MozReview-Commit-ID: 1cNukxBgfey
--HG--
extra : rebase_source : 4f0fcce2bcea0fb78ba70e7c052638ca2c5b8a3d
extra : intermediate-source : ba5cbf4e06a550993e5216f816dcf0ccd3938b2e
extra : source : f744bd2fbcd3ae9b90851dcd12307c15d04f8bea
2016-06-27 22:57:44 +00:00
c9059e6a53
Backed out 3 changesets (bug 1280231) for decision task failures CLOSED TREE
...
Backed out changeset 4b1d94901b4d (bug 1280231)
Backed out changeset 7898d1ab1afc (bug 1280231)
Backed out changeset ba5cbf4e06a5 (bug 1280231)
--HG--
rename : taskcluster/taskgraph/taskgraph.py => taskcluster/taskgraph/types.py
2016-06-27 15:45:44 -07:00
4cfe2b182c
Bug 1280231: refactor task kinds to task classes; r=jonasfj
...
MozReview-Commit-ID: 1cNukxBgfey
--HG--
extra : source : f744bd2fbcd3ae9b90851dcd12307c15d04f8bea
2016-06-20 20:45:15 +00:00
d788a52aeb
Bug 1274611: implement docker image builds as a distinct kind; r=wcosta
...
MozReview-Commit-ID: 81Ad9LcBdx6
--HG--
extra : rebase_source : 3a2c06040f7af8b757eec2991360a7a4d3cf61cb
2016-06-05 19:15:53 +00:00
Jonas Finnemann Jensen
Dustin J. Mitchell
CuriousLearner
Hammad Akhtar
Gregory Szorc
Joel Maher
Alexandre Lissy
Wes Kocher