gecko-dev/taskcluster/taskgraph/transforms/build.py

45 строки
1.5 KiB
Python
Исходник Обычный вид История

# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
"""
Apply some defaults and minor modifications to the jobs defined in the build
kind.
"""
from __future__ import absolute_import, print_function, unicode_literals
from taskgraph.transforms.base import TransformSequence
Bug 1359976: base worker payload generation on worker-type; r=wcosta r=aki To date we have variously specified both worker-type and worker-implementation, often manually coordinated. We also embedded a few awkward assumptions such as that the native engine only runs on OS X. But a worker type has one and only one implementation, and that implementation is stable over time (as changing it would require simultaneous landings on all trees). Instead, this change makes worker-type the primary configuration, and derives both a worker implementation (defining the payload format) and worker OS (determining what to include in the payload) from that value. The derivation occurs when deciding how to implement a particular job, where the run_using functions are distinguished by worker implementation. The two-part logic to determine how and where to run a test task based on its platform is combined into a single transform, `set_worker_type`. This contains some other related changes: - MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker is used - the URL to test packages is factored out into a shared function - docker-worker test defaults are applied in `mozharness_test.py` - the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is split - the 'invalid' workerType is assigned an 'invalid' implementation - all tasks that do not use job descriptions but use docker-worker, etc. have `worker.os` added Tested to not produce a substantially different taskgraph for a regular push, a try push, or a nightly cron. MozReview-Commit-ID: LDHrmrpBo7I --HG-- extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
2017-05-09 01:53:50 +03:00
from taskgraph.util.workertypes import worker_type_implementation
transforms = TransformSequence()
@transforms.add
def set_defaults(config, jobs):
"""Set defaults, including those that differ per worker implementation"""
for job in jobs:
job['treeherder'].setdefault('kind', 'build')
job['treeherder'].setdefault('tier', 1)
Bug 1359976: base worker payload generation on worker-type; r=wcosta r=aki To date we have variously specified both worker-type and worker-implementation, often manually coordinated. We also embedded a few awkward assumptions such as that the native engine only runs on OS X. But a worker type has one and only one implementation, and that implementation is stable over time (as changing it would require simultaneous landings on all trees). Instead, this change makes worker-type the primary configuration, and derives both a worker implementation (defining the payload format) and worker OS (determining what to include in the payload) from that value. The derivation occurs when deciding how to implement a particular job, where the run_using functions are distinguished by worker implementation. The two-part logic to determine how and where to run a test task based on its platform is combined into a single transform, `set_worker_type`. This contains some other related changes: - MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker is used - the URL to test packages is factored out into a shared function - docker-worker test defaults are applied in `mozharness_test.py` - the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is split - the 'invalid' workerType is assigned an 'invalid' implementation - all tasks that do not use job descriptions but use docker-worker, etc. have `worker.os` added Tested to not produce a substantially different taskgraph for a regular push, a try push, or a nightly cron. MozReview-Commit-ID: LDHrmrpBo7I --HG-- extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
2017-05-09 01:53:50 +03:00
_, worker_os = worker_type_implementation(job['worker-type'])
worker = job.setdefault('worker', {})
worker.setdefault('env', {})
Bug 1359976: base worker payload generation on worker-type; r=wcosta r=aki To date we have variously specified both worker-type and worker-implementation, often manually coordinated. We also embedded a few awkward assumptions such as that the native engine only runs on OS X. But a worker type has one and only one implementation, and that implementation is stable over time (as changing it would require simultaneous landings on all trees). Instead, this change makes worker-type the primary configuration, and derives both a worker implementation (defining the payload format) and worker OS (determining what to include in the payload) from that value. The derivation occurs when deciding how to implement a particular job, where the run_using functions are distinguished by worker implementation. The two-part logic to determine how and where to run a test task based on its platform is combined into a single transform, `set_worker_type`. This contains some other related changes: - MOZ_AUTOMATION is set in specific jobs, rather than everywhere docker-worker is used - the URL to test packages is factored out into a shared function - docker-worker test defaults are applied in `mozharness_test.py` - the WORKER_TYPE array in `task.py`, formerly mixing two types of keys, is split - the 'invalid' workerType is assigned an 'invalid' implementation - all tasks that do not use job descriptions but use docker-worker, etc. have `worker.os` added Tested to not produce a substantially different taskgraph for a regular push, a try push, or a nightly cron. MozReview-Commit-ID: LDHrmrpBo7I --HG-- extra : rebase_source : 4cdfe6b8d9874b0c156671515b213d820b48482f
2017-05-09 01:53:50 +03:00
if worker_os == "linux":
worker.setdefault('docker-image', {'in-tree': 'desktop-build'})
worker['chain-of-trust'] = True
elif worker_os == "windows":
worker['chain-of-trust'] = True
yield job
@transforms.add
def set_env(config, jobs):
"""Set extra environment variables from try command line."""
env = {}
if config.params['try_mode'] == 'try_option_syntax':
env = config.params['try_options']['env'] or {}
for job in jobs:
if env:
job['worker']['env'].update(dict(x.split('=') for x in env))
yield job