Bug 1352113 - Shippable Builds - Support shippable where 'nightly' is used. r=aki

Makes most kinds that reference nightly attribute reference the shippable attribute.
Also makes most transforms that use nightly use shippable

Transfers dependencies/ownership for some things to shippable from nightly where it was harder to support both.

In no particular order, full list:
Send shippable attribute down to dep tasks.
Set tests as shippable attribute
Add new signing routes
Add shippable routes to repackage_routes transform
Adjust target tasks
Add shippable nightly-l10n
Add nightly-signing and as a side affect add repackage and repackage-signing
Add support for proper balrog platforms for shippable
Add shippable to the nightly sccache guard
Fix TC_PLATFORM_PER_FTP in partners.py to use shippable
Add shippable to mozharness_test variants
Only actually used for android which doesn't have shippable at this time.
Add shippable variant to beetmover transforms
Do nightly signing for mars on shippable
Support shippable in partner-repack transform
Support shippable in amo langpacks transform
Use proper signing for shippable tasks in repackage transforms
Set upload symbols to use shippable too
Use shippable as deps for geckodriver extraction
Use shippable as dep for autograph-stage signing
Do not run beetmover-l10n for shippable
Run shippable style jobs for repackage signing
Set build_platform for update verify and uvc to be shippable
Run repackage-msi for shippable
Add shippable to osx partner repack signing
add shippable to emefree repackage
add shippable to emefree repackage signing
add shippable to beetmover checksums
Add shippable to partner repack repackage signing
add partner repack beetmover
Add shippable to mar signing
Add shippable to mar-signing-l10n
add shippable to eme free beetmover checksums
Add shippable to upload-generated-sources
Add beetmover langpacks to shippable
Add repackage-l10n to shippable
Add shippable to partner repack chunk-dummy
Do eme free builds with shippable
Add signing of language packs to shippable
Add snap repackage for shippable
Add shippable for release-eme-free repack signing
Add partials for shippable
Add partner repack repackage for shippable
Add emefree beetmover for shippable
Add checksums-signing to shippable
Switch partner repacks to shippable
Add shippable to beetmover-repackage
Add secondary update verify configs for shippable
secondary update verify for shippable

Differential Revision: https://phabricator.services.mozilla.com/D24699

--HG--
extra : moz-landing-system : lando
This commit is contained in:
Justin Wood 2019-03-26 01:37:06 +00:00
Родитель ad86445b8c
Коммит 417805e566
56 изменённых файлов: 257 добавлений и 108 удалений

Просмотреть файл

@ -17,6 +17,7 @@ kind-dependencies:
only-for-attributes:
- nightly
- shippable
not-for-build-platforms:
- android-api-16-nightly/opt

Просмотреть файл

@ -14,6 +14,7 @@ kind-dependencies:
only-for-attributes:
- nightly
- shippable
job-template:
shipping-phase: promote

Просмотреть файл

@ -17,6 +17,12 @@ only-for-attributes:
- nightly
not-for-build-platforms:
- linux-shippable/opt
- linux64-shippable/opt
- macosx64-shippable/opt
- win32-shippable/opt
- win64-shippable/opt
- win64-aarch64-shippable/opt
- linux-nightly/opt
- linux64-nightly/opt
- macosx64-nightly/opt

Просмотреть файл

@ -32,6 +32,12 @@ primary-dependency:
- repackage-l10n
only-for-build-platforms:
- linux-shippable/opt
- linux64-shippable/opt
- macosx64-shippable/opt
- win32-shippable/opt
- win64-shippable/opt
- win64-aarch64-shippable/opt
- linux-nightly/opt
- linux64-nightly/opt
- macosx64-nightly/opt

Просмотреть файл

@ -16,3 +16,4 @@ kind-dependencies:
only-for-attributes:
- nightly
- shippable

Просмотреть файл

@ -32,7 +32,7 @@ jobs:
treeherder:
platform: linux32/opt
dependencies:
build-linux/opt: build-linux-nightly/opt
build-linux/opt: build-linux-shippable/opt
fetches:
build-linux/opt:
- artifact: target.common.tests.tar.gz
@ -51,7 +51,7 @@ jobs:
treeherder:
platform: linux64/opt
dependencies:
build-linux64/opt: build-linux64-nightly/opt
build-linux64/opt: build-linux64-shippable/opt
fetches:
build-linux64/opt:
- artifact: target.common.tests.tar.gz
@ -70,7 +70,7 @@ jobs:
treeherder:
platform: osx-cross/opt
dependencies:
build-macosx64/opt: build-macosx64-nightly/opt
build-macosx64/opt: build-macosx64-shippable/opt
fetches:
build-macosx64/opt:
- artifact: target.common.tests.tar.gz
@ -89,7 +89,7 @@ jobs:
treeherder:
platform: windows2012-32/opt
dependencies:
build-win32/opt: build-win32-nightly/opt
build-win32/opt: build-win32-shippable/opt
fetches:
build-win32/opt:
- artifact: target.common.tests.tar.gz
@ -108,7 +108,7 @@ jobs:
treeherder:
platform: windows2012-64/opt
dependencies:
build-win64/opt: build-win64-nightly/opt
build-win64/opt: build-win64-shippable/opt
fetches:
build-win64/opt:
- artifact: target.common.tests.tar.gz

Просмотреть файл

@ -13,7 +13,7 @@ kind-dependencies:
- repackage
only-for-build-platforms:
- linux64-nightly/opt
- linux64-shippable/opt
job-template:
shipping-phase: null

Просмотреть файл

@ -13,6 +13,12 @@ kind-dependencies:
- repackage-l10n
only-for-build-platforms:
- linux-shippable/opt
- linux64-shippable/opt
- macosx64-shippable/opt
- win32-shippable/opt
- win64-shippable/opt
- win64-aarch64-shippable/opt
- linux-nightly/opt
- linux-devedition-nightly/opt
- linux64-nightly/opt

Просмотреть файл

@ -13,11 +13,16 @@ kind-dependencies:
- repackage
only-for-build-platforms:
- linux-shippable/opt
- linux64-shippable/opt
- macosx64-shippable/opt
- win32-shippable/opt
- win64-shippable/opt
- win64-aarch64-shippable/opt
- linux-nightly/opt
- linux-devedition-nightly/opt
- linux64-nightly/opt
- linux64-devedition-nightly/opt
- linux64-asan-reporter-nightly/opt
- macosx64-nightly/opt
- macosx64-devedition-nightly/opt
- win32-nightly/opt

Просмотреть файл

@ -15,3 +15,4 @@ kind-dependencies:
only-for-attributes:
- nightly
- shippable

Просмотреть файл

@ -21,13 +21,19 @@ kind-dependencies:
primary-dependency: build
only-for-build-platforms:
- linux64-nightly/opt
- linux-nightly/opt
- android-api-16-nightly/opt
- linux64-nightly/opt
- linux64-shippable/opt
- linux-nightly/opt
- linux-shippable/opt
- macosx64-nightly/opt
- macosx64-shippable/opt
- win32-nightly/opt
- win32-shippable/opt
- win64-nightly/opt
- win64-shippable/opt
- win64-aarch64-nightly/opt
- win64-aarch64-shippable/opt
- linux64-devedition-nightly/opt
- linux-devedition-nightly/opt
- macosx64-devedition-nightly/opt
@ -74,18 +80,13 @@ job-template:
- android-sdk-linux
tooltool:
by-build-platform:
default: public
android-api-16-nightly: internal
macosx64-nightly: internal
macosx64-devedition-nightly: internal
win32-nightly: internal
win32-devedition-nightly: internal
win64-nightly: internal
win64-devedition-nightly: internal
win64-aarch64-nightly: internal
win64-aarch64-devedition-nightly: internal
default: internal
linux.*: public
index:
type: nightly-l10n
type:
by-build-platform:
.*nightly.*: nightly-l10n
default: shippable-l10n
product:
by-build-platform:
default: firefox
@ -99,6 +100,12 @@ job-template:
win32-nightly: win32-opt
win64-nightly: win64-opt
win64-aarch64-nightly: win64-aarch64-opt
linux-shippable: linux-opt
linux64-shippable: linux64-opt
macosx64-shippable: macosx64-opt
win32-shippable: win32-opt
win64-shippable: win64-opt
win64-aarch64-shippable: win64-aarch64-opt
linux-devedition-nightly: linux-devedition-opt
linux64-devedition-nightly: linux64-devedition-opt
macosx64-devedition-nightly: macosx64-devedition-opt
@ -125,6 +132,12 @@ job-template:
win32-nightly: windows2012-32/opt
win64-nightly: windows2012-64/opt
win64-aarch64-nightly: windows2012-aarch64/opt
linux64-shippable: linux64-shippable/opt
linux-shippable: linux32-shippable/opt
macosx64-shippable: osx-shippable/opt
win32-shippable: windows2012-32-shippable/opt
win64-shippable: windows2012-64-shippable/opt
win64-aarch64-shippable: windows2012-aarch64-shippable/opt
linux64-devedition-nightly: linux64-devedition/opt
linux-devedition-nightly: linux32-devedition/opt
macosx64-devedition-nightly: osx-cross-devedition/opt
@ -153,37 +166,37 @@ job-template:
mozharness:
config:
by-build-platform:
linux-nightly:
linux-(?:nightly|shippable):
- single_locale/firefox.py
- single_locale/linux32.py
- single_locale/tc_common.py
- single_locale/tc_linux_common.py
- taskcluster_nightly.py
linux64-nightly:
linux64-(?:nightly|shippable):
- single_locale/firefox.py
- single_locale/linux64.py
- single_locale/tc_common.py
- single_locale/tc_linux_common.py
- taskcluster_nightly.py
macosx64-nightly:
macosx64-(?:nightly|shippable):
- single_locale/firefox.py
- single_locale/macosx64.py
- single_locale/tc_common.py
- single_locale/tc_macosx64.py
- taskcluster_nightly.py
win32-nightly:
win32-(?:nightly|shippable):
- single_locale/firefox.py
- single_locale/win32.py
- single_locale/tc_common.py
- single_locale/tc_win32.py
- taskcluster_nightly.py
win64-nightly:
win64-(?:nightly|shippable):
- single_locale/firefox.py
- single_locale/win64.py
- single_locale/tc_common.py
- single_locale/tc_win64.py
- taskcluster_nightly.py
win64-aarch64-nightly:
win64-aarch64-(?:nightly|shippable):
- single_locale/firefox.py
- single_locale/win64-aarch64.py
- single_locale/tc_common.py

Просмотреть файл

@ -15,18 +15,25 @@ kind-dependencies:
only-for-attributes:
- nightly
- shippable
only-for-build-platforms:
- macosx64-shippable/opt
- macosx64-nightly/opt
- macosx64-devedition-nightly/opt
- win32-shippable/opt
- win32-nightly/opt
- win32-devedition-nightly/opt
- win64-shippable/opt
- win64-nightly/opt
- win64-devedition-nightly/opt
- win64-aarch64-shippable/opt
- win64-aarch64-nightly/opt
- win64-aarch64-devedition-nightly/opt
- linux-shippable/opt
- linux-nightly/opt
- linux-devedition-nightly/opt
- linux64-shippable/opt
- linux64-nightly/opt
- linux64-devedition-nightly/opt
- linux64-asan-reporter-nightly/opt

Просмотреть файл

@ -14,6 +14,7 @@ kind-dependencies:
only-for-attributes:
- nightly
- shippable
job-template:
description: Beetmover submission for platform-independent langpacks {locales} in {platform} directory

Просмотреть файл

@ -13,6 +13,9 @@ kind-dependencies:
- release-eme-free-repack-beetmover
only-for-build-platforms:
- macosx64-shippable/opt
- win32-shippable/opt
- win64-shippable/opt
- macosx64-nightly/opt
- win32-nightly/opt
- win64-nightly/opt

Просмотреть файл

@ -13,6 +13,9 @@ kind-dependencies:
- release-eme-free-repack-repackage-signing
only-for-build-platforms:
- macosx64-shippable/opt
- win32-shippable/opt
- win64-shippable/opt
- macosx64-nightly/opt
- win32-nightly/opt
- win64-nightly/opt

Просмотреть файл

@ -13,6 +13,9 @@ kind-dependencies:
- release-eme-free-repack-repackage
only-for-build-platforms:
- win32-shippable/opt
- win32-nightly/opt
- win64-shippable/opt
- win64-nightly/opt
- macosx64-shippable/opt
- macosx64-nightly/opt

Просмотреть файл

@ -18,8 +18,11 @@ kind-dependencies:
- toolchain
only-for-build-platforms:
- macosx64-shippable/opt
- macosx64-nightly/opt
- win32-shippable/opt
- win32-nightly/opt
- win64-shippable/opt
- win64-nightly/opt
job-template:

Просмотреть файл

@ -15,6 +15,7 @@ kind-dependencies:
- release-eme-free-repack
only-for-build-platforms:
- macosx64-shippable/opt
- macosx64-nightly/opt
job-template:

Просмотреть файл

@ -41,38 +41,38 @@ job-defaults:
production: false
jobs:
macosx64-nightly:
macosx64-shippable:
treeherder:
symbol: EME
platform: macosx64/opt
kind: test
tier: 1
attributes:
build_platform: macosx64-nightly
build_platform: macosx64-shippable
build_type: opt
artifact_prefix: releng/partner
nightly: true
win32-nightly:
win32-shippable:
treeherder:
symbol: EME
platform: win32/opt
kind: test
tier: 1
attributes:
build_platform: win32-nightly
build_platform: win32-shippable
build_type: opt
artifact_prefix: releng/partner
nightly: true
win64-nightly:
win64-shippable:
treeherder:
symbol: EME
platform: win64/opt
kind: test
tier: 1
attributes:
build_platform: win64-nightly
build_platform: win64-shippable
build_type: opt
artifact_prefix: releng/partner
nightly: true

Просмотреть файл

@ -13,6 +13,12 @@ kind-dependencies:
- release-partner-repack-repackage-signing
only-for-build-platforms:
- linux-shippable/opt
- linux64-shippable/opt
- macosx64-shippable/opt
- win32-shippable/opt
- win64-shippable/opt
- win64-aarch64-shippable/opt
- linux-nightly/opt
- linux64-nightly/opt
- macosx64-nightly/opt

Просмотреть файл

@ -22,6 +22,8 @@ kind-dependencies:
- release-partner-repack
only-for-build-platforms:
- linux-shippable/opt
- linux64-shippable/opt
- linux-nightly/opt
- linux64-nightly/opt

Просмотреть файл

@ -14,6 +14,12 @@ kind-dependencies:
- release-partner-repack-repackage # Windows, Mac
only-for-build-platforms:
- linux-shippable/opt
- linux64-shippable/opt
- macosx64-shippable/opt
- win32-shippable/opt
- win64-shippable/opt
- win64-aarch64-shippable/opt
- linux-nightly/opt
- linux64-nightly/opt
- macosx64-nightly/opt

Просмотреть файл

@ -18,6 +18,10 @@ kind-dependencies:
- toolchain
only-for-build-platforms:
- macosx64-shippable/opt
- win32-shippable/opt
- win64-shippable/opt
- win64-aarch64-shippable/opt
- macosx64-nightly/opt
- win32-nightly/opt
- win64-nightly/opt

Просмотреть файл

@ -16,6 +16,7 @@ kind-dependencies:
only-for-build-platforms:
- macosx64-nightly/opt
- macosx64-shippable/opt
job-template:
shipping-product: firefox

Просмотреть файл

@ -41,44 +41,44 @@ job-defaults:
production: false
jobs:
linux-nightly:
linux-shippable:
attributes:
build_platform: linux-nightly
build_platform: linux-shippable
build_type: opt
artifact_prefix: releng/partner
nightly: true
shippable: true
linux64-nightly:
linux64-shippable:
attributes:
build_platform: linux64-nightly
build_platform: linux64-shippable
build_type: opt
artifact_prefix: releng/partner
nightly: true
shippable: true
macosx64-nightly:
macosx64-shippable:
attributes:
build_platform: macosx64-nightly
build_platform: macosx64-shippable
build_type: opt
artifact_prefix: releng/partner
nightly: true
shippable: true
win32-nightly:
win32-shippable:
attributes:
build_platform: win32-nightly
build_platform: win32-shippable
build_type: opt
artifact_prefix: releng/partner
nightly: true
shippable: true
win64-nightly:
win64-shippable:
attributes:
build_platform: win64-nightly
build_platform: win64-shippable
build_type: opt
artifact_prefix: releng/partner
nightly: true
shippable: true
win64-aarch64-nightly:
win64-aarch64-shippable:
attributes:
build_platform: win64-aarch64-nightly
build_platform: win64-aarch64-shippable
build_type: opt
artifact_prefix: releng/partner
nightly: true
shippable: true

Просмотреть файл

@ -58,7 +58,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: linux-nightly
build_platform: linux-shippable
extra:
platform: linux-i686
updater-platform: linux-x86_64
@ -70,7 +70,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: linux64-nightly
build_platform: linux64-shippable
extra:
platform: linux-x86_64
updater-platform: linux-x86_64
@ -82,7 +82,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: macosx64-nightly
build_platform: macosx64-shippable
extra:
platform: mac
updater-platform: linux-x86_64
@ -94,7 +94,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: win32-nightly
build_platform: win32-shippable
extra:
platform: win32
updater-platform: linux-x86_64
@ -106,7 +106,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: win64-nightly
build_platform: win64-shippable
extra:
platform: win64
updater-platform: linux-x86_64

Просмотреть файл

@ -46,7 +46,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: linux64-nightly
build_platform: linux64-shippable
firefox-secondary-linux:
description: linux secondary channel update verify
@ -57,7 +57,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: linux-nightly
build_platform: linux-shippable
firefox-secondary-win64:
description: win64 secondary channel update verify
@ -68,7 +68,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: win64-nightly
build_platform: win64-shippable
firefox-secondary-win32:
description: win32 secondary channel update verify
@ -79,7 +79,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: win32-nightly
build_platform: win32-shippable
firefox-secondary-macosx64:
description: macosx64 secondary channel update verify
@ -90,4 +90,4 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: macosx64-nightly
build_platform: macosx64-shippable

Просмотреть файл

@ -14,9 +14,9 @@ kind-dependencies:
only-for-build-platforms:
- linux64-nightly/opt # addons.mozilla.org only support 1 platform per locale. That's why we use linux64
- linux64-shippable/opt # addons.mozilla.org only support 1 platform per locale. That's why we use linux64
- linux64-devedition-nightly/opt
- macosx64-nightly/opt # Although, we need the special locale "ja-JP-Mac" from this platform
- macosx64-shippable/opt # Although, we need the special locale "ja-JP-Mac" from this platform
- macosx64-devedition-nightly/opt
@ -39,6 +39,7 @@ job-template:
by-platform:
# ja-JP-mac is only langpack on mac, and is unlisted always
macosx64-nightly.*: unlisted
macosx64-shippable.*: unlisted
macosx64-devedition.*: unlisted
linux.*-devedition.*: unlisted
default: listed

Просмотреть файл

@ -19,7 +19,7 @@ job-defaults:
shipping-phase: promote
scopes: []
treeherder:
platform: linux64/opt
platform: linux64-shippable/opt
kind: build
tier: 2
worker-type: aws-provisioner-v1/gecko-{level}-b-linux
@ -59,6 +59,6 @@ jobs:
firefox:
shipping-product: firefox
attributes:
build_platform: linux64-nightly
build_platform: linux64-shippable
treeherder:
symbol: Snap(r)

Просмотреть файл

@ -77,7 +77,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: linux-nightly
build_platform: linux-shippable
extra:
product: firefox
platform: linux-i686
@ -101,7 +101,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: linux64-nightly
build_platform: linux64-shippable
extra:
product: firefox
platform: linux-x86_64
@ -125,7 +125,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: macosx64-nightly
build_platform: macosx64-shippable
extra:
product: firefox
platform: mac
@ -149,7 +149,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: win32-nightly
build_platform: win32-shippable
extra:
product: firefox
platform: win32
@ -173,7 +173,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: win64-nightly
build_platform: win64-shippable
extra:
product: firefox
platform: win64

Просмотреть файл

@ -53,7 +53,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: linux64-nightly
build_platform: linux64-shippable
firefox-linux:
description: linux update verify
@ -73,7 +73,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: linux-nightly
build_platform: linux-shippable
firefox-win64:
description: win64 update verify
@ -93,7 +93,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: win64-nightly
build_platform: win64-shippable
firefox-win32:
description: win32 update verify
@ -113,7 +113,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: win32-nightly
build_platform: win32-shippable
firefox-macosx64:
description: macosx64 update verify
@ -133,7 +133,7 @@ jobs:
kind: test
tier: 1
attributes:
build_platform: macosx64-nightly
build_platform: macosx64-shippable
devedition-linux64:
description: linux64 update verify

Просмотреть файл

@ -17,6 +17,12 @@ kind-dependencies:
- toolchain
only-for-build-platforms:
- linux-shippable/opt
- linux64-shippable/opt
- macosx64-shippable/opt
- win32-shippable/opt
- win64-shippable/opt
- win64-aarch64-shippable/opt
- linux-nightly/opt
- linux64-nightly/opt
- macosx64-nightly/opt

Просмотреть файл

@ -17,7 +17,9 @@ kind-dependencies:
- fetch
only-for-build-platforms:
- win32-shippable/opt
- win32-nightly/opt
- win64-shippable/opt
- win64-nightly/opt
- win32-devedition-nightly/opt
- win64-devedition-nightly/opt

Просмотреть файл

@ -14,12 +14,15 @@ kind-dependencies:
- repackage-l10n
only-for-build-platforms:
- win32-shippable/opt
- win32-nightly/opt
- win32-devedition-nightly/opt
- win32/opt
- win64-shippable/opt
- win64-nightly/opt
- win64-devedition-nightly/opt
- win64/opt
- win64-aarch64-shippable/opt
- win64-aarch64-nightly/opt
- win64-aarch64-devedition-nightly/opt
- win64-asan-reporter-nightly/opt

Просмотреть файл

@ -13,9 +13,11 @@ kind-dependencies:
- repackage-msi
only-for-build-platforms:
- win32-shippable/opt
- win32-nightly/opt
- win32-devedition-nightly/opt
- win32/opt
- win64-shippable/opt
- win64-nightly/opt
- win64-devedition-nightly/opt
- win64/opt

Просмотреть файл

@ -14,11 +14,10 @@ kind-dependencies:
only-for-attributes:
- nightly
- shippable
job-template:
description: Upload generated source files from build
attributes:
nightly: true
worker-type: aws-provisioner-v1/gecko-t-linux-xlarge
treeherder:
symbol: Ugs

Просмотреть файл

@ -51,6 +51,8 @@ def loader(kind, path, config, params, loaded_tasks):
test['test-name'] = test_name
if test_platform.get('nightly'):
test.setdefault('attributes', {})['nightly'] = True
if test_platform.get('shippable'):
test.setdefault('attributes', {})['shippable'] = True
logger.debug("Generating tasks for test {} on platform {}".format(
test_name, test['test-platform']))
@ -97,6 +99,9 @@ def get_test_platforms(test_platforms_cfg, builds_by_platform, signed_builds_by_
if builds_by_platform[build_platform].attributes.get('nightly'):
test_platforms[test_platform]['nightly'] = \
builds_by_platform[build_platform].attributes['nightly']
if builds_by_platform[build_platform].attributes.get('shippable'):
test_platforms[test_platform]['shippable'] = \
builds_by_platform[build_platform].attributes['shippable']
test_platforms[test_platform].update(cfg)

Просмотреть файл

@ -25,7 +25,10 @@ def get_method(method):
def filter_out_nightly(task, parameters):
return not task.attributes.get('nightly')
return (
not task.attributes.get('nightly') and
task.attributes.get('shipping_phase') in (None, 'build')
)
def filter_out_cron(task, parameters):
@ -468,9 +471,13 @@ def make_desktop_nightly_filter(platforms):
return all([
filter_on_platforms(task, platforms),
filter_for_project(task, parameters),
task.attributes.get('nightly', False),
any([
task.attributes.get('nightly', False),
task.attributes.get('shippable', False),
]),
# Tests and nightly only builds don't have `shipping_product` set
task.attributes.get('shipping_product') in {None, "firefox", "thunderbird"},
task.kind not in {'l10n'}, # no on-change l10n
])
return filter
@ -480,7 +487,9 @@ def target_tasks_nightly_linux(full_task_graph, parameters, graph_config):
"""Select the set of tasks required for a nightly build of linux. The
nightly build process involves a pipeline of builds, signing,
and, eventually, uploading the tasks to balrog."""
filter = make_desktop_nightly_filter({'linux64-nightly', 'linux-nightly'})
filter = make_desktop_nightly_filter({
'linux64-nightly', 'linux-nightly', 'linux64-shippable', 'linux-shippable'
})
return [l for l, t in full_task_graph.tasks.iteritems() if filter(t, parameters)]
@ -489,7 +498,7 @@ def target_tasks_nightly_macosx(full_task_graph, parameters, graph_config):
"""Select the set of tasks required for a nightly build of macosx. The
nightly build process involves a pipeline of builds, signing,
and, eventually, uploading the tasks to balrog."""
filter = make_desktop_nightly_filter({'macosx64-nightly'})
filter = make_desktop_nightly_filter({'macosx64-nightly', 'macosx64-shippable'})
return [l for l, t in full_task_graph.tasks.iteritems() if filter(t, parameters)]
@ -498,7 +507,7 @@ def target_tasks_nightly_win32(full_task_graph, parameters, graph_config):
"""Select the set of tasks required for a nightly build of win32 and win64.
The nightly build process involves a pipeline of builds, signing,
and, eventually, uploading the tasks to balrog."""
filter = make_desktop_nightly_filter({'win32-nightly'})
filter = make_desktop_nightly_filter({'win32-nightly', 'win32-shippable'})
return [l for l, t in full_task_graph.tasks.iteritems() if filter(t, parameters)]
@ -507,7 +516,7 @@ def target_tasks_nightly_win64(full_task_graph, parameters, graph_config):
"""Select the set of tasks required for a nightly build of win32 and win64.
The nightly build process involves a pipeline of builds, signing,
and, eventually, uploading the tasks to balrog."""
filter = make_desktop_nightly_filter({'win64-nightly'})
filter = make_desktop_nightly_filter({'win64-nightly', 'win64-shippable'})
return [l for l, t in full_task_graph.tasks.iteritems() if filter(t, parameters)]
@ -516,7 +525,7 @@ def target_tasks_nightly_win64_aarch64(full_task_graph, parameters, graph_config
"""Select the set of tasks required for a nightly build of win32 and win64.
The nightly build process involves a pipeline of builds, signing,
and, eventually, uploading the tasks to balrog."""
filter = make_desktop_nightly_filter({'win64-aarch64-nightly'})
filter = make_desktop_nightly_filter({'win64-aarch64-nightly', 'win64-aarch64-shippable'})
return [l for l, t in full_task_graph.tasks.iteritems() if filter(t, parameters)]

Просмотреть файл

@ -79,7 +79,7 @@ _DESKTOP_UPSTREAM_ARTIFACTS_UNSIGNED_L10N = [
# with a beetmover patch in https://github.com/mozilla-releng/beetmoverscript/.
# See example in bug 1348286
UPSTREAM_ARTIFACT_UNSIGNED_PATHS = _compile_regex_mapping({
r'^(linux(|64)|macosx64)(|-devedition)-nightly$':
r'^(linux(|64)|macosx64)(|-devedition)-(nightly|shippable)$':
_DESKTOP_UPSTREAM_ARTIFACTS_UNSIGNED_EN_US + [
'host/bin/mar',
'host/bin/mbsdiff',
@ -96,12 +96,12 @@ UPSTREAM_ARTIFACT_UNSIGNED_PATHS = _compile_regex_mapping({
"host/bin/mar.exe",
"host/bin/mbsdiff.exe",
]),
r'^win(32|64(|-aarch64))(|-devedition)-nightly$':
r'^win(32|64(|-aarch64))(|-devedition)-(nightly|shippable)$':
_DESKTOP_UPSTREAM_ARTIFACTS_UNSIGNED_EN_US + [
'host/bin/mar.exe',
'host/bin/mbsdiff.exe',
],
r'^(linux(|64)|macosx64|win(32|64))(|-devedition)-nightly-l10n$':
r'^(linux(|64)|macosx64|win(32|64))(|-devedition)-(nightly|shippable)-l10n$':
_DESKTOP_UPSTREAM_ARTIFACTS_UNSIGNED_L10N,
})
@ -110,9 +110,10 @@ UPSTREAM_ARTIFACT_UNSIGNED_PATHS = _compile_regex_mapping({
# with a beetmover patch in https://github.com/mozilla-releng/beetmoverscript/.
# See example in bug 1348286
UPSTREAM_ARTIFACT_SIGNED_PATHS = _compile_regex_mapping({
r'^linux(|64)(|-devedition|-asan-reporter)-nightly(|-l10n)$':
r'^linux(|64)(|-devedition|-asan-reporter)-(nightly|shippable)(|-l10n)$':
['target.tar.bz2', 'target.tar.bz2.asc'],
r'^win(32|64)(|-aarch64)(|-devedition|-asan-reporter)-nightly(|-l10n)$': ['target.zip'],
r'^win(32|64)(|-aarch64)(|-devedition|-asan-reporter)-(nightly|shippable)(|-l10n)$':
['target.zip'],
})
# Until bug 1331141 is fixed, if you are adding any new artifacts here that

Просмотреть файл

@ -24,6 +24,14 @@ def add_signed_routes(config, jobs):
dep_job = job['primary-dependency']
job['routes'] = []
if dep_job.attributes.get('shippable'):
for dep_route in dep_job.task.get('routes', []):
if not dep_route.startswith('index.gecko.v2'):
continue
branch = dep_route.split(".")[3]
rest = ".".join(dep_route.split(".")[4:])
job['routes'].append(
'index.gecko.v2.{}.signed.{}'.format(branch, rest))
if dep_job.attributes.get('nightly'):
for dep_route in dep_job.task.get('routes', []):
if not dep_route.startswith('index.gecko.v2'):

Просмотреть файл

@ -21,6 +21,7 @@ import os
VARIANTS = [
'nightly',
'shippable',
'devedition',
'pgo',
'asan',
@ -53,7 +54,8 @@ def test_packages_url(taskdesc):
'target.test_packages.json'))
# for android nightly we need to add 'en-US' to the artifact url
test = taskdesc['run']['test']
if get_variant(test['test-platform']) == "nightly" and 'android' in test['test-platform']:
if 'android' in test['test-platform'] and (
get_variant(test['test-platform']) in ("nightly", 'shippable')):
head, tail = os.path.split(artifact_url)
artifact_url = os.path.join(head, 'en-US', tail)
return artifact_url

Просмотреть файл

@ -80,7 +80,7 @@ l10n_description_schema = schema.extend({
Required('job-name'): _by_platform(basestring),
# Type of index
Optional('type'): basestring,
Optional('type'): _by_platform(basestring),
},
# Description of the localized task
Required('description'): _by_platform(basestring),
@ -245,6 +245,7 @@ def handle_keyed_by(config, jobs):
"mozharness.script",
"treeherder.tier",
"treeherder.platform",
"index.type",
"index.product",
"index.job-name",
"when.files-changed",

Просмотреть файл

@ -135,7 +135,11 @@ def make_task_description(config, jobs):
upstream_artifacts = generate_complete_artifacts(dep_job, config.kind)
build_platform = dep_job.attributes.get('build_platform')
is_nightly = job.get('nightly', dep_job.attributes.get('nightly'))
is_nightly = job.get(
'nightly', # First check current job
dep_job.attributes.get(
'nightly', # Then dep job for 'nightly'
dep_job.attributes.get('shippable'))) # lastly dep job for 'shippable'
signing_cert_scope = get_signing_cert_scope_per_platform(
build_platform, is_nightly, config
)

Просмотреть файл

@ -66,16 +66,19 @@ def add_command_arguments(config, tasks):
all_locales.update(sub_partner.get('locales', []))
for task in tasks:
# add the MOZHARNESS_OPTIONS, eg version=61.0, build-number=1, platform=win64
if not task['attributes']['build_platform'].endswith('-nightly'):
if not task['attributes']['build_platform'].endswith('-shippable') and \
not task['attributes']['build_platform'].endswith('-nightly'):
raise Exception(
"Unexpected partner repack platform: {}".format(
task['attributes']['build_platform'],
),
)
platform = task['attributes']['build_platform'].partition('-shippable')[0]
platform = platform.partition('-nightly')[0]
task['run']['options'] = [
'version={}'.format(release_config['version']),
'build-number={}'.format(release_config['build_number']),
'platform={}'.format(task['attributes']['build_platform'].partition('-nightly')[0]),
'platform={}'.format(platform),
]
if task['extra']['limit-locales']:
for locale in all_locales:

Просмотреть файл

@ -82,9 +82,13 @@ def filter_out_macos_jobs_but_mac_only_locales(config, jobs):
for job in jobs:
build_platform = job['primary-dependency'].attributes.get('build_platform')
if build_platform in ('linux64-nightly', 'linux64-devedition-nightly'):
if build_platform in (
'linux64-nightly', 'linux64-devedition-nightly',
'linux64-shippable'):
yield job
elif build_platform in ('macosx64-nightly', 'macosx64-devedition-nightly') and \
elif build_platform in (
'macosx64-nightly', 'macosx64-devedition-nightly',
'macosx64-shippable') and \
'ja-JP-mac' in job['attributes']['chunk_locales']:
# Other locales of the same job shouldn't be processed
job['attributes']['chunk_locales'] = ['ja-JP-mac']

Просмотреть файл

@ -21,8 +21,12 @@ def add_indexes(config, jobs):
job_name = '{}-{}'.format(build_platform, repackage_type)
product = job.get('index', {}).get('product', 'firefox')
index_type = 'generic'
if job['attributes'].get('shippable') and job['attributes'].get('locale'):
index_type = 'shippable-l10n'
if job['attributes'].get('nightly') and job['attributes'].get('locale'):
index_type = 'nightly-l10n'
if job['attributes'].get('nightly'):
index_type = 'shippable'
if job['attributes'].get('nightly'):
index_type = 'nightly'
if job['attributes'].get('locale'):

Просмотреть файл

@ -89,7 +89,7 @@ def make_repackage_signing_description(config, jobs):
)
build_platform = dep_job.attributes.get('build_platform')
is_nightly = dep_job.attributes.get('nightly')
is_nightly = dep_job.attributes.get('nightly', dep_job.attributes.get('shippable'))
signing_cert_scope = get_signing_cert_scope_per_platform(
build_platform, is_nightly, config
)

Просмотреть файл

@ -41,7 +41,7 @@ def make_repackage_signing_description(config, jobs):
repack_id = dep_job.task['extra']['repack_id']
attributes = dep_job.attributes
build_platform = dep_job.attributes.get('build_platform')
is_nightly = dep_job.attributes.get('nightly')
is_nightly = dep_job.attributes.get('nightly', dep_job.attributes.get('shippable'))
# Mac & windows
label = dep_job.label.replace("repackage-", "repackage-signing-")

Просмотреть файл

@ -88,7 +88,8 @@ def make_task_description(config, jobs):
add_scope_prefix(config, 'signing:format:{}'.format(format))
)
is_nightly = dep_job.attributes.get('nightly', False)
is_nightly = dep_job.attributes.get(
'nightly', dep_job.attributes.get('shippable', False))
treeherder = None
if 'partner' not in config.kind and 'eme-free' not in config.kind:
treeherder = job.get('treeherder', {})

Просмотреть файл

@ -127,7 +127,7 @@ task_description_schema = Schema({
# Type of gecko v2 index to use
'type': Any('generic', 'nightly', 'l10n', 'nightly-with-multi-l10n',
'release', 'nightly-l10n', 'shippable'),
'release', 'nightly-l10n', 'shippable', 'shippable-l10n'),
# The rank that the task will receive in the TaskCluster
# index. A newly completed task supercedes the currently

Просмотреть файл

@ -29,6 +29,10 @@ def add_task_info(config, jobs):
dep_th = dep_task.task['extra']['treeherder']
job.setdefault('attributes', {})
job['attributes']['build_platform'] = dep_task.attributes.get('build_platform')
if dep_task.attributes.get('nightly'):
job['attributes']['nightly'] = True
if dep_task.attributes.get('shippable'):
job['attributes']['shippable'] = True
plat = '{}/{}'.format(dep_th['machine']['platform'], dep_task.attributes.get('build_type'))
job['treeherder']['platform'] = plat
job['treeherder']['tier'] = dep_th['tier']

Просмотреть файл

@ -28,7 +28,7 @@ def check_nightlies(config, tasks):
for task in tasks:
dep = task['primary-dependency']
if config.params['project'] in RELEASE_PROJECTS and \
dep.attributes.get('nightly') and \
dep.attributes.get('nightly', dep.attributes.get('shippable')) and \
not dep.attributes.get('enable-full-crashsymbols'):
raise Exception('Nightly job %s should have enable-full-crashsymbols attribute '
'set to true to enable symbol upload to crash-stats' % dep.label)
@ -61,6 +61,8 @@ def fill_template(config, tasks):
attributes['build_type'] = build_type
if dep.attributes.get('nightly'):
attributes['nightly'] = True
if dep.attributes.get('shippable'):
attributes['shippable'] = True
treeherder = task.get('treeherder', {})
th = dep.task.get('extra')['treeherder']
@ -82,8 +84,12 @@ def fill_template(config, tasks):
if dep.attributes.get('nightly'):
# For nightly builds, we want to run these tasks if the build is run.
task['run-on-projects'] = dep.attributes.get('run_on_projects')
elif dep.attributes.get('shippable'):
# For shippable builds, we want to run these tasks if the build is run.
# XXX Better to run this on promote phase instead?
task['run-on-projects'] = dep.attributes.get('run_on_projects')
else:
# For non nightly builds, these can be requested to upload to the try symbol sever.
# For other builds, these can be requested to upload to the try symbol sever.
task['run-on-projects'] = ['try']
# clear out the stuff that's not part of a task description

Просмотреть файл

@ -43,6 +43,7 @@ _OPTIONAL_ATTRIBUTES = (
'shipping_product',
'stub-installer',
'update-channel',
'shippable',
)

Просмотреть файл

@ -18,6 +18,7 @@ PLATFORM_RENAMES = {
'windows2012-64': 'win64',
'windows2012-aarch64': 'win64-aarch64',
'osx-cross': 'macosx64',
'osx': 'macosx64',
}
BALROG_PLATFORM_MAP = {
@ -80,6 +81,8 @@ def get_balrog_platform_name(platform):
platform = platform.replace('-nightly', '')
if '-devedition' in platform:
platform = platform.replace('-devedition', '')
if '-shippable' in platform:
platform = platform.replace('-shippable', '')
return PLATFORM_RENAMES.get(platform, platform)

Просмотреть файл

@ -121,12 +121,12 @@ REPACK_CFG_QUERY = """query{
# Map platforms in repack.cfg into their equivalents in taskcluster
TC_PLATFORM_PER_FTP = {
'linux-i686': 'linux-nightly',
'linux-x86_64': 'linux64-nightly',
'mac': 'macosx64-nightly',
'win32': 'win32-nightly',
'win64': 'win64-nightly',
'win64-aarch64': 'win64-aarch64-nightly',
'linux-i686': 'linux-shippable',
'linux-x86_64': 'linux64-shippable',
'mac': 'macosx64-shippable',
'win32': 'win32-shippable',
'win64': 'win64-shippable',
'win64-aarch64': 'win64-aarch64-shippable',
}
TASKCLUSTER_PROXY_SECRET_ROOT = 'http://taskcluster/secrets/v1/secret'

Просмотреть файл

@ -222,7 +222,7 @@ def verify_always_optimized(task, taskgraph, scratch_pad, graph_config):
@verifications.add('full_task_graph')
def verify_nightly_no_sccache(task, taskgraph, scratch_pad, graph_config):
if task and task.attributes.get('nightly'):
if task and any([task.attributes.get('nightly'), task.attributes.get('shippable')]):
if task.task.get('payload', {}).get('env', {}).get('USE_SCCACHE'):
raise Exception(
'Nightly job {} cannot use sccache'.format(task.label))