зеркало из https://github.com/mozilla/gecko-dev.git
Bug 1460777 - Taskgraph tasks for retrieving remote content; r=dustin, glandium
Currently, many tasks fetch content from the Internets. A problem with that is fetching from the Internets is unreliable: servers may have outages or be slow; content may disappear or change out from under us. The unreliability of 3rd party services poses a risk to Firefox CI. If services aren't available, we could potentially not run some CI tasks. In the worst case, we might not be able to release Firefox. That would be bad. In fact, as I write this, gmplib.org has been unavailable for ~24 hours and Firefox CI is unable to retrieve the GMP source code. As a result, building GCC toolchains is failing. A solution to this is to make tasks more hermetic by depending on fewer network services (which by definition aren't reliable over time and therefore introduce instability). This commit attempts to mitigate some external service dependencies by introducing the *fetch* task kind. The primary goal of the *fetch* kind is to obtain remote content and re-expose it as a task artifact. By making external content available as a cached task artifact, we allow dependent tasks to consume this content without touching the service originally providing that content, thus eliminating a run-time dependency and making tasks more hermetic and reproducible over time. We introduce a single "fetch-url" "using" flavor to define tasks that fetch single URLs and then re-expose that URL as an artifact. Powering this is a new, minimal "fetch" Docker image that contains a "fetch-content" Python script that does the work for us. We have added tasks to fetch source archives used to build the GCC toolchains. Fetching remote content and re-exposing it as an artifact is not very useful by itself: the value is in having tasks use those artifacts. We introduce a taskgraph transform that allows tasks to define an array of "fetches." Each entry corresponds to the name of a "fetch" task kind. When present, the corresponding "fetch" task is added as a dependency. And the task ID and artifact path from that "fetch" task is added to the MOZ_FETCHES environment variable of the task depending on it. Our "fetch-content" script has a "task-artifacts" sub-command that tasks can execute to perform retrieval of all artifacts listed in MOZ_FETCHES. To prove all of this works, the code for fetching dependencies when building GCC toolchains has been updated to use `fetch-content`. The now-unused legacy code has been deleted. This commit improves the reliability and efficiency of GCC toolchain tasks. Dependencies now all come from task artifacts and should always be available in the common case. In addition, `fetch-content` downloads and extracts files concurrently. This makes it faster than the serial application which we were previously using. There are some things I don't like about this commit. First, a new Docker image and Python script for downloading URLs feels a bit heavyweight. The Docker image is definitely overkill as things stand. I can eventually justify it because I want to implement support for fetching and repackaging VCS repositories and for caching Debian packages. These will require more packages than what I'm comfortable installing on the base Debian image, therefore justifying a dedicated image. The `fetch-content static-url` sub-command could definitely be implemented as a shell script. But Python is readily available and is more pleasant to maintain than shell, so I wrote it in Python. `fetch-content task-artifacts` is more advanced and writing it in Python is more justified, IMO. FWIW, the script is Python 3 only, which conveniently gives us access to `concurrent.futures`, which facilitates concurrent download. `fetch-content` also duplicates functionality found elsewhere. generic-worker's task payload supports a "mounts" feature which facilitates downloading remote content, including from a task artifact. However, this feature doesn't exist on docker-worker. So we have to implement downloading inside the task rather than at the worker level. I concede that if all workers had generic-worker's "mounts" feature and supported concurrent download, `fetch-content` wouldn't need to exist. `fetch-content` also duplicates functionality of `mach artifact toolchain`. I probably could have used `mach artifact toolchain` instead of writing `fetch-content task-artifacts`. However, I didn't want to introduce the requirement of a VCS checkout. `mach artifact toolchain` has its origins in providing a feature to the build system. And "fetching artifacts from tasks" is a more generic feature than that. I think it should be implemented as a generic feature and not something that is "toolchain" specific. I think the best place for a generic "fetch content" feature is in the worker, where content can be defined in the task payload. But as explained above, that feature isn't universally available. The next best place is probably run-task. run-task already performs generic, very-early task preparation steps, such as performing a VCS checkout. I would like to fold `fetch-content` into run-task and make it all driven by environment variables. But run-task is currently Python 2 and achieving concurrency would involve a bit of programming (or adding package dependencies). I may very well port run-task to Python 3 and then fold fetch-content into it. Or maybe we leave `fetch-content` as a standalone script. MozReview-Commit-ID: AGuTcwNcNJR --HG-- extra : source : 0b941cbdca76fb2fbb98dc5bbc1a0237c69954d0 extra : histedit_source : a3e43bdd8a9a58550bef02fec3be832ca304ea93
This commit is contained in:
Gregory Szorc
Родитель
cf83defe06
Коммит
8922082362
|
|
@ -7,89 +7,6 @@ make_flags="-j$(nproc)"
|
|||
|
||||
. $data_dir/download-tools.sh
|
||||
|
||||
prepare() {
|
||||
pushd $root_dir
|
||||
download_and_check https://ftp.gnu.org/gnu/binutils binutils-$binutils_version.tar.$binutils_ext.sig
|
||||
tar xaf $TMPDIR/binutils-$binutils_version.tar.$binutils_ext
|
||||
|
||||
case "$gcc_version" in
|
||||
*-*)
|
||||
download https://gcc.gnu.org/pub/gcc/snapshots/$gcc_version/gcc-$gcc_version.tar.$gcc_ext
|
||||
;;
|
||||
*)
|
||||
download_and_check https://ftp.gnu.org/gnu/gcc/gcc-$gcc_version gcc-$gcc_version.tar.$gcc_ext.sig
|
||||
;;
|
||||
esac
|
||||
tar xaf $TMPDIR/gcc-$gcc_version.tar.$gcc_ext
|
||||
cd gcc-$gcc_version
|
||||
|
||||
(
|
||||
# Divert commands that download_prerequisites use
|
||||
ln() { :; }
|
||||
tar() { :; }
|
||||
sed() { :; }
|
||||
wget() {
|
||||
# Get last argument given to wget.
|
||||
eval echo \$$# >&3
|
||||
}
|
||||
|
||||
# In GCC >= 7, the download_prerequisites script tried to do its own
|
||||
# verification, but we have ours, so disable it.
|
||||
set -- --no-verify
|
||||
. ./contrib/download_prerequisites
|
||||
) 3>&1 > /dev/null | while read url; do
|
||||
file=$(basename $url)
|
||||
case "$file" in
|
||||
gmp-*.tar.*)
|
||||
# If download_prerequisites wants 4.3.2, use 5.1.3 instead.
|
||||
file=${file/4.3.2/5.1.3}
|
||||
download_and_check https://ftp.gnu.org/gnu/gmp/ $file.sig
|
||||
;;
|
||||
mpfr-*.tar.*)
|
||||
# If download_prerequisites wants 2.4.2, use 3.1.5 instead.
|
||||
file=${file/2.4.2/3.1.5}
|
||||
download_and_check https://ftp.gnu.org/gnu/mpfr/ $file.sig
|
||||
;;
|
||||
mpc-*.tar.*)
|
||||
# If download_prerequisites wants 0.8.1, use 0.8.2 instead.
|
||||
file=${file/0.8.1/0.8.2}
|
||||
case "$file" in
|
||||
*-0.8.2.tar*|*-0.9.tar*|*-1.0.tar*)
|
||||
ext=asc
|
||||
;;
|
||||
*)
|
||||
ext=sig
|
||||
;;
|
||||
esac
|
||||
case "$file" in
|
||||
*-0.8.2.tar*)
|
||||
# The ftp.gnu.org mirror doesn't have 0.8.2, so we use
|
||||
# a debian source tarball instead. It has a different name,
|
||||
# so we can't use `download`. Manually do what it does, but
|
||||
# handling the difference in file name.
|
||||
wget -c --progress=dot:mega -O $TMPDIR/$file http://snapshot.debian.org/archive/debian/20100527T162226Z/pool/main/m/mpclib/mpclib_0.8.2.orig.tar.gz
|
||||
(cd $TMPDIR; sha256sum $file) >> $root_dir/downloads
|
||||
;;
|
||||
*)
|
||||
download_and_check https://ftp.gnu.org/gnu/mpc $file.$ext
|
||||
;;
|
||||
esac
|
||||
;;
|
||||
*)
|
||||
download $(dirname $url) $file
|
||||
;;
|
||||
esac
|
||||
tar xaf $TMPDIR/$file
|
||||
ln -sf ${file%.tar.*} ${file%-*}
|
||||
done
|
||||
|
||||
# Check all the downloads we did are in the checksums list, and that the
|
||||
# checksums match.
|
||||
diff -u <(sort -k 2 $root_dir/downloads) $root_dir/checksums
|
||||
|
||||
popd
|
||||
}
|
||||
|
||||
prepare_mingw() {
|
||||
export prefix=/tools/mingw32
|
||||
export install_dir=$root_dir$prefix
|
||||
|
|
|
|||
|
|
@ -83,6 +83,9 @@ jobs:
|
|||
android-build:
|
||||
symbol: I(agb)
|
||||
parent: debian9-base
|
||||
fetch:
|
||||
symbol: I(fetch)
|
||||
parent: debian9-base
|
||||
mingw32-build:
|
||||
symbol: I(mingw)
|
||||
parent: debian9-base
|
||||
|
|
|
|||
|
|
@ -0,0 +1,24 @@
|
|||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
loader: taskgraph.loader.transform:loader
|
||||
|
||||
transforms:
|
||||
- taskgraph.transforms.try_job:transforms
|
||||
- taskgraph.transforms.job:transforms
|
||||
- taskgraph.transforms.task:transforms
|
||||
|
||||
job-defaults:
|
||||
run-on-projects: []
|
||||
treeherder:
|
||||
kind: build
|
||||
platform: fetch/opt
|
||||
tier: 1
|
||||
worker-type: aws-provisioner-v1/gecko-{level}-images
|
||||
worker:
|
||||
docker-image: {in-tree: fetch}
|
||||
max-run-time: 900
|
||||
|
||||
jobs-from:
|
||||
- toolchains.yml
|
||||
|
|
@ -0,0 +1,195 @@
|
|||
binutils-2.25.1:
|
||||
description: binutils 2.25.1 source code
|
||||
treeherder:
|
||||
symbol: binutils2.25.1
|
||||
run:
|
||||
using: fetch-url
|
||||
url: ftp://ftp.gnu.org/gnu/binutils/binutils-2.25.1.tar.bz2
|
||||
sha256: b5b14added7d78a8d1ca70b5cb75fef57ce2197264f4f5835326b0df22ac9f22
|
||||
size: 24163561
|
||||
gpg-signature:
|
||||
sig-url: "{url}.sig"
|
||||
key-path: build/unix/build-gcc/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
|
||||
|
||||
binutils-2.27:
|
||||
description: binutils 2.27 source code
|
||||
treeherder:
|
||||
symbol: binutils2.27
|
||||
run:
|
||||
using: fetch-url
|
||||
url: ftp://ftp.gnu.org/gnu/binutils/binutils-2.27.tar.bz2
|
||||
sha256: 369737ce51587f92466041a97ab7d2358c6d9e1b6490b3940eb09fb0a9a6ac88
|
||||
size: 26099568
|
||||
gpg-signature:
|
||||
sig-url: "{url}.sig"
|
||||
key-path: build/unix/build-gcc/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
|
||||
|
||||
binutils-2.28.1:
|
||||
description: binutils 2.28.1 source code
|
||||
treeherder:
|
||||
symbol: binutils2.28.1
|
||||
run:
|
||||
using: fetch-url
|
||||
url: ftp://ftp.gnu.org/gnu/binutils/binutils-2.28.1.tar.xz
|
||||
sha256: 16328a906e55a3c633854beec8e9e255a639b366436470b4f6245eb0d2fde942
|
||||
size: 19440112
|
||||
gpg-signature:
|
||||
sig-url: "{url}.sig"
|
||||
key-path: build/unix/build-gcc/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
|
||||
|
||||
cloog-0.18.1:
|
||||
description: cloog source code
|
||||
treeherder:
|
||||
symbol: cloog0.18.1
|
||||
run:
|
||||
using: fetch-url
|
||||
url: ftp://gcc.gnu.org/pub/gcc/infrastructure/cloog-0.18.1.tar.gz
|
||||
sha256: 02500a4edd14875f94fe84cbeda4290425cb0c1c2474c6f75d75a303d64b4196
|
||||
size: 3857324
|
||||
|
||||
gcc-4.9.4:
|
||||
description: GCC 4.9.4 source code
|
||||
treeherder:
|
||||
symbol: gcc4.9.4
|
||||
run:
|
||||
using: fetch-url
|
||||
url: ftp://ftp.gnu.org/gnu/gcc/gcc-4.9.4/gcc-4.9.4.tar.bz2
|
||||
sha256: 6c11d292cd01b294f9f84c9a59c230d80e9e4a47e5c6355f046bb36d4f358092
|
||||
size: 90097606
|
||||
gpg-signature:
|
||||
sig-url: "{url}.sig"
|
||||
key-path: build/unix/build-gcc/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
|
||||
|
||||
gcc-6.4.0:
|
||||
description: GCC 6.4.0 source code
|
||||
treeherder:
|
||||
symbol: gcc6.4.0
|
||||
run:
|
||||
using: fetch-url
|
||||
url: ftp://ftp.gnu.org/gnu/gcc/gcc-6.4.0/gcc-6.4.0.tar.xz
|
||||
sha256: 850bf21eafdfe5cd5f6827148184c08c4a0852a37ccf36ce69855334d2c914d4
|
||||
size: 76156220
|
||||
gpg-signature:
|
||||
sig-url: "{url}.sig"
|
||||
key-path: build/unix/build-gcc/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
|
||||
|
||||
gcc-7.3.0:
|
||||
description: GCC 7.3.0 source code
|
||||
treeherder:
|
||||
symbol: gcc7.3.0
|
||||
run:
|
||||
using: fetch-url
|
||||
url: ftp://ftp.gnu.org/gnu/gcc/gcc-7.3.0/gcc-7.3.0.tar.xz
|
||||
sha256: 832ca6ae04636adbb430e865a1451adf6979ab44ca1c8374f61fba65645ce15c
|
||||
size: 62462388
|
||||
gpg-signature:
|
||||
sig-url: "{url}.sig"
|
||||
key-path: build/unix/build-gcc/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
|
||||
|
||||
gmp-5.1.3:
|
||||
description: GMP 5.1.3 source code
|
||||
treeherder:
|
||||
symbol: gmp5.1.3
|
||||
run:
|
||||
using: fetch-url
|
||||
url: https://ftp.gnu.org/gnu/gmp/gmp-5.1.3.tar.bz2
|
||||
sha256: 752079520b4690531171d0f4532e40f08600215feefede70b24fabdc6f1ab160
|
||||
size: 2196480
|
||||
gpg-signature:
|
||||
sig-url: "{url}.sig"
|
||||
key-path: build/unix/build-gcc/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
|
||||
|
||||
gmp-6.1.0:
|
||||
description: GMP 6.1.0 source code
|
||||
treeherder:
|
||||
symbol: gmp6.1.0
|
||||
run:
|
||||
using: fetch-url
|
||||
url: https://ftp.gnu.org/gnu/gmp/gmp-6.1.0.tar.bz2
|
||||
sha256: 498449a994efeba527885c10405993427995d3f86b8768d8cdf8d9dd7c6b73e8
|
||||
size: 2383840
|
||||
gpg-signature:
|
||||
sig-url: "{url}.sig"
|
||||
key-path: build/unix/build-gcc/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
|
||||
|
||||
isl-0.12.2:
|
||||
description: ISL 0.12.2 source code
|
||||
treeherder:
|
||||
symbol: isl0.12.2
|
||||
run:
|
||||
using: fetch-url
|
||||
url: ftp://gcc.gnu.org/pub/gcc/infrastructure/isl-0.12.2.tar.bz2
|
||||
sha256: f4b3dbee9712850006e44f0db2103441ab3d13b406f77996d1df19ee89d11fb4
|
||||
size: 1319434
|
||||
|
||||
isl-0.15:
|
||||
description: ISL 0.15 source code
|
||||
treeherder:
|
||||
symbol: isl0.15
|
||||
run:
|
||||
using: fetch-url
|
||||
url: ftp://gcc.gnu.org/pub/gcc/infrastructure/isl-0.15.tar.bz2
|
||||
sha256: 8ceebbf4d9a81afa2b4449113cee4b7cb14a687d7a549a963deb5e2a41458b6b
|
||||
size: 1574964
|
||||
|
||||
isl-0.16.1:
|
||||
description: ISL 0.16.1 source code
|
||||
treeherder:
|
||||
symbol: isl0.16.1
|
||||
run:
|
||||
using: fetch-url
|
||||
url: ftp://gcc.gnu.org/pub/gcc/infrastructure/isl-0.16.1.tar.bz2
|
||||
sha256: 412538bb65c799ac98e17e8cfcdacbb257a57362acfaaff254b0fcae970126d2
|
||||
size: 1626446
|
||||
|
||||
mpc-0.8.2:
|
||||
description: mpc 0.8.2 source code
|
||||
treeherder:
|
||||
symbol: mpc0.8.2
|
||||
run:
|
||||
using: fetch-url
|
||||
url: http://www.multiprecision.org/downloads/mpc-0.8.2.tar.gz
|
||||
sha256: ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7
|
||||
size: 548401
|
||||
gpg-signature:
|
||||
sig-url: "{url}.asc"
|
||||
key-path: build/unix/build-gcc/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
|
||||
|
||||
mpc-1.0.3:
|
||||
description: mpc 1.0.3 source code
|
||||
treeherder:
|
||||
symbol: mpc1.0.3
|
||||
run:
|
||||
using: fetch-url
|
||||
url: http://www.multiprecision.org/downloads/mpc-1.0.3.tar.gz
|
||||
sha256: 617decc6ea09889fb08ede330917a00b16809b8db88c29c31bfbb49cbf88ecc3
|
||||
size: 669925
|
||||
gpg-signature:
|
||||
sig-url: "{url}.sig"
|
||||
key-path: build/unix/build-gcc/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
|
||||
|
||||
mpfr-3.1.4:
|
||||
description: mpfr 3.1.4 source code
|
||||
treeherder:
|
||||
symbol: mpfr3.1.4
|
||||
run:
|
||||
using: fetch-url
|
||||
url: http://www.mpfr.org/mpfr-3.1.4/mpfr-3.1.4.tar.bz2
|
||||
sha256: d3103a80cdad2407ed581f3618c4bed04e0c92d1cf771a65ead662cc397f7775
|
||||
size: 1279284
|
||||
gpg-signature:
|
||||
sig-url: "{url}.asc"
|
||||
key-path: build/unix/build-gcc/07F3DBBECC1A39605078094D980C197698C3739D.key
|
||||
|
||||
mpfr-3.1.5:
|
||||
description: mpfr 3.1.5 source code
|
||||
treeherder:
|
||||
symbol: mpfr3.1.5
|
||||
run:
|
||||
using: fetch-url
|
||||
url: http://www.mpfr.org/mpfr-3.1.5/mpfr-3.1.5.tar.bz2
|
||||
sha256: ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658
|
||||
size: 1279489
|
||||
gpg-signature:
|
||||
sig-url: "{url}.asc"
|
||||
key-path: build/unix/build-gcc/07F3DBBECC1A39605078094D980C197698C3739D.key
|
||||
|
|
@ -4,8 +4,12 @@
|
|||
|
||||
loader: taskgraph.loader.transform:loader
|
||||
|
||||
kind-dependencies:
|
||||
- fetch
|
||||
|
||||
transforms:
|
||||
- taskgraph.transforms.try_job:transforms
|
||||
- taskgraph.transforms.use_fetches:transforms
|
||||
- taskgraph.transforms.use_toolchains:transforms
|
||||
- taskgraph.transforms.job:transforms
|
||||
- taskgraph.transforms.task:transforms
|
||||
|
|
|
|||
|
|
@ -161,6 +161,14 @@ linux64-gcc-4.9:
|
|||
resources:
|
||||
- 'build/unix/build-gcc/build-gcc.sh'
|
||||
toolchain-artifact: public/build/gcc.tar.xz
|
||||
fetches:
|
||||
- binutils-2.25.1
|
||||
- cloog-0.18.1
|
||||
- gcc-4.9.4
|
||||
- gmp-5.1.3
|
||||
- isl-0.12.2
|
||||
- mpc-0.8.2
|
||||
- mpfr-3.1.5
|
||||
|
||||
linux64-gcc-6:
|
||||
description: "GCC 6 toolchain build"
|
||||
|
|
@ -179,6 +187,13 @@ linux64-gcc-6:
|
|||
- 'build/unix/build-gcc/build-gcc.sh'
|
||||
toolchain-alias: linux64-gcc
|
||||
toolchain-artifact: public/build/gcc.tar.xz
|
||||
fetches:
|
||||
- binutils-2.28.1
|
||||
- gcc-6.4.0
|
||||
- gmp-5.1.3
|
||||
- isl-0.15
|
||||
- mpc-0.8.2
|
||||
- mpfr-3.1.5
|
||||
|
||||
linux64-gcc-7:
|
||||
description: "GCC 7 toolchain build"
|
||||
|
|
@ -196,6 +211,13 @@ linux64-gcc-7:
|
|||
resources:
|
||||
- 'build/unix/build-gcc/build-gcc.sh'
|
||||
toolchain-artifact: public/build/gcc.tar.xz
|
||||
fetches:
|
||||
- binutils-2.28.1
|
||||
- gcc-7.3.0
|
||||
- gmp-6.1.0
|
||||
- isl-0.16.1
|
||||
- mpc-1.0.3
|
||||
- mpfr-3.1.4
|
||||
|
||||
linux64-gcc-sixgill:
|
||||
description: "sixgill GCC plugin build"
|
||||
|
|
@ -216,6 +238,13 @@ linux64-gcc-sixgill:
|
|||
toolchain-artifact: public/build/sixgill.tar.xz
|
||||
toolchains:
|
||||
- linux64-gcc-6
|
||||
fetches:
|
||||
- binutils-2.28.1
|
||||
- isl-0.15
|
||||
- gcc-6.4.0
|
||||
- gmp-5.1.3
|
||||
- mpc-0.8.2
|
||||
- mpfr-3.1.5
|
||||
|
||||
linux64-llvm-dsymutil:
|
||||
description: "llvm-dsymutil toolchain build"
|
||||
|
|
@ -570,6 +599,13 @@ linux64-mingw32-gcc:
|
|||
resources:
|
||||
- 'build/unix/build-gcc/build-gcc.sh'
|
||||
toolchain-artifact: public/build/mingw32.tar.xz
|
||||
fetches:
|
||||
- binutils-2.27
|
||||
- gcc-6.4.0
|
||||
- gmp-5.1.3
|
||||
- isl-0.15
|
||||
- mpc-0.8.2
|
||||
- mpfr-3.1.5
|
||||
|
||||
linux64-mingw32-nsis:
|
||||
description: "NSIS build for MinGW32 Cross Compile"
|
||||
|
|
|
|||
|
|
@ -0,0 +1,9 @@
|
|||
# %ARG DOCKER_IMAGE_PARENT
|
||||
FROM $DOCKER_IMAGE_PARENT
|
||||
|
||||
RUN apt-get update && \
|
||||
apt-get install \
|
||||
gnupg
|
||||
|
||||
# %include taskcluster/scripts/misc/fetch-content
|
||||
ADD topsrcdir/taskcluster/scripts/misc/fetch-content /builds/worker/bin/fetch-content
|
||||
|
|
@ -176,6 +176,11 @@ repackage_type
|
|||
This is the type of repackage. Can be ``repackage`` or
|
||||
``repackage_signing``.
|
||||
|
||||
fetch-artifact
|
||||
==============
|
||||
|
||||
For fetch jobs, this is the path to the artifact for that fetch operation.
|
||||
|
||||
toolchain-artifact
|
||||
==================
|
||||
For toolchain jobs, this is the path to the artifact for that toolchain.
|
||||
|
|
|
|||
|
|
@ -461,6 +461,13 @@ post-langpack-dummy
|
|||
------------------------------
|
||||
Dummy tasks to consolidate language pack beetmover dependencies to avoid taskcluster limits on number of dependencies per task.
|
||||
|
||||
fetch
|
||||
-----
|
||||
|
||||
Tasks that obtain something from a remote service and re-expose it as a
|
||||
task artifact. These tasks are used to effectively cache and re-host
|
||||
remote content so it is reliably and deterministically available.
|
||||
|
||||
packages
|
||||
--------
|
||||
Tasks used to build packages for use in docker images.
|
||||
|
|
|
|||
|
|
@ -146,6 +146,7 @@ following ``run-using`` are available
|
|||
* ``debian-package``
|
||||
* ``toolchain-script``
|
||||
* ``always-optimized``
|
||||
* ``fetch-url``
|
||||
|
||||
|
||||
Task Descriptions
|
||||
|
|
|
|||
|
|
@ -17,28 +17,16 @@ gcc_ext=bz2
|
|||
binutils_version=2.25.1
|
||||
binutils_ext=bz2
|
||||
|
||||
# GPG key used to sign GCC
|
||||
$GPG --import $data_dir/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
|
||||
# GPG key used to sign binutils
|
||||
$GPG --import $data_dir/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
|
||||
# GPG key used to sign GMP
|
||||
$GPG --import $data_dir/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
|
||||
# GPG key used to sign MPFR
|
||||
$GPG --import $data_dir/07F3DBBECC1A39605078094D980C197698C3739D.key
|
||||
# GPG key used to sign MPC
|
||||
$GPG --import $data_dir/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
|
||||
$HOME_DIR/src/taskcluster/scripts/misc/fetch-content task-artifacts -d $root_dir $MOZ_FETCHES
|
||||
|
||||
cat > $HOME_DIR/checksums <<EOF
|
||||
b5b14added7d78a8d1ca70b5cb75fef57ce2197264f4f5835326b0df22ac9f22 binutils-2.25.1.tar.bz2
|
||||
02500a4edd14875f94fe84cbeda4290425cb0c1c2474c6f75d75a303d64b4196 cloog-0.18.1.tar.gz
|
||||
6c11d292cd01b294f9f84c9a59c230d80e9e4a47e5c6355f046bb36d4f358092 gcc-4.9.4.tar.bz2
|
||||
752079520b4690531171d0f4532e40f08600215feefede70b24fabdc6f1ab160 gmp-5.1.3.tar.bz2
|
||||
f4b3dbee9712850006e44f0db2103441ab3d13b406f77996d1df19ee89d11fb4 isl-0.12.2.tar.bz2
|
||||
ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7 mpc-0.8.2.tar.gz
|
||||
ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658 mpfr-3.1.5.tar.bz2
|
||||
EOF
|
||||
pushd $root_dir/gcc-$gcc_version
|
||||
ln -sf ../cloog-0.18.1 cloog
|
||||
ln -sf ../gmp-5.1.3 gmp
|
||||
ln -sf ../mpc-0.8.2 mpc
|
||||
ln -sf ../isl-0.12.2 isl
|
||||
ln -sf ../mpfr-3.1.5 mpfr
|
||||
popd
|
||||
|
||||
prepare
|
||||
apply_patch $data_dir/PR64905.patch
|
||||
build_binutils
|
||||
build_gcc
|
||||
|
|
|
|||
|
|
@ -17,27 +17,15 @@ gcc_ext=xz
|
|||
binutils_version=2.28.1
|
||||
binutils_ext=xz
|
||||
|
||||
# GPG key used to sign GCC
|
||||
$GPG --import $data_dir/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
|
||||
# GPG key used to sign binutils
|
||||
$GPG --import $data_dir/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
|
||||
# GPG key used to sign GMP
|
||||
$GPG --import $data_dir/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
|
||||
# GPG key used to sign MPFR
|
||||
$GPG --import $data_dir/07F3DBBECC1A39605078094D980C197698C3739D.key
|
||||
# GPG key used to sign MPC
|
||||
$GPG --import $data_dir/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
|
||||
$HOME_DIR/src/taskcluster/scripts/misc/fetch-content task-artifacts -d $root_dir $MOZ_FETCHES
|
||||
|
||||
cat > $HOME_DIR/checksums <<EOF
|
||||
16328a906e55a3c633854beec8e9e255a639b366436470b4f6245eb0d2fde942 binutils-2.28.1.tar.xz
|
||||
850bf21eafdfe5cd5f6827148184c08c4a0852a37ccf36ce69855334d2c914d4 gcc-6.4.0.tar.xz
|
||||
752079520b4690531171d0f4532e40f08600215feefede70b24fabdc6f1ab160 gmp-5.1.3.tar.bz2
|
||||
8ceebbf4d9a81afa2b4449113cee4b7cb14a687d7a549a963deb5e2a41458b6b isl-0.15.tar.bz2
|
||||
ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7 mpc-0.8.2.tar.gz
|
||||
ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658 mpfr-3.1.5.tar.bz2
|
||||
EOF
|
||||
pushd $root_dir/gcc-$gcc_version
|
||||
ln -sf ../gmp-5.1.3 gmp
|
||||
ln -sf ../isl-0.15 isl
|
||||
ln -sf ../mpc-0.8.2 mpc
|
||||
ln -sf ../mpfr-3.1.5 mpfr
|
||||
popd
|
||||
|
||||
prepare
|
||||
build_binutils
|
||||
build_gcc
|
||||
|
||||
|
|
|
|||
|
|
@ -17,27 +17,15 @@ gcc_ext=xz
|
|||
binutils_version=2.28.1
|
||||
binutils_ext=xz
|
||||
|
||||
# GPG key used to sign GCC
|
||||
$GPG --import $data_dir/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
|
||||
# GPG key used to sign binutils
|
||||
$GPG --import $data_dir/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
|
||||
# GPG key used to sign GMP
|
||||
$GPG --import $data_dir/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
|
||||
# GPG key used to sign MPFR
|
||||
$GPG --import $data_dir/07F3DBBECC1A39605078094D980C197698C3739D.key
|
||||
# GPG key used to sign MPC
|
||||
$GPG --import $data_dir/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
|
||||
$HOME_DIR/src/taskcluster/scripts/misc/fetch-content task-artifacts -d $root_dir $MOZ_FETCHES
|
||||
|
||||
cat > $HOME_DIR/checksums <<EOF
|
||||
16328a906e55a3c633854beec8e9e255a639b366436470b4f6245eb0d2fde942 binutils-2.28.1.tar.xz
|
||||
832ca6ae04636adbb430e865a1451adf6979ab44ca1c8374f61fba65645ce15c gcc-7.3.0.tar.xz
|
||||
498449a994efeba527885c10405993427995d3f86b8768d8cdf8d9dd7c6b73e8 gmp-6.1.0.tar.bz2
|
||||
412538bb65c799ac98e17e8cfcdacbb257a57362acfaaff254b0fcae970126d2 isl-0.16.1.tar.bz2
|
||||
617decc6ea09889fb08ede330917a00b16809b8db88c29c31bfbb49cbf88ecc3 mpc-1.0.3.tar.gz
|
||||
d3103a80cdad2407ed581f3618c4bed04e0c92d1cf771a65ead662cc397f7775 mpfr-3.1.4.tar.bz2
|
||||
EOF
|
||||
pushd $root_dir/gcc-$gcc_version
|
||||
ln -sf ../gmp-6.1.0 gmp
|
||||
ln -sf ../isl-0.16.1 isl
|
||||
ln -sf ../mpc-1.0.3 mpc
|
||||
ln -sf ../mpfr-3.1.4 mpfr
|
||||
popd
|
||||
|
||||
prepare
|
||||
build_binutils
|
||||
build_gcc
|
||||
|
||||
|
|
|
|||
|
|
@ -20,28 +20,15 @@ binutils_ext=bz2
|
|||
binutils_configure_flags="--target=i686-w64-mingw32"
|
||||
mingw_version=bcf1f29d6dc80b6025b416bef104d2314fa9be57
|
||||
|
||||
# GPG keys used to sign GCC (collected from 5.1.0, 5.4.0, 6.4.0)
|
||||
$GPG --import $data_dir/33C235A34C46AA3FFB293709A328C3A2C3C45C06.key
|
||||
$GPG --import $data_dir/7F74F97C103468EE5D750B583AB00996FC26A641.key
|
||||
# GPG key used to sign binutils
|
||||
$GPG --import $data_dir/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
|
||||
# GPG key used to sign GMP
|
||||
$GPG --import $data_dir/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
|
||||
# GPG key used to sign MPFR
|
||||
$GPG --import $data_dir/07F3DBBECC1A39605078094D980C197698C3739D.key
|
||||
# GPG key used to sign MPC
|
||||
$GPG --import $data_dir/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
|
||||
$HOME_DIR/src/taskcluster/scripts/misc/fetch-content task-artifacts -d $root_dir $MOZ_FETCHES
|
||||
|
||||
cat > $root_dir/checksums <<EOF
|
||||
369737ce51587f92466041a97ab7d2358c6d9e1b6490b3940eb09fb0a9a6ac88 binutils-2.27.tar.bz2
|
||||
850bf21eafdfe5cd5f6827148184c08c4a0852a37ccf36ce69855334d2c914d4 gcc-6.4.0.tar.xz
|
||||
752079520b4690531171d0f4532e40f08600215feefede70b24fabdc6f1ab160 gmp-5.1.3.tar.bz2
|
||||
8ceebbf4d9a81afa2b4449113cee4b7cb14a687d7a549a963deb5e2a41458b6b isl-0.15.tar.bz2
|
||||
ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7 mpc-0.8.2.tar.gz
|
||||
ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658 mpfr-3.1.5.tar.bz2
|
||||
EOF
|
||||
pushd $root_dir/gcc-$gcc_version
|
||||
ln -sf ../gmp-5.1.3 gmp
|
||||
ln -sf ../isl-0.15 isl
|
||||
ln -sf ../mpc-0.8.2 mpc
|
||||
ln -sf ../mpfr-3.1.5 mpfr
|
||||
popd
|
||||
|
||||
prepare
|
||||
prepare_mingw
|
||||
build_binutils
|
||||
build_gcc_and_mingw
|
||||
|
|
|
|||
|
|
@ -27,28 +27,15 @@ sixgill_repo=https://hg.mozilla.org/users/sfink_mozilla.com/sixgill
|
|||
|
||||
. $data_dir/build-gcc.sh
|
||||
|
||||
# GPG key used to sign GCC
|
||||
$GPG --import $data_dir/13975A70E63C361C73AE69EF6EEB81F8981C74C7.key
|
||||
# GPG key used to sign binutils
|
||||
$GPG --import $data_dir/EAF1C276A747E9ED86210CBAC3126D3B4AE55E93.key
|
||||
# GPG key used to sign GMP
|
||||
$GPG --import $data_dir/343C2FF0FBEE5EC2EDBEF399F3599FF828C67298.key
|
||||
# GPG key used to sign MPFR
|
||||
$GPG --import $data_dir/07F3DBBECC1A39605078094D980C197698C3739D.key
|
||||
# GPG key used to sign MPC
|
||||
$GPG --import $data_dir/AD17A21EF8AED8F1CC02DBD9F7D5C9BF765C61E3.key
|
||||
$HOME_DIR/src/taskcluster/scripts/misc/fetch-content task-artifacts -d $root_dir $MOZ_FETCHES
|
||||
|
||||
cat > $HOME_DIR/checksums <<EOF
|
||||
16328a906e55a3c633854beec8e9e255a639b366436470b4f6245eb0d2fde942 binutils-2.28.1.tar.xz
|
||||
850bf21eafdfe5cd5f6827148184c08c4a0852a37ccf36ce69855334d2c914d4 gcc-6.4.0.tar.xz
|
||||
752079520b4690531171d0f4532e40f08600215feefede70b24fabdc6f1ab160 gmp-5.1.3.tar.bz2
|
||||
8ceebbf4d9a81afa2b4449113cee4b7cb14a687d7a549a963deb5e2a41458b6b isl-0.15.tar.bz2
|
||||
ae79f8d41d8a86456b68607e9ca398d00f8b7342d1d83bcf4428178ac45380c7 mpc-0.8.2.tar.gz
|
||||
ca498c1c7a74dd37a576f353312d1e68d490978de4395fa28f1cbd46a364e658 mpfr-3.1.5.tar.bz2
|
||||
EOF
|
||||
|
||||
# Download GCC + related, and unpack.
|
||||
prepare
|
||||
pushd $root_dir/gcc-$gcc_version
|
||||
ln -sf ../binutils-2.28.1 binutils
|
||||
ln -sf ../gmp-5.1.3 gmp
|
||||
ln -sf ../isl-0.15 isl
|
||||
ln -sf ../mpc-0.8.2 mpc
|
||||
ln -sf ../mpfr-3.1.5 mpfr
|
||||
popd
|
||||
|
||||
export TMPDIR=${TMPDIR:-/tmp/}
|
||||
export gcc_bindir=$root_dir/src/gcc/bin
|
||||
|
|
|
|||
|
|
@ -0,0 +1,268 @@
|
|||
#!/usr/bin/python3 -u
|
||||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
import argparse
|
||||
import concurrent.futures
|
||||
import hashlib
|
||||
import multiprocessing
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
|
||||
ARTIFACT_URL = ('https://queue.taskcluster.net/v1/task/{task}/artifacts/'
|
||||
'{artifact}')
|
||||
|
||||
CONCURRENCY = multiprocessing.cpu_count()
|
||||
|
||||
|
||||
class IntegrityError(Exception):
|
||||
"""Represents an integrity error when downloading a URL."""
|
||||
|
||||
|
||||
def stream_download(url, sha256=None, size=None):
|
||||
"""Download a URL to a generator, optionally with content verification.
|
||||
|
||||
If ``sha256`` or ``size`` are defined, the downloaded URL will be
|
||||
validated against those requirements and ``IntegrityError`` will be
|
||||
raised if expectations do not match.
|
||||
|
||||
Because verification cannot occur until the file is completely downloaded
|
||||
it is recommended for consumers to not do anything meaningful with the
|
||||
data if content verification is being used. To securely handle retrieved
|
||||
content, it should be streamed to a file or memory and only operated
|
||||
on after the generator is exhausted without raising.
|
||||
"""
|
||||
print('downloading %s' % url)
|
||||
|
||||
h = hashlib.sha256()
|
||||
length = 0
|
||||
|
||||
t0 = time.time()
|
||||
with urllib.request.urlopen(url) as fh:
|
||||
while True:
|
||||
chunk = fh.read(65536)
|
||||
if not chunk:
|
||||
break
|
||||
|
||||
h.update(chunk)
|
||||
length += len(chunk)
|
||||
|
||||
yield chunk
|
||||
|
||||
duration = time.time() - t0
|
||||
digest = h.hexdigest()
|
||||
|
||||
print('%s resolved to %d bytes with sha256 %s in %.3fs' % (
|
||||
url, length, digest, duration))
|
||||
|
||||
if size:
|
||||
if size == length:
|
||||
print('verified size of %s' % url)
|
||||
else:
|
||||
raise IntegrityError('size mismatch on %s: wanted %d; got %d' % (
|
||||
url, size, length))
|
||||
|
||||
if sha256:
|
||||
if digest == sha256:
|
||||
print('verified sha256 integrity of %s' % url)
|
||||
else:
|
||||
raise IntegrityError('sha256 mismatch on %s: wanted %s; got %s' % (
|
||||
url, sha256, digest))
|
||||
|
||||
|
||||
def download_to_path(url, path, sha256=None, size=None):
|
||||
"""Download a URL to a filesystem path, possibly with verification."""
|
||||
|
||||
# We download to a temporary file and rename at the end so there's
|
||||
# no chance of the final file being partially written or containing
|
||||
# bad data.
|
||||
try:
|
||||
path.unlink()
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
tmp = path.with_name('%s.tmp' % path.name)
|
||||
|
||||
print('downloading %s to %s' % (url, tmp))
|
||||
|
||||
try:
|
||||
with tmp.open('wb') as fh:
|
||||
for chunk in stream_download(url, sha256=sha256, size=size):
|
||||
fh.write(chunk)
|
||||
|
||||
print('renaming to %s' % path)
|
||||
tmp.rename(path)
|
||||
except IntegrityError:
|
||||
tmp.unlink()
|
||||
raise
|
||||
|
||||
|
||||
def gpg_verify_path(path: pathlib.Path, public_key_data: bytes,
|
||||
signature_data: bytes):
|
||||
"""Verify that a filesystem path verifies using GPG.
|
||||
|
||||
Takes a Path defining a file to verify. ``public_key_data`` contains
|
||||
bytes with GPG public key data. ``signature_data`` contains a signed
|
||||
GPG document to use with ``gpg --verify``.
|
||||
"""
|
||||
print('validating GPG signature of %s' % path)
|
||||
print('GPG key data:\n%s' % public_key_data.decode('ascii'))
|
||||
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
try:
|
||||
# --batch since we're running unattended.
|
||||
gpg_args = ['gpg', '--homedir', td, '--batch']
|
||||
|
||||
print('importing GPG key...')
|
||||
subprocess.run(gpg_args + ['--import'],
|
||||
input=public_key_data,
|
||||
check=True)
|
||||
|
||||
print('verifying GPG signature...')
|
||||
subprocess.run(gpg_args + ['--verify', '-', '%s' % path],
|
||||
input=signature_data,
|
||||
check=True)
|
||||
|
||||
print('GPG signature verified!')
|
||||
finally:
|
||||
# There is a race between the agent self-terminating and
|
||||
# shutil.rmtree() from the temporary directory cleanup that can
|
||||
# lead to exceptions. Kill the agent before cleanup to prevent this.
|
||||
env = dict(os.environ)
|
||||
env['GNUPGHOME'] = td
|
||||
subprocess.run(['gpgconf', '--kill', 'gpg-agent'], env=env)
|
||||
|
||||
|
||||
def extract_archive(path, dest_dir):
|
||||
"""Extract an archive to a destination directory."""
|
||||
|
||||
if re.search('\.tar\..*$', path.name):
|
||||
args = ['tar', 'xaf', str(path)]
|
||||
elif path.name.endswith('.zip'):
|
||||
args = ['unzip', str(path)]
|
||||
else:
|
||||
print('%s is unknown archive format; ignoring' % path)
|
||||
return False
|
||||
|
||||
print('extracting %s to %s using %r' % (path, dest_dir, args))
|
||||
t0 = time.time()
|
||||
subprocess.check_call(args, cwd=str(dest_dir), bufsize=0)
|
||||
print('%s extracted in %.3fs' % (path, time.time() - t0))
|
||||
return True
|
||||
|
||||
|
||||
def fetch_and_extract(url, dest_dir, sha256=None, size=None):
|
||||
"""Fetch a URL and extract it to a destination path.
|
||||
|
||||
If the downloaded URL is an archive, it is extracted automatically
|
||||
and the archive is deleted. Otherwise the file remains in place in
|
||||
the destination directory.
|
||||
"""
|
||||
|
||||
basename = url.split('/')[-1]
|
||||
dest_path = dest_dir / basename
|
||||
|
||||
download_to_path(url, dest_path, sha256=sha256, size=size)
|
||||
|
||||
if extract_archive(dest_path, dest_dir):
|
||||
print('removing %s' % dest_path)
|
||||
dest_path.unlink()
|
||||
|
||||
|
||||
def fetch_urls(urls, dest):
|
||||
"""Fetch URLs pairs to a pathlib.Path."""
|
||||
dest.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
with concurrent.futures.ThreadPoolExecutor(CONCURRENCY) as e:
|
||||
fs = []
|
||||
|
||||
for url in urls:
|
||||
fs.append(e.submit(fetch_and_extract, url, dest))
|
||||
|
||||
for f in fs:
|
||||
f.result()
|
||||
|
||||
|
||||
def command_static_url(args):
|
||||
gpg_sig_url = args.gpg_sig_url
|
||||
gpg_env_key = args.gpg_key_env
|
||||
|
||||
if bool(gpg_sig_url) != bool(gpg_env_key):
|
||||
print('--gpg-sig-url and --gpg-key-env must both be defined')
|
||||
return 1
|
||||
|
||||
if gpg_sig_url:
|
||||
gpg_signature = b''.join(stream_download(gpg_sig_url))
|
||||
gpg_key = os.environb[gpg_env_key.encode('ascii')]
|
||||
|
||||
dest = pathlib.Path(args.dest)
|
||||
dest.parent.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
try:
|
||||
download_to_path(args.url, dest, sha256=args.sha256, size=args.size)
|
||||
|
||||
if gpg_sig_url:
|
||||
gpg_verify_path(dest, gpg_key, gpg_signature)
|
||||
|
||||
except Exception:
|
||||
try:
|
||||
dest.unlink()
|
||||
except FileNotFoundError:
|
||||
pass
|
||||
|
||||
raise
|
||||
|
||||
|
||||
def command_task_artifacts(args):
|
||||
urls = []
|
||||
|
||||
for word in args.artifacts:
|
||||
# Format is name@task
|
||||
artifact, task_id = word.split('@', 1)
|
||||
|
||||
urls.append(ARTIFACT_URL.format(task=task_id,
|
||||
artifact=artifact))
|
||||
|
||||
fetch_urls(urls, pathlib.Path(args.dest))
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser()
|
||||
subparsers = parser.add_subparsers(title='sub commands')
|
||||
|
||||
url = subparsers.add_parser('static-url', help='Download a static URL')
|
||||
url.set_defaults(func=command_static_url)
|
||||
url.add_argument('--sha256', required=True,
|
||||
help='SHA-256 of downloaded content')
|
||||
url.add_argument('--size', required=True, type=int,
|
||||
help='Size of downloaded content, in bytes')
|
||||
url.add_argument('--gpg-sig-url',
|
||||
help='URL containing signed GPG document validating '
|
||||
'URL to fetch')
|
||||
url.add_argument('--gpg-key-env',
|
||||
help='Environment variable containing GPG key to validate')
|
||||
url.add_argument('url', help='URL to fetch')
|
||||
url.add_argument('dest', help='Destination path')
|
||||
|
||||
artifacts = subparsers.add_parser('task-artifacts',
|
||||
help='Fetch task artifacts')
|
||||
artifacts.set_defaults(func=command_task_artifacts)
|
||||
artifacts.add_argument('-d', '--dest', help='Destination directory')
|
||||
artifacts.add_argument('artifacts', nargs='+',
|
||||
help='Artifacts to fetch. Of form path@task_id')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
return args.func(args)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
sys.exit(main())
|
||||
|
|
@ -0,0 +1,138 @@
|
|||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
# Support for running tasks that download remote content and re-export
|
||||
# it as task artifacts.
|
||||
|
||||
from __future__ import absolute_import, unicode_literals
|
||||
|
||||
import os
|
||||
|
||||
from voluptuous import (
|
||||
Optional,
|
||||
Required,
|
||||
)
|
||||
|
||||
import taskgraph
|
||||
|
||||
from . import (
|
||||
run_job_using,
|
||||
)
|
||||
from ...util.cached_tasks import (
|
||||
add_optimization,
|
||||
)
|
||||
from ...util.schema import (
|
||||
Schema,
|
||||
)
|
||||
|
||||
|
||||
CACHE_TYPE = 'content.v1'
|
||||
|
||||
|
||||
url_schema = Schema({
|
||||
Required('using'): 'fetch-url',
|
||||
|
||||
# Base work directory used to set up the task.
|
||||
Required('workdir'): basestring,
|
||||
|
||||
# The URL to download.
|
||||
Required('url'): basestring,
|
||||
|
||||
# The SHA-256 of the downloaded content.
|
||||
Required('sha256'): basestring,
|
||||
|
||||
# Size of the downloaded entity, in bytes.
|
||||
Required('size'): int,
|
||||
|
||||
# GPG signature verification.
|
||||
Optional('gpg-signature'): {
|
||||
# URL where GPG signature document can be obtained. Can contain the
|
||||
# value ``{url}``, which will be substituted with the value from
|
||||
# ``url``.
|
||||
Required('sig-url'): basestring,
|
||||
# Path to file containing GPG public key(s) used to validate download.
|
||||
Required('key-path'): basestring,
|
||||
},
|
||||
|
||||
# The name to give to the generated artifact.
|
||||
Optional('artifact-name'): basestring,
|
||||
|
||||
# IMPORTANT: when adding anything that changes the behavior of the task,
|
||||
# it is important to update the digest data used to compute cache hits.
|
||||
})
|
||||
|
||||
|
||||
@run_job_using('docker-worker', 'fetch-url',
|
||||
schema=url_schema)
|
||||
def cache_url(config, job, taskdesc):
|
||||
"""Configure a task to download a URL and expose it as an artifact."""
|
||||
run = job['run']
|
||||
|
||||
worker = taskdesc['worker']
|
||||
worker['chain-of-trust'] = True
|
||||
|
||||
# Fetch tasks are idempotent and immutable. Have them live for
|
||||
# essentially forever.
|
||||
if config.params['level'] == '3':
|
||||
expires = '1000 years'
|
||||
else:
|
||||
expires = '28 days'
|
||||
|
||||
taskdesc['expires-after'] = expires
|
||||
|
||||
artifact_name = run.get('artifact-name')
|
||||
if not artifact_name:
|
||||
artifact_name = run['url'].split('/')[-1]
|
||||
|
||||
worker.setdefault('artifacts', []).append({
|
||||
'type': 'directory',
|
||||
'name': 'public',
|
||||
'path': '/builds/worker/artifacts',
|
||||
})
|
||||
|
||||
env = worker.setdefault('env', {})
|
||||
|
||||
args = [
|
||||
'/builds/worker/bin/fetch-content', 'static-url',
|
||||
'--sha256', run['sha256'],
|
||||
'--size', '%d' % run['size'],
|
||||
]
|
||||
|
||||
if 'gpg-signature' in run:
|
||||
sig_url = run['gpg-signature']['sig-url'].format(url=run['url'])
|
||||
key_path = os.path.join(taskgraph.GECKO, run['gpg-signature'][
|
||||
'key-path'])
|
||||
|
||||
with open(key_path, 'rb') as fh:
|
||||
gpg_key = fh.read()
|
||||
|
||||
env['FETCH_GPG_KEY'] = gpg_key
|
||||
args.extend([
|
||||
'--gpg-sig-url', sig_url,
|
||||
'--gpg-key-env', 'FETCH_GPG_KEY',
|
||||
])
|
||||
|
||||
args.extend([
|
||||
run['url'], '/builds/worker/artifacts/%s' % artifact_name,
|
||||
])
|
||||
|
||||
worker['command'] = ['/builds/worker/bin/run-task', '--'] + args
|
||||
|
||||
attributes = taskdesc.setdefault('attributes', {})
|
||||
attributes['fetch-artifact'] = 'public/%s' % artifact_name
|
||||
|
||||
if not taskgraph.fast:
|
||||
cache_name = taskdesc['label'].replace('{}-'.format(config.kind), '', 1)
|
||||
|
||||
# This adds the level to the index path automatically.
|
||||
add_optimization(
|
||||
config,
|
||||
taskdesc,
|
||||
cache_type=CACHE_TYPE,
|
||||
cache_name=cache_name,
|
||||
# We don't include the GPG signature in the digest because it isn't
|
||||
# materially important for caching: GPG signatures are supplemental
|
||||
# trust checking beyond what the shasum already provides.
|
||||
digest_data=[run['sha256'], '%d' % run['size'], artifact_name],
|
||||
)
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
# This Source Code Form is subject to the terms of the Mozilla Public
|
||||
# License, v. 2.0. If a copy of the MPL was not distributed with this
|
||||
# file, You can obtain one at http://mozilla.org/MPL/2.0/.
|
||||
|
||||
from __future__ import absolute_import, unicode_literals
|
||||
|
||||
from .base import (
|
||||
TransformSequence,
|
||||
)
|
||||
|
||||
|
||||
transforms = TransformSequence()
|
||||
|
||||
|
||||
def get_attribute(dict, key, attributes, attribute_name):
|
||||
'''Get `attribute_name` from the given `attributes` dict, and if there
|
||||
is a corresponding value, set `key` in `dict` to that value.'''
|
||||
value = attributes.get(attribute_name)
|
||||
if value:
|
||||
dict[key] = value
|
||||
|
||||
|
||||
@transforms.add
|
||||
def use_fetches(config, jobs):
|
||||
artifacts = {}
|
||||
|
||||
for task in config.kind_dependencies_tasks:
|
||||
if task.kind != 'fetch':
|
||||
continue
|
||||
|
||||
name = task.label.replace('%s-' % task.kind, '')
|
||||
get_attribute(artifacts, name, task.attributes, 'fetch-artifact')
|
||||
|
||||
for job in jobs:
|
||||
fetches = job.pop('fetches', [])
|
||||
|
||||
for fetch in fetches:
|
||||
if fetch not in artifacts:
|
||||
raise Exception('Missing fetch job for %s-%s: %s' % (
|
||||
config.kind, job['name'], fetch))
|
||||
|
||||
if not artifacts[fetch].startswith('public/'):
|
||||
raise Exception('non-public artifacts not supported')
|
||||
|
||||
if fetches:
|
||||
job.setdefault('dependencies', {}).update(
|
||||
('fetch-%s' % f, 'fetch-%s' % f)
|
||||
for f in fetches)
|
||||
|
||||
env = job.setdefault('worker', {}).setdefault('env', {})
|
||||
env['MOZ_FETCHES'] = {'task-reference': ' '.join(
|
||||
'%s@<fetch-%s>' % (artifacts[f], f)
|
||||
for f in fetches)}
|
||||
|
||||
yield job
|
||||
Загрузка…
Ссылка в новой задаче