gecko-dev/taskcluster/docker
Dustin J. Mitchell b884691b8f Bug 1337903: add support for OS X in run-task; r=ahal
* add run.using = 'run-task' for native-engine
* modify run-task to run on OS X
  - not as root
  - without assuming /home/worker (using ~ and os.expanduser instead)
  - hg is in /usr/local/bin on OS X; trust the PATH
* add_build_dependency isn't docker-worker specific, so just rename
* support_vcs_checkout modified to omit caches on native-engine
* don't download fingerprints on OS X; these hosts are configured with
  the proper fingerprint via puppet

MozReview-Commit-ID: C83XClXtcn4

--HG--
extra : rebase_source : 2ef1e8dced12ccc4acb7706d7f4587df19a379fc
2017-03-03 22:49:23 +00:00
..
android-gradle-build Backed out changeset fe7303de56be (bug 1302763) for failing webdriver tests on Linux x64 debug (geckodriver.manifest fetch failed). r=backout 2016-12-19 21:15:07 +01:00
base-build Merge mozilla-central to mozilla-inbound 2016-12-22 16:25:35 +01:00
base-test Merge mozilla-central to mozilla-inbound 2016-12-22 16:25:35 +01:00
beet-mover Bug 1343107 - Move release images under taskcluster/docker r=nthomas a=release 2017-03-02 19:37:51 -05:00
centos6-build Merge mozilla-central to mozilla-inbound 2016-12-22 16:25:35 +01:00
centos6-build-upd Merge mozilla-central to mozilla-inbound 2016-12-22 16:25:35 +01:00
decision Merge mozilla-central to mozilla-inbound 2016-12-22 16:25:35 +01:00
desktop-build Bug 1331957 - Part 8: Build cctools-port in the infrastructure; r=froydnj,dustin 2017-02-03 10:19:28 -05:00
desktop-test Backed out changeset fe7303de56be (bug 1302763) for failing webdriver tests on Linux x64 debug (geckodriver.manifest fetch failed). r=backout 2016-12-19 21:15:07 +01:00
desktop1604-test Bug 1345105 - Disable gnome-software-service on desktop1604-test. r=dustin 2017-03-07 15:42:52 +01:00
firefox-snap Bug 1343107 - Move release images under taskcluster/docker r=nthomas a=release 2017-03-02 19:37:51 -05:00
funsize-balrog-submitter Bug 1343107 - Fix linter failures r=nthomas a=release 2017-03-02 22:40:09 -05:00
funsize-update-generator Bug 1343107 - Fix linter failures r=nthomas a=release 2017-03-02 22:40:09 -05:00
image_builder Bug 1329701 - robustcheckout.py moved r=gps 2017-01-09 11:29:38 -06:00
lint Bug 1331065 - Upgrade the Lint docker image to node v6.9.1. r=jmaher 2017-01-20 10:57:47 +00:00
recipes Bug 1337903: add support for OS X in run-task; r=ahal 2017-03-03 22:49:23 +00:00
rust-build Bug 1336155 - Update rust repack script for 1.15.0 stable. r=mshal 2017-02-02 11:02:41 -08:00
tester Merge mozilla-central to mozilla-inbound 2016-12-22 16:25:35 +01:00
upload-symbols Backed out changeset fe7303de56be (bug 1302763) for failing webdriver tests on Linux x64 debug (geckodriver.manifest fetch failed). r=backout 2016-12-19 21:15:07 +01:00
README.md Backed out changeset fe7303de56be (bug 1302763) for failing webdriver tests on Linux x64 debug (geckodriver.manifest fetch failed). r=backout 2016-12-19 21:15:07 +01:00
REGISTRY Backed out changeset fe7303de56be (bug 1302763) for failing webdriver tests on Linux x64 debug (geckodriver.manifest fetch failed). r=backout 2016-12-19 21:15:07 +01:00

README.md

Docker Images for use in TaskCluster

This folder contains various docker images used in taskcluster as well as other misc docker images which may be useful for hacking on gecko.

Organization

Each folder describes a single docker image. We have two types of images that can be defined:

  1. Task Images (build-on-push)
  2. Docker Images (prebuilt)

These images depend on one another, as described in the FROM line at the top of the Dockerfile in each folder.

Images could either be an image intended for pushing to a docker registry, or one that is meant either for local testing or being built as an artifact when pushed to vcs.

Task Images (build-on-push)

Images can be uploaded as a task artifact, indexed under a given namespace, and used in other tasks by referencing the task ID.

Important to note, these images do not require building and pushing to a docker registry, and are build per push (if necessary) and uploaded as task artifacts.

The decision task that is run per push will determine if the image needs to be built based on the hash of the context directory and if the image exists under the namespace for a given branch.

As an additional convenience, and a precaution to loading images per branch, if an image has been indexed with a given context hash for mozilla-central, any tasks requiring that image will use that indexed task. This is to ensure there are not multiple images built/used that were built from the same context. In summary, if the image has been built for mozilla-central, pushes to any branch will use that already built image.

To use within an in-tree task definition, the format is:

image:
  type: 'task-image'
  path: 'public/image.tar.zst'
  taskId: '{{#task_id_for_image}}builder{{/task_id_for_image}}'
Context Directory Hashing

Decision tasks will calculate the sha256 hash of the contents of the image directory and will determine if the image already exists for a given branch and hash or if a new image must be built and indexed.

Note: this is the contents of only the context directory, not the image contents.

The decision task will:

  1. Recursively collect the paths of all files within the context directory
  2. Sort the filenames alphabetically to ensure the hash is consistently calculated
  3. Generate a sha256 hash of the contents of each file.
  4. All file hashes will then be combined with their path and used to update the hash of the context directory.

This ensures that the hash is consistently calculated and path changes will result in different hashes being generated.

Task Image Index Namespace

Images that are built on push and uploaded as an artifact of a task will be indexed under the following namespaces.

  • docker.images.v2.level-{level}.{image_name}.latest
  • docker.images.v2.level-{level}.{image_name}.pushdate.{year}.{month}-{day}-{pushtime}
  • docker.images.v2.level-{level}.{image_name}.hash.{context_hash}

Not only can images be browsed by the pushdate and context hash, but the 'latest' namespace is meant to view the latest built image. This functions similarly to the 'latest' tag for docker images that are pushed to a registry.

Docker Registry Images (prebuilt)

Deprecation Warning: Use of prebuilt images should only be used for base images (those that other images will inherit from), or private images that must be stored in a private docker registry account. Existing public images will be converted to images that are built on push and any newly added image should follow this pattern.

These are images that are intended to be pushed to a docker registry and used by specifying the folder name in task definitions. This information is automatically populated by using the 'docker_image' convenience method in task definitions.

Example: image: {#docker_image}builder{/docker_image}

Each image has a hash and a version, given by its HASH and VERSION files. When rebuilding a prebuilt image the VERSION should be bumped. Once a new version of the image has been built the HASH file should be updated with the hash of the image.

The HASH file is the image hash as computed by docker, this is always on the format sha256:<digest>. In production images will be referenced by image hash. This mitigates attacks against the registry as well as simplifying validate of correctness. The VERSION file only serves to provide convenient names, such that old versions are easy to discover in the registry (and ensuring old versions aren't deleted by garbage-collection).

This way, older tasks which were designed to run on an older version of the image can still be executed in taskcluster, while new tasks can use the new version. Further more, this mitigates attacks against the registry as docker will verify the image hash when loading the image.

Each image also has a REGISTRY, defaulting to the REGISTRY in this directory, and specifying the image registry to which the completed image should be uploaded.

Building images

Generally, images can be pulled from the registry rather than built locally, however, for developing new images it's often helpful to hack on them locally.

To build an image, invoke mach taskcluster-build-image with the name of the folder (without a trailing slash):

./mach taskcluster-build-image <image-name>

This is a tiny wrapper around docker build -t $REGISTRY/$FOLDER:$VERSION. Once a new version image has been built and pushed to the remote registry using docker push $REGISTRY/$FOLDER:$VERSION the HASH file must be updated for the change to effect in production.

Note: If no "VERSION" file present in the image directory, the tag 'latest' will be used and no registry will be defined. The image is only meant to run locally and will overwrite any existing image with the same name and tag.

Adding a new image

The docker image primitives are very basic building block for constructing an "image" but generally don't help much with tagging it for deployment so we have a wrapper (./build.sh) which adds some sugar to help with tagging/versioning... Each folder should look something like this:

  - your_amazing_image/
    - your_amazing_image/Dockerfile: Standard docker file syntax
    - your_amazing_image/VERSION: The version of the docker file
      (required* used during tagging)
    - your_amazing_image/REGISTRY: Override default registry
      (useful for secret registries)

Conventions

In some image folders you will see .env files these can be used in conjunction with the --env-file flag in docker to provide a environment with the given environment variables. These are primarily for convenience when manually hacking on the images.

You will also see a system-setup.sh script used to build the image. Do not replicate this technique - prefer to include the commands and options directly in the Dockerfile.