Bug 1498640 - deploy latest image_builder image r=glandium

This uses the latest image_builder image (on docker hub) to build even the
image_builder image.

The change to `docker.py` handles a new API response (`aux`) from the Docker
daemon.  It's unclear what this key means, but displaying it is simple.

Differential Revision: https://phabricator.services.mozilla.com/D8441

--HG--
extra : rebase_source : b6a2c2de231bd623521a0a7a0dc595fed059b758
extra : intermediate-source : aa32ba9e6df3e20aab896a63538c8e8a432b4988
extra : source : f33b9e6aca06b7883f048c6c068704680cbfbbb2

taskcluster/docker/image_builder/VERSION

@@ -0,0 +1 @@
+3.0.0

taskcluster/docs/docker-images.rst

@@ -94,8 +94,8 @@ for docker images that are pushed to a registry.
 Docker Registry Images (prebuilt)
 :::::::::::::::::::::::::::::::::
 
-***Warning: Use of prebuilt images should only be used for base images (those that other images
+***Warning: Registry images are only used for ``decision`` and
-will inherit from), or private images that must be stored in a private docker registry account.***
+``image_builder`` images.***
 
 These are images that are intended to be pushed to a docker registry and used
 by specifying the docker image name in task definitions.  They are generally
@@ -107,17 +107,16 @@ Example:
 
     image: taskcluster/decision:0.1.10@sha256:c5451ee6c655b3d97d4baa3b0e29a5115f23e0991d4f7f36d2a8f793076d6854
 
-Each image has a repo digest and a version. The repo digest is stored in the
+Such images must always be referred to with both a version and a repo digest.
-``HASH`` file in the image directory and used to refer to the image as above.
+For the decision image, the repo digest is stored in the ``HASH`` file in the
-The version is in ``VERSION``.
+image directory and used to refer to the image as above.  The version for both
+images is in ``VERSION``.
 
-The version file only serves to provide convenient names, such that old
+The version file serves to help users identify which image is being used, and makes old
-versions are easy to discover in the registry (and ensuring old versions aren't
+versions easy to discover in the registry.
-deleted by garbage-collection).
 
-Each image directory also has a ``REGISTRY``, defaulting to the ``REGISTRY`` in
+The file ``taskcluster/docker/REGISTRY`` specifies the image registry to which
-the ``taskcluster/docker`` directory, and specifying the image registry to
+the completed image should be uploaded.
-which the completed image should be uploaded.
 
 Docker Hashes and Digests
 .........................
@@ -164,13 +163,19 @@ Docker Registry Images
 
 Landing docker registry images takes a little more care.
 
-Once a new version of the image has been built and tested locally, push it to
+Begin by bumping the ``VERSION``.  Once the new version of the image has been
-the docker registry and make note of the resulting repo digest.  Put this value
+built and tested locally, push it to the docker registry and make note of the
-in the ``HASH`` file, and update any references to the image in the code or
+resulting repo digest.  Put this value in the ``HASH`` file for the
-task definitions.
+``decision`` image and in ``taskcluster/taskgraph/transforms/docker_image.py``
+for the ``image_builder`` image.
 
 The change is now safe to use in Try pushes.
 
+Note that ``image_builder`` change can be tested directly in try pushes without
+using a registry, as the in-registry ``image_builder`` image is used to build a
+task image which is then used to build other images.  It is referenced by hash
+in ``taskcluster/taskgraph/transforms/docker_image.py``.
+
 Special Dockerfile Syntax
 -------------------------
 

taskcluster/taskgraph/transforms/docker_image.py

@@ -186,11 +186,12 @@ def fill_template(config, tasks):
 
         # We use the in-tree image_builder image to build docker images, but
         # that can't be used to build the image_builder image itself,
-        # obviously. So we fall back to the last snapshot of the image that
+        # obviously. So we fall back to an image on docker hub, identified
-        # was uploaded to docker hub.
+        # by hash.  After the image-builder image is updated, it's best to push
+        # and update this hash as well, to keep image-builder builds up to date.
         if image_name == 'image_builder':
-            worker['docker-image'] = 'taskcluster/image_builder@sha256:' + \
+            hash = 'sha256:c6622fd3e5794842ad83d129850330b26e6ba671e39c58ee288a616a3a1c4c73'
-                '24ce54a1602453bc93515aecd9d4ad25a22115fbc4b209ddb5541377e9a37315'
+            worker['docker-image'] = 'taskcluster/image_builder@' + hash
             # Keep in sync with the Dockerfile used to generate the
             # docker image whose digest is referenced above.
             worker['volumes'] = [

taskcluster/taskgraph/util/docker.py

@@ -103,6 +103,8 @@ def post_to_docker(tar, api_path, **kwargs):
                 sys.stderr.write('{}\n'.format(data['status']))
         elif 'stream' in data:
             sys.stderr.write(data['stream'])
+        elif 'aux' in data:
+            sys.stderr.write(repr(data['aux']))
         elif 'error' in data:
             sys.stderr.write('{}\n'.format(data['error']))
             # Sadly, docker doesn't give more than a plain string for errors,