mozilla
/
gecko-dev
зеркало из https://github.com/mozilla/gecko-dev.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
gecko-dev/taskcluster/docs/caches.rst

99 строки
3.7 KiB
ReStructuredText
Исходник Обычный вид История

Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
.. taskcluster_caches:
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin Today, cache names are mostly static and are brittle as a result. In theory, when a backwards incompatible change is performed on something that touches a cache, the cache name needs to be changed to ensure tasks running the old code don't see cached data from the new task. (Alternatively, all code is forward compatible, but that is hard to implement in practice.) For many things, the process works as planned. However, not everyone knows that cache names need changed. And, it isn't always obvious that some things require fresh caches. When mistakes are made, tasks break intermittently due to cache wonkiness. One area where we get into trouble is with UID and GID mismatch. Task A will use a Docker image where our standard "worker" user/group is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is common when mixing Debian and RedHel based distros.) If they use the same cache, then Task B needs to chown/chmod all files in the cache or there could be a permissions problem. This is exactly why run-task recursively chowns certain paths before dropping root privileges. Permissions setting in run-task solves permissions problems. But it doesn't solve content incompatibility problems. For that, you need to change cache names, not use caches, or blow away content when incompatibilities are detected. This commit starts the process of adding a little bit more coherence to our caching story. There are two main features in this commit: 1) Cache names tied to run-task content 2) Cache validation in run-task Taskgraph now detects when a task is using caches with run-task. When caches and run-task are both being used, the cache name is adjusted to contain a hash of run-task's content. When run-task changes, the cache name changes. So, changing run-task ensures that all caches from that point forward are "clean." This frees run-task and any functionality related to run-task (such as maintaining version control checkouts) from having to maintain backwards or forwards compatibility with any other version of run-task. This does mean that any changes to run-task effectively wipe out caches. But changes to run-task tend to be seldom, so this should be acceptable. The second part of this change is code in run-task to record per-cache properties and validate whether a populated cache is appropriate for use. To enable this, taskgraph passes a list of cache paths via an environment variable. For each cache path, run-task looks for a well-defined file containing a list of "requirements." Right now, that list is simply a version string. But other features will be worked into it. If the cache is empty, we simply write out a new requirements file and are done. If the file exists, we compare requirements and fail fast if there is a mismatch. If the cache has content but not this special file, then we abort (because this should never happen). The "requirements" validation isn't very useful now because the only entry comes from run-task's source code and modifying run-task will change the hash and cause a new cache to be used. The implementation at this point is more demonstrating the concept than doing anything terribly useful with it. MozReview-Commit-ID: HtpXIc7OD1k --HG-- extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-19 00:07:03 +03:00
Caches
======
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
There are various caches used by the in-tree tasks. This page attempts to
document them and their appropriate use.
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin Today, cache names are mostly static and are brittle as a result. In theory, when a backwards incompatible change is performed on something that touches a cache, the cache name needs to be changed to ensure tasks running the old code don't see cached data from the new task. (Alternatively, all code is forward compatible, but that is hard to implement in practice.) For many things, the process works as planned. However, not everyone knows that cache names need changed. And, it isn't always obvious that some things require fresh caches. When mistakes are made, tasks break intermittently due to cache wonkiness. One area where we get into trouble is with UID and GID mismatch. Task A will use a Docker image where our standard "worker" user/group is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is common when mixing Debian and RedHel based distros.) If they use the same cache, then Task B needs to chown/chmod all files in the cache or there could be a permissions problem. This is exactly why run-task recursively chowns certain paths before dropping root privileges. Permissions setting in run-task solves permissions problems. But it doesn't solve content incompatibility problems. For that, you need to change cache names, not use caches, or blow away content when incompatibilities are detected. This commit starts the process of adding a little bit more coherence to our caching story. There are two main features in this commit: 1) Cache names tied to run-task content 2) Cache validation in run-task Taskgraph now detects when a task is using caches with run-task. When caches and run-task are both being used, the cache name is adjusted to contain a hash of run-task's content. When run-task changes, the cache name changes. So, changing run-task ensures that all caches from that point forward are "clean." This frees run-task and any functionality related to run-task (such as maintaining version control checkouts) from having to maintain backwards or forwards compatibility with any other version of run-task. This does mean that any changes to run-task effectively wipe out caches. But changes to run-task tend to be seldom, so this should be acceptable. The second part of this change is code in run-task to record per-cache properties and validate whether a populated cache is appropriate for use. To enable this, taskgraph passes a list of cache paths via an environment variable. For each cache path, run-task looks for a well-defined file containing a list of "requirements." Right now, that list is simply a version string. But other features will be worked into it. If the cache is empty, we simply write out a new requirements file and are done. If the file exists, we compare requirements and fail fast if there is a mismatch. If the cache has content but not this special file, then we abort (because this should never happen). The "requirements" validation isn't very useful now because the only entry comes from run-task's source code and modifying run-task will change the hash and cause a new cache to be used. The implementation at this point is more demonstrating the concept than doing anything terribly useful with it. MozReview-Commit-ID: HtpXIc7OD1k --HG-- extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-19 00:07:03 +03:00
Caches are essentially isolated filesystems that are persisted between
tasks. For example, if 2 tasks run on a worker - one after the other -
and both tasks request the same cache, the subsequent task will be
able to see files in the cache that were created by the first task.
It's also worth noting that TaskCluster workers ensure a cache can only
be used by 1 task at a time. If a worker is simultaneously running
multiple tasks requesting the same named cache, the worker will
have multiple caches of the same name on the worker.
Caches and ``run-task``
Bug 1391776: cleanup of taskgraph docs; r=ahal * eliminate heading for test kinds, of which there is now only one * make the caches document have a single heading in the TOC * break out mach commands into a separate document, add ./mach taskgraph morphed * remove docs for YAML templates support (the .yml file wasn't actually used -- I expect it was a merge leftover); these are still used for actions.yml, but once that is gone the code should be removed, too. * break try out into its own document, edit to distinguish "how to run try" from "how to generate config" MozReview-Commit-ID: 76ZopWA9TPL --HG-- extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
2017-08-23 22:22:10 +03:00
-----------------------
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin Today, cache names are mostly static and are brittle as a result. In theory, when a backwards incompatible change is performed on something that touches a cache, the cache name needs to be changed to ensure tasks running the old code don't see cached data from the new task. (Alternatively, all code is forward compatible, but that is hard to implement in practice.) For many things, the process works as planned. However, not everyone knows that cache names need changed. And, it isn't always obvious that some things require fresh caches. When mistakes are made, tasks break intermittently due to cache wonkiness. One area where we get into trouble is with UID and GID mismatch. Task A will use a Docker image where our standard "worker" user/group is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is common when mixing Debian and RedHel based distros.) If they use the same cache, then Task B needs to chown/chmod all files in the cache or there could be a permissions problem. This is exactly why run-task recursively chowns certain paths before dropping root privileges. Permissions setting in run-task solves permissions problems. But it doesn't solve content incompatibility problems. For that, you need to change cache names, not use caches, or blow away content when incompatibilities are detected. This commit starts the process of adding a little bit more coherence to our caching story. There are two main features in this commit: 1) Cache names tied to run-task content 2) Cache validation in run-task Taskgraph now detects when a task is using caches with run-task. When caches and run-task are both being used, the cache name is adjusted to contain a hash of run-task's content. When run-task changes, the cache name changes. So, changing run-task ensures that all caches from that point forward are "clean." This frees run-task and any functionality related to run-task (such as maintaining version control checkouts) from having to maintain backwards or forwards compatibility with any other version of run-task. This does mean that any changes to run-task effectively wipe out caches. But changes to run-task tend to be seldom, so this should be acceptable. The second part of this change is code in run-task to record per-cache properties and validate whether a populated cache is appropriate for use. To enable this, taskgraph passes a list of cache paths via an environment variable. For each cache path, run-task looks for a well-defined file containing a list of "requirements." Right now, that list is simply a version string. But other features will be worked into it. If the cache is empty, we simply write out a new requirements file and are done. If the file exists, we compare requirements and fail fast if there is a mismatch. If the cache has content but not this special file, then we abort (because this should never happen). The "requirements" validation isn't very useful now because the only entry comes from run-task's source code and modifying run-task will change the hash and cause a new cache to be used. The implementation at this point is more demonstrating the concept than doing anything terribly useful with it. MozReview-Commit-ID: HtpXIc7OD1k --HG-- extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-19 00:07:03 +03:00
``run-task`` is our generic task wrapper script. It does common activities
like ensure a version control checkout is present.
One of the roles of ``run-task`` is to verify and sanitize caches.
It does this by storing state in a cache on its first use. If the recorded
*capabilities* of an existing cache don't match expectations for the
current task, ``run-task`` bails. This ensures that caches are only
reused by tasks with similar execution *profiles*. This prevents
accidental cache use across incompatible tasks. It also allows run-task
to make assumptions about the state of caches, which can help avoid
costly operations.
In addition, the hash of ``run-task`` is used to derive the cache name.
So any time ``run-task`` changes, a new set of caches are used. This
ensures that any backwards incompatible changes or bug fixes to
``run-task`` result in fresh caches.
Some caches are reserved for use with run-task. That property will be denoted
below.
Common Caches
Bug 1391776: cleanup of taskgraph docs; r=ahal * eliminate heading for test kinds, of which there is now only one * make the caches document have a single heading in the TOC * break out mach commands into a separate document, add ./mach taskgraph morphed * remove docs for YAML templates support (the .yml file wasn't actually used -- I expect it was a merge leftover); these are still used for actions.yml, but once that is gone the code should be removed, too. * break try out into its own document, edit to distinguish "how to run try" from "how to generate config" MozReview-Commit-ID: 76ZopWA9TPL --HG-- extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
2017-08-23 22:22:10 +03:00
-------------
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin Today, cache names are mostly static and are brittle as a result. In theory, when a backwards incompatible change is performed on something that touches a cache, the cache name needs to be changed to ensure tasks running the old code don't see cached data from the new task. (Alternatively, all code is forward compatible, but that is hard to implement in practice.) For many things, the process works as planned. However, not everyone knows that cache names need changed. And, it isn't always obvious that some things require fresh caches. When mistakes are made, tasks break intermittently due to cache wonkiness. One area where we get into trouble is with UID and GID mismatch. Task A will use a Docker image where our standard "worker" user/group is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is common when mixing Debian and RedHel based distros.) If they use the same cache, then Task B needs to chown/chmod all files in the cache or there could be a permissions problem. This is exactly why run-task recursively chowns certain paths before dropping root privileges. Permissions setting in run-task solves permissions problems. But it doesn't solve content incompatibility problems. For that, you need to change cache names, not use caches, or blow away content when incompatibilities are detected. This commit starts the process of adding a little bit more coherence to our caching story. There are two main features in this commit: 1) Cache names tied to run-task content 2) Cache validation in run-task Taskgraph now detects when a task is using caches with run-task. When caches and run-task are both being used, the cache name is adjusted to contain a hash of run-task's content. When run-task changes, the cache name changes. So, changing run-task ensures that all caches from that point forward are "clean." This frees run-task and any functionality related to run-task (such as maintaining version control checkouts) from having to maintain backwards or forwards compatibility with any other version of run-task. This does mean that any changes to run-task effectively wipe out caches. But changes to run-task tend to be seldom, so this should be acceptable. The second part of this change is code in run-task to record per-cache properties and validate whether a populated cache is appropriate for use. To enable this, taskgraph passes a list of cache paths via an environment variable. For each cache path, run-task looks for a well-defined file containing a list of "requirements." Right now, that list is simply a version string. But other features will be worked into it. If the cache is empty, we simply write out a new requirements file and are done. If the file exists, we compare requirements and fail fast if there is a mismatch. If the cache has content but not this special file, then we abort (because this should never happen). The "requirements" validation isn't very useful now because the only entry comes from run-task's source code and modifying run-task will change the hash and cause a new cache to be used. The implementation at this point is more demonstrating the concept than doing anything terribly useful with it. MozReview-Commit-ID: HtpXIc7OD1k --HG-- extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-19 00:07:03 +03:00
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
Version Control Caches
Bug 1391776: cleanup of taskgraph docs; r=ahal * eliminate heading for test kinds, of which there is now only one * make the caches document have a single heading in the TOC * break out mach commands into a separate document, add ./mach taskgraph morphed * remove docs for YAML templates support (the .yml file wasn't actually used -- I expect it was a merge leftover); these are still used for actions.yml, but once that is gone the code should be removed, too. * break try out into its own document, edit to distinguish "how to run try" from "how to generate config" MozReview-Commit-ID: 76ZopWA9TPL --HG-- extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
2017-08-23 22:22:10 +03:00
::::::::::::::::::::::
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin Today, cache names are mostly static and are brittle as a result. In theory, when a backwards incompatible change is performed on something that touches a cache, the cache name needs to be changed to ensure tasks running the old code don't see cached data from the new task. (Alternatively, all code is forward compatible, but that is hard to implement in practice.) For many things, the process works as planned. However, not everyone knows that cache names need changed. And, it isn't always obvious that some things require fresh caches. When mistakes are made, tasks break intermittently due to cache wonkiness. One area where we get into trouble is with UID and GID mismatch. Task A will use a Docker image where our standard "worker" user/group is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is common when mixing Debian and RedHel based distros.) If they use the same cache, then Task B needs to chown/chmod all files in the cache or there could be a permissions problem. This is exactly why run-task recursively chowns certain paths before dropping root privileges. Permissions setting in run-task solves permissions problems. But it doesn't solve content incompatibility problems. For that, you need to change cache names, not use caches, or blow away content when incompatibilities are detected. This commit starts the process of adding a little bit more coherence to our caching story. There are two main features in this commit: 1) Cache names tied to run-task content 2) Cache validation in run-task Taskgraph now detects when a task is using caches with run-task. When caches and run-task are both being used, the cache name is adjusted to contain a hash of run-task's content. When run-task changes, the cache name changes. So, changing run-task ensures that all caches from that point forward are "clean." This frees run-task and any functionality related to run-task (such as maintaining version control checkouts) from having to maintain backwards or forwards compatibility with any other version of run-task. This does mean that any changes to run-task effectively wipe out caches. But changes to run-task tend to be seldom, so this should be acceptable. The second part of this change is code in run-task to record per-cache properties and validate whether a populated cache is appropriate for use. To enable this, taskgraph passes a list of cache paths via an environment variable. For each cache path, run-task looks for a well-defined file containing a list of "requirements." Right now, that list is simply a version string. But other features will be worked into it. If the cache is empty, we simply write out a new requirements file and are done. If the file exists, we compare requirements and fail fast if there is a mismatch. If the cache has content but not this special file, then we abort (because this should never happen). The "requirements" validation isn't very useful now because the only entry comes from run-task's source code and modifying run-task will change the hash and cause a new cache to be used. The implementation at this point is more demonstrating the concept than doing anything terribly useful with it. MozReview-Commit-ID: HtpXIc7OD1k --HG-- extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-19 00:07:03 +03:00
``level-{{level}}-checkouts-{{hash}}``
This cache holds version control checkouts, each in a subdirectory named
after the repo (e.g., ``gecko``).
Checkouts should be read-only. If a task needs to create new files from
content of a checkout, this content should be written in a separate
directory/cache (like a workspace).
This cache name pattern is managed by ``run-task`` and must only be
used with ``run-task``.
Bug 1390700 - Support sparse checkouts in run-task; r=dustin `run-task` is taught a --sparse-profile argument to be passed down to `hg robustcheckout` for the main source checkout. It does what you expect: performs a sparse checkout using the named profile. The Taskgraph YAML for run-task is taught a "sparse-profile" property to define the sparse profile. When defined, --sparse-profile will be passed down to `run-task` and the cache name will be updated to reflect the use of sparse checkout. Our cache checking transform is updated to audit for the use of --sparse-profile without the corresponding "-sparse" cache name variation. The reason we need a distinct cache name for sparse is because clients that aren't sparse aware will be unable to read checkouts that are sparse. By forcing sparse and non-sparse into different cache pools, we avoid compatibility issues. In the ideal world, we probably support sparse profiles on all the VCS checkouts that `run-task` supports (e.g. --tools-checkout). Perfect is the enemy of done. All of this is defined in-tree and it is easy enough to change atomically. MozReview-Commit-ID: 79k7Vul0hHO --HG-- extra : rebase_source : babe9b42e2796c2341bffc6ecfe829f4daff9e0f
2017-08-24 04:54:14 +03:00
``level-{{level}}-checkouts-sparse-{{hash}}``
This is like the above but is used when the checkout is sparse (contains
a subset of files).
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin Today, cache names are mostly static and are brittle as a result. In theory, when a backwards incompatible change is performed on something that touches a cache, the cache name needs to be changed to ensure tasks running the old code don't see cached data from the new task. (Alternatively, all code is forward compatible, but that is hard to implement in practice.) For many things, the process works as planned. However, not everyone knows that cache names need changed. And, it isn't always obvious that some things require fresh caches. When mistakes are made, tasks break intermittently due to cache wonkiness. One area where we get into trouble is with UID and GID mismatch. Task A will use a Docker image where our standard "worker" user/group is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is common when mixing Debian and RedHel based distros.) If they use the same cache, then Task B needs to chown/chmod all files in the cache or there could be a permissions problem. This is exactly why run-task recursively chowns certain paths before dropping root privileges. Permissions setting in run-task solves permissions problems. But it doesn't solve content incompatibility problems. For that, you need to change cache names, not use caches, or blow away content when incompatibilities are detected. This commit starts the process of adding a little bit more coherence to our caching story. There are two main features in this commit: 1) Cache names tied to run-task content 2) Cache validation in run-task Taskgraph now detects when a task is using caches with run-task. When caches and run-task are both being used, the cache name is adjusted to contain a hash of run-task's content. When run-task changes, the cache name changes. So, changing run-task ensures that all caches from that point forward are "clean." This frees run-task and any functionality related to run-task (such as maintaining version control checkouts) from having to maintain backwards or forwards compatibility with any other version of run-task. This does mean that any changes to run-task effectively wipe out caches. But changes to run-task tend to be seldom, so this should be acceptable. The second part of this change is code in run-task to record per-cache properties and validate whether a populated cache is appropriate for use. To enable this, taskgraph passes a list of cache paths via an environment variable. For each cache path, run-task looks for a well-defined file containing a list of "requirements." Right now, that list is simply a version string. But other features will be worked into it. If the cache is empty, we simply write out a new requirements file and are done. If the file exists, we compare requirements and fail fast if there is a mismatch. If the cache has content but not this special file, then we abort (because this should never happen). The "requirements" validation isn't very useful now because the only entry comes from run-task's source code and modifying run-task will change the hash and cause a new cache to be used. The implementation at this point is more demonstrating the concept than doing anything terribly useful with it. MozReview-Commit-ID: HtpXIc7OD1k --HG-- extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-19 00:07:03 +03:00
``level-{{level}}-checkouts-{{version}}`` (deprecated)
Bug 1302776: document caches; r=gps MozReview-Commit-ID: Dy9aXGo5O4u --HG-- extra : rebase_source : 265f7548bb83db2cc5d8bc5e7bdfbfd81342c7e5
2016-10-20 17:44:26 +03:00
This cache holds version control checkouts, each in a subdirectory named
after the repo (e.g., ``gecko``).
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
Bug 1302776: document caches; r=gps MozReview-Commit-ID: Dy9aXGo5O4u --HG-- extra : rebase_source : 265f7548bb83db2cc5d8bc5e7bdfbfd81342c7e5
2016-10-20 17:44:26 +03:00
Checkouts should be read-only. If a task needs to create new files from
content of a checkout, this content should be written in a separate
directory/cache (like a workspace).
Bug 1312475 - Add a version parameter to checkouts cache; r=dustin ff5a4bab0813 (bug 1311791) and 332a08725ed0 (bug 1292071) changed behavior of the VCS caches. First, the store cache / path was merged into the checkouts cache. Then the path of the Mercurial shared store was moved within the cache to always be rooted at the cache root. Caches are shared across tasks. Tasks can execute on any revision configured to use a cache. So, when interacting with caches, it is important to consider how every revision configured to use that cache will interact with it. Take this scenario for example. A worker executes a task where the hg shared store is rooted at /home/worker/checkouts/src/hg-shared. Then the worker executes a task where the hg shared store is rooted at /home/worker/checkouts/hg-store. `hg robustcheckout` will see the checkout from the first task. But then it sees that the store it is pointing to is at an unexpected location (checkouts/src/hg-shared instead of checkouts/hg-store). `hg robustcheckout` aggressively normalizes state to ensure consistency. So when it sees this mismatch, it blows away the checkout and creates one from checkouts/hg-store to replace it. That's a lot of overhead. And this cycle can repeat itself if the right combination of revisions run on the worker! A solution to this problem is to create a clean break from caches when cache semantics change. In TaskCluster, that means using a different cache. This commit introduces a "version" component to the checkouts cache name. By doing so, we create a clean break from all previous caches, ensuring all revisions this point forward won't encounter an hg shared store at an unexpected location. This also paves the road for easily making additional clean breaks in the future. MozReview-Commit-ID: JT8yuULKpch --HG-- extra : rebase_source : 2e5d321e4314ff7543423d3c7837b770b7c8a3a3
2016-10-24 19:58:01 +03:00
A ``version`` parameter appears in the cache name to allow
backwards-incompatible changes to the cache's behavior.
Bug 1380833 - Document level-*-checkouts-v1's hg-store and the HG_STORE_PATH environment variable; r=dustin MozReview-Commit-ID: 9yRuKONHTvP --HG-- extra : rebase_source : dd8f28f9d96748bf74b4a7dd1bfdbe5cbaa4c950
2017-07-14 01:01:41 +03:00
The ``hg-store`` contains a `shared store <https://www.mercurial-scm.org/wiki/ShareExtension>`
that is is used by ``hg robustcheckout``. If you are using ``run-task`` you
should set the ``HG_STORE_PATH`` environment variable to point to this
directory. If you are using ``hg robustcheckout``, pass this directory to the
``--sharebase`` option.
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
Workspace Caches
Bug 1391776: cleanup of taskgraph docs; r=ahal * eliminate heading for test kinds, of which there is now only one * make the caches document have a single heading in the TOC * break out mach commands into a separate document, add ./mach taskgraph morphed * remove docs for YAML templates support (the .yml file wasn't actually used -- I expect it was a merge leftover); these are still used for actions.yml, but once that is gone the code should be removed, too. * break try out into its own document, edit to distinguish "how to run try" from "how to generate config" MozReview-Commit-ID: 76ZopWA9TPL --HG-- extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
2017-08-23 22:22:10 +03:00
::::::::::::::::
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
Bug 1302776: document caches; r=gps MozReview-Commit-ID: Dy9aXGo5O4u --HG-- extra : rebase_source : 265f7548bb83db2cc5d8bc5e7bdfbfd81342c7e5
2016-10-20 17:44:26 +03:00
``level-{{level}}-*-workspace``
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
These caches (of various names typically ending with ``workspace``)
contain state to be shared between task invocations. Use cases are
dependent on the task.
Bug 1302776: document caches; r=gps MozReview-Commit-ID: Dy9aXGo5O4u --HG-- extra : rebase_source : 265f7548bb83db2cc5d8bc5e7bdfbfd81342c7e5
2016-10-20 17:44:26 +03:00
Other
Bug 1391776: cleanup of taskgraph docs; r=ahal * eliminate heading for test kinds, of which there is now only one * make the caches document have a single heading in the TOC * break out mach commands into a separate document, add ./mach taskgraph morphed * remove docs for YAML templates support (the .yml file wasn't actually used -- I expect it was a merge leftover); these are still used for actions.yml, but once that is gone the code should be removed, too. * break try out into its own document, edit to distinguish "how to run try" from "how to generate config" MozReview-Commit-ID: 76ZopWA9TPL --HG-- extra : rebase_source : 6946d866f9df6eec591b9a05ddedc6467dd69e4b
2017-08-23 22:22:10 +03:00
:::::
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin Today, cache names are mostly static and are brittle as a result. In theory, when a backwards incompatible change is performed on something that touches a cache, the cache name needs to be changed to ensure tasks running the old code don't see cached data from the new task. (Alternatively, all code is forward compatible, but that is hard to implement in practice.) For many things, the process works as planned. However, not everyone knows that cache names need changed. And, it isn't always obvious that some things require fresh caches. When mistakes are made, tasks break intermittently due to cache wonkiness. One area where we get into trouble is with UID and GID mismatch. Task A will use a Docker image where our standard "worker" user/group is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is common when mixing Debian and RedHel based distros.) If they use the same cache, then Task B needs to chown/chmod all files in the cache or there could be a permissions problem. This is exactly why run-task recursively chowns certain paths before dropping root privileges. Permissions setting in run-task solves permissions problems. But it doesn't solve content incompatibility problems. For that, you need to change cache names, not use caches, or blow away content when incompatibilities are detected. This commit starts the process of adding a little bit more coherence to our caching story. There are two main features in this commit: 1) Cache names tied to run-task content 2) Cache validation in run-task Taskgraph now detects when a task is using caches with run-task. When caches and run-task are both being used, the cache name is adjusted to contain a hash of run-task's content. When run-task changes, the cache name changes. So, changing run-task ensures that all caches from that point forward are "clean." This frees run-task and any functionality related to run-task (such as maintaining version control checkouts) from having to maintain backwards or forwards compatibility with any other version of run-task. This does mean that any changes to run-task effectively wipe out caches. But changes to run-task tend to be seldom, so this should be acceptable. The second part of this change is code in run-task to record per-cache properties and validate whether a populated cache is appropriate for use. To enable this, taskgraph passes a list of cache paths via an environment variable. For each cache path, run-task looks for a well-defined file containing a list of "requirements." Right now, that list is simply a version string. But other features will be worked into it. If the cache is empty, we simply write out a new requirements file and are done. If the file exists, we compare requirements and fail fast if there is a mismatch. If the cache has content but not this special file, then we abort (because this should never happen). The "requirements" validation isn't very useful now because the only entry comes from run-task's source code and modifying run-task will change the hash and cause a new cache to be used. The implementation at this point is more demonstrating the concept than doing anything terribly useful with it. MozReview-Commit-ID: HtpXIc7OD1k --HG-- extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-19 00:07:03 +03:00
``level-{{level}}-tooltool-cache-{{hash}}
Tooltool invocations should use this cache. Tooltool will store files here
indexed by their hash.
Bug 1302776: document caches; r=gps MozReview-Commit-ID: Dy9aXGo5O4u --HG-- extra : rebase_source : 265f7548bb83db2cc5d8bc5e7bdfbfd81342c7e5
2016-10-20 17:44:26 +03:00
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin Today, cache names are mostly static and are brittle as a result. In theory, when a backwards incompatible change is performed on something that touches a cache, the cache name needs to be changed to ensure tasks running the old code don't see cached data from the new task. (Alternatively, all code is forward compatible, but that is hard to implement in practice.) For many things, the process works as planned. However, not everyone knows that cache names need changed. And, it isn't always obvious that some things require fresh caches. When mistakes are made, tasks break intermittently due to cache wonkiness. One area where we get into trouble is with UID and GID mismatch. Task A will use a Docker image where our standard "worker" user/group is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is common when mixing Debian and RedHel based distros.) If they use the same cache, then Task B needs to chown/chmod all files in the cache or there could be a permissions problem. This is exactly why run-task recursively chowns certain paths before dropping root privileges. Permissions setting in run-task solves permissions problems. But it doesn't solve content incompatibility problems. For that, you need to change cache names, not use caches, or blow away content when incompatibilities are detected. This commit starts the process of adding a little bit more coherence to our caching story. There are two main features in this commit: 1) Cache names tied to run-task content 2) Cache validation in run-task Taskgraph now detects when a task is using caches with run-task. When caches and run-task are both being used, the cache name is adjusted to contain a hash of run-task's content. When run-task changes, the cache name changes. So, changing run-task ensures that all caches from that point forward are "clean." This frees run-task and any functionality related to run-task (such as maintaining version control checkouts) from having to maintain backwards or forwards compatibility with any other version of run-task. This does mean that any changes to run-task effectively wipe out caches. But changes to run-task tend to be seldom, so this should be acceptable. The second part of this change is code in run-task to record per-cache properties and validate whether a populated cache is appropriate for use. To enable this, taskgraph passes a list of cache paths via an environment variable. For each cache path, run-task looks for a well-defined file containing a list of "requirements." Right now, that list is simply a version string. But other features will be worked into it. If the cache is empty, we simply write out a new requirements file and are done. If the file exists, we compare requirements and fail fast if there is a mismatch. If the cache has content but not this special file, then we abort (because this should never happen). The "requirements" validation isn't very useful now because the only entry comes from run-task's source code and modifying run-task will change the hash and cause a new cache to be used. The implementation at this point is more demonstrating the concept than doing anything terribly useful with it. MozReview-Commit-ID: HtpXIc7OD1k --HG-- extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-19 00:07:03 +03:00
This cache name pattern is reserved for use with ``run-task`` and must only
be used by ``run-task``
Bug 1391789 - Make tooltool cache level dependent; r=dustin Caches shared across levels scare me, even if readers are purported to perform content verification. We shouldn't take any risks with released Firefox builds being contaminated by e.g. Try tasks. Also, the old cache name interferes with my desire to make cache names dynamic. This requires dynamic scopes. We already have have level-{{level}}-* scopes for caches. So having all caches prefixed with this makes things flexible. MozReview-Commit-ID: LsrcxIYoEh1 --HG-- extra : rebase_source : dfe97f92a726059200ed79afe215ef2cf1fd7bf1
2017-08-19 02:15:44 +03:00
``tooltool-cache`` (deprecated)
Bug 1391789 - Improve cache coherence via run-task integration; r=dustin Today, cache names are mostly static and are brittle as a result. In theory, when a backwards incompatible change is performed on something that touches a cache, the cache name needs to be changed to ensure tasks running the old code don't see cached data from the new task. (Alternatively, all code is forward compatible, but that is hard to implement in practice.) For many things, the process works as planned. However, not everyone knows that cache names need changed. And, it isn't always obvious that some things require fresh caches. When mistakes are made, tasks break intermittently due to cache wonkiness. One area where we get into trouble is with UID and GID mismatch. Task A will use a Docker image where our standard "worker" user/group is UID/GID 1000:1000. Then Task B will use UID/GID 500:500. (This is common when mixing Debian and RedHel based distros.) If they use the same cache, then Task B needs to chown/chmod all files in the cache or there could be a permissions problem. This is exactly why run-task recursively chowns certain paths before dropping root privileges. Permissions setting in run-task solves permissions problems. But it doesn't solve content incompatibility problems. For that, you need to change cache names, not use caches, or blow away content when incompatibilities are detected. This commit starts the process of adding a little bit more coherence to our caching story. There are two main features in this commit: 1) Cache names tied to run-task content 2) Cache validation in run-task Taskgraph now detects when a task is using caches with run-task. When caches and run-task are both being used, the cache name is adjusted to contain a hash of run-task's content. When run-task changes, the cache name changes. So, changing run-task ensures that all caches from that point forward are "clean." This frees run-task and any functionality related to run-task (such as maintaining version control checkouts) from having to maintain backwards or forwards compatibility with any other version of run-task. This does mean that any changes to run-task effectively wipe out caches. But changes to run-task tend to be seldom, so this should be acceptable. The second part of this change is code in run-task to record per-cache properties and validate whether a populated cache is appropriate for use. To enable this, taskgraph passes a list of cache paths via an environment variable. For each cache path, run-task looks for a well-defined file containing a list of "requirements." Right now, that list is simply a version string. But other features will be worked into it. If the cache is empty, we simply write out a new requirements file and are done. If the file exists, we compare requirements and fail fast if there is a mismatch. If the cache has content but not this special file, then we abort (because this should never happen). The "requirements" validation isn't very useful now because the only entry comes from run-task's source code and modifying run-task will change the hash and cause a new cache to be used. The implementation at this point is more demonstrating the concept than doing anything terribly useful with it. MozReview-Commit-ID: HtpXIc7OD1k --HG-- extra : rebase_source : 2424696b1fde59f20152617a6ebb2afe14b94678
2017-08-19 00:07:03 +03:00
Legacy location for tooltool files. Use the per-level one instead.