mozilla
/
gecko-dev
зеркало из https://github.com/mozilla/gecko-dev.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
gecko-dev/taskcluster/docs/caches.rst

52 строки
1.7 KiB
ReStructuredText
Исходник Обычный вид История

Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
.. taskcluster_caches:
=============
Common Caches
=============
There are various caches used by the in-tree tasks. This page attempts to
document them and their appropriate use.
Version Control Caches
======================
Bug 1312475 - Add a version parameter to checkouts cache; r=dustin ff5a4bab0813 (bug 1311791) and 332a08725ed0 (bug 1292071) changed behavior of the VCS caches. First, the store cache / path was merged into the checkouts cache. Then the path of the Mercurial shared store was moved within the cache to always be rooted at the cache root. Caches are shared across tasks. Tasks can execute on any revision configured to use a cache. So, when interacting with caches, it is important to consider how every revision configured to use that cache will interact with it. Take this scenario for example. A worker executes a task where the hg shared store is rooted at /home/worker/checkouts/src/hg-shared. Then the worker executes a task where the hg shared store is rooted at /home/worker/checkouts/hg-store. `hg robustcheckout` will see the checkout from the first task. But then it sees that the store it is pointing to is at an unexpected location (checkouts/src/hg-shared instead of checkouts/hg-store). `hg robustcheckout` aggressively normalizes state to ensure consistency. So when it sees this mismatch, it blows away the checkout and creates one from checkouts/hg-store to replace it. That's a lot of overhead. And this cycle can repeat itself if the right combination of revisions run on the worker! A solution to this problem is to create a clean break from caches when cache semantics change. In TaskCluster, that means using a different cache. This commit introduces a "version" component to the checkouts cache name. By doing so, we create a clean break from all previous caches, ensuring all revisions this point forward won't encounter an hg shared store at an unexpected location. This also paves the road for easily making additional clean breaks in the future. MozReview-Commit-ID: JT8yuULKpch --HG-- extra : rebase_source : 2e5d321e4314ff7543423d3c7837b770b7c8a3a3
2016-10-24 19:58:01 +03:00
``level-{{level}}-checkouts-{{version}}``
Bug 1302776: document caches; r=gps MozReview-Commit-ID: Dy9aXGo5O4u --HG-- extra : rebase_source : 265f7548bb83db2cc5d8bc5e7bdfbfd81342c7e5
2016-10-20 17:44:26 +03:00
This cache holds version control checkouts, each in a subdirectory named
after the repo (e.g., ``gecko``).
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
Bug 1302776: document caches; r=gps MozReview-Commit-ID: Dy9aXGo5O4u --HG-- extra : rebase_source : 265f7548bb83db2cc5d8bc5e7bdfbfd81342c7e5
2016-10-20 17:44:26 +03:00
Checkouts should be read-only. If a task needs to create new files from
content of a checkout, this content should be written in a separate
directory/cache (like a workspace).
Bug 1312475 - Add a version parameter to checkouts cache; r=dustin ff5a4bab0813 (bug 1311791) and 332a08725ed0 (bug 1292071) changed behavior of the VCS caches. First, the store cache / path was merged into the checkouts cache. Then the path of the Mercurial shared store was moved within the cache to always be rooted at the cache root. Caches are shared across tasks. Tasks can execute on any revision configured to use a cache. So, when interacting with caches, it is important to consider how every revision configured to use that cache will interact with it. Take this scenario for example. A worker executes a task where the hg shared store is rooted at /home/worker/checkouts/src/hg-shared. Then the worker executes a task where the hg shared store is rooted at /home/worker/checkouts/hg-store. `hg robustcheckout` will see the checkout from the first task. But then it sees that the store it is pointing to is at an unexpected location (checkouts/src/hg-shared instead of checkouts/hg-store). `hg robustcheckout` aggressively normalizes state to ensure consistency. So when it sees this mismatch, it blows away the checkout and creates one from checkouts/hg-store to replace it. That's a lot of overhead. And this cycle can repeat itself if the right combination of revisions run on the worker! A solution to this problem is to create a clean break from caches when cache semantics change. In TaskCluster, that means using a different cache. This commit introduces a "version" component to the checkouts cache name. By doing so, we create a clean break from all previous caches, ensuring all revisions this point forward won't encounter an hg shared store at an unexpected location. This also paves the road for easily making additional clean breaks in the future. MozReview-Commit-ID: JT8yuULKpch --HG-- extra : rebase_source : 2e5d321e4314ff7543423d3c7837b770b7c8a3a3
2016-10-24 19:58:01 +03:00
A ``version`` parameter appears in the cache name to allow
backwards-incompatible changes to the cache's behavior.
Bug 1380833 - Document level-*-checkouts-v1's hg-store and the HG_STORE_PATH environment variable; r=dustin MozReview-Commit-ID: 9yRuKONHTvP --HG-- extra : rebase_source : dd8f28f9d96748bf74b4a7dd1bfdbe5cbaa4c950
2017-07-14 01:01:41 +03:00
The ``hg-store`` contains a `shared store <https://www.mercurial-scm.org/wiki/ShareExtension>`
that is is used by ``hg robustcheckout``. If you are using ``run-task`` you
should set the ``HG_STORE_PATH`` environment variable to point to this
directory. If you are using ``hg robustcheckout``, pass this directory to the
``--sharebase`` option.
Bug 1302776: document caches; r=gps MozReview-Commit-ID: Dy9aXGo5O4u --HG-- extra : rebase_source : 265f7548bb83db2cc5d8bc5e7bdfbfd81342c7e5
2016-10-20 17:44:26 +03:00
``level-{{level}}-{{project}}-tc-vcs`` (deprecated)
This cache is used internally by ``tc-vcs``. This tool is deprecated and
should be replaced with ``hg robustcheckout``.
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
Bug 1380833 - Document level-*-checkouts-v1's hg-store and the HG_STORE_PATH environment variable; r=dustin MozReview-Commit-ID: 9yRuKONHTvP --HG-- extra : rebase_source : dd8f28f9d96748bf74b4a7dd1bfdbe5cbaa4c950
2017-07-14 01:01:41 +03:00
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
Workspace Caches
================
Bug 1302776: document caches; r=gps MozReview-Commit-ID: Dy9aXGo5O4u --HG-- extra : rebase_source : 265f7548bb83db2cc5d8bc5e7bdfbfd81342c7e5
2016-10-20 17:44:26 +03:00
``level-{{level}}-*-workspace``
Bug 1289643 - Change path for checkouts from "workspace" to "checkouts"; r=dustin Currently, TaskCluster tasks tend to use the "workspace" directory as a cache that manages the source checkout *and* additional state. Historically at Mozilla, we've lumped "source checkout" and "workspace" (sometimes known as an "objdir") into the same directory. This is not ideal. Ideally, there is an immutable, read-only source checkout and all files produced from that source live in a separate directory. In this commit, the "workspace" directory for the "lint" image has been renamed to "checkouts" and all tasks using the image have been updated accordingly. By having "checkout" in the name, we clearly identify this cache as being relevant to source checkouts, which IMO can serve a different role from "workspaces." This distinction is important, as the next commit will prevent the "checkouts" cache from getting optimized out in certain tasks. To hammer this point home, documentation on common caches has been introduced. MozReview-Commit-ID: BSEc4dM5YCt --HG-- extra : rebase_source : 5a62939e066d3723736b41e14007112d92346684
2016-07-29 20:44:19 +03:00
These caches (of various names typically ending with ``workspace``)
contain state to be shared between task invocations. Use cases are
dependent on the task.
Bug 1302776: document caches; r=gps MozReview-Commit-ID: Dy9aXGo5O4u --HG-- extra : rebase_source : 265f7548bb83db2cc5d8bc5e7bdfbfd81342c7e5
2016-10-20 17:44:26 +03:00
Other
=====
Bug 1391789 - Make tooltool cache level dependent; r=dustin Caches shared across levels scare me, even if readers are purported to perform content verification. We shouldn't take any risks with released Firefox builds being contaminated by e.g. Try tasks. Also, the old cache name interferes with my desire to make cache names dynamic. This requires dynamic scopes. We already have have level-{{level}}-* scopes for caches. So having all caches prefixed with this makes things flexible. MozReview-Commit-ID: LsrcxIYoEh1 --HG-- extra : rebase_source : dfe97f92a726059200ed79afe215ef2cf1fd7bf1
2017-08-19 02:15:44 +03:00
``level-{{level}}-tooltool-cache
Tooltool invocations should use this cache. Tooltool will store files here
indexed by their hash.
``tooltool-cache`` (deprecated)
Legacy location for tooltool files. Use the per-level one instead.