* Allow for cache alias

* Bump task version
This commit is contained in:
Ethan Dennis 2019-07-02 17:12:20 -07:00 коммит произвёл GitHub
Родитель 12aae5a871
Коммит 255505f08b
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
15 изменённых файлов: 358 добавлений и 33 удалений

Просмотреть файл

@ -11,9 +11,9 @@ This build task is meant to add an easy way to provide caching of intermediate b
```yaml
- task: 1ESLighthouseEng.PipelineArtifactCaching.RestoreCacheV1.RestoreCache@1
inputs:
keyfile: '**/yarn.lock, !**/node_modules/**/yarn.lock, !**/.*/**/yarn.lock'
targetfolder: '**/node_modules, !**/node_modules/**/node_modules'
vstsFeed: '$(ArtifactFeed)'
keyfile: "**/yarn.lock, !**/node_modules/**/yarn.lock, !**/.*/**/yarn.lock"
targetfolder: "**/node_modules, !**/node_modules/**/node_modules"
vstsFeed: "$(ArtifactFeed)"
- script: |
yarn install
@ -21,14 +21,15 @@ This build task is meant to add an easy way to provide caching of intermediate b
- task: 1ESLighthouseEng.PipelineArtifactCaching.SaveCacheV1.SaveCache@1
inputs:
keyfile: '**/yarn.lock, !**/node_modules/**/yarn.lock, !**/.*/**/yarn.lock'
targetfolder: '**/node_modules, !**/node_modules/**/node_modules'
vstsFeed: '$(ArtifactFeed)'
keyfile: "**/yarn.lock, !**/node_modules/**/yarn.lock, !**/.*/**/yarn.lock"
targetfolder: "**/node_modules, !**/node_modules/**/node_modules"
vstsFeed: "$(ArtifactFeed)"
```
Conceptually, this snippet creates a lookup key from the `keyfile` argument and checks the `vstsFeed` for a matching entry. If one exists, it will be downloaded and unpacked. After more `node_modules` are restored via `yarn` the `SaveCache` task runs to create a cache entry if it wasn't available previously (if a cache entry was downloaded, this is a no-op).
Inputs:
- `keyfile`: The file or pattern of files to use for creating the lookup key of the cache. Due to the nature of `node_modules` potentially having their own `yarn.lock` file, this snippet explicitly excludes that pattern to ensure there is a consistent lookup key before and after package restoration.
- `targetfolder`: The file/folder or pattern of files/folders that you want to cache. The matching files/folders will be represented as the universal package that is uploaded to your Azure DevOps artifact feed.
- `vstsFeed`: The guid representing the artifact feed in Azure DevOps meant to store the build's caches.
@ -38,9 +39,9 @@ If you do not want to add two build steps to your build definition, you can also
```yaml
- task: 1ESLighthouseEng.PipelineArtifactCaching.RestoreAndSaveCacheV1.RestoreAndSaveCache@1
inputs:
keyfile: '**/yarn.lock, !**/node_modules/**/yarn.lock, !**/.*/**/yarn.lock'
targetfolder: '**/node_modules, !**/node_modules/**/node_modules'
vstsFeed: '$(ArtifactFeed)'
keyfile: "**/yarn.lock, !**/node_modules/**/yarn.lock, !**/.*/**/yarn.lock"
targetfolder: "**/node_modules, !**/node_modules/**/node_modules"
vstsFeed: "$(ArtifactFeed)"
- script: |
yarn install
@ -56,9 +57,9 @@ In the following example, the 'yarn' task will only run if there was not a cache
```yaml
- task: 1ESLighthouseEng.PipelineArtifactCaching.RestoreAndSaveCacheV1.RestoreAndSaveCache@1
inputs:
keyfile: '**/yarn.lock, !**/node_modules/**/yarn.lock, !**/.*/**/yarn.lock'
targetfolder: '**/node_modules, !**/node_modules/**/node_modules'
vstsFeed: '$(ArtifactFeed)'
keyfile: "**/yarn.lock, !**/node_modules/**/yarn.lock, !**/.*/**/yarn.lock"
targetfolder: "**/node_modules, !**/node_modules/**/node_modules"
vstsFeed: "$(ArtifactFeed)"
- script: |
yarn install
@ -66,6 +67,26 @@ In the following example, the 'yarn' task will only run if there was not a cache
condition: ne(variables['CacheRestored'], 'true')
```
### Cache aliases
By default, the name of the variable used for optimistic cache restoration defaults to `CacheRestored`. However, this can be problematic in restoring multiple caches in the same build (E.g. caches for build output and for packages). To work around this, you may set an optional task variable to control the naming of the `CacheRestored` variable.
For example:
```yaml
- task: 1ESLighthouseEng.PipelineArtifactCaching.RestoreAndSaveCacheV1.RestoreAndSaveCache@1
inputs:
keyfile: "yarn.lock"
targetfolder: "node_modules"
vstsFeed: "$(ArtifactFeed)"
alias: "Packages"
- script: |
yarn install
displayName: Install Dependencies
condition: ne(variables['CacheRestored-Packages'], 'true')
```
## Platform independent caches
By default, cached archives are platform _dependent_ to support the small differences that may occur in packages produced for a specific platform. If you are certain that the cached archive will be platform _independent_, you can set the task variable `platformIndependent` to true and all platforms will restore the same archive.
@ -79,7 +100,6 @@ For example:
targetfolder: bin
vstsFeed: $(ArtifactFeed)
platformIndependent: true
```
## Onboarding
@ -133,13 +153,13 @@ npm install
### Build
The following instructions demonstrate how to build and test either all or a specific task. The output will be sent to
the `_build` directory. You can then use the tfx client to upload this to your server for testing.
the `_build` directory. You can then use the tfx client to upload this to your server for testing.
The build will also generate a `task.loc.json` and an english strings file under `Strings` in your source tree. You should check these back in. Another localization process will create the other strings files.
To build all tasks:
``` bash
```bash
npm run build
```
@ -151,7 +171,7 @@ node make.js build --task RestoreCacheV1
## Contributing
This project welcomes contributions and suggestions. Most contributions require you to agree to a
This project welcomes contributions and suggestions. Most contributions require you to agree to a
Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us
the rights to use your contribution. For details, visit https://cla.microsoft.com.

Просмотреть файл

@ -62,6 +62,10 @@ export class cacheUtilities {
try {
const result = await universalPackages.download(hash, tmp_cache);
const alias = tl.getInput("alias", false);
const output =
alias && alias.length > 0 ? `CacheRestored-${alias}` : "CacheRestored";
if (!result.toolRan) {
tl.warning("Issue running universal packages tools");
} else if (result.success) {
@ -70,14 +74,14 @@ export class cacheUtilities {
// Set variable to track whether or not we downloaded cache (i.e. it already existed)
tl.setVariable(hash, "true");
tl.setVariable("CacheRestored", "true");
tl.setVariable(output, "true");
return;
} catch (err) {
console.log(err);
}
} else {
console.log("Cache miss: ", hash);
tl.setVariable("CacheRestored", "false");
tl.setVariable(output, "false");
tl.setVariable(hash, "false");
}
} catch (err) {

Просмотреть файл

@ -9,6 +9,7 @@
"loc.input.help.targetfolder": "The folder/file or wildcard of items to cache. For example, node projects can cache packages with '**/node_modules, !**/node_modules/**/node_modules'.",
"loc.input.label.feedList": "Feed",
"loc.input.label.platformIndependent": "Platform Independent?",
"loc.input.label.alias": "Cache alias",
"loc.input.label.verbosity": "Verbosity",
"loc.input.help.verbosity": "Specifies the amount of detail displayed in the output.",
"loc.messages.PackagesDownloadedSuccessfully": "Package were downloaded successfully",

Просмотреть файл

@ -0,0 +1,88 @@
import * as tmrm from "azure-pipelines-task-lib/mock-run";
import * as path from "path";
import * as fs from "fs";
import { TaskLibAnswers } from "azure-pipelines-task-lib/mock-answer";
import { UniversalMockHelper } from "packaging-common/Tests/UniversalMockHelper";
import { Constants } from "./Constants";
const taskPath = path.join(__dirname, "..", "restorecache.js");
const tmr: tmrm.TaskMockRunner = new tmrm.TaskMockRunner(taskPath);
const a: TaskLibAnswers = {
findMatch: {
"**/*/yarn.lock": ["src/webapi/yarn.lock", "src/application/yarn.lock"],
"**/*/node_modules": []
},
rmRF: {
"/users/home/directory/tmp_cache": { success: true }
},
checkPath: {},
exec: {},
exist: {},
which: {}
};
tmr.setAnswers(a);
const umh: UniversalMockHelper = new UniversalMockHelper(
tmr,
a,
"/users/tmp/ArtifactTool.exe"
);
umh.mockUniversalCommand(
"download",
"node-package-feed",
"builddefinition1",
`1.0.0-${process.platform}-${Constants.Hash}`,
"/users/home/directory/tmp_cache",
{
code: 0,
stdout: "ArtifactTool.exe output",
stderr: ""
}
);
tmr.setInput("keyFile", "**/*/yarn.lock");
tmr.setInput("targetFolders", "**/*/node_modules");
tmr.setInput("alias", "Build");
// mock a specific module function called in task
tmr.registerMock("fs", {
readFileSync(
path: string,
options:
| string
| {
encoding: string;
flag?: string;
}
): string {
if (path.endsWith("/yarn.lock")) {
const segments = path.split("/");
return segments.splice(segments.length - 3).join("/");
}
return fs.readFileSync(path, options);
},
chmodSync: fs.chmodSync,
writeFileSync: fs.writeFileSync,
readdirSync: fs.readdirSync,
mkdirSync: fs.mkdirSync,
copyFileSync: fs.copyFileSync,
statSync: fs.statSync,
linkSync: fs.linkSync,
symlinkSync: fs.symlinkSync
});
tmr.registerMock("shelljs", {
exec(command: string) {
console.log(`Mock executing command: ${command}`);
return {
code: 0,
stdout: "shelljs output",
stderr: null
};
}
});
tmr.run();

Просмотреть файл

@ -0,0 +1,88 @@
import * as tmrm from "azure-pipelines-task-lib/mock-run";
import * as path from "path";
import * as fs from "fs";
import { TaskLibAnswers } from "azure-pipelines-task-lib/mock-answer";
import { UniversalMockHelper } from "packaging-common/Tests/UniversalMockHelper";
import { Constants } from "./Constants";
const taskPath = path.join(__dirname, "..", "restorecache.js");
const tmr: tmrm.TaskMockRunner = new tmrm.TaskMockRunner(taskPath);
const a: TaskLibAnswers = {
findMatch: {
"**/*/yarn.lock": ["src/webapi/yarn.lock", "src/application/yarn.lock"],
"**/*/node_modules": []
},
rmRF: {
"/users/home/directory/tmp_cache": { success: true }
},
checkPath: {},
exec: {},
exist: {},
which: {}
};
tmr.setAnswers(a);
const umh: UniversalMockHelper = new UniversalMockHelper(
tmr,
a,
"/users/tmp/ArtifactTool.exe"
);
umh.mockUniversalCommand(
"download",
"node-package-feed",
"builddefinition1",
`1.0.0-${process.platform}-${Constants.Hash}`,
"/users/home/directory/tmp_cache",
{
code: 1,
stdout: "ArtifactTool.exe output",
stderr: "Can't find the package "
}
);
tmr.setInput("keyFile", "**/*/yarn.lock");
tmr.setInput("targetFolders", "**/*/node_modules");
tmr.setInput("alias", "Build");
// mock a specific module function called in task
tmr.registerMock("fs", {
readFileSync(
path: string,
options:
| string
| {
encoding: string;
flag?: string;
}
): string {
if (path.endsWith("/yarn.lock")) {
const segments = path.split("/");
return segments.splice(segments.length - 3).join("/");
}
return fs.readFileSync(path, options);
},
chmodSync: fs.chmodSync,
writeFileSync: fs.writeFileSync,
readdirSync: fs.readdirSync,
mkdirSync: fs.mkdirSync,
copyFileSync: fs.copyFileSync,
statSync: fs.statSync,
linkSync: fs.linkSync,
symlinkSync: fs.symlinkSync
});
tmr.registerMock("shelljs", {
exec(command: string) {
console.log(`Mock executing command: ${command}`);
return {
code: 0,
stdout: "shelljs output",
stderr: null
};
}
});
tmr.run();

Просмотреть файл

@ -129,6 +129,41 @@ describe("RestoreCache tests", function() {
done();
});
it("RestoreCache runs successfully if cache hit for cache alias", (done: MochaDone) => {
const tp = path.join(__dirname, "RestoreCacheCacheHitCacheAlias.js");
const tr: ttm.MockTestRunner = new ttm.MockTestRunner(tp);
tr.run();
assert(tr.invokedToolCount === 1, "should have run ArtifactTool once");
assert(
tr.ran(
`/users/tmp/ArtifactTool.exe universal download --feed node-package-feed --service https://example.visualstudio.com/defaultcollection --package-name builddefinition1 --package-version 1.0.0-${
process.platform
}-${
Constants.Hash
} --path /users/home/directory/tmp_cache --patvar UNIVERSAL_DOWNLOAD_PAT --verbosity verbose`
),
"it should have run ArtifactTool"
);
assert(
tr.stdOutContained("ArtifactTool.exe output"),
"should have ArtifactTool output"
);
assert(tr.succeeded, "should have succeeded");
assert.equal(tr.errorIssues.length, 0, "should have no errors");
assert(
tr.stdOutContained("set CacheRestored-Build=true"),
"'CacheRestored-Build' variable should be set to true"
);
assert(
tr.stdOutContained(`${process.platform}-${Constants.Hash}=true`),
"variable should be set to mark key as valid in build"
);
done();
});
it("RestoreCache runs successfully if cache miss", (done: MochaDone) => {
const tp = path.join(__dirname, "RestoreCacheCacheMiss.js");
const tr: ttm.MockTestRunner = new ttm.MockTestRunner(tp);
@ -168,6 +203,45 @@ describe("RestoreCache tests", function() {
done();
});
it("RestoreCache runs successfully if cache miss for cache alias", (done: MochaDone) => {
const tp = path.join(__dirname, "RestoreCacheCacheMissCacheAlias.js");
const tr: ttm.MockTestRunner = new ttm.MockTestRunner(tp);
tr.run();
assert(tr.invokedToolCount === 1, "should have run ArtifactTool once");
assert(
tr.ran(
`/users/tmp/ArtifactTool.exe universal download --feed node-package-feed --service https://example.visualstudio.com/defaultcollection --package-name builddefinition1 --package-version 1.0.0-${
process.platform
}-${
Constants.Hash
} --path /users/home/directory/tmp_cache --patvar UNIVERSAL_DOWNLOAD_PAT --verbosity verbose`
),
"it should have run ArtifactTool"
);
assert(
tr.stdOutContained("ArtifactTool.exe output"),
"should have ArtifactTool output"
);
assert(
tr.stdOutContained(`Cache miss: ${process.platform}-${Constants.Hash}`),
"should have output stating cache miss"
);
assert(tr.succeeded, "should have succeeded");
assert.equal(tr.errorIssues.length, 0, "should have no errors");
assert(
tr.stdOutContained("set CacheRestored-Build=false"),
"'CacheRestored' variable should be set to false"
);
assert(
tr.stdOutContained(`${process.platform}-${Constants.Hash}=false`),
"variable should be set to mark key as valid in build"
);
done();
});
it("RestoreCache handles artifact permissions errors gracefully", (done: MochaDone) => {
const tp = path.join(__dirname, "RestoreCachePermissionsError.js");
const tr: ttm.MockTestRunner = new ttm.MockTestRunner(tp);

Просмотреть файл

@ -9,7 +9,7 @@
"version": {
"Major": 1,
"Minor": 0,
"Patch": 12
"Patch": 13
},
"instanceNameFormat": "Restore and save artifact based on: $(keyfile)",
"inputs": [
@ -45,6 +45,14 @@
"defaultValue": false,
"required": "false"
},
{
"name": "alias",
"type": "string",
"label": "Cache alias",
"description": "An optional alias to the cache to control the name of the output variable (E.g. An alias of 'Build' sets the output variable 'CacheRestored-Build').",
"defaultValue": "",
"required": "false"
},
{
"name": "verbosity",
"type": "pickList",

Просмотреть файл

@ -9,7 +9,7 @@
"version": {
"Major": 1,
"Minor": 0,
"Patch": 11
"Patch": 12
},
"instanceNameFormat": "ms-resource:loc.instanceNameFormat",
"inputs": [
@ -47,6 +47,14 @@
"defaultValue": false,
"required": "false"
},
{
"name": "alias",
"type": "string",
"label": "ms-resource:loc.input.label.alias",
"description": "An optional alias to the cache to control the name of the output variable (E.g. An alias of 'Build' sets the output variable 'CacheRestored-Build').",
"defaultValue": "",
"required": "false"
},
{
"name": "verbosity",
"type": "pickList",

Просмотреть файл

@ -9,6 +9,7 @@
"loc.input.help.targetfolder": "The folder/file or wildcard of items to cache. For example, node projects can cache packages with '**/node_modules, !**/node_modules/**/node_modules'.",
"loc.input.label.feedList": "Feed",
"loc.input.label.platformIndependent": "Platform Independent?",
"loc.input.label.alias": "Cache alias",
"loc.input.label.verbosity": "Verbosity",
"loc.input.help.verbosity": "Specifies the amount of detail displayed in the output.",
"loc.messages.PackagesDownloadedSuccessfully": "Package were downloaded successfully",

Просмотреть файл

@ -9,7 +9,7 @@
"version": {
"Major": 1,
"Minor": 0,
"Patch": 12
"Patch": 13
},
"instanceNameFormat": "Restore artifact based on: $(keyfile)",
"inputs": [
@ -45,6 +45,14 @@
"defaultValue": false,
"required": "false"
},
{
"name": "alias",
"type": "string",
"label": "Cache alias",
"description": "An optional alias to the cache to control the name of the output variable (E.g. An alias of 'Build' sets the output variable 'CacheRestored-Build').",
"defaultValue": "",
"required": "false"
},
{
"name": "verbosity",
"type": "pickList",

Просмотреть файл

@ -9,7 +9,7 @@
"version": {
"Major": 1,
"Minor": 0,
"Patch": 11
"Patch": 12
},
"instanceNameFormat": "ms-resource:loc.instanceNameFormat",
"inputs": [
@ -47,6 +47,14 @@
"defaultValue": false,
"required": "false"
},
{
"name": "alias",
"type": "string",
"label": "ms-resource:loc.input.label.alias",
"description": "An optional alias to the cache to control the name of the output variable (E.g. An alias of 'Build' sets the output variable 'CacheRestored-Build').",
"defaultValue": "",
"required": "false"
},
{
"name": "verbosity",
"type": "pickList",

Просмотреть файл

@ -9,6 +9,7 @@
"loc.input.help.targetfolder": "The folder/file or wildcard of items to cache. For example, node projects can cache packages with '**/node_modules, !**/node_modules/**/node_modules'.",
"loc.input.label.feedList": "Feed",
"loc.input.label.platformIndependent": "Platform Independent?",
"loc.input.label.alias": "Cache alias",
"loc.input.label.verbosity": "Verbosity",
"loc.input.help.verbosity": "Specifies the amount of detail displayed in the output.",
"loc.messages.PackagesDownloadedSuccessfully": "Package were downloaded successfully",

Просмотреть файл

@ -9,7 +9,7 @@
"version": {
"Major": 1,
"Minor": 0,
"Patch": 12
"Patch": 13
},
"instanceNameFormat": "Save artifact based on: $(keyfile)",
"inputs": [
@ -45,6 +45,14 @@
"defaultValue": false,
"required": "false"
},
{
"name": "alias",
"type": "string",
"label": "Cache alias",
"description": "An optional alias to the cache to control the name of the output variable (E.g. An alias of 'Build' sets the output variable 'CacheRestored-Build').",
"defaultValue": "",
"required": "false"
},
{
"name": "verbosity",
"type": "pickList",

Просмотреть файл

@ -9,7 +9,7 @@
"version": {
"Major": 1,
"Minor": 0,
"Patch": 11
"Patch": 12
},
"instanceNameFormat": "ms-resource:loc.instanceNameFormat",
"inputs": [
@ -47,6 +47,14 @@
"defaultValue": false,
"required": "false"
},
{
"name": "alias",
"type": "string",
"label": "ms-resource:loc.input.label.alias",
"description": "An optional alias to the cache to control the name of the output variable (E.g. An alias of 'Build' sets the output variable 'CacheRestored-Build').",
"defaultValue": "",
"required": "false"
},
{
"name": "verbosity",
"type": "pickList",

Просмотреть файл

@ -34,22 +34,22 @@ jobs:
verbose: false
- task: Gulp@0
displayName: 'gulp build'
displayName: "gulp build"
inputs:
targets: build
- task: Gulp@0
displayName: 'gulp test'
displayName: "gulp test"
inputs:
targets: test
arguments: '--testResults=TESTRESULTS.xml'
arguments: "--testResults=TESTRESULTS.xml"
- task: PublishTestResults@2
displayName: 'Publish Test Results **/TESTRESULTS.xml'
displayName: "Publish Test Results **/TESTRESULTS.xml"
inputs:
testResultsFiles: '**/TESTRESULTS.xml'
testResultsFiles: "**/TESTRESULTS.xml"
condition: succeededOrFailed()
- job: Package
dependsOn: BuildAndTest
pool:
@ -69,12 +69,12 @@ jobs:
verbose: false
- task: Gulp@0
displayName: 'gulp build'
displayName: "gulp build"
inputs:
targets: build
- task: ms-devlabs.vsts-developer-tools-build-tasks.tfx-installer-build-task.TfxInstaller@1
displayName: 'Use Node CLI for Azure DevOps (tfx-cli): v0.7.x'
displayName: "Use Node CLI for Azure DevOps (tfx-cli): v0.7.x"
inputs:
version: v0.7.x
@ -90,6 +90,6 @@ jobs:
inputs:
Contents: "**/*.vsix"
TargetFolder: "$(Build.ArtifactStagingDirectory)"
- task: PublishBuildArtifacts@1
displayName: "Publish build artifacts"