Move vcpkg-ce development into the vcpkg-tool repo. (#428)

* Move vcpkg-ce development into the vcpkg-tool repo.

This change obsoletes https://github.com/microsoft/vcpkg-ce , which will be archived shortly after this merges.

ce development did not maintain a linear history but vcpkg-tool does maintain a linear history, so I have not attempted to preserve "blame".

The following files were not copied over:

* `.git/**/*`
* `.github/**/*` (workflows changes merged into pipelines.yaml)
* `.scripts/verify-pr.yaml`
* `.scripts/signing/**/*`
* `LICENSE.md`
* `test/vcpkg-ce.test.build.log`
* `azure-pipelines.yml`

The following files were modified:

* `.vscode/**/*` was placed in the root, had paths rewritten to target the "ce" subdirectory, and a few tasks renamed to indicate that they target the "ce" subset.
* `CODE_OF_CONDUCT.md`/`SECURITY.md`/`SUPPORT.md`/`PolicheckExclusion.xml` were placed in the root.
* `.gitattributes` was merged with the existing one in the root.
* `.gitignore` had a few things explicitly copied into the root but most were discarded.
* `readme.md` had some content merged.

* Add missing gitignores

* Remove broken fast-xml-parser reference.

* Fix some more vcpkg-ce names.

* Fix some repo docs.

* Add back overzealous vcpkg- removal for vcpkg-init.

* Delete support.md as it's irrelevant in the tool repo.

* Replay https://github.com/microsoft/vcpkg-ce/pull/22/

* Remove big test assets.

* Update ce/getting-started.md

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>

* Delete 'docs' folder.

Co-authored-by: Robert Schumacher <roschuma@microsoft.com>

.gitattributes

@@ -1 +1,20 @@
+# Don't allow people to merge changes to these generated files, because the result
+# may be invalid.  You need to run "rush update" again.
+ce/pnpm-lock.yaml               merge=binary
+ce/shrinkwrap.yaml              merge=binary
+ce/npm-shrinkwrap.json          merge=binary
+
+
+# Disable line ending smudges entirely.
 * -text
+
+# Rush's JSON config files use JavaScript-style code comments.  The rule below prevents pedantic
+# syntax highlighters such as GitHub's from highlighting these comments as errors.  Your text editor
+# may also require a special configuration to allow comments in JSON.
+#
+# For more information, see this issue: https://github.com/Microsoft/web-build-tools/issues/1088
+#
+*.json                       linguist-language=JSON-with-Comments
+*.png               binary
+*.gif               binary
+*.mp4               binary

.gitignore

@@ -8,8 +8,17 @@
 /CMakeSettings.json
 /out
 .DS_Store
-# Qt Creator CMake project files
 CMakeLists.txt.user
 .cache
 /vcpkg-ce.zip
-/vcpkg-ce/**
+node_modules/
+**/.rush/
+**/dist/
+/ce/test/**/*.d.ts
+/ce/test/**/*.map
+/ce/test/**/*.js
+/ce/ce/vcpkg-ce.build.log
+/ce/common/config/rush/pnpm-lock.yaml
+/ce/test/vcpkg-ce.test.build.log
+/ce/common/temp
+/vcpkg-root

CODE_OF_CONDUCT.md

@@ -0,0 +1,9 @@
+# Microsoft Open Source Code of Conduct
+
+This project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).
+
+Resources:
+
+- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)
+- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)
+- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns

README.md

@@ -16,6 +16,50 @@ tracking, and edits to which libraries are available.
 This repository contains the contents formerly at https://github.com/microsoft/vcpkg in the
 "toolsrc" tree, and build support.
 
+# Vcpkg-ce: "Configure Environment" / artifacts
+
+Parts of vcpkg powered by "ce" are currently in 'preview' -- there will most certainly be changes between now
+and when the tool is 'released' based on feedback. 
+
+You can use it, but be forewarned that we may change formats, commands, etc. 
+
+Think of it as a manifest-driven desired state configuration for C/C++ projects. 
+
+It 
+ - integrates itself into your shell (PowerShell, CMD, bash/zsh)
+ - can restore artifacts according to a manifest that follows one’s code 
+ - provides discoverability interfaces
+
+## Installation
+
+While the usage of `ce` is the same on all platforms, the installation/loading/removal is slightly different depending on the platform you're using.
+
+`ce` doesn't persist any changes to the environment, nor does it automatically add itself to the start-up environment. If you wish to make it load in a window, you can just execute the script. Manually adding that in your profile will load it in every new window.
+
+<hr>
+
+## Install/Use/Remove
+
+| OS              | Install                                             | Use                   | Remove                          |
+|-----------------|-----------------------------------------------------|-----------------------|---------------------------------|
+| **PowerShell/Pwsh** |`iex (iwr -useb https://aka.ms/vcpkg-init.ps1)`              |` . ~/.vcpkg/vcpkg-init.ps1`          | `rmdir -recurse ~/.vcpkg`          |
+| **Linux/OSX**       |`. <(curl https://aka.ms/vcpkg-init.sh -L)`                  |` . ~/.vcpkg/vcpkg-init.sh`          | `rm -rf ~/.ce`                  |
+| **CMD Shell**       |`curl -LO https://aka.ms/vcpkg-init.cmd && .\vcpkg-init.cmd` |`%USERPROFILE%\.vcpkg\vcpkg-init.cmd` | `rmdir /s /q %USERPROFILE%\.vcpkg` |
+
+## Glossary
+
+| Term       | Description                                         |
+|------------|-----------------------------------------------------|
+| `artifact` | An archive (.zip or .tar.gz-like), package (.nupkg, .vsix) binary inside which build tools or components thereof are stored. |
+| `artifact metadata` | A description of the locations one or more artifacts describing rules for which ones are deployed given selection of a host architecture, target architecture, or other properties|
+| `artifact identity` | A short string that uniquely describes a moniker that a given artifact (and its metadata) can be referenced by. They can have one of the following forms:<br> `full/identity/path` - the full identity of an artifact that is in the built-in artifact source<br>`sourcename:full/identity/path` - the full identity of an artifact that is in the artifact source specified by the sourcename prefix<br>`shortname` - the shortened unique name of an artifact that is in the built-in artifact source<br>`sourcename:shortname` - the shortened unique name of an artifact that is in the artifact source specified by the sourcename prefix<br>Shortened names are generated based off the shortest unique identity path in the given source. |
+| `artifact source` | Also known as a “feed”. An Artifact Source is a location that hosts metadata to locate artifacts. (_There is only one source currently_) |
+| `project profile` | The per-project configuration file (`environment.yaml` or `environment.json`) 
+| `AMF`&nbsp;or&nbsp;`Metadata`&nbsp;`Format` | The schema / format of the YAML/JSON files for project profiles, global settings, and artifacts metadata. |
+| `activation` | The process by which a particular set of artifacts are acquired and enabled for use in a calling command program.|
+| `versions` | Version numbers are specified using the Semver format. If a version for a particular operation isn't specified, a range for the latest version ( `*` ) is assumed. A version or version range can be specified using the npm semver matching syntax. When a version is stored, it can be stored using the version range specified, a space and then the version found. (ie, the first version is what was asked for, the second is what was installed. No need for a separate lock file.) |
+
+
 # Contributing
 
 Please refer to the "contributing" section of the
@@ -36,6 +80,14 @@ with any additional questions or comments.
 The product code in this repository is licensed under the [MIT License](LICENSE.txt). The tests
 contain 3rd party code as documented in `NOTICE.txt`.
 
+# Trademarks
+
+This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft 
+trademarks or logos is subject to and must follow 
+[Microsoft's Trademark & Brand Guidelines](https://www.microsoft.com/en-us/legal/intellectualproperty/trademarks/usage/general).
+Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship.
+Any use of third-party trademarks or logos are subject to those third-party's policies.
+
 # Telemetry
 
 vcpkg collects usage data in order to help us improve your experience.

SECURITY.md

@@ -0,0 +1,41 @@
+<!-- BEGIN MICROSOFT SECURITY.MD V0.0.5 BLOCK -->
+
+## Security
+
+Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).
+
+If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below.
+
+## Reporting Security Issues
+
+**Please do not report security vulnerabilities through public GitHub issues.**
+
+Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report).
+
+If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com).  If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc).
+
+You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). 
+
+Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
+
+  * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
+  * Full paths of source file(s) related to the manifestation of the issue
+  * The location of the affected source code (tag/branch/commit or direct URL)
+  * Any special configuration required to reproduce the issue
+  * Step-by-step instructions to reproduce the issue
+  * Proof-of-concept or exploit code (if possible)
+  * Impact of the issue, including how an attacker might exploit the issue
+
+This information will help us triage your report more quickly.
+
+If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs.
+
+## Preferred Languages
+
+We prefer all communications to be in English.
+
+## Policy
+
+Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd).
+
+<!-- END MICROSOFT SECURITY.MD BLOCK -->

azure-pipelines/arch-independent-signing.signproj

@@ -11,14 +11,14 @@
   </ItemGroup>
 
   <ItemGroup>
-    <FilesToSign Include="$(IntermediateOutputPath)\vcpkg-ce\**\*.js" Exclude="$(IntermediateOutputPath)\vcpkg-ce\**\node_modules\**\*.js">
+    <FilesToSign Include="$(IntermediateOutputPath)\ce\**\*.js" Exclude="$(IntermediateOutputPath)\ce\**\node_modules\**\*.js">
       <Authenticode>Microsoft400</Authenticode>
     </FilesToSign>
-    <FilesToSign Include="$(IntermediateOutputPath)\vcpkg-ce\scripts\ce.ps1">
+    <FilesToSign Include="$(IntermediateOutputPath)\ce\scripts\ce.ps1">
       <Authenticode>Microsoft400</Authenticode>
     </FilesToSign>
-    <FilesToSign Include="$(IntermediateOutputPath)\vcpkg-ce\**\node_modules\**\*.js"
-                 Exclude="$(IntermediateOutputPath)\vcpkg-ce\common\temp\node_modules\.pnpm\**\node_modules\fast-xml-parser\src\xmlbuilder\prettifyJs2Xml.js">
+    <FilesToSign Include="$(IntermediateOutputPath)\ce\**\node_modules\**\*.js"
+                 Exclude="$(IntermediateOutputPath)\ce\common\temp\node_modules\.pnpm\**\node_modules\fast-xml-parser\src\xmlbuilder\prettifyJs2Xml.js">
       <Authenticode>3PartyScriptsSHA2</Authenticode>
     </FilesToSign>
     <FilesToSign Include="$(IntermediateOutputPath)\vcpkg-init.ps1">

azure-pipelines/pipelines.yml

@@ -1,6 +1,6 @@
 jobs:
 - job: linux_gcc_9
-  displayName: 'Ubuntu 20.04 with GCC 9'
+  displayName: 'Ubuntu 20.04 with GCC 9, plus vcpkg-ce'
   pool:
     vmImage: 'ubuntu-20.04'
   variables:
@@ -11,6 +11,34 @@ jobs:
         git clone https://github.com/microsoft/vcpkg "$VCPKG_ROOT" -n
         git -C "$VCPKG_ROOT" checkout `cat vcpkg-init/vcpkg-scripts-sha.txt`
       displayName: "Clone vcpkg repo to serve as root"
+    - task: NodeTool@0
+      inputs:
+        versionSpec: '16.x'
+      displayName: 'Install Node.js'
+    - script: |
+        cd ce
+
+        npm install -g npm
+        rc=$?; if [ $rc -ne 0 ]; then exit $rc ; fi
+
+        npx @microsoft/rush update
+        rc=$?; if [ $rc -ne 0 ]; then exit $rc ; fi
+
+        npx @microsoft/rush rebuild
+        rc=$?; if [ $rc -ne 0 ]; then exit $rc ; fi
+
+        npx @microsoft/rush test
+        rc=$?; if [ $rc -ne 0 ]; then exit $rc ; fi
+
+        npx @microsoft/rush lint
+        rc=$?; if [ $rc -ne 0 ]; then exit $rc ; fi
+
+        if [ -n "$(git status --porcelain)" ]; then
+          echo "ERROR: Working directory is dirty. Are there test output files missing from the PR?"
+          git status
+          exit 1
+        fi
+      displayName: 'Rush install, build and test vcpkg-ce'
     - bash: |
         export CXXFLAGS="-fprofile-arcs -ftest-coverage -fPIC -O0"
         cmake -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON -DVCPKG_DEVELOPMENT_WARNINGS=ON -DVCPKG_WARNINGS_AS_ERRORS=ON -DVCPKG_BUILD_FUZZING=ON -B build.amd64.debug

azure-pipelines/signing.yml

@@ -66,55 +66,42 @@ jobs:
         filePath: vcpkg-init/lock-versions.ps1
         arguments: '-Destination "$(Build.BinariesDirectory)" -VcpkgBaseVersion $(VCPKG_INITIAL_BASE_VERSION)'
     # Build and test vcpkg-ce
-    - task: PowerShell@2
-      displayName: 'Download vcpkg-ce sources'
-      inputs:
-        targetType: 'inline'
-        script: |
-          $sha = Get-Content azure-pipelines/vcpkg-ce-sha.txt -Raw
-          $sha = $sha.Trim()
-          if( test-path vcpkg-ce) { rmdir -recurse -force vcpkg-ce -ea 0 }
-          # this will keep the .git folder, which is used by set-versions to accurately set the version of ce
-          git clone https://github.com/microsoft/vcpkg-ce/ vcpkg-ce
-          cd vcpkg-ce
-          git checkout $sha
-        pwsh: true
     - task: UseNode@1
       displayName: Use Node 16 or later
       inputs:
         version: "16.x"
     - script: npm install -g @microsoft/rush
       displayName: Install Rush
-      workingDirectory: vcpkg-ce
+      workingDirectory: ce
     - script: rush update
       displayName: Install vcpkg-ce Dependencies
-      workingDirectory: vcpkg-ce
+      workingDirectory: ce
     - script: rush lint
       displayName: Check vcpkg-ce for Linting Errors
-      workingDirectory: vcpkg-ce
+      workingDirectory: ce
     - script: rush rebuild
       displayName: Build vcpkg-ce Packages
-      workingDirectory: vcpkg-ce
+      workingDirectory: ce
     - script: rush test
       displayName: Run vcpkg-ce Tests
-      workingDirectory: vcpkg-ce
+      workingDirectory: ce
     - script: |
         rush set-versions
         node -e "const c = require('./ce/package.json'); p = require('./assets/package.json') ; p.version = c.version; require('fs').writeFileSync('./assets/package.json', JSON.stringify(p,undefined,2)); console.log(``set asset version to `${p.version}``);"
       displayName: Set vcpkg-ce Package Versions
-      workingDirectory: vcpkg-ce
-    - script: mkdir "$(Build.BinariesDirectory)\vcpkg-ce" && rush deploy -t "$(Build.BinariesDirectory)\vcpkg-ce"
+      workingDirectory: ce
+    - script: mkdir "$(Build.BinariesDirectory)\ce" && rush deploy -t "$(Build.BinariesDirectory)\ce"
       displayName: Collect vcpkg-ce Dependencies
-      workingDirectory: vcpkg-ce
+      workingDirectory: ce
     - task: ComponentGovernanceComponentDetection@0
       displayName: Detect Components
       inputs:
-        ignoreDirectories: vcpkg-ce/common/temp
+        ignoreDirectories: ce/common/temp
     # Inject the NOTICE file. This must run after component detection.
     - task: msospo.ospo-extension.8d7f9abb-6896-461d-9e25-4f74ed65ddb2.notice@0
       displayName: Generate NOTICE File
       inputs:
-        outputfile: $(Build.BinariesDirectory)/vcpkg-ce/NOTICE.txt
+        outputfile: $(Build.BinariesDirectory)/ce/NOTICE.txt
     - task: MicroBuildSigningPlugin@3
       displayName: Install MicroBuild Signing
       inputs:
@@ -148,10 +135,10 @@ jobs:
         arguments: '-DestinationTarball "$(Build.BinariesDirectory)\vcpkg-standalone-bundle.tar.gz" -TempDir standalone-temp "$(Build.BinariesDirectory)\vcpkg-init.cmd" "$(Build.BinariesDirectory)\vcpkg-init.ps1" "$(Build.BinariesDirectory)\vcpkg-init"'
     - script: npm pack
       displayName: Create vcpkg-ce Pack
-      workingDirectory: $(Build.BinariesDirectory)/vcpkg-ce
+      workingDirectory: $(Build.BinariesDirectory)/ce
     - script: |
         mkdir "$(Build.ArtifactStagingDirectory)\staging"
-        move "$(Build.BinariesDirectory)\vcpkg-ce\vcpkg-ce-*.tgz" "$(Build.ArtifactStagingDirectory)\staging\vcpkg-ce.tgz"
+        move "$(Build.BinariesDirectory)\ce\vcpkg-ce-*.tgz" "$(Build.ArtifactStagingDirectory)\staging\vcpkg-ce.tgz"
         move "$(Build.BinariesDirectory)\vcpkg-standalone-bundle.tar.gz" "$(Build.ArtifactStagingDirectory)\staging\vcpkg-standalone-bundle.tar.gz"
         move "$(Build.BinariesDirectory)\vcpkg-init" "$(Build.ArtifactStagingDirectory)\staging\vcpkg-init"
         move "$(Build.BinariesDirectory)\vcpkg-init.ps1" "$(Build.ArtifactStagingDirectory)\staging\vcpkg-init.ps1"

azure-pipelines/vcpkg-ce-sha.txt

@@ -1 +0,0 @@
-b71a8fd4d01cde1ff34c16df2874fa1c763751b6

ce/.eslintignore

@@ -0,0 +1 @@
+**/*.d.ts

ce/.scripts/for-each.js

@@ -0,0 +1,76 @@
+const { spawn } = require('child_process');
+const { readFileSync } = require('fs');
+const { resolve } = require('path');
+
+/** Reads a json/jsonc file and returns the JSON.
+ * 
+ * Necessary because rush uses jsonc ( >,< )
+ */
+function read(filename) {
+  const txt = readFileSync(filename, "utf8")
+    .replace(/\r/gm, "")
+    .replace(/\n/gm, "«")
+    .replace(/\/\*.*?\*\//gm, "")
+    .replace(/«/gm, "\n")
+    .replace(/\s+\/\/.*/g, "");
+  return JSON.parse(txt);
+}
+
+const repo = `${__dirname}/..`;
+
+const rush = read(`${repo}/rush.json`);
+const pjs = {};
+
+function forEachProject(onEach) {
+  // load all the projects
+  for (const each of rush.projects) {
+    const packageName = each.packageName;
+    const projectFolder = resolve(`${repo}/${each.projectFolder}`);
+    const project = JSON.parse(readFileSync(`${projectFolder}/package.json`));
+    onEach(packageName, projectFolder, project);
+  }
+}
+
+function npmForEach(cmd) {
+  let count = 0;
+  let failing = false;
+  const result = {};
+  const procs = [];
+  const t1 = process.uptime() * 100;
+  forEachProject((name, location, project) => {
+    // checks for the script first
+    if (project.scripts[cmd]) {
+      count++;
+      const proc = spawn("npm", ["--silent", "run", cmd], { cwd: location, shell: true, stdio: "inherit" });
+      procs.push(proc);
+      result[name] = {
+        name, location, project, proc,
+      };
+    }
+  });
+
+  procs.forEach(proc => proc.on("close", (code, signal) => {
+    count--;
+    failing || !!code;
+
+    if (count === 0) {
+      const t2 = process.uptime() * 100;
+
+      console.log('---------------------------------------------------------');
+      if (failing) {
+        console.log(`  Done : command '${cmd}' - ${Math.floor(t2 - t1) / 100} s -- Errors encountered. `)
+      } else {
+        console.log(`  Done : command '${cmd}' - ${Math.floor(t2 - t1) / 100} s -- No Errors `)
+      }
+      console.log('---------------------------------------------------------');
+      process.exit(failing ? 1 : 0);
+    }
+  }));
+
+  return result;
+}
+
+module.exports.forEachProject = forEachProject;
+module.exports.npm = npmForEach;
+module.exports.projectCount = rush.projects.length;
+module.exports.read = read;

ce/.scripts/npm-run.js

@@ -0,0 +1,2 @@
+// Runs the npm run command on each project that has it.
+require('./for-each').npm(process.argv[2]);

ce/.scripts/set-versions.js

@@ -0,0 +1,47 @@
+const { exec } = require('child_process');
+const { writeFileSync } = require('fs');
+const { forEachProject, projectCount } = require('./for-each');
+
+let count = projectCount;
+
+
+function updateVersion(name, project, location, patch) {
+  const origJson = JSON.stringify(project, null, 2);
+
+  // update the third digit
+  const verInfo = project.version.split('.');
+  verInfo[2] = patch;
+  project.version = verInfo.join('.');
+
+  // write the file if it's changed
+  const newJson = JSON.stringify(project, null, 2);
+  if (origJson !== newJson) {
+    console.log(`Writing project '${name}' version to '${project.version}' in '${location}'`);
+    writeFileSync(`${location}/package.json`, newJson)
+  }
+
+  count--;
+  if (count === 0) {
+    // last one!
+    // call sync-versions
+    require('./sync-versions');
+  }
+}
+
+if (process.argv[2] === '--reset') {
+  forEachProject((name, location, project) => {
+    updateVersion(name, project, location, 0);
+  })
+} else {
+
+  // Sets the patch version on each package.json in the project.
+  forEachProject((name, location, project) => {
+    if (!process.argv[2] || process.argv[2] === name) {
+      exec(`git rev-list --parents HEAD --count --full-history ..`, { cwd: location }, (o, stdout) => {
+        const patch = (parseInt(stdout.trim()) + (Number(project.patchOffset) || -1));
+        updateVersion(name, project, location, patch);
+      });
+    }
+  });
+}
+

ce/.scripts/sync-versions.js

@@ -0,0 +1,133 @@
+const { readFileSync, writeFileSync } = require('fs');
+const { read } = require('./for-each');
+
+const packageList = {};
+const rush = read(`${__dirname}/../rush.json`);
+const pjs = {};
+
+function writeIfChanged(filename, content) {
+  const orig = JSON.parse(readFileSync(filename))
+  const origJson = JSON.stringify(orig, null, 2);
+  const json = JSON.stringify(content, null, 2);
+
+  if (origJson !== json) {
+    console.log(`Writing updated file '${filename}'`)
+    writeFileSync(filename, json)
+    return true;
+  }
+  return false;
+}
+
+function versionToInt(ver) {
+  let v = ver.replace(/[^\d\.]/g, '').split('.').slice(0, 3);
+  while (v.length < 3) {
+    v.unshift(0);
+  }
+  let n = 0;
+  for (let i = 0; i < v.length; i++) {
+    n = n + ((2 ** (i * 16)) * parseInt(v[v.length - 1 - i]))
+  }
+  return n;
+}
+
+function setPeerDependencies(dependencies) {
+  for (const dep in dependencies) {
+    const ref = pjs[dep];
+    if (ref) {
+      if (dependencies[dep] !== `~${ref.version}`) {
+        console.log(`updating peer depedency ${dep} to ~${ref.version}`);
+        dependencies[dep] = `~${ref.version}`;
+      }
+    }
+  }
+}
+
+
+function recordDeps(dependencies) {
+  for (const packageName in dependencies) {
+    const packageVersion = dependencies[packageName];
+    if (packageList[packageName]) {
+      // same version?
+      if (packageList[packageName] === packageVersion) {
+        continue;
+      }
+      console.log(`${packageName} has ['${packageList[packageName]}','${packageVersion}']`);
+
+      // pick the higher one
+      const v = versionToInt(packageVersion);
+
+      if (v === 0) {
+        console.error(`Unparsed version ${packageName}:${packageVersion}`);
+        process.exit(1);
+      }
+      const v2 = versionToInt(packageList[packageName]);
+      if (v > v2) {
+
+        packageList[packageName] = packageVersion;
+      }
+    } else {
+      packageList[packageName] = packageVersion;
+    }
+  }
+}
+function fixDeps(pj, dependencies) {
+  for (const packageName in dependencies) {
+    if (dependencies[packageName] !== packageList[packageName]) {
+      console.log(`updating ${pj}:${packageName} from '${dependencies[packageName]}' to '${packageList[packageName]}'`)
+      dependencies[packageName] = packageList[packageName];
+    }
+  }
+}
+
+// load all the projects
+for (const each of rush.projects) {
+  const packageName = each.packageName;
+  const projectFolder = each.projectFolder;
+  pjs[packageName] = JSON.parse(readFileSync(`${__dirname}/../${projectFolder}/package.json`));
+}
+
+// verify that peer dependencies are the same version as they are building.
+for (const pj of Object.getOwnPropertyNames(pjs)) {
+  const each = pjs[pj];
+  setPeerDependencies(each.dependencies);
+  setPeerDependencies(each.devDependencies);
+  if (each['static-link']) {
+    setPeerDependencies(each['static-link'].dependencies);
+  }
+}
+
+// now compare to see if someone has an external package with different version
+// than everyone else.
+for (const pj of Object.getOwnPropertyNames(pjs)) {
+  const each = pjs[pj];
+  recordDeps(each.dependencies);
+  recordDeps(each.devDependencies);
+  if (each['static-link']) {
+    recordDeps(each['static-link'].dependencies);
+  }
+}
+
+for (const pj of Object.getOwnPropertyNames(pjs)) {
+  const each = pjs[pj];
+  fixDeps(pj, each.dependencies);
+  fixDeps(pj, each.devDependencies);
+  if (each['static-link']) {
+    fixDeps(pj, each['static-link'].dependencies);
+  }
+}
+var changed = 0;
+
+// write out the results.
+for (const each of rush.projects) {
+  const packageName = each.packageName;
+  const projectFolder = each.projectFolder;
+  if (writeIfChanged(`${__dirname}/../${projectFolder}/package.json`, pjs[packageName])) {
+    changed++;
+  }
+}
+
+if (changed) {
+  console.log(`Updated ${changed} files.`);
+} else {
+  console.log('No changes made')
+}

ce/.scripts/watch.js

@@ -0,0 +1,33 @@
+var cp = require('child_process');
+
+require('./for-each').forEachProject((packageName, projectFolder, project) => {
+  if (project.scripts && project.scripts.watch) {
+    // NOTE: We deliberately use `tsc --watch --project ${projectFolder}` here
+    // with cwd at the repo root instead of `npm run watch` with cwd at project
+    // folder. This ensures that error messages put source file paths relative
+    // to the repo root, which then allows VS Code to navigate to error
+    // locations correctly.
+    const tsc = `${projectFolder}/node_modules/.bin/tsc`;
+    const args = ['--watch', '--project', projectFolder];
+    console.log(`${tsc} ${args.join(' ')}`);
+
+    const proc = cp.spawn(tsc, args, { cwd: `${__dirname}/../`, shell: true, stdio: "inherit" });
+    proc.on("error", (c, s) => {
+      console.log(packageName);
+      console.error(c);
+      console.error(s);
+    });
+    proc.on('exit', (c, s) => {
+      console.log(packageName);
+      console.error(c);
+      console.error(s);
+    });
+    proc.on('message', (c, s) => {
+      console.log(packageName);
+      console.error(c);
+      console.error(s);
+    })
+  }
+});
+
+

ce/assets/LICENSE.txt

@@ -0,0 +1,216 @@
+MICROSOFT PRE-RELEASE SOFTWARE LICENSE TERMS
+
+MICROSOFT VCPKG-CE PROJECT
+
+These license terms are an agreement between Microsoft Corporation (or based on
+where you live, one of its affiliates) and you. They apply to the pre-release
+software named above. The terms also apply to any Microsoft services or updates
+for the software, except to the extent those have additional terms.
+
+IF YOU COMPLY WITH THESE LICENSE TERMS, YOU HAVE THE RIGHTS BELOW.
+
+1.  INSTALLATION AND USE RIGHTS.
+
+    a. General. You may install and use any number of copies of the software.
+
+    b. Third Party Components. The software may include third party components
+       with separate legal notices or governed by other agreements, as may be
+       described in the “…Notice” file(s) accompanying the software.
+
+2.  PRE-RELEASE SOFTWARE. This software is a pre-release version. It may not
+    operate correctly or work the way a final version will. Microsoft may change
+    it for the final, commercial version. Microsoft is not obligated to provide
+    maintenance, technical support or updates to you for the software.
+
+3.  FEEDBACK. If you give feedback about the software to Microsoft, you give to
+    Microsoft, without charge, the right to use, share and commercialize your
+    feedback in any way and for any purpose. You will not give feedback that is
+    subject to a license that requires Microsoft to license its software or
+    documentation to third parties because we include your feedback in them.
+    These rights survive this agreement.
+
+4.  DATA.
+
+    a. Data Collection. The software may collect information about you and your
+       use of the software, and send that to Microsoft. Microsoft may use this
+       information to provide services and improve our products and services.
+       You may opt-out of many of these scenarios, but not all, as described in
+       the product documentation. There are also some features in the software
+       that may enable you and Microsoft to collect data from users of your
+       applications. If you use these features, you must comply with applicable
+       law, including providing appropriate notices to users of your
+       applications together with a copy of Microsoft’s privacy statement. Our
+       privacy statement is located at
+       https://go.microsoft.com/fwlink/?LinkID=824704. You can learn more about
+       data collection and use in the help documentation and our privacy
+       statement. Your use of the software operates as your consent to these
+       practices.
+
+    b. Processing of Personal Data. To the extent Microsoft is a processor or
+       subprocessor of personal data in connection with the software, Microsoft
+       makes the commitments in the European Union General Data Protection
+       Regulation Terms of the Online Services Terms to all customers effective
+       May 25, 2018, at https://docs.microsoft.com/en-us/legal/gdpr.
+
+5.  TIME-SENSITIVE SOFTWARE.
+
+    a) Period. This agreement is effective on your acceptance and terminates on
+       the earlier of (i) 30 days following first availability of a commercial
+       release of the software or (ii) upon termination by Microsoft. Microsoft
+       may extend this agreement in its discretion.
+
+    b) Notice. You may receive periodic reminder notices of this date through
+       the software.
+
+    c) Access to data. You may not be able to access data used in the software
+       when it stops running.
+
+6.  FEEDBACK. If you give feedback about the software to Microsoft, you give to
+    Microsoft, without charge, the right to use, share and commercialize your
+    feedback in any way and for any purpose. You will not give feedback that is
+    subject to a license that requires Microsoft to license its software or
+    documentation to third parties because we include your feedback in them.
+    These rights survive this agreement.
+
+7.  SCOPE OF LICENSE. The software is licensed, not sold. This agreement only
+    gives you some rights to use the software. Microsoft reserves all other
+    rights. Unless applicable law gives you more rights despite this limitation,
+    you may use the software only as expressly permitted in this agreement. In
+    doing so, you must comply with any technical limitations in the software
+    that only allow you to use it in certain ways. You may not
+
+    - work around any technical limitations in the software;
+
+    - reverse engineer, decompile or disassemble the software, or otherwise
+      attempt to derive the source code for the software, except and to the
+      extent required by third party licensing terms governing use of certain
+      open-source components that may be included with the software;
+
+    - remove, minimize, block or modify any notices of Microsoft or its
+      suppliers in the software;
+
+    - use the software in any way that is against the law;
+
+    - share, publish, rent, or lease the software; or,
+
+    - provide the software as a stand-alone offering or combined with any of
+      your applications for others to use, or transfer the software or this
+      agreement to any third party.
+
+8.  EXPORT RESTRICTIONS. You must comply with all domestic and international
+    export laws and regulations that apply to the software, which include
+    restrictions on destinations, end users and end use. For further information
+    on export restrictions, visit www.microsoft.com/exporting.
+
+9.  SUPPORT SERVICES. Because this software is “as is,” we may not provide
+    support services for it.
+
+10. ENTIRE AGREEMENT. This agreement, and the terms for supplements, updates,
+    Internet-based services and support services that you use, are the entire
+    agreement for the software and support services.
+
+11. APPLICABLE LAW. If you acquired the software in the United States,
+    Washington law applies to interpretation of and claims for breach of this
+    agreement, and the laws of the state where you live apply to all other
+    claims. If you acquired the software in any other country, its laws apply.
+
+12. CONSUMER RIGHTS; REGIONAL VARIATIONS. This agreement describes certain legal
+    rights. You may have other rights, including consumer rights, under the laws
+    of your state or country. Separate and apart from your relationship with
+    Microsoft, you may also have rights with respect to the party from which you
+    acquired the software. This agreement does not change those other rights if
+    the laws of your state or country do not permit it to do so. For example, if
+    you acquired the software in one of the below regions, or mandatory country
+    law applies, then the following provisions apply to you:
+
+    a. Australia. You have statutory guarantees under the Australian Consumer
+       Law and nothing in this agreement is intended to affect those rights.
+
+    b. Canada. If you acquired this software in Canada, you may stop receiving
+       updates by turning off the automatic update feature, disconnecting your
+       device from the Internet (if and when you re-connect to the Internet,
+       however, the software will resume checking for and installing updates),
+       or uninstalling the software. The product documentation, if any, may also
+       specify how to turn off updates for your specific device or software.
+
+    c. Germany and Austria.
+
+        (i) Warranty. The properly licensed software will perform substantially
+            as described in any Microsoft materials that accompany the software.
+            However, Microsoft gives no contractual guarantee in relation to the
+            licensed software.
+
+        (ii) Limitation of Liability. In case of intentional conduct, gross
+            negligence, claims based on the Product Liability Act, as well as,
+            in case of death or personal or physical injury, Microsoft is liable
+            according to the statutory law.
+
+        Subject to the foregoing clause (ii), Microsoft will only be liable for
+        slight negligence if Microsoft is in breach of such material contractual
+        obligations, the fulfillment of which facilitate the due performance of
+        this agreement, the breach of which would endanger the purpose of this
+        agreement and the compliance with which a party may constantly trust in
+        (so-called "cardinal obligations"). In other cases of slight negligence,
+        Microsoft will not be liable for slight negligence.
+
+13. DISCLAIMER OF WARRANTY. THE SOFTWARE IS LICENSED “AS-IS.” YOU BEAR THE RISK
+    OF USING IT. MICROSOFT GIVES NO EXPRESS WARRANTIES, GUARANTEES OR
+    CONDITIONS. TO THE EXTENT PERMITTED UNDER YOUR LOCAL LAWS, MICROSOFT
+    XCLUDES THE IMPLIED WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR
+    PURPOSE AND NON-INFRINGEMENT.
+
+14. LIMITATION ON AND EXCLUSION OF DAMAGES. YOU CAN RECOVER FROM MICROSOFT AND
+    ITS SUPPLIERS ONLY DIRECT DAMAGES UP TO U.S. $5.00. YOU CANNOT RECOVER ANY
+    OTHER DAMAGES, INCLUDING CONSEQUENTIAL, LOST PROFITS, SPECIAL, INDIRECT OR
+    INCIDENTAL DAMAGES.
+
+    This limitation applies to (a) anything related to the software, services,
+    content (including code) on third party Internet sites, or third party
+    applications; and (b) claims for breach of contract, breach of warranty,
+    guarantee or condition, strict liability, negligence, or other tort to the
+    extent permitted by applicable law.
+
+    It also applies even if Microsoft knew or should have known about the
+    possibility of the damages. The above limitation or exclusion may not apply
+    to you because your country may not allow the exclusion or limitation of
+    incidental, consequential or other damages.
+
+Please note: As this software is distributed in Quebec, Canada, some of the
+clauses in this agreement are provided below in French.
+
+Remarque : Ce logiciel étant distribué au Québec, Canada, certaines des clauses
+dans ce contrat sont fournies ci-dessous en français.
+
+EXONÉRATION DE GARANTIE. Le logiciel visé par une licence est offert
+« tel quel ». Toute utilisation de ce logiciel est à votre seule risque et
+péril. Microsoft n’accorde aucune autre garantie expresse. Vous pouvez
+bénéficier de droits additionnels en vertu du droit local sur la protection des
+consommateurs, que ce contrat ne peut modifier. La ou elles sont permises par le
+droit locale, les garanties implicites de qualité marchande, d’adéquation à un
+usage particulier et d’absence de contrefaçon sont exclues.
+
+LIMITATION DES DOMMAGES-INTÉRÊTS ET EXCLUSION DE RESPONSABILITÉ POUR LES
+DOMMAGES. Vous pouvez obtenir de Microsoft et de ses fournisseurs une
+indemnisation en cas de dommages directs uniquement à hauteur de 5,00 $ US. Vous
+ne pouvez prétendre à aucune indemnisation pour les autres dommages, y compris
+les dommages spéciaux, indirects ou accessoires et pertes de bénéfices.
+
+Cette limitation concerne:
+
+- tout ce qui est relié au logiciel, aux services ou au contenu (y compris le
+  code) figurant sur des sites Internet tiers ou dans des programmes tiers ; et
+
+- les réclamations au titre de violation de contrat ou de garantie, ou au titre
+  de responsabilité stricte, de négligence ou d’une autre faute dans la limite
+  autorisée par la loi en vigueur.
+
+Elle s’applique également, même si Microsoft connaissait ou devrait connaître
+l’éventualité d’un tel dommage. Si votre pays n’autorise pas l’exclusion ou la
+limitation de responsabilité pour les dommages indirects, accessoires ou de
+quelque nature que ce soit, il se peut que la limitation ou l’exclusion
+ci-dessus ne s’appliquera pas à votre égard.
+
+EFFET JURIDIQUE. Le présent contrat décrit certains droits juridiques. Vous
+pourriez avoir d’autres droits prévus par les lois de votre pays. Le présent
+contrat ne modifie pas les droits que vous confèrent les lois de votre pays si
+celles-ci ne le permettent pas.

ce/assets/NOTICE.txt

@@ -0,0 +1 @@
+

ce/assets/package.json

@@ -0,0 +1,39 @@
+{
+  "name": "vcpkg-ce",
+  "version": "0.7.0",
+  "description": "vcpkg-ce CLI",
+  "main": "ce/dist/main.js",
+  "bin": {
+    "ce_": "./ce/dist/main.js"
+  },
+  "directories": {
+    "doc": "docs"
+  },
+  "engines": {
+    "node": ">=14.17.0"
+  },
+  "scripts": {
+    "postinstall": "node ./create-links create && node ./wrapper-scripts create",
+    "uninstall": "node ./create-links remove && node ./wrapper-scripts remove",
+    "prepack": "npx rimraf ./common/temp/node_modules/.pnpm/typescript* ./common/temp/node_modules/.pnpm/translate-strings*  ./common/temp/node_modules/.pnpm/ts-morph*  ./common/temp/node_modules/.pnpm/@types* && node ./prepare-deploy.js"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/Microsoft/vcpkg-tool.git"
+  },
+  "files": [
+    "**/*"
+  ],
+  "keywords": [
+    "vcpkg-ce",
+    "vcpkg",
+    "ce"
+  ],
+  "author": "Microsoft",
+  "license": "MIT",
+  "bugs": {
+    "url": "https://github.com/Microsoft/vcpkg/issues"
+  },
+  "homepage": "https://github.com/Microsoft/vcpkg#readme",
+  "readme": "https://github.com/Microsoft/vcpkg/blob/master/readme.md"
+}

ce/assets/prepare-deploy.js

@@ -0,0 +1,10 @@
+const { unlinkSync, writeFileSync } = require('fs');
+const { join } = require('path');
+
+const file = join(__dirname, 'deploy-metadata.json');
+const metadata = require(file);
+
+const links = metadata.links;
+metadata.links = links.filter(link => link.kind !== 'folderLink' || (link.linkPath.indexOf('/@types') === -1 && link.targetPath.indexOf('/typescript@') === -1));
+writeFileSync(file, JSON.stringify(metadata, null, 2));
+unlinkSync(__filename);

ce/assets/scripts/ce

@@ -0,0 +1,481 @@
+#!/bin/sh
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+# wrapper script for ce.
+# this is intended to be dot-sourced and then you can use the ce() function.
+
+# DEBUGGING :
+# set | grep -i ^VCPKG
+
+# check to see if we've been dot-sourced (should work for most POSIX shells)
+sourced=0
+
+if [ -n "$ZSH_EVAL_CONTEXT" ]; then
+  case $ZSH_EVAL_CONTEXT in *:file) sourced=1;; esac
+elif [ -n "$KSH_VERSION" ]; then
+  [ "$(cd $(dirname -- $0) && pwd -P)/$(basename -- $0)" != "$(cd $(dirname -- ${.sh.file}) && pwd -P)/$(basename -- ${.sh.file})" ] && sourced=1
+elif [ -n "$BASH_VERSION" ]; then
+  (return 0 2>/dev/null) && sourced=1
+else # All other shells: examine $0 for known shell binary filenames
+  # Detects `sh` and `dash`; add additional shell filenames as needed.
+  case ${0##*/} in sh|dash) sourced=1;; esac
+fi
+
+if [ $sourced -eq 0 ]; then
+  echo 'This script is expected to be dot-sourced so that it may load ce into the'
+  echo 'current environment and not require permanent changes to the system when you activate.'
+  echo ''
+  echo "You should instead run '. $(basename $0)' first to import ce into the current session."
+  exit
+fi
+
+# GLOBALS
+VCPKG_NODE_LATEST=16.12.0
+VCPKG_NODE_REMOTE=https://nodejs.org/dist/
+VCPKG_PWD=`pwd`
+
+VCPKG_init() {
+  VCPKG_OS="$(uname | sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/')"
+  VCPKG_ARCH="$(uname -m | sed -e 's/x86_64/x64/;s/i86pc/x64/;s/i686/x86/;s/aarch64/arm64/')"
+
+  case $VCPKG_OS in
+    ( mingw64_nt* | msys_nt* ) VCPKG_OS="win";;
+    ( darwin*|*bsd*) VCPKG_IS_THIS_BSD=TRUE;;
+    ( aix ) VCPKG_ARCH="ppc64" ;;
+  esac
+
+  if [ ! -z "$VCPKG_IS_THIS_BSD" ]; then
+    VCPKG_START_TIME=$(date +%s)
+  else
+    VCPKG_START_TIME=$(($(date +%s%N)/1000000))
+  fi
+
+  # find important cmdline args
+  VCPKG_ARGS=()
+  for each in "$@"; do case $each in
+    --reset-ce) VCPKG_RESET=TRUE;;
+    --remove-ce) VCPKG_REMOVE=TRUE;;
+    --debug) VCPKG_DEBUG=TRUE && VCPKG_ARGS+=("$each");;
+    *) VCPKG_ARGS+=("$each");;
+  esac ;done
+}
+
+VCPKG_init "$@"
+shift $#
+
+# at this point, we're pretty sure we've been dot-sourced
+# Try to locate the $VCPKG_ROOT folder, where the ce is installed.
+if [ -n "$VCPKG_ROOT" ]; then
+  # specify it on the command line if you like, we'll export it
+  export VCPKG_ROOT=$VCPKG_ROOT
+else
+  # default is off the home folder
+  export VCPKG_ROOT=~/.vcpkg
+fi;
+
+CE=${VCPKG_ROOT}
+mkdir -p $CE
+
+VCPKG_DOWNLOADS=${CE}/downloads
+
+VCPKG_NODE=${CE}/downloads/bin/node
+VCPKG_NPM=${CE}/bin/npm
+
+VCPKG_debug() {
+  if [ ! -z "$VCPKG_IS_THIS_BSD" ]; then
+    local NOW=$(date +%s)
+    local OFFSET="$(( $NOW - $VCPKG_START_TIME )) sec"
+  else
+    local NOW=$(($(date +%s%N)/1000000))
+    local OFFSET="$(( $NOW - $VCPKG_START_TIME )) msec"
+  fi
+
+  if [ ! -z "$VCPKG_DEBUG" ]; then
+    if [ -n "$VCPKG_NESTED" ]; then
+      echo "[NESTED $OFFSET] $*"
+    else
+      echo "[$OFFSET] $*"
+    fi
+  fi
+
+  if [ -n "$VCPKG_NESTED" ]; then
+    echo "[NESTED $OFFSET] $*" >> $VCPKG_ROOT/log.txt
+  else
+    echo "[$OFFSET] $*" >> $VCPKG_ROOT/log.txt
+  fi
+}
+
+if [ ! -z "$VCPKG_RESET" ]; then
+  if [ -d $CE/node_modules ]; then
+    echo Forcing reinstall of vcpkg-ce package
+    rm -rf $CE/node_modules
+  fi
+
+  if [ -d $CE/bin ]; then
+    rm -rf $CE/bin
+  fi
+
+  if [ -d $CE/lib ]; then
+    rm -rf $CE/lib
+  fi
+fi
+
+if [ ! -z "$VCPKG_REMOVE" ]; then
+  if [ -d $CE/node_modules ]; then
+    echo Removing vcpkg-ce package
+    rm -rf $CE/node_modules
+  fi
+
+  if [ -d $CE/bin ]; then
+    rm -rf $CE/bin
+  fi
+
+  if [ -d $CE/lib ]; then
+    rm -rf $CE/lib
+  fi
+
+  if [ -f $VCPKG_ROOT/ce.ps1 ]; then
+    rm -f $VCPKG_ROOT/ce.ps1
+  fi
+
+  if [ -f $VCPKG_ROOT/ce ]; then
+    rm -f $VCPKG_ROOT/ce
+  fi
+
+  if [ -f $VCPKG_ROOT/NOTICE.txt ]; then
+    rm -f $VCPKG_ROOT/NOTICE.txt
+  fi
+
+  if [ -f $VCPKG_ROOT/LICENSE.txt ]; then
+    rm -f $VCPKG_ROOT/LICENSE.txt
+  fi
+
+  # cleanup environment.
+  . <(set | grep -i ^VCPKG | sed -e 's/[=| |\W].*//;s/^/unset /')
+
+  # remove functions (zsh)
+  which functions  > /dev/null 2>&1  && . <(functions | grep -i ^vcpkg | sed -e 's/[=| |\W].*//;s/^/unset -f /')
+  return
+fi
+
+VCPKG_cleanup() {
+  # clear things that we're not going to need for the long term
+  unset VCPKG_NODE_LATEST
+  unset VCPKG_NODE_REMOTE
+  unset VCPKG_PWD
+  unset VCPKG_NPM
+  unset VCPKG_NESTED
+  unset VCPKG_OS
+  unset VCPKG_ARCH
+  unset VCPKG_IS_THIS_BSD
+  unset VCPKG_DEBUG
+  unset -f VCPKG_bootstrap_node > /dev/null 2>&1
+  unset -f VCPKG_bootstrap_ce > /dev/null 2>&1
+  unset VCPKG_REMOVE
+  unset VCPKG_RESET
+  unset VCPKG_START_TIME
+  unset VCPKG_ARGS
+  if [ -f "${Z_VCPKG_POSTSCRIPT}" ]; then
+    command rm "${Z_VCPKG_POSTSCRIPT}"
+  fi
+	unset Z_VCPKG_POSTSCRIPT
+}
+VCPKG_verify_node() {
+  # $1 should be the folder to check
+  local NODE_EXE="node"
+  if [ "${VCPKG_OS}" = "win" ]; then
+    NODE_EXE="node.exe"
+  fi
+  
+  local N=$(which $1/$NODE_EXE)
+  
+    if [ ! -z "$N" ]; then
+      if [ -f $N ]; then
+        if [ $($N -e "[major, minor, patch ] = process.versions.node.split('.'); console.log( !!(major>16 || major == 16 & minor >= 12) )") = "true" ]; then
+          VCPKG_NODE=$N
+          VCPKG_NPM=$(which $1/npm)
+          VCPKG_debug using node in $1
+          return 0
+        fi
+      fi
+    fi
+    return 1;
+}
+
+VCPKG_find_node() {
+  local NODES=$(find $1 | grep -i /bin/node)
+  
+  for each in $NODES; do 
+    local d=$(dirname "$each")
+    VCPKG_verify_node $d
+    if [ $? -eq 0 ]; then
+      return 0;
+    fi
+  done
+  return 1;
+}
+
+VCPKG_bootstrap_node() {
+  VCPKG_debug starting VCPKG_bootstrap_node
+
+  # did we put one in downloads at some point?
+  VCPKG_find_node $VCPKG_DOWNLOADS
+  if [ $? -eq 0 ]; then
+    return 0;
+  fi
+
+  # is there one on the path?
+  VCPKG_find_node $(dirname $(which node))
+  if [ $? -eq 0 ]; then
+    return 0;
+  fi
+
+  local NODE_EXE="node"
+  if [ "${VCPKG_OS}" = "win" ]; then
+    NODE_EXE="node.exe"
+  fi
+
+  # we don't seem to have a suitable nodejs on the path
+  # let's grab a well-known one, cache it and use it.
+
+  local VCPKG_ARCHIVE_EXT=".tar.gz"
+	local TAR_FLAGS="-zxvf"
+	if [ "${VCPKG_OS}" = "win" ]; then
+	  VCPKG_ARCHIVE_EXT=".zip"
+  fi
+
+  local NODE_FULLNAME="node-v${VCPKG_NODE_LATEST}-${VCPKG_OS}-${VCPKG_ARCH}"
+  local NODE_URI="${VCPKG_NODE_REMOTE}v${VCPKG_NODE_LATEST}/${NODE_FULLNAME}${VCPKG_ARCHIVE_EXT}"
+  local VCPKG_ARCHIVE="${VCPKG_DOWNLOADS}/${NODE_FULLNAME}${VCPKG_ARCHIVE_EXT}"
+
+  if [ ! -d "${VCPKG_DOWNLOADS}" ]; then
+    command mkdir -p "${VCPKG_DOWNLOADS}"
+  fi
+
+  echo "Downloading node from ${NODE_URI} to ${VCPKG_ARCHIVE}"
+  if type noglob > /dev/null 2>&1; then
+    noglob curl -L -# "${NODE_URI}" -o "${VCPKG_ARCHIVE}"
+  else
+    curl -L -# "${NODE_URI}" -o "${VCPKG_ARCHIVE}"
+  fi
+
+   if [ ! -f "${VCPKG_ARCHIVE}" ]; then
+    echo "Failed to download node binary."
+    return 1
+  fi
+
+  # UNPACK IT
+  if [ "${VCPKG_OS}" = "aix" ]; then
+    gunzip "${VCPKG_ARCHIVE}" | tar -xvC "${VCPKG_DOWNLOADS}" "${NODE_FULLNAME}/bin/${NODE_EXE}" >> $VCPKG_ROOT/log.txt 2>&1
+  else
+    tar $TAR_FLAGS "${VCPKG_ARCHIVE}" -C "${VCPKG_DOWNLOADS}" >> $VCPKG_ROOT/log.txt 2>&1
+  fi
+  
+  # OK, we good?
+  VCPKG_find_node $VCPKG_DOWNLOADS
+  if [ $? -eq 0 ]; then
+    return 0;
+  fi
+
+  if [ ! -f $VCPKG_NPM ]; then
+    echo "ERROR! Unable to find/get npm"
+    return 1;
+  fi
+
+  VCPKG_debug installed node in ce
+  return 0
+}
+
+VCPKG_SCRIPT=${CE}/node_modules/.bin/ce
+VCPKG_MAIN=${CE}/node_modules/vcpkg-ce
+
+VCPKG_bootstrap_ce() {
+  VCPKG_debug checking for installed ce $VCPKG_SCRIPT
+
+  if [ -f $VCPKG_SCRIPT ]; then
+    VCPKG_debug ce is installed.
+    return 0
+  fi
+
+  # it's not there!
+  # let's install it where we want it
+
+  # ensure we have a node_modules here, so npm won't search for one up the tree.
+  command mkdir -p $CE/node_modules
+
+  echo Installing vcpkg-ce in $VCPKG_ROOT
+  unset VCPKG_RESET
+
+  cd $CE
+  $VCPKG_NODE $VCPKG_NPM cache clean --force >> $VCPKG_ROOT/log.txt 2>&1
+  local OLD_PATH=$PATH
+  PATH=`dirname $VCPKG_NODE`:$PATH
+  if [ ! -z "$USE_LOCAL_VCPKG_PKG" ]; then
+    echo USING LOCAL CE PACKAGE $USE_LOCAL_VCPKG_PKG
+
+    $VCPKG_NODE $VCPKG_NPM --force install --no-save --no-lockfile --scripts-prepend-node-path=true $USE_LOCAL_VCPKG_PKG  >> $VCPKG_ROOT/log.txt 2>&1
+  else
+    $VCPKG_NODE $VCPKG_NPM --force install --no-save --no-lockfile --scripts-prepend-node-path=true https://aka.ms/vcpkg-ce.tgz  >> $VCPKG_ROOT/log.txt 2>&1
+  fi
+
+  PATH=$OLD_PATH
+# go back where we were
+  cd $VCPKG_PWD
+
+  cp $CE/node_modules/.bin/ce* $VCPKG_ROOT/
+
+  # Copy the NOTICE and LICENSE files to $VCPKG_ROOT to improve discoverability.
+  cp $CE/node_modules/vcpkg-ce/NOTICE.txt $VCPKG_ROOT/
+  cp $CE/node_modules/vcpkg-ce/LICENSE.txt $VCPKG_ROOT/
+
+  if [  ! -f $VCPKG_SCRIPT ]; then
+    echo "ERROR! Unable to find/get ce script command $VCPKG_SCRIPT"
+    return 1;
+  fi
+
+  VCPKG_debug ce is installed
+  return 0;
+}
+
+# first, let's make sure we have a good copy of node
+VCPKG_bootstrap_node
+
+if [ $? -eq 1 ]; then
+  VCPKG_debug failed to acquire node.js
+  VCPKG_cleanup
+  return 1;
+fi
+
+VCPKG_bootstrap_ce
+
+# is ce installed?
+if [ $? -eq 1 ]; then
+  VCPKG_debug failed to bootstrap ce
+  VCPKG_cleanup
+  return 1;
+fi
+
+if [ -z $VCPKG_NESTED ]; then
+  VCPKG_debug executing final script: $VCPKG_SCRIPT
+  VCPKG_NESTED=TRUE
+  . $VCPKG_SCRIPT $VCPKG_ARGS
+  return # let the real script take over from here.
+fi
+
+# So, we're the real script then.
+VCPKG_debug 'real ce adding function'
+
+ce() {
+  # set | grep -i ^VCPKG
+
+  local cst=$VCPKG_START_TIME
+
+  VCPKG_init "$@"
+
+  if [ ! -z "$VCPKG_RESET" ]; then
+    if [ -d $CE/node_modules ]; then
+      echo Forcing reinstall of vcpkg-ce package
+      rm -rf $CE/node_modules
+    fi
+
+    if [ -d $CE/bin ]; then
+      rm -rf $CE/bin
+    fi
+
+    if [ -d $CE/lib ]; then
+      rm -rf $CE/lib
+    fi
+    unset VCPKG_RESET
+
+    if [ ! -z "$USE_LOCAL_VCPKG_SCRIPT" ]; then
+      echo USING LOCAL CE SCRIPT $USE_LOCAL_VCPKG_SCRIPT
+      . <(cat $USE_LOCAL_VCPKG_SCRIPT) "${VCPKG_ARGS[@]}"
+    else
+      . <(curl -L -# aka.ms/install-ce.sh) "${VCPKG_ARGS[@]}"
+    fi
+
+    return 0
+  fi
+
+  if [ ! -z "$VCPKG_REMOVE" ]; then
+    if [ -d $CE/node_modules ]; then
+      unset VCPKG_REMOVE
+      echo Removing vcpkg-ce package
+      rm -rf $CE/node_modules
+    fi
+
+    if [ -d $CE/bin ]; then
+      rm -rf $CE/bin
+    fi
+
+    if [ -d $CE/lib ]; then
+      rm -rf $CE/lib
+    fi
+
+    if [ -f $VCPKG_ROOT/ce.ps1 ]; then
+      rm -f $VCPKG_ROOT/ce.ps1
+    fi
+
+    if [ -f $VCPKG_ROOT/ce ]; then
+      rm -f $VCPKG_ROOT/ce
+    fi
+
+    if [ -f $VCPKG_ROOT/NOTICE.txt ]; then
+      rm -f $VCPKG_ROOT/NOTICE.txt
+    fi
+
+    if [ -f $VCPKG_ROOT/LICENSE.txt ]; then
+      rm -f $VCPKG_ROOT/LICENSE.txt
+    fi
+
+    # cleanup environment
+    . <(set | grep -i ^VCPKG | sed -e 's/[=| |\W].*//;s/^/unset /')
+
+    # remove functions (zsh)
+    which functions  > /dev/null 2>&1  && . <(functions | grep -i ^vcpkg | sed -e 's/[=| |\W].*//;s/^/unset -f /')
+    return 0
+  fi
+
+  if [ ! -f $VCPKG_NODE ]; then
+    echo The installation of nodejs $VCPKG_NODE that ce is using is missing
+    echo You may need to reacquire ce with '. <(curl aka.ms/install-ce.sh -L)'
+    echo or fix your nodejs installation.
+  fi
+
+  if [ ! -d $VCPKG_MAIN ]; then
+    echo The installation of ce is corrupted. $VCPKG_MAIN
+    echo You may need to reacquire ce with '. <(curl aka.ms/install-ce.sh -L)'
+  fi
+
+  # set the response file
+	# Generate 32 bits of randomness, to avoid clashing with concurrent executions.
+	export Z_VCPKG_POSTSCRIPT="${VCPKG_ROOT}/VCPKG_tmp_$(dd if=/dev/urandom count=1 2> /dev/null | cksum | cut -f1 -d" ").sh"
+
+  # call ce.js
+  # it picks up the Z_VCPKG_POSTSCRIPT environment variable to know where to dump the postscript
+  $VCPKG_NODE --harmony $VCPKG_MAIN ${VCPKG_ARGS[@]}
+
+  VCPKG_debug called ce.js
+  # modify the environment
+
+  # Call the post-invocation script if it is present, then delete it.
+	# This allows the invocation to potentially modify the caller's environment (e.g. PATH)
+	if [ -f "${Z_VCPKG_POSTSCRIPT}" ]; then
+		. "${Z_VCPKG_POSTSCRIPT}"
+		command rm "${Z_VCPKG_POSTSCRIPT}"
+		unset Z_VCPKG_POSTSCRIPT
+	fi
+
+  VCPKG_cleanup
+
+  VCPKG_START_TIME=$cst
+}
+
+# did they dotsource and have args go ahead and run it then!
+if [ -n "$VCPKG_ARGS" ]; then
+  ce "${VCPKG_ARGS[@]}"
+fi
+
+VCPKG_cleanup

ce/assets/scripts/ce.ps1

@@ -0,0 +1,452 @@
+@(echo off) > $null
+if #ftw NEQ '' goto :init
+($true){ $Error.clear(); }
+
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+# wrapper script for ce.
+# this is intended to be dot-sourced and then you can use the ce() function
+
+# unpack arguments if they came from CMD
+$hash=@{};
+get-item env:argz* |% { $hash[$_.name] = $_.value }
+if ($hash.count -gt 0) {
+  $args=for ($i=0; $i -lt $hash.count;$i++) { $hash["ARGZ[$i]"] }
+}
+# force the array to be an arraylist since we like to mutate it.
+$args=[System.Collections.ArrayList][System.Array]$args
+
+# GLOBALS
+$VCPKG_NODE_LATEST='16.12.0'
+$VCPKG_NODE_REMOTE='https://nodejs.org/dist/'
+$VCPKG_START_TIME=get-date
+
+function resolve([string]$name) {
+  $name = Resolve-Path $name -ErrorAction 0 -ErrorVariable _err
+  if (-not($name)) { return $_err[0].TargetObject }
+  $Error.clear()
+  return $name
+}
+
+$SCRIPT:DEBUG=( $args.indexOf('--debug') -gt -1 )
+
+function ce-debug() {
+  $t = [int32]((get-date).Subtract(($VCPKG_START_TIME)).ticks/10000)
+  if($SCRIPT:DEBUG) {
+    write-host -fore green "[$t msec] " -nonewline
+    write-host -fore gray $args
+  }
+  write-output "[$t msec] $args" >> $VCPKG_ROOT/log.txt
+}
+
+function download($url, $path) {
+  $wc = New-Object net.webclient
+
+  if( test-path -ea 0 $path) {
+    # check to see if the size is a match before downloading
+    $s = $wc.OpenRead($url)
+    $len = $wc.ResponseHeaders['Content-Length']
+    $s.Dispose()
+    if( (get-item $path).Length -eq $len ){
+      $wc.Dispose();
+      ce-debug "skipping download of '$url' - '$path' is ok."
+      return $path;
+    }
+  }
+  ce-debug "Downloading '$url' -> '$path'"
+  $wc.DownloadFile($url, $path);
+  $wc.Dispose();
+  if( (get-item $path).Length -ne $wc.ResponseHeaders['Content-Length'] ) {
+    throw "Download of '$url' failed.  Check your internet connection."
+  }
+  ce-debug "Completed Download of $url"
+  return $path
+}
+
+# set the home path.
+if( $ENV:VCPKG_ROOT ) {
+  $SCRIPT:VCPKG_ROOT=(resolve $ENV:VCPKG_ROOT)
+  $ENV:VCPKG_ROOT=$VCPKG_ROOT
+} else {
+  $SCRIPT:VCPKG_ROOT=(resolve "$HOME/.vcpkg")
+  $ENV:VCPKG_ROOT=$VCPKG_ROOT
+}
+
+# set the download path
+if( $ENV:VCPKG_DOWNLOADS ) {
+  $SCRIPT:VCPKG_DOWNLOADS= (resolve $ENV:VCPKG_DOWNLOADS)
+  $ENV:VCPKG_DOWNLOADS=$VCPKG_DOWNLOADS
+} else {
+  $SCRIPT:VCPKG_DOWNLOADS= (resolve "$VCPKG_ROOT/downloads")
+  $ENV:VCPKG_DOWNLOADS=$VCPKG_DOWNLOADS
+}
+
+$CE = "${VCPKG_ROOT}"
+$MODULES= "$CE/node_modules"
+
+$SCRIPT:VCPKG_SCRIPT=(resolve $MODULES/.bin/ce.ps1)
+$SCRIPT:CE_MODULE=(resolve $MODULES/vcpkg-ce )
+
+$reset = $args.IndexOf('--reset-ce') -gt -1
+$remove = $args.IndexOf('--remove-ce') -gt -1
+
+if( $reset -or -$remove ) {
+  $args.remove('--reset-ce');
+  $args.remove('--remove-ce');
+
+  if( $reset ) {
+    write-host "Resetting vcpkg-ce"
+  }
+
+  remove-item -recurse -force -ea 0 "$MODULES/.bin","$MODULES"
+  remove-item -force -ea 0 "${VCPKG_ROOT}/ce.ps1","${VCPKG_ROOT}/ce.cmd","${VCPKG_ROOT}/ce","${VCPKG_ROOT}/NOTICE.txt","${VCPKG_ROOT}/LICENSE.txt"
+  $error.clear();
+
+  if( $remove ) {
+    write-host "Removing vcpkg-ce"
+    exit
+  }
+}
+
+function verify-node($NODE) {
+  if( $NODE -and (get-command -ea 0 $NODE) -and ( (& $NODE -p "/(^\d*\.\d*)/g.exec( process.versions.node)[0]") -ge 16.12 ) ) {
+    # it's a good version of node, let's set the variables
+    $SCRIPT:VCPKG_NODE=$NODE
+    $error.clear();
+    return $TRUE;
+  }
+  $error.clear();
+  return $FALSE
+}
+
+function find-node() {
+  $PLACES= @($VCPKG_DOWNLOADS,"$ENV:LOCALAPPDATA/vcpkg/downloads/tools")
+  for( $i=0; $i -lt $PLACES.count; $i++ ) {
+    $p = $PLACES[$i]
+    if( $p ) {
+      $NODES= @()+((get-childitem -ea 0 $p -recurse |? {$_.name -in @('node.exe', 'node')}).FullName)
+      for( $j=0; $j -lt $NODES.count; $j++ ) {
+        $NODE=$NODES[$j]
+        if( verify-node $NODE ) {
+          return $NODE
+        }
+      }
+    }
+  }
+}
+
+function bootstrap-node {
+  # if we have a custom ce node let's use that first
+  $NODE=find-node
+  if( $NODE ) {
+    ce-debug "Node: $NODE"
+    return $TRUE;
+  }
+
+  # check the node on the path.
+  if( (verify-node ((get-command node -ea 0).source ))) {
+    ce-debug "Node: ${VCPKG_NODE}"
+    return $TRUE;
+  }
+
+  # not there, or not good enough
+  if((($PSVersionTable.OS -match "windows") -or ($PSVersionTable.PSEdition -match 'desktop') ) ) {   # windows
+    $NODE_OS='win'
+    switch($ENV:PROCESSOR_ARCHITECTURE) {
+      'AMD64' { $NODE_ARCH='x64' }
+      'ARM64' { $NODE_ARCH='arm64' }
+      Default { $NODE_ARCH='x86' }
+    }
+    $NODE_ARCHIVE_EXT=".zip"
+  } else {
+    $NODE_OS=(uname | sed 'y/ABCDEFGHIJKLMNOPQRSTUVWXYZ/abcdefghijklmnopqrstuvwxyz/')
+    $NODE_ARCH=(uname -m | sed -e 's/x86_64/x64/;s/i86pc/x64/;s/i686/x86/;s/aarch64/arm64/')
+    if ( $NODE_OS -eq "aix" ) { $NODE_ARCH="ppc64" } #aix special
+    $NODE_ARCHIVE_EXT=".tar.gz"
+  }
+
+  $NODE_FULLNAME="node-v${VCPKG_NODE_LATEST}-${NODE_OS}-${NODE_ARCH}"
+  $NODE_URI="${VCPKG_NODE_REMOTE}v${VCPKG_NODE_LATEST}/${NODE_FULLNAME}${NODE_ARCHIVE_EXT}"
+  $NODE_FOLDER=resolve "${VCPKG_DOWNLOADS}/${NODE_FULLNAME}"
+  $NODE_ARCHIVE=resolve "$VCPKG_DOWNLOADS/${NODE_FULLNAME}${NODE_ARCHIVE_EXT}"
+
+  write-host "Installing node runtime"
+
+  $ProgressPreference = 'SilentlyContinue'
+  ce-debug "Downloading Node: ${NODE_URI}"
+  download $NODE_URI $NODE_ARCHIVE
+  $shh = new-item -type directory -ea 0 $NODE_FOLDER 
+  
+  switch($NODE_OS){
+    'win' {
+      if( get-command -ea 0 tar.exe ) {
+        tar "-xvf" "${NODE_ARCHIVE}" -C "${NODE_FOLDER}"  2>&1  > $null
+      } else {
+        $shh= expand-archive -path $NODE_ARCHIVE -destinationpath "$NODE_FOLDER"
+      }
+    }
+    'aix' {
+      $shh = gunzip "${NODE_ARCHIVE}" | tar -xvC "$NODE_FOLDER" "${NODE_FULLNAME}/bin/"
+    }
+    default {
+      $shh = tar "-zxvf" "${NODE_ARCHIVE}" -C "$NODE_FOLDER"
+    }
+  }
+
+  $NODE=find-node
+  if( $NODE ) {
+    ce-debug "Node: $NODE"
+    return $TRUE;
+  }
+
+  write-error 'Unable to resolve nodejs'
+  return $FALSE;
+}
+
+function bootstrap-vcpkg-ce {
+  if(test-path "${ce}/ce.ps1") {
+    if( test-path $VCPKG_SCRIPT ) {
+      return $TRUE;
+    }
+
+    ## if we're running from an installed module location, we'll keep that.
+    $MODULE=(resolve ${PSScriptRoot}/node_modules/vcpkg-ce )
+
+    if( test-path $MODULE ) {
+      $SCRIPT:CE_MODULE=$MODULE
+      return $TRUE
+    }
+  }
+
+  # cleanup the yarn cache.
+  ce-debug "Clearing YARN cache"
+  $shh = & $VCPKG_NODE $YARN cache clean --force 2>&1
+  $error.clear();
+
+  write-host "Installing vcpkg-ce to ${VCPKG_ROOT}"
+
+  if( $ENV:USE_LOCAL_VCPKG_PKG )  {
+    $USE_LOCAL_VCPKG_PKG=$ENV:USE_LOCAL_VCPKG_PKG 
+  }
+
+  $PKG = $USE_LOCAL_VCPKG_PKG
+  if( -not $PKG ) {
+    $PKG = 'https://aka.ms/vcpkg-ce.tgz'
+  }
+  pushd $CE
+
+  $PATH = $ENV:PATH
+  $N_DIR=(resolve "$VCPKG_NODE/..")
+  $ENV:PATH="$N_DIR;$PATH"
+  
+  &$VCPKG_NODE $YARN add $PKG --no-lockfile --force --scripts-prepend-node-path=true --modules-folder=$MODULES 2>&1 >> $VCPKG_ROOT/log.txt
+  $ENV:PATH = $PATH
+
+  remove-item -path $ce/package.json -ea 0
+
+  popd
+
+  ce-debug 'yarn finished.'
+  if( $error.count -gt 0 ) {
+    $error |% { add-content -encoding UTF8 $VCPKG_ROOT/log.txt $_ }
+    $Error.clear()
+  }
+
+  # we should also copy the .bin files into the $VCPKG_ROOT folder to make reactivation (without being on the PATH) easy
+  copy-item "$MODULES/.bin/ce*" $VCPKG_ROOT
+
+  # Copy the NOTICE and LICENSE files to $VCPKG_ROOT to improve discoverability.
+  copy-item "$CE_MODULE/NOTICE.txt","$CE_MODULE/LICENSE.txt" $VCPKG_ROOT
+
+  ce-debug "Bootstrapped vcpkg-ce: ${VCPKG_ROOT}"
+
+  if( -not (test-path $CE_MODULE )) {
+    write-error "ERROR! Unable to find/get vcpkg-ce module $CE_MODULE"
+    return $false;
+  }
+  return $true;
+}
+
+# ensure it's there.
+$shh = new-item -type directory $CE,$MODULES,"$CE/scripts",$VCPKG_DOWNLOADS -ea 0
+
+# grab the yarn cli script
+$SCRIPT:YARN = resolve "$CE/scripts/yarn.js"
+if( -not (test-path $SCRIPT:YARN )) {
+  $SCRIPT:YARN = download https://aka.ms/yarn.js $YARN
+}
+
+if( -not (bootstrap-node )) {
+  write-error "Unable to acquire an appropriate version of Node."
+  write-error "You should install the LTS version or greater of NodeJS"
+  throw "Installation Unsuccessful."
+}
+
+if( -not (bootstrap-vcpkg-ce )) {
+  write-error "Unable to install vcpkg-ce."
+  throw "Installation Unsuccessful."
+}
+
+# export vcpkg-ce to the current shell.
+$shh = New-Module -name vcpkg-ce -ArgumentList @($VCPKG_NODE,$CE_MODULE,$VCPKG_ROOT) -ScriptBlock {
+  param($VCPKG_NODE,$CE_MODULE,$VCPKG_ROOT)
+
+  function resolve([string]$name) {
+    $name = Resolve-Path $name -ErrorAction 0 -ErrorVariable _err
+    if (-not($name)) { return $_err[0].TargetObject }
+    $Error.clear()
+    return $name
+  }
+
+  function ce() {
+    if( ($args.indexOf('--remove-ce') -gt -1) -or ($args.indexOf('--reset-ce') -gt -1)) {
+      # we really want to do call the ps1 script to do this.
+      if( test-path "${VCPKG_ROOT}/ce.ps1" ) {
+        & "${VCPKG_ROOT}/ce.ps1" @args
+      }
+      return
+    }
+
+    if( -not (test-path $CE_MODULE )) {
+      write-error "vcpkg-ce is not installed."
+      write-host -nonewline "You can reinstall vcpkg-ce by running "
+      write-host -fore green "iex (iwr -useb aka.ms/install-ce.ps1)"
+      return
+    }
+
+    # setup the postscript file
+    # Generate 31 bits of randomness, to avoid clashing with concurrent executions.
+    $env:Z_VCPKG_POSTSCRIPT = resolve "${VCPKG_ROOT}/VCPKG_tmp_$(Get-Random -SetSeed $PID).ps1"
+
+    & $VCPKG_NODE --harmony $CE_MODULE @args
+
+    # dot-source the postscript file to modify the environment
+    if ($env:Z_VCPKG_POSTSCRIPT -and (Test-Path $env:Z_VCPKG_POSTSCRIPT)) {
+      # write-host (get-content -raw $env:Z_VCPKG_POSTSCRIPT)
+      $postscr = get-content -raw $env:Z_VCPKG_POSTSCRIPT
+      if( $postscr ) {
+        iex $postscr
+      }
+      Remove-Item -Force -ea 0 $env:Z_VCPKG_POSTSCRIPT,env:Z_VCPKG_POSTSCRIPT
+    }
+  }
+}
+
+# finally, if this was run with some arguments, then let's just pass it
+if( $args.length -gt 0 ) {
+  ce @args
+}
+
+return
+<#
+:set
+set ARGZ[%i%]=%1&set /a i+=1 & goto :eof
+
+:unset
+set %1=& goto :eof
+
+:init
+if exist $null erase $null
+
+:: do anything we need to before calling into powershell
+if exist $null erase $null
+
+IF "%VCPKG_ROOT%"=="" SET VCPKG_ROOT=%USERPROFILE%\.vcpkg
+
+if exist %~dp0ce\node_modules\vcpkg-ce\package.json (
+  :: we're running the wrapper script for a module-installed vcpkg-ce
+  set VCPKG_CMD=%~dpf0
+  set VCPKG_MODULE=%~dp0ce\node_modules\vcpkg-ce
+  goto INVOKE
+)
+
+:: we're running vcpkg-ce from the ce home folder
+set VCPKG_CMD=%VCPKG_ROOT%\ce\node_modules\vcpkg-ce\ce.cmd
+
+:: if we're being asked to reset the install, call bootstrap
+if "%1" EQU "--reset-ce" goto BOOTSTRAP
+
+:: if we're being asked to remove the install, call bootstrap
+if "%1" EQU "--remove-ce" (
+  set REMOVE_CE=TRUE
+  doskey ce=
+  goto BOOTSTRAP
+)
+
+:: do we even have it installed?
+if NOT exist "%VCPKG_CMD%" goto BOOTSTRAP
+
+set VCPKG_MODULE="%VCPKG_ROOT%\ce\node_modules\vcpkg-ce"
+
+:: if this is the actual installed vcpkg-ce, let's get to the invocation
+if "%~dfp0" == "%VCPKG_CMD%" goto INVOKE
+
+:: this is not the 'right' ce cmd, let's forward this on to that one.
+call %VCPKG_CMD% %*
+set VCPKG_EXITCODE=%ERRORLEVEL%
+goto :eof
+
+:INVOKE
+:: Generate 30 bits of randomness, to avoid clashing with concurrent executions.
+SET /A Z_VCPKG_POSTSCRIPT=%RANDOM% * 32768 + %RANDOM%
+SET Z_VCPKG_POSTSCRIPT=%VCPKG_ROOT%\VCPKG_tmp_%Z_VCPKG_POSTSCRIPT%.cmd
+
+:: find the right node
+if exist %VCPKG_ROOT%\ce\bin\node.exe set VCPKG_NODE=%VCPKG_ROOT%\ce\bin\node.exe
+if "%VCPKG_NODE%" EQU "" (
+  for %%i in (node.exe) do set VCPKG_NODE=%%~$PATH:i
+)
+if "%VCPKG_NODE%" EQU "" goto OHNONONODE:
+
+:: call the program
+"%VCPKG_NODE%" --harmony "%VCPKG_MODULE%" %*
+set VCPKG_EXITCODE=%ERRORLEVEL%
+doskey ce="%VCPKG_CMD%" $*
+
+:POSTSCRIPT
+:: Call the post-invocation script if it is present, then delete it.
+:: This allows the invocation to potentially modify the caller's environment (e.g. PATH).
+IF NOT EXIST "%Z_VCPKG_POSTSCRIPT%" GOTO :fin
+CALL "%Z_VCPKG_POSTSCRIPT%"
+DEL "%Z_VCPKG_POSTSCRIPT%"
+
+goto :fin
+
+:OHNONONODE
+set VCPKG_EXITCODE=1
+echo "Unable to find the nodejs for ce to run."
+goto fin:
+
+:BOOTSTRAP
+:: add the cmdline args to the environment so powershell can use them
+set /a i=0 & for %%a in (%*) do call :set %%a
+
+set POWERSHELL_EXE=
+for %%i in (pwsh.exe powershell.exe) do (
+  if EXIST "%%~$PATH:i" set POWERSHELL_EXE=%%~$PATH:i & goto :gotpwsh
+)
+:gotpwsh
+
+"%POWERSHELL_EXE%" -noprofile -executionpolicy unrestricted -command "iex (get-content %~dfp0 -raw)#" &&  set REMOVE_CE=
+set VCPKG_EXITCODE=%ERRORLEVEL%
+
+:: clear out the argz
+@for /f "delims==" %%_ in ('set ^|  findstr -i argz') do call :unset %%_
+
+:: if we're being asked to remove it,we're done.
+if "%REMOVE_CE%" EQU "TRUE" (
+  goto :fin
+)
+
+:CREATEALIAS
+doskey ce="%VCPKG_ROOT%\ce.cmd" $*
+
+:fin
+SET Z_VCPKG_POSTSCRIPT=
+SET VCPKG_CMD=
+set VCPKG_NODE=
+
+EXIT /B %VCPKG_EXITCODE%
+goto :eof
+#>

ce/assets/wrapper-scripts.js

@@ -0,0 +1,112 @@
+const { existsSync: exists, chmod, chmodSync } = require('fs');
+const { stat, copyFile, unlink } = require('fs').promises;
+const { join } = require('path');
+
+/**
+ * This script creates/removes custom wrapper scripts for vcpkg-ce.
+ */
+async function findScriptFolder() {
+  const root = `${__dirname}`;
+  let s = root;
+  while (true) {
+    s = join(s, '..');
+
+    // did we find a folder where the script is in the folder (windows style)
+    if (exists(s) && (await stat(s)).isDirectory() && (
+      exists(join(s, 'ce_.ps1')) ||
+      exists(join(s, 'ce_.cmd')) ||
+      exists(join(s, 'ce.ps1')) ||
+      exists(join(s, 'ce.cmd')))
+    ) {
+      return s;
+    }
+
+    // find it in a bin folder 
+    for (const f of ['.bin', 'bin']) {
+      const b1 = join(s, f);
+      if (exists(b1) && (await stat(b1)).isDirectory() && (
+        exists(join(b1, 'ce_')) ||
+        exists(join(b1, 'ce')) ||
+        exists(join(b1, 'ce.ps1')) ||
+        exists(join(b1, 'ce_.ps1')))
+      ) {
+        return b1;
+      }
+    }
+
+    if (s === join(s, '..')) {
+      return undefined;
+    }
+  }
+}
+
+async function create() {
+  const folder = await findScriptFolder();
+  if (!folder) {
+    console.error("Unable to find install'd folder. Aborting.")
+    return process.exit(1);
+  }
+  const files = {
+    'ce': {
+      source: 'ce',
+      install: process.platform !== 'win32'
+    },
+    'ce.ps1': {
+      source: 'ce.ps1',
+      install: true
+    },
+    'ce.cmd': {
+      source: 'ce.ps1',
+      install: process.platform === 'win32'
+    }
+  }
+
+  for (const file of ['ce_', 'ce_.ps1', 'ce_.cmd']) {
+    // remove the normally created scripts 
+    const target = join(folder, file);
+    if (exists(target)) {
+      await unlink(target);
+    }
+  }
+
+  // we install all of these, because an installation from bash can still work with powershell
+  for (const file of Object.keys(files)) {
+    console.log(`file: ${file} <== ${files[file].source} if ${files[file].install}`)
+    if (files[file].install) {
+      const target = join(folder, file);
+
+      // remove the symlink/script file if it exists
+      if (exists(target)) {
+        await unlink(target);
+      }
+      // copy the shell script into it's place
+      console.log(`copyFile: ${join(__dirname, "scripts", files[file].source)}  ==>  ${target} }`);
+      await copyFile(join(__dirname, "scripts", files[file].source), target);
+
+      chmodSync(target, 0o765);
+    }
+  }
+}
+
+async function remove() {
+  const folder = await findScriptFolder();
+  if (!folder) {
+    return process.exit(0);
+  }
+
+  for (const file of ['ce', 'ce.ps1', 'ce.cmd']) {
+    // remove the custom created scripts 
+    const target = join(folder, file);
+    if (exists(target)) {
+      await unlink(target);
+    }
+  }
+}
+
+if (process.argv[2] !== 'remove') {
+  console.error('Installing Scripts');
+  create();
+} else {
+  console.error('After this is uninstalled, you should close this terminal.');
+  remove()
+}

ce/ce.ps1

@@ -0,0 +1,40 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT License.
+
+$ENV:NODE_OPTIONS="--enable-source-maps"
+
+
+function resolve {
+    param ( [string] $name )
+    $name = Resolve-Path $name -ErrorAction 0 -ErrorVariable _err
+    if (-not($name)) { return $_err[0].TargetObject }
+    $Error.clear()
+    return $name
+}
+
+
+if( $ENV:VCPKG_ROOT ) {
+  $SCRIPT:VCPKG_ROOT=(resolve $ENV:VCPKG_ROOT)
+  $ENV:VCPKG_ROOT=$VCPKG_ROOT
+} else {
+  $SCRIPT:VCPKG_ROOT=(resolve "$HOME/.vcpkg")
+  $ENV:VCPKG_ROOT=$VCPKG_ROOT
+}
+
+# setup the postscript file
+# Generate 31 bits of randomness, to avoid clashing with concurrent executions.
+$env:Z_VCPKG_POSTSCRIPT = resolve "${VCPKG_ROOT}/VCPKG_tmp_$(Get-Random -SetSeed $PID).ps1"
+
+node $PSScriptRoot/ce @args 
+
+# dot-source the postscript file to modify the environment
+if ($env:Z_VCPKG_POSTSCRIPT -and (Test-Path $env:Z_VCPKG_POSTSCRIPT)) {
+  # write-host (get-content -raw $env:Z_VCPKG_POSTSCRIPT)
+  $content = get-content -raw $env:Z_VCPKG_POSTSCRIPT
+
+  if( $content ) {
+    iex $content
+  }
+  Remove-Item -Force $env:Z_VCPKG_POSTSCRIPT
+  remove-item -ea 0 -force env:Z_VCPKG_POSTSCRIPT
+}

ce/ce/.eslintignore

@@ -0,0 +1,3 @@
+**/*.d.ts
+test/scenarios/**
+dist/**

ce/ce/.eslintrc.yaml

@@ -0,0 +1,10 @@
+---
+# configure plugins first
+parser: "@typescript-eslint/parser"
+plugins:
+- "@typescript-eslint"
+- "notice"
+ 
+# then inherit the common settings
+extends:
+- "../common/.default-eslintrc.yaml"

ce/ce/.npmignore

@@ -0,0 +1,20 @@
+!dist/**/*
+src/
+dist/test/
+package/
+.npmignore
+tsconfig.json
+*.ts
+changelog.md
+.eslint*
+!*.d.ts
+*.tgz
+.vscode
+.scripts
+attic/
+generated/
+notes.md
+Examples/
+samples/
+*.log
+package-deps.json

ce/ce/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) Microsoft Corporation. All rights reserved.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE

ce/ce/amf/Requires.ts

@@ -0,0 +1,33 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { Scalar } from 'yaml';
+import { VersionReference as IVersionReference } from '../interfaces/metadata/version-reference';
+import { CustomScalarMap } from '../yaml/CustomScalarMap';
+import { Yaml, YAMLDictionary } from '../yaml/yaml-types';
+import { VersionReference } from './version-reference';
+
+export class Requires extends CustomScalarMap<VersionReference> {
+  constructor(node?: YAMLDictionary, parent?: Yaml, key?: string) {
+    super(VersionReference, node, parent, key);
+  }
+
+  override set(key: string, value: VersionReference | IVersionReference | string) {
+    if (typeof value === 'string') {
+      this.assert(true);   // if we don't have a node at the moment, we need to create one.
+      this.node.set(key, new Scalar(value));
+      return;
+    }
+    if (value.raw) {
+      this.assert(true);   // if we don't have a node at the moment, we need to create one.
+      this.node.set(key, new Scalar(value.raw));
+    }
+    if (value.resolved) {
+      this.assert(true);   // if we don't have a node at the moment, we need to create one.
+      this.node.set(key, new Scalar(`${value.range} ${value.resolved}`));
+    } else {
+      this.assert(true);   // if we don't have a node at the moment, we need to create one.
+      this.node.set(key, new Scalar(value.range));
+    }
+  }
+}

ce/ce/amf/contact.ts

@@ -0,0 +1,36 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { Dictionary } from '../interfaces/collections';
+import { Contact as IContact } from '../interfaces/metadata/contact';
+import { ValidationError } from '../interfaces/validation-error';
+import { Entity } from '../yaml/Entity';
+import { EntityMap } from '../yaml/EntityMap';
+import { Strings } from '../yaml/strings';
+import { Yaml, YAMLDictionary } from '../yaml/yaml-types';
+
+export class Contact extends Entity implements IContact {
+  get email(): string | undefined { return this.asString(this.getMember('email')); }
+  set email(value: string | undefined) { this.setMember('email', value); }
+
+  readonly roles = new Strings(undefined, this, 'role');
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    yield* super.validate();
+  }
+}
+
+export class Contacts extends EntityMap<YAMLDictionary, Contact> implements Dictionary<IContact> {
+  constructor(node?: YAMLDictionary, parent?: Yaml, key?: string) {
+    super(Contact, node, parent, key);
+  }
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    yield* super.validate();
+    if (this.exists()) {
+      for (const [key, contact] of this) {
+        yield* contact.validate();
+      }
+    }
+  }
+}

ce/ce/amf/demands.ts

@@ -0,0 +1,352 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { stream } from 'fast-glob';
+import { lstat, Stats } from 'fs';
+import { delimiter, join, resolve } from 'path';
+import { isMap, isScalar } from 'yaml';
+import { Activation } from '../artifacts/activation';
+import { i } from '../i18n';
+import { ErrorKind } from '../interfaces/error-kind';
+import { AlternativeFulfillment } from '../interfaces/metadata/alternative-fulfillment';
+import { ValidationError } from '../interfaces/validation-error';
+import { parseQuery } from '../mediaquery/media-query';
+import { Session } from '../session';
+import { Evaluator } from '../util/evaluator';
+import { cmdlineToArray, execute } from '../util/exec-cmd';
+import { createSandbox } from '../util/safeEval';
+import { Entity } from '../yaml/Entity';
+import { EntityMap } from '../yaml/EntityMap';
+import { Strings } from '../yaml/strings';
+import { Primitive, Yaml, YAMLDictionary } from '../yaml/yaml-types';
+import { Installs } from './installer';
+import { Requires } from './Requires';
+import { Settings } from './settings';
+
+/** sandboxed eval function for evaluating expressions */
+const safeEval: <T>(code: string, context?: any) => T = createSandbox();
+const hostFeatures = new Set<string>(['x64', 'x86', 'arm', 'arm64', 'windows', 'linux', 'osx', 'freebsd']);
+
+const ignore = new Set<string>(['info', 'contacts', 'error', 'message', 'warning', 'requires', 'see-also']);
+/**
+ * A map of mediaquery to DemandBlock
+ */
+export class Demands extends EntityMap<YAMLDictionary, DemandBlock> {
+  constructor(node?: YAMLDictionary, parent?: Yaml, key?: string) {
+    super(DemandBlock, node, parent, key);
+  }
+
+  override get keys() {
+    return super.keys.filter(each => !ignore.has(each));
+  }
+
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    yield* super.validate();
+
+    for (const [mediaQuery, demandBlock] of this) {
+      if (ignore.has(mediaQuery)) {
+        continue;
+      }
+      if (!isMap(demandBlock.node)) {
+        yield {
+          message: `Conditional demand '${mediaQuery}' is not an object`,
+          range: demandBlock.node!.range || [0, 0, 0],
+          category: ErrorKind.IncorrectType
+        };
+        continue;
+      }
+
+      const query = parseQuery(mediaQuery);
+      if (!query.isValid) {
+        yield { message: i`Error parsing conditional demand '${mediaQuery}'- ${query.error?.message}`, range: this.sourcePosition(mediaQuery)/* mediaQuery.range! */, rangeOffset: query.error, category: ErrorKind.ParseError };
+        continue;
+      }
+
+      yield* demandBlock.validate();
+    }
+  }
+}
+
+export class DemandBlock extends Entity {
+  #environment: Record<string, string | number | boolean | undefined> = {};
+  #activation?: Activation;
+  #data?: Record<string, string>;
+
+  setActivation(activation?: Activation) {
+    this.#activation = activation;
+  }
+
+  setData(data: Record<string, string>) {
+    this.#data = data;
+  }
+
+  setEnvironment(env: Record<string, string | number | boolean | undefined>) {
+    this.#environment = env;
+  }
+
+  protected get evaluationBlock() {
+    return new Evaluator(this.#data || {}, this.#environment, this.#activation?.output || {});
+  }
+  get error(): string | undefined { return this.usingAlternative ? this.unless.error : this.asString(this.getMember('error')); }
+  set error(value: string | undefined) { this.setMember('error', value); }
+
+  get warning(): string | undefined { return this.usingAlternative ? this.unless.warning : this.asString(this.getMember('warning')); }
+  set warning(value: string | undefined) { this.setMember('warning', value); }
+
+  get message(): string | undefined { return this.usingAlternative ? this.unless.warning : this.asString(this.getMember('message')); }
+  set message(value: string | undefined) { this.setMember('message', value); }
+
+  get seeAlso(): Requires {
+    return this.usingAlternative ? this.unless.seeAlso : this._seeAlso;
+  }
+
+  get requires(): Requires {
+    return this.usingAlternative ? this.unless.requires : this._requires;
+  }
+
+  get settings(): Settings {
+    return this.usingAlternative ? this.unless.settings : this._settings;
+  }
+
+  get install(): Installs {
+    return this.usingAlternative ? this.unless.install : this._install;
+  }
+
+  protected readonly _seeAlso = new Requires(undefined, this, 'seeAlso');
+  protected readonly _requires = new Requires(undefined, this, 'requires');
+  protected readonly _settings = new Settings(undefined, this, 'settings');
+  protected readonly _install = new Installs(undefined, this, 'install');
+
+  readonly unless!: Unless;
+
+  protected usingAlternative: boolean | undefined;
+
+  constructor(node?: YAMLDictionary, parent?: Yaml, key?: string) {
+    super(node, parent, key);
+    if (key !== 'unless') {
+      this.unless = new Unless(undefined, this, 'unless');
+    }
+  }
+
+  /**
+   * Async Initializer.
+   *
+   * checks the alternative demand resolution.
+   * when this runs, if the alternative is met, the rest of the demand is redirected to the alternative.
+   */
+  async init(session: Session): Promise<DemandBlock> {
+    this.#environment = session.environment;
+    if (this.usingAlternative === undefined && this.has('unless')) {
+      await this.unless.init(session);
+      this.usingAlternative = this.unless.usingAlternative;
+    }
+    return this;
+  }
+
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    yield* super.validate();
+    if (this.exists()) {
+      yield* this.settings.validate();
+      yield* this.requires.validate();
+      yield* this.seeAlso.validate();
+      yield* this.install.validate();
+    }
+  }
+
+  override asString(value: any): string | undefined {
+    if (value === undefined) {
+      return value;
+    }
+    return this.evaluationBlock.evaluate(isScalar(value) ? value.value : value);
+  }
+
+  override asPrimitive(value: any): Primitive | undefined {
+    if (value === undefined) {
+      return value;
+    }
+    if (isScalar(value)) {
+      value = value.value;
+    }
+    switch (typeof value) {
+      case 'boolean':
+      case 'number':
+        return value;
+
+      case 'string': {
+        return this.evaluationBlock.evaluate(value);
+      }
+    }
+    return undefined;
+  }
+}
+
+/** Expands string variables in a string */
+function expandStrings(sandboxData: Record<string, any>, value: string) {
+  let n = undefined;
+
+  // allow $PATH instead of ${PATH} -- simplifies YAML strings
+  value = value.replace(/\$([a-zA-Z0-9.]+)/g, '${$1}');
+
+  const parts = value.split(/(\${\S+?})/g).filter(each => each).map((each, i) => {
+    const v = each.replace(/^\${(.*)}$/, (m, match) => safeEval(match, sandboxData) ?? each);
+
+    if (v.indexOf(delimiter) !== -1) {
+      n = i;
+    }
+
+    return v;
+  });
+
+  if (n === undefined) {
+    return parts.join('');
+  }
+
+  const front = parts.slice(0, n).join('');
+  const back = parts.slice(n + 1).join('');
+
+  return parts[n].split(delimiter).filter(each => each).map(each => `${front}${each}${back}`).join(delimiter);
+}
+
+/** filters output and produces a sandbox context object */
+function filter(expression: string, content: string) {
+  const parsed = /^\/(.*)\/(\w*)$/.exec(expression);
+  const output = <any>{
+    $content: content
+  };
+  if (parsed) {
+    const filtered = new RegExp(parsed[1], parsed[2]).exec(content);
+
+    if (filtered) {
+      for (const [i, v] of filtered.entries()) {
+        if (i === 0) {
+          continue;
+        }
+        output[`$${i}`] = v;
+      }
+    }
+  }
+  return output;
+}
+
+export class Unless extends DemandBlock implements AlternativeFulfillment {
+
+  readonly from = new Strings(undefined, this, 'from');
+  readonly where = new Strings(undefined, this, 'where');
+
+  get run(): string | undefined { return this.asString(this.getMember('run')); }
+  set run(value: string | undefined) { this.setMember('run', value); }
+
+  get select(): string | undefined { return this.asString(this.getMember('select')); }
+  set select(value: string | undefined) { this.setMember('select', value); }
+
+  get matches(): string | undefined { return this.asString(this.getMember('is')); }
+  set matches(value: string | undefined) { this.setMember('is', value); }
+
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    // todo: what other validations do we need?
+    yield* super.validate();
+    if (this.has('unless')) {
+      yield {
+        message: '"unless" is not supported in an unless block',
+        range: this.sourcePosition('unless'),
+        category: ErrorKind.InvalidDefinition
+      };
+    }
+  }
+
+  override async init(session: Session): Promise<Unless> {
+    this.setEnvironment(session.environment);
+    if (this.usingAlternative === undefined) {
+      this.usingAlternative = false;
+      if (this.from.length > 0 && this.where.length > 0) {
+        // we're doing some kind of check.
+        const locations = [...this.from].map(each => expandStrings(this.evaluationBlock, each).split(delimiter)).flat();
+        const binaries = [...this.where].map(each => expandStrings(this.evaluationBlock, each));
+
+        const search = locations.map(location => binaries.map(binary => join(location, binary).replace(/\\/g, '/'))).flat();
+
+        // when we find an adequate match, we stop looking
+        // to do so and not work hrd
+
+        const Break = <NodeJS.ErrnoException>{};
+        for await (const item of stream(search, {
+          concurrency: 1,
+          stats: false, fs: <any>{
+            lstat: (path: string, callback: (error: NodeJS.ErrnoException | null, stats: Stats) => void) => {
+              // if we're done iterating, always return an error.
+              if (this.usingAlternative) {
+                return callback(Break, <Stats><any>undefined);
+              }
+
+              return lstat(path, (error, stats) => {
+                // just return an error, as we don't want more results.
+                if (this.usingAlternative) {
+                  // just return an error, as we don't want more results.
+                  return callback(Break, <Stats><any>undefined);
+                }
+
+                // symlink'd binaries on windows give us errors when it interrogates it too much.
+                if (stats && stats.mode === 41398) {
+                  stats.mode = stats.mode & ~8192;
+                }
+                return callback(error, stats);
+              });
+            }
+          }
+        })) {
+          // we found something that looks promising.
+          let filtered = <any>{ $0: item };
+          this.setData(filtered);
+          if (this.run) {
+
+            const commandline = cmdlineToArray(this.run.replace('$0', item.toString()));
+            const result = await execute(resolve(commandline[0]), commandline.slice(1));
+            if (result.code !== 0) {
+              continue;
+            }
+
+            filtered = filter(this.select || '', result.log);
+            filtered.$0 = item;
+
+            // if we have a match expression, let's check it.
+            if (this.matches && !safeEval(this.matches, filtered)) {
+              continue; // not a match, move on
+            }
+
+            // it did match, or it's just presence check
+            this.usingAlternative = true;
+            // set the data output of the check
+            // this is used later to fill in the settings.
+            this.setData(filtered);
+            return this;
+          }
+        }
+      }
+    }
+    return this;
+  }
+
+  override get error(): string | undefined { return this.asString(this.getMember('error')); }
+  override get warning(): string | undefined { return this.asString(this.getMember('warning')); }
+  override get message(): string | undefined { return this.asString(this.getMember('message')); }
+
+  override get seeAlso(): Requires {
+    return this._seeAlso;
+  }
+
+  override get requires(): Requires {
+    return this._requires;
+  }
+
+  override get settings(): Settings {
+    return this._settings;
+  }
+
+  override get install(): Installs {
+    return this._install;
+  }
+
+}

ce/ce/amf/document-context.ts

@@ -0,0 +1,15 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { LineCounter } from 'yaml';
+import { Session } from '../session';
+import { Uri } from '../util/uri';
+
+
+export interface DocumentContext {
+  session: Session;
+  filename: string;
+  file: Uri;
+  folder: Uri;
+  lineCounter: LineCounter;
+}

ce/ce/amf/global-settings.ts

@@ -0,0 +1,9 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { ScalarMap } from '../yaml/ScalarMap';
+
+export class GlobalSettings extends ScalarMap {
+  // global settings is just a map at this point.
+}
+

ce/ce/amf/info.ts

@@ -0,0 +1,58 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../i18n';
+import { ErrorKind } from '../interfaces/error-kind';
+import { Info as IInfo } from '../interfaces/metadata/info';
+import { ValidationError } from '../interfaces/validation-error';
+import { Entity } from '../yaml/Entity';
+import { Flags } from '../yaml/Flags';
+
+
+export class Info extends Entity implements IInfo {
+  get version(): string { return this.asString(this.getMember('version')) || ''; }
+  set version(value: string) { this.setMember('version', value); }
+
+  get id(): string { return this.asString(this.getMember('id')) || ''; }
+  set id(value: string) { this.setMember('id', value); }
+
+  get summary(): string | undefined { return this.asString(this.getMember('summary')); }
+  set summary(value: string | undefined) { this.setMember('summary', value); }
+
+  get priority(): number | undefined { return this.asNumber(this.getMember('priority')) || 0; }
+  set priority(value: number | undefined) { this.setMember('priority', value); }
+
+  get description(): string | undefined { return this.asString(this.getMember('description')); }
+  set description(value: string | undefined) { this.setMember('description', value); }
+
+  private flags = new Flags(undefined, this, 'options');
+
+  get dependencyOnly(): boolean { return this.flags.has('dependencyOnly'); }
+  set dependencyOnly(value: boolean) { this.flags.set('dependencyOnly', value); }
+
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    yield* super.validate();
+
+    if (!this.has('id')) {
+      yield { message: i`Missing identity '${'info.id'}'`, range: this, category: ErrorKind.FieldMissing };
+    } else if (!this.is('id', 'string')) {
+      yield { message: i`info.id should be of type 'string', found '${this.kind('id')}'`, range: this.sourcePosition('id'), category: ErrorKind.IncorrectType };
+    }
+
+    if (!this.has('version')) {
+      yield { message: i`Missing version '${'info.version'}'`, range: this, category: ErrorKind.FieldMissing };
+    } else if (!this.is('version', 'string')) {
+      yield { message: i`info.version should be of type 'string', found '${this.kind('version')}'`, range: this.sourcePosition('version'), category: ErrorKind.IncorrectType };
+    }
+    if (this.is('summary', 'string') === false) {
+      yield { message: i`info.summary should be of type 'string', found '${this.kind('summary')}'`, range: this.sourcePosition('summary'), category: ErrorKind.IncorrectType };
+    }
+    if (this.is('description', 'string') === false) {
+      yield { message: i`info.description should be of type 'string', found '${this.kind('description')}'`, range: this.sourcePosition('description'), category: ErrorKind.IncorrectType };
+    }
+    if (this.is('options', 'sequence') === false) {
+      yield { message: i`info.options should be a sequence, found '${this.kind('options')}'`, range: this.sourcePosition('options'), category: ErrorKind.IncorrectType };
+    }
+  }
+}

ce/ce/amf/installer.ts

@@ -0,0 +1,194 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { isMap, isSeq } from 'yaml';
+import { GitInstaller } from '../interfaces/metadata/installers/git';
+import { Installer as IInstaller } from '../interfaces/metadata/installers/Installer';
+import { NupkgInstaller } from '../interfaces/metadata/installers/nupkg';
+import { UnTarInstaller } from '../interfaces/metadata/installers/tar';
+import { UnZipInstaller } from '../interfaces/metadata/installers/zip';
+import { Entity } from '../yaml/Entity';
+import { EntitySequence } from '../yaml/EntitySequence';
+import { Flags } from '../yaml/Flags';
+import { Strings } from '../yaml/strings';
+import { Node, Yaml, YAMLDictionary } from '../yaml/yaml-types';
+
+export class Installs extends EntitySequence<Installer> {
+  constructor(node?: YAMLDictionary, parent?: Yaml, key?: string) {
+    super(Installer, node, parent, key);
+  }
+
+  override *[Symbol.iterator](): Iterator<Installer> {
+    if (isMap(this.node)) {
+      yield this.createInstance(this.node);
+    }
+    if (isSeq(this.node)) {
+      for (const item of this.node.items) {
+        yield this.createInstance(item);
+      }
+    }
+  }
+
+  protected createInstance(node: Node): Installer {
+    if (isMap(node)) {
+      if (node.has('unzip')) {
+        return new UnzipNode(node, this);
+      }
+      if (node.has('nupkg')) {
+        return new NupkgNode(node, this);
+      }
+      if (node.has('untar')) {
+        return new UnTarNode(node, this);
+      }
+      if (node.has('git')) {
+        return new GitCloneNode(node, this);
+      }
+    }
+    throw new Error('Unsupported node type');
+  }
+}
+
+
+export class Installer extends Entity implements IInstaller {
+  get installerKind(): string {
+    throw new Error('abstract type, should not get here.');
+  }
+
+  get lang() {
+    return this.asString(this.getMember('lang'));
+  }
+
+  get nametag() {
+    return this.asString(this.getMember('nametag'));
+  }
+}
+
+abstract class FileInstallerNode extends Installer {
+  get sha256() {
+    return this.asString(this.getMember('sha256'));
+  }
+
+  set sha256(value: string | undefined) {
+    this.setMember('sha256', value);
+  }
+
+  get sha512() {
+    return this.asString(this.getMember('sha512'));
+  }
+
+  set sha512(value: string | undefined) {
+    this.setMember('sha512', value);
+  }
+
+  get strip() {
+    return this.asNumber(this.getMember('strip'));
+  }
+
+  set strip(value: number | undefined) {
+    this.setMember('1', value);
+  }
+
+  readonly transform = new Strings(undefined, this, 'transform');
+}
+class UnzipNode extends FileInstallerNode implements UnZipInstaller {
+  override get installerKind() { return 'unzip'; }
+
+  readonly location = new Strings(undefined, this, 'unzip');
+
+}
+class UnTarNode extends FileInstallerNode implements UnTarInstaller {
+  override get installerKind() { return 'untar'; }
+  location = new Strings(undefined, this, 'untar');
+
+}
+class NupkgNode extends Installer implements NupkgInstaller {
+  get location() {
+    return this.asString(this.getMember('nupkg'))!;
+  }
+
+  set location(value: string) {
+    this.setMember('nupkg', value);
+  }
+
+  override get installerKind() { return 'nupkg'; }
+
+  get strip() {
+    return this.asNumber(this.getMember('strip'));
+  }
+
+  set strip(value: number | undefined) {
+    this.setMember('1', value);
+  }
+
+  get sha256() {
+    return this.asString(this.getMember('sha256'));
+  }
+
+  set sha256(value: string | undefined) {
+    this.setMember('sha256', value);
+  }
+
+  get sha512() {
+    return this.asString(this.getMember('sha512'));
+  }
+
+  set sha512(value: string | undefined) {
+    this.setMember('sha512', value);
+  }
+
+  readonly transform = new Strings(undefined, this, 'transform');
+
+}
+class GitCloneNode extends Installer implements GitInstaller {
+  override get installerKind() { return 'git'; }
+
+  get location() {
+    return this.asString(this.getMember('git'))!;
+  }
+
+  set location(value: string) {
+    this.setMember('git', value);
+  }
+
+  get commit() {
+    return this.asString(this.getMember('commit'));
+  }
+
+  set commit(value: string | undefined) {
+    this.setMember('commit', value);
+  }
+
+  private flags = new Flags(undefined, this, 'options');
+
+  get full() {
+    return this.flags.has('full');
+  }
+
+  set full(value: boolean) {
+    this.flags.set('full', value);
+  }
+
+  get recurse() {
+    return this.flags.has('recurse');
+  }
+
+  set recurse(value: boolean) {
+    this.flags.set('recurse', value);
+  }
+
+  get subdirectory() {
+    return this.asString(this.getMember('subdirectory'));
+  }
+
+  set subdirectory(value: string | undefined) {
+    this.setMember('subdirectory', value);
+  }
+
+  get espidf() {
+    return this.flags.has('espidf');
+  }
+
+  set espidf(value: boolean) {
+    this.flags.set('espidf', value);
+  }
+}

ce/ce/amf/metadata-file.ts

@@ -0,0 +1,216 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+
+import { extname } from 'path';
+import { Document, isMap, LineCounter, parseDocument, YAMLMap } from 'yaml';
+import { Activation } from '../artifacts/activation';
+import { Registry } from '../artifacts/registry';
+import { i } from '../i18n';
+import { ErrorKind } from '../interfaces/error-kind';
+import { Profile } from '../interfaces/metadata/metadata-format';
+import { ValidationError } from '../interfaces/validation-error';
+import { Session } from '../session';
+import { Uri } from '../util/uri';
+import { BaseMap } from '../yaml/BaseMap';
+import { toYAML } from '../yaml/yaml';
+import { Yaml, YAMLDictionary } from '../yaml/yaml-types';
+import { Contacts } from './contact';
+import { DemandBlock, Demands } from './demands';
+import { DocumentContext } from './document-context';
+import { GlobalSettings } from './global-settings';
+import { Info } from './info';
+import { Registries } from './registries';
+
+
+export class MetadataFile extends BaseMap implements Profile {
+  readonly context: DocumentContext;
+  session!: Session;
+
+  private constructor(protected document: Document.Parsed, public readonly filename: string, public lineCounter: LineCounter, public readonly registry: Registry | undefined) {
+    super(<YAMLMap<string, any>><any>document.contents);
+    this.context = <DocumentContext>{
+      filename,
+      lineCounter,
+    };
+  }
+
+  async init(session: Session): Promise<MetadataFile> {
+    this.context.session = session;
+    this.context.file = session.parseUri(this.context.filename);
+    this.context.folder = this.context.file.parent;
+    return this;
+  }
+
+  static async parseMetadata(uri: Uri, session: Session, registry?: Registry): Promise<MetadataFile> {
+    return MetadataFile.parseConfiguration(uri.path, await uri.readUTF8(), session, registry);
+  }
+
+  static async parseConfiguration(filename: string, content: string, session: Session, registry?: Registry): Promise<MetadataFile> {
+    const lc = new LineCounter();
+    if (!content || content === 'null') {
+      content = '{\n}';
+    }
+    const doc = parseDocument(content, { prettyErrors: false, lineCounter: lc, strict: true });
+    return new MetadataFile(doc, filename, lc, registry).init(session);
+  }
+
+  info = new Info(undefined, this, 'info');
+
+  contacts = new Contacts(undefined, this, 'contacts');
+  registries = new Registries(undefined, this, 'registries');
+  globalSettings = new GlobalSettings(undefined, this, 'global');
+
+  // rather than re-implement it, use encapsulatiob with a demand block
+  private demandBlock = new DemandBlock(this.node, undefined);
+
+  get error(): string | undefined { return this.demandBlock.error; }
+  set error(value: string | undefined) { this.demandBlock.error = value; }
+
+  get warning(): string | undefined { return this.demandBlock.warning; }
+  set warning(value: string | undefined) { this.demandBlock.warning = value; }
+
+  get message(): string | undefined { return this.demandBlock.message; }
+  set message(value: string | undefined) { this.demandBlock.message = value; }
+
+  get seeAlso() { return this.demandBlock.seeAlso; }
+  get requires() { return this.demandBlock.requires; }
+  get settings() { return this.demandBlock.settings; }
+  get install() { return this.demandBlock.install; }
+  get unless() { return this.demandBlock.unless; }
+
+  setActivation(activation: Activation): void {
+    this.demandBlock.setActivation(activation);
+  }
+
+  conditionalDemands = new Demands(undefined, this, 'demands');
+
+  get isFormatValid(): boolean {
+    return this.document.errors.length === 0;
+  }
+
+  get content() {
+    return toYAML(this.document.toString());
+  }
+
+  async save(uri: Uri = this.context.file): Promise<void> {
+    // check the filename, and select the format.
+    let content = '';
+
+    switch (extname(uri.path).toLowerCase()) {
+      case '.yaml':
+      case '.yml':
+        // format as yaml
+        content = this.content;
+        break;
+
+      case '.json':
+        content = JSON.stringify(this.document.toJSON(), null, 2);
+        break;
+      default:
+        throw new Error(`Unsupported file type ${extname(uri.path)}`);
+    }
+    if (!content || content === 'null') {
+      content = '{\n}';
+    }
+    await uri.writeUTF8(content);
+  }
+  #errors!: Array<string>;
+  get formatErrors(): Array<string> {
+    const t = this;
+    return this.#errors || (this.#errors = this.document.errors.map(each => {
+      const message = each.message;
+      const line = each.linePos?.[0].line || 1;
+      const column = each.linePos?.[0].col || 1;
+      return t.formatMessage(each.name, message, line, column);
+    }));
+  }
+
+  /** @internal */ formatMessage(category: ErrorKind | string, message: string, line?: number, column?: number): string {
+    if (line !== undefined && column !== undefined) {
+      return `${this.filename}:${line}:${column} ${category}, ${message}`;
+    } else {
+      return `${this.filename}: ${category}, ${message}`;
+    }
+  }
+
+  get isValid(): boolean {
+    return this.validationErrors.length === 0;
+  }
+
+  #validationErrors!: Array<string>;
+  get validationErrors(): Array<string> {
+    if (this.#validationErrors) {
+      return this.#validationErrors;
+    }
+
+    const errs = new Set<string>();
+    for (const { message, range, rangeOffset, category } of this.validate()) {
+      const r = Array.isArray(range) ? range : range?.sourcePosition();
+
+      const { line, column } = this.positionAt(r, rangeOffset);
+      errs.add(this.formatMessage(category, message, line, column));
+    }
+    this.#validationErrors = [...errs];
+    return this.#validationErrors;
+  }
+
+  private positionAt(range?: [number, number, number?], offset?: { line: number, column: number }) {
+    const { line, col } = this.lineCounter.linePos(range?.[0] || 0);
+
+    return offset ? {
+      // adds the offset values (which can come from the mediaquery parser) to the line & column. If MQ doesn't have a position, it's zero.
+      line: line + (offset.line - 1),
+      column: col + (offset.column - 1),
+    } :
+      {
+        line, column: col
+      };
+  }
+
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    yield* super.validate();
+
+    // verify that we have info
+    if (!this.document.has('info')) {
+      yield { message: i`Missing section '${'info'}'`, range: this, category: ErrorKind.SectionNotFound };
+    } else {
+      yield* this.info.validate();
+    }
+
+
+    if (this.document.has('contacts')) {
+      for (const each of this.contacts.values) {
+        yield* each.validate();
+      }
+    }
+
+    const set = new Set<string>();
+
+    for (const [mediaQuery, demandBlock] of this.conditionalDemands) {
+
+      if (set.has(mediaQuery)) {
+        yield { message: i`Duplicate keys detected in manifest: '${mediaQuery}'`, range: demandBlock, category: ErrorKind.DuplicateKey };
+      }
+      set.add(mediaQuery);
+
+      yield* demandBlock.validate();
+    }
+    yield* this.conditionalDemands.validate();
+    yield* this.install.validate();
+    yield* this.registries.validate();
+    yield* this.contacts.validate();
+    yield* this.settings.validate();
+    yield* this.globalSettings.validate();
+    yield* this.requires.validate();
+    yield* this.seeAlso.validate();
+  }
+
+  /** @internal */override assert(recreateIfDisposed = false, node = this.node): asserts this is Yaml<YAMLDictionary> & { node: YAMLDictionary } {
+    if (!isMap(this.node)) {
+      this.document = parseDocument('{\n}', { prettyErrors: false, lineCounter: this.context.lineCounter, strict: true });
+      this.node = <YAMLMap<string, any>><any>this.document.contents;
+    }
+  }
+}

ce/ce/amf/registries.ts

@@ -0,0 +1,216 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { isMap, isSeq, YAMLMap } from 'yaml';
+import { Dictionary } from '../interfaces/collections';
+import { ErrorKind } from '../interfaces/error-kind';
+import { RegistryDeclaration } from '../interfaces/metadata/metadata-format';
+import { Registry as IRegistry } from '../interfaces/metadata/registries/artifact-registry';
+import { ValidationError } from '../interfaces/validation-error';
+import { isFilePath, Uri } from '../util/uri';
+import { Entity } from '../yaml/Entity';
+import { Strings } from '../yaml/strings';
+import { Node, Yaml, YAMLDictionary, YAMLSequence } from '../yaml/yaml-types';
+
+export class Registries extends Yaml<YAMLDictionary | YAMLSequence> implements Dictionary<RegistryDeclaration>, Iterable<[string, RegistryDeclaration]>  {
+  *[Symbol.iterator](): Iterator<[string, RegistryDeclaration]> {
+    if (isMap(this.node)) {
+      for (const { key, value } of this.node.items) {
+        const v = this.createRegistry(value);
+        if (v) {
+          yield [key, v];
+        }
+      }
+    }
+    if (isSeq(this.node)) {
+      for (const item of this.node.items) {
+        if (isMap(item)) {
+          const name = this.asString(item.get('name'));
+          if (name) {
+            const v = this.createRegistry(item);
+            if (v) {
+              yield [name, v];
+            }
+          }
+        }
+      }
+    }
+  }
+
+  clear(): void {
+    this.dispose(true);
+  }
+
+  override createNode() {
+    return new YAMLSequence();
+  }
+
+  add(name: string, location?: Uri, kind?: string): RegistryDeclaration {
+    if (this.get(name)) {
+      throw new Error(`Registry ${name} already exists.`);
+    }
+
+    this.assert(true);
+    if (isMap(this.node)) {
+      throw new Error('Not Implemented as a map right now.');
+    }
+    if (isSeq(this.node)) {
+      const m = new YAMLMap();
+      this.node.add(m);
+      m.set('name', name);
+      m.set('location', location?.formatted);
+      m.set('kind', kind);
+    }
+    return this.get(name)!;
+  }
+  delete(key: string): boolean {
+    const n = this.node;
+    if (isMap(n)) {
+      const result = n.delete(key);
+      this.dispose();
+      return result;
+    }
+    if (isSeq(n)) {
+      let removed = false;
+      const items = n.items;
+      for (let i = items.length - 1; i >= 0; i--) {
+        const item = items[i];
+        if (isMap(item) && item.get('name') === key) {
+          removed ||= n.delete(i);
+        }
+      }
+      this.dispose();
+      return removed;
+    }
+    return false;
+  }
+  get(key: string): RegistryDeclaration | undefined {
+    const n = this.node;
+    if (isMap(n)) {
+      return this.createRegistry(<Node>n.get(key, true));
+    }
+    if (isSeq(n)) {
+      for (const item of n.items) {
+        if (isMap(item) && item.get('name') === key) {
+          return this.createRegistry(<Node>item);
+        }
+      }
+    }
+    return undefined;
+  }
+
+  has(key: string): boolean {
+    const n = this.node;
+    if (isMap(n)) {
+      return n.has(key);
+    }
+    if (isSeq(n)) {
+      for (const item of n.items) {
+        if (isMap(item) && item.get('name') === key) {
+          return true;
+        }
+      }
+    }
+    return false;
+  }
+
+  get length(): number {
+    if (isMap(this.node) || isSeq(this.node)) {
+      return this.node.items.length;
+    }
+    return 0;
+  }
+  get keys(): Array<string> {
+    if (isMap(this.node)) {
+      return this.node.items.map(({ key }) => this.asString(key) || '');
+    }
+    if (isSeq(this.node)) {
+      const result = new Array<string>();
+      for (const item of this.node.items) {
+        if (isMap(item)) {
+          const n = this.asString(item.get('name'));
+          if (n) {
+            result.push(n);
+          }
+        }
+      }
+      return result;
+    }
+    return [];
+  }
+
+  protected createRegistry(node: Node) {
+    if (isMap(node)) {
+      const k = this.asString(node.get('kind'));
+      const l = this.asString(node.get('location'));
+
+      // simplistic check to see if we're pointing to a file or a https:// url
+      if (k === 'artifact' && l) {
+        const ll = l?.toLowerCase();
+        if (ll.startsWith('https://')) {
+          return new RemoteRegistry(node, this);
+        }
+        if (isFilePath(l)) {
+          return new LocalRegistry(node, this);
+        }
+      }
+
+    }
+    return undefined;
+  }
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    if (this.exists()) {
+      for (const [key, registry] of this) {
+        yield* registry.validate();
+      }
+    }
+  }
+}
+
+export class Registry extends Entity implements IRegistry {
+
+  get registryKind(): string | undefined { return this.asString(this.getMember('kind')); }
+  set registryKind(value: string | undefined) { this.setMember('kind', value); }
+
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    //
+    if (this.registryKind === undefined) {
+      yield {
+        message: 'Registry missing \'kind\'',
+        range: this,
+        category: ErrorKind.FieldMissing,
+      };
+    }
+  }
+}
+
+class LocalRegistry extends Registry {
+  readonly location = new Strings(undefined, this, 'location');
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    //
+    if (this.registryKind !== 'artifact') {
+      yield {
+        message: 'Registry \'kind\' is not correct for LocalRegistry ',
+        range: this,
+        category: ErrorKind.IncorrectType,
+      };
+    }
+  }
+}
+
+class RemoteRegistry extends Registry {
+  readonly location = new Strings(undefined, this, 'location');
+  override *validate(): Iterable<ValidationError> {
+    //
+    if (this.registryKind !== 'artifact') {
+      yield {
+        message: 'Registry \'kind\' is not correct for LocalRegistry ',
+        range: this,
+        category: ErrorKind.IncorrectType,
+      };
+    }
+  }
+}

ce/ce/amf/settings.ts

@@ -0,0 +1,24 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+
+import { Settings as ISettings } from '../interfaces/metadata/Settings';
+import { ValidationError } from '../interfaces/validation-error';
+import { BaseMap } from '../yaml/BaseMap';
+import { ScalarMap } from '../yaml/ScalarMap';
+import { StringsMap } from '../yaml/strings';
+
+export class Settings extends BaseMap implements ISettings {
+  paths: StringsMap = new StringsMap(undefined, this, 'paths');
+  locations: ScalarMap<string> = new ScalarMap<string>(undefined, this, 'locations');
+  properties: StringsMap = new StringsMap(undefined, this, 'properties');
+  variables: StringsMap = new StringsMap(undefined, this, 'variables');
+  tools: ScalarMap<string> = new ScalarMap<string>(undefined, this, 'tools');
+  defines: ScalarMap<string> = new ScalarMap<string>(undefined, this, 'defines');
+
+  /** @internal */
+  override *validate(): Iterable<ValidationError> {
+    // todo: what validations do we need?
+
+  }
+}

ce/ce/amf/version-reference.ts

@@ -0,0 +1,92 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { Range, SemVer } from 'semver';
+import { VersionReference as IVersionReference } from '../interfaces/metadata/version-reference';
+import { Yaml, YAMLScalar } from '../yaml/yaml-types';
+
+
+// nuget-semver parser doesn't have a ts typings package
+// eslint-disable-next-line @typescript-eslint/no-var-requires
+const parseRange: any = require('@snyk/nuget-semver/lib/range-parser');
+
+export class VersionReference extends Yaml<YAMLScalar> implements IVersionReference {
+  get raw(): string | undefined {
+    return this.node?.value || undefined;
+  }
+
+  set raw(value: string | undefined) {
+    if (value === undefined) {
+      this.dispose(true);
+    } else {
+      this.node = new YAMLScalar(value);
+    }
+  }
+
+  static override create(): YAMLScalar {
+    return new YAMLScalar('');
+  }
+
+  private split(): [Range, SemVer | undefined] {
+
+    const v = this.raw;
+    if (v) {
+
+      const [, a, b] = /(.+)\s+([\d\\.]+)/.exec(v) || [];
+
+      if (/\[|\]|\(|\)/.exec(v)) {
+        // looks like a nuget version range.
+        try {
+          const range = parseRange(a || v);
+          let str = '';
+          if (range._components[0].minOperator) {
+            str = `${range._components[0].minOperator} ${range._components[0].minOperand}`;
+          }
+          if (range._components[0].maxOperator) {
+            str = `${str} ${range._components[0].maxOperator} ${range._components[0].maxOperand}`;
+          }
+          const newRange = new Range(str);
+          newRange.raw = a || v;
+
+          if (b) {
+            const ver = new SemVer(b, true);
+            return [newRange, ver];
+          }
+
+          return [newRange, undefined];
+
+        } catch (E) {
+          // ignore and fall thru
+        }
+      }
+
+      if (a) {
+        // we have at least a range going on here.
+        try {
+          const range = new Range(a, true);
+          const ver = new SemVer(b, true);
+          return [range, ver];
+        } catch (E) {
+          // ignore and fall thru
+        }
+      }
+      // the range or version didn't resolve correctly.
+      // must be a range alone.
+      return [new Range(v, true), undefined];
+    }
+    return [new Range('*', true), undefined];
+  }
+  get range() {
+    return this.split()[0];
+  }
+  set range(ver: Range) {
+    this.raw = `${ver.raw} ${this.resolved?.raw || ''}`.trim();
+  }
+
+  get resolved() {
+    return this.split()[1];
+  }
+  set resolved(ver: SemVer | undefined) {
+    this.raw = `${this.range.raw} ${ver?.raw || ''}`.trim();
+  }
+}

ce/ce/archivers/ZipUnpacker.ts

@@ -0,0 +1,71 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { ProgressTrackingStream } from '../fs/streams';
+import { InstallEvents } from '../interfaces/events';
+import { Session } from '../session';
+import { PercentageScaler } from '../util/percentage-scaler';
+import { Queue } from '../util/promise';
+import { Uri } from '../util/uri';
+import { UnpackOptions } from './options';
+import { pipeline, Unpacker } from './unpacker';
+import { ZipEntry, ZipFile } from './unzip';
+
+export class ZipUnpacker extends Unpacker {
+  constructor(private readonly session: Session) {
+    super();
+  }
+
+  async unpackFile(file: ZipEntry, archiveUri: Uri, outputUri: Uri, options: UnpackOptions) {
+    const extractPath = Unpacker.implementOutputOptions(file.name, options);
+    if (extractPath) {
+      const destination = outputUri.join(extractPath);
+      const fileEntry = {
+        archiveUri,
+        destination,
+        path: file.name,
+        extractPath
+      };
+
+      this.fileProgress(fileEntry, 0);
+      this.session.channels.debug(`unpacking ZIP file ${archiveUri}/${file.name} => ${destination}`);
+      await destination.parent.createDirectory();
+      const readStream = await file.read();
+      const mode = ((file.attr >> 16) & 0xfff);
+
+      const writeStream = await destination.writeStream({ mtime: file.time, mode: mode ? mode : undefined });
+      const progressStream = new ProgressTrackingStream(0, file.size);
+      progressStream.on('progress', (filePercentage) => this.fileProgress(fileEntry, filePercentage));
+      await pipeline(readStream, progressStream, writeStream);
+      this.fileProgress(fileEntry, 100);
+      this.unpacked(fileEntry);
+    }
+  }
+
+  async unpack(archiveUri: Uri, outputUri: Uri, events: Partial<InstallEvents>, options: UnpackOptions): Promise<void> {
+    this.subscribe(events);
+    try {
+      this.session.channels.debug(`unpacking ZIP ${archiveUri} => ${outputUri}`);
+
+      const openedFile = await archiveUri.openFile();
+      const zipFile = await ZipFile.read(openedFile);
+
+      const archiveProgress = new PercentageScaler(0, zipFile.files.size);
+      this.progress(0);
+      const q = new Queue();
+      let count = 0;
+      for (const file of zipFile.files.values()) {
+
+        void q.enqueue(async () => {
+          await this.unpackFile(file, archiveUri, outputUri, options);
+          this.progress(archiveProgress.scalePosition(count++));
+        });
+      }
+      await q.done;
+      await zipFile.close();
+      this.progress(100);
+    } finally {
+      this.unsubscribe(events);
+    }
+  }
+}

ce/ce/archivers/git.ts

@@ -0,0 +1,147 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { InstallEvents } from '../interfaces/events';
+import { Session } from '../session';
+import { Credentials } from '../util/credentials';
+import { execute } from '../util/exec-cmd';
+import { isFilePath, Uri } from '../util/uri';
+
+export interface CloneOptions {
+  force?: boolean;
+  credentials?: Credentials;
+}
+
+/** @internal */
+export class Git {
+  #session: Session;
+  #toolPath: string;
+  #targetFolder: string;
+  #environment: NodeJS.ProcessEnv;
+
+  constructor(session: Session, toolPath: string, environment: NodeJS.ProcessEnv, targetFolder: Uri) {
+    this.#session = session;
+    this.#toolPath = toolPath;
+    this.#targetFolder = targetFolder.fsPath;
+    this.#environment = environment;
+  }
+
+  /**
+   * Method that clones a git repo into a desired location and with various options.
+   * @param repo The Uri of the remote repository that is desired to be cloned.
+   * @param events The events that may need to be updated in order to track progress.
+   * @param options The options that will modify how the clone will be called.
+   * @returns Boolean representing whether the execution was completed without error, this is not necessarily
+   *  a gaurantee that the clone did what we expected.
+   */
+  async clone(repo: Uri, events: Partial<InstallEvents>, options: { recursive?: boolean, depth?: number } = {}) {
+    const remote = await isFilePath(repo) ? repo.fsPath : repo.toString();
+
+    const result = await execute(this.#toolPath, [
+      'clone',
+      remote,
+      this.#targetFolder,
+      options.recursive ? '--recursive' : '',
+      options.depth ? `--depth=${options.depth}` : '',
+      '--progress'
+    ], {
+      env: this.#environment,
+      onStdErrData: (chunk) => {
+        // generate progress events
+        // this.#session.channels.debug(chunk.toString());
+        const regex = /\s([0-9]*?)%/;
+        chunk.toString().split(/^/gim).map((x: string) => x.trim()).filter((each: any) => each).forEach((line: string) => {
+          const match_array = line.match(regex);
+          if (match_array !== null) {
+            events.heartbeat?.(line.trim());
+          }
+        });
+      }
+    });
+
+    if (result.code) {
+      return false;
+    }
+
+    return true;
+  }
+
+  /**
+   * Fetches a 'tag', this could theoretically be a commit, a tag, or a branch.
+   * @param remoteName Remote name to fetch from. Typically will be 'origin'.
+   * @param events Events that may be called in order to present progress.
+   * @param options Options to modify how fetch is called.
+   * @returns Boolean representing whether the execution was completed without error, this is not necessarily
+   *  a gaurantee that the fetch did what we expected.
+   */
+  async fetch(remoteName: string, events: Partial<InstallEvents>, options: { commit?: string, recursive?: boolean, depth?: number } = {}) {
+    const result = await execute(this.#toolPath, [
+      '-C',
+      this.#targetFolder,
+      'fetch',
+      remoteName,
+      options.commit ? options.commit : '',
+      options.recursive ? '--recurse-submodules' : '',
+      options.depth ? `--depth=${options.depth}` : ''
+    ], {
+      env: this.#environment
+    });
+
+    if (result.code) {
+      return false;
+    }
+
+    return true;
+  }
+
+  /**
+   * Checks out a specific commit. If no commit is given, the default behavior of a checkout will be
+   * used. (Checking out the current branch)
+   * @param events Events to possibly track progress.
+   * @param options Passing along a commit or branch to checkout, optionally.
+   * @returns Boolean representing whether the execution was completed without error, this is not necessarily
+   *  a gaurantee that the checkout did what we expected.
+   */
+  async checkout(events: Partial<InstallEvents>, options: { commit?: string } = {}) {
+    const result = await execute(this.#toolPath, [
+      '-C',
+      this.#targetFolder,
+      'checkout',
+      options.commit ? options.commit : ''
+    ], {
+      env: this.#environment
+    });
+
+    if (result.code) {
+      return false;
+    }
+
+    return true;
+  }
+
+  /**
+   * Performs a reset on the git repo.
+   * @param events Events to possibly track progress.
+   * @param options Options to control how the reset is called.
+   * @returns Boolean representing whether the execution was completed without error, this is not necessarily
+   *  a gaurantee that the reset did what we expected.
+   */
+  async reset(events: Partial<InstallEvents>, options: { commit?: string, recurse?: boolean, hard?: boolean } = {}) {
+    const result = await execute(this.#toolPath, [
+      '-C',
+      this.#targetFolder,
+      'reset',
+      options.commit ? options.commit : '',
+      options.recurse ? '--recurse-submodules' : '',
+      options.hard ? '--hard' : ''
+    ], {
+      env: this.#environment
+    });
+
+    if (result.code) {
+      return false;
+    }
+
+    return true;
+  }
+}

ce/ce/archivers/options.ts

@@ -0,0 +1,18 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+/** Unpacker output options */
+
+export interface UnpackOptions {
+  /**
+   * Strip # directories from the path
+   *
+   * Typically used to remove excessive nested folders off the front of the paths in an archive.
+  */
+  strip?: number;
+
+  /**
+   * A regular expression to transform filenames during unpack. If the resulting file name is empty, it is not emitted.
+   */
+  transform?: Array<string>;
+}

ce/ce/archivers/tar.ts

@@ -0,0 +1,141 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { fail } from 'assert';
+import { pipeline as origPipeline, Readable, Transform } from 'stream';
+import { extract as tarExtract, Headers } from 'tar-stream';
+import { promisify } from 'util';
+import { createGunzip } from 'zlib';
+import { ProgressTrackingStream } from '../fs/streams';
+import { UnifiedFileSystem } from '../fs/unified-filesystem';
+import { i } from '../i18n';
+import { InstallEvents } from '../interfaces/events';
+import { Session } from '../session';
+import { Uri } from '../util/uri';
+import { UnpackOptions } from './options';
+import { Unpacker } from './unpacker';
+
+export const pipeline = promisify(origPipeline);
+// eslint-disable-next-line @typescript-eslint/no-var-requires
+const bz2 = require('unbzip2-stream');
+
+abstract class BasicTarUnpacker extends Unpacker {
+  constructor(protected readonly session: Session) {
+    super();
+  }
+
+  async maybeUnpackEntry(archiveUri: Uri, outputUri: Uri, events: Partial<InstallEvents>, options: UnpackOptions, header: Headers, stream: Readable): Promise<void> {
+    const streamPromise = new Promise((accept, reject) => {
+      stream.on('end', accept);
+      stream.on('error', reject);
+    });
+
+    try {
+      const extractPath = Unpacker.implementOutputOptions(header.name, options);
+      let destination: Uri | undefined = undefined;
+      if (extractPath) {
+        destination = outputUri.join(extractPath);
+      }
+
+      if (destination) {
+        switch (header?.type) {
+          case 'symlink': {
+            const linkTargetUri = destination?.parent.join(header.linkname!) || fail('');
+            await destination.parent.createDirectory();
+            await (<UnifiedFileSystem>this.session.fileSystem).filesystem(linkTargetUri).createSymlink(linkTargetUri, destination!);
+          }
+            return;
+
+          case 'link': {
+            // this should be a 'hard-link' -- but I'm not sure if we can make hardlinks on windows. todo: find out
+            const linkTargetUri = outputUri.join(Unpacker.implementOutputOptions(header.linkname!, options)!);
+            // quick hack
+            await destination.parent.createDirectory();
+            await (<UnifiedFileSystem>this.session.fileSystem).filesystem(linkTargetUri).createSymlink(linkTargetUri, destination!);
+          }
+            return;
+
+          case 'directory':
+            this.session.channels.debug(`in ${archiveUri.fsPath} skipping directory ${header.name}`);
+            return;
+
+          case 'file':
+            // files handle below
+            break;
+
+          default:
+            this.session.channels.warning(i`in ${archiveUri.fsPath} skipping ${header.name} because it is a ${header?.type || ''}`);
+            return;
+        }
+
+        const fileEntry = {
+          archiveUri: archiveUri,
+          destination: destination,
+          path: header.name,
+          extractPath: extractPath
+        };
+
+        this.session.channels.debug(`unpacking TAR ${archiveUri}/${header.name} => ${destination}`);
+        this.fileProgress(fileEntry, 0);
+
+        if (header.size) {
+          const parentDirectory = destination.parent;
+          await parentDirectory.createDirectory();
+          const fileProgress = new ProgressTrackingStream(0, header.size);
+          fileProgress.on('progress', (filePercentage) => this.fileProgress(fileEntry, filePercentage));
+          fileProgress.on('progress', (filePercentage) => events?.fileProgress?.(fileEntry, filePercentage));
+          const writeStream = await destination.writeStream({ mtime: header.mtime, mode: header.mode });
+          await pipeline(stream, fileProgress, writeStream);
+        }
+
+        this.fileProgress(fileEntry, 100);
+        this.unpacked(fileEntry);
+      }
+
+    } finally {
+      stream.resume();
+      await streamPromise;
+    }
+  }
+
+  protected async unpackTar(archiveUri: Uri, outputUri: Uri, events: Partial<InstallEvents>, options: UnpackOptions, decompressor?: Transform): Promise<void> {
+    this.subscribe(events);
+    const archiveSize = await archiveUri.size();
+    const archiveFileStream = await archiveUri.readStream(0, archiveSize);
+    const archiveProgress = new ProgressTrackingStream(0, archiveSize);
+    const tarExtractor = tarExtract();
+
+    tarExtractor.on('entry', (header, stream, next) =>
+      this.maybeUnpackEntry(archiveUri, outputUri, events, options, header, stream).then(() => {
+        this.progress(archiveProgress.currentPercentage);
+        next();
+      }).catch(err => (<any>next)(err)));
+
+    if (decompressor) {
+      await pipeline(archiveFileStream, archiveProgress, decompressor, tarExtractor);
+    } else {
+      await pipeline(archiveFileStream, archiveProgress, tarExtractor);
+    }
+  }
+}
+
+export class TarUnpacker extends BasicTarUnpacker {
+  unpack(archiveUri: Uri, outputUri: Uri, events: Partial<InstallEvents>, options: UnpackOptions): Promise<void> {
+    this.session.channels.debug(`unpacking TAR ${archiveUri} => ${outputUri}`);
+    return this.unpackTar(archiveUri, outputUri, events, options);
+  }
+}
+
+export class TarGzUnpacker extends BasicTarUnpacker {
+  unpack(archiveUri: Uri, outputUri: Uri, events: Partial<InstallEvents>, options: UnpackOptions): Promise<void> {
+    this.session.channels.debug(`unpacking TAR.GZ ${archiveUri} => ${outputUri}`);
+    return this.unpackTar(archiveUri, outputUri, events, options, createGunzip());
+  }
+}
+
+export class TarBzUnpacker extends BasicTarUnpacker {
+  unpack(archiveUri: Uri, outputUri: Uri, events: Partial<InstallEvents>, options: UnpackOptions): Promise<void> {
+    this.session.channels.debug(`unpacking TAR.BZ2 ${archiveUri} => ${outputUri}`);
+    return this.unpackTar(archiveUri, outputUri, events, options, bz2());
+  }
+}

ce/ce/archivers/unpacker.ts

@@ -0,0 +1,97 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { sed } from 'sed-lite';
+import { pipeline as origPipeline } from 'stream';
+import { promisify } from 'util';
+import { InstallEvents, UnpackEvents } from '../interfaces/events';
+import { ExtendedEmitter } from '../util/events';
+import { Uri } from '../util/uri';
+import { UnpackOptions } from './options';
+
+export const pipeline = promisify(origPipeline);
+
+export interface FileEntry {
+  archiveUri: Uri;
+  destination: Uri | undefined;
+  path: string;
+  extractPath: string | undefined;
+}
+
+/** Unpacker base class definition */
+export abstract class Unpacker extends ExtendedEmitter<UnpackEvents> {
+  /* Event Emitters */
+
+  /** EventEmitter: progress, at least once per file */
+  protected progress(archivePercentage: number): void {
+    this.emit('progress', archivePercentage);
+  }
+  protected fileProgress(entry: Readonly<FileEntry>, filePercentage: number): void {
+    this.emit('fileProgress', entry, filePercentage);
+  }
+  /** EventEmitter: unpacked, emitted per file (not per archive)  */
+  protected unpacked(entry: Readonly<FileEntry>) {
+    this.emit('unpacked', entry);
+  }
+
+  abstract unpack(archiveUri: Uri, outputUri: Uri, events: Partial<InstallEvents>, options: UnpackOptions): Promise<void>;
+
+  /**
+ * Returns a new path string such that the path has prefixCount path elements removed, and directory
+ * separators normalized to a single forward slash.
+ * If prefixCount is greater than or equal to the number of path elements in the path, undefined is returned.
+ */
+  public static stripPath(path: string, prefixCount: number): string | undefined {
+    const elements = path.split(/[\\/]+/);
+    const hasLeadingSlash = elements.length !== 0 && elements[0].length === 0;
+    const hasTrailingSlash = elements.length !== 0 && elements[elements.length - 1].length === 0;
+    let countForUndefined = prefixCount;
+    if (hasLeadingSlash) {
+      ++countForUndefined;
+    }
+
+    if (hasTrailingSlash) {
+      ++countForUndefined;
+    }
+
+    if (elements.length <= countForUndefined) {
+      return undefined;
+    }
+
+    if (hasLeadingSlash) {
+      return '/' + elements.splice(prefixCount + 1).join('/');
+    }
+
+    return elements.splice(prefixCount).join('/');
+  }
+
+  /**
+ * Apply OutputOptions to a path before extraction.
+ * @param entry The initial path to a file to unpack.
+ * @param options Options to apply to that file name.
+ * @returns If the file is to be emitted, the path to use; otherwise, undefined.
+ */
+  protected static implementOutputOptions(path: string, options: UnpackOptions): string | undefined {
+    if (options.strip) {
+      const maybeStripped = Unpacker.stripPath(path, options.strip);
+      if (maybeStripped) {
+        path = maybeStripped;
+      } else {
+        return undefined;
+      }
+    }
+
+    if (options.transform) {
+      for (const transformExpr of options.transform) {
+        if (!path) {
+          break;
+        }
+
+        const sedTransformExpr = sed(transformExpr);
+        path = sedTransformExpr(path);
+      }
+    }
+
+    return path;
+  }
+}

ce/ce/archivers/unzip.ts

@@ -0,0 +1,797 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+/**
+ * @license node-stream-zip | (c) 2020 Antelle | https://github.com/antelle/node-stream-zip/blob/master/LICENSE
+ * Portions copyright https://github.com/cthackers/adm-zip | https://raw.githubusercontent.com/cthackers/adm-zip/master/LICENSE
+ */
+
+import { Readable, Transform, TransformCallback } from 'stream';
+import { constants, createInflateRaw } from 'zlib';
+import { ReadHandle } from '../fs/filesystem';
+
+interface ZipParseState {
+  window: FileWindowBuffer;
+  totalReadLength: number;
+  minPos: number;
+  lastPos: any;
+  chunkSize: any;
+  firstByte: number;
+  sig: number;
+  lastBufferPosition?: number;
+  pos: number;
+  entry: ZipEntry | null;
+  entriesLeft: number;
+  move: boolean;
+}
+
+export class ZipFile {
+  private centralDirectory!: CentralDirectoryHeader;
+  private chunkSize = 0;
+  private state!: ZipParseState;
+  private fileSize = 0;
+
+  /** file entries in the zip file */
+  public files = new Map<string, ZipEntry>();
+
+  /** folder entries in the zip file */
+  public folders = new Map<string, ZipEntry>();
+
+  /**
+   * archive comment
+   */
+  public comment?: string;
+
+  static async read(readHandle: ReadHandle) {
+    const result = new ZipFile(readHandle);
+    await result.readCentralDirectory();
+    return result;
+  }
+
+  close() {
+    return this.readHandle.close();
+  }
+
+  private constructor(private readHandle: ReadHandle) {
+  }
+
+  private async readUntilFound(): Promise<void> {
+    let pos = this.state.lastPos;
+    let bufferPosition = pos - this.state.window.position;
+    const buffer = this.state.window.buffer;
+    const minPos = this.state.minPos;
+    while (--pos >= minPos && --bufferPosition >= 0) {
+      if (buffer.length - bufferPosition >= 4 && buffer[bufferPosition] === this.state.firstByte) {
+        // quick check first signature byte
+        if (buffer.readUInt32LE(bufferPosition) === this.state.sig) {
+          this.state.lastBufferPosition = bufferPosition;
+          return;
+        }
+      }
+    }
+    if (pos === minPos) {
+      throw new Error('Bad archive');
+    }
+
+    this.state.lastPos = pos + 1;
+    this.state.chunkSize *= 2;
+    if (pos <= minPos) {
+      throw new Error('Bad archive');
+    }
+
+    const expandLength = Math.min(this.state.chunkSize, pos - minPos);
+    await this.state.window.expandLeft(expandLength);
+    return this.readUntilFound();
+  }
+
+  async readCentralDirectory() {
+    this.fileSize = await this.readHandle.size();
+    this.chunkSize = this.chunkSize || Math.round(this.fileSize / 1000);
+    this.chunkSize = Math.max(
+      Math.min(this.chunkSize, Math.min(128 * 1024, this.fileSize)),
+      Math.min(1024, this.fileSize)
+    );
+
+    const totalReadLength = Math.min(consts.ENDHDR + consts.MAXFILECOMMENT, this.fileSize);
+    this.state = <ZipParseState>{
+      window: new FileWindowBuffer(this.readHandle),
+      totalReadLength,
+      minPos: this.fileSize - totalReadLength,
+      lastPos: this.fileSize,
+      chunkSize: Math.min(1024, this.chunkSize),
+      firstByte: consts.ENDSIGFIRST,
+      sig: consts.ENDSIG,
+    };
+    await this.state.window.read(this.fileSize - this.state.chunkSize, this.state.chunkSize);
+    await this.readUntilFound();
+
+    const buffer = this.state.window.buffer;
+    const pos = this.state.lastBufferPosition || 0;
+
+    this.centralDirectory = new CentralDirectoryHeader(buffer.slice(pos, pos + consts.ENDHDR));
+
+    this.centralDirectory.headerOffset = this.state.window.position + pos;
+    this.comment = this.centralDirectory.commentLength ? buffer.slice(pos + consts.ENDHDR, pos + consts.ENDHDR + this.centralDirectory.commentLength).toString() : undefined;
+
+    if ((this.centralDirectory.volumeEntries === consts.EF_ZIP64_OR_16 && this.centralDirectory.totalEntries === consts.EF_ZIP64_OR_16) || this.centralDirectory.size === consts.EF_ZIP64_OR_32 || this.centralDirectory.offset === consts.EF_ZIP64_OR_32) {
+      return this.readZip64CentralDirectoryLocator();
+    } else {
+      this.state = <ZipParseState>{};
+      return this.readEntries();
+    }
+  }
+
+  private async readZip64CentralDirectoryLocator() {
+    const length = consts.ENDL64HDR;
+    if (this.state.lastBufferPosition! > length) {
+      this.state.lastBufferPosition! -= length;
+    } else {
+      this.state = <ZipParseState>{
+        window: this.state.window,
+        totalReadLength: length,
+        minPos: this.state.window.position - length,
+        lastPos: this.state.window.position,
+        chunkSize: this.state.chunkSize,
+        firstByte: consts.ENDL64SIGFIRST,
+        sig: consts.ENDL64SIG,
+      };
+      await this.state.window.read(this.state.lastPos - this.state.chunkSize, this.state.chunkSize);
+      await this.readUntilFound();
+    }
+
+    let buffer = this.state.window.buffer;
+    const locHeader = new CentralDirectoryLoc64Header(
+      buffer.slice(this.state.lastBufferPosition, this.state.lastBufferPosition! + consts.ENDL64HDR)
+    );
+    const readLength = this.fileSize - locHeader.headerOffset;
+    this.state = <ZipParseState>{
+      window: this.state.window,
+      totalReadLength: readLength,
+      minPos: locHeader.headerOffset,
+      lastPos: this.state.lastPos,
+      chunkSize: this.state.chunkSize,
+      firstByte: consts.END64SIGFIRST,
+      sig: consts.END64SIG,
+    };
+    await this.state.window.read(this.fileSize - this.state.chunkSize, this.state.chunkSize);
+    await this.readUntilFound();
+
+    buffer = this.state.window.buffer;
+    const zip64cd = new CentralDirectoryZip64Header(buffer.slice(this.state.lastBufferPosition, this.state.lastBufferPosition! + consts.END64HDR));
+    this.centralDirectory.volumeEntries = zip64cd.volumeEntries;
+    this.centralDirectory.totalEntries = zip64cd.totalEntries;
+    this.centralDirectory.size = zip64cd.size;
+    this.centralDirectory.offset = zip64cd.offset;
+
+    this.state = <ZipParseState>{};
+    return this.readEntries();
+  }
+
+  private async readEntries() {
+    this.state = <ZipParseState>{
+      window: new FileWindowBuffer(this.readHandle),
+      pos: this.centralDirectory.offset,
+      chunkSize: this.chunkSize,
+      entriesLeft: this.centralDirectory.volumeEntries,
+    };
+
+    await this.state.window.read(this.state.pos, Math.min(this.chunkSize, this.fileSize - this.state.pos));
+
+    while (this.state.entriesLeft) {
+      let bufferPos = this.state.pos - this.state.window.position;
+      let entry = this.state.entry;
+      const buffer = this.state.window.buffer;
+      const bufferLength = buffer.length;
+
+      if (!entry) {
+        entry = new ZipEntry(this, buffer, bufferPos);
+        entry.headerOffset = this.state.window.position + bufferPos;
+        this.state.entry = entry;
+        this.state.pos += consts.CENHDR;
+        bufferPos += consts.CENHDR;
+      }
+      const entryHeaderSize = entry.fnameLen + entry.extraLen + entry.comLen;
+      const advanceBytes = entryHeaderSize + (this.state.entriesLeft > 1 ? consts.CENHDR : 0);
+
+      // if there isn't enough bytes read, read 'em.
+      if (bufferLength - bufferPos < advanceBytes) {
+        await this.state.window.moveRight(bufferPos);
+        continue;
+      }
+
+      entry.processFilename(buffer, bufferPos);
+      entry.validateName();
+      if (entry.isDirectory) {
+        this.folders.set(entry.name, entry);
+      } else {
+        this.files.set(entry.name, entry);
+      }
+
+      this.state.entry = entry = null;
+      this.state.entriesLeft--;
+      this.state.pos += entryHeaderSize;
+      bufferPos += entryHeaderSize;
+    }
+  }
+
+  private dataOffset(entry: ZipEntry) {
+    return entry.offset + consts.LOCHDR + entry.fnameLen + entry.extraLen;
+  }
+
+  /** @internal */
+  async openEntry(entry: ZipEntry) {
+    // is this a file?
+    if (!entry.isFile) {
+      throw new Error(`Entry ${entry} not is not a file`);
+    }
+
+    // let's check to see if it's encrypted.
+    const buffer = Buffer.alloc(consts.LOCHDR);
+    await this.readHandle.readComplete(buffer, 0, buffer.length, entry.offset);
+
+    entry.parseDataHeader(buffer);
+    if (entry.encrypted) {
+      throw new Error('Entry encrypted');
+    }
+
+    const offset = this.dataOffset((<ZipEntry>entry));
+    let entryStream = this.readHandle.readStream(offset, offset + entry.compressedSize - 1);
+
+    if (entry.method === consts.STORED) {
+      // nothing to do
+    } else if (entry.method === consts.DEFLATED) {
+      entryStream = entryStream.pipe(createInflateRaw({ flush: constants.Z_SYNC_FLUSH }));
+    } else {
+      throw new Error('Unknown compression method: ' + entry.method);
+    }
+
+    // should check CRC?
+    if ((entry.flags & 0x8) === 0x8) {
+      entryStream = entryStream.pipe(createVerifier(entry.crc, entry.size));
+    }
+
+    return entryStream;
+  }
+}
+
+const consts = {
+  /* The local file header */
+  LOCHDR: 30, // LOC header size
+  LOCSIG: 0x04034b50, // "PK\003\004"
+  LOCVER: 4, // version needed to extract
+  LOCFLG: 6, // general purpose bit flag
+  LOCHOW: 8, // compression method
+  LOCTIM: 10, // modification time (2 bytes time, 2 bytes date)
+  LOCCRC: 14, // uncompressed file crc-32 value
+  LOCSIZ: 18, // compressed size
+  LOCLEN: 22, // uncompressed size
+  LOCNAM: 26, // filename length
+  LOCEXT: 28, // extra field length
+
+  /* The Data descriptor */
+  EXTSIG: 0x08074b50, // "PK\007\008"
+  EXTHDR: 16, // EXT header size
+  EXTCRC: 4, // uncompressed file crc-32 value
+  EXTSIZ: 8, // compressed size
+  EXTLEN: 12, // uncompressed size
+
+  /* The central directory file header */
+  CENHDR: 46, // CEN header size
+  CENSIG: 0x02014b50, // "PK\001\002"
+  CENVEM: 4, // version made by
+  CENVER: 6, // version needed to extract
+  CENFLG: 8, // encrypt, decrypt flags
+  CENHOW: 10, // compression method
+  CENTIM: 12, // modification time (2 bytes time, 2 bytes date)
+  CENCRC: 16, // uncompressed file crc-32 value
+  CENSIZ: 20, // compressed size
+  CENLEN: 24, // uncompressed size
+  CENNAM: 28, // filename length
+  CENEXT: 30, // extra field length
+  CENCOM: 32, // file comment length
+  CENDSK: 34, // volume number start
+  CENATT: 36, // internal file attributes
+  CENATX: 38, // external file attributes (host system dependent)
+  CENOFF: 42, // LOC header offset
+
+  /* The entries in the end of central directory */
+  ENDHDR: 22, // END header size
+  ENDSIG: 0x06054b50, // "PK\005\006"
+  ENDSIGFIRST: 0x50,
+  ENDSUB: 8, // number of entries on this disk
+  ENDTOT: 10, // total number of entries
+  ENDSIZ: 12, // central directory size in bytes
+  ENDOFF: 16, // offset of first CEN header
+  ENDCOM: 20, // zip file comment length
+  MAXFILECOMMENT: 0xffff,
+
+  /* The entries in the end of ZIP64 central directory locator */
+  ENDL64HDR: 20, // ZIP64 end of central directory locator header size
+  ENDL64SIG: 0x07064b50, // ZIP64 end of central directory locator signature
+  ENDL64SIGFIRST: 0x50,
+  ENDL64OFS: 8, // ZIP64 end of central directory offset
+
+  /* The entries in the end of ZIP64 central directory */
+  END64HDR: 56, // ZIP64 end of central directory header size
+  END64SIG: 0x06064b50, // ZIP64 end of central directory signature
+  END64SIGFIRST: 0x50,
+  END64SUB: 24, // number of entries on this disk
+  END64TOT: 32, // total number of entries
+  END64SIZ: 40,
+  END64OFF: 48,
+
+  /* Compression methods */
+  STORED: 0, // no compression
+  SHRUNK: 1, // shrunk
+  REDUCED1: 2, // reduced with compression factor 1
+  REDUCED2: 3, // reduced with compression factor 2
+  REDUCED3: 4, // reduced with compression factor 3
+  REDUCED4: 5, // reduced with compression factor 4
+  IMPLODED: 6, // imploded
+  // 7 reserved
+  DEFLATED: 8, // deflated
+  ENHANCED_DEFLATED: 9, // deflate64
+  PKWARE: 10, // PKWare DCL imploded
+  // 11 reserved
+  BZIP2: 12, //  compressed using BZIP2
+  // 13 reserved
+  LZMA: 14, // LZMA
+  // 15-17 reserved
+  IBM_TERSE: 18, // compressed using IBM TERSE
+  IBM_LZ77: 19, //IBM LZ77 z
+
+  /* General purpose bit flag */
+  FLG_ENC: 0, // encrypted file
+  FLG_COMP1: 1, // compression option
+  FLG_COMP2: 2, // compression option
+  FLG_DESC: 4, // data descriptor
+  FLG_ENH: 8, // enhanced deflation
+  FLG_STR: 16, // strong encryption
+  FLG_LNG: 1024, // language encoding
+  FLG_MSK: 4096, // mask header values
+  FLG_ENTRY_ENC: 1,
+
+  /* 4.5 Extensible data fields */
+  EF_ID: 0,
+  EF_SIZE: 2,
+
+  /* Header IDs */
+  ID_ZIP64: 0x0001,
+  ID_AVINFO: 0x0007,
+  ID_PFS: 0x0008,
+  ID_OS2: 0x0009,
+  ID_NTFS: 0x000a,
+  ID_OPENVMS: 0x000c,
+  ID_UNIX: 0x000d,
+  ID_FORK: 0x000e,
+  ID_PATCH: 0x000f,
+  ID_X509_PKCS7: 0x0014,
+  ID_X509_CERTID_F: 0x0015,
+  ID_X509_CERTID_C: 0x0016,
+  ID_STRONGENC: 0x0017,
+  ID_RECORD_MGT: 0x0018,
+  ID_X509_PKCS7_RL: 0x0019,
+  ID_IBM1: 0x0065,
+  ID_IBM2: 0x0066,
+  ID_POSZIP: 0x4690,
+
+  EF_ZIP64_OR_32: 0xffffffff,
+  EF_ZIP64_OR_16: 0xffff,
+};
+
+class CentralDirectoryHeader {
+  volumeEntries!: number;
+  totalEntries!: number;
+  size!: number;
+  offset!: number;
+  commentLength!: number;
+  headerOffset!: number;
+
+  constructor(data: Buffer) {
+    if (data.length !== consts.ENDHDR || data.readUInt32LE(0) !== consts.ENDSIG) {
+      throw new Error('Invalid central directory');
+    }
+    // number of entries on this volume
+    this.volumeEntries = data.readUInt16LE(consts.ENDSUB);
+    // total number of entries
+    this.totalEntries = data.readUInt16LE(consts.ENDTOT);
+    // central directory size in bytes
+    this.size = data.readUInt32LE(consts.ENDSIZ);
+    // offset of first CEN header
+    this.offset = data.readUInt32LE(consts.ENDOFF);
+    // zip file comment length
+    this.commentLength = data.readUInt16LE(consts.ENDCOM);
+  }
+}
+
+class CentralDirectoryLoc64Header {
+  headerOffset!: number;
+
+  constructor(data: Buffer) {
+    if (data.length !== consts.ENDL64HDR || data.readUInt32LE(0) !== consts.ENDL64SIG) {
+      throw new Error('Invalid zip64 central directory locator');
+    }
+    // ZIP64 EOCD header offset
+    this.headerOffset = readUInt64LE(data, consts.ENDSUB);
+  }
+}
+
+class CentralDirectoryZip64Header {
+  volumeEntries!: number;
+  totalEntries!: number;
+  size!: number;
+  offset!: number;
+
+  constructor(data: Buffer) {
+    if (data.length !== consts.END64HDR || data.readUInt32LE(0) !== consts.END64SIG) {
+      throw new Error('Invalid central directory');
+    }
+    // number of entries on this volume
+    this.volumeEntries = readUInt64LE(data, consts.END64SUB);
+    // total number of entries
+    this.totalEntries = readUInt64LE(data, consts.END64TOT);
+    // central directory size in bytes
+    this.size = readUInt64LE(data, consts.END64SIZ);
+    // offset of first CEN header
+    this.offset = readUInt64LE(data, consts.END64OFF);
+  }
+}
+
+export class ZipEntry {
+  /**
+  * file name
+  */
+  name!: string;
+
+  /**
+   * true if it's a directory entry
+   */
+  isDirectory!: boolean;
+
+  /**
+   * file comment
+   */
+  comment!: string | null;
+
+  /**
+   * version made by
+   */
+  verMade: number;
+
+  /**
+   * version needed to extract
+   */
+  version: number;
+
+  /**
+   * encrypt, decrypt flags
+   */
+  flags: number;
+
+  /**
+   * compression method
+   */
+  method: number;
+
+  /**
+   * modification time
+   */
+  time: Date;
+
+  /**
+   * uncompressed file crc-32 value
+   */
+  crc: number;
+
+  /**
+   * compressed size
+   */
+  compressedSize: number;
+
+  /**
+   * uncompressed size
+   */
+  size: number;
+
+  /**
+   * volume number start
+   */
+  diskStart: number;
+
+  /**
+   * internal file attributes
+   */
+  inattr: number;
+
+  /**
+   * external file attributes
+   */
+  attr: number;
+
+  /**
+   * LOC header offset
+   */
+  offset: number;
+
+  fnameLen: number;
+  extraLen: number;
+  comLen: number;
+  headerOffset!: number;
+
+  constructor(private zipFile: ZipFile, data: Buffer, offset: number) {
+    // data should be 46 bytes and start with "PK 01 02"
+    if (data.length < offset + consts.CENHDR || data.readUInt32LE(offset) !== consts.CENSIG) {
+      throw new Error('Invalid entry header');
+    }
+    // version made by
+    this.verMade = data.readUInt16LE(offset + consts.CENVEM);
+    // version needed to extract
+    this.version = data.readUInt16LE(offset + consts.CENVER);
+    // encrypt, decrypt flags
+    this.flags = data.readUInt16LE(offset + consts.CENFLG);
+    // compression method
+    this.method = data.readUInt16LE(offset + consts.CENHOW);
+    // modification time (2 bytes time, 2 bytes date)
+    const timebytes = data.readUInt16LE(offset + consts.CENTIM);
+    const datebytes = data.readUInt16LE(offset + consts.CENTIM + 2);
+    this.time = parseZipTime(timebytes, datebytes);
+
+    // uncompressed file crc-32 value
+    this.crc = data.readUInt32LE(offset + consts.CENCRC);
+    // compressed size
+    this.compressedSize = data.readUInt32LE(offset + consts.CENSIZ);
+    // uncompressed size
+    this.size = data.readUInt32LE(offset + consts.CENLEN);
+    // filename length
+    this.fnameLen = data.readUInt16LE(offset + consts.CENNAM);
+    // extra field length
+    this.extraLen = data.readUInt16LE(offset + consts.CENEXT);
+    // file comment length
+    this.comLen = data.readUInt16LE(offset + consts.CENCOM);
+    // volume number start
+    this.diskStart = data.readUInt16LE(offset + consts.CENDSK);
+    // internal file attributes
+    this.inattr = data.readUInt16LE(offset + consts.CENATT);
+    // external file attributes
+    this.attr = data.readUInt32LE(offset + consts.CENATX);
+    // LOC header offset
+    this.offset = data.readUInt32LE(offset + consts.CENOFF);
+  }
+
+  read(): Promise<Readable> {
+    return this.zipFile.openEntry(this);
+  }
+
+  parseDataHeader(data: Buffer) {
+    // 30 bytes and should start with "PK\003\004"
+    if (data.readUInt32LE(0) !== consts.LOCSIG) {
+      throw new Error('Invalid local header');
+    }
+    // version needed to extract
+    this.version = data.readUInt16LE(consts.LOCVER);
+    // general purpose bit flag
+    this.flags = data.readUInt16LE(consts.LOCFLG);
+    // compression method
+    this.method = data.readUInt16LE(consts.LOCHOW);
+    // modification time (2 bytes time ; 2 bytes date)
+    const timebytes = data.readUInt16LE(consts.LOCTIM);
+    const datebytes = data.readUInt16LE(consts.LOCTIM + 2);
+    this.time = parseZipTime(timebytes, datebytes);
+
+    // uncompressed file crc-32 value
+    this.crc = data.readUInt32LE(consts.LOCCRC) || this.crc;
+    // compressed size
+    const compressedSize = data.readUInt32LE(consts.LOCSIZ);
+    if (compressedSize && compressedSize !== consts.EF_ZIP64_OR_32) {
+      this.compressedSize = compressedSize;
+    }
+    // uncompressed size
+    const size = data.readUInt32LE(consts.LOCLEN);
+    if (size && size !== consts.EF_ZIP64_OR_32) {
+      this.size = size;
+    }
+    // filename length
+    this.fnameLen = data.readUInt16LE(consts.LOCNAM);
+    // extra field length
+    this.extraLen = data.readUInt16LE(consts.LOCEXT);
+  }
+
+  processFilename(data: Buffer, offset: number) {
+    this.name = data.slice(offset, (offset += this.fnameLen)).toString();
+    const lastChar = data[offset - 1];
+    this.isDirectory = lastChar === 47 || lastChar === 92;
+
+    if (this.extraLen) {
+      this.readExtra(data, offset);
+      offset += this.extraLen;
+    }
+    this.comment = this.comLen ? data.slice(offset, offset + this.comLen).toString() : null;
+  }
+
+  validateName() {
+    if (/\\|^\w+:|^\/|(^|\/)\.\.(\/|$)/.test(this.name)) {
+      throw new Error('Malicious entry: ' + this.name);
+    }
+  }
+
+  readExtra(data: Buffer, offset: number) {
+    let signature, size;
+    const maxPos = offset + this.extraLen;
+    while (offset < maxPos) {
+      signature = data.readUInt16LE(offset);
+      offset += 2;
+      size = data.readUInt16LE(offset);
+      offset += 2;
+      if (consts.ID_ZIP64 === signature) {
+        this.parseZip64Extra(data, offset, size);
+      }
+      offset += size;
+    }
+  }
+
+  parseZip64Extra(data: Buffer, offset: number, length: number) {
+    if (length >= 8 && this.size === consts.EF_ZIP64_OR_32) {
+      this.size = readUInt64LE(data, offset);
+      offset += 8;
+      length -= 8;
+    }
+    if (length >= 8 && this.compressedSize === consts.EF_ZIP64_OR_32) {
+      this.compressedSize = readUInt64LE(data, offset);
+      offset += 8;
+      length -= 8;
+    }
+    if (length >= 8 && this.offset === consts.EF_ZIP64_OR_32) {
+      this.offset = readUInt64LE(data, offset);
+      offset += 8;
+      length -= 8;
+    }
+    if (length >= 4 && this.diskStart === consts.EF_ZIP64_OR_16) {
+      this.diskStart = data.readUInt32LE(offset);
+      // offset += 4; length -= 4;
+    }
+  }
+
+  get encrypted() {
+    return (this.flags & consts.FLG_ENTRY_ENC) === consts.FLG_ENTRY_ENC;
+  }
+
+  get isFile() {
+    return !this.isDirectory;
+  }
+}
+
+class FileWindowBuffer {
+  position = 0;
+  buffer = Buffer.alloc(0);
+  constructor(public readHandle: ReadHandle) {
+  }
+
+  async read(pos: number, length: number) {
+    if (this.buffer.length < length) {
+      this.buffer = Buffer.alloc(length);
+    }
+    this.position = pos;
+    await this.readHandle.readComplete(this.buffer, 0, length, this.position);
+  }
+
+  async expandLeft(length: number) {
+    this.buffer = Buffer.concat([Buffer.alloc(length), this.buffer]);
+    this.position -= length;
+    if (this.position < 0) {
+      this.position = 0;
+    }
+    await this.readHandle.readComplete(this.buffer, 0, length, this.position);
+  }
+
+  async expandRight(length: number,) {
+    const offset = this.buffer.length;
+    this.buffer = Buffer.concat([this.buffer, Buffer.alloc(length)]);
+    await this.readHandle.readComplete(this.buffer, offset, length, this.position + offset);
+  }
+
+  async moveRight(shift: number) {
+    if (shift) {
+      this.buffer.copy(this.buffer, 0, shift);
+    }
+    this.position += shift;
+    await this.readHandle.readComplete(this.buffer, this.buffer.length - shift, shift, this.position + this.buffer.length - shift);
+  }
+}
+
+function createVerifier(crc: number, size: number) {
+  const verify = new Verifier(crc, size);
+  return new Transform({
+    transform: (data: any, unused: BufferEncoding, passThru: TransformCallback) => {
+      let err;
+      try {
+        verify.data(data);
+      } catch (e: any) {
+        err = e;
+      }
+      passThru(err, data);
+    }
+  });
+}
+
+function createCrcTable() {
+  const crcTable = [];
+  const b = Buffer.alloc(4);
+  for (let n = 0; n < 256; n++) {
+    let c = n;
+    for (let k = 8; --k >= 0;) {
+      if ((c & 1) !== 0) {
+        c = 0xedb88320 ^ (c >>> 1);
+      } else {
+        c = c >>> 1;
+      }
+    }
+    if (c < 0) {
+      b.writeInt32LE(c, 0);
+      c = b.readUInt32LE(0);
+    }
+    crcTable[n] = c;
+  }
+  return crcTable;
+}
+
+const crcTable = createCrcTable();
+
+class Verifier {
+  state = {
+    crc: ~0,
+    size: 0,
+  };
+
+  constructor(public crc: number, public size: number) {
+  }
+
+  data(data: Buffer) {
+    let crc = this.state.crc;
+    let off = 0;
+    let len = data.length;
+    while (--len >= 0) {
+      crc = crcTable[(crc ^ data[off++]) & 0xff] ^ (crc >>> 8);
+    }
+    this.state.crc = crc;
+    this.state.size += data.length;
+    if (this.state.size >= this.size) {
+      const buf = Buffer.alloc(4);
+      buf.writeInt32LE(~this.state.crc & 0xffffffff, 0);
+      crc = buf.readUInt32LE(0);
+      if (crc !== this.crc) {
+        throw new Error(`Invalid CRC Expected: ${this.crc} found:${crc} `);
+      }
+      if (this.state.size !== this.size) {
+        throw new Error('Invalid size');
+      }
+    }
+  }
+}
+
+function parseZipTime(timebytes: number, datebytes: number) {
+  const timebits = toBits(timebytes, 16);
+  const datebits = toBits(datebytes, 16);
+
+  const mt = {
+    h: parseInt(timebits.slice(0, 5).join(''), 2),
+    m: parseInt(timebits.slice(5, 11).join(''), 2),
+    s: parseInt(timebits.slice(11, 16).join(''), 2) * 2,
+    Y: parseInt(datebits.slice(0, 7).join(''), 2) + 1980,
+    M: parseInt(datebits.slice(7, 11).join(''), 2),
+    D: parseInt(datebits.slice(11, 16).join(''), 2),
+  };
+  const dt_str = [mt.Y, mt.M, mt.D].join('-') + ' ' + [mt.h, mt.m, mt.s].join(':') + ' GMT+0';
+  return new Date(dt_str);
+}
+
+function toBits(dec: number, size: number) {
+  let b = (dec >>> 0).toString(2);
+  while (b.length < size) {
+    b = '0' + b;
+  }
+  return b.split('');
+}
+
+function readUInt64LE(buffer: Buffer, offset: number) {
+  return buffer.readUInt32LE(offset + 4) * 0x0000000100000000 + buffer.readUInt32LE(offset);
+}

ce/ce/artifacts/SetOfDemands.ts

@@ -0,0 +1,86 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { MetadataFile } from '../amf/metadata-file';
+import { Demands } from '../interfaces/metadata/demands';
+import { VersionReference } from '../interfaces/metadata/version-reference';
+import { parseQuery } from '../mediaquery/media-query';
+import { Session } from '../session';
+import { MultipleInstallsMatched } from '../util/exceptions';
+import { Dictionary, linq } from '../util/linq';
+import { Activation } from './activation';
+
+
+export class SetOfDemands {
+  _demands = new Map<string, Demands>();
+
+  constructor(metadata: MetadataFile, session: Session) {
+    this._demands.set('', metadata);
+
+    for (const [query, demands] of metadata.conditionalDemands) {
+      if (parseQuery(query).match(session.context)) {
+        session.channels.debug(`Matching demand query: '${query}'`);
+        this._demands.set(query, demands);
+      }
+    }
+  }
+
+  setActivation(activation: Activation) {
+    for (const [, demandBlock] of this._demands.entries()) {
+      demandBlock.setActivation(activation);
+    }
+  }
+
+  /** Async Initializer */
+  async init(session: Session) {
+    for (const [query, demands] of this._demands) {
+      await demands.init(session);
+    }
+  }
+
+  get installer() {
+    const install = linq.entries(this._demands).where(([query, demand]) => demand.install.length > 0).toArray();
+
+    if (install.length > 1) {
+      // bad. There should only ever be one install block.
+      throw new MultipleInstallsMatched(install.map(each => each[0]));
+    }
+
+    return install[0]?.[1].install || [];
+  }
+
+  get errors() {
+    return linq.values(this._demands).selectNonNullable(d => d.error).toArray();
+  }
+  get warnings() {
+    return linq.values(this._demands).selectNonNullable(d => d.warning).toArray();
+  }
+  get messages() {
+    return linq.values(this._demands).selectNonNullable(d => d.message).toArray();
+  }
+  get settings() {
+    return linq.values(this._demands).selectNonNullable(d => d.settings).toArray();
+  }
+  get seeAlso() {
+    return linq.values(this._demands).selectNonNullable(d => d.seeAlso).toArray();
+  }
+
+  get requires() {
+    const d = this._demands;
+    const rq1 = linq.values(d).selectNonNullable(d => d.requires).toArray();
+    const result = new Dictionary<VersionReference>();
+    for (const dict of rq1) {
+      for (const [query, demands] of dict) {
+        result[query] = demands;
+      }
+    }
+    const rq = [...d.values()].map(each => each.requires).filter(each => each);
+
+    for (const dict of rq) {
+      for (const [query, demands] of dict) {
+        result[query] = demands;
+      }
+    }
+    return result;
+  }
+}

ce/ce/artifacts/activation.ts

@@ -0,0 +1,166 @@
+/* eslint-disable keyword-spacing */
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { delimiter } from 'path';
+import { Session } from '../session';
+import { linq } from '../util/linq';
+import { Uri } from '../util/uri';
+import { toXml } from '../util/xml';
+import { Artifact } from './artifact';
+
+export class Activation {
+  #session: Session;
+  constructor(session: Session) {
+    this.#session = session;
+  }
+
+  /** gets a flattend object representation of the activation */
+  get output() {
+    return {
+      defines: Object.fromEntries(this.defines),
+      locations: Object.fromEntries([... this.locations.entries()].map(([k, v]) => [k, v.fsPath])),
+      properties: Object.fromEntries([... this.properties.entries()].map(([k, v]) => [k, v.join(',')])),
+      environment: { ...process.env, ...Object.fromEntries([... this.environment.entries()].map(([k, v]) => [k, v.join(' ')])) },
+      tools: Object.fromEntries(this.tools),
+      paths: Object.fromEntries([...this.paths.entries()].map(([k, v]) => [k, v.map(each => each.fsPath).join(delimiter)])),
+      aliases: Object.fromEntries(this.aliases)
+    };
+  }
+
+  generateMSBuild(artifacts: Iterable<Artifact>): string {
+    const msbuildFile = {
+      Project: {
+        $xmlns: 'http://schemas.microsoft.com/developer/msbuild/2003',
+        PropertyGroup: <Array<Record<string, any>>>[]
+      }
+    };
+
+    if (this.locations.size) {
+      msbuildFile.Project.PropertyGroup.push({ $Label: 'Locations', ...linq.entries(this.locations).toObject(([key, value]) => [key, value.fsPath]) });
+    }
+
+    if (this.properties.size) {
+      msbuildFile.Project.PropertyGroup.push({ $Label: 'Properties', ...linq.entries(this.properties).toObject(([key, value]) => [key, value.join(';')]) });
+    }
+
+    if (this.tools.size) {
+      msbuildFile.Project.PropertyGroup.push({ $Label: 'Tools', ...linq.entries(this.tools).toObject(each => each) });
+    }
+
+    if (this.environment.size) {
+      msbuildFile.Project.PropertyGroup.push({ $Label: 'Environment', ...linq.entries(this.environment).toObject(each => each) });
+    }
+
+    if (this.paths.size) {
+      msbuildFile.Project.PropertyGroup.push({ $Label: 'Paths', ...linq.entries(this.paths).toObject(([key, value]) => [key, value.map(each => each.fsPath).join(';')]) });
+    }
+
+    if (this.defines.size) {
+      msbuildFile.Project.PropertyGroup.push({ $Label: 'Defines', DEFINES: linq.entries(this.defines).select(([key, value]) => `${key}=${value}`).join(';') });
+    }
+
+    if (this.aliases.size) {
+      msbuildFile.Project.PropertyGroup.push({ $Label: 'Aliases', ...linq.entries(this.environment).toObject(each => each) });
+    }
+
+    const propertyGroup = <any>{ $Label: 'Artifacts', Artifacts: { Artifact: [] } };
+
+    for (const artifact of artifacts) {
+      propertyGroup.Artifacts.Artifact.push({ $id: artifact.metadata.info.id, '#text': artifact.targetLocation.fsPath });
+    }
+
+    if (propertyGroup.Artifacts.Artifact.length > 0) {
+      msbuildFile.Project.PropertyGroup.push(propertyGroup);
+    }
+
+    return toXml(msbuildFile);
+  }
+
+  /** a collection of #define declarations that would assumably be applied to all compiler calls. */
+  defines = new Map<string, string>();
+
+  /** a collection of tool definitions from artifacts (think shell 'aliases')  */
+  tools = new Map<string, string>();
+
+  /** Aliases are tools that get exposed to the user as shell aliases */
+  aliases = new Map<string, string>();
+
+  /** a collection of 'published locations' from artifacts. useful for msbuild */
+  locations = new Map<string, Uri>();
+
+  /** a collection of environment variables from artifacts that are intended to be combinined into variables that have PATH delimiters */
+  paths = new Map<string, Array<Uri>>();
+
+  /** environment variables from artifacts */
+  environment = new Map<string, Array<string>>();
+
+  /** a collection of arbitrary properties from artifacts. useful for msbuild */
+  properties = new Map<string, Array<string>>();
+
+  get Paths() {
+    // return just paths that have contents.
+    return [... this.paths.entries()].filter(([k, v]) => v.length > 0);
+  }
+
+  get Variables() {
+    // tools + environment
+    const result = new Array<[string, string]>();
+
+    // combine variables with spaces
+    for (const [key, values] of this.environment) {
+      result.push([key, values.join(' ')]);
+    }
+
+    // add tools to the list
+    for (const [key, value] of this.tools) {
+      result.push([key, value]);
+    }
+    return result;
+  }
+
+  get Defines(): Array<[string, string]> {
+    return linq.entries(this.defines).toArray();
+  }
+
+  get Locations(): Array<[string, string]> {
+    return linq.entries(this.locations).select(([k, v]) => <[string, string]>[k, v.fsPath]).where(([k, v]) => v.length > 0).toArray();
+  }
+
+  get Properties(): Array<[string, Array<string>]> {
+    return linq.entries(this.properties).toArray();
+  }
+
+  /** produces an environment block that can be passed to child processes to leverage dependent artifacts during installtion/activation. */
+  get environmentBlock(): NodeJS.ProcessEnv {
+    const result = this.#session.environment;
+
+    // add environment variables
+    for (const [k, v] of this.Variables) {
+      result[k] = v;
+    }
+
+    // update environment paths
+    for (const [variable, values] of this.Paths) {
+      if (values.length) {
+        const s = new Set(values.map(each => each.fsPath));
+        const originalVariable = result[variable] || '';
+        if (originalVariable) {
+          for (const p of originalVariable.split(delimiter)) {
+            if (p) {
+              s.add(p);
+            }
+          }
+        }
+        result[variable] = originalVariable;
+      }
+    }
+
+    // define tool environment variables
+    for (const [key, value] of this.tools) {
+      result[key] = value;
+    }
+
+    return result;
+  }
+}

ce/ce/artifacts/artifact.ts

@@ -0,0 +1,358 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { fail } from 'assert';
+import { resolve } from 'path';
+import { MetadataFile } from '../amf/metadata-file';
+import { gitArtifact, gitUniqueIdPrefix, latestVersion } from '../constants';
+import { i } from '../i18n';
+import { InstallEvents } from '../interfaces/events';
+import { Registries } from '../registries/registries';
+import { Session } from '../session';
+import { linq } from '../util/linq';
+import { Uri } from '../util/uri';
+import { Activation } from './activation';
+import { Registry } from './registry';
+import { SetOfDemands } from './SetOfDemands';
+
+export type Selections = Map<string, string>;
+export type UID = string;
+export type ID = string;
+export type VersionRange = string;
+export type Selection = [Artifact, ID, VersionRange]
+
+export class ArtifactMap extends Map<UID, Selection>{
+  get artifacts() {
+    return [...linq.values(this).select(([artifact, id, range]) => artifact)].sort((a, b) => (b.metadata.info.priority || 0) - (a.metadata.info.priority || 0));
+  }
+}
+
+class ArtifactBase {
+  public registries: Registries;
+  readonly applicableDemands: SetOfDemands;
+
+  constructor(protected session: Session, public readonly metadata: MetadataFile) {
+    this.applicableDemands = new SetOfDemands(this.metadata, this.session);
+    this.registries = new Registries(session);
+
+    // load the registries from the project file
+    for (const [name, registry] of this.metadata.registries) {
+      const reg = session.loadRegistry(registry.location.get(0), registry.registryKind || 'artifact');
+      if (reg) {
+        this.registries.add(reg, name);
+      }
+    }
+  }
+
+  /** Async Initializer */
+  async init(session: Session) {
+    await this.applicableDemands.init(session);
+    return this;
+  }
+
+  async resolveDependencies(artifacts = new ArtifactMap(), recurse = true) {
+    // find the dependencies and add them to the set
+
+    let dependency: [Registry, string, Artifact] | undefined;
+    for (const [id, version] of linq.entries(this.applicableDemands.requires)) {
+      dependency = undefined;
+      if (id.indexOf(':') === -1) {
+        if (this.metadata.registry) {
+          // let's assume the dependency is in the same registry as the artifact
+          const [registry, b, artifacts] = (await this.metadata.registry.search(this.registries, { idOrShortName: id, version: version.raw }))[0];
+          dependency = [registry, b, artifacts[0]];
+          if (!dependency) {
+            throw new Error(i`Dependency '${id}' version '${version.raw}' does not specify the registry.`);
+          }
+        }
+      }
+      dependency = dependency || await this.registries.getArtifact(id, version.raw);
+      if (!dependency) {
+        throw new Error(i`Unable to resolve dependency ${id}: ${version.raw}`);
+      }
+      const artifact = dependency[2];
+      if (!artifacts.has(artifact.uniqueId)) {
+        artifacts.set(artifact.uniqueId, [artifact, id, version.raw || latestVersion]);
+
+        if (recurse) {
+          // process it's dependencies too.
+          await artifact.resolveDependencies(artifacts);
+        }
+      }
+    }
+
+
+    if (!linq.startsWith(artifacts, gitUniqueIdPrefix)) {
+      // check if anyone needs git and add it if it isn't there
+      for (const each of this.applicableDemands.installer) {
+        if (each.installerKind === 'git') {
+          const [reg, id, art] = await this.registries.getArtifact(gitArtifact, latestVersion) || [];
+          if (art) {
+            artifacts.set(gitArtifact, [art, gitArtifact, latestVersion]);
+            break;
+          }
+        }
+      }
+    }
+    return artifacts;
+  }
+}
+
+export class Artifact extends ArtifactBase {
+  isPrimary = false;
+
+  constructor(session: Session, metadata: MetadataFile, public shortName: string = '', public targetLocation: Uri, public readonly registryId: string, public readonly registryUri: Uri) {
+    super(session, metadata);
+  }
+
+  get id() {
+    return this.metadata.info.id;
+  }
+
+  get reference() {
+    return `${this.registryId}:${this.id}`;
+  }
+
+  get version() {
+    return this.metadata.info.version;
+  }
+
+  get isInstalled() {
+    return this.targetLocation.exists('artifact.yaml');
+  }
+
+  get uniqueId() {
+    return `${this.registryUri.toString()}::${this.id}::${this.version}`;
+  }
+
+  async install(activation: Activation, events: Partial<InstallEvents>, options: { force?: boolean, allLanguages?: boolean, language?: string }): Promise<boolean> {
+    let installing = false;
+    try {
+      // is it installed?
+      const applicableDemands = this.applicableDemands;
+      applicableDemands.setActivation(activation);
+
+      let isFailing = false;
+      for (const error of applicableDemands.errors) {
+        this.session.channels.error(error);
+        isFailing = true;
+      }
+
+      if (isFailing) {
+        throw Error('errors present');
+      }
+
+      // warnings
+      for (const warning of applicableDemands.warnings) {
+        this.session.channels.warning(warning);
+      }
+
+      // messages
+      for (const message of applicableDemands.messages) {
+        this.session.channels.message(message);
+      }
+
+      if (await this.isInstalled && !options.force) {
+        await this.loadActivationSettings(activation);
+        return false;
+      }
+      installing = true;
+
+      if (options.force) {
+        try {
+          await this.uninstall();
+        } catch {
+          // if a file is locked, it may not get removed. We'll deal with this later.
+        }
+      }
+
+      // ok, let's install this.
+      for (const installInfo of applicableDemands.installer) {
+        if (installInfo.lang && !options.allLanguages && options.language && options.language.toLowerCase() !== installInfo.lang.toLowerCase()) {
+          continue;
+        }
+
+        const installer = this.session.artifactInstaller(installInfo);
+        if (!installer) {
+          fail(i`Unknown installer type ${installInfo!.installerKind}`);
+        }
+        await installer(this.session, activation, this.id, this.targetLocation, installInfo, events, options);
+      }
+
+      // after we unpack it, write out the installed manifest
+      await this.writeManifest();
+      await this.loadActivationSettings(activation);
+      return true;
+    } catch (err) {
+      if (installing) {
+        // if we started installing, and then had an error, we need to remove the artifact.
+        try {
+          await this.uninstall();
+        } catch {
+          // if a file is locked, it may not get removed. We'll deal with this later.
+        }
+      }
+      throw err;
+    }
+  }
+
+  get name() {
+    return `${this.metadata.info.id.replace(/[^\w]+/g, '.')}-${this.metadata.info.version}`;
+  }
+
+  async writeManifest() {
+    await this.targetLocation.createDirectory();
+    await this.metadata.save(this.targetLocation.join('artifact.yaml'));
+  }
+
+  async uninstall() {
+    await this.targetLocation.delete({ recursive: true, useTrash: false });
+  }
+
+  async loadActivationSettings(activation: Activation) {
+    // construct paths (bin, lib, include, etc.)
+    // construct tools
+    // compose variables
+    // defines
+
+    const l = this.targetLocation.toString().length + 1;
+    const allPaths = (await this.targetLocation.readDirectory(undefined, { recursive: true })).select(([name, stat]) => name.toString().substr(l));
+
+    for (const settingBlock of this.applicableDemands.settings) {
+      // **** defines ****
+      // eslint-disable-next-line prefer-const
+      for (let [key, value] of settingBlock.defines) {
+        if (value === 'true') {
+          value = '1';
+        }
+
+        const v = activation.defines.get(key);
+        if (v && v !== value) {
+          // conflict. todo: what do we want to do?
+          this.session.channels.warning(i`Duplicate define ${key} during activation. New value will replace old `);
+        }
+        activation.defines.set(key, value);
+      }
+
+      // **** paths ****
+      for (const key of settingBlock.paths.keys) {
+        if (!key) {
+          continue;
+        }
+
+        const pathEnvVariable = key.toUpperCase();
+        const p = activation.paths.getOrDefault(pathEnvVariable, []);
+        const l = settingBlock.paths.get(key);
+        const uris = new Set<Uri>();
+
+        for (const location of l ?? []) {
+          // check that each path is an actual path.
+          const path = await this.sanitizeAndValidatePath(location);
+          if (path && !uris.has(path)) {
+            uris.add(path);
+            p.push(path);
+          }
+        }
+      }
+
+      // **** tools ****
+      for (const key of settingBlock.tools.keys) {
+        const envVariable = key.toUpperCase();
+
+        if (activation.tools.has(envVariable)) {
+          this.session.channels.error(i`Duplicate tool declared ${key} during activation. Probably not a good thing?`);
+        }
+
+        const p = settingBlock.tools.get(key) || '';
+        const uri = await this.sanitizeAndValidatePath(p);
+        if (uri) {
+          activation.tools.set(envVariable, uri.fsPath);
+        } else {
+          if (p) {
+            activation.tools.set(envVariable, p);
+            // this.session.channels.warning(i`Invalid tool path '${p}'`);
+          }
+        }
+      }
+
+      // **** variables ****
+      for (const [key, value] of settingBlock.variables) {
+        const envKey = activation.environment.getOrDefault(key, []);
+        envKey.push(...value);
+      }
+
+      // **** properties ****
+      for (const [key, value] of settingBlock.properties) {
+        const envKey = activation.properties.getOrDefault(key, []);
+        envKey.push(...value);
+      }
+
+      // **** locations ****
+      for (const locationName of settingBlock.locations.keys) {
+        if (activation.locations.has(locationName)) {
+          this.session.channels.error(i`Duplicate location declared ${locationName} during activation. Probably not a good thing?`);
+        }
+
+        const p = settingBlock.locations.get(locationName) || '';
+        const uri = await this.sanitizeAndValidatePath(p);
+        if (uri) {
+          activation.locations.set(locationName, uri);
+        }
+      }
+    }
+  }
+
+  async sanitizeAndValidatePath(path: string) {
+    if (!path.startsWith('.')) {
+      try {
+        const loc = this.session.fileSystem.file(resolve(path));
+        if (await loc.exists()) {
+          return loc;
+        }
+      } catch {
+        // no worries, treat it like a relative path.
+      }
+    }
+    const loc = this.targetLocation.join(sanitizePath(path));
+    if (await loc.exists()) {
+      return loc;
+    }
+    return undefined;
+  }
+}
+
+export function sanitizePath(path: string) {
+  return path.
+    replace(/[\\/]+/g, '/').     // forward slahses please
+    replace(/[?<>:|"]/g, ''). // remove illegal characters.
+    // eslint-disable-next-line no-control-regex
+    replace(/[\x00-\x1f\x80-\x9f]/g, ''). // remove unicode control codes
+    replace(/^(con|prn|aux|nul|com[0-9]|lpt[0-9])$/i, ''). // no reserved names
+    replace(/^[/.]*\//, ''). // dots and slashes off the front.
+    replace(/[/.]+$/, ''). // dots and slashes off the back.
+    replace(/\/\.+\//g, '/'). // no parts made just of dots.
+    replace(/\/+/g, '/'); // duplicate slashes.
+}
+
+export function sanitizeUri(u: string) {
+  return u.
+    replace(/[\\/]+/g, '/').     // forward slahses please
+    replace(/[?<>|"]/g, ''). // remove illegal characters.
+    // eslint-disable-next-line no-control-regex
+    replace(/[\x00-\x1f\x80-\x9f]/g, ''). // remove unicode control codes
+    replace(/^(con|prn|aux|nul|com[0-9]|lpt[0-9])$/i, ''). // no reserved names
+    replace(/^[/.]*\//, ''). // dots and slashes off the front.
+    replace(/[/.]+$/, ''). // dots and slashes off the back.
+    replace(/\/\.+\//g, '/'). // no parts made just of dots.
+    replace(/\/+/g, '/'); // duplicate slashes.
+}
+
+export class ProjectManifest extends ArtifactBase {
+
+}
+
+export class InstalledArtifact extends Artifact {
+  constructor(session: Session, metadata: MetadataFile) {
+    super(session, metadata, '', Uri.invalid, 'OnDisk?', Uri.invalid); /* fixme ? */
+  }
+}

ce/ce/artifacts/registry.ts

@@ -0,0 +1,25 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { Registries } from '../registries/registries';
+import { Uri } from '../util/uri';
+import { Artifact } from './artifact';
+
+export interface SearchCriteria {
+  idOrShortName?: string;
+  version?: string
+  keyword?: string;
+}
+
+export interface Registry {
+  readonly count: number;
+  readonly location: Uri;
+  readonly loaded: boolean;
+
+  search(parent: Registries, criteria?: SearchCriteria): Promise<Array<[Registry, string, Array<Artifact>]>>;
+
+  load(force?: boolean): Promise<void>;
+  save(): Promise<void>;
+  update(): Promise<void>;
+  regenerate(): Promise<void>;
+}

ce/ce/cli/argument.ts

@@ -0,0 +1,15 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { Command } from './command';
+import { Help } from './command-line';
+
+export abstract class Argument implements Help {
+  readonly abstract argument: string;
+  readonly title = '';
+  readonly abstract help: Array<string>;
+
+  constructor(protected command: Command) {
+    command.arguments.push(this);
+  }
+}

ce/ce/cli/artifacts.ts

@@ -0,0 +1,153 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { MultiBar, SingleBar } from 'cli-progress';
+import { Activation } from '../artifacts/activation';
+import { Artifact, ArtifactMap } from '../artifacts/artifact';
+import { i } from '../i18n';
+import { trackAcquire } from '../insights';
+import { Registries } from '../registries/registries';
+import { Session } from '../session';
+import { artifactIdentity, artifactReference } from './format';
+import { Table } from './markdown-table';
+import { debug, error, log } from './styling';
+
+export async function showArtifacts(artifacts: Iterable<Artifact>, options?: { force?: boolean }) {
+  let failing = false;
+  const table = new Table(i`Artifact`, i`Version`, i`Status`, i`Dependency`, i`Summary`);
+  for (const artifact of artifacts) {
+    const name = artifactIdentity(artifact.registryId, artifact.id, artifact.shortName);
+    if (!artifact.metadata.isValid) {
+      failing = true;
+      for (const err of artifact.metadata.validationErrors) {
+        error(err);
+      }
+    }
+    table.push(name, artifact.version, options?.force || await artifact.isInstalled ? 'installed' : 'will install', artifact.isPrimary ? ' ' : '*', artifact.metadata.info.summary || '');
+  }
+  log(table.toString());
+
+  return !failing;
+}
+
+export type Selections = Map<string, string>;
+
+export async function selectArtifacts(selections: Selections, registries: Registries): Promise<false | ArtifactMap> {
+  const artifacts = new ArtifactMap();
+
+  for (const [identity, version] of selections) {
+    const [registry, id, artifact] = await registries.getArtifact(identity, version) || [];
+
+    if (!artifact) {
+      error(`Unable to resolve artifact: ${artifactReference('', identity, version)}`);
+      return false;
+    }
+
+    artifacts.set(artifact.uniqueId, [artifact, identity, version]);
+    artifact.isPrimary = true;
+    await artifact.resolveDependencies(artifacts);
+  }
+  return artifacts;
+}
+
+export async function installArtifacts(session: Session, artifacts: Iterable<Artifact>, options?: { force?: boolean, allLanguages?: boolean, language?: string }): Promise<[boolean, Map<Artifact, boolean>, Activation]> {
+  // resolve the full set of artifacts to install.
+  const installed = new Map<Artifact, boolean>();
+  const activation = new Activation(session);
+
+  const bar = new MultiBar({
+    clearOnComplete: true, hideCursor: true, format: '{name} {bar}\u25A0 {percentage}% {action} {current}',
+    barCompleteChar: '\u25A0',
+    barIncompleteChar: ' ',
+    etaBuffer: 40
+  });
+  let dl: SingleBar | undefined;
+  let p: SingleBar | undefined;
+  let spinnerValue = 0;
+
+  for (const artifact of artifacts) {
+    const id = artifact.id;
+    const registryName = artifact.registryId;
+
+    try {
+      const actuallyInstalled = await artifact.install(activation, {
+        verifying: (name, percent) => {
+          if (percent >= 100) {
+            p?.update(percent);
+            p = undefined;
+            return;
+          }
+          if (percent) {
+            if (!p) {
+              p = bar.create(100, 0, { action: i`verifying`, name: artifactIdentity(registryName, id), current: name });
+            }
+            p?.update(percent);
+          }
+        },
+        download: (name, percent) => {
+          if (percent >= 100) {
+            if (dl) {
+              dl.update(percent);
+            }
+            dl = undefined;
+            return;
+          }
+          if (percent) {
+            if (!dl) {
+              dl = bar.create(100, 0, { action: i`downloading`, name: artifactIdentity(registryName, id), current: name });
+            }
+            dl.update(percent);
+          }
+        },
+        fileProgress: (entry) => {
+          p?.update({ action: i`unpacking`, name: artifactIdentity(registryName, id), current: entry.extractPath });
+        },
+        progress: (percent: number) => {
+          if (percent >= 100) {
+            if (p) {
+              p.update(percent, { action: i`unpacked`, name: artifactIdentity(registryName, id), current: '' });
+            }
+            p = undefined;
+            return;
+          }
+          if (percent) {
+            if (!p) {
+              p = bar.create(100, 0, { action: i`unpacking`, name: artifactIdentity(registryName, id), current: '' });
+            }
+            p.update(percent);
+          }
+        },
+        heartbeat: (text: string) => {
+          if (!p) {
+            p = bar.create(3, 0, { action: i`working`, name: artifactIdentity(registryName, id), current: '' });
+          }
+          p?.update((spinnerValue++) % 4, { action: i`working`, name: artifactIdentity(registryName, id), current: text });
+        }
+      }, options || {});
+      // remember what was actually installed
+      installed.set(artifact, actuallyInstalled);
+      if (actuallyInstalled) {
+        trackAcquire(artifact.id, artifact.version);
+      }
+    } catch (e: any) {
+      bar.stop();
+      debug(e);
+      debug(e.stack);
+      error(i`Error installing ${artifactIdentity(registryName, id)} - ${e} `);
+      return [false, installed, activation];
+    }
+
+    bar.stop();
+  }
+  return [true, installed, activation];
+}
+
+export async function activateArtifacts(session: Session, artifacts: Iterable<Artifact>) {
+  const activation = new Activation(session);
+  for (const artifact of artifacts) {
+    if (await artifact.isInstalled) {
+      await artifact.loadActivationSettings(activation);
+    }
+  }
+  return activation;
+}

ce/ce/cli/command-line.ts

@@ -0,0 +1,167 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { strict } from 'assert';
+import { tmpdir } from 'os';
+import { join, resolve } from 'path';
+import { i } from '../i18n';
+import { intersect } from '../util/intersect';
+import { Command } from './command';
+import { cmdSwitch } from './format';
+
+export type switches = {
+  [key: string]: Array<string>;
+}
+
+export interface Help {
+  readonly help: Array<string>;
+  readonly title: string;
+}
+
+class Ctx {
+  constructor(cmdline: CommandLine) {
+    this.os =
+      cmdline.isSet('windows') ? 'win32' :
+        cmdline.isSet('osx') ? 'darwin' :
+          cmdline.isSet('linux') ? 'linux' :
+            cmdline.isSet('freebsd') ? 'freebsd' :
+              process.platform;
+    this.arch = cmdline.isSet('x64') ? 'x64' :
+      cmdline.isSet('x86') ? 'x32' :
+        cmdline.isSet('arm') ? 'arm' :
+          cmdline.isSet('arm64') ? 'arm64' :
+            process.arch;
+  }
+
+  readonly os: string;
+  readonly arch: string;
+
+  get windows(): boolean {
+    return this.os === 'win32';
+  }
+
+  get linux(): boolean {
+    return this.os === 'linux';
+  }
+
+  get freebsd(): boolean {
+    return this.os === 'freebsd';
+  }
+
+  get osx(): boolean {
+    return this.os === 'darwin';
+  }
+
+  get x64(): boolean {
+    return this.arch === 'x64';
+  }
+
+  get x86(): boolean {
+    return this.arch === 'x32';
+  }
+
+  get arm(): boolean {
+    return this.arch === 'arm';
+  }
+
+  get arm64(): boolean {
+    return this.arch === 'arm64';
+  }
+}
+
+export function resolvePath(v: string | undefined) {
+  return v?.startsWith('.') ? resolve(v) : v;
+}
+
+export class CommandLine {
+  readonly commands = new Array<Command>();
+  readonly inputs = new Array<string>();
+  readonly switches: switches = {};
+  readonly context: Ctx & switches;
+
+  #home?: string;
+  get homeFolder() {
+    // home folder is determined by
+    // command line (--vcpkg_root, --vcpkg-root )
+    // environment (VCPKG_ROOT)
+    // default 1 $HOME/.vcpkg
+    // default 2 <tmpdir>/.vcpkg
+
+    // note, this does not create the folder, that would happen when the session is initialized.
+
+    return this.#home || (this.#home = resolvePath(
+      this.switches['vcpkg-root']?.[0] ||
+      this.switches['vcpkg_root']?.[0] ||
+      process.env['VCPKG_ROOT'] ||
+      join(process.env['HOME'] || process.env['USERPROFILE'] || tmpdir(), '.vcpkg')));
+  }
+
+  get force() {
+    return !!this.switches['force'];
+  }
+
+  get debug() {
+    return !!this.switches['debug'];
+  }
+
+  get fromVCPKG() {
+    return !!this.switches['from-vcpkg'];
+  }
+
+  get language() {
+    const l = this.switches['language'] || [];
+    strict.ok((l?.length || 0) < 2, i`Expected a single value for ${cmdSwitch('language')} - found multiple`);
+    return l[0] || Intl.DateTimeFormat().resolvedOptions().locale;
+  }
+
+  get allLanguages(): boolean {
+    const l = this.switches['all-languages'] || [];
+    strict.ok((l?.length || 0) < 2, i`Expected a single value for ${cmdSwitch('all-languages')} - found multiple`);
+    return !!l[0];
+  }
+
+  isSet(sw: string) {
+    const s = this.switches[sw];
+    if (s && s.last !== 'false') {
+      return true;
+    }
+    return false;
+  }
+
+  claim(sw: string) {
+    const v = this.switches[sw];
+    delete this.switches[sw];
+    return v;
+  }
+
+  addCommand(command: Command) {
+    this.commands.push(command);
+  }
+
+  /** parses the command line and returns the command that has been requested */
+  get command() {
+    return this.commands.find(cmd => cmd.command === this.inputs[0] || !!cmd.aliases.find(alias => alias === this.inputs[0]));
+  }
+
+  constructor(args: Array<string>) {
+    for (let i = 0; i < args.length; i++) {
+      const arg = args[i];
+      // eslint-disable-next-line prefer-const
+      let [, name, sep, value] = /^--([^=:]+)([=:])?(.+)?$/g.exec(arg) || [];
+      if (name) {
+        if (!value) {
+          if (i + 1 < args.length && !args[i + 1].startsWith('--')) {
+            // if you say --foo bar then bar is the value
+            value = args[++i];
+          }
+        }
+        this.switches[name] = this.switches[name] === undefined ? [] : this.switches[name];
+        this.switches[name].push(value);
+        continue;
+      }
+      this.inputs.push(arg);
+    }
+
+    this.context = intersect(new Ctx(this), this.switches);
+  }
+}

ce/ce/cli/command.ts

@@ -0,0 +1,80 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../i18n';
+import { Argument } from './argument';
+import { CommandLine, Help } from './command-line';
+import { blank, cli } from './constants';
+import { cmdSwitch, command, heading, optional } from './format';
+import { Switch } from './switch';
+import { Debug } from './switches/debug';
+import { Force } from './switches/force';
+
+/** @internal */
+
+export abstract class Command implements Help {
+  readonly abstract command: string;
+  readonly abstract argumentsHelp: Array<string>;
+
+  readonly switches = new Array<Switch>();
+  readonly arguments = new Array<Argument>();
+
+  readonly abstract seeAlso: Array<Help>;
+  readonly abstract aliases: Array<string>;
+
+  abstract get summary(): string;
+  abstract get description(): Array<string>;
+
+  readonly force = new Force(this);
+  readonly debug = new Debug(this);
+
+  get synopsis(): Array<string> {
+    return [
+      heading(i`Synopsis`, 2),
+      ` ${command(`${cli} ${this.command} ${this.arguments.map(each => `<${each.argument}>`).join(' ')}`)}${this.switches.flatMap(each => optional(`[--${each.switch}]`)).join(' ')}`,
+    ];
+  }
+
+  get title() {
+    return `${cli} ${this.command}`;
+  }
+
+  constructor(public commandLine: CommandLine) {
+    commandLine.addCommand(this);
+  }
+
+  get inputs() {
+    return this.commandLine.inputs.slice(1);
+  }
+
+  get help() {
+    return [
+      heading(this.title),
+      blank,
+      this.summary,
+      blank,
+      ...this.synopsis,
+      blank,
+      heading(i`Description`, 2),
+      blank,
+      ...this.description,
+      ...this.argumentsHelp,
+      ...(this.switches.length ? [
+        blank,
+        heading(i`Switches`, 2),
+        blank,
+        ...this.switches.flatMap(each => ` ${cmdSwitch(each.switch)}: ${each.help.join(' ')}`)
+      ] : []),
+      ...(this.seeAlso.length ? [
+        heading(i`See Also`, 2),
+        ...this.seeAlso.flatMap(each => each.title)
+      ] : []),
+    ];
+  }
+
+  async run() {
+    // do something
+    return true;
+  }
+
+}

ce/ce/cli/commands/acquire.ts

@@ -0,0 +1,84 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { countWhere } from '../../util/linq';
+import { installArtifacts, selectArtifacts, showArtifacts } from '../artifacts';
+import { Command } from '../command';
+import { blank } from '../constants';
+import { cmdSwitch } from '../format';
+import { debug, error, log, warning } from '../styling';
+import { Registry } from '../switches/registry';
+import { Version } from '../switches/version';
+import { WhatIf } from '../switches/whatIf';
+
+export class AcquireCommand extends Command {
+  readonly command = 'acquire';
+  readonly aliases = ['install'];
+  seeAlso = [];
+  argumentsHelp = [];
+  version = new Version(this);
+  whatIf = new WhatIf(this);
+  registrySwitch = new Registry(this);
+
+  get summary() {
+    return i`Acquire artifacts in the registry`;
+  }
+
+  get description() {
+    return [
+      i`This allows the consumer to acquire (download and unpack) artifacts. Artifacts must be activated to be used`,
+    ];
+  }
+
+  override async run() {
+    if (this.inputs.length === 0) {
+      error(i`No artifacts specified`);
+      return false;
+    }
+
+    const registries = await this.registrySwitch.loadRegistries(session);
+
+    const versions = this.version.values;
+    if (versions.length && this.inputs.length !== versions.length) {
+      error(i`Multiple packages specified, but not an equal number of ${cmdSwitch('version')} switches.`);
+      return false;
+    }
+
+    const artifacts = await selectArtifacts(new Map(this.inputs.map((v, i) => [v, versions[i] || '*'])), registries);
+
+    if (!artifacts) {
+      debug('No artifacts selected - stopping');
+      return false;
+    }
+
+    if (!await showArtifacts(artifacts.artifacts, this.commandLine)) {
+      warning(i`No artifacts are acquired`);
+      return false;
+    }
+
+    const numberOfArtifacts = await countWhere(artifacts.artifacts, async (artifact) => !(!this.commandLine.force && await artifact.isInstalled));
+
+    if (!numberOfArtifacts) {
+      log(blank);
+      log(i`All artifacts are already installed`);
+      return true;
+    }
+
+    debug(`Installing ${numberOfArtifacts} artifacts`);
+
+    const [success] = await installArtifacts(session, artifacts.artifacts, { force: this.commandLine.force, language: this.commandLine.language, allLanguages: this.commandLine.allLanguages });
+
+    if (success) {
+      log(blank);
+      log(i`${numberOfArtifacts} artifacts installed successfuly`);
+      return true;
+    }
+
+    log(blank);
+    log(i`Installation failed -- stopping`);
+
+    return false;
+  }
+}

ce/ce/cli/commands/activate.ts

@@ -0,0 +1,51 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Command } from '../command';
+import { projectFile } from '../format';
+import { activateProject } from '../project';
+import { debug, error } from '../styling';
+import { MSBuildProps } from '../switches/msbuild-props';
+import { Project } from '../switches/project';
+import { WhatIf } from '../switches/whatIf';
+
+export class ActivateCommand extends Command {
+  readonly command = 'activate';
+  readonly aliases = [];
+  seeAlso = [];
+  argumentsHelp = [];
+  whatIf = new WhatIf(this);
+  project: Project = new Project(this);
+  msbuildProps: MSBuildProps = new MSBuildProps(this);
+
+  get summary() {
+    return i`Activates the tools required for a project`;
+  }
+
+  get description() {
+    return [
+      i`This allows the consumer to Activate the tools required for a project. If the tools are not already installed, this will force them to be downloaded and installed before activation.`,
+    ];
+  }
+
+  override async run() {
+    const projectManifest = await this.project.manifest;
+
+    if (!projectManifest) {
+      error(i`Unable to find project in folder (or parent folders) for ${session.currentDirectory.fsPath}`);
+      return false;
+    }
+
+    debug(i`Deactivating project ${projectFile(projectManifest.metadata.context.file)}`);
+    await session.deactivate();
+
+    return await activateProject(projectManifest, {
+      force: this.commandLine.force,
+      allLanguages: this.commandLine.allLanguages,
+      language: this.commandLine.language,
+      msbuildProps: await this.msbuildProps.value
+    });
+  }
+}

ce/ce/cli/commands/add.ts

@@ -0,0 +1,90 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { selectArtifacts } from '../artifacts';
+import { Command } from '../command';
+import { cmdSwitch, projectFile } from '../format';
+import { activateProject } from '../project';
+import { debug, error } from '../styling';
+import { Project } from '../switches/project';
+import { Registry } from '../switches/registry';
+import { Version } from '../switches/version';
+import { WhatIf } from '../switches/whatIf';
+
+export class AddCommand extends Command {
+  readonly command = 'add';
+  readonly aliases = [];
+  seeAlso = [];
+  argumentsHelp = [];
+
+  version = new Version(this);
+  project: Project = new Project(this);
+  whatIf = new WhatIf(this);
+  registrySwitch = new Registry(this);
+
+
+  get summary() {
+    return i`Adds an artifact to the project`;
+  }
+
+  get description() {
+    return [
+      i`This allows the consumer to add an artifact to the project. This will activate the project as well.`,
+    ];
+  }
+
+  override async run() {
+    const projectManifest = await this.project.manifest;
+
+    if (!projectManifest) {
+      error(i`Unable to find project in folder (or parent folders) for ${session.currentDirectory.fsPath}`);
+      return false;
+    }
+
+    // pull in any registries that are on the command line
+    await this.registrySwitch.loadRegistries(session);
+
+    if (this.inputs.length === 0) {
+      error(i`No artifacts specified`);
+      return false;
+    }
+
+    const versions = this.version.values;
+    if (versions.length && this.inputs.length !== versions.length) {
+      error(i`Multiple artifacts specified, but not an equal number of ${cmdSwitch('version')} switches`);
+      return false;
+    }
+
+    const selections = new Map(this.inputs.map((v, i) => [v, versions[i] || '*']));
+    const selectedArtifacts = await selectArtifacts(selections, projectManifest.registries);
+
+    if (!selectedArtifacts) {
+      return false;
+    }
+
+    for (const [artifact, id, requested] of selectedArtifacts.values()) {
+      // make sure the registry is in the project
+      const registry = projectManifest.registries.getRegistry(artifact.registryUri);
+      if (!registry) {
+        const r = projectManifest.metadata.registries.add(artifact.registryId, artifact.registryUri, 'artifact');
+
+      }
+
+
+      // add the artifact to the project
+      const fulfilled = artifact.version.toString();
+      const v = requested !== fulfilled ? `${requested} ${fulfilled}` : fulfilled;
+      projectManifest.metadata.requires.set(artifact.reference, <any>v);
+    }
+
+    // write the file out.
+    await projectManifest.metadata.save();
+
+    debug(i`Deactivating project ${projectFile(projectManifest.metadata.context.file)}`);
+    await session.deactivate();
+
+    return await activateProject(projectManifest, this.commandLine);
+  }
+}

ce/ce/cli/commands/apply-vsman.ts

@@ -0,0 +1,139 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { MetadataFile } from '../../amf/metadata-file';
+import { acquireArtifactFile } from '../../fs/acquire';
+import { FileType } from '../../fs/filesystem';
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Session } from '../../session';
+import { Uri } from '../../util/uri';
+import { templateAmfApplyVsManifestInformation } from '../../willow/template-amf';
+import { parseVsManFromChannel, VsManDatabase } from '../../willow/willow';
+import { Command } from '../command';
+import { log } from '../styling';
+import { Switch } from '../switch';
+
+class ChannelUri extends Switch {
+  readonly switch = 'channel';
+
+  get help() {
+    return [
+      i`The URI to the Visual Studio channel to apply.`
+    ];
+  }
+}
+
+class RepoRoot extends Switch {
+  readonly switch = 'repo';
+
+  get help() {
+    return [
+      i`The directory path to the root of the repo into which artifact metadata is to be generated.`
+    ];
+  }
+}
+
+export class ApplyVsManCommand extends Command {
+  readonly command = 'z-apply-vsman';
+  readonly seeAlso = [];
+  readonly argumentsHelp = [];
+  readonly aliases = [];
+
+  readonly channelUri = new ChannelUri(this);
+  readonly repoRoot = new RepoRoot(this);
+
+  get summary() {
+    return i`Apply Visual Studio Channel (.vsman) information to a prototypical artifact metadata.`;
+  }
+
+  get description() {
+    return [
+      i`This is used to mint artifacts metadata exactly corresponding to a release state in a Visual Studio channel.`,
+    ];
+  }
+
+  /**
+   * Process an input file.
+   */
+  static async processFile(session: Session, inputUri: Uri, repoRoot: Uri, vsManLookup: VsManDatabase) {
+    const inputPath = inputUri.fsPath;
+    session.channels.debug(i`Processing ${inputPath}...`);
+    const inputContent = await inputUri.readUTF8();
+    const outputContent = templateAmfApplyVsManifestInformation(session, inputPath, inputContent, vsManLookup);
+    if (!outputContent) {
+      session.channels.warning(i`Skipped processing ${inputPath}`);
+      return 0;
+    }
+
+    const outputAmf = await MetadataFile.parseConfiguration(inputPath, outputContent, session);
+    if (!outputAmf.isValid) {
+      const errors = outputAmf.validationErrors.join('\n');
+      session.channels.warning(i`After transformation, ${inputPath} did not result in a valid AMF; skipping:\n${outputContent}\n${errors}`);
+      return 0;
+    }
+
+    const outputId = outputAmf.info.id;
+    const outputIdLast = outputId.slice(outputId.lastIndexOf('/'));
+    const outputVersion = outputAmf.info.version;
+    const outputRelativePath = `${outputId}/${outputIdLast}-${outputVersion}.yaml`;
+    const outputFullPath = repoRoot.join(outputRelativePath);
+    let doWrite = true;
+    try {
+      const outputExistingContent = await outputFullPath.readUTF8();
+      if (outputExistingContent === outputContent) {
+        doWrite = false;
+      } else {
+        session.channels.warning(i`After transformation, ${inputPath} results in ${outputFullPath.toString()} which already exists; overwriting.`);
+      }
+    } catch {
+      // nothing to do
+    }
+
+    if (doWrite) {
+      await outputFullPath.writeUTF8(outputContent);
+    }
+
+    session.channels.debug(i`-> ${outputFullPath.toString()}`);
+    return 1;
+  }
+
+  /**
+   * Process an input file or directory, recursively.
+   */
+  static async processInput(session: Session, inputDirectoryEntry: [Uri, FileType], repoRoot: Uri, vsManLookup: VsManDatabase): Promise<number> {
+    if ((inputDirectoryEntry[1] & FileType.Directory) !== 0) {
+      let total = 0;
+      for (const child of await inputDirectoryEntry[0].readDirectory()) {
+        total += await ApplyVsManCommand.processInput(session, child, repoRoot, vsManLookup);
+      }
+
+      return total;
+    } else if ((inputDirectoryEntry[1] & FileType.File) !== 0) {
+      return await ApplyVsManCommand.processFile(session, inputDirectoryEntry[0], repoRoot, vsManLookup);
+    }
+
+    return 0;
+  }
+
+  override async run() {
+    const channelUriStr = this.channelUri.requiredValue;
+    const repoRoot = session.fileSystem.file(this.repoRoot.requiredValue);
+    log(i`Downloading channel manifest from ${channelUriStr}`);
+    const channelUriUri = session.parseUri(channelUriStr);
+    const channelFile = await acquireArtifactFile(session, [channelUriUri], 'channel.chman', {});
+    const vsManPayload = parseVsManFromChannel(await channelFile.readUTF8());
+    log(i`Downloading Visual Studio manifest version ${vsManPayload.version} (${vsManPayload.url})`);
+    const vsManUri = await acquireArtifactFile(session, [session.parseUri(vsManPayload.url)], vsManPayload.fileName, {});
+    const vsManLookup = new VsManDatabase(await vsManUri.readUTF8());
+    let totalProcessed = 0;
+    for (const inputPath of this.inputs) {
+      const inputUri = session.fileSystem.file(inputPath);
+      const inputStat = await inputUri.stat();
+      totalProcessed += await ApplyVsManCommand.processInput(session, [inputUri, inputStat.type], repoRoot, vsManLookup);
+    }
+
+    session.channels.message(i`Processed ${totalProcessed} templates.`);
+    return true;
+  }
+}

ce/ce/cli/commands/cache.ts

@@ -0,0 +1,62 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { basename } from 'path';
+import { FileType } from '../../fs/filesystem';
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Uri } from '../../util/uri';
+import { Command } from '../command';
+import { Table } from '../markdown-table';
+import { log } from '../styling';
+import { Clear } from '../switches/clear';
+import { WhatIf } from '../switches/whatIf';
+
+export class CacheCommand extends Command {
+  readonly command = 'cache';
+  readonly aliases = [];
+  seeAlso = [];
+  argumentsHelp = [];
+  clear = new Clear(this);
+  whatIf = new WhatIf(this);
+
+  get summary() {
+    return i`Manages the download cache`;
+  }
+
+  get description() {
+    return [
+      i`Manages the download cache.`,
+    ];
+  }
+
+  override async run() {
+    if (this.clear.active) {
+      await session.cache.delete({ recursive: true });
+      await session.cache.createDirectory();
+      log(i`Cache folder cleared (${session.cache.fsPath}) `);
+      return true;
+    }
+    let files: Array<[Uri, FileType]> = [];
+    try {
+      files = await session.cache.readDirectory();
+    } catch {
+      // shh
+    }
+
+    if (!files.length) {
+      log('The download cache is empty');
+      return true;
+    }
+
+    const table = new Table('File', 'Size', 'Date');
+    for (const [file, type] of files) {
+      const stat = await file.stat();
+      table.push(basename(file.fsPath), stat.size.toString(), new Date(stat.mtime).toString());
+    }
+    log(table.toString());
+    log();
+
+    return true;
+  }
+}

ce/ce/cli/commands/clean.ts

@@ -0,0 +1,78 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Command } from '../command';
+import { debug, log } from '../styling';
+import { Switch } from '../switch';
+import { WhatIf } from '../switches/whatIf';
+
+export class All extends Switch {
+  switch = 'all';
+  get help() {
+    return [
+      i`cleans out everything (cache, installed artifacts)`
+    ];
+  }
+}
+
+export class Cache extends Switch {
+  switch = 'cache';
+  get help() {
+    return [
+      i`cleans out the cache`
+    ];
+  }
+}
+
+export class Artifacts extends Switch {
+  switch = 'artifacts';
+  get help() {
+    return [
+      i`removes all the artifacts that are installed`
+    ];
+  }
+}
+
+export class CleanCommand extends Command {
+  readonly command = 'clean';
+  readonly aliases = [];
+  seeAlso = [];
+  argumentsHelp = [];
+  all = new All(this);
+  artifacts = new Artifacts(this);
+  cache = new Cache(this);
+  whatIf = new WhatIf(this);
+
+  get summary() {
+    return i`cleans up`;
+  }
+
+  get description() {
+    return [
+      i`Allows the user to clean out the cache, installed artifacts, etc.`,
+    ];
+  }
+
+  override async run() {
+
+    if (this.all.active || this.artifacts.active) {
+      // if we're removing artifacts
+      debug(i`Deactivating project`);
+      await session.deactivate();
+
+      await session.installFolder.delete({ recursive: true });
+      await session.installFolder.createDirectory();
+      log(i`Installed Artifact folder cleared (${session.installFolder.fsPath}) `);
+    }
+
+    if (this.all.active || this.cache.active) {
+      await session.cache.delete({ recursive: true });
+      await session.cache.createDirectory();
+      log(i`Cache folder cleared (${session.cache.fsPath}) `);
+    }
+
+    return true;
+  }
+}

ce/ce/cli/commands/deactivate.ts

@@ -0,0 +1,41 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Command } from '../command';
+import { projectFile } from '../format';
+import { log } from '../styling';
+import { Project } from '../switches/project';
+import { WhatIf } from '../switches/whatIf';
+
+export class DeactivateCommand extends Command {
+  readonly command = 'deactivate';
+  readonly aliases = [];
+  seeAlso = [];
+  argumentsHelp = [];
+  project = new Project(this);
+  whatIf = new WhatIf(this);
+
+  get summary() {
+    return i`Deactivates the current session`;
+  }
+
+  get description() {
+    return [
+      i`This allows the consumer to remove environment settings for the currently active session.`,
+    ];
+  }
+
+  override async run() {
+    const project = await this.project.value;
+    if (!project) {
+      return false;
+    }
+
+    log(i`Deactivating project ${projectFile(project)}`);
+    await session.deactivate();
+
+    return true;
+  }
+}

ce/ce/cli/commands/delete.ts

@@ -0,0 +1,42 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Command } from '../command';
+import { Version } from '../switches/version';
+import { WhatIf } from '../switches/whatIf';
+
+export class DeleteCommand extends Command {
+  readonly command = 'delete';
+  readonly aliases = ['uninstall'];
+  seeAlso = [];
+  argumentsHelp = [];
+  version = new Version(this);
+  whatIf = new WhatIf(this);
+
+  get summary() {
+    return i`Deletes an artifact from the artifact folder`;
+  }
+
+  get description() {
+    return [
+      i`This allows the consumer to remove an artifact from disk.`,
+    ];
+  }
+
+  override async run() {
+    const artifacts = await session.getInstalledArtifacts();
+    for (const input of this.inputs) {
+      for (const { artifact, id, folder } of artifacts) {
+        if (input === id) {
+          if (await folder.exists()) {
+            session.channels.message(i`Deleting artifact ${id} from ${folder.fsPath}`);
+            await artifact.uninstall();
+          }
+        }
+      }
+    }
+    return true;
+  }
+}

ce/ce/cli/commands/find.ts

@@ -0,0 +1,58 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Registries } from '../../registries/registries';
+import { Command } from '../command';
+import { artifactIdentity } from '../format';
+import { Table } from '../markdown-table';
+import { debug, log } from '../styling';
+import { Project } from '../switches/project';
+import { Registry } from '../switches/registry';
+import { Version } from '../switches/version';
+
+export class FindCommand extends Command {
+  readonly command = 'find';
+  readonly aliases = ['search'];
+  seeAlso = [];
+  argumentsHelp = [];
+
+  version = new Version(this);
+  registrySwitch = new Registry(this);
+  project = new Project(this);
+
+  get summary() {
+    return i`Find artifacts in the registry`;
+  }
+
+  get description() {
+    return [
+      i`This allows the user to find artifacts based on some criteria.`,
+    ];
+  }
+
+  override async run() {
+    // load registries (from the current project too if available)
+    let registries: Registries = await this.registrySwitch.loadRegistries(session);
+    registries = (await this.project.manifest)?.registries ?? registries;
+
+    debug(`using registries: ${[...registries].map(([registry, registryNames]) => registryNames[0]).join(', ')}`);
+    const table = new Table('Artifact', 'Version', 'Summary');
+
+    for (const each of this.inputs) {
+      for (const [registry, id, artifacts] of await registries.search({ idOrShortName: each, version: this.version.value })) {
+        const latest = artifacts[0];
+        if (!latest.metadata.info.dependencyOnly) {
+          const name = artifactIdentity(latest.registryId, id, latest.shortName);
+          table.push(name, latest.metadata.info.version, latest.metadata.info.summary || '');
+        }
+      }
+    }
+
+    log(table.toString());
+    log();
+    return true;
+  }
+}

ce/ce/cli/commands/help.ts

@@ -0,0 +1,84 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { Argument } from '../argument';
+import { Command } from '../command';
+import { blank, cli } from '../constants';
+import { command as formatCommand, heading, hint } from '../format';
+import { error, indent, log } from '../styling';
+
+class CommandName extends Argument {
+  argument = 'command';
+
+  get help() {
+    return [
+      i`the name of the command for which you want help`
+    ];
+  }
+}
+
+/**@internal */
+export class HelpCommand extends Command {
+  readonly command = 'help';
+  readonly aliases = [];
+  seeAlso = [];
+  commandName: CommandName = new CommandName(this);
+
+  get argumentsHelp() {
+    return [indent(i` <${this.commandName.argument}> : ${this.commandName.help.join(' ')}`)];
+  }
+
+  get summary() {
+    return i`get help on ${cli} or one of the commands`;
+  }
+
+  get description() {
+    return [
+      i`Gets detailed help on ${cli}, or one of the commands`,
+      blank,
+      i`Arguments:`
+    ];
+  }
+
+  override async run() {
+    const cmd = ['-h', '-help', '-?', '/?'].find(each => (this.commandLine.inputs.indexOf(each) > -1)) ? this.commandLine.inputs[0] : this.commandLine.inputs[1];
+    // did they ask for help on a command?
+
+    if (cmd) {
+      const target = this.commandLine.commands.find(each => each.command === cmd);
+      if (target) {
+        log(target.help.join('\n'));
+        log(blank);
+        return true;
+      }
+
+      // I don't know the command
+      error(i`Unrecognized command '${cmd}'`);
+      log(hint(i`Use ${formatCommand(`${cli} ${this.command}`)} to get the list of available commands`));
+      return false;
+    }
+
+    // general help. return the general help info
+
+    log(heading(i`Usage`, 2));
+    log(blank);
+    log(indent(i`${cli} COMMAND <arguments> [--switches]`));
+    log(blank);
+
+    log(heading(i`Available ${cli} commands:`, 2));
+    log(blank);
+    const max = Math.max(...this.commandLine.commands.map(each => each.command.length));
+    for (const command of this.commandLine.commands) {
+      if (command.command.startsWith('z-')) {
+        // don't show internal commands
+        continue;
+      }
+      log(indent(i`${formatCommand(command.command.padEnd(max))} : ${command.summary}`));
+    }
+
+    log(blank);
+
+    return true;
+  }
+}

ce/ce/cli/commands/list.ts

@@ -0,0 +1,48 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Command } from '../command';
+import { artifactIdentity } from '../format';
+import { Table } from '../markdown-table';
+import { log } from '../styling';
+import { Installed } from '../switches/installed';
+
+export class ListCommand extends Command {
+  readonly command = 'list';
+  readonly aliases = ['show'];
+  seeAlso = [];
+  argumentsHelp = [];
+  installed = new Installed(this);
+
+  get summary() {
+    return i`Lists the artifacts`;
+  }
+
+  get description() {
+    return [
+      i`This allows the consumer to list artifacts.`,
+    ];
+  }
+
+  override async run() {
+    if (this.installed.active) {
+      const artifacts = await session.getInstalledArtifacts();
+      const table = new Table('Artifact', 'Version', 'Summary');
+
+      for (const { artifact, id, folder } of artifacts) {
+
+        const name = artifactIdentity('<registry-name-goes-here>', id); //todo: fixme
+        table.push(name, artifact.version, artifact.metadata.info.summary || '');
+      }
+      log(table.toString());
+      log();
+    }
+    else {
+      log('use --installed for now');
+    }
+
+    return true;
+  }
+}

ce/ce/cli/commands/new.ts

@@ -0,0 +1,40 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { MetadataFile } from '../../amf/metadata-file';
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Command } from '../command';
+import { project } from '../constants';
+import { log } from '../styling';
+import { WhatIf } from '../switches/whatIf';
+
+export class NewCommand extends Command {
+  readonly command = 'new';
+  readonly aliases = [];
+  seeAlso = [];
+  argumentsHelp = [];
+  whatIf = new WhatIf(this);
+
+  get summary() {
+    return i`Creates a new project file`;
+  }
+
+  get description() {
+    return [
+      i`This allows the consumer create a new project file ('${project}').`,
+    ];
+  }
+
+  override async run() {
+    if (await session.currentDirectory.exists(project)) {
+      log(i`The folder at ${session.currentDirectory.fsPath} already contains a project file '${project}'`);
+      return false;
+    }
+    const prjFile = session.currentDirectory.join(project);
+
+    await (await MetadataFile.parseConfiguration(prjFile.toString(), '# Environment configuration\n', session)).save(session.currentDirectory.join(project));
+
+    return true;
+  }
+}

ce/ce/cli/commands/regenerate-index.ts

@@ -0,0 +1,82 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { Registry } from '../../artifacts/registry';
+import { registryIndexFile } from '../../constants';
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Registries } from '../../registries/registries';
+import { Uri } from '../../util/uri';
+import { Command } from '../command';
+import { cli } from '../constants';
+import { error, log, writeException } from '../styling';
+import { Project } from '../switches/project';
+import { Registry as RegSwitch } from '../switches/registry';
+import { WhatIf } from '../switches/whatIf';
+
+export class RegenerateCommand extends Command {
+  readonly command = 'regenerate';
+  project = new Project(this);
+  readonly aliases = ['regen'];
+  readonly regSwitch = new RegSwitch(this, { required: true });
+  seeAlso = [];
+  argumentsHelp = [];
+
+  whatIf = new WhatIf(this);
+  get summary() {
+    return i`regenerate the index for a registry`;
+  }
+
+  get description() {
+    return [
+      i`This allows the user to regenerate the ${registryIndexFile} files for a ${cli} registry.`,
+    ];
+  }
+
+  override async run() {
+    let registries: Registries = await this.regSwitch.loadRegistries(session);
+    registries = (await this.project.manifest)?.registries ?? registries;
+
+    for (const registryNameOrLocation of this.inputs) {
+      let registry: Registry | undefined;
+      try {
+        if (registries.has(registryNameOrLocation)) {
+          // check for named registries first.
+          registry = registries.getRegistry(registryNameOrLocation);
+          await registry?.load();
+        } else {
+          // see if the name is a location
+          const location = await session.parseLocation(registryNameOrLocation);
+          registry = location ?
+            session.loadRegistry(location, 'artifact') :  // a folder
+            registries.getRegistry(registryNameOrLocation); // a registry name or other location.
+        }
+        if (registry) {
+          if (Uri.isInvalid(registry.location)) {
+            error(i`Registry: '${registryNameOrLocation}' does not have an index to regenerate.`);
+            return false;
+          }
+          log(i`Regenerating index for ${registry.location.formatted}`);
+          await registry.regenerate();
+          if (registry.count) {
+            await registry.save();
+            log(i`Regeneration complete. Index contains ${registry.count} metadata files`);
+            continue;
+          }
+          // looks like  the registry contained no items
+          error(i`Registry: '${registry.location.formatted}' contains no artifacts.`);
+          continue;
+        }
+
+        error(i`Unrecognized registry: ${registryNameOrLocation}`);
+        return false;
+
+      } catch (e) {
+        log(i`Regeneration failed for ${registryNameOrLocation.toString()}`);
+        writeException(e);
+        return false;
+      }
+    }
+    return true;
+  }
+}

ce/ce/cli/commands/remove.ts

@@ -0,0 +1,64 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Command } from '../command';
+import { projectFile } from '../format';
+import { activateProject } from '../project';
+import { debug, error, log } from '../styling';
+import { Project } from '../switches/project';
+import { WhatIf } from '../switches/whatIf';
+
+export class RemoveCommand extends Command {
+  readonly command = 'remove';
+  readonly aliases = [];
+  seeAlso = [];
+  argumentsHelp = [];
+  whatIf = new WhatIf(this);
+  project: Project = new Project(this);
+
+  get summary() {
+    return i`Removes an artifact from a project`;
+  }
+
+  get description() {
+    return [
+      i`This allows the consumer to remove an artifact from the project. Forces reactivation in this window.`,
+    ];
+  }
+
+  override async run() {
+    const projectManifest = await this.project.manifest;
+
+    if (!projectManifest) {
+      error(i`Unable to find project in folder (or parent folders) for ${session.currentDirectory.fsPath}`);
+      return false;
+    }
+
+    if (this.inputs.length === 0) {
+      error(i`No artifacts specified`);
+      return false;
+    }
+
+
+    const req = projectManifest.metadata.requires.keys;
+    for (const input of this.inputs) {
+      if (req.indexOf(input) !== -1) {
+        projectManifest.metadata.requires.delete(input);
+        log(i`Removing ${input} from project manifest`);
+      } else {
+        error(i`unable to find artifact ${input} in the project manifest`);
+        return false;
+      }
+    }
+
+    // write the file out.
+    await projectManifest.metadata.save();
+
+    debug(`Deactivating project ${projectFile(projectManifest.metadata.context.file)}`);
+    await session.deactivate();
+
+    return await activateProject(projectManifest, this.commandLine);
+  }
+}

ce/ce/cli/commands/update.ts

@@ -0,0 +1,91 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+
+import { Registry } from '../../artifacts/registry';
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { RemoteFileUnavailable } from '../../util/exceptions';
+import { Command } from '../command';
+import { CommandLine } from '../command-line';
+import { count } from '../format';
+import { error, log, writeException } from '../styling';
+import { Registry as RegSwitch } from '../switches/registry';
+import { WhatIf } from '../switches/whatIf';
+
+export class UpdateCommand extends Command {
+  readonly command = 'update';
+  readonly aliases = [];
+  seeAlso = [];
+  argumentsHelp = [];
+  whatIf = new WhatIf(this);
+  registrySwitch = new RegSwitch(this);
+
+  get summary() {
+    return i`update the registry from the remote`;
+  }
+
+  get description() {
+    return [
+      i`This downloads the latest contents of the registry from the remote service.`,
+    ];
+  }
+
+  override async run() {
+    const registries = await this.registrySwitch.loadRegistries(session);
+
+    // process named registries
+    for (let registryName of this.inputs) {
+      if (registryName.indexOf(':') !== -1) {
+        registryName = session.parseUri(registryName).toString();
+      }
+      const registry = registries.getRegistryWithNameOrLocation(registryName);
+      if (registry) {
+        try {
+          log(i`Downloading registry data`);
+          await registry.update();
+          await registry.load();
+          log(i`Updated ${registryName}. registry contains ${count(registry.count)} metadata files`);
+        } catch (e) {
+          if (e instanceof RemoteFileUnavailable) {
+            log(i`Unable to download registry snapshot`);
+            return false;
+          }
+          writeException(e);
+          return false;
+        }
+      } else {
+        error(i`Unable to find registry ${registryName}`);
+      }
+    }
+
+    return true;
+  }
+
+  static async update(registry: Registry) {
+    log(i`Artifact registry data is not loaded`);
+    log(i`Attempting to update artifact registry`);
+    const update = new UpdateCommand(new CommandLine([]));
+
+    let success = true;
+    try {
+      success = await update.run();
+    } catch (e) {
+      writeException(e);
+      success = false;
+    }
+    if (!success) {
+      error(i`Unable to load registry index`);
+      return false;
+    }
+    try {
+      await registry.load();
+    } catch (e) {
+      writeException(e);
+      // it just doesn't want to load.
+      error(i`Unable to load registry index`);
+      return false;
+    }
+    return true;
+  }
+}

ce/ce/cli/commands/use.ts

@@ -0,0 +1,79 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Registries } from '../../registries/registries';
+import { installArtifacts, selectArtifacts, showArtifacts } from '../artifacts';
+import { Command } from '../command';
+import { cmdSwitch } from '../format';
+import { error, log, warning } from '../styling';
+import { MSBuildProps } from '../switches/msbuild-props';
+import { Project } from '../switches/project';
+import { Registry } from '../switches/registry';
+import { Version } from '../switches/version';
+import { WhatIf } from '../switches/whatIf';
+
+export class UseCommand extends Command {
+  readonly command = 'use';
+  readonly aliases = [];
+  seeAlso = [];
+  argumentsHelp = [];
+  version = new Version(this);
+  whatIf = new WhatIf(this);
+  registrySwitch = new Registry(this);
+  project = new Project(this);
+  msbuildProps = new MSBuildProps(this);
+
+  get summary() {
+    return i`Instantly activates an artifact outside of the project`;
+  }
+
+  get description() {
+    return [
+      i`This will instantly activate an artifact .`,
+    ];
+  }
+
+  override async run() {
+    if (this.inputs.length === 0) {
+      error(i`No artifacts specified`);
+      return false;
+    }
+
+    // load registries (from the current project too if available)
+    let registries: Registries = await this.registrySwitch.loadRegistries(session);
+    registries = (await this.project.manifest)?.registries ?? registries;
+
+    const versions = this.version.values;
+    if (versions.length && this.inputs.length !== versions.length) {
+      error(i`Multiple packages specified, but not an equal number of ${cmdSwitch('version')} switches`);
+      return false;
+    }
+
+    const selections = new Map(this.inputs.map((v, i) => [v, versions[i] || '*']));
+    const artifacts = await selectArtifacts(selections, registries);
+
+    if (!artifacts) {
+      return false;
+    }
+
+    if (!await showArtifacts(artifacts.artifacts, this.commandLine)) {
+      warning(i`No artifacts are being acquired`);
+      return false;
+    }
+
+    const [success, artifactStatus, activation] = await installArtifacts(session, artifacts.artifacts, { force: this.commandLine.force, language: this.commandLine.language, allLanguages: this.commandLine.allLanguages });
+    if (success) {
+      log(i`Activating individual artifacts`);
+      await session.setActivationInPostscript(activation, false);
+      const propsFile = this.msbuildProps.value;
+      if (propsFile) {
+        await propsFile.writeUTF8(activation.generateMSBuild(artifactStatus.keys()));
+      }
+    } else {
+      return false;
+    }
+    return true;
+  }
+}

ce/ce/cli/commands/version.ts

@@ -0,0 +1,111 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { strict } from 'assert';
+import { parse } from 'semver';
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Version } from '../../version';
+import { Command } from '../command';
+import { cli, product } from '../constants';
+import { debug, error, log } from '../styling';
+import { Switch } from '../switch';
+
+class Check extends Switch {
+  switch = 'check';
+  get help() {
+    return [
+      i`check to see if a newer version of ${cli} is available`
+    ];
+  }
+}
+
+class Update extends Switch {
+  switch = 'update';
+  get help() {
+    return [
+      i`will update the current installation of ${cli} if a newer version is available`
+    ];
+  }
+}
+
+export class VersionCommand extends Command {
+  readonly command = 'version';
+  readonly aliases = ['ver'];
+  seeAlso = [];
+  argumentsHelp = [];
+  check = new Check(this);
+  update = new Update(this);
+
+  versionUrl = session.parseUri('https://aka.ms/vcpkg-ce.version');
+
+  get summary() {
+    return i`manage the version of ${cli}`;
+  }
+
+  get description() {
+    return [
+      i`This allows the user to get the current verison information for ${cli}`,
+      i`as well as checking if a new version is available, and can upgrade the current installation to the latest version.`,
+    ];
+  }
+
+  private async getRemoteVersion() {
+    const version = await this.versionUrl.readUTF8();
+    const semver = parse(version.trim());
+    strict.ok(semver, i`Unable to parse version ${version}`);
+
+    return semver;
+  }
+
+  override async run() {
+    if (this.update.active) {
+      // check for a new version, and update if necessary
+      debug(i`checking to see if there is a new version of the ${cli}, and updating if there is`);
+      try {
+        const semver = await this.getRemoteVersion();
+
+        if (semver.compare(Version) > 0) {
+          // we can update the tool.
+          debug('An update is available, we can install it. ');
+          debug('(we can not do it yet, waiting for download support');
+        }
+
+      } catch (err) {
+        error('Failed to get latest version number');
+        return false;
+      }
+      return true;
+    }
+
+    if (this.check.active) {
+      // check for a new version
+      debug(i`checking to see if there is a new version of the ${cli}`);
+      try {
+        const semver = await this.getRemoteVersion();
+
+        if (semver.compare(Version) > 0) {
+          log(i`There is a new version (${semver.version}) of ${cli} available`);
+        }
+        return true;
+      } catch (err) {
+        if (err instanceof Error) {
+          error(i`Failed to get latest version number. (${err.message})`);
+          log(err.stack || '');
+        }
+      }
+      return false;
+    }
+
+    // dump version information
+    log(i`${product} version information\n`);
+    log(i`  version: ${Version} `);
+
+
+    // Make the NOTICE and LICENSE files discoverable. NOTICE is generated during the official build.
+    log(i`Usage of vcpkg-ce is subject to license terms available at ${session.homeFolder.join('LICENSE.txt').fsPath}`);
+    log(i`Third-party license information is available at ${session.homeFolder.join('NOTICE.txt').fsPath}`);
+    return true;
+  }
+
+}

ce/ce/cli/constants.ts

@@ -0,0 +1,7 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+export const cli = 'ce';
+export const product = 'vcpkg-ce';
+export const project = 'vcpkg-configuration.json';
+export const blank = '\n';

ce/ce/cli/format.ts

@@ -0,0 +1,55 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { bold, cyan, gray, green, greenBright, grey, underline, whiteBright, yellowBright } from 'chalk';
+import { Uri } from '../util/uri';
+
+export function projectFile(uri: Uri): string {
+  return cyan(uri.fsPath);
+}
+
+export function artifactIdentity(registryName: string, identity: string, alias?: string) {
+  if (alias) {
+    return `${registryName}:${identity.substr(0, identity.length - alias.length)}${yellowBright(alias)}`;
+  }
+  return yellowBright(identity);
+}
+
+export function artifactReference(registryName: string, identity: string, version: string) {
+  return `${artifactIdentity(registryName, identity)}-v${gray(version)}`;
+}
+
+export function heading(text: string, level = 1) {
+  switch (level) {
+    case 1:
+      return `${underline.bold(text)}\n`;
+    case 2:
+      return `${greenBright(text)}\n`;
+    case 3:
+      return `${green(text)}\n`;
+  }
+  return `${bold(text)}\n`;
+}
+
+export function optional(text: string) {
+  return gray(text);
+}
+export function cmdSwitch(text: string) {
+  return optional(`--${text}`);
+}
+
+export function command(text: string) {
+  return whiteBright.bold(text);
+}
+
+export function hint(text: string) {
+  return green.dim(text);
+}
+
+export function count(num: number) {
+  return grey(`${num}`);
+}
+
+export function position(text: string) {
+  return grey(`${text}`);
+}

ce/ce/cli/markdown-table.ts

@@ -0,0 +1,21 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { strict } from 'assert';
+
+export class Table {
+  private readonly rows = new Array<string>();
+  private numberOfColumns = 0;
+  constructor(...columnNames: Array<string>) {
+    this.numberOfColumns = columnNames.length;
+    this.rows.push(`|${columnNames.join('|')}|`);
+    this.rows.push(`${'|--'.repeat(this.numberOfColumns)}|`);
+  }
+  push(...values: Array<string>) {
+    strict.equal(values.length, this.numberOfColumns, 'unexpected number of arguments in table row');
+    this.rows.push(`|${values.join('|')}|`);
+  }
+  toString() {
+    return this.rows.join('\n');
+  }
+}

ce/ce/cli/project.ts

@@ -0,0 +1,65 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { ArtifactMap, ProjectManifest } from '../artifacts/artifact';
+import { i } from '../i18n';
+import { trackActivation } from '../insights';
+import { session } from '../main';
+import { Uri } from '../util/uri';
+import { installArtifacts, showArtifacts } from './artifacts';
+import { blank } from './constants';
+import { projectFile } from './format';
+import { error, log } from './styling';
+
+class ActivationOptions {
+  force?: boolean;
+  allLanguages?: boolean;
+  language?: string;
+  msbuildProps?: Uri;
+}
+
+export async function openProject(location: Uri): Promise<ProjectManifest> {
+  // load the project
+  return new ProjectManifest(session, await session.openManifest(location));
+}
+
+export async function activate(artifacts: ArtifactMap, options?: ActivationOptions) {
+  // install the items in the project
+  const [success, artifactStatus, activation] = await installArtifacts(session, artifacts.artifacts, options);
+
+  if (success) {
+    // create an MSBuild props file if indicated by the user
+    const propsFile = options?.msbuildProps;
+    if (propsFile) {
+      await propsFile.writeUTF8(activation.generateMSBuild(artifactStatus.keys()));
+    }
+
+    // activate all the tools in the project
+    await session.setActivationInPostscript(activation);
+  }
+
+  return success;
+}
+
+export async function activateProject(project: ProjectManifest, options?: ActivationOptions) {
+  // track what got installed
+  const artifacts = await project.resolveDependencies();
+
+  // print the status of what is going to be activated.
+  if (!await showArtifacts(artifacts.artifacts, options)) {
+    error(i`Unable to activate project`);
+    return false;
+  }
+
+  if (await activate(artifacts, options)) {
+    trackActivation();
+    log(blank);
+    log(i`Project ${projectFile(project.metadata.context.folder)} activated`);
+    return true;
+  }
+
+  log(blank);
+  log(i`Failed to activate project ${projectFile(project.metadata.context.folder)}`);
+
+  return false;
+}

ce/ce/cli/styling.ts

@@ -0,0 +1,104 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { blue, cyan, gray, green, red, white, yellow } from 'chalk';
+import * as renderer from 'marked-terminal';
+import { argv } from 'process';
+import { Session } from '../session';
+import { CommandLine } from './command-line';
+
+// eslint-disable-next-line @typescript-eslint/no-var-requires
+const marked = require('marked');
+
+function formatTime(t: number) {
+  return (
+    t < 3600000 ? [Math.floor(t / 60000) % 60, Math.floor(t / 1000) % 60, t % 1000] :
+      t < 86400000 ? [Math.floor(t / 3600000) % 24, Math.floor(t / 60000) % 60, Math.floor(t / 1000) % 60, t % 1000] :
+        [Math.floor(t / 86400000), Math.floor(t / 3600000) % 24, Math.floor(t / 60000) % 60, Math.floor(t / 1000) % 60, t % 1000]).map(each => each.toString().padStart(2, '0')).join(':').replace(/(.*):(\d)/, '$1.$2');
+}
+
+// setup markdown renderer
+marked.setOptions({
+  renderer: new renderer({
+    tab: 2,
+    emoji: true,
+    showSectionPrefix: false,
+    firstHeading: green.underline.bold,
+    heading: green.underline,
+    codespan: white.bold,
+    link: blue.bold,
+    href: blue.bold.underline,
+    code: gray,
+    tableOptions: {
+      chars: {
+        'top': '', 'top-mid': '', 'top-left': '', 'top-right': ''
+        , 'bottom': '', 'bottom-mid': '', 'bottom-left': '', 'bottom-right': ''
+        , 'left': '', 'left-mid': '', 'mid': '', 'mid-mid': ''
+        , 'right': '', 'right-mid': '', 'middle': ''
+      }
+    }
+  }),
+  gfm: true,
+});
+
+export function indent(text: string): string
+export function indent(text: Array<string>): Array<string>
+export function indent(text: string | Array<string>): string | Array<string> {
+  if (Array.isArray(text)) {
+    return text.map(each => indent(each));
+  }
+  return `  ${text}`;
+}
+
+function md(text = '', session?: Session): string {
+  if (text) {
+    text = marked.marked(`${text}`.replace(/\\\./g, '\\\\.')); // work around md messing up paths with .\ in them.
+
+    // rewrite file:// urls to be locl filesystem urls.
+    return (!!text && !!session) ? text.replace(/(file:\/\/\S*)/g, (s, a) => yellow.dim(session.parseUri(a).fsPath)) : text;
+  }
+  return '';
+}
+
+const stdout = console['log'];
+
+export let log: (message?: any, ...optionalParams: Array<any>) => void = stdout;
+export let error: (message?: any, ...optionalParams: Array<any>) => void = console.error;
+export let warning: (message?: any, ...optionalParams: Array<any>) => void = console.error;
+export let debug: (message?: any, ...optionalParams: Array<any>) => void = (text) => {
+  if (argv.any(arg => arg === '--debug')) {
+    stdout(`${cyan.bold('debug: ')}${text}`);
+  }
+};
+
+export function writeException(e: any) {
+  if (e instanceof Error) {
+    debug(e.message);
+    debug(e.stack);
+    return;
+  }
+  debug(e && e.toString ? e.toString() : e);
+}
+
+export function initStyling(commandline: CommandLine, session: Session) {
+  log = (text) => stdout((md(text, session).trim()));
+  error = (text) => stdout(`${red.bold('ERROR: ')}${md(text, session).trim()}`);
+  warning = (text) => stdout(`${yellow.bold('WARNING: ')}${md(text, session).trim()}`);
+  debug = (text) => { if (commandline.debug) { stdout(`${cyan.bold('DEBUG: ')}${md(text, session).trim()}`); } };
+
+  session.channels.on('message', (text: string, context: any, msec: number) => {
+    log(text);
+  });
+
+  session.channels.on('error', (text: string, context: any, msec: number) => {
+    error(text);
+  });
+
+  session.channels.on('debug', (text: string, context: any, msec: number) => {
+    debug(`${cyan.bold(`[${formatTime(msec)}]`)} ${md(text, session)}`);
+  });
+
+  session.channels.on('warning', (text: string, context: any, msec: number) => {
+    warning(text);
+  });
+}

ce/ce/cli/switch.ts

@@ -0,0 +1,49 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { strict } from 'assert';
+import { i } from '../i18n';
+import { Command } from './command';
+import { Help } from './command-line';
+import { cmdSwitch } from './format';
+
+
+export abstract class Switch implements Help {
+  readonly abstract switch: string;
+  readonly title = '';
+  readonly abstract help: Array<string>;
+  readonly required: boolean;
+  readonly multipleAllowed: boolean;
+
+  constructor(protected command: Command, options?: { multipleAllowed?: boolean, required?: boolean }) {
+    command.switches.push(this);
+    this.multipleAllowed = options?.multipleAllowed || false;
+    this.required = options?.required || false;
+  }
+
+  get valid() {
+    return this.required || this.active;
+  }
+
+  #values?: Array<string>;
+  get values() {
+    return this.#values || (this.#values = this.command.commandLine.claim(this.switch) || []);
+  }
+
+  get value(): any | undefined {
+    const v = this.values;
+    strict.ok(v.length < 2, i`Expected a single value for ${cmdSwitch(this.switch)} - found multiple`);
+    return v[0];
+  }
+
+  get requiredValue(): string {
+    const v = this.values;
+    strict.ok(v.length == 1 && v[0], i`Expected a single value for '--${this.switch}'.`);
+    return v[0];
+  }
+
+  get active(): boolean {
+    const v = this.values;
+    return !!v && v.length > 0 && v[0] !== 'false';
+  }
+}

ce/ce/cli/switches/clear.ts

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { Switch } from '../switch';
+
+export class Clear extends Switch {
+  switch = 'clear';
+  get help() {
+    return [
+      i`removes all files in the local cache`
+    ];
+  }
+}

ce/ce/cli/switches/debug.ts

@@ -0,0 +1,15 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { cli } from '../constants';
+import { Switch } from '../switch';
+
+export class Debug extends Switch {
+  switch = 'debug';
+  get help() {
+    return [
+      i`enables debug mode, displays internal messsages about how ${cli} works`
+    ];
+  }
+}

ce/ce/cli/switches/force.ts

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { Switch } from '../switch';
+
+export class Force extends Switch {
+  switch = 'force';
+  get help() {
+    return [
+      i`proceeds with the (potentially dangerous) action without confirmation`
+    ];
+  }
+}

ce/ce/cli/switches/installed.ts

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { Switch } from '../switch';
+
+export class Installed extends Switch {
+  switch = 'installed';
+  get help() {
+    return [
+      i`shows the _installed_ artifacts`
+    ];
+  }
+}

ce/ce/cli/switches/msbuild-props.ts

@@ -0,0 +1,23 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Uri } from '../../util/uri';
+import { resolvePath } from '../command-line';
+import { Switch } from '../switch';
+
+export class MSBuildProps extends Switch {
+  switch = 'msbuild-props';
+  override multipleAllowed = false;
+  get help() {
+    return [
+      i`Full path to the file in which MSBuild properties will be written.`
+    ];
+  }
+
+  override get value(): Uri | undefined {
+    const v = resolvePath(super.value);
+    return v ? session.fileSystem.file(v) : undefined;
+  }
+}

ce/ce/cli/switches/project.ts

@@ -0,0 +1,58 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { ProjectManifest } from '../../artifacts/artifact';
+import { FileType } from '../../fs/filesystem';
+import { i } from '../../i18n';
+import { session } from '../../main';
+import { Uri } from '../../util/uri';
+import { resolvePath } from '../command-line';
+import { projectFile } from '../format';
+import { debug, error } from '../styling';
+import { Switch } from '../switch';
+
+export class Project extends Switch {
+  switch = 'project';
+  get help() {
+    return [
+      i`override the path to the project folder`
+    ];
+  }
+
+  async getProjectFolder() {
+    const v = resolvePath(super.value);
+    if (v) {
+      const uri = session.fileSystem.file(v);
+      const stat = await uri.stat();
+
+      if (stat.type & FileType.File) {
+        return uri;
+      }
+      if (stat.type & FileType.Directory) {
+        const project = await session.findProjectProfile(uri, false);
+        if (project) {
+          return project;
+        }
+      }
+      error(i`Unable to find project environment ${projectFile(uri)}`);
+      return undefined;
+    }
+    return session.findProjectProfile();
+  }
+
+  override get value(): Promise<Uri | undefined> {
+    return this.getProjectFolder();
+  }
+
+  get manifest(): Promise<ProjectManifest | undefined> {
+    return this.value.then(async (project) => {
+      if (!project) {
+        debug('No project manifest');
+        return undefined;
+      }
+
+      debug(`Loading project manifest ${project} `);
+      return new ProjectManifest(session, await session.openManifest(project));
+    });
+  }
+}

ce/ce/cli/switches/registry.ts

@@ -0,0 +1,48 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { sanitizeUri } from '../../artifacts/artifact';
+import { i } from '../../i18n';
+import { Session } from '../../session';
+import { UpdateCommand } from '../commands/update';
+import { Switch } from '../switch';
+
+export class Registry extends Switch {
+  switch = 'registry';
+  get help() {
+    return [
+      i`override the path to the registry`
+    ];
+  }
+
+  async loadRegistries(session: Session, more: Array<string> = []) {
+    for (const registry of new Set([...this.values, ...more].map(each => sanitizeUri(each)))) {
+      if (registry) {
+        const uri = session.parseUri(registry);
+        if (await session.isLocalRegistry(uri) || await session.isRemoteRegistry(uri)) {
+
+          const r = session.loadRegistry(uri, 'artifact');
+          if (r) {
+            try {
+              await r.load();
+            } catch (e) {
+              // try to update the repo
+              if (!await UpdateCommand.update(r)) {
+                session.channels.error(i`failed to load registry ${uri.toString()}`);
+                continue;
+              }
+            }
+            // registry is loaded
+            // it should be added to the aggregator
+            session.defaultRegistry.add(r, registry);
+          }
+          continue;
+        }
+        session.channels.error(i`Invalid registry ${registry}`);
+      }
+    }
+
+    return session.defaultRegistry;
+  }
+
+}

ce/ce/cli/switches/verbose.ts

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { Switch } from '../switch';
+
+export class Verbose extends Switch {
+  switch = 'verbose';
+  get help() {
+    return [
+      i`enables verbose mode, displays verbose messsages about the process`
+    ];
+  }
+}

ce/ce/cli/switches/version.ts

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { Switch } from '../switch';
+
+export class Version extends Switch {
+  switch = 'version';
+  get help() {
+    return [
+      i`a version or version range to match`
+    ];
+  }
+}

ce/ce/cli/switches/whatIf.ts

@@ -0,0 +1,14 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { i } from '../../i18n';
+import { Switch } from '../switch';
+
+export class WhatIf extends Switch {
+  switch = 'what-if';
+  get help() {
+    return [
+      i`does not actually perform the action, shows only what would be done`
+    ];
+  }
+}

ce/ce/constants.ts

@@ -0,0 +1,26 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+export const project = 'environment.yaml';
+export const undo = 'Z_VCPKG_UNDO';
+export const postscriptVarible = 'Z_VCPKG_POSTSCRIPT';
+export const blank = '\n';
+export const gitUniqueIdPrefix = 'https://aka.ms/vcpkg-ce-default::tools/git::';
+export const gitArtifact = 'microsoft:tools/git';
+export const latestVersion = '*';
+export const vcpkgDownloadFolder = 'VCPKG_DOWNLOADS';
+export const globalConfigurationFile = 'vcpkg-configuration.global.json';
+export const profileNames = ['vcpkg-configuration.json', 'vcpkg-configuration.yaml', 'environment.yaml', 'environment.yml', 'environment.json'];
+export const registryIndexFile = 'index.yaml';
+
+export const defaultConfig =
+  `{
+  "registries": [
+    {
+      "kind": "artifact",
+      "name": "microsoft",
+      "location": "https://aka.ms/vcpkg-ce-default"
+    }
+  ]
+}
+`;

ce/ce/exports.ts

@@ -0,0 +1,167 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { ManyMap } from './util/linq';
+import { Queue } from './util/promise';
+
+/** This adds the expected declarations to the Array type. */
+declare global {
+  interface Array<T> {
+    /**
+    * Returns the elements of an array that meet the condition specified in a callback function.
+    * @param callbackfn A function that accepts up to three arguments. The filter method calls the callbackfn function one time for each element in the array.
+    */
+    where<S extends T>(callbackfn: (value: T, index: number, array: Array<T>) => value is S): Array<S>;
+    /**
+     * Returns the elements of an array that meet the condition specified in a callback function.
+     * @param callbackfn A function that accepts up to three arguments. The filter method calls the callbackfn function one time for each element in the array.
+     */
+    where(callbackfn: (value: T, index: number, array: Array<T>) => unknown): Array<T>;
+
+    /**
+    * Calls a defined callback function on each element of an array, and returns an array that contains the results.
+    */
+    select<U>(callbackfn: (value: T, index: number, array: Array<T>) => U): Array<U>;
+
+    /**
+     * Determines whether the specified callback function returns true for any element of an array.
+     * @param callbackfn A function that accepts up to three arguments. The some method calls
+     * the callbackfn function for each element in the array until the callbackfn returns a value
+     * which is coercible to the Boolean value true, or until the end of the array.
+     * @param thisArg An object to which the this keyword can refer in the callbackfn function.
+     * If thisArg is omitted, undefined is used as the this value.
+     */
+    any(callbackfn: (value: T, index: number, array: Array<T>) => unknown, thisArg?: any): boolean;
+    /**
+     * Determines whether all the members of an array satisfy the specified test.
+     * @param callbackfn A function that accepts up to three arguments. The every method calls
+     * the callbackfn function for each element in the array until the callbackfn returns a value
+     * which is coercible to the Boolean value false, or until the end of the array.
+     * @param thisArg An object to which the this keyword can refer in the callbackfn function.
+     * If thisArg is omitted, undefined is used as the this value.
+     */
+    all(callbackfn: (value: T, index: number, array: Array<T>) => unknown, thisArg?: any): boolean;
+
+    /**
+       * Removes elements from an array and, if necessary, inserts new elements in their place, returning the deleted elements.
+       * @param start The zero-based location in the array from which to start removing elements.
+       * @param deleteCount The number of elements to remove.
+       * @param items Elements to insert into the array in place of the deleted elements.
+       */
+    insert(start: number, ...items: Array<T>): Array<T>;
+
+    /**
+      * Removes elements from an array returning the deleted elements.
+      * @param start The zero-based location in the array from which to start removing elements.
+      * @param deleteCount The number of elements to remove.
+      */
+    remove(start: number, deleteCount?: number): Array<T>;
+
+    /**
+     * Iterates on a collection to create a Queue that will throttle
+     * the async operation 'fn' to a reasonable degree of parallelism.
+     * @param fn the async Fn to call on each
+     */
+    forEachAsync<S>(fn: (v: T) => Promise<S>): Queue;
+
+    selectMany<U>(callbackfn: (value: T, index: number, array: Array<T>) => U): Array<U extends ReadonlyArray<infer InnerArr> ? InnerArr : U>;
+    groupByMap<TKey, TValue>(keySelector: (each: T) => TKey, selector: (each: T) => TValue): Map<TKey, Array<TValue>>;
+    groupBy<TValue>(keySelector: (each: T) => string, selector: (each: T) => TValue): { [s: string]: Array<TValue> };
+    count(predicate: (each: T) => Promise<boolean>): Promise<number>,
+    count(predicate: (each: T) => boolean): number,
+    readonly last: T | undefined;
+    readonly first: T | undefined;
+  }
+}
+
+declare global {
+  interface Map<K, V> {
+    getOrDefault(key: K, defaultValue: V | (() => V)): V;
+  }
+}
+
+if (!Map.prototype.getOrDefault) {
+  Object.defineProperties(Map.prototype, {
+    getOrDefault: {
+      value: function (key: any, defaultValue: any) {
+        let v = this.get(key);
+        if (!v) {
+          this.set(key, v = typeof defaultValue === 'function' ? defaultValue() : defaultValue);
+        }
+        return v;
+      }
+    }
+  });
+}
+
+if (!Array.prototype.insert) {
+  /**
+   * adding some linq-like functionality to the Array type
+   */
+  Object.defineProperties(Array.prototype, {
+    where: { value: Array.prototype.filter },
+    select: { value: Array.prototype.map },
+    any: { value: Array.prototype.some },
+    all: { value: Array.prototype.every },
+    insert: { value: function (position: number, items: Array<any>) { return (<Array<any>>this).splice(position, 0, ...items); } },
+    selectMany: { value: Array.prototype.flatMap },
+    count: {
+      value: function (predicate: (e: any) => boolean | Promise<boolean>) {
+        let v = 0;
+        const all = [];
+        for (const each of this) {
+          const test = <any>predicate(each);
+          if (test.then) {
+            all.push(test.then((antecedent: any) => {
+              if (antecedent) {
+                v++;
+              }
+            }));
+            continue;
+          }
+          if (test) {
+            v++;
+          }
+        }
+        if (all.length) {
+          return Promise.all(all).then(() => v);
+        }
+        return v;
+      }
+    },
+    groupByMap: {
+      value: function (keySelector: (each: any) => any, selector: (each: any) => any) {
+        const result = new ManyMap<any, any>();
+        for (const each of this) {
+          result.push(keySelector(each), selector(each));
+        }
+        return result;
+      }
+    },
+    groupBy: {
+      value: function (keySelector: (each: any) => any, selector: (each: any) => any) {
+        const result = <any>{};
+        for (const each of this) {
+          const key = keySelector(each);
+          (result[key] = result[key] || new Array<any>()).push(selector(each));
+        }
+        return result;
+      }
+    },
+    last: {
+      get() {
+        return this[this.length - 1];
+      }
+    },
+    first: {
+      get() {
+        return this[0];
+      }
+    },
+    forEachAsync: {
+      value: function (fn: (i: any) => Promise<any>) {
+        return new Queue().enqueueMany(this, fn);
+      }
+    }
+  });
+}

ce/ce/fs/acquire.ts

@@ -0,0 +1,249 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { strict } from 'assert';
+import { pipeline as origPipeline } from 'stream';
+import { promisify } from 'util';
+import { i } from '../i18n';
+import { AcquireEvents } from '../interfaces/events';
+import { Session } from '../session';
+import { Credentials } from '../util/credentials';
+import { ExtendedEmitter } from '../util/events';
+import { RemoteFileUnavailable } from '../util/exceptions';
+import { Algorithm, Hash } from '../util/hash';
+import { Uri } from '../util/uri';
+import { get, getStream, RemoteFile, resolveRedirect } from './https';
+import { ProgressTrackingStream } from './streams';
+
+const pipeline = promisify(origPipeline);
+
+const size32K = 1 << 15;
+const size64K = 1 << 16;
+
+export interface AcquireOptions extends Hash {
+  /** force a redownload even if it's in cache */
+  force?: boolean;
+  credentials?: Credentials;
+}
+
+export async function acquireArtifactFile(session: Session, uris: Array<Uri>, outputFilename: string, events: Partial<AcquireEvents>, options?: AcquireOptions) {
+  await session.cache.createDirectory();
+  const outputFile = session.cache.join(outputFilename);
+  session.channels.debug(`Acquire file '${outputFilename}' from [${uris.map(each => each.toString()).join(',')}]`);
+
+  if (options?.algorithm && options?.value) {
+    session.channels.debug(`We have a hash: ${options.algorithm}/${options.value}`);
+
+    // if we have hash data, check to see if the output file is good.
+    if (await outputFile.isFile()) {
+      session.channels.debug(`There is an output file already, verifying: ${outputFile.fsPath}`);
+
+      if (await outputFile.hashValid(events, options)) {
+        session.channels.debug(`Cached file matched hash: ${outputFile.fsPath}`);
+        return outputFile;
+      }
+    }
+  }
+
+  // is the file present on a local filesystem?
+  for (const uri of uris) {
+    if (uri.isLocal) {
+      // we have a local file
+
+      if (options?.algorithm && options?.value) {
+        // we have a hash.
+        // is it valid?
+        if (await uri.hashValid(events, options)) {
+          session.channels.debug(`Local file matched hash: ${uri.fsPath}`);
+          return uri;
+        }
+      } else if (await uri.exists()) {
+        // we don't have a hash, but the file is local, and it exists.
+        // we have to return it
+        session.channels.debug(`Using local file (no hash, unable to verify): ${uri.fsPath}`);
+        return uri;
+      }
+      // do we have a filename
+    }
+  }
+
+  // we don't have a local file
+  // https is all that we know at the moment.
+  const webUris = uris.where(each => each.isHttps);
+  if (webUris.length === 0) {
+    // wait, no web uris?
+    throw new RemoteFileUnavailable(uris);
+  }
+
+  return https(session, webUris, outputFilename, events, options);
+}
+
+/** */
+async function https(session: Session, uris: Array<Uri>, outputFilename: string, events: Partial<AcquireEvents>, options?: AcquireOptions) {
+  const ee = new ExtendedEmitter<AcquireEvents>();
+  ee.subscribe(events);
+  session.channels.debug(`Attempting to download file '${outputFilename}' from [${uris.map(each => each.toString()).join(',')}]`);
+
+  let resumeAtOffset = 0;
+  await session.cache.createDirectory();
+  const outputFile = session.cache.join(outputFilename);
+
+  if (options?.force) {
+    session.channels.debug(`Acquire '${outputFilename}': force specified, forcing download`);
+    // is force specified; delete the current file
+    await outputFile.delete();
+  }
+
+  // start this peeking at the target uris.
+  session.channels.debug(`Acquire '${outputFilename}': checking remote connections`);
+  const locations = new RemoteFile(uris, { credentials: options?.credentials });
+  let url: Uri | undefined;
+
+  // is there a file in the cache
+  if (await outputFile.exists()) {
+    session.channels.debug(`Acquire '${outputFilename}': local file exists`);
+    if (options?.algorithm) {
+      // does it match a hash that we have?
+      if (await outputFile.hashValid(events, options)) {
+        session.channels.debug(`Acquire '${outputFilename}': local file hash matches metdata`);
+        // yes it does. let's just return done.
+        return outputFile;
+      }
+    }
+
+    // it doesn't match a known hash.
+    const contentLength = await locations.contentLength;
+    session.channels.debug(`Acquire '${outputFilename}': remote connection info is back`);
+    const onDiskSize = await outputFile.size();
+    if (!await locations.availableLocation) {
+      if (locations.failures.all(each => each.code === 404)) {
+        let msg = i`Unable to download file`;
+        if (options?.credentials) {
+          msg += (i` - It could be that your authentication credentials are not correct`);
+        }
+
+        session.channels.error(msg);
+        throw new RemoteFileUnavailable(uris);
+      }
+    }
+    // first, make sure that there is a remote that is accessible.
+    strict.ok(!!await locations.availableLocation, `Requested file ${outputFilename} has no accessible locations ${uris.map(each => each.toString()).join(',')}`);
+
+    url = await locations.resumableLocation;
+    // ok, does it support resume?
+    if (url) {
+      // yes, let's check what the size is expected to be.
+
+      if (!options?.algorithm) {
+
+        if (contentLength === onDiskSize) {
+          session.channels.debug(`Acquire '${outputFilename}': on disk file matches length of remote file`);
+          const algorithm = <Algorithm>(await locations.algorithm);
+          const value = await locations.hash;
+          session.channels.debug(`Acquire '${outputFilename}': remote alg/hash: '${algorithm}'/'${value}`);
+          if (algorithm && value && outputFile.hashValid(events, { algorithm, value, ...options })) {
+            session.channels.debug(`Acquire '${outputFilename}': on disk file hash matches the server hash`);
+            // so *we* don't have the hash, but ... if the server has a hash, we could see if what we have is what they have?
+            // it does match what the server has.
+            // I call this an win.
+            return outputFile;
+          }
+
+          // we don't have a hash, or what we have doesn't match.
+          // maybe we will get a match below (or resume)
+        }
+      }
+
+      if (onDiskSize > size64K) {
+        // it's bigger than 64k. Good. otherwise, we're just wasting time.
+
+        // so, how big is the remote
+        if (contentLength >= onDiskSize) {
+          session.channels.debug(`Acquire '${outputFilename}': local file length is less than or equal to remote file length`);
+          // looks like there could be more remotely than we have.
+          // lets compare the first 32k and the last 32k of what we have
+          // against what they have and see if they match.
+          const top = (await get(url, { start: 0, end: size32K - 1, credentials: options?.credentials })).rawBody;
+          const bottom = (await get(url, { start: onDiskSize - size32K, end: onDiskSize - 1, credentials: options?.credentials })).rawBody;
+
+          const onDiskTop = await outputFile.readBlock(0, size32K - 1);
+          const onDiskBottom = await outputFile.readBlock(onDiskSize - size32K, onDiskSize - 1);
+
+          if (top.compare(onDiskTop) === 0 && bottom.compare(onDiskBottom) === 0) {
+            session.channels.debug(`Acquire '${outputFilename}': first/last blocks are equal`);
+            // the start and end of what we have does match what they have.
+            // is this file the same size?
+            if (contentLength === onDiskSize) {
+              // same file size, front and back match, let's accept this. begrudgingly
+              session.channels.debug(`Acquire '${outputFilename}': file size is identical. keeping this one`);
+              return outputFile;
+            }
+            // looks like we can continue from here.
+            session.channels.debug(`Acquire '${outputFilename}': ok to resume`);
+            resumeAtOffset = onDiskSize;
+          }
+        }
+      }
+    }
+  }
+
+  if (resumeAtOffset === 0) {
+    // clearly we mean to not resume. clean any existing file.
+    session.channels.debug(`Acquire '${outputFilename}': not resuming file, full download`);
+    await outputFile.delete();
+  }
+
+  url = url || await locations.availableLocation;
+  strict.ok(!!url, `Requested file ${outputFilename} has no accessible locations ${uris.map(each => each.toString()).join(',')}`);
+  session.channels.debug(`Acquire '${outputFilename}': initiating download`);
+  const length = await locations.contentLength;
+
+  const inputStream = getStream(url, { start: resumeAtOffset, end: length > 0 ? length : undefined, credentials: options?.credentials });
+  let progressStream;
+  if (length > 0) {
+    progressStream = new ProgressTrackingStream(resumeAtOffset, length);
+    progressStream.on('progress', (filePercentage) => ee.emit('download', outputFilename, filePercentage));
+  }
+  const outputStream = await outputFile.writeStream({ append: true });
+  ee.emit('download', outputFilename, 0);
+
+  // whoooosh. write out the file
+  if (progressStream) {
+    await pipeline(inputStream, progressStream, outputStream);
+  } else {
+    await pipeline(inputStream, outputStream);
+  }
+
+  // we've downloaded the file, let's see if it matches the hash we have.
+  if (options?.algorithm) {
+    session.channels.debug(`Acquire '${outputFilename}': checking downloaded file hash`);
+    // does it match the hash that we have?
+    if (!await outputFile.hashValid(events, options)) {
+      await outputFile.delete();
+      throw new Error(i`Downloaded file '${outputFile.fsPath}' did not have the correct hash (${options.algorithm}: ${options.value}) `);
+    }
+    session.channels.debug(`Acquire '${outputFilename}': downloaded file hash matches specified hash`);
+  }
+
+  session.channels.debug(`Acquire '${outputFilename}': downloading file successful`);
+  ee.emit('download', outputFilename, 1000);
+  ee.emit('complete');
+  return outputFile;
+}
+
+export async function resolveNugetUrl(session: Session, pkg: string) {
+  const [, name, version] = pkg.match(/^(.*)\/(.*)$/) ?? [];
+  strict.ok(version, i`package reference '${pkg}' is not a valid nuget package reference ({name}/{version})`);
+
+  // let's resolve the redirect first, since nuget servers don't like us getting HEAD data on the targets via a redirect.
+  // even if this wasn't the case, this is lower cost now rather than later.
+  const url = await resolveRedirect(session.parseUri(`https://www.nuget.org/api/v2/package/${name}/${version}`));
+
+  session.channels.debug(`Resolving nuget package for '${pkg}' to '${url}'`);
+  return url;
+}
+
+export async function acquireNugetFile(session: Session, pkg: string, outputFilename: string, events: Partial<AcquireEvents>, options?: AcquireOptions): Promise<Uri> {
+  return https(session, [await resolveNugetUrl(session, pkg)], outputFilename, events, options);
+}
+

ce/ce/fs/filesystem.ts

@@ -0,0 +1,410 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+/* eslint-disable @typescript-eslint/ban-types */
+
+import { EventEmitter } from 'ee-ts';
+import { Readable, Writable } from 'stream';
+import { Session } from '../session';
+import { Uri } from '../util/uri';
+
+const size64K = 1 << 16;
+const size32K = 1 << 15;
+
+/**
+ * The `FileStat`-type represents metadata about a file
+ */
+export interface FileStat {
+  /**
+   * The type of the file, e.g. is a regular file, a directory, or symbolic link
+   * to a file.
+   *
+   * *Note:* This value might be a bitmask, e.g. `FileType.File | FileType.SymbolicLink`.
+   */
+  type: FileType;
+  /**
+   * The creation timestamp in milliseconds elapsed since January 1, 1970 00:00:00 UTC.
+   */
+  ctime: number;
+  /**
+   * The modification timestamp in milliseconds elapsed since January 1, 1970 00:00:00 UTC.
+   *
+   * *Note:* If the file changed, it is important to provide an updated `mtime` that advanced
+   * from the previous value. Otherwise there may be optimizations in place that will not show
+   * the updated file contents in an editor for example.
+   */
+  mtime: number;
+  /**
+   * The size in bytes.
+   *
+   * *Note:* If the file changed, it is important to provide an updated `size`. Otherwise there
+   * may be optimizations in place that will not show the updated file contents in an editor for
+   * example.
+   */
+  size: number;
+  /**
+   * The file mode (unix permissions).
+   */
+  mode: number;
+}
+
+/**
+* Enumeration of file types. The types `File` and `Directory` can also be
+* a symbolic links, in that case use `FileType.File | FileType.SymbolicLink` and
+* `FileType.Directory | FileType.SymbolicLink`.
+*/
+export enum FileType {
+  /**
+   * The file type is unknown.
+   */
+  Unknown = 0,
+  /**
+   * A regular file.
+   */
+  File = 1,
+  /**
+   * A directory.
+   */
+  Directory = 2,
+  /**
+   * A symbolic link to a file.
+   */
+  SymbolicLink = 64
+}
+
+export interface WriteStreamOptions {
+  append?: boolean;
+  mode?: number;
+  mtime?: Date;
+}
+
+/**
+ * A random-access reading interface to access a file in a FileSystem.
+ *
+ * Ideally, we keep reads in a file to a forward order, so that this can be implemented on filesystems
+ * that do not support random access (ie, please do your best to order reads so that they go forward only as much as possible)
+ *
+ * Underneath on FSes that do not support random access, this would likely require multiple 'open' operation for the same
+ * target file.
+ */
+export abstract class ReadHandle {
+  /**
+   * Reads a block from a file
+   *
+   * @param buffer The buffer that the data will be written to.
+   * @param offset The offset in the buffer at which to start writing.
+   * @param length The number of bytes to read.
+   * @param position The offset from the beginning of the file from which data should be read. If `null`, data will be read from the current position.
+   */
+  abstract read<TBuffer extends Uint8Array>(buffer: TBuffer, offset?: number | null, length?: number | null, position?: number | null): Promise<{ bytesRead: number, buffer: TBuffer }>;
+
+  async readComplete<TBuffer extends Uint8Array>(buffr: TBuffer, offset = 0, length = buffr.byteLength, position: number | null = null, totalRead = 0): Promise<{ bytesRead: number, buffer: TBuffer }> {
+    const { bytesRead, buffer } = await this.read(buffr, offset, length, position);
+    if (length) {
+      if (bytesRead && bytesRead < length) {
+        return await this.readComplete(buffr, offset + bytesRead, length - bytesRead, position ? position + bytesRead : null, bytesRead + totalRead);
+      }
+    }
+    return { bytesRead: bytesRead + totalRead, buffer };
+  }
+  /**
+   * Returns a Readable for consuming an opened ReadHandle
+   * @param start the first byte to read of the target
+   * @param end the last byte to read of the target (inclusive!)
+   */
+  readStream(start = 0, end = Infinity): Readable {
+    return Readable.from(asyncIterableOverHandle(start, end, this), {});
+  }
+
+  abstract size(): Promise<number>;
+
+  abstract close(): Promise<void>;
+
+  range(start: number, length: number) {
+    return new RangeReadHandle(this, start, length);
+  }
+}
+
+class RangeReadHandle extends ReadHandle {
+
+  pos = 0;
+  readHandle?: ReadHandle;
+
+  constructor(readHandle: ReadHandle, private start: number, private length: number) {
+    super();
+    this.readHandle = readHandle;
+  }
+
+  async read<TBuffer extends Uint8Array>(buffer: TBuffer, offset?: number | null, length?: number | null, position?: number | null): Promise<{ bytesRead: number; buffer: TBuffer; }> {
+    if (this.readHandle) {
+      position = position !== undefined && position !== null ? (position + this.start) : (this.pos + this.start);
+      length = length === null ? this.length : length;
+
+      const result = await this.readHandle.read(buffer, offset, length, position);
+      this.pos += result.bytesRead;
+      return result;
+    }
+
+    return {
+      bytesRead: 0, buffer
+    };
+
+  }
+
+  async size(): Promise<number> {
+    return this.length;
+  }
+
+  async close(): Promise<void> {
+    this.readHandle = undefined;
+  }
+
+}
+
+/**
+ * Picks a reasonable buffer size. Not more than 64k
+ *
+ * @param length
+ */
+function reasonableBuffer(length: number) {
+  return Buffer.alloc(length > size64K ? size32K : length);
+}
+
+/**
+ * Creates an AsyncIterable<Buffer> over a ReadHandle
+ * @param start the first byte in the target read from
+ * @param end the last byte in the target to read from
+ * @param handle the ReadHandle
+ */
+async function* asyncIterableOverHandle(start: number, end: number, handle: ReadHandle): AsyncIterable<Buffer> {
+  while (start < end) {
+    // buffer alloc must be inside the loop; zlib will hold the buffers until it can deal with a whole stream.
+    const buffer = reasonableBuffer(1 + end - start);
+    const count = Math.min(1 + end - start, buffer.byteLength);
+    const b = await handle.read(buffer, 0, count, start);
+    if (b.bytesRead === 0) {
+      return;
+    }
+    start += b.bytesRead;
+    // return only what was actually read. (just a view)
+    if (b.bytesRead === buffer.byteLength) {
+      yield buffer;
+    }
+    else {
+      yield buffer.slice(0, b.bytesRead);
+    }
+  }
+}
+
+export abstract class FileSystem extends EventEmitter<FileSystemEvents> {
+
+  protected baseUri?: Uri;
+
+  /**
+ * Creates a new URI from a file system path, e.g. `c:\my\files`,
+ * `/usr/home`, or `\\server\share\some\path`.
+ *
+ * associates this FileSystem with the Uri
+ *
+ * @param path A file system path (see `URI#fsPath`)
+ */
+  file(path: string): Uri {
+    return Uri.file(this, path);
+  }
+
+  /** construct an Uri from the various parts */
+  from(components: {
+    scheme: string;
+    authority?: string;
+    path?: string;
+    query?: string;
+    fragment?: string;
+  }): Uri {
+    return Uri.from(this, components);
+  }
+
+  /**
+ * Creates a new URI from a string, e.g. `https://www.msft.com/some/path`,
+ * `file:///usr/home`, or `scheme:with/path`.
+ *
+ * @param value A string which represents an URI (see `URI#toString`).
+ */
+  parse(value: string, _strict?: boolean): Uri {
+    return Uri.parse(this, value, _strict);
+  }
+
+  /**
+   * Retrieve metadata about a file.
+   *
+   * @param uri The uri of the file to retrieve metadata about.
+   * @return The file metadata about the file.
+   */
+  abstract stat(uri: Uri, options?: {}): Promise<FileStat>;
+
+  /**
+   * Retrieve all entries of a [directory](#FileType.Directory).
+   *
+   * @param uri The uri of the folder.
+   * @return An array of name/type-tuples or a Promise that resolves to such.
+   */
+  abstract readDirectory(uri: Uri, options?: { recursive?: boolean }): Promise<Array<[Uri, FileType]>>;
+
+  /**
+   * Create a new directory (Note, that new files are created via `write`-calls).
+   *
+   * *Note* that missing directories are created automatically, e.g this call has
+   * `mkdirp` semantics.
+   *
+   * @param uri The uri of the new folder.
+   */
+  abstract createDirectory(uri: Uri, options?: {}): Promise<void>;
+
+  /**
+   * Read the entire contents of a file.
+   *
+   * @param uri The uri of the file.
+   * @return An array of bytes or a Promise that resolves to such.
+   */
+  abstract readFile(uri: Uri, options?: {}): Promise<Uint8Array>;
+
+  /**
+   * Creates a stream to read a file from the filesystem
+   *
+   * @param uri The uri of the file.
+   * @return a Readable stream
+   */
+  abstract readStream(uri: Uri, options?: { start?: number, end?: number }): Promise<Readable>;
+
+  /**
+   * Write data to a file, replacing its entire contents.
+   *
+   * @param uri The uri of the file.
+   * @param content The new content of the file.
+   */
+  abstract writeFile(uri: Uri, content: Uint8Array): Promise<void>;
+
+  /**
+   * Creates a stream to write a file to the filesystem
+   *
+   * @param uri The uri of the file.
+   * @return a Writeable stream
+   */
+  abstract writeStream(uri: Uri, options?: WriteStreamOptions): Promise<Writable>;
+
+  /**
+   * Delete a file.
+   *
+   * @param uri The resource that is to be deleted.
+   * @param options Defines if trash can should be used and if deletion of folders is recursive
+   */
+  abstract delete(uri: Uri, options?: { recursive?: boolean, useTrash?: boolean }): Promise<void>;
+
+  /**
+   * Rename a file or folder.
+   *
+   * @param oldUri The existing file.
+   * @param newUri The new location.
+   * @param options Defines if existing files should be overwritten.
+   */
+  abstract rename(source: Uri, target: Uri, options?: { overwrite?: boolean }): Promise<void>;
+
+  abstract openFile(uri: Uri): Promise<ReadHandle>;
+
+  /**
+   * Copy files or folders.
+   *
+   * @param source The existing file.
+   * @param destination The destination location.
+   * @param options Defines if existing files should be overwritten.
+   */
+  abstract copy(source: Uri, target: Uri, options?: { overwrite?: boolean }): Promise<number>;
+
+  abstract createSymlink(symlink: Uri, target: Uri): Promise<void>;
+
+  /** checks to see if the target exists */
+  async exists(uri: Uri) {
+    try {
+      return !!(await this.stat(uri));
+    } catch (e) {
+      // if this fails, we're assuming false
+    }
+    return false;
+  }
+
+  /** checks to see if the target is a directory/folder */
+  async isDirectory(uri: Uri) {
+    try {
+      return !!((await this.stat(uri)).type & FileType.Directory);
+    } catch {
+      // if this fails, we're assuming false
+    }
+    return false;
+  }
+
+  /** checks to see if the target is a file */
+  async isFile(uri: Uri) {
+    try {
+      const s = await this.stat(uri);
+
+      return !!(s.type & FileType.File);
+    } catch {
+      // if this fails, we're assuming false
+    }
+    return false;
+  }
+
+  /** checks to see if the target is a symbolic link */
+  async isSymlink(uri: Uri) {
+    try {
+      return !!((await this.stat(uri)) && FileType.SymbolicLink);
+    } catch {
+      // if this fails, we're assuming false
+    }
+    return false;
+  }
+
+  constructor(protected session: Session) {
+    super();
+  }
+
+  /** EventEmitter for when files are read */
+  protected read(path: Uri, context?: any) {
+    this.emit('read', path, context, this.session.stopwatch.total);
+  }
+
+  /** EventEmitter for when files are written */
+  protected write(path: Uri, context?: any) {
+    this.emit('write', path, context, this.session.stopwatch.total);
+  }
+
+  /** EventEmitter for when files are deleted */
+  protected deleted(path: Uri, context?: any) {
+    this.emit('deleted', path, context, this.session.stopwatch.total);
+  }
+
+  /** EventEmitter for when files are renamed */
+  protected renamed(path: Uri, context?: any) {
+    this.emit('renamed', path, context, this.session.stopwatch.total);
+  }
+
+  /** EventEmitter for when directories are read */
+  protected directoryRead(path: Uri, contents?: Promise<Array<[Uri, FileType]>>) {
+    this.emit('directoryRead', path, contents, this.session.stopwatch.total);
+  }
+
+  /** EventEmitter for when direcotries are created */
+  protected directoryCreated(path: Uri, context?: any) {
+    this.emit('directoryCreated', path, context, this.session.stopwatch.total);
+  }
+}
+
+
+/** Event definitions for FileSystem events */
+interface FileSystemEvents {
+  read(path: Uri, context: any, msec: number): void;
+  write(path: Uri, context: any, msec: number): void;
+  deleted(path: Uri, context: any, msec: number): void;
+  renamed(path: Uri, context: any, msec: number): void;
+  directoryRead(path: Uri, contents: Promise<Array<[Uri, FileType]>> | undefined, msec: number): void;
+  directoryCreated(path: Uri, context: any, msec: number): void;
+}

ce/ce/fs/http-filesystem.ts

@@ -0,0 +1,93 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { Readable, Writable } from 'stream';
+import { Uri } from '../util/uri';
+import { FileStat, FileSystem, FileType, ReadHandle } from './filesystem';
+import { get, getStream, head } from './https';
+
+/**
+ * HTTPS Filesystem
+ *
+ */
+export class HttpsFileSystem extends FileSystem {
+
+  async stat(uri: Uri): Promise<FileStat> {
+    const result = await head(uri);
+
+    return {
+      type: FileType.File,
+      mtime: Date.parse(result.headers.date || ''),
+      ctime: Date.parse(result.headers.date || ''),
+      size: Number.parseInt(result.headers['content-length'] || '0'),
+      mode: 0o555 // https is read only but always 'executable'
+    };
+  }
+  readDirectory(uri: Uri): Promise<Array<[Uri, FileType]>> {
+    throw new Error('Method not implemented');
+  }
+  createDirectory(uri: Uri): Promise<void> {
+    throw new Error('Method not implemented');
+  }
+  async readFile(uri: Uri): Promise<Uint8Array> {
+    return (await get(uri)).rawBody;
+  }
+  writeFile(uri: Uri, content: Uint8Array): Promise<void> {
+    throw new Error('Method not implemented');
+  }
+  delete(uri: Uri, options?: { recursive?: boolean | undefined; useTrash?: boolean | undefined; }): Promise<void> {
+    throw new Error('Method not implemented');
+  }
+  rename(source: Uri, target: Uri, options?: { overwrite?: boolean | undefined; }): Promise<void> {
+    throw new Error('Method not implemented');
+  }
+  copy(source: Uri, target: Uri, options?: { overwrite?: boolean | undefined; }): Promise<number> {
+    throw new Error('Method not implemented');
+  }
+  async createSymlink(original: Uri, symlink: Uri): Promise<void> {
+    throw new Error('Method not implemented');
+  }
+  async readStream(uri: Uri, options?: { start?: number, end?: number }): Promise<Readable> {
+    return getStream(uri, options);
+  }
+  writeStream(uri: Uri): Promise<Writable> {
+    throw new Error('Method not implemented');
+  }
+
+  async openFile(uri: Uri): Promise<ReadHandle> {
+    return new HttpsReadHandle(uri);
+  }
+}
+
+
+class HttpsReadHandle extends ReadHandle {
+  position = 0;
+  constructor(private target: Uri) {
+    super();
+  }
+
+  async read<TBuffer extends Uint8Array>(buffer: TBuffer, offset = 0, length = buffer.byteLength, position: number | null = null): Promise<{ bytesRead: number; buffer: TBuffer; }> {
+    if (position !== null) {
+      this.position = position;
+    }
+
+    const r = getStream(this.target, { start: this.position, end: this.position + length });
+    let bytesRead = 0;
+
+    for await (const chunk of r) {
+      const c = <Buffer>chunk;
+      c.copy(buffer, offset);
+      bytesRead += c.length;
+      offset += c.length;
+    }
+    return { bytesRead, buffer };
+  }
+
+  async size(): Promise<number> {
+    return this.target.size();
+  }
+
+  async close() {
+    //return this.handle.close();
+  }
+}

ce/ce/fs/https.ts

@@ -0,0 +1,225 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { default as got, Headers, HTTPError, Response } from 'got';
+import { Credentials } from '../util/credentials';
+import { anyWhere } from '../util/promise';
+import { Uri } from '../util/uri';
+
+/**
+ * Resolves an HTTPS GET redirect by doing the GET, grabbing the redirects and then cancelling the rest of the request
+ * @param location the URL to get the final location of
+ */
+export async function resolveRedirect(location: Uri) {
+  let finalUrl = location;
+
+  const stream = got.get(location.toUrl(), { timeout: 15000, isStream: true });
+
+  // when the response comes thru, we can grab the headers & stuff from it
+  stream.on('response', (response: Response) => {
+    finalUrl = location.fileSystem.parse(response.redirectUrls.last || finalUrl.toString());
+  });
+
+  // we have to get at least some data for the response event to trigger.
+  for await (const chunk of stream) {
+    // but we don't need any of it :D
+    break;
+  }
+  stream.destroy();
+  return finalUrl;
+}
+
+/**
+ * Does an HTTPS HEAD request, and on a 404, tries to do an HTTPS GET and see if we get a redirect, and harvest the headers from that.
+ * @param location the target URL
+ * @param headers any headers to put in the request.
+ */
+export async function head(location: Uri, headers: Headers = {}, credentials?: Credentials): Promise<Response<string>> {
+  try {
+    setCredentials(headers, location, credentials);
+    // on a successful HEAD request, do nothing different
+    return await got.head(location.toUrl(), { timeout: 15000, headers });
+  } catch (E) {
+    // O_o
+    //
+    // So, it turns out that nuget servers (maybe others too?) don't do redirects on HEAD requests,
+    // and instead issue a 404.
+    // let's retry the request as a GET, and dump it after the first chunk.
+    // typically, a HEAD request should see a 300-400msec response time
+    // and yes, this does stretch that out to 500-700msec, but whatcha gonna do?
+    if (E instanceof HTTPError && E.response.statusCode === 404) {
+      try {
+        const syntheticResponse = <Response<string>>{};
+        const stream = got.get(location.toUrl(), { timeout: 15000, headers, isStream: true });
+
+        // when the response comes thru, we can grab the headers & stuff from it
+        stream.on('response', (response: Response) => {
+          syntheticResponse.headers = response.headers;
+          syntheticResponse.statusCode = response.statusCode;
+          syntheticResponse.redirectUrls = response.redirectUrls;
+        });
+
+        // we have to get at least some data for the response event to trigger.
+        for await (const chunk of stream) {
+          // but we don't need any of it :D
+          break;
+        }
+        stream.destroy();
+        return syntheticResponse;
+      }
+      catch {
+        // whatever, it didn't work. let the rethrow happen.
+      }
+    }
+    throw E;
+  }
+}
+
+/** HTTPS Get request, returns a buffer  */
+export function get(location: Uri, options?: { start?: number, end?: number, headers?: Headers, credentials?: Credentials }) {
+  let headers: Headers | undefined = undefined;
+  headers = setRange(headers, options?.start, options?.end);
+  headers = setCredentials(headers, location, options?.credentials);
+
+  return got.get(location.toUrl(), { headers });
+}
+
+function setRange(headers: Headers | undefined, start?: number, end?: number) {
+  if (start !== undefined || end !== undefined) {
+    headers = headers || {};
+    headers['range'] = `bytes=${start !== undefined ? start : ''}-${end !== undefined ? end : ''}`;
+  }
+  return headers;
+}
+
+
+function setCredentials(headers: Headers | undefined, target: Uri, credentials?: Credentials) {
+  if (credentials) {
+    // todo: if we have to add some credential headers, we'd do it here.
+    // we've removed github auth support until we actually need such a thing
+  }
+  return headers;
+}
+
+/** HTTPS Get request, returns a stream
+ * @internal
+*/
+export function getStream(location: Uri, options?: { start?: number, end?: number, headers?: Headers, credentials?: Credentials }) {
+  let headers: Headers | undefined = options?.headers;
+  headers = setRange(headers, options?.start, undefined);
+  headers = setCredentials(headers, location, options?.credentials);
+
+  return got.get(location.toUrl(), { isStream: true, retry: 3, headers });
+}
+
+export interface Info {
+  failed?: boolean;
+  location: Uri;
+  resumeable: boolean;
+  contentLength: number;
+  hash?: string;
+  algorithm?: string;
+}
+
+function digest(headers: Headers) {
+  let hash = hashAlgorithm(headers['digest'], 'sha-256');
+
+  // any of the sha* hashes..
+  if (hash) {
+    return { hash, algorithm: 'sha256' };
+  }
+  hash = hashAlgorithm(headers['digest'], 'sha-384');
+  if (hash) {
+    return { hash, algorithm: 'sha384' };
+  }
+  hash = hashAlgorithm(headers['digest'], 'sha-512');
+  if (hash) {
+    return { hash, algorithm: 'sha512' };
+  }
+
+  // nothing we know about.
+  return { hash: undefined, algorithm: undefined };
+}
+
+/**
+ * RemoteFile is a class that represents a single remote file, but mapped to multiple mirrored URLs
+ * on creation, it kicks off HEAD requests to each URL so that we can get hash/digest, length, resumability etc
+ *
+ * the properties are Promises<> to the results, where it grabs data from the first returning valid query without
+ * blocking elsewhere.
+ *
+*/
+export class RemoteFile {
+  info: Array<Promise<Info>>;
+  constructor(protected locations: Array<Uri>, options?: { credentials?: Credentials }) {
+    this.info = locations.map(location => {
+      return head(location, setCredentials({
+        'want-digest': 'sha-256;q=1, sha-512;q=0.9',
+        'accept-encoding': 'identity;q=0', // we need to know the content length without gzip encoding,
+      }, location, options?.credentials)).then(data => {
+        if (data.statusCode === 200) {
+          const { hash, algorithm } = digest(data.headers);
+          return {
+            location,
+            resumeable: data.headers['accept-ranges'] === 'bytes',
+            contentLength: Number.parseInt(data.headers['content-length']!) || -1, // -1 means we were not told.
+            hash,
+            algorithm,
+          };
+        }
+        this.failures.push({
+          code: data.statusCode,
+          reason: `A non-ok status code was returned: ${data.statusMessage}`
+        });
+        throw new Error(`A non-ok status code was returned: ${data.statusCode}`);
+      }, err => {
+        this.failures.push({
+          code: err?.response?.statusCode,
+          reason: `A non-ok status code was returned: ${err?.response?.statusMessage}`
+        });
+        throw err;
+      });
+    });
+
+
+    // lazy properties (which do not throw on errors.)
+    this.availableLocation = Promise.any(this.info).then(success => success.location, fail => undefined);
+    this.resumable = anyWhere(this.info, each => each.resumeable).then(success => true, fail => false);
+    this.resumableLocation = anyWhere(this.info, each => each.resumeable).then(success => success.location, fail => undefined);
+    this.contentLength = anyWhere(this.info, each => !!each.contentLength).then(success => success.contentLength, fail => -2);
+    this.hash = anyWhere(this.info, each => !!each.hash).then(success => success.hash, fail => undefined);
+    this.algorithm = anyWhere(this.info, each => !!each.algorithm).then(success => success.algorithm, fail => undefined);
+  }
+
+  resumable: Promise<boolean>;
+  contentLength: Promise<number>;
+  hash: Promise<string | undefined>;
+  algorithm: Promise<string | undefined>;
+  availableLocation: Promise<Uri | undefined>;
+  resumableLocation: Promise<Uri | undefined>;
+  failures = new Array<{ code: number, reason: string }>();
+}
+
+/**
+ * Digest/hash in headers are base64 encoded strings.
+ * @param data the base64 encoded string
+ */
+function decode(data?: string): string | undefined {
+  return data ? Buffer.from(data, 'base64').toString('hex').toLowerCase() : undefined;
+}
+
+/**
+ * Get the hash alg/hash from the digest.
+ * @param digest the digest header
+ * @param algorithm the algorithm we're trying to match
+ */
+function hashAlgorithm(digest: string | Array<string> | undefined, algorithm: 'sha-256' | 'sha-384' | 'sha-512'): string | undefined {
+  for (const each of (digest ? Array.isArray(digest) ? digest : [digest] : [])) {
+    if (each.startsWith(algorithm)) {
+      return decode(each.substr(8));
+    }
+  }
+
+  // nothing.
+  return undefined;
+}

ce/ce/fs/local-filesystem.ts

@@ -0,0 +1,232 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { strict } from 'assert';
+import { COPYFILE_EXCL } from 'constants';
+import { close, createReadStream, createWriteStream, futimes, NoParamCallback, open as openFd, Stats, write as writeFd, writev as writevFd } from 'fs';
+import { copyFile, FileHandle, mkdir, open, readdir, readFile, rename, rm, stat, symlink, writeFile } from 'fs/promises';
+import { basename, join } from 'path';
+import { Readable, Writable } from 'stream';
+import { i } from '../i18n';
+import { delay } from '../util/events';
+import { TargetFileCollision } from '../util/exceptions';
+import { Queue } from '../util/promise';
+import { Uri } from '../util/uri';
+import { FileStat, FileSystem, FileType, ReadHandle, WriteStreamOptions } from './filesystem';
+
+function getFileType(stats: Stats) {
+  return FileType.Unknown |
+    (stats.isDirectory() ? FileType.Directory : 0) |
+    (stats.isFile() ? FileType.File : 0) |
+    (stats.isSymbolicLink() ? FileType.SymbolicLink : 0);
+}
+
+class LocalFileStats implements FileStat {
+  constructor(private stats: Stats) {
+    strict.ok(stats, i`stats may not be undefined`);
+  }
+  get type() {
+    return getFileType(this.stats);
+  }
+  get ctime() {
+    return this.stats.ctimeMs;
+  }
+  get mtime() {
+    return this.stats.mtimeMs;
+  }
+  get size() {
+    return this.stats.size;
+  }
+  get mode() {
+    return this.stats.mode;
+  }
+}
+
+
+/**
+ * Implementation of the Local File System
+ *
+ * This is used to handle the access to the local disks.
+ */
+export class LocalFileSystem extends FileSystem {
+  async stat(uri: Uri): Promise<FileStat> {
+    const path = uri.fsPath;
+    const s = await stat(path);
+    return new LocalFileStats(s);
+  }
+
+  async readDirectory(uri: Uri, options?: { recursive?: boolean }): Promise<Array<[Uri, FileType]>> {
+    let retval!: Promise<Array<[Uri, FileType]>>;
+    try {
+      const folder = uri.fsPath;
+      const retval = new Array<[Uri, FileType]>();
+
+      // use forEachAsync instead so we can throttle this appropriately.
+      await (await readdir(folder)).forEachAsync(async each => {
+        const path = uri.fileSystem.file(join(folder, each));
+        const type = getFileType(await stat(uri.join(each).fsPath));
+        retval.push(<[Uri, FileType]>[path, type]);
+        if (options?.recursive && type === FileType.Directory) {
+          retval.push(... await this.readDirectory(path, options));
+        }
+      }).done;
+
+      return retval;
+    } finally {
+      // log that.
+      this.directoryRead(uri, retval);
+    }
+  }
+
+  async createDirectory(uri: Uri): Promise<void> {
+    await mkdir(uri.fsPath, { recursive: true });
+    this.directoryCreated(uri);
+  }
+
+  createSymlink(original: Uri, slink: Uri): Promise<void> {
+    return symlink(original.fsPath, slink.fsPath, 'file');
+  }
+
+  async readFile(uri: Uri): Promise<Uint8Array> {
+    let contents!: Promise<Uint8Array>;
+    try {
+      contents = readFile(uri.fsPath);
+      return await contents;
+    } finally {
+      this.read(uri, contents);
+    }
+  }
+
+  async writeFile(uri: Uri, content: Uint8Array): Promise<void> {
+    try {
+      await uri.parent.createDirectory();
+      return writeFile(uri.fsPath, content);
+    } finally {
+      this.write(uri, content);
+    }
+  }
+
+  async delete(uri: Uri, options?: { recursive?: boolean | undefined; useTrash?: boolean | undefined; }): Promise<void> {
+    try {
+      options = options || { recursive: false };
+      await rm(uri.fsPath, { recursive: options.recursive, force: true, maxRetries: 3, retryDelay: 20 });
+      // todo: Hack -- on windows, when something is used and then deleted, the delete might not actually finish
+      // before the Promise is resolved. Adding a delay fixes this (but probably is an underlying node bug)
+      await delay(50);
+      return;
+    } finally {
+      this.deleted(uri);
+    }
+  }
+
+  rename(source: Uri, target: Uri, options?: { overwrite?: boolean | undefined; }): Promise<void> {
+    try {
+      strict.equal(source.fileSystem, target.fileSystem, i`Cannot rename files across filesystems`);
+      return rename(source.fsPath, target.fsPath);
+    } finally {
+      this.renamed(source, { target, options });
+    }
+  }
+
+  async copy(source: Uri, target: Uri, options?: { overwrite?: boolean | undefined; }): Promise<number> {
+    const { type } = await source.stat();
+    const opts = <any>(options || {});
+    const overwrite = opts.overwrite ? 0 : COPYFILE_EXCL;
+
+    if (type & FileType.File) {
+      // make sure the target folder is there
+      await target.parent.createDirectory();
+      await copyFile(source.fsPath, target.fsPath, overwrite);
+      return 1;
+    }
+
+    strict.ok(type & FileType.Directory, 'Unknown file type should never happen during copy');
+
+    let targetIsFile = false;
+    try {
+      targetIsFile = !!((await target.stat()).type & FileType.File);
+    } catch {
+      // not a file
+    }
+
+    // if it's a folder, then the target has to be a folder, or not exist
+    if (targetIsFile) {
+      throw new TargetFileCollision(target, i`Copy failed: source (${source.fsPath}) is a folder, target (${target.fsPath}) is a file`);
+    }
+
+    // make sure the target folder exists
+    await target.createDirectory();
+
+    // only the initial call gets to wait for everybody to finish.
+    let queue: Queue | undefined;
+
+    // track the count, starting at the base folder.
+    if (opts.queue === undefined) {
+      queue = opts.queue = new Queue();
+    }
+
+    // loop thru the contents of this folder
+    for (const [sourceUri, fileType] of await source.readDirectory()) {
+      const targetUri = target.join(basename(sourceUri.path));
+      if (fileType & FileType.Directory) {
+        await this.copy(sourceUri, targetUri, opts);
+        continue;
+      }
+      // queue up the copy file
+      void opts.queue.enqueue(() => copyFile(sourceUri.fsPath, targetUri.fsPath, overwrite));
+    }
+    return queue ? queue.done : -1 /* innerloop */;
+  }
+
+  async readStream(uri: Uri, options?: { start?: number, end?: number }): Promise<Readable> {
+    this.read(uri);
+    return createReadStream(uri.fsPath, options);
+  }
+
+  async writeStream(uri: Uri, options?: WriteStreamOptions): Promise<Writable> {
+    this.write(uri);
+    const flags = options?.append ? 'a' : 'w';
+    const createWriteOptions: any = { flags, mode: options?.mode, autoClose: true, emitClose: true };
+    if (options?.mtime) {
+      const mtime = options.mtime;
+      // inject futimes call as part of close
+      createWriteOptions.fs = {
+        open: openFd,
+        write: writeFd,
+        writev: writevFd,
+        close: (fd: number, callback: NoParamCallback) => {
+          futimes(fd, new Date(), mtime, (futimesErr) => {
+            close(fd, (closeErr) => {
+              callback(futimesErr || closeErr);
+            });
+          });
+        }
+      };
+    }
+
+    return createWriteStream(uri.fsPath, createWriteOptions);
+  }
+
+  async openFile(uri: Uri): Promise<ReadHandle> {
+    return new LocalReadHandle(await open(uri.fsPath, 'r'));
+  }
+}
+
+class LocalReadHandle extends ReadHandle {
+  constructor(private handle: FileHandle) {
+    super();
+  }
+
+  read<TBuffer extends Uint8Array>(buffer: TBuffer, offset = 0, length = buffer.byteLength, position: number | null = null): Promise<{ bytesRead: number; buffer: TBuffer; }> {
+    return this.handle.read(buffer, offset, length, position);
+  }
+
+  async size(): Promise<number> {
+    const stat = await this.handle.stat();
+    return stat.size;
+  }
+
+  async close() {
+    return this.handle.close();
+  }
+}

ce/ce/fs/streams.ts

@@ -0,0 +1,42 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import EventEmitter = require('events');
+import { Transform, TransformCallback } from 'stream';
+import { Stopwatch } from '../util/channels';
+import { PercentageScaler } from '../util/percentage-scaler';
+
+export interface Progress {
+  progress(percent: number, bytes: number, msec: number): void;
+}
+
+export interface ProgressTrackingEvents extends EventEmitter {
+  on(event: 'progress', callback: (progress: number, currentPosition: number, msec: number) => void): this;
+}
+
+export class ProgressTrackingStream extends Transform implements ProgressTrackingEvents {
+  private readonly stopwatch = new Stopwatch;
+  private readonly scaler: PercentageScaler;
+  private currentPosition: number;
+
+  constructor(start: number, end: number) {
+    super();
+    this.scaler = new PercentageScaler(start, end);
+    this.currentPosition = start;
+  }
+
+  override _transform(chunk: any, encoding: BufferEncoding, callback: TransformCallback): void {
+    if (<string>encoding !== 'buffer') {
+      return callback(new Error('unexpected chunk type'));
+    }
+
+    const chunkBuffer = <Buffer>chunk;
+    this.currentPosition += chunkBuffer.byteLength;
+    this.emit('progress', this.scaler.scalePosition(this.currentPosition), this.currentPosition, this.stopwatch.total);
+    return callback(null, chunk);
+  }
+
+  get currentPercentage() {
+    return this.scaler.scalePosition(this.currentPosition);
+  }
+}

ce/ce/fs/unified-filesystem.ts

@@ -0,0 +1,119 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { strict } from 'assert';
+import { Readable, Writable } from 'stream';
+import { i } from '../i18n';
+import { Dictionary } from '../util/linq';
+import { Uri } from '../util/uri';
+import { FileStat, FileSystem, FileType, ReadHandle, WriteStreamOptions } from './filesystem';
+
+/**
+ * gets the scheme off the front of an uri.
+ * @param uri the uri to get the scheme for.
+ * @returns the scheme, undefined if the uri has no scheme (colon)
+ */
+export function schemeOf(uri: string) {
+  strict.ok(uri, i`Uri may not be empty`);
+  return /^(\w*):/.exec(uri)?.[1];
+}
+
+export class UnifiedFileSystem extends FileSystem {
+
+  private filesystems: Dictionary<FileSystem> = {};
+
+  /** registers a scheme to a given filesystem
+   *
+   * @param scheme the Uri scheme to reserve
+   * @param fileSystem the filesystem to associate with the scheme
+   */
+  register(scheme: string | Array<string>, fileSystem: FileSystem) {
+    if (Array.isArray(scheme)) {
+      for (const each of scheme) {
+        this.register(each, fileSystem);
+      }
+      return this;
+    }
+    strict.ok(!this.filesystems[scheme], i`scheme '${scheme}' already registered`);
+    this.filesystems[scheme] = fileSystem;
+    return this;
+  }
+
+  /**
+   * gets the filesystem for the given uri.
+   *
+   * @param uri the uri to check the filesystem for
+   *
+   * @returns the filesystem. Will throw if no filesystem is valid.
+   */
+  public filesystem(uri: string | Uri) {
+    const scheme = schemeOf(uri.toString());
+
+    strict.ok(scheme, i`uri ${uri.toString()} has no scheme`);
+
+    const filesystem = this.filesystems[scheme];
+    strict.ok(filesystem, i`scheme ${scheme} has no filesystem associated with it`);
+
+    return filesystem;
+  }
+
+  /**
+  * Creates a new URI from a string, e.g. `https://www.msft.com/some/path`,
+  * `file:///usr/home`, or `scheme:with/path`.
+  *
+  * @param uri A string which represents an URI (see `URI#toString`).
+  */
+  override parse(uri: string, _strict?: boolean): Uri {
+    return this.filesystem(uri).parse(uri);
+  }
+
+
+  stat(uri: Uri): Promise<FileStat> {
+    return this.filesystem(uri).stat(uri);
+  }
+
+  async readDirectory(uri: Uri, options?: { recursive?: boolean }): Promise<Array<[Uri, FileType]>> {
+    return this.filesystem(uri).readDirectory(uri, options);
+  }
+
+  createDirectory(uri: Uri): Promise<void> {
+    return this.filesystem(uri).createDirectory(uri);
+  }
+
+  readFile(uri: Uri): Promise<Uint8Array> {
+    return this.filesystem(uri).readFile(uri);
+  }
+
+  openFile(uri: Uri): Promise<ReadHandle> {
+    return this.filesystem(uri).openFile(uri);
+  }
+
+  writeFile(uri: Uri, content: Uint8Array): Promise<void> {
+    return this.filesystem(uri).writeFile(uri, content);
+  }
+
+  readStream(uri: Uri, options?: { start?: number, end?: number }): Promise<Readable> {
+    return this.filesystem(uri).readStream(uri, options);
+  }
+
+  writeStream(uri: Uri, options?: WriteStreamOptions): Promise<Writable> {
+    return this.filesystem(uri).writeStream(uri, options);
+  }
+
+  delete(uri: Uri, options?: { recursive?: boolean | undefined; useTrash?: boolean | undefined; }): Promise<void> {
+    return this.filesystem(uri).delete(uri, options);
+  }
+
+  rename(source: Uri, target: Uri, options?: { overwrite?: boolean | undefined; }): Promise<void> {
+    strict.ok(source.fileSystem === target.fileSystem, i`may not rename across filesystems`);
+    return source.fileSystem.rename(source, target, options);
+  }
+
+  copy(source: Uri, target: Uri, options?: { overwrite?: boolean | undefined; }): Promise<number> {
+    return target.fileSystem.copy(source, target);
+  }
+
+  createSymlink(original: Uri, symlink: Uri): Promise<void> {
+    return symlink.fileSystem.createSymlink(original, symlink);
+  }
+}

ce/ce/fs/vsix-local-filesystem.ts

@@ -0,0 +1,28 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { Session } from '../session';
+import { Uri } from '../util/uri';
+import { LocalFileSystem } from './local-filesystem';
+
+export class VsixLocalFilesystem extends LocalFileSystem {
+  private readonly vsixBaseUri: Uri | undefined;
+
+  constructor(session: Session) {
+    super(session);
+    const programData = session.environment['ProgramData'];
+    if (programData) {
+      this.vsixBaseUri = this.file(programData).join('Microsoft/VisualStudio/Packages');
+    }
+  }
+
+  /**
+   * Creates a new URI from a string, e.g. `https://www.msft.com/some/path`,
+   * `file:///usr/home`, or `scheme:with/path`.
+   *
+   * @param value A string which represents an URI (see `URI#toString`).
+   */
+  override parse(value: string, _strict?: boolean): Uri {
+    return Uri.parseFilterVsix(this, value, _strict, this.vsixBaseUri);
+  }
+}

ce/ce/i18n.ts

@@ -0,0 +1,74 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { join } from 'path';
+
+/** what a language map looks like. */
+interface language {
+  [key: string]: (...args: Array<any>) => string;
+}
+
+type PrimitiveValue = string | number | boolean | undefined | Date;
+
+let translatorModule: language | undefined = undefined;
+
+function loadTranslatorModule(newLocale: string, basePath?: string) {
+  // eslint-disable-next-line @typescript-eslint/no-var-requires
+  return <language>(require(join(basePath || `${__dirname}/../i18n`, newLocale.toLowerCase())).map);
+}
+
+export function setLocale(newLocale: string, basePath?: string) {
+  try {
+    translatorModule = loadTranslatorModule(newLocale, basePath);
+  } catch {
+    // translation did not load.
+    // let's try to trim the locale and see if it fits
+    const l = newLocale.lastIndexOf('-');
+    if (l > -1) {
+      try {
+        const localeFiltered = newLocale.substr(0, l);
+        translatorModule = loadTranslatorModule(localeFiltered, basePath);
+      } catch {
+        // intentionally fall down to undefined setting below
+      }
+    }
+
+    // fallback to no translation
+    translatorModule = undefined;
+  }
+}
+
+/**
+ * processes a TaggedTemplateLiteral to return either:
+ * - a template string with numbered placeholders
+ * - or to resolve the template with the values given.
+ *
+ * @param literals The templateStringsArray from the templateFunction
+ * @param values the values from the template Function
+ * @param formatter an optional formatter (formats to ${##} if not specified)
+ */
+function normalize(literals: TemplateStringsArray, values: Array<PrimitiveValue>, formatter?: (value: PrimitiveValue) => string) {
+  const content = formatter ? literals.flatMap((k, i) => [k, formatter(values[i])]) : literals.flatMap((k, i) => [k, `$\{${i}}`]);
+  content.length--; // drop the trailing undefined.
+  return content.join('');
+}
+
+/**
+ * Support for tagged template literals for i18n.
+ *
+ * Leverages translation files in ../i18n
+ *
+ * @param literals the literal values in the tagged template
+ * @param values the inserted values in the template
+ *
+ * @translator
+ */
+export function i(literals: TemplateStringsArray, ...values: Array<string | number | boolean | undefined | Date>) {
+  // if the language has no translation, use the default content.
+  if (!translatorModule) {
+    return normalize(literals, values, (content) => `${content}`);
+  }
+  // use the translator module, but fallback to no translation if the file doesn't have a translation.
+  const fn = translatorModule[normalize(literals, values)];
+  return fn ? fn(...values) : normalize(literals, values, (content) => `${content}`);
+}

ce/ce/insights.ts

@@ -0,0 +1,71 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+
+import { defaultClient, DistributedTracingModes, setup } from 'applicationinsights';
+import { createHash } from 'crypto';
+import { session } from './main';
+import { Version } from './version';
+
+process.env['APPLICATION_INSIGHTS_NO_STATSBEAT'] = 'true';
+export const insights = setup('b4e88960-4393-4dd9-ab8e-97e8fe6d7603').
+  setAutoCollectConsole(false).
+  setAutoCollectDependencies(false).
+  setAutoCollectExceptions(false).
+  setAutoCollectHeartbeat(false).
+  setAutoCollectPerformance(false).
+  setAutoCollectPreAggregatedMetrics(false).
+  setAutoCollectRequests(false).
+  setAutoDependencyCorrelation(false).
+  setDistributedTracingMode(DistributedTracingModes.AI).
+  setInternalLogging(false).
+  setSendLiveMetrics(false).
+  setUseDiskRetryCaching(false).
+  start();
+
+defaultClient.context.keys.applicationVersion = Version;
+
+
+// todo: This will be refactored to allow appInsights to be called out-of-proc from the main process.
+//       in order to not potentially slow down or block on activation/etc.
+
+export function flushTelemetry() {
+  session.channels.debug('Ensuring Telemetry data is finished sending.');
+  defaultClient.flush({});
+}
+
+defaultClient.addTelemetryProcessor((envelope, contextObjects) => {
+  if (session.context['printmetrics']) {
+    session.channels.message(`Telemetry Event: \n${JSON.stringify(envelope.data, null, 2)}`);
+  }
+
+  // only actually send telemetry if it's enabled.
+  return session.telemetryEnabled;
+});
+
+export function trackEvent(name: string, properties: { [key: string]: string } = {}) {
+  session.channels.debug(`Triggering Telemetry Event ce.${name}`);
+
+  defaultClient.trackEvent({
+    name: `ce/${name}`,
+    time: new Date(),
+
+    properties: {
+      ...properties,
+    }
+  });
+}
+
+export function trackActivation() {
+  return trackEvent('activate', {});
+}
+
+export function trackAcquire(artifactId: string, artifactVersion: string) {
+  return trackEvent('acquire', {
+    'artifactId': createHash('sha256').update(artifactId, 'ascii').digest('hex'),
+    'artifactVersion': artifactVersion
+  });
+}

ce/ce/installers/git.ts

@@ -0,0 +1,49 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { CloneOptions, Git } from '../archivers/git';
+import { Activation } from '../artifacts/activation';
+import { i } from '../i18n';
+import { InstallEvents, InstallOptions } from '../interfaces/events';
+import { CloneSettings, GitInstaller } from '../interfaces/metadata/installers/git';
+import { Session } from '../session';
+import { linq } from '../util/linq';
+import { Uri } from '../util/uri';
+
+export async function installGit(session: Session, activation: Activation, name: string, targetLocation: Uri, install: GitInstaller, events: Partial<InstallEvents>, options: Partial<InstallOptions & CloneOptions & CloneSettings>): Promise<void> {
+  const gitPath = linq.find(activation.tools, 'git');
+
+  if (!gitPath) {
+    throw new Error(i`Git is not installed`);
+  }
+
+  const repo = session.parseUri(install.location);
+  const targetDirectory = targetLocation.join(options.subdirectory ?? '');
+
+  const gitTool = new Git(session, gitPath, activation.environmentBlock, targetDirectory);
+
+  await gitTool.clone(repo, events, {
+    recursive: install.recurse,
+    depth: install.full ? undefined : 1,
+  });
+
+  if (install.commit) {
+    if (install.full) {
+      await gitTool.reset(events, {
+        commit: install.commit,
+        recurse: install.recurse,
+        hard: true
+      });
+    }
+    else {
+      await gitTool.fetch('origin', events, {
+        commit: install.commit,
+        recursive: install.recurse,
+        depth: install.full ? undefined : 1
+      });
+      await gitTool.checkout(events, {
+        commit: install.commit
+      });
+    }
+  }
+}

ce/ce/installers/nuget.ts

@@ -0,0 +1,24 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { ZipUnpacker } from '../archivers/ZipUnpacker';
+import { Activation } from '../artifacts/activation';
+import { acquireNugetFile } from '../fs/acquire';
+import { InstallEvents, InstallOptions } from '../interfaces/events';
+import { NupkgInstaller } from '../interfaces/metadata/installers/nupkg';
+import { Session } from '../session';
+import { Uri } from '../util/uri';
+import { applyAcquireOptions } from './util';
+
+export async function installNuGet(session: Session, activation: Activation, name: string, targetLocation: Uri, install: NupkgInstaller, events: Partial<InstallEvents>, options: Partial<InstallOptions>): Promise<void> {
+  const file = await acquireNugetFile(session, install.location, `${name}.zip`, events, applyAcquireOptions(options, install));
+
+  return new ZipUnpacker(session).unpack(
+    file,
+    targetLocation,
+    events,
+    {
+      strip: install.strip,
+      transform: [...install.transform],
+    });
+}

ce/ce/installers/untar.ts

@@ -0,0 +1,28 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+import { TarBzUnpacker, TarGzUnpacker, TarUnpacker } from '../archivers/tar';
+import { Unpacker } from '../archivers/unpacker';
+import { Activation } from '../artifacts/activation';
+import { acquireArtifactFile } from '../fs/acquire';
+import { InstallEvents, InstallOptions } from '../interfaces/events';
+import { UnTarInstaller } from '../interfaces/metadata/installers/tar';
+import { Session } from '../session';
+import { Uri } from '../util/uri';
+import { applyAcquireOptions, artifactFileName } from './util';
+
+
+export async function installUnTar(session: Session, activation: Activation, name: string, targetLocation: Uri, install: UnTarInstaller, events: Partial<InstallEvents>, options: Partial<InstallOptions>): Promise<void> {
+  const file = await acquireArtifactFile(session, [...install.location].map(each => session.parseUri(each)), artifactFileName(name, install, '.tar'), events, applyAcquireOptions(options, install));
+  const x = await file.readBlock(0, 128);
+  let unpacker: Unpacker;
+  if (x[0] === 0x1f && x[1] === 0x8b) {
+    unpacker = new TarGzUnpacker(session);
+  } else if (x[0] === 66 && x[1] === 90) {
+    unpacker = new TarBzUnpacker(session);
+  } else {
+    unpacker = new TarUnpacker(session);
+  }
+
+  return unpacker.unpack(file, targetLocation, events, { strip: install.strip, transform: [...install.transform] });
+}