Add `Scan` functionality for assets-maintenance-tool (#6269)
* Add new project `assets-maintenance-tool`. Implement `scan` functionality within. `backup` and `cleanup` still to come. * Add integration tests for new assets-maintenance tool. * Move original `asset-sync` powershell proto script into `tools/assets-automation/` folder * Move `eng/scripts/python/assets-automation/` under `tools/assets-automation/assets-reporting/` * Add ci.yml and tests.yml to invoke integration tests against the azure-sdk-assets-integration Co-authored-by: Konrad Jamrozik <kojamroz@microsoft.com>
This commit is contained in:
Scott Beddall
GitHub
Родитель
7b6c275247
Коммит
81667b192f
|
|
@ -3,14 +3,15 @@ This page contains guidelines for developing or updating powershell scripts used
|
|||
Table of Contents
|
||||
=================
|
||||
|
||||
* [TLDR](#tldr)
|
||||
* [Structure](#structure)
|
||||
* [Functionality](#functionality)
|
||||
* [Style](#style)
|
||||
* [Testing](#testing)
|
||||
* [Unit Testing](#unit-testing)
|
||||
* [Running Pester Tests](#running-pester-tests)
|
||||
* [Local/Pipeline Functional Testing](#localpipeline-functional-testing)
|
||||
- [Table of Contents](#table-of-contents)
|
||||
- [TLDR](#tldr)
|
||||
- [Structure](#structure)
|
||||
- [Functionality](#functionality)
|
||||
- [Style](#style)
|
||||
- [Testing](#testing)
|
||||
- [Unit Testing](#unit-testing)
|
||||
- [Running Pester Tests](#running-pester-tests)
|
||||
- [Local/Pipeline Functional Testing](#localpipeline-functional-testing)
|
||||
|
||||
## TLDR
|
||||
|
||||
|
|
@ -110,13 +111,13 @@ Powershell scripts should be testable, via one or more methods:
|
|||
Unit tests should be written for all scripts, and should utilize [Pester](https://pester.dev/).
|
||||
|
||||
- Tests can be located alongside scripts in a directory called `tests`.
|
||||
- Example pester test suites: [job matrix tests](https://github.com/Azure/azure-sdk-tools/tree/main/eng/common/scripts/job-matrix/tests), [asset sync tests](https://github.com/Azure/azure-sdk-tools/blob/main/tools/asset-sync/assets.Tests.ps1)
|
||||
- Example pester test suites: [job matrix tests](https://github.com/Azure/azure-sdk-tools/tree/main/eng/common/scripts/job-matrix/tests), [asset sync tests](https://github.com/Azure/azure-sdk-tools/blob/main/tools/assets-automation/asset-sync/assets.Tests.ps1)
|
||||
- A CI pipeline should be defined to run scripts unit tests at the very least. See [archetype-sdk-tool-pwsh](https://github.com/Azure/azure-sdk-tools/blob/main/eng/common/pipelines/templates/stages/archetype-sdk-tool-pwsh.yml) for how to do this.
|
||||
- Script code should always be written so as much of the surface area as possible can be run via unit tests. Move code that calls out to external dependencies into modular functions, and simplify context/data structures passed to functions as much as possible to it can be easily mocked.
|
||||
|
||||
#### Running Pester Tests
|
||||
|
||||
(stolen from https://github.com/Azure/azure-sdk-tools/blob/main/tools/asset-sync/contributing.md).
|
||||
(stolen from https://github.com/Azure/azure-sdk-tools/blob/main/tools/assets-automation/asset-sync/contributing.md).
|
||||
|
||||
> **First, ensure you have `pester` installed:**
|
||||
>
|
||||
|
|
|
|||
|
|
@ -0,0 +1,9 @@
|
|||
# Assets automation tooling
|
||||
|
||||
This directory contains tooling pertaining to the _support_ of externalized assets created by the [azure-sdk test-proxy](../test-proxy/Azure.Sdk.Tools.TestProxy/README.md).
|
||||
|
||||
| Directory | Description |
|
||||
|---|---|
|
||||
| [assets-maintenance-tool](./assets-maintenance-tool/README.md) | CLI tool used to scan, backup, and clean azure-sdk assets across all repositories. |
|
||||
| [assets-reporting](./assets-reporting/README.md) | CLI tool used to audit current repositories and find status of test-proxy adoption on a per-package basis. Used to generate weekly reporting. |
|
||||
| [asset-sync](./assets-sync/README.md) | Deprecated initial version of `asset-sync` implementation. The current implementation ended up integrated directly into the `test-proxy` codebase, rather than existing as an external powershell script. |
|
||||
|
|
@ -0,0 +1,36 @@
|
|||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<TargetFramework>net6.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
|
||||
<IsPackable>false</IsPackable>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.1.0" />
|
||||
<PackageReference Include="NUnit" Version="3.13.3" />
|
||||
<PackageReference Include="NUnit3TestAdapter" Version="4.2.1" />
|
||||
<PackageReference Include="NUnit.Analyzers" Version="3.3.0" />
|
||||
<PackageReference Include="coverlet.collector" Version="3.1.2" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<Folder Include="TestResources\basic_output\" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\Azure.Sdk.Tools.Assets.MaintenanceTool\Azure.Sdk.Tools.Assets.MaintenanceTool.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<None Update="TestResources\basic_output\output.json">
|
||||
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
|
||||
</None>
|
||||
<None Update="TestResources\configurations\sample-repo-configuration.json">
|
||||
<CopyToOutputDirectory>Always</CopyToOutputDirectory>
|
||||
</None>
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
|
@ -0,0 +1,59 @@
|
|||
global using NUnit.Framework;
|
||||
using Azure.Sdk.Tools.Assets.MaintenanceTool.Model;
|
||||
using Azure.Sdk.Tools.Assets.MaintenanceTool.Scan;
|
||||
using Microsoft.AspNetCore.Http;
|
||||
using Newtonsoft.Json;
|
||||
|
||||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Tests
|
||||
{
|
||||
public class CLITests
|
||||
{
|
||||
public string TestDirectory { get; protected set; }
|
||||
|
||||
[SetUp]
|
||||
public void Setup()
|
||||
{
|
||||
var workingDirectory = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString());
|
||||
|
||||
if (!Directory.Exists(workingDirectory))
|
||||
{
|
||||
Directory.CreateDirectory(workingDirectory);
|
||||
}
|
||||
|
||||
// copy our static test files there
|
||||
var source = Path.Combine(Directory.GetCurrentDirectory(), "TestResources");
|
||||
var target = Path.Combine(workingDirectory, "TestResources");
|
||||
|
||||
Microsoft.VisualBasic.FileIO.FileSystem.CopyDirectory(source, target);
|
||||
|
||||
TestDirectory = workingDirectory;
|
||||
}
|
||||
|
||||
[TearDown]
|
||||
public void TearDown()
|
||||
{
|
||||
Directory.Delete(TestDirectory, true);
|
||||
}
|
||||
|
||||
[Test]
|
||||
[TestCase("scan", "-c", "")]
|
||||
[TestCase("scan", "--config", "")]
|
||||
public void TestScanOptions(params string[] args)
|
||||
{
|
||||
}
|
||||
|
||||
[Test]
|
||||
[TestCase("scan")]
|
||||
[TestCase("scan", "--config")]
|
||||
public void TestInvalidScanOptions(params string[] args)
|
||||
{
|
||||
var obj = new object();
|
||||
|
||||
var rootCommand = Program.InitializeCommandOptions((DefaultOptions) =>
|
||||
{
|
||||
obj = DefaultOptions;
|
||||
});
|
||||
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.Linq;
|
||||
using System.Text;
|
||||
using System.Threading.Tasks;
|
||||
using NUnit.Framework.Interfaces;
|
||||
using NUnit.Framework.Internal;
|
||||
|
||||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Tests
|
||||
{
|
||||
public class GitTokenSkipAttribute : NUnitAttribute, IApplyToTest
|
||||
{
|
||||
public GitTokenSkipAttribute() { }
|
||||
|
||||
public void ApplyToTest(Test test)
|
||||
{
|
||||
if (!string.IsNullOrEmpty(Environment.GetEnvironmentVariable("TF_BUILD")) && string.IsNullOrEmpty(Environment.GetEnvironmentVariable("GIT_TOKEN")))
|
||||
{
|
||||
new IgnoreAttribute("Skipping this test. Within a CI run, and GIT_TOKEN is not set.").ApplyToTest(test);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,341 @@
|
|||
global using NUnit.Framework;
|
||||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using Azure.Sdk.Tools.Assets.MaintenanceTool.Model;
|
||||
using Azure.Sdk.Tools.Assets.MaintenanceTool.Scan;
|
||||
|
||||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Tests
|
||||
{
|
||||
public class ScanTests
|
||||
{
|
||||
// These tests assume the presence of three integration branches for the purposes of testing.
|
||||
// Azure/azure-sdk-tools
|
||||
// https://github.com/Azure/azure-sdk-tools/tree/integration/assets-test-branch
|
||||
// Azure/azure-sdk-assets-integration
|
||||
// This repo is non-public, and as such a valid GIT_TOKEN must be set or the default git account
|
||||
// on the invoking machine should have permissions to Azure/azure-sdk-assets-integration.
|
||||
//
|
||||
// https://github.com/Azure/azure-sdk-assets-integration/tree/integration/assets-branch-1
|
||||
// https://github.com/Azure/azure-sdk-assets-integration/tree/integration/assets-branch-2
|
||||
private RunConfiguration RunConfiguration
|
||||
{
|
||||
get
|
||||
{
|
||||
return new RunConfiguration()
|
||||
{
|
||||
LanguageRepos = new List<RepoConfiguration>
|
||||
{
|
||||
new RepoConfiguration("azure/azure-sdk-tools")
|
||||
{
|
||||
Branches = new List<string>() { "integration/assets-test-branch" }
|
||||
},
|
||||
new RepoConfiguration("azure/azure-sdk-assets-integration")
|
||||
{
|
||||
Branches = new List<string>()
|
||||
{
|
||||
"integration/assets-branch-1",
|
||||
"integration/assets-branch-2"
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
public string TestDirectory { get; protected set; }
|
||||
|
||||
[SetUp]
|
||||
public void Setup()
|
||||
{
|
||||
var workingDirectory = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString());
|
||||
|
||||
if (!Directory.Exists(workingDirectory))
|
||||
{
|
||||
Directory.CreateDirectory(workingDirectory);
|
||||
}
|
||||
|
||||
// copy our static test files there
|
||||
var source = Path.Combine(Directory.GetCurrentDirectory(), "TestResources");
|
||||
var target = Path.Combine(workingDirectory, "TestResources");
|
||||
|
||||
Microsoft.VisualBasic.FileIO.FileSystem.CopyDirectory(source, target);
|
||||
|
||||
TestDirectory = workingDirectory;
|
||||
}
|
||||
|
||||
[TearDown]
|
||||
public void TearDown()
|
||||
{
|
||||
Directory.Delete(TestDirectory, true);
|
||||
}
|
||||
|
||||
[Test]
|
||||
[GitTokenSkip]
|
||||
public void TestBasicScanSingleBranch()
|
||||
{
|
||||
var scanner = new AssetsScanner(TestDirectory);
|
||||
var config = RunConfiguration;
|
||||
config.LanguageRepos.RemoveAt(0);
|
||||
config.LanguageRepos.First().Branches.RemoveAt(1);
|
||||
var results = scanner.Scan(config);
|
||||
|
||||
Assert.IsNotNull(results);
|
||||
Assert.That(results.Results.Count(), Is.EqualTo(3));
|
||||
|
||||
var payload0 = results.Results[0];
|
||||
var payload1 = results.Results[1];
|
||||
var payload2 = results.Results[2];
|
||||
|
||||
Assert.That(payload0.AssetsLocation, Is.EqualTo("sdk/agrifood/arm-agrifood/assets.json"));
|
||||
Assert.That(payload0.Commit, Is.EqualTo("48bca526a2a9972e4219ec87d29a7aa31438581a"));
|
||||
Assert.That(payload0.LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(payload0.Tag, Is.EqualTo("js/agrifood/arm-agrifood_4f244d09c7"));
|
||||
Assert.That(payload0.AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
|
||||
Assert.That(payload1.AssetsLocation, Is.EqualTo("sdk/storage/storage-blob/assets.json"));
|
||||
Assert.That(payload1.Commit, Is.EqualTo("48bca526a2a9972e4219ec87d29a7aa31438581a"));
|
||||
Assert.That(payload1.LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(payload1.Tag, Is.EqualTo("js/storage/storage-blob_5d5a32b74a"));
|
||||
Assert.That(payload1.AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
|
||||
Assert.That(payload2.AssetsLocation, Is.EqualTo("sdk/agrifood/arm-agrifood/assets.json"));
|
||||
Assert.That(payload2.Commit, Is.EqualTo("f139c4ddf7aaa4d637282ae7da4466b473044281"));
|
||||
Assert.That(payload2.LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(payload2.Tag, Is.EqualTo("js/agrifood/arm-agrifood_4f244d09c7"));
|
||||
Assert.That(payload2.AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
}
|
||||
|
||||
[Test]
|
||||
[GitTokenSkip]
|
||||
public void TestBasicScanMultipleBranches()
|
||||
{
|
||||
var scanner = new AssetsScanner(TestDirectory);
|
||||
var config = RunConfiguration;
|
||||
config.LanguageRepos.RemoveAt(0);
|
||||
var results = scanner.Scan(config);
|
||||
|
||||
Assert.IsNotNull(results);
|
||||
Assert.That(results.Results.Count(), Is.EqualTo(6));
|
||||
|
||||
Assert.That(results.Results[0].AssetsLocation, Is.EqualTo("sdk/agrifood/arm-agrifood/assets.json"));
|
||||
Assert.That(results.Results[0].Commit, Is.EqualTo("48bca526a2a9972e4219ec87d29a7aa31438581a"));
|
||||
Assert.That(results.Results[0].LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(results.Results[0].Tag, Is.EqualTo("js/agrifood/arm-agrifood_4f244d09c7"));
|
||||
Assert.That(results.Results[0].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
|
||||
Assert.That(results.Results[1].AssetsLocation, Is.EqualTo("sdk/storage/storage-blob/assets.json"));
|
||||
Assert.That(results.Results[1].Commit, Is.EqualTo("48bca526a2a9972e4219ec87d29a7aa31438581a"));
|
||||
Assert.That(results.Results[1].LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(results.Results[1].Tag, Is.EqualTo("js/storage/storage-blob_5d5a32b74a"));
|
||||
Assert.That(results.Results[1].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
|
||||
Assert.That(results.Results[2].AssetsLocation, Is.EqualTo("sdk/appconfiguration/app-configuration/assets.json"));
|
||||
Assert.That(results.Results[2].Commit, Is.EqualTo("4b6ee6ea00af2384c0dcc0558e9b96d8051aa8cf"));
|
||||
Assert.That(results.Results[2].LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(results.Results[2].Tag, Is.EqualTo("js/appconfiguration/app-configuration_61261605e2"));
|
||||
Assert.That(results.Results[2].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
|
||||
Assert.That(results.Results[3].AssetsLocation, Is.EqualTo("sdk/keyvault/keyvault-certificates/assets.json"));
|
||||
Assert.That(results.Results[3].Commit, Is.EqualTo("4b6ee6ea00af2384c0dcc0558e9b96d8051aa8cf"));
|
||||
Assert.That(results.Results[3].LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(results.Results[3].Tag, Is.EqualTo("js/keyvault/keyvault-certificates_43821e21b3"));
|
||||
Assert.That(results.Results[3].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
|
||||
Assert.That(results.Results[4].AssetsLocation, Is.EqualTo("sdk/keyvault/keyvault-keys/assets.json"));
|
||||
Assert.That(results.Results[4].Commit, Is.EqualTo("4b6ee6ea00af2384c0dcc0558e9b96d8051aa8cf"));
|
||||
Assert.That(results.Results[4].LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(results.Results[4].Tag, Is.EqualTo("js/keyvault/keyvault-keys_b69a5239e9"));
|
||||
Assert.That(results.Results[4].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
|
||||
Assert.That(results.Results[5].AssetsLocation, Is.EqualTo("sdk/agrifood/arm-agrifood/assets.json"));
|
||||
Assert.That(results.Results[5].Commit, Is.EqualTo("f139c4ddf7aaa4d637282ae7da4466b473044281"));
|
||||
Assert.That(results.Results[5].LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(results.Results[5].Tag, Is.EqualTo("js/agrifood/arm-agrifood_4f244d09c7"));
|
||||
Assert.That(results.Results[5].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
}
|
||||
|
||||
[Test]
|
||||
[GitTokenSkip]
|
||||
public void TestBasicScanMultipleBranchesMultipleRepos()
|
||||
{
|
||||
var scanner = new AssetsScanner(TestDirectory);
|
||||
var config = RunConfiguration;
|
||||
var results = scanner.Scan(config);
|
||||
|
||||
Assert.IsNotNull(results);
|
||||
Assert.That(results.Results.Count(), Is.EqualTo(8));
|
||||
|
||||
Assert.That(results.ByLanguageRepo.Keys.Count(), Is.EqualTo(2));
|
||||
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-tools"][0].AssetsLocation, Is.EqualTo("sdk/formrecognizer/azure-ai-formrecognizer/assets.json"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-tools"][0].Commit, Is.EqualTo("eeeee9e00cc0d0111edf7471962b0da826d9a5cc"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-tools"][0].LanguageRepo, Is.EqualTo("azure/azure-sdk-tools"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-tools"][0].Tag, Is.EqualTo("python/formrecognizer/azure-ai-formrecognizer_f60081bf10"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-tools"][0].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-tools"][1].AssetsLocation, Is.EqualTo("sdk/keyvault/assets.json"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-tools"][1].Commit, Is.EqualTo("eeeee9e00cc0d0111edf7471962b0da826d9a5cc"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-tools"][1].LanguageRepo, Is.EqualTo("azure/azure-sdk-tools"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-tools"][1].Tag, Is.EqualTo("python/keyvault/azure-keyvault-administration_f6e776f55f"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-tools"][1].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-assets-integration"][5].AssetsLocation, Is.EqualTo("sdk/agrifood/arm-agrifood/assets.json"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-assets-integration"][5].Commit, Is.EqualTo("f139c4ddf7aaa4d637282ae7da4466b473044281"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-assets-integration"][5].LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-assets-integration"][5].Tag, Is.EqualTo("js/agrifood/arm-agrifood_4f244d09c7"));
|
||||
Assert.That(results.ByLanguageRepo["azure/azure-sdk-assets-integration"][5].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
}
|
||||
|
||||
[Test]
|
||||
[GitTokenSkip]
|
||||
public void TestScanHonorsPreviousResults()
|
||||
{
|
||||
var specificDirectory = Path.Combine(TestDirectory, "TestResources", "basic_output");
|
||||
var scanner = new AssetsScanner(specificDirectory);
|
||||
|
||||
var config = RunConfiguration;
|
||||
var resultSet = scanner.Scan(config);
|
||||
var scanDate = DateTime.UtcNow.AddSeconds(-10);
|
||||
Assert.That(resultSet, Is.Not.Null);
|
||||
|
||||
// The only elements that _could_ change (given that git is immutable per commit)
|
||||
// are the BackupURIs and the ScanDates. If we have properly loaded previous results,
|
||||
// these should align.
|
||||
var resultsForCommit48b = resultSet.ByOriginSHA["48bca526a2a9972e4219ec87d29a7aa31438581a"];
|
||||
var resultsForf1 = resultSet.ByOriginSHA["f139c4ddf7aaa4d637282ae7da4466b473044281"];
|
||||
var resultsFore6 = resultSet.ByOriginSHA["eeeee9e00cc0d0111edf7471962b0da826d9a5cc"];
|
||||
|
||||
Assert.That(resultsForCommit48b[0].BackupUri, Is.Null);
|
||||
Assert.That(resultsForCommit48b[0].CreationDate, Is.EqualTo(DateTime.Parse("2023-05-11T11:05:59")));
|
||||
Assert.That(resultsForCommit48b[1].BackupUri, Is.Null);
|
||||
Assert.That(resultsForCommit48b[1].CreationDate, Is.EqualTo(DateTime.Parse("2023-05-10T00:00:00")));
|
||||
Assert.That(resultsForCommit48b.Count(), Is.EqualTo(2));
|
||||
|
||||
Assert.That(resultsForf1[0].BackupUri, Is.EqualTo("https://github.com/azure-sdk/azure-sdk-assets/tree/backup_js/agrifood/arm-agrifood_4f244d09c7"));
|
||||
Assert.That(resultsForf1[0].CreationDate, Is.EqualTo(DateTime.Parse("2023-05-08T00:00:00")));
|
||||
Assert.That(resultsForf1.Count(), Is.EqualTo(1));
|
||||
|
||||
Assert.That(resultsFore6[0].AssetsLocation, Is.EqualTo("sdk/formrecognizer/azure-ai-formrecognizer/assets.json"));
|
||||
Assert.That(resultsFore6[0].Commit, Is.EqualTo("eeeee9e00cc0d0111edf7471962b0da826d9a5cc"));
|
||||
Assert.That(resultsFore6[0].LanguageRepo, Is.EqualTo("azure/azure-sdk-tools"));
|
||||
Assert.That(resultsFore6[0].Tag, Is.EqualTo("python/formrecognizer/azure-ai-formrecognizer_f60081bf10"));
|
||||
Assert.That(resultsFore6[0].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
Assert.That(resultsFore6[0].CreationDate, Is.GreaterThanOrEqualTo(scanDate));
|
||||
|
||||
Assert.That(resultsFore6[1].AssetsLocation, Is.EqualTo("sdk/keyvault/assets.json"));
|
||||
Assert.That(resultsFore6[1].Commit, Is.EqualTo("eeeee9e00cc0d0111edf7471962b0da826d9a5cc"));
|
||||
Assert.That(resultsFore6[1].LanguageRepo, Is.EqualTo("azure/azure-sdk-tools"));
|
||||
Assert.That(resultsFore6[1].Tag, Is.EqualTo("python/keyvault/azure-keyvault-administration_f6e776f55f"));
|
||||
Assert.That(resultsFore6[1].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
Assert.That(resultsFore6[1].CreationDate, Is.GreaterThanOrEqualTo(scanDate));
|
||||
Assert.That(resultsFore6.Count(), Is.EqualTo(2));
|
||||
}
|
||||
|
||||
[Test]
|
||||
[GitTokenSkip]
|
||||
public void TestParsePreviouslyOutputResults()
|
||||
{
|
||||
var specificDirectory = Path.Combine(TestDirectory, "TestResources", "basic_output");
|
||||
var scanner = new AssetsScanner(specificDirectory);
|
||||
|
||||
// ensure that we can parse a default set of existing results
|
||||
var resultSet = scanner.ParseExistingResults();
|
||||
|
||||
Assert.That(resultSet, Is.Not.Null);
|
||||
|
||||
Assert.That(resultSet.Results[0].AssetsLocation, Is.EqualTo("sdk/agrifood/arm-agrifood/assets.json"));
|
||||
Assert.That(resultSet.Results[0].Commit, Is.EqualTo("48bca526a2a9972e4219ec87d29a7aa31438581a"));
|
||||
Assert.That(resultSet.Results[0].LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(resultSet.Results[0].Tag, Is.EqualTo("js/agrifood/arm-agrifood_4f244d09c7"));
|
||||
Assert.That(resultSet.Results[0].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
Assert.That(resultSet.Results[0].BackupUri, Is.Null);
|
||||
Assert.That(resultSet.Results[0].CreationDate, Is.EqualTo(DateTime.Parse("2023-05-11T11:05:59")));
|
||||
|
||||
Assert.That(resultSet.Results[1].AssetsLocation, Is.EqualTo("sdk/storage/storage-blob/assets.json"));
|
||||
Assert.That(resultSet.Results[1].Commit, Is.EqualTo("48bca526a2a9972e4219ec87d29a7aa31438581a"));
|
||||
Assert.That(resultSet.Results[1].LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(resultSet.Results[1].Tag, Is.EqualTo("js/storage/storage-blob_5d5a32b74a"));
|
||||
Assert.That(resultSet.Results[1].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
Assert.That(resultSet.Results[1].BackupUri, Is.Null);
|
||||
Assert.That(resultSet.Results[1].CreationDate, Is.EqualTo(DateTime.Parse("2023-05-10T00:00:00")));
|
||||
|
||||
Assert.That(resultSet.Results[2].AssetsLocation, Is.EqualTo("sdk/agrifood/arm-agrifood/assets.json"));
|
||||
Assert.That(resultSet.Results[2].Commit, Is.EqualTo("f139c4ddf7aaa4d637282ae7da4466b473044281"));
|
||||
Assert.That(resultSet.Results[2].LanguageRepo, Is.EqualTo("azure/azure-sdk-assets-integration"));
|
||||
Assert.That(resultSet.Results[2].Tag, Is.EqualTo("js/agrifood/arm-agrifood_4f244d09c7"));
|
||||
Assert.That(resultSet.Results[2].AssetsRepo, Is.EqualTo("Azure/azure-sdk-assets"));
|
||||
Assert.That(resultSet.Results[2].BackupUri, Is.EqualTo("https://github.com/azure-sdk/azure-sdk-assets/tree/backup_js/agrifood/arm-agrifood_4f244d09c7"));
|
||||
Assert.That(resultSet.Results[2].CreationDate, Is.EqualTo(DateTime.Parse("2023-05-08T00:00:00")));
|
||||
}
|
||||
|
||||
[Test]
|
||||
[GitTokenSkip]
|
||||
public void TestLoadConfiguration()
|
||||
{
|
||||
var scanner = new AssetsScanner(TestDirectory);
|
||||
var runConfigurationPath = Path.Combine(TestDirectory, "TestResources", "configurations", "sample-repo-configuration.json");
|
||||
var config = new RunConfiguration(runConfigurationPath);
|
||||
|
||||
var results = scanner.Scan(config);
|
||||
|
||||
Assert.That(results.Results.Count, Is.EqualTo(8));
|
||||
}
|
||||
|
||||
[Test]
|
||||
[GitTokenSkip]
|
||||
public void TestScanOutputsResults()
|
||||
{
|
||||
var scanner = new AssetsScanner(TestDirectory);
|
||||
var config = RunConfiguration;
|
||||
config.LanguageRepos.RemoveAt(0);
|
||||
config.LanguageRepos.First().Branches.RemoveAt(1);
|
||||
var resultSet = scanner.Scan(config);
|
||||
scanner.Save(resultSet);
|
||||
|
||||
var fileThatShouldExist = Path.Combine(TestDirectory, "output.json");
|
||||
var newOutFile = Path.Combine(TestDirectory, "test_output");
|
||||
|
||||
Assert.That(File.Exists(fileThatShouldExist), Is.EqualTo(true));
|
||||
|
||||
var parsedNewResults = scanner.ParseExistingResults();
|
||||
|
||||
using (var stream = System.IO.File.OpenWrite(newOutFile))
|
||||
{
|
||||
stream.Write(Encoding.UTF8.GetBytes(JsonSerializer.Serialize(RunConfiguration)));
|
||||
}
|
||||
|
||||
if (parsedNewResults != null)
|
||||
{
|
||||
AssertResultsSame(resultSet, parsedNewResults);
|
||||
}
|
||||
else
|
||||
{
|
||||
Assert.NotNull(parsedNewResults);
|
||||
}
|
||||
}
|
||||
|
||||
private bool AssertResultsSame(AssetsResultSet a, AssetsResultSet b)
|
||||
{
|
||||
if (a.Results.Count() != b.Results.Count())
|
||||
{
|
||||
return false;
|
||||
}
|
||||
|
||||
for (int i = 0; i < a.Results.Count(); i++)
|
||||
{
|
||||
AssetsResult aResult = a.Results[i];
|
||||
AssetsResult bResult = b.Results[i];
|
||||
|
||||
Assert.That(bResult.CreationDate, Is.EqualTo(aResult.CreationDate));
|
||||
Assert.That(bResult.AssetsLocation, Is.EqualTo(aResult.AssetsLocation));
|
||||
Assert.That(bResult.Tag, Is.EqualTo(aResult.Tag));
|
||||
Assert.That(bResult.AssetsRepo, Is.EqualTo(aResult.AssetsRepo));
|
||||
Assert.That(bResult.Commit, Is.EqualTo(aResult.Commit));
|
||||
Assert.That(bResult.LanguageRepo, Is.EqualTo(aResult.LanguageRepo));
|
||||
Assert.That(bResult.BackupUri, Is.EqualTo(aResult.BackupUri));
|
||||
}
|
||||
|
||||
return true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,29 @@
|
|||
[
|
||||
{
|
||||
"LanguageRepo": "azure/azure-sdk-assets-integration",
|
||||
"Commit": "48bca526a2a9972e4219ec87d29a7aa31438581a",
|
||||
"AssetsLocation": "sdk/agrifood/arm-agrifood/assets.json",
|
||||
"Tag": "js/agrifood/arm-agrifood_4f244d09c7",
|
||||
"AssetsRepo": "Azure/azure-sdk-assets",
|
||||
"BackupUri": null,
|
||||
"CreationDate": "2023-05-11T11:05:59"
|
||||
},
|
||||
{
|
||||
"LanguageRepo": "azure/azure-sdk-assets-integration",
|
||||
"Commit": "48bca526a2a9972e4219ec87d29a7aa31438581a",
|
||||
"AssetsLocation": "sdk/storage/storage-blob/assets.json",
|
||||
"Tag": "js/storage/storage-blob_5d5a32b74a",
|
||||
"AssetsRepo": "Azure/azure-sdk-assets",
|
||||
"BackupUri": null,
|
||||
"CreationDate": "2023-05-10T00:00:00"
|
||||
},
|
||||
{
|
||||
"LanguageRepo": "azure/azure-sdk-assets-integration",
|
||||
"Commit": "f139c4ddf7aaa4d637282ae7da4466b473044281",
|
||||
"AssetsLocation": "sdk/agrifood/arm-agrifood/assets.json",
|
||||
"Tag": "js/agrifood/arm-agrifood_4f244d09c7",
|
||||
"AssetsRepo": "Azure/azure-sdk-assets",
|
||||
"BackupUri": "https://github.com/azure-sdk/azure-sdk-assets/tree/backup_js/agrifood/arm-agrifood_4f244d09c7",
|
||||
"CreationDate": "2023-05-08T00:00:00"
|
||||
}
|
||||
]
|
||||
|
|
@ -0,0 +1,17 @@
|
|||
{
|
||||
"LanguageRepos": [
|
||||
{
|
||||
"LanguageRepo": "azure/azure-sdk-tools",
|
||||
"Branches": [
|
||||
"integration/assets-test-branch"
|
||||
]
|
||||
},
|
||||
{
|
||||
"LanguageRepo": "azure/azure-sdk-assets-integration",
|
||||
"Branches": [
|
||||
"integration/assets-branch-1",
|
||||
"integration/assets-branch-2"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
|
|
@ -0,0 +1,19 @@
|
|||
<Project Sdk="Microsoft.NET.Sdk">
|
||||
|
||||
<PropertyGroup>
|
||||
<OutputType>Exe</OutputType>
|
||||
<TargetFramework>net6.0</TargetFramework>
|
||||
<ImplicitUsings>enable</ImplicitUsings>
|
||||
<Nullable>enable</Nullable>
|
||||
<WarningsAsErrors>nullable</WarningsAsErrors>
|
||||
</PropertyGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="System.CommandLine" Version="2.0.0-beta4.22272.1" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
<ProjectReference Include="..\..\..\test-proxy\Azure.Sdk.Tools.TestProxy\Azure.Sdk.Tools.TestProxy.csproj" />
|
||||
</ItemGroup>
|
||||
|
||||
</Project>
|
||||
|
|
@ -0,0 +1,13 @@
|
|||
using Azure.Sdk.Tools.Assets.MaintenanceTool.Model;
|
||||
|
||||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Cleanup;
|
||||
|
||||
public class AssetsCleanupClient
|
||||
{
|
||||
public AssetsCleanupClient() { }
|
||||
|
||||
public AssetsResultSet Cleanup(RunConfiguration config, AssetsResultSet backupResult)
|
||||
{
|
||||
return new AssetsResultSet(new List<AssetsResult>()); ;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,69 @@
|
|||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Model
|
||||
{
|
||||
/// <summary>
|
||||
/// This class represents a single reference TO the assets repo FROM a language repo. It is populated by combining
|
||||
/// an assets's json content with metadata about which language repo it was discovered in.
|
||||
///
|
||||
/// The "language" repo is the repo from which recordings are being generated. The "assets" repo is the storage medium for a set
|
||||
/// of recordings that are generated by invoking tests in record mode FROM the language repo.
|
||||
/// </summary>
|
||||
public class AssetsResult
|
||||
{
|
||||
public AssetsResult(string repo, string repoCommit, string assetsLocation, string tag, string tagRepo, string? backupUri = null, DateTime? scanDate = null)
|
||||
{
|
||||
LanguageRepo = repo;
|
||||
Commit = repoCommit;
|
||||
AssetsLocation = assetsLocation;
|
||||
Tag = tag;
|
||||
AssetsRepo = tagRepo;
|
||||
BackupUri = backupUri;
|
||||
CreationDate = scanDate??DateTime.UtcNow;
|
||||
}
|
||||
|
||||
public AssetsResult()
|
||||
{
|
||||
LanguageRepo = string.Empty;
|
||||
Commit = string.Empty;
|
||||
AssetsLocation = string.Empty;
|
||||
Tag = string.Empty;
|
||||
AssetsRepo = string.Empty;
|
||||
BackupUri = string.Empty;
|
||||
CreationDate = DateTime.UtcNow;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The containing language repo from within which this result was generated.
|
||||
/// </summary>
|
||||
public string LanguageRepo { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// The SHA of the language repo from which this result was generated.
|
||||
/// </summary>
|
||||
public string Commit { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// The location of the assets.json within the language repo from which this result was generated.
|
||||
/// </summary>
|
||||
public string AssetsLocation { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// What tag in the assets repo is this reference pointed at?
|
||||
/// </summary>
|
||||
public string Tag { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Which git repo is being used to store the assets? In other words, the targeted assets repo.
|
||||
/// </summary>
|
||||
public string AssetsRepo { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// The URI of where we backed this item up. Null when we haven't yet backed up this instance.
|
||||
/// </summary>
|
||||
public string? BackupUri { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// Used as a datapoint for debugging and testing of the scanner. Creation datetime of this result.
|
||||
/// </summary>
|
||||
public DateTime CreationDate { get; set; }
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,52 @@
|
|||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Model;
|
||||
|
||||
/// <summary>
|
||||
/// This class abstracts some common query patterns of a set of scanned results.
|
||||
/// </summary>
|
||||
public class AssetsResultSet
|
||||
{
|
||||
|
||||
public AssetsResultSet(List<AssetsResult> input)
|
||||
{
|
||||
Results = input;
|
||||
CalculateObjects();
|
||||
}
|
||||
public List<AssetsResult> Results { get; set; } = new List<AssetsResult>();
|
||||
|
||||
public Dictionary<string, List<AssetsResult>> ByLanguageRepo { get; private set; } = new();
|
||||
|
||||
public Dictionary<string, List<AssetsResult>> ByTargetTag { get; private set; } = new();
|
||||
|
||||
public Dictionary<string, List<AssetsResult>> ByOriginSHA { get; private set; } = new();
|
||||
|
||||
private void CalculateObjects()
|
||||
{
|
||||
ByLanguageRepo = new Dictionary<string, List<AssetsResult>>();
|
||||
ByTargetTag = new Dictionary<string, List<AssetsResult>>();
|
||||
ByOriginSHA = new Dictionary<string, List<AssetsResult>>();
|
||||
|
||||
// sort to ensure that orderings are always the same
|
||||
Results = Results.OrderBy(asset => asset.AssetsRepo).ThenBy(asset => asset.Commit).ThenBy(asset => asset.Tag).ToList();
|
||||
|
||||
foreach (var result in Results)
|
||||
{
|
||||
if (!ByLanguageRepo.ContainsKey(result.LanguageRepo))
|
||||
{
|
||||
ByLanguageRepo.Add(result.LanguageRepo, new List<AssetsResult>());
|
||||
}
|
||||
ByLanguageRepo[result.LanguageRepo].Add(result);
|
||||
|
||||
if (!ByTargetTag.ContainsKey(result.Tag))
|
||||
{
|
||||
ByTargetTag.Add(result.Tag, new List<AssetsResult>());
|
||||
}
|
||||
ByTargetTag[result.Tag].Add(result);
|
||||
|
||||
if (!ByOriginSHA.ContainsKey(result.Commit))
|
||||
{
|
||||
ByOriginSHA.Add(result.Commit, new List<AssetsResult>());
|
||||
}
|
||||
ByOriginSHA[result.Commit].Add(result);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,34 @@
|
|||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Model;
|
||||
|
||||
/// <summary>
|
||||
/// Configuration class describing options available while targeting a repository for scanning.
|
||||
/// </summary>
|
||||
public class RepoConfiguration
|
||||
{
|
||||
public RepoConfiguration(string repo)
|
||||
{
|
||||
LanguageRepo = repo;
|
||||
}
|
||||
|
||||
public RepoConfiguration() {
|
||||
LanguageRepo = string.Empty;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The full orgname/repo-id identifier to access a repo on github. EG: "azure/azure-sdk-for-net"
|
||||
/// </summary>
|
||||
public string LanguageRepo { get; set; }
|
||||
|
||||
/// <summary>
|
||||
/// The time from which we will search for commits that contain assets.jsons. The current default was chosen
|
||||
/// almost arbitrarily. Official test-proxy began supported external assets in late November of 2022, so we don't
|
||||
/// need to go further back then that when examining the SHAs in the language repos. There is no possibility of an
|
||||
/// assets.json past this date!
|
||||
/// </summary>
|
||||
public DateTime ScanStartDate { get; set; } = DateTime.Parse("2022-12-01");
|
||||
|
||||
/// <summary>
|
||||
/// The set of branches that we will examine. Defaults to just 'main'.
|
||||
/// </summary>
|
||||
public List<string> Branches { get; set; } = new List<string> { "main" };
|
||||
}
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
using System.Text.Json;
|
||||
|
||||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Model;
|
||||
|
||||
/// <summary>
|
||||
/// A RunConfiguration is the basic unit of configuration for this maintenance tool. It contains a set of targeted language
|
||||
/// repositories that should be scanned for for assets.jsons.
|
||||
/// </summary>
|
||||
public class RunConfiguration
|
||||
{
|
||||
|
||||
|
||||
public RunConfiguration() {
|
||||
LanguageRepos = new List<RepoConfiguration>();
|
||||
}
|
||||
|
||||
public RunConfiguration(string configPath)
|
||||
{
|
||||
if (File.Exists(configPath))
|
||||
{
|
||||
LanguageRepos = new List<RepoConfiguration>();
|
||||
|
||||
using var stream = System.IO.File.OpenRead(configPath);
|
||||
using var doc = JsonDocument.Parse(stream);
|
||||
|
||||
var results = JsonSerializer.Deserialize<RunConfiguration>(doc);
|
||||
|
||||
if (results != null)
|
||||
{
|
||||
LanguageRepos = results.LanguageRepos;
|
||||
}
|
||||
}
|
||||
else
|
||||
{
|
||||
throw new ArgumentException($"The configuration file path \"{configPath}\" does not exist.");
|
||||
}
|
||||
}
|
||||
|
||||
public List<RepoConfiguration> LanguageRepos { get; set; }
|
||||
}
|
||||
|
|
@ -0,0 +1,39 @@
|
|||
using System.CommandLine;
|
||||
using System.CommandLine.Binding;
|
||||
|
||||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Options;
|
||||
|
||||
public class BaseOptions
|
||||
{
|
||||
public string ConfigLocation { get; set; } = string.Empty;
|
||||
}
|
||||
|
||||
public class BaseOptionsBinder : BinderBase<BaseOptions>
|
||||
{
|
||||
private readonly Option<string> _configLocationOption;
|
||||
|
||||
public BaseOptionsBinder(Option<string> configLocationOption)
|
||||
{
|
||||
_configLocationOption = configLocationOption;
|
||||
}
|
||||
|
||||
protected override BaseOptions GetBoundValue(BindingContext bindingContext)
|
||||
{
|
||||
var result = bindingContext.ParseResult.GetValueForOption(_configLocationOption);
|
||||
|
||||
if (result != null)
|
||||
{
|
||||
return new BaseOptions
|
||||
{
|
||||
ConfigLocation = result
|
||||
};
|
||||
}
|
||||
else
|
||||
{
|
||||
return new BaseOptions
|
||||
{
|
||||
ConfigLocation = string.Empty
|
||||
};
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,86 @@
|
|||
using System.CommandLine;
|
||||
using Azure.Sdk.Tools.Assets.MaintenanceTool.Model;
|
||||
using Azure.Sdk.Tools.Assets.MaintenanceTool.Options;
|
||||
using Azure.Sdk.Tools.Assets.MaintenanceTool.Scan;
|
||||
|
||||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool;
|
||||
|
||||
public class Program
|
||||
{
|
||||
public static string[] StoredArgs = new string[] { };
|
||||
|
||||
// By default, we will only search under the CWD for the previous output. This may change
|
||||
// in the future, but we are keeping it simple to start.
|
||||
|
||||
// all should honor
|
||||
// --scan-data <-- results from previous scans (todo, currently checks working directory)
|
||||
|
||||
// SCAN
|
||||
// --configuration: -> path to file
|
||||
|
||||
// BACKUP
|
||||
// --configuration provided?
|
||||
// SCAN
|
||||
// BACKUP
|
||||
// as each tag is backed up, it is saved with suffix _backup
|
||||
|
||||
// RESTORE
|
||||
// --input-tag <tag that has been stored away>
|
||||
|
||||
// CLEANUP
|
||||
// --configuration provided?
|
||||
// SCAN
|
||||
// BACKUP
|
||||
// CLEANUP
|
||||
// each tag as found by configuration
|
||||
|
||||
// --input-tag <tag on repo>?
|
||||
// SCAN, BACKUP, and CLEANUP individual tag
|
||||
|
||||
public static void Main(string[] args)
|
||||
{
|
||||
StoredArgs = args;
|
||||
|
||||
var rootCommand = InitializeCommandOptions(Run);
|
||||
var resultCode = rootCommand.Invoke(args);
|
||||
Environment.Exit(resultCode);
|
||||
}
|
||||
|
||||
public static void Run(object commandObj)
|
||||
{
|
||||
switch (commandObj)
|
||||
{
|
||||
case BaseOptions configOptions:
|
||||
AssetsScanner scanner = new AssetsScanner();
|
||||
var runConfig = new RunConfiguration(configOptions.ConfigLocation);
|
||||
AssetsResultSet results = scanner.Scan(runConfig);
|
||||
scanner.Save(results);
|
||||
|
||||
break;
|
||||
default:
|
||||
throw new ArgumentException($"Unable to parse the argument set: {string.Join(" ", StoredArgs)}");
|
||||
}
|
||||
}
|
||||
|
||||
public static RootCommand InitializeCommandOptions(Action<BaseOptions> action)
|
||||
{
|
||||
var root = new RootCommand();
|
||||
var configOption = new Option<string>(
|
||||
name: "--config",
|
||||
description: "The path to the json file containing the repo configuration. A sample repo configuration can be seen under <SolutionDirectory>/integration-test-repo-configuration.yml."
|
||||
) {
|
||||
IsRequired = true
|
||||
};
|
||||
configOption.AddAlias("-c");
|
||||
|
||||
var scanCommand = new Command("scan", "Scan the repositories as configured within the file provided to input argument --config <yml config file>.");
|
||||
scanCommand.AddOption(configOption);
|
||||
scanCommand.SetHandler(
|
||||
(configOpts) => action(configOpts),
|
||||
new BaseOptionsBinder(configOption)
|
||||
);
|
||||
root.Add(scanCommand);
|
||||
|
||||
return root;
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,325 @@
|
|||
using System.Text;
|
||||
using System.Text.Json;
|
||||
using Azure.Sdk.Tools.Assets.MaintenanceTool.Model;
|
||||
using Azure.Sdk.Tools.TestProxy.Common.Exceptions;
|
||||
using Azure.Sdk.Tools.TestProxy.Store;
|
||||
using Microsoft.Extensions.FileSystemGlobbing;
|
||||
|
||||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Scan;
|
||||
|
||||
/// <summary>
|
||||
/// Used to walk through repo configurations and locate all assets.
|
||||
/// </summary>
|
||||
public class AssetsScanner
|
||||
{
|
||||
public string WorkingDirectory { get; set; }
|
||||
public static readonly string GitTokenEnvVar = "GIT_TOKEN";
|
||||
|
||||
private string ResultsFile
|
||||
=> Path.Combine(WorkingDirectory, "output.json");
|
||||
|
||||
public GitProcessHandler handler { get; set; } = new GitProcessHandler();
|
||||
|
||||
public AssetsScanner(string? workingDirectory = null)
|
||||
{
|
||||
WorkingDirectory = workingDirectory ?? Directory.GetCurrentDirectory();
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Walk a run configuration and create a resultSet of all found assets.json references.
|
||||
///
|
||||
/// This function automatically takes previous output into account by checking in the current
|
||||
/// working directory for an "output.json" file that contains the output of a previously run Scan.
|
||||
/// </summary>
|
||||
/// <param name="config"></param>
|
||||
/// <returns>A set of results which combines any previous output with a new scan.</returns>
|
||||
public AssetsResultSet Scan(RunConfiguration config)
|
||||
{
|
||||
var resultSet = new List<AssetsResult>();
|
||||
AssetsResultSet? existingResults = ParseExistingResults();
|
||||
|
||||
Parallel.ForEach(config.LanguageRepos, repoConfig =>
|
||||
{
|
||||
resultSet.AddRange(ScanRepo(repoConfig, existingResults));
|
||||
});
|
||||
|
||||
return new AssetsResultSet(resultSet);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// If the tool is invoked in a directory containing an "output.json" file, that file will be parsed
|
||||
/// for its results. The file itself is merely a List of type AssetsResult serialized to disk.
|
||||
/// </summary>
|
||||
/// <returns></returns>
|
||||
public AssetsResultSet? ParseExistingResults()
|
||||
{
|
||||
if (File.Exists(ResultsFile))
|
||||
{
|
||||
using var stream = System.IO.File.OpenRead(ResultsFile);
|
||||
using var doc = JsonDocument.Parse(stream);
|
||||
|
||||
var results = JsonSerializer.Deserialize<List<AssetsResult>>(doc);
|
||||
|
||||
if (results != null)
|
||||
{
|
||||
return new AssetsResultSet(results);
|
||||
}
|
||||
}
|
||||
|
||||
return null;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Given a repo configuration, scan the repo and return an AssetsResult list from all targeted branches.
|
||||
/// </summary>
|
||||
/// <param name="config"></param>
|
||||
/// <param name="previousOutput"></param>
|
||||
/// <returns></returns>
|
||||
private List<AssetsResult> ScanRepo(RepoConfiguration config, AssetsResultSet? previousOutput)
|
||||
{
|
||||
string? envOverride = Environment.GetEnvironmentVariable(GitTokenEnvVar);
|
||||
var authString = string.Empty;
|
||||
if (!string.IsNullOrWhiteSpace(envOverride))
|
||||
{
|
||||
authString = $"{envOverride}@";
|
||||
}
|
||||
|
||||
var targetRepoUri = $"https://{authString}github.com/{config.LanguageRepo}.git";
|
||||
var workingDirectory = Path.Combine(Path.GetTempPath(), Guid.NewGuid().ToString());
|
||||
var results = new List<AssetsResult>();
|
||||
|
||||
try
|
||||
{
|
||||
if (!Directory.Exists(workingDirectory))
|
||||
{
|
||||
Directory.CreateDirectory(workingDirectory);
|
||||
}
|
||||
|
||||
foreach (var branch in config.Branches)
|
||||
{
|
||||
var commitsOnBranch = GetBranchCommits(targetRepoUri, branch, config.ScanStartDate, workingDirectory);
|
||||
var unretrievedCommits = ResolveUnhandledCommits(commitsOnBranch, previousOutput);
|
||||
|
||||
results.AddRange(GetAssetsResults(config.LanguageRepo, unretrievedCommits, workingDirectory));
|
||||
|
||||
if (previousOutput != null)
|
||||
{
|
||||
foreach (var commit in commitsOnBranch.Where(commit => !unretrievedCommits.Contains(commit)))
|
||||
{
|
||||
results.AddRange(previousOutput.ByOriginSHA[commit]);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
finally
|
||||
{
|
||||
CleanupWorkingDirectory(workingDirectory);
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Clones a specific branch, then returns all commit shas newer than our targeted date.
|
||||
/// </summary>
|
||||
/// <returns>A list of commits (limited to after a startdate) from the targeted branch.</returns>
|
||||
private List<string> GetBranchCommits(string uri, string branch, DateTime since, string workingDirectory)
|
||||
{
|
||||
var commitSHAs = new List<string>();
|
||||
try
|
||||
{
|
||||
// if git is already initialized, we just need to checkout a specific branch
|
||||
if (!Directory.Exists(Path.Combine(workingDirectory, ".git")))
|
||||
{
|
||||
handler.Run($"clone {uri} --branch {branch} --single-branch .", workingDirectory);
|
||||
}
|
||||
else
|
||||
{
|
||||
handler.Run($"fetch origin {branch}", workingDirectory);
|
||||
handler.Run($"branch {branch} FETCH_HEAD", workingDirectory);
|
||||
handler.Run($"checkout {branch}", workingDirectory);
|
||||
Cleanup(workingDirectory);
|
||||
}
|
||||
|
||||
var tagResult = handler.Run($"log --since={since.ToString("yyyy-MM-dd")} --format=format:%H", workingDirectory);
|
||||
commitSHAs.AddRange(tagResult.StdOut.Split(Environment.NewLine).Select(x => x.Trim()).Where(x => !string.IsNullOrWhiteSpace(x)));
|
||||
}
|
||||
catch (GitProcessException gitException)
|
||||
{
|
||||
// special case handling here?
|
||||
Console.WriteLine(gitException.ToString());
|
||||
Environment.Exit(1);
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
Console.WriteLine(e.ToString());
|
||||
Environment.Exit(1);
|
||||
}
|
||||
|
||||
return commitSHAs;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// We only need to process each commit _once_, as commit SHAs are immutable in git. Given that, once we have
|
||||
/// a list of commits from a targeted branch, we need to check against the previous results to ensure we don't
|
||||
/// reprocess those and emit duplicate assetsResults.
|
||||
/// </summary>
|
||||
/// <returns>The set of unprocessed commit SHAs.</returns>
|
||||
private List<string> ResolveUnhandledCommits(List<string> commits, AssetsResultSet? previousResults)
|
||||
{
|
||||
if (previousResults == null)
|
||||
{
|
||||
return commits;
|
||||
}
|
||||
else
|
||||
{
|
||||
return commits.Where(x => !previousResults.ByOriginSHA.ContainsKey(x)).ToList();
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Used to easily parse an assets.json and grab only the properties that this tool cares about.
|
||||
/// </summary>
|
||||
private class Assets
|
||||
{
|
||||
public Assets()
|
||||
{
|
||||
AssetsRepo = string.Empty;
|
||||
Tag = string.Empty;
|
||||
}
|
||||
|
||||
public string AssetsRepo { get; set; }
|
||||
|
||||
public string Tag { get; set; }
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Deserialize an assets.json from disk into a class instance to retrieve the targeted Tag and Assets Repository.
|
||||
/// </summary>
|
||||
/// <param name="assetsJson"></param>
|
||||
/// <returns>A class instance containing the assets.json details.</returns>
|
||||
private Assets? ExtractAssetsData(string assetsJson)
|
||||
{
|
||||
return JsonSerializer.Deserialize<Assets>(File.ReadAllText(assetsJson));
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Find all assets.jsons beneath a targeted folder.
|
||||
/// </summary>
|
||||
/// <returns>AssetsResults for each discovered assets.json, populating other metadata as necessary.</returns>
|
||||
private List<AssetsResult> ScanDirectory(string repo, string commit, string workingDirectory)
|
||||
{
|
||||
Matcher matcher = new();
|
||||
List<AssetsResult> locatedAssets = new List<AssetsResult>();
|
||||
matcher.AddIncludePatterns(new[] { "**/assets.json" });
|
||||
IEnumerable<string> assetsJsons = matcher.GetResultsInFullPath(workingDirectory);
|
||||
|
||||
foreach (var assetsJson in assetsJsons)
|
||||
{
|
||||
var path = Path.GetRelativePath(workingDirectory, assetsJson).Replace("\\", "/");
|
||||
var assetsData = ExtractAssetsData(assetsJson);
|
||||
|
||||
if (assetsData != null)
|
||||
{
|
||||
var newResult = new AssetsResult(repo, commit, path, assetsData.Tag, assetsData.AssetsRepo, null);
|
||||
locatedAssets.Add(newResult);
|
||||
}
|
||||
}
|
||||
|
||||
return locatedAssets;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Walks a set of targeted commits, extracting all available assets.jsons from each.
|
||||
/// </summary>
|
||||
/// <returns>A list of AssetsResults reflecting all discovered assets.jsons from each targeted commit.</returns>
|
||||
private List<AssetsResult> GetAssetsResults(string repo, List<string> commits, string workingDirectory)
|
||||
{
|
||||
var allResults = new List<AssetsResult>();
|
||||
foreach (var commit in commits)
|
||||
{
|
||||
handler.Run($"checkout {commit}", workingDirectory);
|
||||
Cleanup(workingDirectory);
|
||||
allResults.AddRange(ScanDirectory(repo, commit, workingDirectory));
|
||||
}
|
||||
|
||||
return allResults;
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Cleans up a git repo. When swapping between commits, we don't want to accidentally include assets.jsons that are
|
||||
/// present simply because a folder didn't auto delete itself when we switched commits.
|
||||
/// </summary>
|
||||
private void Cleanup(string workingDirectory)
|
||||
{
|
||||
try
|
||||
{
|
||||
handler.Run("clean -xdf", workingDirectory);
|
||||
}
|
||||
catch (GitProcessException gitException)
|
||||
{
|
||||
Console.WriteLine(gitException.ToString());
|
||||
Environment.Exit(1);
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
Console.WriteLine(e.ToString());
|
||||
Environment.Exit(1);
|
||||
}
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Intended to be aimed at a specific .git folder. Walks every file and ensures that
|
||||
/// any wonky permissions that could prevent deletion are removed.
|
||||
///
|
||||
/// This is necessary because certain `.pack` files created by git cannot be deleted without
|
||||
/// adjusting these permissions.
|
||||
/// </summary>
|
||||
private void SetPermissionsAndDelete(string gitfolder)
|
||||
{
|
||||
File.SetAttributes(gitfolder, FileAttributes.Normal);
|
||||
|
||||
string[] files = Directory.GetFiles(gitfolder);
|
||||
string[] dirs = Directory.GetDirectories(gitfolder);
|
||||
|
||||
foreach (string file in files)
|
||||
{
|
||||
File.SetAttributes(file, FileAttributes.Normal);
|
||||
File.Delete(file);
|
||||
}
|
||||
|
||||
foreach (string dir in dirs)
|
||||
{
|
||||
SetPermissionsAndDelete(dir);
|
||||
}
|
||||
|
||||
Directory.Delete(gitfolder, false);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// The .git folder's .pack files can be super finicky to delete from code.
|
||||
/// This function abstracts the necessary permissions update and cleans that folder for us.
|
||||
/// </summary>
|
||||
private void CleanupWorkingDirectory(string workingDirectory)
|
||||
{
|
||||
var gitDir = Path.Combine(workingDirectory, ".git");
|
||||
|
||||
if (Directory.Exists(gitDir))
|
||||
{
|
||||
SetPermissionsAndDelete(gitDir);
|
||||
}
|
||||
|
||||
Directory.Delete(workingDirectory, true);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// Writes a resultSet to disk.
|
||||
/// </summary>
|
||||
public void Save(AssetsResultSet newResults)
|
||||
{
|
||||
using (var stream = System.IO.File.OpenWrite(ResultsFile))
|
||||
{
|
||||
stream.Write(Encoding.UTF8.GetBytes(JsonSerializer.Serialize(newResults.Results)));
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,9 @@
|
|||
namespace Azure.Sdk.Tools.Assets.MaintenanceTool.Store;
|
||||
|
||||
/// <summary>
|
||||
/// Used to write our backup entries.
|
||||
/// </summary>
|
||||
public class AssetsStorageClient
|
||||
{
|
||||
// placeholder for now.
|
||||
}
|
||||
|
|
@ -0,0 +1,37 @@
|
|||
|
||||
Microsoft Visual Studio Solution File, Format Version 12.00
|
||||
# Visual Studio Version 17
|
||||
VisualStudioVersion = 17.5.33530.505
|
||||
MinimumVisualStudioVersion = 10.0.40219.1
|
||||
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Azure.Sdk.Tools.Assets.MaintenanceTool", "Azure.Sdk.Tools.Assets.MaintenanceTool\Azure.Sdk.Tools.Assets.MaintenanceTool.csproj", "{8ECBAB4E-1209-454D-B0AA-F93A61C7C1B9}"
|
||||
EndProject
|
||||
Project("{9A19103F-16F7-4668-BE54-9A1E7A4F7556}") = "Azure.Sdk.Tools.TestProxy", "..\..\test-proxy\Azure.Sdk.Tools.TestProxy\Azure.Sdk.Tools.TestProxy.csproj", "{531B1A4C-B213-4486-B965-330585816BAA}"
|
||||
EndProject
|
||||
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Azure.Sdk.Tools.Assets.MaintenanceTool.Tests", "Azure.Sdk.Tools.Assets.MaintenanceTool.Tests\Azure.Sdk.Tools.Assets.MaintenanceTool.Tests.csproj", "{968EC25C-861D-4C10-967D-A3E0137D35C2}"
|
||||
EndProject
|
||||
Global
|
||||
GlobalSection(SolutionConfigurationPlatforms) = preSolution
|
||||
Debug|Any CPU = Debug|Any CPU
|
||||
Release|Any CPU = Release|Any CPU
|
||||
EndGlobalSection
|
||||
GlobalSection(ProjectConfigurationPlatforms) = postSolution
|
||||
{8ECBAB4E-1209-454D-B0AA-F93A61C7C1B9}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
|
||||
{8ECBAB4E-1209-454D-B0AA-F93A61C7C1B9}.Debug|Any CPU.Build.0 = Debug|Any CPU
|
||||
{8ECBAB4E-1209-454D-B0AA-F93A61C7C1B9}.Release|Any CPU.ActiveCfg = Release|Any CPU
|
||||
{8ECBAB4E-1209-454D-B0AA-F93A61C7C1B9}.Release|Any CPU.Build.0 = Release|Any CPU
|
||||
{531B1A4C-B213-4486-B965-330585816BAA}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
|
||||
{531B1A4C-B213-4486-B965-330585816BAA}.Debug|Any CPU.Build.0 = Debug|Any CPU
|
||||
{531B1A4C-B213-4486-B965-330585816BAA}.Release|Any CPU.ActiveCfg = Release|Any CPU
|
||||
{531B1A4C-B213-4486-B965-330585816BAA}.Release|Any CPU.Build.0 = Release|Any CPU
|
||||
{968EC25C-861D-4C10-967D-A3E0137D35C2}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
|
||||
{968EC25C-861D-4C10-967D-A3E0137D35C2}.Debug|Any CPU.Build.0 = Debug|Any CPU
|
||||
{968EC25C-861D-4C10-967D-A3E0137D35C2}.Release|Any CPU.ActiveCfg = Release|Any CPU
|
||||
{968EC25C-861D-4C10-967D-A3E0137D35C2}.Release|Any CPU.Build.0 = Release|Any CPU
|
||||
EndGlobalSection
|
||||
GlobalSection(SolutionProperties) = preSolution
|
||||
HideSolutionNode = FALSE
|
||||
EndGlobalSection
|
||||
GlobalSection(ExtensibilityGlobals) = postSolution
|
||||
SolutionGuid = {A9C5822D-B16B-4043-A8FD-E7F8B5536855}
|
||||
EndGlobalSection
|
||||
EndGlobal
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
# Assets Maintenance Tooling
|
||||
|
||||
The tool contained within this directory is intended two fulfill three main tasks.
|
||||
|
||||
1. Scan the commits of a set of targeted repos and branches to create a representation of all assets.jsons, their targeted tags, and their origin SHAs.
|
||||
* This is just a map of the complete footprint of `test-proxy` assets that are referenced in azure-sdk repositories.
|
||||
2. Use the map of data created in step 1 to individually backup tags from the assets repository to a target that we can retrieve later.
|
||||
3. Use the map of data created in step 1 to clean up _unnecessary_ tags in the assets repository.
|
||||
|
||||
## Usage
|
||||
|
||||
### Installation
|
||||
|
||||
`<todo>`
|
||||
|
||||
### Scanning
|
||||
|
||||
`<todo>`
|
||||
|
||||
## What does the process look like?
|
||||
|
||||
|
||||
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 131 KiB |
|
|
@ -8,7 +8,7 @@ trigger:
|
|||
- hotfix/*
|
||||
paths:
|
||||
include:
|
||||
- tools/asset-sync
|
||||
- tools/assets-automation
|
||||
|
||||
pr:
|
||||
branches:
|
||||
|
|
@ -19,10 +19,9 @@ pr:
|
|||
- hotfix/*
|
||||
paths:
|
||||
include:
|
||||
- tools/asset-sync
|
||||
- tools/assets-automation
|
||||
|
||||
extends:
|
||||
template: /eng/common/pipelines/templates/stages/archetype-sdk-tool-pwsh.yml
|
||||
template: /eng/pipelines/templates/stages/archetype-sdk-tool-dotnet.yml
|
||||
parameters:
|
||||
TargetDirectory: tools/asset-sync/
|
||||
TargetTags: 'Unit'
|
||||
ToolDirectory: tools/assets-automation/assets-maintenance-tool/
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
trigger: none
|
||||
|
||||
variables:
|
||||
- template: /eng/pipelines/templates/variables/globals.yml
|
||||
|
||||
stages:
|
||||
- stage: IntegrationTests
|
||||
displayName: "Integration Tests"
|
||||
jobs:
|
||||
- job: Solution_Integration_Test
|
||||
displayName: Run Solution Tests
|
||||
|
||||
strategy:
|
||||
matrix:
|
||||
Windows:
|
||||
Pool: 'azsdk-pool-mms-win-2022-general'
|
||||
OS: 'Windows'
|
||||
Linux:
|
||||
Pool: azsdk-pool-mms-ubuntu-2204-general
|
||||
OS: 'Linux'
|
||||
|
||||
pool:
|
||||
name: $(Pool)
|
||||
|
||||
steps:
|
||||
- template: /eng/pipelines/templates/steps/install-dotnet.yml
|
||||
|
||||
- script: 'dotnet test /p:ArtifactsPackagesDir=$(Build.ArtifactStagingDirectory) --logger trx $(Build.SourcesDirectory)/tools/assets-automation/assets-maintenance-tool/'
|
||||
displayName: 'Test'
|
||||
env:
|
||||
DOTNET_SKIP_FIRST_TIME_EXPERIENCE: 1
|
||||
DOTNET_CLI_TELEMETRY_OPTOUT: 1
|
||||
DOTNET_MULTILEVEL_LOOKUP: 0
|
||||
GIT_TOKEN: $(azuresdk-github-pat)
|
||||
GIT_COMMIT_OWNER: azure-sdk
|
||||
GIT_COMMIT_EMAIL: azuresdk@microsoft.com
|
||||
|
||||
- task: PublishTestResults@2
|
||||
condition: succeededOrFailed()
|
||||
inputs:
|
||||
testResultsFiles: '**/*.trx'
|
||||
testRunTitle: '$(OS) Maintenance tool tests against .NET'
|
||||
testResultsFormat: 'VSTest'
|
||||
mergeTestResults: true
|
||||
Загрузка…
Ссылка в новой задаче