Add developer docs and initial dev data endpoint (#201)
This change focuses on enabling a better developer environment. The changes here include: - Add IPlateTileDownloader that takes a plate name and how many levels to retrieve. It will then create a new plate file for use with those levels. - Expose an endpoint `/v2/data/dev_export` to obtain data a developer can work with. This is initially just the first few levels of the `dss.aspx` dataset. The endpoint only allows up to level 4. The result of this is zipped up and cached for future requests. This is currently about 4mb. - Add docs on how to set up an environment - Add docs on how to test with the webclient - Update the cli tool to download plate images up to a certain level
This commit is contained in:
Taylor Southwick
GitHub
Родитель
95b6ac53a9
Коммит
631c826fbc
46
README.md
46
README.md
|
|
@ -21,51 +21,9 @@ As we move the service to a more modern architecture, some endpoints are being d
|
|||
| `/wwtweb/wmsmoon.aspx` | Data is from `onmoon.jpl.nasa.gov` which no longer available. | [#181](https://github.com/WorldWideTelescope/wwt-website/pull/181) |
|
||||
| `/wwtweb/wmstoast.aspx` | Data is from `ms.mars.asu.edu` which no longer available. | [#181](https://github.com/WorldWideTelescope/wwt-website/pull/181) |
|
||||
|
||||
## Installation Instructions
|
||||
|
||||
**To be revised**. Some *old* notes are in [INSTALL.md](./INSTALL.md).
|
||||
|
||||
### Azure Storage Emulator
|
||||
|
||||
This project is configured to use [Azurite](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite), a cross-platform emulator for Azure storage. There are multiple ways to acquire the tool, so please refer to the link given to set up an install.
|
||||
|
||||
Access to the storage is done via the `DefaultAzureTokenCredential` that requires `https` protocol for connection. In order to do that, the following steps must be done to enable development:
|
||||
|
||||
These are steps taken from [here](https://blog.jongallant.com/2020/04/local-azure-storage-development-with-azurite-azuresdks-storage-explorer/):
|
||||
|
||||
1. Install [`mkcert`](https://github.com/FiloSottile/mkcert#installation)
|
||||
1. Trust the mkcert RootCA.pem and create a certificate
|
||||
|
||||
```
|
||||
mkcert -install
|
||||
mkcert 127.0.0.1
|
||||
```
|
||||
1. Chose a directory from which to run Azurite. The local emulator data will be stored here
|
||||
1. Run Azurite with oauth and SSL support:
|
||||
|
||||
```
|
||||
azurite --oauth basic --cert 127.0.0.1.pem --key 127.0.0.1-key.pem
|
||||
```
|
||||
1. The app will now run with default settings.
|
||||
|
||||
In order to configure the [Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/) to run, you'll need to do the following:
|
||||
|
||||
1. Get the RootCA.pem
|
||||
|
||||
```
|
||||
mkcert -CAROOT
|
||||
```
|
||||
1. Open Azure Storage Explorer
|
||||
1. Go to `Edit->SSL Certificates->Import Certificates` and select the file from the first step.
|
||||
1. Restart the storage explorer (you will be prompted to do this)
|
||||
|
||||
### Configuration
|
||||
Configuration in this project uses ConfigurationManager.AppSettings. In order to make
|
||||
it easier to configure outside of web.config, ConfigurationBuilders are supported.
|
||||
Currently, there are three builders enabled: KeyVault, Environment, and User Secrets.
|
||||
For more details, see the [project](https://github.com/aspnet/MicrosoftConfigurationBuilders#config-builders-in-this-project)
|
||||
where they are maintained.
|
||||
## Developer Instructions
|
||||
|
||||
This project is undergoing rapid changes and some of these will be playing with moving pieces. Please submit issues (or PRs!) if you find anything out of date. See [docs/dev-environment.md](instructions) for details on how to build and run locally.
|
||||
|
||||
## Getting involved
|
||||
|
||||
|
|
|
|||
|
|
@ -0,0 +1,83 @@
|
|||
# Dev Environments
|
||||
|
||||
To see a list of technologies that this project uses, see [here](technologies.md).
|
||||
|
||||
This project is heavily data-driven with massive backend datastores of astronomical data. It is not reasonable to require full datasets to work on the service as it can take days to copy the full sets. Thus, we will outline some ways to test the service locally.
|
||||
|
||||
## Required tools
|
||||
|
||||
- [Visual Studio 16.8](https://visualstudio.microsoft.com/) with the ASP.NET and the .NET Core workloads installed. Community edition will work just fine.
|
||||
- [Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/)
|
||||
|
||||
There is work towards moving this to .NET 5. Once that lands, the service will not be tied to Windows or Visual Studio.
|
||||
|
||||
The solution file is `wwt-website.sln` and the start up project should be set to `WWTMVC5`.
|
||||
|
||||
## Set up development datasets
|
||||
|
||||
### Download datasets
|
||||
A subset of data can be accessed by calling `GET http://worldwidetelescope.org/v2/data/dev_export`. This is a zip file that will contain various datapoints and will be expanded as more deve data comes online. Each of the folders align with the expected Azure container it will be put in. Once the dataset is downloaded, the data should be copied over to either Azure or an Azure emulator such as Azurite.
|
||||
|
||||
### Azure Storage Emulator
|
||||
This project is configured to use [Azurite](https://docs.microsoft.com/en-us/azure/storage/common/storage-use-azurite), a cross-platform emulator for Azure storage. There are multiple ways to acquire the tool, so please refer to the link given to set up an install.
|
||||
|
||||
Access to the storage is done via the `DefaultAzureTokenCredential` that requires `https` protocol for connection. In order to do that, the following steps must be done to enable development:
|
||||
|
||||
These are steps taken from [here](https://blog.jongallant.com/2020/04/local-azure-storage-development-with-azurite-azuresdks-storage-explorer/):
|
||||
|
||||
1. Install [`mkcert`](https://github.com/FiloSottile/mkcert#installation)
|
||||
1. Trust the mkcert RootCA.pem and create a certificate
|
||||
|
||||
```
|
||||
mkcert -install
|
||||
mkcert 127.0.0.1
|
||||
```
|
||||
1. Chose a directory from which to run Azurite. The local emulator data will be stored here
|
||||
1. Run Azurite with oauth and SSL support:
|
||||
|
||||
```
|
||||
azurite --oauth basic --cert 127.0.0.1.pem --key 127.0.0.1-key.pem
|
||||
```
|
||||
1. The app will now run with default settings.
|
||||
|
||||
In order to configure the [Azure Storage Explorer](https://azure.microsoft.com/en-us/features/storage-explorer/) to run, you'll need to do the following:
|
||||
|
||||
1. Get the RootCA.pem
|
||||
|
||||
```
|
||||
mkcert -CAROOT
|
||||
```
|
||||
1. Open Azure Storage Explorer
|
||||
1. Go to `Edit->SSL Certificates->Import Certificates` and select the file from the first step.
|
||||
1. Restart the storage explorer (you will be prompted to do this)
|
||||
|
||||
### Hit an endpoint with the developer dataset
|
||||
In order to verify that the dataset is set up correctly, you should be able to `GET /wwtweb/dss.aspx?Q=0,0,0` to verify that it will be served up. Be careful that the service does not verify that the level you request is actually available - it expects specific levels to be there. Use this as a simple verification step or to run specific manual steps.
|
||||
|
||||
## Run service against live webclient
|
||||
Besides hitting endpoints individually, the webclient can be used to test the service. There is no endpoint that says where the client starts, so the easiest way to do this is to intercept the requests. The default dataset is `dss.aspx`, the data for which should be obtained from the developer dataset.
|
||||
|
||||
In order to set this up, we need a way to rewrite requests so that they will be serviced elsewhere. For purposes of this example, we will show it using [requestly.io](https://requestly.io/) which can be added as an extension to enable this. For now, only the `dss.aspx` will be available for testing, so it will be highlighted here. In the future, other datasets may be available for testing as well. Please file any issues if there are specific datasets you need to be able to test.
|
||||
|
||||
1. Set up Requestly rules
|
||||
Requestly.io uses rules to rewrite requests from websites. The call we want to intercept is to the `dss.aspx` endpoint, which we can do by setting up rules similar to:
|
||||
|
||||
|
||||
1. Open browser and its developer tools
|
||||
1. Load webclient by going to http://worldwidetelescope.org
|
||||
1. Inspect the network call results to see the calls to the local service such as below:
|
||||
|
||||
|
||||
## Next steps for developers
|
||||
Here are some of the potential next steps to investigate to simplify the development process:
|
||||
|
||||
1. Update clients to be data driven from an endpoint on the service
|
||||
Currently, the clients have a bunch of hardcoded expectations around what datasets are available and what it should be doing. By exposing an endpoint that provides an initial WTML or some other file of what the initial screen looks like, the client can react more easily to changes in available datasets and don't need to be updated. The service becomes the source of truth for this.
|
||||
1. Consolidate the platefile related endpoints to be a single endpoint that takes a desired plate
|
||||
Currently, there are a number of endpoints that all do roughly the same thing: take coordinates and return images from a plate file. This is difficult to maintain and difficult to extend. This could potentially be combined to use a single endpoint, ie `/api/plate/{name}/{level}/{x},{y}` that could be completely data driven. Some thoughts here:
|
||||
- Can use rewriting tools to maintain current endpoints and have them end up there.
|
||||
- After the move to .NET 5, this can be much more easily done via controllers which will handle parameter binding
|
||||
- The available plates could be a dynamic list generated by searching the available files in the container
|
||||
1. Levels can be calculated given a plate file, so a more data driven approach could calculate the available levels and dynamically adjust from the small develop plate files to the large plate files available in production.
|
||||
1. The access to Azure could be abstracted to use the `IFileProvider` implementations that can compose from multiple locations. That way, a local dev could very easily use a file system, while the deployed system would use an Azure Blob backed file provider.
|
||||
1. If the app were deployed to a container, dev data could be used for client development also without needing access to the internet or the public service.
|
||||
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 23 KiB |
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 114 KiB |
|
|
@ -0,0 +1,10 @@
|
|||
# Technologies
|
||||
|
||||
There are a number of technologies being used within the service. This document will try to keep the high level tools/SDKs so that people can easily ramp on the project.
|
||||
|
||||
## App Configuration
|
||||
Configuration in this project uses ConfigurationManager.AppSettings. In order to make
|
||||
it easier to configure outside of web.config, ConfigurationBuilders are supported.
|
||||
Currently, there are three builders enabled: KeyVault, Environment, and User Secrets.
|
||||
For more details, see the [project](https://github.com/aspnet/MicrosoftConfigurationBuilders#config-builders-in-this-project)
|
||||
where they are maintained.
|
||||
|
|
@ -0,0 +1,51 @@
|
|||
using Microsoft.Extensions.Logging;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using WWT.PlateFiles;
|
||||
using WWTWebservices;
|
||||
|
||||
namespace WWT.Azure
|
||||
{
|
||||
public class AzurePlateFileDownloader : IPlateTileDownloader
|
||||
{
|
||||
private readonly IPlateTilePyramid _plateTiles;
|
||||
private readonly ILogger<AzurePlateFileDownloader> _logger;
|
||||
|
||||
public AzurePlateFileDownloader(IPlateTilePyramid plateTiles, ILogger<AzurePlateFileDownloader> logger)
|
||||
{
|
||||
_plateTiles = plateTiles;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public async Task<int> DownloadPlateFileAsync(string name, Stream stream, int maxLevel, CancellationToken token)
|
||||
{
|
||||
using var file = new PlateTilePyramid(stream, maxLevel);
|
||||
|
||||
for (int level = 0; level < maxLevel; level++)
|
||||
{
|
||||
var size = Math.Pow(2, level);
|
||||
|
||||
_logger.LogInformation("Adding level {Level} to {Plate} ({ImageCount} images)", level, name, size * size);
|
||||
|
||||
for (int x = 0; x < size; x++)
|
||||
{
|
||||
for (int y = 0; y < size; y++)
|
||||
{
|
||||
using var tile = await _plateTiles.GetStreamAsync(string.Empty, name, level, x, y, token);
|
||||
|
||||
await file.AddStreamAsync(tile, level, x, y, token);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
file.UpdateHeaderAndClose();
|
||||
|
||||
return file.Count;
|
||||
}
|
||||
|
||||
public IAsyncEnumerable<string> GetPlateNames(CancellationToken token) => _plateTiles.GetPlateNames(token);
|
||||
}
|
||||
}
|
||||
|
|
@ -9,7 +9,7 @@ namespace WWT.Azure
|
|||
public AzureServiceAccessor(AzureOptions options, TokenCredential credential)
|
||||
{
|
||||
WwtFiles = CreateServiceClient(options.StorageAccount, credential);
|
||||
Mars = CreateServiceClient(options.MarsStorageAccount, credential);
|
||||
Mars = CreateServiceClient(options.MarsStorageAccount, credential) ?? WwtFiles;
|
||||
}
|
||||
|
||||
public BlobServiceClient WwtFiles { get; set; }
|
||||
|
|
@ -22,6 +22,11 @@ namespace WWT.Azure
|
|||
/// </summary>
|
||||
private static BlobServiceClient CreateServiceClient(string storageAccount, TokenCredential credential)
|
||||
{
|
||||
if (storageAccount is null)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
if (Uri.TryCreate(storageAccount, UriKind.Absolute, out var storageUri))
|
||||
{
|
||||
return new BlobServiceClient(storageUri, credential);
|
||||
|
|
|
|||
|
|
@ -57,6 +57,7 @@ namespace WWT.Azure
|
|||
services.Services.AddSingleton(options);
|
||||
services.Services.AddSingleton<IPlateTilePyramid, MarsMolaAwareSeekableAzurePlateTilePyramid>();
|
||||
services.Services.AddSingleton<IKnownPlateFiles, AzureKnownPlateFile>();
|
||||
services.Services.AddSingleton<IPlateTileDownloader, AzurePlateFileDownloader>();
|
||||
|
||||
return services;
|
||||
}
|
||||
|
|
|
|||
|
|
@ -10,6 +10,7 @@
|
|||
<PackageReference Include="Azure.Storage.Blobs" Version="12.7.0" />
|
||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="3.1.9" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="3.1.9" />
|
||||
<PackageReference Include="System.Linq.Async" Version="4.1.1" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
|
|
|||
|
|
@ -3,37 +3,39 @@ using System.IO;
|
|||
using System.Runtime.CompilerServices;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using WWTWebservices;
|
||||
|
||||
namespace WWTWebservices
|
||||
namespace WWT
|
||||
{
|
||||
public class FilePlateTilePyramid : IPlateTilePyramid
|
||||
{
|
||||
private readonly string _directory;
|
||||
|
||||
public FilePlateTilePyramid(string directory)
|
||||
{
|
||||
_directory = directory;
|
||||
}
|
||||
|
||||
public async IAsyncEnumerable<string> GetPlateNames([EnumeratorCancellation] CancellationToken token)
|
||||
{
|
||||
await Task.Yield();
|
||||
yield break;
|
||||
|
||||
foreach (var file in Directory.GetFiles(_directory, "*.plate"))
|
||||
{
|
||||
yield return Path.GetFileName(file);
|
||||
}
|
||||
}
|
||||
|
||||
public Task<Stream> GetStreamAsync(string pathPrefix, string plateName, int level, int x, int y, CancellationToken token)
|
||||
{
|
||||
if (string.IsNullOrEmpty(pathPrefix))
|
||||
{
|
||||
throw new System.ArgumentException($"'{nameof(pathPrefix)}' cannot be null or empty", nameof(pathPrefix));
|
||||
}
|
||||
|
||||
var result = PlateTilePyramid.GetFileStream(Path.Combine(pathPrefix, plateName), level, x, y);
|
||||
var result = PlateTilePyramid.GetFileStream(Path.Combine(_directory, plateName), level, x, y);
|
||||
|
||||
return Task.FromResult(result);
|
||||
}
|
||||
|
||||
public Task<Stream> GetStreamAsync(string pathPrefix, string plateName, int tag, int level, int x, int y, CancellationToken token)
|
||||
{
|
||||
if (string.IsNullOrEmpty(pathPrefix))
|
||||
{
|
||||
throw new System.ArgumentException($"'{nameof(pathPrefix)}' cannot be null or empty", nameof(pathPrefix));
|
||||
}
|
||||
|
||||
var plateFile2 = new PlateFile2(Path.Combine(pathPrefix, plateName));
|
||||
var plateFile2 = new PlateFile2(Path.Combine(_directory, plateName));
|
||||
var result = plateFile2.GetFileStream(tag, level, x, y);
|
||||
|
||||
return Task.FromResult(result);
|
||||
|
|
|
|||
|
|
@ -0,0 +1,14 @@
|
|||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace WWT.PlateFiles
|
||||
{
|
||||
public interface IPlateTileDownloader
|
||||
{
|
||||
IAsyncEnumerable<string> GetPlateNames(CancellationToken token);
|
||||
|
||||
Task<int> DownloadPlateFileAsync(string name, Stream stream, int maxLevel, CancellationToken token);
|
||||
}
|
||||
}
|
||||
|
|
@ -2,6 +2,7 @@ using System;
|
|||
using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using WWT;
|
||||
using WWT.PlateFiles;
|
||||
|
||||
/* Jonathan Fay wrote this, except for the bits polluted by Dinoj, which are between <dinoj>...</dinoj> tags
|
||||
|
|
@ -25,20 +26,10 @@ namespace WWTWebservices
|
|||
{
|
||||
public class PlateTilePyramid : IDisposable
|
||||
{
|
||||
string filename;
|
||||
private FileStream _readStream;
|
||||
private bool _disposed = false;
|
||||
|
||||
public string FileName
|
||||
{
|
||||
get { return filename; }
|
||||
}
|
||||
int levels;
|
||||
public int Levels { get; }
|
||||
|
||||
public int Levels
|
||||
{
|
||||
get { return levels; }
|
||||
}
|
||||
uint currentOffset = 0;
|
||||
LevelInfo[] levelMap;
|
||||
|
||||
|
|
@ -48,84 +39,50 @@ namespace WWTWebservices
|
|||
/// magic number is ceil(0.9876 * 2^31) = 0111 1110 0110 1001 1010 1101 0100 0011 in binary
|
||||
/// this identifies that this plate file has useful header information
|
||||
|
||||
public PlateTilePyramid(string filename)
|
||||
public PlateTilePyramid(Stream stream)
|
||||
{
|
||||
int L = -1;
|
||||
if (GetLevelCount(filename, out L))
|
||||
if (GetLevelCount(stream, out var L))
|
||||
{
|
||||
this.filename = filename;
|
||||
this.levels = L;
|
||||
fileStream = stream;
|
||||
Levels = L;
|
||||
}
|
||||
else
|
||||
{
|
||||
this.filename = ""; // UNCLEAR WHAT TO SET THIS TO
|
||||
this.levels = -1; // UNCLEAR WHAT TO SET THIS TO
|
||||
throw new InvalidOperationException();
|
||||
}
|
||||
}
|
||||
// </dinoj>
|
||||
|
||||
public PlateTilePyramid(string filename, int levels)
|
||||
Stream fileStream = null;
|
||||
|
||||
public PlateTilePyramid(Stream stream, int L)
|
||||
{
|
||||
this.filename = filename;
|
||||
this.levels = levels;
|
||||
}
|
||||
fileStream = stream;
|
||||
Levels = L;
|
||||
|
||||
FileStream fileStream = null;
|
||||
levelMap = new LevelInfo[Levels];
|
||||
|
||||
public void Create()
|
||||
{
|
||||
levelMap = new LevelInfo[levels];
|
||||
|
||||
for (int i = 0; i < levels; i++)
|
||||
for (int i = 0; i < Levels; i++)
|
||||
{
|
||||
levelMap[i] = new LevelInfo(i);
|
||||
}
|
||||
fileStream = File.Open(filename, FileMode.Create, FileAccess.ReadWrite, FileShare.None);
|
||||
|
||||
WriteHeaders();
|
||||
currentOffset = HeaderSize;
|
||||
fileStream.Seek(currentOffset, SeekOrigin.Begin);
|
||||
|
||||
}
|
||||
|
||||
// would be nice to have a version of AddFile that has as its first argument a Bitmap or Stream
|
||||
public void AddFile(string inputFilename, int level, int x, int y)
|
||||
public int Count { get; private set; }
|
||||
|
||||
public async Task AddStreamAsync(Stream inputStream, int level, int x, int y, CancellationToken token)
|
||||
{
|
||||
Count++;
|
||||
|
||||
long start = fileStream.Seek(0, SeekOrigin.End);
|
||||
byte[] buf = null;
|
||||
|
||||
using (FileStream fs = File.Open(inputFilename, FileMode.Open, FileAccess.Read, FileShare.Read))
|
||||
{
|
||||
int len = (int)fs.Length;
|
||||
buf = new byte[fs.Length];
|
||||
|
||||
levelMap[level].fileMap[x, y].start = (uint)start;
|
||||
levelMap[level].fileMap[x, y].size = (uint)len;
|
||||
|
||||
fs.Read(buf, 0, len);
|
||||
fileStream.Write(buf, 0, len);
|
||||
fs.Close();
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
public void AddStream(Stream inputStream, int level, int x, int y)
|
||||
{
|
||||
// closes inputStream once read
|
||||
// added by Dinoj, debugged by Jonathan
|
||||
// TODO: add a while loop so that it can handle very long streams
|
||||
// (for our 256 x 256 bitmaps it's fine)
|
||||
long start = fileStream.Seek(0, SeekOrigin.End);
|
||||
byte[] buf = null;
|
||||
|
||||
int len = (int)inputStream.Length;
|
||||
buf = new byte[inputStream.Length];
|
||||
await inputStream.CopyToAsync(fileStream, token).ConfigureAwait(false);
|
||||
|
||||
levelMap[level].fileMap[x, y].start = (uint)start;
|
||||
levelMap[level].fileMap[x, y].size = (uint)len;
|
||||
inputStream.Seek(0, SeekOrigin.Begin);
|
||||
int lenRead = inputStream.Read(buf, 0, len);
|
||||
fileStream.Write(buf, 0, len);
|
||||
inputStream.Close();
|
||||
levelMap[level].fileMap[x, y].size = (uint)inputStream.Length;
|
||||
}
|
||||
|
||||
public void UpdateHeaderAndClose()
|
||||
|
|
@ -137,46 +94,47 @@ namespace WWTWebservices
|
|||
fileStream = null;
|
||||
}
|
||||
}
|
||||
// <dinoj>
|
||||
static bool HasUsefulHeaders(string plateFileName)
|
||||
{
|
||||
// returns true if plateFileName has the magic number identifying this as a .plate file
|
||||
int L = -1;
|
||||
return GetLevelCount(plateFileName, out L);
|
||||
}
|
||||
|
||||
public static int GetImageCountOneAxis(int level) => (int)Math.Pow(2, level);
|
||||
|
||||
public static bool GetLevelCount(string plateFileName, out int L)
|
||||
{
|
||||
// Returns true if plateFileName has the magic number identifying this as a .plate file
|
||||
// Also returns the number of levels in the .plate file.
|
||||
// This is 10 by default i.e. if no headers are found.
|
||||
L = 10; //
|
||||
bool hasHeadersWithInfo = false;
|
||||
if (File.Exists(plateFileName))
|
||||
{
|
||||
using (FileStream fs = new FileStream(plateFileName, FileMode.Open, FileAccess.Read))
|
||||
using (var fs = new FileStream(plateFileName, FileMode.Open, FileAccess.Read))
|
||||
{
|
||||
if (fs != null)
|
||||
{
|
||||
uint FirstFourBytes, SecondFourBytes;
|
||||
FirstFourBytes = GetNodeInfo(fs, 0, out SecondFourBytes);
|
||||
if (FirstFourBytes == dotPlateFileTypeNumber)
|
||||
{
|
||||
L = (int)SecondFourBytes;
|
||||
hasHeadersWithInfo = true;
|
||||
}
|
||||
}
|
||||
fs.Close();
|
||||
return GetLevelCount(fs, out L);
|
||||
}
|
||||
}
|
||||
return hasHeadersWithInfo;
|
||||
|
||||
L = 10;
|
||||
return false;
|
||||
}
|
||||
|
||||
public static bool GetLevelCount(Stream stream, out int L)
|
||||
{
|
||||
// Returns true if plateFileName has the magic number identifying this as a .plate file
|
||||
// Also returns the number of levels in the .plate file.
|
||||
|
||||
if (stream != null)
|
||||
{
|
||||
uint FirstFourBytes = GetNodeInfo(stream, 0, out var SecondFourBytes);
|
||||
if (FirstFourBytes == dotPlateFileTypeNumber)
|
||||
{
|
||||
L = (int)SecondFourBytes;
|
||||
return true;
|
||||
}
|
||||
}
|
||||
|
||||
// This is 10 by default i.e. if no headers are found.
|
||||
L = 10;
|
||||
return false;
|
||||
}
|
||||
// </dinoj>
|
||||
|
||||
private void WriteHeaders()
|
||||
{
|
||||
// <dinoj>
|
||||
uint L = (uint)levels;
|
||||
uint L = (uint)Levels;
|
||||
byte[] buffer = new byte[8];
|
||||
buffer[0] = (byte)(dotPlateFileTypeNumber % 256);
|
||||
buffer[1] = (byte)((dotPlateFileTypeNumber >> 8) % 256);
|
||||
|
|
@ -209,7 +167,7 @@ namespace WWTWebservices
|
|||
{
|
||||
get
|
||||
{
|
||||
return GetFileIndexOffset(levels, 0, 0);
|
||||
return GetFileIndexOffset(Levels, 0, 0);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -229,14 +187,6 @@ namespace WWTWebservices
|
|||
|
||||
}
|
||||
|
||||
public Stream GetFileStream(int level, int x, int y)
|
||||
{
|
||||
if (filename.Length > 0 && File.Exists(filename) && levels > level)
|
||||
{
|
||||
return GetFileStream(filename, level, x, y);
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
public static async Task<Stream> GetImageStreamAsync(Stream f, int level, int x, int y, CancellationToken token)
|
||||
{
|
||||
|
|
@ -248,7 +198,7 @@ namespace WWTWebservices
|
|||
return new StreamSlice(f, start, length);
|
||||
}
|
||||
|
||||
static public Stream GetFileStream(string filename, int level, int x, int y)
|
||||
public static Stream GetFileStream(string filename, int level, int x, int y)
|
||||
{
|
||||
uint offset = GetFileIndexOffset(level, x, y);
|
||||
uint start;
|
||||
|
|
@ -268,7 +218,7 @@ namespace WWTWebservices
|
|||
return ms;
|
||||
}
|
||||
|
||||
public static async ValueTask<(uint start, uint length)> GetNodeInfoAsync(Stream fs, uint offset, CancellationToken token)
|
||||
private static async ValueTask<(uint start, uint length)> GetNodeInfoAsync(Stream fs, uint offset, CancellationToken token)
|
||||
{
|
||||
Byte[] buf = new Byte[8];
|
||||
fs.Seek(offset, SeekOrigin.Begin);
|
||||
|
|
@ -280,7 +230,7 @@ namespace WWTWebservices
|
|||
return (start, length);
|
||||
}
|
||||
|
||||
public static uint GetNodeInfo(Stream fs, uint offset, out uint length)
|
||||
private static uint GetNodeInfo(Stream fs, uint offset, out uint length)
|
||||
{
|
||||
Byte[] buf = new Byte[8];
|
||||
fs.Seek(offset, SeekOrigin.Begin);
|
||||
|
|
@ -291,7 +241,7 @@ namespace WWTWebservices
|
|||
return (uint)((buf[0] + (buf[1] << 8) + (buf[2] << 16) + (buf[3] << 24)));
|
||||
}
|
||||
|
||||
static public void SetNodeInfo(FileStream fs, uint offset, uint start, uint length)
|
||||
private static void SetNodeInfo(Stream fs, uint offset, uint start, uint length)
|
||||
{
|
||||
Byte[] buf = new Byte[8];
|
||||
buf[0] = (byte)start;
|
||||
|
|
@ -311,6 +261,7 @@ namespace WWTWebservices
|
|||
GC.SuppressFinalize(this);
|
||||
Dispose(true);
|
||||
}
|
||||
|
||||
public void Dispose(bool disposing)
|
||||
{
|
||||
if (!this._disposed)
|
||||
|
|
@ -318,16 +269,12 @@ namespace WWTWebservices
|
|||
// If disposing equals true, dispose all managed and unmanaged resources.
|
||||
if (disposing)
|
||||
{
|
||||
if (_readStream != null)
|
||||
_readStream.Dispose();
|
||||
_readStream = null;
|
||||
}
|
||||
this._disposed = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
public class LevelInfo
|
||||
{
|
||||
public int level;
|
||||
|
|
|
|||
|
|
@ -0,0 +1,16 @@
|
|||
using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace WWT
|
||||
{
|
||||
public static class StreamExtensions
|
||||
{
|
||||
/// <summary>
|
||||
/// This is available on platforms after .NET Standard 2.0, but this mimics the general shape so we don't have deal with a buffer size.
|
||||
/// Per the documentation, the default buffer size is 81920 bytes.
|
||||
/// </summary>
|
||||
public static Task CopyToAsync(this Stream stream, Stream destination, CancellationToken token)
|
||||
=> stream.CopyToAsync(destination, 81920, token);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,42 @@
|
|||
using System.IO;
|
||||
using System.IO.Compression;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using WWT.PlateFiles;
|
||||
|
||||
namespace WWT.Providers
|
||||
{
|
||||
public class DevDataAccessor : IDevDataAccessor
|
||||
{
|
||||
private readonly IPlateTileDownloader _downloader;
|
||||
|
||||
private readonly string[] _plateFiles = new[]
|
||||
{
|
||||
"DSSTerraPixel.plate"
|
||||
};
|
||||
|
||||
public DevDataAccessor(IPlateTileDownloader downloader)
|
||||
{
|
||||
_downloader = downloader;
|
||||
}
|
||||
|
||||
public async Task<Stream> GetDevDataAsync(int maxLevel, CancellationToken token)
|
||||
{
|
||||
var ms = new MemoryStream();
|
||||
|
||||
using (var zip = new ZipArchive(ms, ZipArchiveMode.Update, leaveOpen: true))
|
||||
{
|
||||
foreach (var plateFile in _plateFiles)
|
||||
{
|
||||
var entry = zip.CreateEntry($"coredata/{plateFile}");
|
||||
using var stream = entry.Open();
|
||||
|
||||
await _downloader.DownloadPlateFileAsync(plateFile, stream, maxLevel, token);
|
||||
}
|
||||
}
|
||||
|
||||
ms.Position = 0;
|
||||
return ms;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
using System.IO;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace WWT.Providers
|
||||
{
|
||||
public interface IDevDataAccessor
|
||||
{
|
||||
Task<Stream> GetDevDataAsync(int maxLevel, CancellationToken token);
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,43 @@
|
|||
using Microsoft.Extensions.Logging;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
|
||||
namespace WWT.Providers
|
||||
{
|
||||
[RequestEndpoint("/v2/data/dev_export")]
|
||||
public class DevDataProvider : RequestProvider
|
||||
{
|
||||
private const int MaxLevel = 4;
|
||||
|
||||
private readonly IDevDataAccessor _devData;
|
||||
private readonly ILogger<DevDataProvider> _logger;
|
||||
|
||||
public DevDataProvider(IDevDataAccessor devData, ILogger<DevDataProvider> logger)
|
||||
{
|
||||
_devData = devData;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
public override string ContentType => ContentTypes.Zip;
|
||||
|
||||
public override async Task RunAsync(IWwtContext context, CancellationToken token)
|
||||
{
|
||||
if (!int.TryParse(context.Request.Params["level"], out var level))
|
||||
{
|
||||
level = MaxLevel;
|
||||
}
|
||||
|
||||
if (level > MaxLevel)
|
||||
{
|
||||
_logger.LogInformation("Level {Level} was requested above max {MaxLevel}", level, MaxLevel);
|
||||
level = MaxLevel;
|
||||
}
|
||||
|
||||
_logger.LogInformation("Retrieving dev data for datasets up to {Level}", level);
|
||||
|
||||
using var result = await _devData.GetDevDataAsync(level, token);
|
||||
|
||||
await result.CopyToAsync(context.Response.OutputStream, token);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -28,6 +28,8 @@ namespace WWT.Providers
|
|||
public const string XWtt = "application/x-wtt";
|
||||
|
||||
public const string XWtml = "application/x-wtml";
|
||||
|
||||
public const string Zip = "application/zip";
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -1,10 +1,7 @@
|
|||
using Microsoft.Extensions.DependencyInjection;
|
||||
using System;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
using System.Reflection;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using WWT.Providers.Services;
|
||||
using WWTWebservices;
|
||||
|
||||
|
|
@ -34,6 +31,7 @@ namespace WWT.Providers
|
|||
services.AddSingleton<IOctTileMapBuilder, OctTileMapBuilder>();
|
||||
services.AddSingleton<IMandelbrot, Mandelbrot>();
|
||||
services.AddSingleton<IVirtualEarthDownloader, VirtualEarthDownloader>();
|
||||
services.AddSingleton<IDevDataAccessor, DevDataAccessor>();
|
||||
|
||||
var options = new WwtOptions();
|
||||
|
||||
|
|
@ -41,12 +39,5 @@ namespace WWT.Providers
|
|||
|
||||
services.AddSingleton(options);
|
||||
}
|
||||
|
||||
/// <summary>
|
||||
/// This is available on platforms after .NET Standard 2.0, but this mimics the general shape so we don't have deal with a buffer size.
|
||||
/// Per the documentation, the default buffer size is 81920 bytes.
|
||||
/// </summary>
|
||||
internal static Task CopyToAsync(this Stream stream, Stream destination, CancellationToken token)
|
||||
=> stream.CopyToAsync(destination, 81920, token);
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -24,6 +24,7 @@
|
|||
|
||||
<ItemGroup>
|
||||
<PackageReference Include="Microsoft.Extensions.DependencyInjection.Abstractions" Version="3.1.9" />
|
||||
<PackageReference Include="Microsoft.Extensions.Logging.Abstractions" Version="3.1.9" />
|
||||
<PackageReference Include="System.Net.Http" Version="4.3.4" />
|
||||
</ItemGroup>
|
||||
|
||||
|
|
|
|||
|
|
@ -1,4 +1,5 @@
|
|||
using Microsoft.ApplicationInsights.Extensibility;
|
||||
using Microsoft.Extensions.Caching.Distributed;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using Microsoft.Extensions.Logging.ApplicationInsights;
|
||||
|
|
@ -134,6 +135,7 @@ namespace WWTMVC5
|
|||
.CacheType<IThumbnailAccessor>(plates => plates
|
||||
.Add(nameof(IThumbnailAccessor.GetThumbnailStreamAsync))
|
||||
.Add(nameof(IThumbnailAccessor.GetDefaultThumbnailStreamAsync)))
|
||||
.CacheType<IDevDataAccessor>(dev => dev.Add(nameof(IDevDataAccessor.GetDevDataAsync)))
|
||||
.CacheType<ITourAccessor>(plates => plates
|
||||
.Add(nameof(ITourAccessor.GetAuthorThumbnailAsync))
|
||||
.Add(nameof(ITourAccessor.GetTourAsync))
|
||||
|
|
|
|||
|
|
@ -77,6 +77,7 @@ Content-Type: application/x-wt-->
|
|||
</security>
|
||||
<handlers>
|
||||
<add name="wwtweb" verb="*" path="wwtweb/*" type="WWTMVC5.WwtWebHttpHandler, WWTMVC5" preCondition="managedHandler"/>
|
||||
<add name="v2" verb="*" path="v2/*" type="WWTMVC5.WwtWebHttpHandler, WWTMVC5" preCondition="managedHandler"/>
|
||||
<add name="gettourfile" verb="*" path="GetTourFile.aspx" type="WWTMVC5.WwtWebHttpHandler, WWTMVC5" preCondition="managedHandler"/>
|
||||
<add name="gettourfile2" verb="*" path="GetTourFile2.aspx" type="WWTMVC5.WwtWebHttpHandler, WWTMVC5" preCondition="managedHandler"/>
|
||||
</handlers>
|
||||
|
|
|
|||
|
|
@ -1,250 +0,0 @@
|
|||
using System;
|
||||
using System.Drawing;
|
||||
using System.IO;
|
||||
|
||||
namespace WWTWebservices
|
||||
{
|
||||
/* This contains dinoj's utilities for dealing with image pyramids and PlateTilePyramids
|
||||
*
|
||||
* Things to add
|
||||
*
|
||||
* create .plate file given any of the following
|
||||
* 2^N x 2^N image
|
||||
* X x Y image (place inside a 2^N x 2^N image)
|
||||
* an image pyramid
|
||||
* a level of an image pyramid
|
||||
*
|
||||
* given L X Y and (xfrac,yfrac) within the tile, output L+1 X1 Y1 for corresponding tile in level below
|
||||
* given L X Y, output L-1 X1 Y1 for parent tile
|
||||
* given L X Y and (xfrac,yfrac) within the tile, output L-1 X1 Y1 for parent tile (and corresponding xfrac1 yfrac1?)
|
||||
*
|
||||
*
|
||||
* This file is very much in progress and should only be used as examples
|
||||
*/
|
||||
|
||||
public class ImagePyramid
|
||||
{
|
||||
/*
|
||||
*
|
||||
*
|
||||
*/
|
||||
int maxLevel = -1; // if all levels are filled, this is one less than the number of levels
|
||||
int[] levels = null;
|
||||
string ext = ""; // type of image
|
||||
int side = -1; // length of image side
|
||||
string origPath = ""; // where image pyramid came from
|
||||
bool valid = false; // true iff all the above member data can be properly initialized.
|
||||
|
||||
public string OrigPath
|
||||
{
|
||||
get { return origPath; }
|
||||
set { origPath = value; }
|
||||
}
|
||||
|
||||
public int MaxLevel
|
||||
{
|
||||
get { return maxLevel; }
|
||||
}
|
||||
public string Ext
|
||||
{
|
||||
get { return ext; }
|
||||
}
|
||||
public int Side
|
||||
{
|
||||
get { return side; }
|
||||
}
|
||||
|
||||
public ImagePyramid(string dir)
|
||||
{
|
||||
if (dir.Length > 0 && Directory.Exists(dir))
|
||||
{
|
||||
origPath = dir;
|
||||
if (! origPath[origPath.Length - 1].Equals('\\'))
|
||||
{
|
||||
origPath += "\\";
|
||||
}
|
||||
// origPath now ends with a \ , emphasizing that this is a directory
|
||||
string[] tmp = Directory.GetDirectories(origPath);
|
||||
|
||||
int count = 0;
|
||||
maxLevel = -1;
|
||||
for (int i = 0; i < tmp.Length; i++)
|
||||
{
|
||||
if (tmp[i].Length > 0)
|
||||
{
|
||||
int k = Convert.ToInt32(Path.GetFileNameWithoutExtension(tmp[i]));
|
||||
if (k >= 0)
|
||||
{
|
||||
maxLevel = (maxLevel > k ? maxLevel : k);
|
||||
count++;
|
||||
}
|
||||
}
|
||||
}
|
||||
levels = new int[count];
|
||||
|
||||
count = 0;
|
||||
for (int i = 0; i < tmp.Length; i++)
|
||||
{
|
||||
if (tmp[i].Length > 0)
|
||||
{
|
||||
int k = Convert.ToInt32(Path.GetFileNameWithoutExtension(tmp[i]));
|
||||
if (k >= 0)
|
||||
{
|
||||
levels[count++] = k;
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
if (levels.Length > 0)
|
||||
{
|
||||
string dire = origPath + string.Format("{0}\\0", levels[0]);
|
||||
if (Directory.Exists(dire))
|
||||
{
|
||||
string[] files = Directory.GetFiles(dire);
|
||||
if (files.Length > 0)
|
||||
{
|
||||
ext = Path.GetExtension(files[0]);
|
||||
using (Bitmap b = new Bitmap(files[0]))
|
||||
{
|
||||
if (b.Height == b.Width && b.Height > 0)
|
||||
{
|
||||
side = b.Height;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if (side > 0)
|
||||
{
|
||||
valid = true;
|
||||
}
|
||||
}
|
||||
|
||||
public bool BuildupPyramid(bool overwrite)
|
||||
{
|
||||
// creates more levels
|
||||
return false;
|
||||
}
|
||||
|
||||
public bool FillPyramid()
|
||||
{
|
||||
// Fills up levels of pyramid in origPath
|
||||
// Returns true if pyramid is full by the time the function ends
|
||||
// not implemented yet
|
||||
if (maxLevel != levels.Length - 1)
|
||||
{
|
||||
// implement this
|
||||
// if new levels are added, add them to the array levels.
|
||||
}
|
||||
return (maxLevel == levels.Length - 1);
|
||||
}
|
||||
|
||||
public Bitmap createMosaic(int level)
|
||||
{
|
||||
// creates a large bitmap from a level
|
||||
bool levelExists = false;
|
||||
if (level <= maxLevel)
|
||||
{
|
||||
foreach (int L in levels)
|
||||
{
|
||||
if (L.Equals(level))
|
||||
{
|
||||
levelExists = true;
|
||||
break;
|
||||
}
|
||||
}
|
||||
}
|
||||
if (!levelExists)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
int L2 = (int)Math.Pow(2, level);
|
||||
int bSide = side * L2;
|
||||
Bitmap b = new Bitmap(bSide, bSide);
|
||||
Graphics g = Graphics.FromImage(b);
|
||||
string tileFileName;
|
||||
for (int y = 0; y < L2; y++)
|
||||
{
|
||||
for (int x = 0; x < L2; x++)
|
||||
{
|
||||
tileFileName = string.Format("{0}{1}\\{3}\\{3}_{2}.png", origPath, level, x, y);
|
||||
if (File.Exists(tileFileName))
|
||||
{
|
||||
using (Bitmap bTile = new Bitmap(tileFileName))
|
||||
{
|
||||
g.DrawImage(bTile, x * side, y * side);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return b;
|
||||
}
|
||||
|
||||
public PlateTilePyramid createPTP(string dotPlateFile)
|
||||
{
|
||||
/*
|
||||
* Creates a .plate file with the name dotPlateFile (which should include path info)
|
||||
* from the current image pyramid. Adds ".plate" extension if not present.
|
||||
* Assumes that the current image pyramid is full i.e. has all levels
|
||||
* Returns the PlateTilePyramid object created, though this function will
|
||||
* often be called without any return value.
|
||||
*
|
||||
*
|
||||
*/
|
||||
if (!valid)
|
||||
{
|
||||
return null;
|
||||
}
|
||||
|
||||
/* CAUTION: IF this stuff is added, then an exception is thrown
|
||||
* if it is unable to create a full pyramid to work from.
|
||||
|
||||
if (maxLevel != levels.Length - 1)
|
||||
{
|
||||
if (FillPyramid())
|
||||
{
|
||||
throw new System.ArgumentException("Need a full pyramid to work off ");
|
||||
}
|
||||
}
|
||||
*
|
||||
*/
|
||||
|
||||
|
||||
// dotPlateFile has name of .plate file, with or without extension
|
||||
if (Path.GetExtension(dotPlateFile).Length == 0 && dotPlateFile.IndexOf(".") < 0)
|
||||
{
|
||||
dotPlateFile = dotPlateFile + ".plate";
|
||||
}
|
||||
// dotPlateFile has name of .plate file, with extension
|
||||
|
||||
string dotPlateDir = Path.GetDirectoryName(dotPlateFile);
|
||||
if (Directory.Exists(dotPlateDir))
|
||||
{
|
||||
Directory.CreateDirectory(dotPlateDir);
|
||||
}
|
||||
|
||||
PlateTilePyramid ptp = new PlateTilePyramid(dotPlateFile, maxLevel+1);
|
||||
ptp.Create();
|
||||
for (int level = maxLevel; level >=0; level--)
|
||||
{
|
||||
int count = (int)Math.Pow(2, level);
|
||||
for (int y = 0; y < count; y++)
|
||||
{
|
||||
for (int x = 0; x < count; x++)
|
||||
{
|
||||
string tileFileName = string.Format("{0}{1}\\{3}\\{3}_{2}.png", origPath, level, x, y);
|
||||
if (File.Exists(tileFileName))
|
||||
{
|
||||
ptp.AddFile(tileFileName, level, x, y);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
ptp.UpdateHeaderAndClose();
|
||||
return ptp;
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
}
|
||||
|
|
@ -0,0 +1,104 @@
|
|||
using Microsoft.Extensions.Logging;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
using System.Linq;
|
||||
using System.Threading;
|
||||
using System.Threading.Tasks;
|
||||
using WWT.Azure;
|
||||
using WWT.PlateFiles;
|
||||
|
||||
namespace PlateManager.Download
|
||||
{
|
||||
class DownloadCommand : ICommand
|
||||
{
|
||||
private readonly DownloadCommandOptions _options;
|
||||
private readonly IPlateTileDownloader _downloader;
|
||||
private readonly ILogger<DownloadCommand> _logger;
|
||||
|
||||
public DownloadCommand(DownloadCommandOptions options, IPlateTileDownloader downloader, ILogger<DownloadCommand> logger)
|
||||
{
|
||||
_options = options;
|
||||
_downloader = downloader;
|
||||
_logger = logger;
|
||||
}
|
||||
|
||||
private IAsyncEnumerable<string> GetAvailableContainers(CancellationToken token)
|
||||
{
|
||||
if (_options.Plate is null)
|
||||
{
|
||||
_logger.LogInformation("No plates were specified. Defaulting to all available.");
|
||||
|
||||
return _downloader.GetPlateNames(token);
|
||||
}
|
||||
|
||||
_logger.LogInformation("Using supplied plate names");
|
||||
|
||||
return _options.Plate.ToAsyncEnumerable();
|
||||
}
|
||||
|
||||
public async Task RunAsync(CancellationToken token)
|
||||
{
|
||||
if (!_options.Output.Exists)
|
||||
{
|
||||
_options.Output.Create();
|
||||
}
|
||||
|
||||
await foreach (var plate in GetAvailableContainers(token).WithCancellation(token))
|
||||
{
|
||||
var outputFile = new FileInfo(Path.Combine(_options.Output.FullName, plate));
|
||||
|
||||
if (_options.SkipExisting && outputFile.Exists)
|
||||
{
|
||||
_logger.LogWarning("Plate {PlateFile} already exists. Skipping.");
|
||||
|
||||
continue;
|
||||
}
|
||||
|
||||
_logger.LogInformation("Downloading {PlateFile}", outputFile.Name);
|
||||
|
||||
try
|
||||
{
|
||||
using (var fs = outputFile.OpenWrite())
|
||||
{
|
||||
fs.SetLength(0);
|
||||
|
||||
var count = await _downloader.DownloadPlateFileAsync(plate, fs, _options.Levels, token);
|
||||
|
||||
if (count == 0)
|
||||
{
|
||||
_logger.LogWarning("No content added for {PlateFile}", outputFile.Name);
|
||||
}
|
||||
}
|
||||
}
|
||||
catch (Exception e)
|
||||
{
|
||||
_logger.LogError(e, "Unexpected error getting {PlateName}", outputFile.Name);
|
||||
|
||||
if (outputFile.Exists)
|
||||
{
|
||||
outputFile.Delete();
|
||||
}
|
||||
}
|
||||
|
||||
var size = FormatSize(outputFile.Length);
|
||||
|
||||
_logger.LogInformation("Completed {PlateFile} download. Resulting file is {Size}", outputFile.Name, size);
|
||||
}
|
||||
}
|
||||
|
||||
private readonly string[] sizes = new[] { "B", "KB", "MB", "GB", "TB" };
|
||||
|
||||
private string FormatSize(double len)
|
||||
{
|
||||
int order = 0;
|
||||
while (len >= 1024 && order < sizes.Length - 1)
|
||||
{
|
||||
order++;
|
||||
len /= 1024;
|
||||
}
|
||||
|
||||
return string.Format("{0:0.##} {1}", len, sizes[order]);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
using Microsoft.Extensions.DependencyInjection;
|
||||
using System.Collections.Generic;
|
||||
using System.IO;
|
||||
|
||||
namespace PlateManager.Download
|
||||
{
|
||||
class DownloadCommandOptions : BaseOptions, IServiceRegistration
|
||||
{
|
||||
public IEnumerable<string> Plate { get; set; }
|
||||
|
||||
public DirectoryInfo Output { get; set; }
|
||||
|
||||
public int Levels { get; set; }
|
||||
|
||||
public void AddServices(IServiceCollection services)
|
||||
{
|
||||
services.AddSingleton<ICommand, DownloadCommand>();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -20,6 +20,7 @@
|
|||
<PackageReference Include="Serilog.Sinks.File" Version="4.1.0" />
|
||||
<PackageReference Include="System.CommandLine" Version="2.0.0-beta1.20371.2" />
|
||||
<PackageReference Include="System.CommandLine.DragonFruit" Version="0.3.0-alpha.20371.2" />
|
||||
<PackageReference Include="System.Linq.Async" Version="4.1.1" />
|
||||
</ItemGroup>
|
||||
|
||||
<ItemGroup>
|
||||
|
|
|
|||
|
|
@ -1,18 +1,14 @@
|
|||
using Azure.Identity;
|
||||
using Microsoft.Extensions.Azure;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.DependencyInjection;
|
||||
using Microsoft.Extensions.Logging;
|
||||
using PlateManager.List;
|
||||
using Serilog;
|
||||
using Serilog.Events;
|
||||
using System;
|
||||
using System.Collections.Generic;
|
||||
using System.CommandLine;
|
||||
using System.CommandLine.Invocation;
|
||||
using System.IO;
|
||||
using System.Threading.Tasks;
|
||||
using WWT.Azure;
|
||||
using WWTWebservices;
|
||||
|
||||
namespace PlateManager
|
||||
{
|
||||
|
|
@ -23,6 +19,7 @@ namespace PlateManager
|
|||
var root = new RootCommand
|
||||
{
|
||||
CreateUploadCommand(),
|
||||
CreateDownloadCommand(),
|
||||
CreateListCommand(),
|
||||
};
|
||||
|
||||
|
|
@ -64,6 +61,22 @@ namespace PlateManager
|
|||
return command;
|
||||
}
|
||||
|
||||
static Command CreateDownloadCommand()
|
||||
{
|
||||
var command = new Command("download")
|
||||
{
|
||||
new Option<string>("--plate") { IsRequired = true },
|
||||
new Option<DirectoryInfo>(new[] { "--output", "-o" }) { IsRequired = true },
|
||||
new Option<int>("--levels", () => 2),
|
||||
};
|
||||
|
||||
AddDefaultOptions(command);
|
||||
|
||||
command.Handler = CommandHandler.Create<Download.DownloadCommandOptions>(Run);
|
||||
|
||||
return command;
|
||||
}
|
||||
|
||||
private static void AddDefaultOptions(Command command)
|
||||
{
|
||||
command.Add(new Option<string>("--storage", () => "https://127.0.0.1:10000/devstoreaccount1/"));
|
||||
|
|
@ -96,35 +109,19 @@ namespace PlateManager
|
|||
options.AddServices(services);
|
||||
|
||||
services.AddSingleton(options);
|
||||
services.AddSingleton<AzurePlateTilePyramid>();
|
||||
services.AddSingleton<IPlateTilePyramid>(ctx => ctx.GetRequiredService<AzurePlateTilePyramid>());
|
||||
|
||||
services.AddSingleton(new AzurePlateTilePyramidOptions
|
||||
{
|
||||
CreateContainer = true,
|
||||
SkipIfExists = options.SkipExisting,
|
||||
OverwriteExisting = !options.SkipExisting,
|
||||
Container = options.AzureContainer
|
||||
});
|
||||
|
||||
services.AddAzureClients(builder =>
|
||||
{
|
||||
if (Uri.TryCreate(options.Storage, UriKind.Absolute, out var storageUri))
|
||||
services
|
||||
.AddAzureServices(opt =>
|
||||
{
|
||||
// Use the storage URI and register the credential provider
|
||||
builder.AddBlobServiceClient(storageUri);
|
||||
|
||||
if (options.Interactive)
|
||||
builder.UseCredential(new InteractiveBrowserCredential());
|
||||
else
|
||||
builder.UseCredential(new DefaultAzureCredential());
|
||||
}
|
||||
else
|
||||
opt.StorageAccount = options.Storage;
|
||||
})
|
||||
.AddPlateFiles(opt =>
|
||||
{
|
||||
// this is actually a storage connection string with included credentials
|
||||
builder.AddBlobServiceClient(options.Storage);
|
||||
}
|
||||
});
|
||||
opt.CreateContainer = true;
|
||||
opt.SkipIfExists = options.SkipExisting;
|
||||
opt.OverwriteExisting = !options.SkipExisting;
|
||||
opt.Container = options.AzureContainer;
|
||||
});
|
||||
|
||||
await using var container = services.BuildServiceProvider();
|
||||
|
||||
|
|
|
|||
Загрузка…
Ссылка в новой задаче