Added TenGB option + download links

This commit is contained in:
Tomas Kovarik 2022-08-18 07:24:02 +02:00
Родитель 4e59d50295
Коммит 414bf569e1
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4FAD4C7ABE120439
3 изменённых файлов: 22 добавлений и 4 удалений

Просмотреть файл

@ -14,6 +14,7 @@ namespace BenchmarkLogGenerator
public enum BenchmarkDataSize
{
OneGB,
TenGB,
OneTB,
HundredTB
}

Просмотреть файл

@ -106,6 +106,8 @@ namespace BenchmarkLogGenerator
{
case BenchmarkDataSize.OneGB:
return 1;
case BenchmarkDataSize.TenGB:
return 2;
case BenchmarkDataSize.OneTB:
return 6;
case BenchmarkDataSize.HundredTB:
@ -122,6 +124,8 @@ namespace BenchmarkLogGenerator
{
case BenchmarkDataSize.OneGB:
return 1;
case BenchmarkDataSize.TenGB:
return 1;
case BenchmarkDataSize.OneTB:
return 1;
case BenchmarkDataSize.HundredTB:
@ -155,6 +159,8 @@ namespace BenchmarkLogGenerator
{
case BenchmarkDataSize.OneGB:
return 1;
case BenchmarkDataSize.TenGB:
return 5;
case BenchmarkDataSize.OneTB:
return 100;
case BenchmarkDataSize.HundredTB:
@ -170,6 +176,8 @@ namespace BenchmarkLogGenerator
{
case BenchmarkDataSize.OneGB:
return 1;
case BenchmarkDataSize.TenGB:
return 1;
case BenchmarkDataSize.OneTB:
return 1;
case BenchmarkDataSize.HundredTB:
@ -190,7 +198,7 @@ namespace BenchmarkLogGenerator
esb.AppendLine("-localPath: The root folder");
esb.AppendLine("-azureStorageAccountConnections: A comma separated list of Azure storage account connections (can be single connection), containers will be created automaticly using the following template: logsBenchmark-{size}-p{partition}");
esb.AppendLine("-eventHubConnection: The connection string for Azure EventHub");
esb.AppendLine("-size: The output size, possible values are OneGB, OneTB, HundredTB. Default is OneGB");
esb.AppendLine("-size: The output size, possible values are OneGB, TenGB, OneTB, HundredTB. Default is OneGB");
esb.AppendLine("-partition: The applicable partition, between -1 to 9, where -1 means single partition. Only relevant for HundredTB size. Default is -1");
esb.AppendLine();
CommandLineArgsParser.PrintUsage(esb);

Просмотреть файл

@ -4,7 +4,7 @@ This tool supports following configurable command line parameters –
1. **-output**: This is the location where output should be written to.
Supported values are: LocalDisk, AzureStorage, EventHub
2. **-size**: The data size to be generated.
Supported values are: OneGB, OneTB, HundredTB
Supported values are: OneGB, TenGB, OneTB, HundredTB
Default value is OneGB
3. **-partition**: The value for data partition, it could be between -1 to 9, where -1 means single partition. Default value is -1. Its only relevant for HundredTB data size.
@ -18,15 +18,24 @@ Default value is OneGB
`BenchmarkLogGenerator -output:EventHub -eventHubConnection:Endpoint=sb://EHNAMESPACE.servicebus.windows.net/;EntityPath=EHNAME;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=KEYVALUE -size:OneGB"`
**Note** – Data size is restricted to 1 GB for event hub.
**Note** – We host the generated files with default seed value at https://logsbenchmark00.blob.core.windows.net/logsbenchmark-onegb?sp=rl&st=2022-08-18T00:00:00Z&se=2030-01-01T00:00:00Z&spr=https&sv=2021-06-08&sr=c&sig=5pjOow5An3%2BTs5mZ%2FyosJBPtDvV7%2FXfDO8pLEeeylVc%3D. Please use Azure Storage Explorer and attach a blob container specifying this URL or leverage AzCopy to download it.
2. **OneTB size**
2. **TenGB size**
`BenchmarkLogGenerator.exe -output:AzureStorage -size:TenGB -cc:BLOB STORAGE CONN STR`
`BenchmarkLogGenerator -output:LocalDisk -size:TenGB -localPath:"C:\DATA"`
**Note** – We host the generated files with default seed value at https://logsbenchmark00.blob.core.windows.net/logsbenchmark-tengb?sp=rl&st=2022-08-18T00:00:00Z&se=2030-01-01T00:00:00Z&spr=https&sv=2021-06-08&sr=c&sig=AcZvWrUj9EHWoV6%2BIKeo3dC12f06iq%2Fo42IRI6h4t8o%3D. Please use Azure Storage Explorer and attach a blob container specifying this URL or leverage AzCopy to download it.
3. **OneTB size**
`BenchmarkLogGenerator.exe -output:AzureStorage -size:OneTB -cc:BLOB STORAGE CONN STR`
`BenchmarkLogGenerator -output:LocalDisk -size:OneTB -localPath:"C:\DATA"`
You can also use below mentioned Azure batch templates to generate 1 TB data, 3 Standard_D32_v3 VMs would be enough to generate 1 TB .
3. **HundredTB size**
4. **HundredTB size**
Use Azure Batch compute to generate 100 TBs of data. Tool has Azure batch templates for generating required batch pools and jobs.
Follow these steps to generate 100TBs of data -