added execute db script for easy data load to ADX

This commit is contained in:
Tomas Kovarik 2022-08-18 16:29:36 +02:00
Родитель 414bf569e1
Коммит 68aa54ede5
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4FAD4C7ABE120439
1 изменённых файлов: 39 добавлений и 2 удалений

Просмотреть файл

@ -18,7 +18,26 @@ Default value is OneGB
`BenchmarkLogGenerator -output:EventHub -eventHubConnection:Endpoint=sb://EHNAMESPACE.servicebus.windows.net/;EntityPath=EHNAME;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=KEYVALUE -size:OneGB"`
**Note** – Data size is restricted to 1 GB for event hub.
**Note** – We host the generated files with default seed value at https://logsbenchmark00.blob.core.windows.net/logsbenchmark-onegb?sp=rl&st=2022-08-18T00:00:00Z&se=2030-01-01T00:00:00Z&spr=https&sv=2021-06-08&sr=c&sig=5pjOow5An3%2BTs5mZ%2FyosJBPtDvV7%2FXfDO8pLEeeylVc%3D. Please use Azure Storage Explorer and attach a blob container specifying this URL or leverage AzCopy to download it.
**Note** – We host the generated files with default seed value at https://logsbenchmark00.blob.core.windows.net/logsbenchmark-onegb?sp=rl&st=2022-08-18T00:00:00Z&se=2030-01-01T00:00:00Z&spr=https&sv=2021-06-08&sr=c&sig=5pjOow5An3%2BTs5mZ%2FyosJBPtDvV7%2FXfDO8pLEeeylVc%3D. Please use Azure Storage Explorer and attach a blob container specifying this URL or leverage AzCopy to download it. If you plan to load the data to an instance of Azure Data Explorer, use the following snippet to do so directly.
```kql
.execute database script <|
// Define external table pointing to blob storage.
.create-or-alter external table LogsExternal(Timestamp:datetime, Source:string, Node:string, Level:string, Component:string, ClientRequestId:string, Message:string, Properties:dynamic)
kind=storage
partition by (Date:datetime)
pathformat = (datetime_pattern("yyyy/MM/dd/HH", Date))
dataformat=csv
(
h'https://logsbenchmark00.blob.core.windows.net/logsbenchmark-onegb?sp=rl&st=2022-08-18T00:00:00Z&se=2030-01-01T00:00:00Z&spr=https&sv=2021-06-08&sr=c&sig=5pjOow5An3%2BTs5mZ%2FyosJBPtDvV7%2FXfDO8pLEeeylVc%3D'
)
with (includeHeaders='None', compressed=true)
//
// Ingest data from external table to internal table named Logs.
// The command executes asynchronously in the background. Use ".show operations <<OperationId>>" to monitor its progress.
.set-or-replace async Logs <| external_table('LogsExternal')
```
2. **TenGB size**
@ -26,7 +45,25 @@ Default value is OneGB
`BenchmarkLogGenerator -output:LocalDisk -size:TenGB -localPath:"C:\DATA"`
**Note** – We host the generated files with default seed value at https://logsbenchmark00.blob.core.windows.net/logsbenchmark-tengb?sp=rl&st=2022-08-18T00:00:00Z&se=2030-01-01T00:00:00Z&spr=https&sv=2021-06-08&sr=c&sig=AcZvWrUj9EHWoV6%2BIKeo3dC12f06iq%2Fo42IRI6h4t8o%3D. Please use Azure Storage Explorer and attach a blob container specifying this URL or leverage AzCopy to download it.
**Note** – We host the generated files with default seed value at https://logsbenchmark00.blob.core.windows.net/logsbenchmark-tengb?sp=rl&st=2022-08-18T00:00:00Z&se=2030-01-01T00:00:00Z&spr=https&sv=2021-06-08&sr=c&sig=AcZvWrUj9EHWoV6%2BIKeo3dC12f06iq%2Fo42IRI6h4t8o%3D. Please use Azure Storage Explorer and attach a blob container specifying this URL or leverage AzCopy to download it. If you plan to load the data to an instance of Azure Data Explorer, use the following snippet to do so directly.
```kql
.execute database script <|
// Define external table pointing to blob storage.
.create-or-alter external table LogsExternal(Timestamp:datetime, Source:string, Node:string, Level:string, Component:string, ClientRequestId:string, Message:string, Properties:dynamic)
kind=storage
partition by (Date:datetime)
pathformat = (datetime_pattern("yyyy/MM/dd/HH", Date))
dataformat=csv
(
h'https://logsbenchmark00.blob.core.windows.net/logsbenchmark-tengb?sp=rl&st=2022-08-18T00:00:00Z&se=2030-01-01T00:00:00Z&spr=https&sv=2021-06-08&sr=c&sig=AcZvWrUj9EHWoV6%2BIKeo3dC12f06iq%2Fo42IRI6h4t8o%3D'
)
with (includeHeaders='None', compressed=true)
//
// Ingest data from external table to internal table named Logs.
// The command executes asynchronously in the background. Use ".show operations <<OperationId>>" to monitor its progress.
.set-or-replace async Logs <| external_table('LogsExternal')
```
3. **OneTB size**