Documentation update (#78)
* Update documentation * Update parameters * Add AKS pipeline option * Add images for pipelines * Update documentation * Update parameters * Add AKS pipeline option * Add images for pipelines * Update based on feedback * Update based on feedback * Correct typo and grammatical errors * Add support.md file * Add clarification on folder structure * Remove support section * Add removed section from Support.md * Update support text
This commit is contained in:
Kungumaraj Nachimuthu
GitHub
Родитель
0341d734c1
Коммит
e4a3482123
|
|
@ -0,0 +1,98 @@
|
|||
# Glossary
|
||||
|
||||
|
||||
**AOI** -
|
||||
Area of Interest represents the specific geographic extent used to define a focus area for your analysis.
|
||||
|
||||
**band** -
|
||||
One layer of a multispectral image representing data values for a specific range of the electromagnetic spectrum of reflected light or heat (e.g., ultraviolet, blue, green, red, near infrared, infrared, thermal, radar, etc.). Also, other user-specified values derived by manipulation of original image bands. A standard color display of a multispectral image shows
|
||||
three bands, one each for red, green and blue. Satellite imagery such as LANDSAT TM and SPOT provide multispectral images of the Earth, some containing seven or more bands.
|
||||
|
||||
**coordinate** -
|
||||
A set of numeric quantities that describe the location of a point in a geographic reference system. A coordinate pair describes the location of a point or node in two dimensions (ususally x-y), and a coordinate triplet describes a point in three dimensions (x-y-z). A series of points (two or more) is used to describe lines and the edges of polygons or areas. Coordinates represent locations on the Earth's surface relative to other locations.
|
||||
|
||||
**coordinate system** -
|
||||
A reference system used to measure horizontal and vertical distances on a planimetric map. A coordinate system is usually defined by a map projection, a spheroid of reference, a datum, one or more standard parallels, a central meridian, and possible shifts in the x- and y-directions to locate x,y positions of point, line, and area features. A common coordinate system is used to spatially register geographic data for the same area.
|
||||
|
||||
**coverage** -
|
||||
1. A digital version of a map forming the basic unit of vector data storage in ARC/INFO. A coverage stores geographic features as primary features (such as arcs, nodes, polygons, and
|
||||
label points) and secondary features (such as tics, map extent, links, and annotation). Associated feature attribute tables describe and store attributes of the geographic features.
|
||||
2. A set of thematically associated data considered as a unit. A coverage usually represents a single theme such as soils, streams, roads, or land use
|
||||
|
||||
**datum** -
|
||||
A set of parameters and control points used to accurately define the three-dimensional shape of the Earth (e.g., as a spheroid). The datum is the basis for a planar coordinate system. For
|
||||
example, the North American Datum for 1983 (NAD83) is a common datum for map projections and coordinates within the United States and throughout North America.
|
||||
|
||||
**feature** -
|
||||
In a GIS, a physical object or location of an event. Features can be points (a tree or a traffic accident), lines (a road or river), or areas (a forest or a parking lot).
|
||||
|
||||
**format** -
|
||||
The pattern into which data are systematically arranged for use on a computer. A file format is the specific design of how information is organized in the file. DLG, DEM, and TIGER are geographic data sets with different file formats.
|
||||
|
||||
**geodatabase** -
|
||||
An object-based GIS data model developed by ESRI for ArcGIS that stores each feature as rows in a table. Personal geodatabases store data in a Microsoft Access .mdb file. Corporate
|
||||
geodatabases store data in a DBMS such as SQLserver or Oracle. This data structure supports rules-based topology and allows the user to assign behavior to data.
|
||||
|
||||
**geometry** -
|
||||
Geometry deals with the measures and properties of points, lines and surfaces. In a GIS, geometry is used to represent the spatial component of geographic features.
|
||||
|
||||
**georeference** -
|
||||
To establish the relationship between page coordinates on a planar map and known real-world coordinates
|
||||
|
||||
**latitude** -
|
||||
The north/south component of a location on the surface of an ellipsoid. Latitude is an angular measurement north or south of the equator. Traditionally latitudes north of the equator are considered as positive and those south of the equator as negative.
|
||||
|
||||
**longitude** -
|
||||
The East/West component of a location on the surface of the Earth. Longitude is usually measured as an angular value East or West of the Greenwich meridian (London, England). Traditionally longitudes East of Greenwich are considered as positive and those West of Greenwich as negative. This is a negative value in Montana.
|
||||
|
||||
**map projection** -
|
||||
A mathematical model that transforms the locations of features on the Earth's surface to locations on a two- dimensional surface. Because the Earth is three-dimensional, some method must be used to depict a map in two dimensions. Some projections preserve shape; others preserve accuracy of area, distance, or direction. See also coordinate system. Map projections project the Earth's surface onto a flat plane. However, any such representation distorts some parameter of the Earth's surface be it distance, area, shape, or direction.
|
||||
|
||||
**map scale** -
|
||||
The reduction needed to display a representation of the Earth's surface on a map. A statement of a measure on the map and the equivalent measure on the Earth's surface, often expressed as a representative fraction of distance, such as 1:24,000 (one unit of distance on the map represents 24,000 of the same units of distance on the Earth). 'Large scale' refers to a large fractional value such as 1/24,000. A large-scale map shows a small geographic area in greater detail. 'Small scale' refers to a smaller fractional value such as 1/1,000,000. A small-scale map shows a large geographic area in less detail.
|
||||
|
||||
**point-in-polygon** -
|
||||
A topological overlay procedure which determines the spatial coincidence of points and polygons. Points are assigned the attributes of the polygons within which they fall.
|
||||
|
||||
**polygon** -
|
||||
A coverage feature class used to represent areas. A polygon is defined by the arcs that make up its boundary and a point inside its boundary for identification.
|
||||
|
||||
**projection** -
|
||||
See map projection.
|
||||
|
||||
**raster** -
|
||||
A cellular data structure composed of rows and columns for storing images. Each unit in the grid is assigned a value associating it with the corresponding attribute data. Selection of grid
|
||||
size forces a tradeoff between data resolution (and detail) and system storage requirements. Data can be converted to vector data through the process of vectorization.
|
||||
|
||||
**resampling** -
|
||||
The process of reducing image data set size by representing a group of pixels with a single pixel. Thus, pixel count is lowered, individual pixel size is increased, and overall image
|
||||
geographic extent is retained. Resampled images are "coarse" and have less information than the images from which they are taken. Conversely, this process can also be executed in the reverse.
|
||||
|
||||
**resolution** -
|
||||
1. Resolution is the accuracy at which a given map scale can depict the location and shape of geographic features. The larger the map scale, the higher the possible resolution. As map scale decreases, resolution diminishes and feature boundaries must be smoothed, simplified, or not shown at all. For example, small areas may have to be represented as points.
|
||||
2. Distance between sample points in a lattice.
|
||||
3. Size of the smallest feature that can be represented in a surface.
|
||||
4. The number of points in x and y in a grid or lattice (e.g., the resolution of a U.S. Geological Survey one-degree DEM is 1201 x 1201 mesh points).
|
||||
|
||||
**scale** -
|
||||
See map scale.
|
||||
|
||||
**TlFF** -
|
||||
Tagged interchange (image) file format. An industry-standard raster data format. TlFF supports black-and-white, gray-scale, pseudocolor, and true-color images, all of which can be stored in a
|
||||
compressed or uncompressed format. TlFF is commonly used in desktop publishing and serves as an interface to numerous scanners and graphic arts packages.
|
||||
|
||||
**transformation** -
|
||||
The process that converts coordinates from one coordinate system to another through translation, rotation, and scaling. ARC/lNFO supports these transformations: similarity, affine, piecewise linear, projective, NADCON datum adjustment using minimum-derived curvature transformation, and a polynomial transformation to warp grids and images.
|
||||
|
||||
**vector** -
|
||||
Data type comprised of x-y coordinate representations of locations on the earth that take the form of single points, strings of points (lines or arcs) or closed lines (polygons) known as features. Each feature has an associated attribute code for identification. Data can be converted to raster data through a process know as rasterization.
|
||||
|
||||
**WGS-84** -
|
||||
A set of parameters, established by the U.S. Defense Mapping Agency, for determining geometric and physical geodetic relationships on a global scale. The system includes a geocentric reference ellipsoid; a coordinate system; and a gravity field model.
|
||||
|
||||
**zoom** -
|
||||
To enlarge and display greater detail of a portion of a geographic data set.
|
||||
|
||||
### References:
|
||||
|
||||
- [Cartographic and GIS Technical Note (Glossary of Common GIS and GPS Terms)](https://www.nrcs.usda.gov/Internet/FSE_DOCUMENTS/nrcs144p2_051844.pdf)
|
||||
|
|
@ -0,0 +1,173 @@
|
|||
# Pipeline Activities
|
||||
|
||||
The purpose of this document is to provide a detailed walkthrough of the Synapse pipeline including the child pipelines, activities and their parameters. As we discuss each pipeline, their corresponding pipeline and supporting files checked in this repository will be linked under each topic.
|
||||
|
||||
|
||||
## Pipeline Folder Structure
|
||||
|
||||
All supporting files for the pipeline are checked in to the [Workflow](../src/workflow) folder. Below is the folder structure.
|
||||
|
||||
|
||||
|
||||
The folder structure is defined by Azure Synapse Analytics service when exporting your Synapse pipeline to a ZIP file or checking them to a Github/Azure DevOps repository. A quick overview of each folder is in the table below:
|
||||
|
||||
No | Folder | Purpose
|
||||
---|------|----------
|
||||
1 | .package | An internal folder for Synapse pipeline to hold package manifest files
|
||||
2 | bigDataPool | Contains JSON definition files to create Spark pool
|
||||
3 | dataflow | Contains JSON definition files to create dataflows in Synapse Workspace
|
||||
4 | dataset | Contains JSON definition files to create datasets in Synapse Workspace
|
||||
5 | linkedService | Contains JSON definition files to create Linked Services in Synapse Workspace
|
||||
6 | notebook | Contains JSON definition files to create Notebooks in Synapse Workspace
|
||||
7 | pipeline | Contains JSON definition files to create pipelines in Synapse Workspace
|
||||
8 | sparkJobDefinition | Contains JSON definition files to create Spark Job definitions in Synapse Workspace
|
||||
9 | publish_config.json | Contains configuration details such as the branch in your repository to use as pubish branch
|
||||
|
||||
**Note:** Individual files and folders in the [Workflow](../src/workflow) parent folder are not intended to be imported into your workspace individually. You will need to run the `package.sh` in the deploy folder to create the zip file and then use the contents of the zip files to check-in your code to a repository.
|
||||
|
||||
## Main Pipeline
|
||||
|
||||
|
||||
|
||||
Two variants of the pipeline are available.
|
||||
|
||||
a. First variant of the [pipeline](../src/workflow/custom-vision-model-v2/pipeline/E2E%20Custom%20Vision%20Model%20Flow.json) uses [Azure Batch Account]() to run AI Model as a container to process the raster data and provide the output.
|
||||
b. Second variant of the [pipeline](../src/workflow/custom-vision-model-v2/pipeline/E2E%20Custom%20Vision%20Model%20Flow_aks.json) uses [Azure Kubernetes Service (AKS)]() to run AI Model as a container to process the raster data and provide the output.
|
||||
|
||||
You have the option to pick which Azure service will be used to run the AI Model at the time of deployment. Refer to the [README.md](../deploy/README.md) document to understand the deployment steps and options available during deployment.
|
||||
|
||||
The main pipeline contains two steps.
|
||||
|
||||
- It starts by invoking the [Custom Vision Object Transform v2](../src/workflow/custom-vision-model-v2/pipeline/Custom%20Vision%20Model%20Transforms%20v2.json) pipeline to transform your data before it can be consumed by the AI Model.
|
||||
|
||||
- Next, the [Custom Vision Model Transform v2](../src/workflow/custom-vision-model-v2/pipeline/Custom%20Vision%20Object%20Detection%20v2.json) or [Custom Vision Model Transform v2_aks](../src/workflow/custom-vision-model-v2/pipeline/Custom%20Vision%20Object%20Detection%20v2_aks.json) pipeline is triggered to process the data through the AI Model running as container in either Azure Batch Account or Azure Kubernetes Services respectively.
|
||||
|
||||
For Custom Vision model, the output is a set of JSON files that contains the location of the swimming pools detected in the image passed to the AI Model as 512 by 512 images.
|
||||
|
||||
The final steps is to process the location of the detected swimming pools into geolocation (latitude, longitude). This will be explained in detail later in this document as we dive deeper into each child pipeline.
|
||||
|
||||
|
||||
## Custom Vision Model Transform
|
||||
|
||||
|
||||
|
||||
This is a simple, one-activity pipeline that serves one purpose. It transforms the raw raster geotiff (Aerial Imagery) and outputs a set of 512 by 512 tiles of PNG file format.
|
||||
|
||||
To convert the raw raster geotiff, it first stitches the geotiff (in case there are more than one geotiff to be processed), crops the geotiff to the Area of Interest, converts geotiff to PNG and finally tiles them to 512 by 512 sizes.
|
||||
|
||||
Below are the parameters for this pipeline:
|
||||
|
||||
No | Parameter | Purpose
|
||||
---|------|----------
|
||||
1 | Prefix | This refers to the Storage Account Container created for the specific pipeline run. Refer to the [README.md](../deploy/README.md) document to understand the steps to run the pipeline.
|
||||
2 | StorageAccountName | Name of the Storage Account where the data is stored
|
||||
3 | AOI | Area of Interest refers to specific geographic extent to focus the analysis
|
||||
|
||||
Inside the pipeline, there is one activity that submits a Spark Job to the Azure Synapse Spark Pool provisioned by the Bicep template. The Spark Job takes the below parameters as input:
|
||||
|
||||
No | Parameter | Purpose
|
||||
---|------|----------
|
||||
1 | storage_account_name | Name of the Storage Account where the data is stored
|
||||
2 | storage_container | Name of the Container in the Storage Account where the data is stored
|
||||
3 | key_vault_name | Name of the Key Vault where the secrets related to the pipeline are stored. There is only one Key Vault provisioned through the Bicep template. This is the Key Vault that will be used to store for all pipeline related secrets
|
||||
4 | storage_account_key_secret_name | Access Key for the Storage Account is stored as a secret in Key Vault. This parameter is used to pass the name of the secret in the Key Vault that holds the access key
|
||||
5 | linked_service_name | Synapse allows adding certain services like Key Vault, Storage Account, etc as Linked Service. This parameters is used to pass the name of the Key Vault added to the Synapse instance as linked service.
|
||||
6 | aoi | Area of Interest refers to specific geographic extent to focus the analysis
|
||||
|
||||
|
||||
|
||||
|
||||
## Custom Vision Model Detection (Using Azure Batch Account)
|
||||
|
||||
This is the core of the pipeline where the actual AI Model is run against the data submitted to the pipeline. The steps prior to this pipeline, prepares the data to be in a format that is consumable by the AI Model.
|
||||
|
||||
The AI Model is packaged as a [Container Image](https://github.com/Azure/Azure-Orbital-Analytics-Samples/pkgs/container/azure-orbital-analytics-samples%2Fcustom_vision_offline) that is publicly available to be pulled and run locally on your Azure Batch Account or Azure Kubernetes Service.
|
||||
|
||||
Below are the parameters for this pipeline:
|
||||
|
||||
No | Parameter | Purpose
|
||||
---|------|----------
|
||||
1 | Prefix | This refers to the Storage Account Container created for the specific pipeline run. Refer to the [README.md](../deploy/README.md) document to understand the steps to run the pipeline.
|
||||
2 | BatchName | Name of the Azure Batch Account where the AI Model container will be run
|
||||
3 | JobName | Name of the Job inside the Azure Batch Account where the AI Model container will be run
|
||||
4 | BatchLocation | Location/Region where the Batch Account is provisioned. For Example - for East US region, use **eastus**
|
||||
5 | StorageAccountName | Name of the Storage Account where the data is stored
|
||||
|
||||
|
||||
|
||||
**1. Read Spec Document**
|
||||
|
||||
As the name suggests, this activity reads the spec (specification) document.
|
||||
|
||||
A **specification document** is a JSON file that defines the parameters for running the AI Model container. It includes location of the image, credentials to pull the image, path to look for input file, output file path, log file path, CPU, GPU and memory among other parameters that define the minimal runtime environment.
|
||||
|
||||
Data read from the spec document will be used as parameter to one or more of the activities in the later part of the pipeline.
|
||||
|
||||
No | Parameter | Purpose
|
||||
---|------|----------
|
||||
1 | filename | Name of JSON specification document
|
||||
2 | folderpath | Name of the folder that contains the JSON specification document. This is relative folder path (relative to the container in Azure Storage Account)
|
||||
3 | containername | Name of the container that contains the JSON specification document
|
||||
|
||||
**2. Copy and Delete File(s)**
|
||||
|
||||
Data for the AI Model is in a Container in Azure Storage Account. For the AI Model, the data needs to be copied to a File Share in Azure Storage Account. This set of steps serves to copy the input files from the Containers to File Shares in the Storage Account.
|
||||
|
||||
There is also a need to create empty Folders like output folder and log folder what will be used by the AI Model to store the outputs and logs after running the AI Model. However, since Azure Data Factory (ADF) activities do not allow creation of empty folders, we create placeholder files under the folder and then delete the placeholder file to create the empty folder.
|
||||
|
||||
No | Parameter | Purpose
|
||||
---|------|----------
|
||||
1 | container_name | Name of the destination container in Azure Storage Account where the files are to be copied
|
||||
2 | folder_path | Name of the destination folder in Azure Storage Account where the files are to be copied
|
||||
|
||||
**3. Custom Vision kick off**
|
||||
|
||||
During this step, the container is scheduled in Azure Batch Account to run as a task. This is done by calling the Azure Batch Account API and sending the parameters required.
|
||||
|
||||
**Note:** There is no specific custom parameters defined for this activity.
|
||||
|
||||
**4. Wait for Custom Vision**
|
||||
|
||||
The previous step kicks off the task in Azure Batch Account and exits. AI Model takes several minutes to hours to complete depending on the input file sizes. This steps monitors the AI Model running in the Azure Batch Account to check if they are complete. This set of activities are set to run in a loop and exit only if the AI Model completes.
|
||||
|
||||
**Note:** There is no specific custom parameters defined for this activity.
|
||||
|
||||
**5. & 6. Copy File(s)**
|
||||
|
||||
At this stage of the pipeline, the AI Model is run and the output is available in File Share in Azure Storage Account. It needs to be copied from File Share to Container in Azure Storage Account. This step copies the output in JSON and XML format to the container in Azure Storage Account.
|
||||
|
||||
**Note:** There is no specific custom parameters defined for this activity.
|
||||
|
||||
**7. Pool Geolocation**
|
||||
|
||||
The output from running the AI Model (Custom Vision Model) gives the location of the Swimming pool in the tiles in terms of pixel location. This step converts the pixel location of the detected objects to the geolocation (latitude, longitude).
|
||||
|
||||
No | Parameter | Purpose
|
||||
---|------|----------
|
||||
1 | storage_account_name | Name of the Storage Account where the input data is stored
|
||||
2 | storage_container | Name of the container in Storage Account where the input data is stored
|
||||
3 | src_folder_name | Name of the folder under the container in Storage Account where the input data is stored
|
||||
4 | key_vault_name | Name of the Key Vault to use for pulling the secrets
|
||||
5 | storage_account_key_secret_name | Name of the secret in Key Vault that stores the Access Key of Storage Account
|
||||
6 | linked_service_name | Name of the Linked Service in Synapse which links the Key Vaults to Synapse
|
||||
|
||||
|
||||
## Custom Vision Model Detection (Using Azure Kubernetes Service)
|
||||
|
||||
This pipeline performs the same functionality as the **Custom Vision Model Detection (Using Azure Batch Account)**. However, it uses Azure Kubernetes Service in place of Azure Batch Account to run the AI Model.
|
||||
|
||||
The pipeline remains the same as the **Custom Vision Model Detection (Using Azure Batch Account)**. There are two activities that are changes to accomodate the switch from Azure Batch Account to Azure Kubernetes Service. **We'll discuss only the components that are different in this pipeline.**
|
||||
|
||||
|
||||
|
||||
**3. Invoke Workload in AKS**
|
||||
|
||||
During this step, the container is scheduled in Azure Kubernetes Service to run as a job. This is done by calling the Azure Kubernetes Service API and sending the parameters required.
|
||||
|
||||
**Note:** There is no specific custom parameters defined for this activity.
|
||||
|
||||
**4. Wait for Custom Vision**
|
||||
|
||||
The previous step kicks off the task in Azure Kubernetes Service and exits. AI Model takes several minutes to hours to complete depending on the input file sizes. This steps monitors the AI Model running in the Azure Kubernetes Service to check if they are complete. This set of activities are set to run in a loop and exit only if the AI Model completes.
|
||||
|
||||
**Note:** There is no specific custom parameters defined for this activity.
|
||||
|
|
@ -0,0 +1,16 @@
|
|||
# Support
|
||||
|
||||
|
||||
The solution and samples provided in this repository is for learning purpose only. They're intended to explore the possibilites of the Azure Services and is a starting point to developing your own solution using Azure Synapse Analytics. We recommend that you follow the security best practices as per Microsoft documentation for individual services.
|
||||
|
||||
|
||||
## How to file issues and get help
|
||||
|
||||
|
||||
This project uses GitHub Issues to track bugs and feature requests. Please [search](https://github.com/Azure/Azure-Orbital-Analytics-Samples/issues) the existing issues before filing new issues to avoid duplicates. For new issues, file your bug or feature request as [a new issue](https://github.com/Azure/Azure-Orbital-Analytics-Samples/issues/new). The project maintainers will respond to the best of their abilities, and the response time is not guaranteed.
|
||||
|
||||
|
||||
## Microsoft Support Policy
|
||||
|
||||
|
||||
Support for this project is limited to the resources listed above. This project is **not** covered by the [Microsoft Azure support policy](https://azure.microsoft.com/en-us/support/).
|
||||
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 155 KiB |
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 146 KiB |
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 38 KiB |
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 14 KiB |
Двоичный файл не отображается.
|
После Ширина: | Высота: | Размер: 45 KiB |
|
|
@ -1,20 +0,0 @@
|
|||
{
|
||||
"ai_model_infra_type": {
|
||||
"batch-account" : {
|
||||
"exclude" : [
|
||||
"../pipeline/Custom Vision Object Detection v2_aks.json",
|
||||
"../pipeline/E2E Custom Vision Model Flow_aks.json",
|
||||
"../linkedService/GenBase64Encoding.json"
|
||||
]
|
||||
},
|
||||
"aks": {
|
||||
"exclude" : [
|
||||
"../pipeline/Custom Vision Object Detection v2.json",
|
||||
"../pipeline/E2E Custom Vision Model Flow.json"
|
||||
],
|
||||
"rename" : {
|
||||
"../pipeline/E2E Custom Vision Model Flow_aks.json": "../pipeline/E2E Custom Vision Model Flow.json"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
{
|
||||
"name": "__synapse_pool_name__",
|
||||
"location": "__location__",
|
||||
"properties": {
|
||||
"autoPause": {
|
||||
"enabled": true,
|
||||
"delayInMinutes": 15
|
||||
},
|
||||
"autoScale": {
|
||||
"enabled": true,
|
||||
"maxNodeCount": 4,
|
||||
"minNodeCount": 3
|
||||
},
|
||||
"nodeCount": 0,
|
||||
"nodeSize": "Medium",
|
||||
"nodeSizeFamily": "MemoryOptimized",
|
||||
"sparkVersion": "3.1",
|
||||
"libraryRequirements": {
|
||||
"content": "name: aoidemo\r\nchannels:\r\n - conda-forge\r\n - defaults\r\ndependencies:\r\n - gdal=3.3.0\r\n - pip>=20.1.1\r\n - azure-storage-file-datalake\r\n - libgdal\r\n - shapely\r\n - pyproj\r\n - pip:\r\n - rasterio\r\n - geopandas",
|
||||
"filename": "environment.yml",
|
||||
"time": "2022-02-22T00:52:46.8995063Z"
|
||||
},
|
||||
"isComputeIsolationEnabled": false,
|
||||
"sparkConfigProperties": {
|
||||
"configurationType": "File",
|
||||
"filename": "config.txt",
|
||||
"content": "spark.storage.synapse.linkedServiceName \"AOI Geospatial v2\"\rfs.azure.account.oauth.provider.type com.microsoft.azure.synapse.tokenlibrary.LinkedServiceBasedTokenProvider",
|
||||
"time": "2022-02-22T00:52:46.8995063Z"
|
||||
},
|
||||
"sessionLevelPackagesEnabled": true,
|
||||
"annotations": []
|
||||
}
|
||||
}
|
||||
|
|
@ -1,12 +0,0 @@
|
|||
source(allowSchemaDrift: true,
|
||||
validateSchema: false,
|
||||
ignoreNoFilesFound: false,
|
||||
documentForm: 'arrayOfDocuments') ~> source
|
||||
source sink(allowSchemaDrift: true,
|
||||
validateSchema: false,
|
||||
skipDuplicateMapInputs: true,
|
||||
skipDuplicateMapOutputs: true,
|
||||
store: 'cache',
|
||||
format: 'inline',
|
||||
output: true,
|
||||
saveOrder: 1) ~> sink
|
||||
|
|
@ -1,37 +0,0 @@
|
|||
{
|
||||
"name": "ReadSpecDocumentFlow",
|
||||
"properties": {
|
||||
"type": "MappingDataFlow",
|
||||
"typeProperties": {
|
||||
"sources": [
|
||||
{
|
||||
"dataset": {
|
||||
"referenceName": "spec_doc_specification",
|
||||
"type": "DatasetReference"
|
||||
},
|
||||
"name": "source"
|
||||
}
|
||||
],
|
||||
"sinks": [
|
||||
{
|
||||
"name": "sink"
|
||||
}
|
||||
],
|
||||
"transformations": [],
|
||||
"scriptLines": [
|
||||
"source(allowSchemaDrift: true,",
|
||||
" validateSchema: false,",
|
||||
" ignoreNoFilesFound: false,",
|
||||
" documentForm: 'arrayOfDocuments') ~> source",
|
||||
"source sink(allowSchemaDrift: true,",
|
||||
" validateSchema: false,",
|
||||
" skipDuplicateMapInputs: true,",
|
||||
" skipDuplicateMapOutputs: true,",
|
||||
" store: 'cache',",
|
||||
" format: 'inline',",
|
||||
" output: true,",
|
||||
" saveOrder: 1) ~> sink"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
{
|
||||
"name": "gld",
|
||||
"properties": {
|
||||
"linkedServiceName": {
|
||||
"referenceName": "AOI Geospatial v2",
|
||||
"type": "LinkedServiceReference"
|
||||
},
|
||||
"parameters": {
|
||||
"DestinationFolderPath": {
|
||||
"type": "string"
|
||||
},
|
||||
"DestinationContainerName": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"annotations": [],
|
||||
"type": "Binary",
|
||||
"typeProperties": {
|
||||
"location": {
|
||||
"type": "AzureBlobFSLocation",
|
||||
"folderPath": {
|
||||
"value": "@dataset().DestinationFolderPath",
|
||||
"type": "Expression"
|
||||
},
|
||||
"fileSystem": {
|
||||
"value": "@dataset().DestinationContainerName",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/datasets"
|
||||
}
|
||||
|
|
@ -1,17 +0,0 @@
|
|||
{
|
||||
"name": "gls",
|
||||
"properties": {
|
||||
"linkedServiceName": {
|
||||
"referenceName": "AOI Geospatial v2 FS",
|
||||
"type": "LinkedServiceReference"
|
||||
},
|
||||
"annotations": [],
|
||||
"type": "Binary",
|
||||
"typeProperties": {
|
||||
"location": {
|
||||
"type": "AzureFileStorageLocation"
|
||||
}
|
||||
}
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/datasets"
|
||||
}
|
||||
|
|
@ -1,33 +0,0 @@
|
|||
{
|
||||
"name": "rawtifs",
|
||||
"properties": {
|
||||
"linkedServiceName": {
|
||||
"referenceName": "AOI Data Storage Account v2",
|
||||
"type": "LinkedServiceReference"
|
||||
},
|
||||
"parameters": {
|
||||
"containername": {
|
||||
"type": "string"
|
||||
},
|
||||
"folderpath": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"annotations": [],
|
||||
"type": "Binary",
|
||||
"typeProperties": {
|
||||
"location": {
|
||||
"type": "AzureBlobStorageLocation",
|
||||
"folderPath": {
|
||||
"value": "@dataset().folderpath",
|
||||
"type": "Expression"
|
||||
},
|
||||
"container": {
|
||||
"value": "@dataset().containername",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/datasets"
|
||||
}
|
||||
|
|
@ -1,41 +0,0 @@
|
|||
{
|
||||
"name": "spec_doc_specification",
|
||||
"properties": {
|
||||
"linkedServiceName": {
|
||||
"referenceName": "AOI Data Storage Account v2",
|
||||
"type": "LinkedServiceReference"
|
||||
},
|
||||
"parameters": {
|
||||
"filename": {
|
||||
"type": "string"
|
||||
},
|
||||
"folderpath": {
|
||||
"type": "string"
|
||||
},
|
||||
"containername": {
|
||||
"type": "string"
|
||||
}
|
||||
},
|
||||
"annotations": [],
|
||||
"type": "Json",
|
||||
"typeProperties": {
|
||||
"location": {
|
||||
"type": "AzureBlobStorageLocation",
|
||||
"fileName": {
|
||||
"value": "@dataset().filename",
|
||||
"type": "Expression"
|
||||
},
|
||||
"folderPath": {
|
||||
"value": "@dataset().folderpath",
|
||||
"type": "Expression"
|
||||
},
|
||||
"container": {
|
||||
"value": "@dataset().containername",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
},
|
||||
"schema": {}
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/datasets"
|
||||
}
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
{
|
||||
"name": "AOI Batch Storage",
|
||||
"properties": {
|
||||
"annotations": [],
|
||||
"type": "AzureBlobStorage",
|
||||
"typeProperties": {
|
||||
"connectionString": "DefaultEndpointsProtocol=https;AccountName=__batch_storage_account__;EndpointSuffix=core.windows.net;",
|
||||
"accountKey": {
|
||||
"type": "AzureKeyVaultSecret",
|
||||
"store": {
|
||||
"referenceName": "AOI Pipeline Key Vault",
|
||||
"type": "LinkedServiceReference"
|
||||
},
|
||||
"secretName": "PackageStorageAccountKey"
|
||||
}
|
||||
},
|
||||
"connectVia": {
|
||||
"referenceName": "AutoResolveIntegrationRuntime",
|
||||
"type": "IntegrationRuntimeReference"
|
||||
}
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/linkedservices"
|
||||
}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
{
|
||||
"name": "AOI Batch",
|
||||
"properties": {
|
||||
"annotations": [],
|
||||
"type": "AzureBatch",
|
||||
"typeProperties": {
|
||||
"batchUri": "https://__batch_account__.__location__.batch.azure.com",
|
||||
"poolName": "__batch_pool_name__",
|
||||
"accountName": "__batch_account__",
|
||||
"linkedServiceName": {
|
||||
"referenceName": "AOI Batch Storage",
|
||||
"type": "LinkedServiceReference"
|
||||
},
|
||||
"accessKey": {
|
||||
"type": "AzureKeyVaultSecret",
|
||||
"store": {
|
||||
"referenceName": "AOI Pipeline Key Vault",
|
||||
"type": "LinkedServiceReference"
|
||||
},
|
||||
"secretName": "GeospatialBatchAccountKey"
|
||||
}
|
||||
},
|
||||
"connectVia": {
|
||||
"referenceName": "AutoResolveIntegrationRuntime",
|
||||
"type": "IntegrationRuntimeReference"
|
||||
}
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/linkedservices"
|
||||
}
|
||||
|
|
@ -1,23 +0,0 @@
|
|||
{
|
||||
"name": "AOI Data Storage Account v2",
|
||||
"properties": {
|
||||
"annotations": [],
|
||||
"type": "AzureBlobStorage",
|
||||
"typeProperties": {
|
||||
"connectionString": "DefaultEndpointsProtocol=https;AccountName=__raw_data_storage_account__;EndpointSuffix=core.windows.net;",
|
||||
"accountKey": {
|
||||
"type": "AzureKeyVaultSecret",
|
||||
"store": {
|
||||
"referenceName": "AOI Pipeline Key Vault",
|
||||
"type": "LinkedServiceReference"
|
||||
},
|
||||
"secretName": "GeospatialStorageAccountKey"
|
||||
}
|
||||
},
|
||||
"connectVia": {
|
||||
"referenceName": "AutoResolveIntegrationRuntime",
|
||||
"type": "IntegrationRuntimeReference"
|
||||
}
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/linkedservices"
|
||||
}
|
||||
|
|
@ -1,24 +0,0 @@
|
|||
{
|
||||
"name": "AOI Geospatial v2 FS",
|
||||
"properties": {
|
||||
"annotations": [],
|
||||
"type": "AzureFileStorage",
|
||||
"typeProperties": {
|
||||
"connectionString": "DefaultEndpointsProtocol=https;AccountName=__raw_data_storage_account__;EndpointSuffix=core.windows.net;",
|
||||
"accountKey": {
|
||||
"type": "AzureKeyVaultSecret",
|
||||
"store": {
|
||||
"referenceName": "AOI Pipeline Key Vault",
|
||||
"type": "LinkedServiceReference"
|
||||
},
|
||||
"secretName": "GeospatialStorageAccountKey"
|
||||
},
|
||||
"fileShare": "volume-a"
|
||||
},
|
||||
"connectVia": {
|
||||
"referenceName": "AutoResolveIntegrationRuntime",
|
||||
"type": "IntegrationRuntimeReference"
|
||||
}
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/linkedservices"
|
||||
}
|
||||
|
|
@ -1,15 +0,0 @@
|
|||
{
|
||||
"name": "AOI Geospatial v2",
|
||||
"type": "Microsoft.Synapse/workspaces/linkedservices",
|
||||
"properties": {
|
||||
"annotations": [],
|
||||
"type": "AzureBlobFS",
|
||||
"typeProperties": {
|
||||
"url": "https://__raw_data_storage_account__.dfs.core.windows.net"
|
||||
},
|
||||
"connectVia": {
|
||||
"referenceName": "AutoResolveIntegrationRuntime",
|
||||
"type": "IntegrationRuntimeReference"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,11 +0,0 @@
|
|||
{
|
||||
"name": "AOI Pipeline Key Vault",
|
||||
"type": "Microsoft.Synapse/workspaces/linkedservices",
|
||||
"properties": {
|
||||
"annotations": [],
|
||||
"type": "AzureKeyVault",
|
||||
"typeProperties": {
|
||||
"baseUrl": "https://__linked_key_vault__.vault.azure.net/"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,20 +0,0 @@
|
|||
{
|
||||
"name": "GenBase64Encoding",
|
||||
"properties": {
|
||||
"annotations": [],
|
||||
"type": "AzureFunction",
|
||||
"typeProperties": {
|
||||
"functionAppUrl": "__base64encodedzipcontent_functionapp_url__",
|
||||
"functionKey": {
|
||||
"type": "AzureKeyVaultSecret",
|
||||
"store": {
|
||||
"referenceName": "AOI Pipeline Key Vault",
|
||||
"type": "LinkedServiceReference"
|
||||
},
|
||||
"secretName": "GenBase64EncondingFunctionKey"
|
||||
},
|
||||
"authentication": "Anonymous"
|
||||
}
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/linkedservices"
|
||||
}
|
||||
|
|
@ -1,377 +0,0 @@
|
|||
{
|
||||
"name": "Custom Vision Model Transforms v2",
|
||||
"properties": {
|
||||
"activities": [
|
||||
{
|
||||
"name": "GetFilesToMosaic",
|
||||
"type": "GetMetadata",
|
||||
"dependsOn": [],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"dataset": {
|
||||
"referenceName": "rawtifs",
|
||||
"type": "DatasetReference",
|
||||
"parameters": {
|
||||
"containername": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
},
|
||||
"folderpath": "raw"
|
||||
}
|
||||
},
|
||||
"fieldList": [
|
||||
"childItems"
|
||||
],
|
||||
"storeSettings": {
|
||||
"type": "AzureBlobStorageReadSettings",
|
||||
"recursive": true,
|
||||
"enablePartitionDiscovery": false
|
||||
},
|
||||
"formatSettings": {
|
||||
"type": "BinaryReadSettings"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Crop",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "More than one GeoTiff",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Crop",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/raster_crop/src/crop.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--storage_account_key",
|
||||
"@pipeline().parameters.StorageAccountKey",
|
||||
"--storage_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder_name",
|
||||
"@variables('CropSourceFolder')",
|
||||
"--config_file_name",
|
||||
"config-aoi.json"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Convert",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Crop",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Convert",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/raster_convert/src/convert.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--storage_account_key",
|
||||
"@pipeline().parameters.StorageAccountKey",
|
||||
"--storage_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder_name",
|
||||
"crop",
|
||||
"--config_file_name",
|
||||
"config-img-convert-png.json"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Tiling",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Convert",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Tiling",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/raster_tiling/src/tiling.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--storage_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder_name",
|
||||
"convert",
|
||||
"--file_name",
|
||||
"output.png",
|
||||
"--tile_size",
|
||||
"512",
|
||||
"--storage_account_key",
|
||||
"@pipeline().parameters.StorageAccountKey"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "More than one GeoTiff",
|
||||
"type": "IfCondition",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "For Each File to Mosaic",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"expression": {
|
||||
"value": "@greater(length(activity('GetFilesToMosaic').output.childItems),1)",
|
||||
"type": "Expression"
|
||||
},
|
||||
"ifFalseActivities": [
|
||||
{
|
||||
"name": "Set Crop Source Folder to raw",
|
||||
"type": "SetVariable",
|
||||
"dependsOn": [],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"variableName": "CropSourceFolder",
|
||||
"value": "raw"
|
||||
}
|
||||
}
|
||||
],
|
||||
"ifTrueActivities": [
|
||||
{
|
||||
"name": "Mosaic",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Mosaic",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/raster_mosaic/src/mosaic.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--storage_account_key",
|
||||
"@pipeline().parameters.StorageAccountKey",
|
||||
"--storage_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder_name",
|
||||
"raw"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Set Crop Source Folder to mosaic",
|
||||
"type": "SetVariable",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Mosaic",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"variableName": "CropSourceFolder",
|
||||
"value": "mosaic"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "For Each File to Mosaic",
|
||||
"type": "ForEach",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "GetFilesToMosaic",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"items": {
|
||||
"value": "@activity('GetFilesToMosaic').output.childItems",
|
||||
"type": "Expression"
|
||||
},
|
||||
"isSequential": true,
|
||||
"activities": [
|
||||
{
|
||||
"name": "Set Mosaic File Names",
|
||||
"type": "SetVariable",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Store Temp Mosaic File Names",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"variableName": "MosaicFileNames",
|
||||
"value": {
|
||||
"value": "@concat(variables('TempMosaicFileNames'), if(equals(variables('TempMosaicFileNames'), ''),'',','), item().name)",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Store Temp Mosaic File Names",
|
||||
"type": "SetVariable",
|
||||
"dependsOn": [],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"variableName": "TempMosaicFileNames",
|
||||
"value": {
|
||||
"value": "@variables('MosaicFileNames')",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Prefix": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"StorageAccountName": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"StorageAccountKey": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
"FunctionCompleted": {
|
||||
"type": "String",
|
||||
"defaultValue": "None"
|
||||
},
|
||||
"FunctionError": {
|
||||
"type": "String"
|
||||
},
|
||||
"MosaicFileNames": {
|
||||
"type": "String"
|
||||
},
|
||||
"TempMosaicFileNames": {
|
||||
"type": "String"
|
||||
},
|
||||
"CropSourceFolder": {
|
||||
"type": "String"
|
||||
}
|
||||
},
|
||||
"annotations": [],
|
||||
"lastPublishTime": "2022-03-06T06:06:58Z"
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/pipelines"
|
||||
}
|
||||
|
|
@ -1,597 +0,0 @@
|
|||
{
|
||||
"name": "Custom Vision Object Detection v2",
|
||||
"properties": {
|
||||
"activities": [
|
||||
{
|
||||
"name": "Wait for Custom Vision",
|
||||
"type": "Until",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Custom Vision",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"expression": {
|
||||
"value": "@and(and(not(equals(string(variables('FunctionCompleted')), 'running')), not(equals(string(variables('FunctionCompleted')), 'active'))), not(equals(string(variables('FunctionCompleted')), 'preparing')))",
|
||||
"type": "Expression"
|
||||
},
|
||||
"activities": [
|
||||
{
|
||||
"name": "Wait for Custom Vision Check",
|
||||
"type": "Wait",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Set FunctionCompleted Custom Vision",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"waitTimeInSeconds": 30
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Check Status Custom Vision",
|
||||
"type": "WebActivity",
|
||||
"dependsOn": [],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"url": {
|
||||
"value": "@concat('https://',pipeline().parameters.BatchName,'.',pipeline().parameters.BatchLocation,'.batch.azure.com/jobs/',pipeline().parameters.JobName,'/tasks/aoi-cv-task-', pipeline().RunId, '?api-version=2022-01-01.15.0')",
|
||||
"type": "Expression"
|
||||
},
|
||||
"connectVia": {
|
||||
"referenceName": "AutoResolveIntegrationRuntime",
|
||||
"type": "IntegrationRuntimeReference"
|
||||
},
|
||||
"method": "GET",
|
||||
"authentication": {
|
||||
"type": "MSI",
|
||||
"resource": "https://batch.core.windows.net/"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Set FunctionCompleted Custom Vision",
|
||||
"type": "SetVariable",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Check Status Custom Vision",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"variableName": "FunctionCompleted",
|
||||
"value": {
|
||||
"value": "@activity('Check Status Custom Vision').output['state']",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Set FunctionError",
|
||||
"type": "SetVariable",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Check Status Custom Vision",
|
||||
"dependencyConditions": [
|
||||
"Failed"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"variableName": "FunctionError",
|
||||
"value": {
|
||||
"value": "@activity('Check Status Custom Vision').output['executionInfo']['failureInfo']",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"timeout": "7.00:00:00"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Custom Vision",
|
||||
"type": "WebActivity",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Copy Config file",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
},
|
||||
{
|
||||
"activity": "Copy Xml From Convert Transform",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"url": {
|
||||
"value": "@concat('https://',pipeline().parameters.BatchName,'.',pipeline().parameters.BatchLocation,'.batch.azure.com/jobs/',pipeline().parameters.JobName,'/tasks?api-version=2020-03-01.11.0')",
|
||||
"type": "Expression"
|
||||
},
|
||||
"connectVia": {
|
||||
"referenceName": "AutoResolveIntegrationRuntime",
|
||||
"type": "IntegrationRuntimeReference"
|
||||
},
|
||||
"method": "POST",
|
||||
"headers": {
|
||||
"Content-type": "application/json; odata=minimalmetadata; charset=utf-8"
|
||||
},
|
||||
"body": {
|
||||
"value": "@json(concat('{\n \"id\": \"aoi-cv-task-', pipeline().RunId, '\",\n \"commandLine\": \"\",\n \"containerSettings\": {\n \"imageName\": \"', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['algImageName'], '\",\n \"containerRunOptions\": \"--rm --workdir / -v /mnt/batch/tasks/fsmounts/S/', pipeline().parameters.Prefix, ':', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['mountedDirectory'], '\"\n },\n \"environmentSettings\": [\n {\n \"name\": \"APP_INPUT_DIR\",\n \"value\": \"/mnt/batch/tasks/fsmounts/S/', pipeline().parameters.Prefix,'/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['submissionDirectory'],'\"\n },\n {\n \"name\": \"APP_OUTPUT_DIR\",\n \"value\": \"/mnt/batch/tasks/fsmounts/S/', pipeline().parameters.Prefix,'/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['resultsDirectory'],'\"\n },\n {\n \"name\": \"APP_CONFIG_DIR\",\n \"value\": \"/mnt/batch/tasks/fsmounts/S/', pipeline().parameters.Prefix,'/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['contextFileName'],'\"\n }\n ]\n}'))",
|
||||
"type": "Expression"
|
||||
},
|
||||
"authentication": {
|
||||
"type": "MSI",
|
||||
"resource": "https://batch.core.windows.net/"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Pool Geolocation",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Copy Xml",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
},
|
||||
{
|
||||
"activity": "Copy Json",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Pool Geolocation",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/pool_geolocation/src/main.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--storage_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder_name",
|
||||
"detections",
|
||||
"--key_vault_name",
|
||||
"__linked_key_vault__",
|
||||
"--storage_account_key_secret_name",
|
||||
"GeospatialStorageAccountKey",
|
||||
"--linked_service_name",
|
||||
"AOI Pipeline Key Vault",
|
||||
"--config_file_name",
|
||||
"config-pool-geolocation.json"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Copy Tiles",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Read Spec Document",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Copy noop",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/copy_noop/src/main.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--src_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder",
|
||||
"tiles",
|
||||
"--key_vault_name",
|
||||
"__linked_key_vault__",
|
||||
"--storage_account_key_secret_name",
|
||||
"GeospatialStorageAccountKey",
|
||||
"--linked_service_name",
|
||||
"AOI Pipeline Key Vault",
|
||||
"--dst_fileshare",
|
||||
"volume-a",
|
||||
"--dst_folder",
|
||||
"@concat(pipeline().parameters.Prefix,'/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['submissionDirectory'])",
|
||||
"--folders_to_create",
|
||||
"@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['submissionDirectory'])",
|
||||
"--folders_to_create",
|
||||
"@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['resultsDirectory'])",
|
||||
"--folders_to_create",
|
||||
"@concat(pipeline().parameters.Prefix,'/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['logsDirectory'])"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Copy Config file",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Copy Tiles",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Copy noop",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/copy_noop/src/main.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--src_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder",
|
||||
"@concat('config/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['contextFileName'])",
|
||||
"--key_vault_name",
|
||||
"__linked_key_vault__",
|
||||
"--storage_account_key_secret_name",
|
||||
"GeospatialStorageAccountKey",
|
||||
"--linked_service_name",
|
||||
"AOI Pipeline Key Vault",
|
||||
"--dst_fileshare",
|
||||
"volume-a",
|
||||
"--dst_folder",
|
||||
"@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['contextFileName'])"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Copy Json",
|
||||
"type": "Copy",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Wait for Custom Vision",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"source": {
|
||||
"type": "BinarySource",
|
||||
"storeSettings": {
|
||||
"type": "AzureFileStorageReadSettings",
|
||||
"recursive": true,
|
||||
"wildcardFolderPath": {
|
||||
"value": "@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['resultsDirectory'], '/json')",
|
||||
"type": "Expression"
|
||||
},
|
||||
"wildcardFileName": "*.json",
|
||||
"deleteFilesAfterCompletion": false
|
||||
},
|
||||
"formatSettings": {
|
||||
"type": "BinaryReadSettings"
|
||||
}
|
||||
},
|
||||
"sink": {
|
||||
"type": "BinarySink",
|
||||
"storeSettings": {
|
||||
"type": "AzureBlobFSWriteSettings"
|
||||
}
|
||||
},
|
||||
"enableStaging": false
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"referenceName": "gls",
|
||||
"type": "DatasetReference"
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"referenceName": "gld",
|
||||
"type": "DatasetReference",
|
||||
"parameters": {
|
||||
"DestinationFolderPath": "detections",
|
||||
"DestinationContainerName": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Copy Xml",
|
||||
"type": "Copy",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Wait for Custom Vision",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"source": {
|
||||
"type": "BinarySource",
|
||||
"storeSettings": {
|
||||
"type": "AzureFileStorageReadSettings",
|
||||
"recursive": true,
|
||||
"wildcardFolderPath": {
|
||||
"value": "@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['resultsDirectory'], '/other')",
|
||||
"type": "Expression"
|
||||
},
|
||||
"wildcardFileName": "*.xml",
|
||||
"deleteFilesAfterCompletion": false
|
||||
},
|
||||
"formatSettings": {
|
||||
"type": "BinaryReadSettings"
|
||||
}
|
||||
},
|
||||
"sink": {
|
||||
"type": "BinarySink",
|
||||
"storeSettings": {
|
||||
"type": "AzureBlobFSWriteSettings"
|
||||
}
|
||||
},
|
||||
"enableStaging": false
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"referenceName": "gls",
|
||||
"type": "DatasetReference"
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"referenceName": "gld",
|
||||
"type": "DatasetReference",
|
||||
"parameters": {
|
||||
"DestinationFolderPath": "detections",
|
||||
"DestinationContainerName": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Read Spec Document",
|
||||
"type": "ExecuteDataFlow",
|
||||
"dependsOn": [],
|
||||
"policy": {
|
||||
"timeout": "1.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"dataflow": {
|
||||
"referenceName": "ReadSpecDocumentFlow",
|
||||
"type": "DataFlowReference",
|
||||
"datasetParameters": {
|
||||
"source": {
|
||||
"filename": "custom_vision_object_detection.json",
|
||||
"folderpath": "config",
|
||||
"containername": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"compute": {
|
||||
"coreCount": 8,
|
||||
"computeType": "General"
|
||||
},
|
||||
"traceLevel": "None",
|
||||
"cacheSinks": {
|
||||
"firstRowOnly": true
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Copy Xml From Convert Transform",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Copy Tiles",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 0,
|
||||
"retryIntervalInSeconds": 30,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Copy noop",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/copy_noop/src/main.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--src_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder",
|
||||
"convert/output.png.aux.xml",
|
||||
"--key_vault_name",
|
||||
"__linked_key_vault__",
|
||||
"--storage_account_key_secret_name",
|
||||
"GeospatialStorageAccountKey",
|
||||
"--linked_service_name",
|
||||
"AOI Pipeline Key Vault",
|
||||
"--dst_fileshare",
|
||||
"volume-a",
|
||||
"--dst_folder",
|
||||
"@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['submissionDirectory'], '/output.png.aux.xml')"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Prefix": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"BatchName": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"JobName": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"BatchLocation": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"StorageAccountName": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"StorageAccountKey": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
"FunctionCompleted": {
|
||||
"type": "String"
|
||||
},
|
||||
"FunctionError": {
|
||||
"type": "String"
|
||||
}
|
||||
},
|
||||
"annotations": [],
|
||||
"lastPublishTime": "2022-03-06T05:52:44Z"
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/pipelines"
|
||||
}
|
||||
|
|
@ -1,646 +0,0 @@
|
|||
{
|
||||
"name": "Custom Vision Object Detection v2_aks",
|
||||
"properties": {
|
||||
"activities": [
|
||||
{
|
||||
"name": "Pool Geolocation",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Copy Xml",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
},
|
||||
{
|
||||
"activity": "Copy Json",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 5,
|
||||
"retryIntervalInSeconds": 60,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Pool Geolocation",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/pool_geolocation/src/main.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--storage_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder_name",
|
||||
"detections",
|
||||
"--key_vault_name",
|
||||
"__linked_key_vault__",
|
||||
"--storage_account_key_secret_name",
|
||||
"GeospatialStorageAccountKey",
|
||||
"--linked_service_name",
|
||||
"AOI Pipeline Key Vault",
|
||||
"--config_file_name",
|
||||
"config-pool-geolocation.json"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Copy Tiles",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Read Spec Document",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 5,
|
||||
"retryIntervalInSeconds": 60,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Copy noop",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/copy_noop/src/main.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--src_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder",
|
||||
"tiles",
|
||||
"--key_vault_name",
|
||||
"__linked_key_vault__",
|
||||
"--storage_account_key_secret_name",
|
||||
"GeospatialStorageAccountKey",
|
||||
"--linked_service_name",
|
||||
"AOI Pipeline Key Vault",
|
||||
"--dst_fileshare",
|
||||
"volume-a",
|
||||
"--dst_folder",
|
||||
"@concat(pipeline().parameters.Prefix,'/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['submissionDirectory'])",
|
||||
"--folders_to_create",
|
||||
"@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['submissionDirectory'])",
|
||||
"--folders_to_create",
|
||||
"@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['resultsDirectory'])",
|
||||
"--folders_to_create",
|
||||
"@concat(pipeline().parameters.Prefix,'/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['logsDirectory'])"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Copy Config file",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Copy Tiles",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 5,
|
||||
"retryIntervalInSeconds": 60,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Copy noop",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/copy_noop/src/main.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--src_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder",
|
||||
"@concat('config/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['contextFileName'])",
|
||||
"--key_vault_name",
|
||||
"__linked_key_vault__",
|
||||
"--storage_account_key_secret_name",
|
||||
"GeospatialStorageAccountKey",
|
||||
"--linked_service_name",
|
||||
"AOI Pipeline Key Vault",
|
||||
"--dst_fileshare",
|
||||
"volume-a",
|
||||
"--dst_folder",
|
||||
"@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['contextFileName'])"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Copy Json",
|
||||
"type": "Copy",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Until1",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 5,
|
||||
"retryIntervalInSeconds": 60,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"source": {
|
||||
"type": "BinarySource",
|
||||
"storeSettings": {
|
||||
"type": "AzureFileStorageReadSettings",
|
||||
"recursive": true,
|
||||
"wildcardFolderPath": {
|
||||
"value": "@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['resultsDirectory'], '/json')",
|
||||
"type": "Expression"
|
||||
},
|
||||
"wildcardFileName": "*.json",
|
||||
"deleteFilesAfterCompletion": false
|
||||
},
|
||||
"formatSettings": {
|
||||
"type": "BinaryReadSettings"
|
||||
}
|
||||
},
|
||||
"sink": {
|
||||
"type": "BinarySink",
|
||||
"storeSettings": {
|
||||
"type": "AzureBlobFSWriteSettings"
|
||||
}
|
||||
},
|
||||
"enableStaging": false
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"referenceName": "gls",
|
||||
"type": "DatasetReference"
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"referenceName": "gld",
|
||||
"type": "DatasetReference",
|
||||
"parameters": {
|
||||
"DestinationFolderPath": "detections",
|
||||
"DestinationContainerName": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Copy Xml",
|
||||
"type": "Copy",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Until1",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 5,
|
||||
"retryIntervalInSeconds": 60,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"source": {
|
||||
"type": "BinarySource",
|
||||
"storeSettings": {
|
||||
"type": "AzureFileStorageReadSettings",
|
||||
"recursive": true,
|
||||
"wildcardFolderPath": {
|
||||
"value": "@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['resultsDirectory'], '/other')",
|
||||
"type": "Expression"
|
||||
},
|
||||
"wildcardFileName": "*.xml",
|
||||
"deleteFilesAfterCompletion": false
|
||||
},
|
||||
"formatSettings": {
|
||||
"type": "BinaryReadSettings"
|
||||
}
|
||||
},
|
||||
"sink": {
|
||||
"type": "BinarySink",
|
||||
"storeSettings": {
|
||||
"type": "AzureBlobFSWriteSettings"
|
||||
}
|
||||
},
|
||||
"enableStaging": false
|
||||
},
|
||||
"inputs": [
|
||||
{
|
||||
"referenceName": "gls",
|
||||
"type": "DatasetReference"
|
||||
}
|
||||
],
|
||||
"outputs": [
|
||||
{
|
||||
"referenceName": "gld",
|
||||
"type": "DatasetReference",
|
||||
"parameters": {
|
||||
"DestinationFolderPath": "detections",
|
||||
"DestinationContainerName": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "Read Spec Document",
|
||||
"type": "ExecuteDataFlow",
|
||||
"dependsOn": [],
|
||||
"policy": {
|
||||
"timeout": "1.00:00:00",
|
||||
"retry": 5,
|
||||
"retryIntervalInSeconds": 60,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"dataflow": {
|
||||
"referenceName": "ReadSpecDocumentFlow",
|
||||
"type": "DataFlowReference",
|
||||
"datasetParameters": {
|
||||
"source": {
|
||||
"filename": "custom_vision_object_detection.json",
|
||||
"folderpath": "config",
|
||||
"containername": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"compute": {
|
||||
"coreCount": 8,
|
||||
"computeType": "General"
|
||||
},
|
||||
"traceLevel": "None",
|
||||
"cacheSinks": {
|
||||
"firstRowOnly": true
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Copy Xml From Convert Transform",
|
||||
"type": "SparkJob",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Copy Tiles",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 5,
|
||||
"retryIntervalInSeconds": 60,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"sparkJob": {
|
||||
"referenceName": "Copy noop",
|
||||
"type": "SparkJobDefinitionReference"
|
||||
},
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/copy_noop/src/main.py",
|
||||
"args": [
|
||||
"--storage_account_name",
|
||||
"@pipeline().parameters.StorageAccountName",
|
||||
"--src_container",
|
||||
"@pipeline().parameters.Prefix",
|
||||
"--src_folder",
|
||||
"convert/output.png.aux.xml",
|
||||
"--key_vault_name",
|
||||
"__linked_key_vault__",
|
||||
"--storage_account_key_secret_name",
|
||||
"GeospatialStorageAccountKey",
|
||||
"--linked_service_name",
|
||||
"AOI Pipeline Key Vault",
|
||||
"--dst_fileshare",
|
||||
"volume-a",
|
||||
"--dst_folder",
|
||||
"@concat(pipeline().parameters.Prefix, '/', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['submissionDirectory'], '/output.png.aux.xml')"
|
||||
],
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"executorSize": "Medium",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.minExecutors": 2,
|
||||
"spark.dynamicAllocation.maxExecutors": 3
|
||||
},
|
||||
"driverSize": "Medium",
|
||||
"numExecutors": 2
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Gen Kubectl Task Cmd",
|
||||
"type": "SetVariable",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Copy Config file",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
},
|
||||
{
|
||||
"activity": "Copy Xml From Convert Transform",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"variableName": "KubeTaskJob",
|
||||
"value": {
|
||||
"value": "@concat('{\n \"apiVersion\": \"batch/v1\",\n \"kind\": \"Job\",\n \"metadata\": {\n \"name\": \"aoi-cv-task\",\n \"namespace\": \"vision\",\n \"labels\": {\n \"run_id\": \"',pipeline().RunId, '\",\n }\n },\n \"spec\": {\n \"ttlSecondsAfterFinished\": 5,\n \"template\": {\n \"spec\": {\n \"containers\": [\n {\n \"name\": \"aoi-cv-task-xyz\",\n \"image\": \"', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['algImageName'] ,'\",\n \"env\": [\n {\n \"name\": \"APP_INPUT_DIR\",\n \"value\": \"', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['mountedDirectory'] , '/',pipeline().parameters.Prefix,'/' , activity('Read Spec Document').output['runStatus'].output.sink.value[0]['submissionDirectory'],'\"\n },\n {\n \"name\": \"APP_OUTPUT_DIR\",\n \"value\": \"', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['mountedDirectory'] , '/',pipeline().parameters.Prefix,'/' , activity('Read Spec Document').output['runStatus'].output.sink.value[0]['resultsDirectory'],'\"\n },\n {\n \"name\": \"APP_CONFIG_DIR\",\n \"value\": \"', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['mountedDirectory'] , '/',pipeline().parameters.Prefix,'/' , activity('Read Spec Document').output['runStatus'].output.sink.value[0]['contextFileName'],'\"\n }\n ],\n \"volumeMounts\": [\n {\n \"name\": \"azure\",\n \"mountPath\": \"', activity('Read Spec Document').output['runStatus'].output.sink.value[0]['mountedDirectory'], '\"\n }\n ]\n }\n ],\n \"volumes\": [\n {\n \"name\": \"azure\",\n \"persistentVolumeClaim\": {\n \"claimName\": \"', pipeline().parameters.PersistentVolumeClaim, '\"\n }\n }\n ],\n \"automountServiceAccountToken\": false,\n \"restartPolicy\": \"Never\"\n }\n }\n }\n}')",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Gen SpecZipBase64Encoding",
|
||||
"type": "AzureFunctionActivity",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Gen Kubectl Task Cmd",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 5,
|
||||
"retryIntervalInSeconds": 60,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"functionName": "base64EncodedZipContent",
|
||||
"method": "POST",
|
||||
"headers": {
|
||||
"Content-Type": "application/json"
|
||||
},
|
||||
"body": {
|
||||
"value": "@json(variables('KubeTaskJob'))",
|
||||
"type": "Expression"
|
||||
}
|
||||
},
|
||||
"linkedServiceName": {
|
||||
"referenceName": "GenBase64Encoding",
|
||||
"type": "LinkedServiceReference"
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Set SpecZipBase64Encoded",
|
||||
"type": "SetVariable",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Gen SpecZipBase64Encoding",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"variableName": "KubeCmdSpecInBase64Encoded",
|
||||
"value": {
|
||||
"value": "@activity('Gen SpecZipBase64Encoding').output.Response",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Invoke Workload in AKS",
|
||||
"type": "WebActivity",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Set SpecZipBase64Encoded",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 3,
|
||||
"retryIntervalInSeconds": 120,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"url": "@pipeline().parameters.AksManagementRestApiURL",
|
||||
"connectVia": {
|
||||
"referenceName": "AutoResolveIntegrationRuntime",
|
||||
"type": "IntegrationRuntimeReference"
|
||||
},
|
||||
"method": "POST",
|
||||
"body": {
|
||||
"value": "@json(concat(\n'{',\n'\"command\" : \"', 'kubectl apply -f aoi-cv-task.json', '\",',\n'\"context\" : \"', variables('KubeCmdSpecInBase64Encoded'), '\"',\n'}'\n))",
|
||||
"type": "Expression"
|
||||
},
|
||||
"authentication": {
|
||||
"type": "MSI",
|
||||
"resource": "https://management.core.windows.net"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Until1",
|
||||
"type": "Until",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Invoke Workload in AKS",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"expression": {
|
||||
"value": "@contains(variables('KubeJobStatusCheckResult'),'No resources found')",
|
||||
"type": "Expression"
|
||||
},
|
||||
"activities": [
|
||||
{
|
||||
"name": "Check Job Status",
|
||||
"type": "WebActivity",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Wait",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"policy": {
|
||||
"timeout": "7.00:00:00",
|
||||
"retry": 5,
|
||||
"retryIntervalInSeconds": 60,
|
||||
"secureOutput": false,
|
||||
"secureInput": false
|
||||
},
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"url": "@pipeline().parameters.AksManagementRestApiURL",
|
||||
"connectVia": {
|
||||
"referenceName": "AutoResolveIntegrationRuntime",
|
||||
"type": "IntegrationRuntimeReference"
|
||||
},
|
||||
"method": "POST",
|
||||
"body": {
|
||||
"value": "@concat(\n'{',\n'\"command\" : \"', 'kubectl -n vision get jobs --selector run_id=', pipeline().RunId ,' \"',\n'}'\n)",
|
||||
"type": "Expression"
|
||||
},
|
||||
"authentication": {
|
||||
"type": "MSI",
|
||||
"resource": "https://management.core.windows.net"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Set JobStatus",
|
||||
"type": "SetVariable",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Check Job Status",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"variableName": "KubeJobStatusCheckResult",
|
||||
"value": {
|
||||
"value": "@activity('Check Job Status').output['properties']['logs']\n\n",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Wait",
|
||||
"type": "Wait",
|
||||
"dependsOn": [],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"waitTimeInSeconds": 15
|
||||
}
|
||||
}
|
||||
],
|
||||
"timeout": "3.00:00:00"
|
||||
}
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Prefix": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"StorageAccountName": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"StorageAccountKey": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"AksManagementRestApiURL": {
|
||||
"type": "string",
|
||||
"defaultValue": "__aks_management_rest_url__"
|
||||
},
|
||||
"PersistentVolumeClaim": {
|
||||
"type": "string",
|
||||
"defaultValue": "__persistent_volume_claim__"
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
"KubeTaskJob": {
|
||||
"type": "String"
|
||||
},
|
||||
"KubeCmdSpecInBase64Encoded": {
|
||||
"type": "String"
|
||||
},
|
||||
"KubeJobStatusCheckResult": {
|
||||
"type": "String"
|
||||
}
|
||||
},
|
||||
"annotations": [],
|
||||
"lastPublishTime": "2022-03-06T05:52:44Z"
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/pipelines"
|
||||
}
|
||||
|
|
@ -1,114 +0,0 @@
|
|||
{
|
||||
"name": "E2E Custom Vision Model Flow",
|
||||
"properties": {
|
||||
"activities": [
|
||||
{
|
||||
"name": "Transforms",
|
||||
"type": "ExecutePipeline",
|
||||
"dependsOn": [],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"pipeline": {
|
||||
"referenceName": "Custom Vision Model Transforms v2",
|
||||
"type": "PipelineReference"
|
||||
},
|
||||
"waitOnCompletion": true,
|
||||
"parameters": {
|
||||
"Prefix": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
},
|
||||
"StorageAccountName": {
|
||||
"value": "@pipeline().parameters.StorageAccountName",
|
||||
"type": "Expression"
|
||||
},
|
||||
"StorageAccountKey": {
|
||||
"value": "@pipeline().parameters.StorageAccountKey",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Custom Vision Object Detection",
|
||||
"type": "ExecutePipeline",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Transforms",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"pipeline": {
|
||||
"referenceName": "Custom Vision Object Detection v2",
|
||||
"type": "PipelineReference"
|
||||
},
|
||||
"waitOnCompletion": true,
|
||||
"parameters": {
|
||||
"Prefix": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
},
|
||||
"BatchName": {
|
||||
"value": "@pipeline().parameters.BatchAccountName",
|
||||
"type": "Expression"
|
||||
},
|
||||
"JobName": {
|
||||
"value": "@pipeline().parameters.BatchJobName",
|
||||
"type": "Expression"
|
||||
},
|
||||
"BatchLocation": {
|
||||
"value": "@pipeline().parameters.BatchLocation",
|
||||
"type": "Expression"
|
||||
},
|
||||
"StorageAccountName": {
|
||||
"value": "@pipeline().parameters.StorageAccountName",
|
||||
"type": "Expression"
|
||||
},
|
||||
"StorageAccountKey": {
|
||||
"value": "@pipeline().parameters.StorageAccountKey",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Prefix": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"StorageAccountName": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"StorageAccountKey": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"BatchAccountName": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"BatchJobName": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"BatchLocation": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
"Storage_Account_Conn_String": {
|
||||
"type": "String"
|
||||
}
|
||||
},
|
||||
"annotations": [],
|
||||
"lastPublishTime": "2022-03-06T05:42:39Z"
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/pipelines"
|
||||
}
|
||||
|
|
@ -1,106 +0,0 @@
|
|||
{
|
||||
"name": "E2E Custom Vision Model Flow_aks",
|
||||
"properties": {
|
||||
"activities": [
|
||||
{
|
||||
"name": "Transforms",
|
||||
"type": "ExecutePipeline",
|
||||
"dependsOn": [],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"pipeline": {
|
||||
"referenceName": "Custom Vision Model Transforms v2",
|
||||
"type": "PipelineReference"
|
||||
},
|
||||
"waitOnCompletion": true,
|
||||
"parameters": {
|
||||
"Prefix": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
},
|
||||
"StorageAccountName": {
|
||||
"value": "@pipeline().parameters.StorageAccountName",
|
||||
"type": "Expression"
|
||||
},
|
||||
"StorageAccountKey": {
|
||||
"value": "@pipeline().parameters.StorageAccountKey",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "Custom Vision Object Detection with AKS",
|
||||
"type": "ExecutePipeline",
|
||||
"dependsOn": [
|
||||
{
|
||||
"activity": "Transforms",
|
||||
"dependencyConditions": [
|
||||
"Succeeded"
|
||||
]
|
||||
}
|
||||
],
|
||||
"userProperties": [],
|
||||
"typeProperties": {
|
||||
"pipeline": {
|
||||
"referenceName": "Custom Vision Object Detection v2_aks",
|
||||
"type": "PipelineReference"
|
||||
},
|
||||
"waitOnCompletion": true,
|
||||
"parameters": {
|
||||
"Prefix": {
|
||||
"value": "@pipeline().parameters.Prefix",
|
||||
"type": "Expression"
|
||||
},
|
||||
"StorageAccountName": {
|
||||
"value": "@pipeline().parameters.StorageAccountName",
|
||||
"type": "Expression"
|
||||
},
|
||||
"StorageAccountKey": {
|
||||
"value": "@pipeline().parameters.StorageAccountKey",
|
||||
"type": "Expression"
|
||||
},
|
||||
"AksManagementRestApiURL": {
|
||||
"value": "@pipeline().parameters.AksManagementRestApiURL",
|
||||
"type": "Expression"
|
||||
},
|
||||
"PersistentVolumeClaim": {
|
||||
"value": "@pipeline().parameters.PersistentVolumeClaim",
|
||||
"type": "Expression"
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
],
|
||||
"parameters": {
|
||||
"Prefix": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"StorageAccountName": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"StorageAccountKey": {
|
||||
"type": "string",
|
||||
"defaultValue": ""
|
||||
},
|
||||
"AksManagementRestApiURL": {
|
||||
"type": "string",
|
||||
"defaultValue": "__aks_management_rest_url__"
|
||||
},
|
||||
"PersistentVolumeClaim": {
|
||||
"type": "string",
|
||||
"defaultValue": "__persistent_volume_claim__"
|
||||
}
|
||||
},
|
||||
"variables": {
|
||||
"Storage_Account_Conn_String": {
|
||||
"type": "String"
|
||||
}
|
||||
},
|
||||
"annotations": [],
|
||||
"lastPublishTime": "2022-03-06T05:42:39Z"
|
||||
},
|
||||
"type": "Microsoft.Synapse/workspaces/pipelines"
|
||||
}
|
||||
|
|
@ -1 +0,0 @@
|
|||
{"publishBranch":"workspace_test"}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
{
|
||||
"name": "Convert",
|
||||
"properties": {
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"requiredSparkVersion": "3.1",
|
||||
"language": "python",
|
||||
"jobProperties": {
|
||||
"name": "Convert",
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/raster_convert/src/convert.py",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.enabled": "false",
|
||||
"spark.dynamicAllocation.minExecutors": "1",
|
||||
"spark.dynamicAllocation.maxExecutors": "2",
|
||||
"spark.autotune.trackingId": "72aef2fd-aaae-40ed-8a09-7b2e87353ace"
|
||||
},
|
||||
"args": [],
|
||||
"jars": [],
|
||||
"files": [],
|
||||
"driverMemory": "56g",
|
||||
"driverCores": 8,
|
||||
"executorMemory": "56g",
|
||||
"executorCores": 8,
|
||||
"numExecutors": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,30 +0,0 @@
|
|||
{
|
||||
"name": "Copy noop",
|
||||
"properties": {
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"requiredSparkVersion": "3.1",
|
||||
"language": "python",
|
||||
"jobProperties": {
|
||||
"name": "Copy noop",
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/copy_noop/src/main.py",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.enabled": "false",
|
||||
"spark.dynamicAllocation.minExecutors": "1",
|
||||
"spark.dynamicAllocation.maxExecutors": "2",
|
||||
"spark.autotune.trackingId": "01767b3a-cede-4abf-8b79-52cb6d0ff80d"
|
||||
},
|
||||
"args": [],
|
||||
"jars": [],
|
||||
"files": [],
|
||||
"driverMemory": "56g",
|
||||
"driverCores": 8,
|
||||
"executorMemory": "56g",
|
||||
"executorCores": 8,
|
||||
"numExecutors": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
{
|
||||
"name": "Crop",
|
||||
"properties": {
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"requiredSparkVersion": "3.1",
|
||||
"language": "python",
|
||||
"jobProperties": {
|
||||
"name": "Crop",
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/raster_crop/src/crop.py",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.enabled": "false",
|
||||
"spark.dynamicAllocation.minExecutors": "1",
|
||||
"spark.dynamicAllocation.maxExecutors": "2",
|
||||
"spark.autotune.trackingId": "f4cbbafe-9d98-476f-9bd4-e5bfc7bad06c"
|
||||
},
|
||||
"args": [],
|
||||
"jars": [],
|
||||
"files": [
|
||||
"abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/raster_crop/src/utils.py"
|
||||
],
|
||||
"driverMemory": "56g",
|
||||
"driverCores": 8,
|
||||
"executorMemory": "56g",
|
||||
"executorCores": 8,
|
||||
"numExecutors": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
{
|
||||
"name": "Mosaic",
|
||||
"properties": {
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"requiredSparkVersion": "3.1",
|
||||
"language": "python",
|
||||
"jobProperties": {
|
||||
"name": "Mosaic",
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/raster_mosaic/src/mosaic.py",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.enabled": "false",
|
||||
"spark.dynamicAllocation.minExecutors": "3",
|
||||
"spark.dynamicAllocation.maxExecutors": "3",
|
||||
"spark.autotune.trackingId": "811de002-982f-4b4b-9732-147d3565c502"
|
||||
},
|
||||
"args": [],
|
||||
"jars": [],
|
||||
"files": [],
|
||||
"driverMemory": "56g",
|
||||
"driverCores": 8,
|
||||
"executorMemory": "56g",
|
||||
"executorCores": 8,
|
||||
"numExecutors": 3
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,31 +0,0 @@
|
|||
{
|
||||
"name": "Pool Geolocation",
|
||||
"properties": {
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"requiredSparkVersion": "3.1",
|
||||
"language": "python",
|
||||
"jobProperties": {
|
||||
"name": "Pool Geolocation",
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/pool_geolocation/src/main.py",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.enabled": "false",
|
||||
"spark.dynamicAllocation.minExecutors": "1",
|
||||
"spark.dynamicAllocation.maxExecutors": "2",
|
||||
"spark.autotune.trackingId": "0d715b42-8d99-4e74-8a24-860c7275f387"
|
||||
},
|
||||
"args": [],
|
||||
"jars": [],
|
||||
"files": [
|
||||
"abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/pool_geolocation/src/utils.py"
|
||||
],
|
||||
"driverMemory": "56g",
|
||||
"driverCores": 8,
|
||||
"executorMemory": "56g",
|
||||
"executorCores": 8,
|
||||
"numExecutors": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
{
|
||||
"name": "Tiling",
|
||||
"properties": {
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"requiredSparkVersion": "3.1",
|
||||
"language": "python",
|
||||
"jobProperties": {
|
||||
"name": "Tiling",
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/raster_tiling/src/tiling.py",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.enabled": "false",
|
||||
"spark.dynamicAllocation.minExecutors": "1",
|
||||
"spark.dynamicAllocation.maxExecutors": "2",
|
||||
"spark.autotune.trackingId": "65be76e5-ef21-47ec-be7a-38039b2abfd4"
|
||||
},
|
||||
"args": [],
|
||||
"jars": [],
|
||||
"files": [],
|
||||
"driverMemory": "56g",
|
||||
"driverCores": 8,
|
||||
"executorMemory": "56g",
|
||||
"executorCores": 8,
|
||||
"numExecutors": 2
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -1,29 +0,0 @@
|
|||
{
|
||||
"name": "Warp",
|
||||
"properties": {
|
||||
"targetBigDataPool": {
|
||||
"referenceName": "__synapse_pool_name__",
|
||||
"type": "BigDataPoolReference"
|
||||
},
|
||||
"requiredSparkVersion": "3.1",
|
||||
"language": "python",
|
||||
"jobProperties": {
|
||||
"name": "Warp",
|
||||
"file": "abfss://spark-jobs@__synapse_storage_account__.dfs.core.windows.net/raster_warp/src/warp.py",
|
||||
"conf": {
|
||||
"spark.dynamicAllocation.enabled": "false",
|
||||
"spark.dynamicAllocation.minExecutors": "3",
|
||||
"spark.dynamicAllocation.maxExecutors": "3",
|
||||
"spark.autotune.trackingId": "335dd1ad-fc75-4734-ad92-03a79e9ad399"
|
||||
},
|
||||
"args": [],
|
||||
"jars": [],
|
||||
"files": [],
|
||||
"driverMemory": "56g",
|
||||
"driverCores": 8,
|
||||
"executorMemory": "56g",
|
||||
"executorCores": 8,
|
||||
"numExecutors": 3
|
||||
}
|
||||
}
|
||||
}
|
||||
Загрузка…
Ссылка в новой задаче