John Rampono 66ad02966e Major changes in this version including: ... - Complete refactor of ADF Code Components to improve naming conventions and consistency - Addition of Node Web App to facilitate configuration editing On branch development/0.1 Changes to be committed: modified: README.md new file: documentation/images/DeploymentForm.png renamed: solution/DataFactory/dataset/ADLSBinary_AZ_IR.json -> solution/DataFactory/ADF/dataset/ADLSBinary_IRA.json renamed: solution/DataFactory/dataset/ADLSDelimitedText_SH_IR.json -> solution/DataFactory/ADF/dataset/ADLSDelimitedText_IRA.json renamed: solution/DataFactory/dataset/ADLSExcel_SH_IR.json -> solution/DataFactory/ADF/dataset/ADLSExcel_IRA.json renamed: solution/DataFactory/dataset/ADLSJson_SH_IR.json -> solution/DataFactory/ADF/dataset/ADLSJson_IRA.json renamed: solution/DataFactory/dataset/ADLSParquet_AZ_IR.json -> solution/DataFactory/ADF/dataset/ADLSParquet_IRA.json renamed: solution/DataFactory/dataset/AzureSqlTable_AZ_IR.json -> solution/DataFactory/ADF/dataset/AzureSqlTable_IRA.json renamed: solution/DataFactory/dataset/BlobBinary_AZ_IR.json -> solution/DataFactory/ADF/dataset/BlobBinary_IRA.json renamed: solution/DataFactory/dataset/BlobDelimitedText_SH_IR.json -> solution/DataFactory/ADF/dataset/BlobDelimitedText_IRA.json renamed: solution/DataFactory/dataset/BlobExcel_SH_IR.json -> solution/DataFactory/ADF/dataset/BlobExcel_IRA.json renamed: solution/DataFactory/dataset/BlobJson_SH_IR.json -> solution/DataFactory/ADF/dataset/BlobJson_IRA.json renamed: solution/DataFactory/dataset/BlobParquet_AZ_IR.json -> solution/DataFactory/ADF/dataset/BlobParquet_IRA.json renamed: solution/DataFactory/dataset/FileBinary_SH_IR.json -> solution/DataFactory/ADF/dataset/FileBinary_IRA.json renamed: solution/DataFactory/dataset/FileParquet_SH_IR.json -> solution/DataFactory/ADF/dataset/FileParquet_IRA.json renamed: solution/DataFactory/dataset/SqlServerTable_SH_IR.json -> solution/DataFactory/ADF/dataset/SqlServerTable_IRA.json renamed: solution/DataFactory/factory/adsgofastdatakakeacceladf.json -> solution/DataFactory/ADF/factory/adsgofastdatakakeacceladf.json new file: solution/DataFactory/ADF/integrationRuntime/IRA.json renamed: solution/DataFactory/linkedService/AdsGoFastKeyVault.json -> solution/DataFactory/ADF/linkedService/AdsGoFastKeyVault.json renamed: solution/DataFactory/linkedService/AzureFunctionAdsGoFastDataLakeAccelFunApp.json -> solution/DataFactory/ADF/linkedService/AzureFunctionAdsGoFastDataLakeAccelFunApp.json renamed: solution/DataFactory/linkedService/GenericAzureDataLakeStorage_SH_IR.json -> solution/DataFactory/ADF/linkedService/GenericAzureDataLakeStorage_IRA.json renamed: solution/DataFactory/linkedService/GenericAzureKeyVault.json -> solution/DataFactory/ADF/linkedService/GenericAzureKeyVault_IRA.json renamed: solution/DataFactory/linkedService/GenericAzureSQL_SH_IR.json -> solution/DataFactory/ADF/linkedService/GenericAzureSQL_IRA.json renamed: solution/DataFactory/linkedService/GenericBlobStorage_SH_IR.json -> solution/DataFactory/ADF/linkedService/GenericBlobStorage_IRA.json renamed: solution/DataFactory/linkedService/GenericFileServer_SH_IR.json -> solution/DataFactory/ADF/linkedService/GenericFileServer_IRA.json renamed: solution/DataFactory/linkedService/GenericMsSqlServer_SH_IR.json -> solution/DataFactory/ADF/linkedService/GenericMsSqlServer_IRA.json new file: solution/DataFactory/ADF/pipeline/AZ_Function_Generic.json renamed: solution/DataFactory/pipeline/AZ-SQL-AZ-Storage-Parquet-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_SQL_AZ_Storage_Parquet_IRA.json renamed: solution/DataFactory/pipeline/AZ-SQL-Create-Table-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_SQL_Create_Table_IRA.json renamed: solution/DataFactory/pipeline/AZ-SQL-Full-Load-Chunk-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_SQL_Full_Load_Chunk_IRA.json renamed: solution/DataFactory/pipeline/AZ-SQL-Full-Load-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_SQL_Full_Load_IRA.json renamed: solution/DataFactory/pipeline/AZ-SQL-Post-Copy-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_SQL_Post_Copy_IRA.json renamed: solution/DataFactory/pipeline/AZ-SQL-StoredProcedure-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_SQL_StoredProcedure_IRA.json renamed: solution/DataFactory/pipeline/AZ-SQL-Watermark-Chunk-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_SQL_Watermark_Chunk_IRA.json renamed: solution/DataFactory/pipeline/AZ-SQL-Watermark-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_SQL_Watermark_IRA.json renamed: solution/DataFactory/pipeline/AZ-Storage-Binary-AZ-Storage-Binary-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_Storage_Binary_AZ_Storage_Binary_IRA.json renamed: solution/DataFactory/pipeline/AZ-Storage-CSV-AZ-SQL-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_Storage_CSV_AZ_SQL_IRA.json renamed: solution/DataFactory/pipeline/AZ-Storage-Excel-AZ-SQL-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_Storage_Excel_AZ_SQL_IRA.json renamed: solution/DataFactory/pipeline/AZ-Storage-Excel-AZ-Storage-CSV-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_Storage_Excel_AZ_Storage_CSV_IRA.json renamed: solution/DataFactory/pipeline/AZ-Storage-JSON-AZ-SQL-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_Storage_JSON_AZ_SQL_IRA.json renamed: solution/DataFactory/pipeline/AZ-Storage-Parquet-AZ-SQL-SH-IR.json -> solution/DataFactory/ADF/pipeline/AZ_Storage_Parquet_AZ_SQL_IRA.json renamed: solution/DataFactory/pipeline/Create-Task-Master-AZ-SQL-SH-IR.json -> solution/DataFactory/ADF/pipeline/Create_Task_Master_AZ_SQL_IRA.json renamed: solution/DataFactory/pipeline/GEN-File-Binary-AZ-Storage-Binary-SH-IR.json -> solution/DataFactory/ADF/pipeline/GEN_File_Binary_AZ_Storage_Binary_IRA.json renamed: solution/DataFactory/pipeline/Master.json -> solution/DataFactory/ADF/pipeline/Master_IRA.json renamed: solution/DataFactory/pipeline/SH-SQL-AZ-Storage-Parquet-SH-IR.json -> solution/DataFactory/ADF/pipeline/SH_SQL_AZ_Storage_Parquet_IRA.json renamed: solution/DataFactory/pipeline/SH-SQL-Full-Load-Chunk-SH-IR.json -> solution/DataFactory/ADF/pipeline/SH_SQL_Full_Load_Chunk_IRA.json renamed: solution/DataFactory/pipeline/SH-SQL-Full-Load-SH-IR.json -> solution/DataFactory/ADF/pipeline/SH_SQL_Full_Load_IRA.json renamed: solution/DataFactory/pipeline/OnP-SQL-GEN-File-Parquet-OnP-SH-IR.json -> solution/DataFactory/ADF/pipeline/SH_SQL_GEN_File_Parquet_IRA.json renamed: solution/DataFactory/pipeline/SH-SQL-Watermark-Chunk-SH-IR.json -> solution/DataFactory/ADF/pipeline/SH_SQL_Watermark_Chunk_IRA.json renamed: solution/DataFactory/pipeline/SH-SQL-Watermark-SH-IR.json -> solution/DataFactory/ADF/pipeline/SH_SQL_Watermark_IRA.json renamed: solution/DataFactory/trigger/Trigger on File End.json -> solution/DataFactory/ADF/trigger/Trigger on File End.json new file: solution/DataFactory/GenerateADFArtefacts.ps1 new file: solution/DataFactory/IRConfig.json renamed: solution/DataFactory/dataset/ADLSBinary_SH_IR.json -> solution/DataFactory/Templates/dataset/ADLSBinary_{IR}.json new file: solution/DataFactory/Templates/dataset/ADLSDelimitedText_{IR}.json new file: solution/DataFactory/Templates/dataset/ADLSExcel_{IR}.json new file: solution/DataFactory/Templates/dataset/ADLSJson_{IR}.json renamed: solution/DataFactory/dataset/ADLSParquet_SH_IR.json -> solution/DataFactory/Templates/dataset/ADLSParquet_{IR}.json renamed: solution/DataFactory/dataset/AzureSqlTable_SH_IR.json -> solution/DataFactory/Templates/dataset/AzureSqlTable_{IR}.json renamed: solution/DataFactory/dataset/BlobBinary_SH_IR.json -> solution/DataFactory/Templates/dataset/BlobBinary_{IR}.json new file: solution/DataFactory/Templates/dataset/BlobDelimitedText_{IR}.json new file: solution/DataFactory/Templates/dataset/BlobExcel_{IR}.json new file: solution/DataFactory/Templates/dataset/BlobJson_{IR}.json renamed: solution/DataFactory/dataset/BlobParquet_SH_IR.json -> solution/DataFactory/Templates/dataset/BlobParquet_{IR}.json renamed: solution/DataFactory/dataset/FileBinary_OnPrem_SH_IR.json -> solution/DataFactory/Templates/dataset/FileBinary_{IR}.json renamed: solution/DataFactory/dataset/FileParquet_OnPrem_SH_IR.json -> solution/DataFactory/Templates/dataset/FileParquet_{IR}.json renamed: solution/DataFactory/dataset/SqlServerTable_OnPrem_SH_IR.json -> solution/DataFactory/Templates/dataset/SqlServerTable_{IR}.json new file: solution/DataFactory/Templates/factory/adsgofastdatakakeacceladf.json new file: solution/DataFactory/Templates/integrationRuntime/IRA.json new file: solution/DataFactory/Templates/integrationRuntime/IRB.json renamed: solution/DataFactory/linkedService/GenericAzureDataLakeStorage_OnPrem_Net.json -> solution/DataFactory/Templates/linkedService/GenericAzureDataLakeStorage_{IR}.json new file: solution/DataFactory/Templates/linkedService/GenericAzureKeyVault_{IR}.json renamed: solution/DataFactory/linkedService/GenericAzureSQL.json -> solution/DataFactory/Templates/linkedService/GenericAzureSQL_{IR}.json renamed: solution/DataFactory/linkedService/GenericBlobStorage_OnPrem_Net.json -> solution/DataFactory/Templates/linkedService/GenericBlobStorage_{IR}.json renamed: solution/DataFactory/linkedService/GenericFileServer_OnPrem_Net.json -> solution/DataFactory/Templates/linkedService/GenericFileServer_{IR}.json renamed: solution/DataFactory/linkedService/GenericMsSqlServer_OnPrem_Net.json -> solution/DataFactory/Templates/linkedService/GenericMsSqlServer_{IR}.json new file: solution/DataFactory/Templates/linkedService/NamingConventions.md renamed: solution/DataFactory/pipeline/AZ-Function-Generic.json -> solution/DataFactory/Templates/pipeline/AZ_Function_Generic.json new file: solution/DataFactory/Templates/pipeline/AZ_SQL_AZ_Storage_Parquet_{IR}.json new file: solution/DataFactory/Templates/pipeline/AZ_SQL_Create_Table_{IR}.json renamed: solution/DataFactory/pipeline/OnP-SQL-Full-Load-Chunk-OnP-SH-IR.json -> solution/DataFactory/Templates/pipeline/AZ_SQL_Full_Load_Chunk_{IR}.json new file: solution/DataFactory/Templates/pipeline/AZ_SQL_Full_Load_{IR}.json new file: solution/DataFactory/Templates/pipeline/AZ_SQL_Post_Copy_{IR}.json new file: solution/DataFactory/Templates/pipeline/AZ_SQL_StoredProcedure_{IR}.json renamed: solution/DataFactory/pipeline/OnP-SQL-Watermark-Chunk-OnP-SH-IR.json -> solution/DataFactory/Templates/pipeline/AZ_SQL_Watermark_Chunk_{IR}.json new file: solution/DataFactory/Templates/pipeline/AZ_SQL_Watermark_{IR}.json renamed: solution/DataFactory/pipeline/AZ-Storage-Binary-AZ-Storage-Binary-AZ-IR.json -> solution/DataFactory/Templates/pipeline/AZ_Storage_Binary_AZ_Storage_Binary_{IR}.json new file: solution/DataFactory/Templates/pipeline/AZ_Storage_CSV_AZ_SQL_{IR}.json new file: solution/DataFactory/Templates/pipeline/AZ_Storage_Excel_AZ_SQL_{IR}.json new file: solution/DataFactory/Templates/pipeline/AZ_Storage_Excel_AZ_Storage_CSV_{IR}.json new file: solution/DataFactory/Templates/pipeline/AZ_Storage_JSON_AZ_SQL_{IR}.json new file: solution/DataFactory/Templates/pipeline/AZ_Storage_Parquet_AZ_SQL_{IR}.json renamed: solution/DataFactory/pipeline/Create-Task-Master-AZ-SQL-OnP-SH-IR.json -> solution/DataFactory/Templates/pipeline/Create_Task_Master_AZ_SQL_{IR}.json renamed: solution/DataFactory/pipeline/GEN-File-Binary-AZ-Storage-Binary-OnP-SH-IR.json -> solution/DataFactory/Templates/pipeline/GEN_File_Binary_AZ_Storage_Binary_{IR}.json new file: solution/DataFactory/Templates/pipeline/Master_{IR}.json renamed: solution/DataFactory/pipeline/OnP-SQL-AZ-Storage-Parquet-OnP-SH-IR.json -> solution/DataFactory/Templates/pipeline/SH_SQL_AZ_Storage_Parquet_{IR}.json new file: solution/DataFactory/Templates/pipeline/SH_SQL_Full_Load_Chunk_{IR}.json renamed: solution/DataFactory/pipeline/OnP-SQL-Full-Load-OnP-SH-IR.json -> solution/DataFactory/Templates/pipeline/SH_SQL_Full_Load_{IR}.json new file: solution/DataFactory/Templates/pipeline/SH_SQL_GEN_File_Parquet_{IR}.json new file: solution/DataFactory/Templates/pipeline/SH_SQL_Watermark_Chunk_{IR}.json renamed: solution/DataFactory/pipeline/OnP-SQL-Watermark-OnP-SH-IR.json -> solution/DataFactory/Templates/pipeline/SH_SQL_Watermark_{IR}.json new file: solution/DataFactory/Templates/trigger/Trigger on File End.json deleted: solution/DataFactory/dataset/ADLSBinary_OnPrem_SH_IR.json deleted: solution/DataFactory/dataset/ADLSParquet_OnPrem_SH_IR.json deleted: solution/DataFactory/dataset/BlobBinary_OnPrem_SH_IR.json deleted: solution/DataFactory/dataset/BlobParquet_OnPrem_SH_IR.json deleted: solution/DataFactory/integrationRuntime/SelfHostedIntegrationRuntime-Azure-VNET.json deleted: solution/DataFactory/integrationRuntime/SelfHostedIntegrationRuntime-OnPem-Net.json deleted: solution/DataFactory/linkedService/GenericAzureDataLakeStorage.json deleted: solution/DataFactory/linkedService/GenericBlobStorage.json deleted: solution/DataFactory/pipeline/AZ-SQL-AZ-Storage-Parquet-AZ-IR.json deleted: solution/DataFactory/pipeline/AZ-Storage-Parquet-AZ-SQL-AZ-IR.json modified: solution/Database/ADSGoFastDbUp/AdsGoFastDbUp/0-1-0-beta/A-Journaled/006-SampleTasks_TaskMaster.sql modified: solution/Database/ADSGoFastDbUp/AdsGoFastDbUp/0-1-0-beta/A-Journaled/008-SampleTasks_SourceAndTargetSystems.sql new file: solution/Database/ADSGoFastDbUp/AdsGoFastDbUp/0-1-0-beta/A-Journaled/010-TaskTypeMapping_Update.sql modified: solution/Deployment/.devcontainer/devcontainer.json new file: solution/Deployment/arm/AppService_Func.bicep new file: solution/Deployment/arm/AppService_Web.bicep new file: solution/Deployment/arm/ApplicationInsights.bicep modified: solution/Deployment/arm/AzureSQLServer.json new file: solution/Deployment/arm/AzureSqlServer.bicep new file: solution/Deployment/arm/DataFactory.bicep new file: solution/Deployment/arm/FunctionApp.bicep new file: solution/Deployment/arm/KeyVault.bicep new file: solution/Deployment/arm/LogAnalytics.bicep new file: solution/Deployment/arm/Networking.bicep new file: solution/Deployment/arm/Storage_ADLS.bicep new file: solution/Deployment/arm/Storage_Blob.bicep new file: solution/Deployment/arm/Storage_Logging.bicep new file: solution/Deployment/arm/VirtualMachine.bicep new file: solution/Deployment/arm/WebApp.bicep modified: solution/Deployment/arm/WebApp.json new file: solution/Deployment/environments/EditSettings.html new file: solution/Deployment/environments/Node/package-lock.json new file: solution/Deployment/environments/Node/package.json new file: solution/Deployment/environments/Node/server.js modified: solution/Deployment/environments/development.json new file: solution/Deployment/environments/development_bak.json new file: solution/Deployment/environments/development_test.json modified: solution/Deployment/environments/environment.schema.json modified: solution/Deployment/workflows/CD_0a_CreateServicePrincipals_AAD_Elevated.ps1 modified: solution/Deployment/workflows/CD_1a_DeployServices.ps1 modified: solution/Deployment/workflows/CD_2a_CreateMSIs_AAD_Elevated.ps1 modified: solution/Deployment/workflows/CD_2b_ConfigureServices.ps1 modified: solution/Deployment/workflows/LocalDevOnly_EnvironmentSetUp.ps1 modified: solution/Deployment/workflows/Steps/CD_ConfigureADF.ps1 modified: solution/Deployment/workflows/Steps/CD_ConfigureAzureSQLServer.ps1 modified: solution/Deployment/workflows/Steps/CD_ConfigureAzureSqlServer_UpdateTaskTypeMappingJson.ps1 modified: solution/Deployment/workflows/Steps/CD_ConfigureFunctionApp.ps1 modified: solution/Deployment/workflows/Steps/CD_ConfigureKeyVault.ps1 new file: solution/Deployment/workflows/Steps/CD_ConfigureSampleData.ps1 modified: solution/Deployment/workflows/Steps/CD_ConfigureVnet.ps1 modified: solution/Deployment/workflows/Steps/CD_ConfigureWebApp.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployADF.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployADFOnPremSHIR.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployAppInsights.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployAppService.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployAzureSqlServer.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployFunctionApp.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployKeyVault.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployLogAnalytics.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployResourceGroup.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployStorageADLS.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployStorageBlob.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployStorageForLogging.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployVnet.ps1 modified: solution/Deployment/workflows/Steps/CD_DeployWebSite.ps1 modified: solution/Deployment/workflows/Steps/CD_GrantRBAC.ps1 modified: solution/Deployment/workflows/Steps/CD_SetResourceGroupHash.ps1 modified: solution/Deployment/workflows/Steps/CI_BuildDataFactory.ps1 modified: solution/Deployment/workflows/Steps/InstallGatewayFunctions.ps1 modified: solution/Deployment/workflows/Steps/PushEnvFileIntoVariables.ps1 modified: solution/FunctionApp/FunctionApp.csproj modified: solution/FunctionApp/Startup.cs modified: solution/FunctionApp/TaskMetaData/TaskInstance.cs new file: solution/SampleFiles/yellow_tripdata_2017-03.csv new file: solution/SampleFiles/yellow_tripdata_2017-03.xlsx renamed: solution/TaskTypeJson/AZ-SQL-AZ-Storage-Parquet-SH-IR.json -> solution/TaskTypeJson/AZ_SQL_AZ_Storage_Parquet_IRA.json renamed: solution/TaskTypeJson/AZ-SQL-StoredProcedure-SH-IR.json -> solution/TaskTypeJson/AZ_SQL_StoredProcedure_IRA.json renamed: solution/TaskTypeJson/AZ-Storage-Binary-AZ-Storage-Binary-SH-IR.json -> solution/TaskTypeJson/AZ_Storage_Binary_AZ_Storage_Binary_IRA.json renamed: solution/TaskTypeJson/AZ-Storage-CSV-AZ-SQL-SH-IR.json -> solution/TaskTypeJson/AZ_Storage_CSV_AZ_SQL_IRA.json renamed: solution/TaskTypeJson/AZ-Storage-Excel-AZ-SQL-SH-IR.json -> solution/TaskTypeJson/AZ_Storage_Excel_AZ_SQL_IRA.json renamed: solution/TaskTypeJson/AZ-Storage-Excel-AZ-Storage-CSV-SH-IR.json -> solution/TaskTypeJson/AZ_Storage_Excel_AZ_Storage_CSV_IRA.json renamed: solution/TaskTypeJson/AZ-Storage-JSON-AZ-SQL-SH-IR.json -> solution/TaskTypeJson/AZ_Storage_JSON_AZ_SQL_IRA.json renamed: solution/TaskTypeJson/AZ-Storage-Parquet-AZ-SQL-SH-IR.json -> solution/TaskTypeJson/AZ_Storage_Parquet_AZ_SQL_IRA.json renamed: solution/TaskTypeJson/GEN-File-Binary-AZ-Storage-Binary-OnP-SH-IR.json -> solution/TaskTypeJson/GEN_File_Binary_AZ_Storage_Binary_IRA.json renamed: solution/TaskTypeJson/GEN-File-Binary-AZ-Storage-Binary-SH-IR.json -> solution/TaskTypeJson/GEN_File_Binary_AZ_Storage_Binary_IRB.json renamed: solution/TaskTypeJson/OnP-SQL-GEN-File-Parquet-OnP-SH-IR.json -> solution/TaskTypeJson/OnP_SQL_AZ_Storage_Parquet_IRB.json renamed: solution/TaskTypeJson/OnP-SQL-AZ-Storage-Parquet-OnP-SH-IR.json -> solution/TaskTypeJson/OnP_SQL_GEN_File_Parquet_IRB.json modified: solution/WebApplication/WebApplication/Controllers/TaskMasterController.cs modified: solution/WebApplication/WebApplication/Startup.cs modified: solution/WebApplication/WebApplication/WebApplication.csproj		2021-11-21 19:55:34 +08:00
..
DeploymentForm.png	Major changes in this version including:	2021-11-21 19:55:34 +08:00