Граф коммитов

40 Коммитов

Автор SHA1 Сообщение Дата
Rohit Agrawal - MSFT af58f22662
Update dockerfile to pull in dotnet image from mcr (#235)
* Update dockerfile to pull in dotnet image from mcr

Update dockerfile to pull in dotnet image from mcr.microsoft.com

* Update Dockerfile
2022-06-01 22:35:38 -07:00
Rohit Agrawal - MSFT b8bd32802d
Fix regex and add support for custom ambari SQL db in HDI (#195) 2021-10-20 11:51:49 -07:00
Rohit Agrawal - MSFT 774b7d6ddd
Switch from KeyVaultClient to SecretClient and remove sqldb dependency (#194)
* Switch from KeyVaultClient to SecretClient and remove sqldb dependency

* throw exception when data is sent to SQLDB
2021-10-06 12:29:19 -07:00
Rohit Agrawal - MSFT d368b8d95b
1. DataProcessing – (#187)
a. Rev packages flagged by component governance

2. ARM Template –
a. Change SF durability to silver.
b. Set min TLS for HDInsight to 1.2 .
c. Add zookeeper node config for HDInsight

3. Services –
a. Add API for start and stop all spark jobs

4. Website –
a. Accessibility: Tweak borderline color for tabs on flow definition page, charts on metric page and flows list table on flows page to achieve minimum 3:1 luminosity contrast
b. Rev packages flagged by component governance
c. Add button to ‘start all jobs’ and ‘stop all jobs’ on the jobs page
2021-09-24 15:56:53 -07:00
Rohit Agrawal - MSFT 605c7457bf
1. Use existing cert from keyvault for Service fabric in ARM template (#183)
2. Parameterize SF VM image SKU in ARM template
3. Make DataX compatible with HDInsight 4.0
2021-07-01 10:54:09 -07:00
jozavala 86e959af67
Scenario Tester Deployment Scripts (#161)
* Initial commit for scenario tester arm deployment

* Initial commit for deployment of Scenario Tester through an ARM Template

* Some improvements: Event hub Id detection based in if it is a iot device or not

* Add smart is data bricks function

* Error fix: connection string property was not correctly set

* Initial commit for deployment scripts and utilities

* Deploy a default parameter file

* Configure properly a system identity to the new scenario tester app + grant access to the service key vault. Do the same for spark keyvault as well. Make the app always on too.

* Retrieve most of the information in the deployment resources existing in the environment already

* Fix some errors in the script and template. Load from package was not really supported and there was a typo in the template.

* typo fix in blob storage uri

* Use the correct connection string from the array

* Adding missing entries into the spark kv + make cert validation an argument to the deployment script entrypoint

* Pass key vault names as parameters to json deployment file + make data bricks token resource conditional to the environment spark type

* Add comments for deployment scripts for scenario tester.

* Add a few more details

* Add basic readme file for scenario tester directory

* Enrich readme even further
2020-08-25 11:15:54 -07:00
Rohit Agrawal - MSFT 24f93833bf
rev version of components based on CG recommendation (#169) 2020-04-14 16:38:56 -07:00
Anand Kishore 4358d2a8e8
Added check for mdbc PSModule version (#166)
* Added check for mdbc PSModule version

Mdbc PSModule doesn't support New-MdbcQuery cmdlet from v6.0.0. we have to make sure correct Mdbc module (v5.1.4) is installed.
Deployment uses New-MdbcQuery cmdlet to setup CosmosDb. 
$qry = New-MdbcQuery -Name "_id" -Exists

* Install required version of PS modules

Installing and importing the correct required version of PS modules.
2020-03-30 16:01:50 -07:00
msandg 942d6bcd8d
enable to deploy the HDI cluster 4.0 (#158)
* enable to deploy the HDI cluster 4.0

* add the update to the cosmos db config

* fix typo

* include the jackson jar in the spark nuget package
2019-11-20 15:46:22 -08:00
Kwangje Cho e6a8f5f8c5
Add a batch sample (#160)
* Added eventhub batch sample

* changed to oneTime from recurring
2019-10-21 12:46:35 -07:00
Rohit Agrawal - MSFT 0ba78714ac
Add more output sinks to samples (#157)
* Add more output sinks to samples

* Rename setupCosmosdbAndSqldbSample parameter to setupAdditionalResourcesForSample

* edit comment

* PR feedback

* Make sqlDocDBName token accessible to setup-secret and add estimated time log for setting up SQL type cosmosdb resource
2019-10-21 10:12:15 -07:00
msandg 5dd07fcc13
fix the DataXGateway Audience field (#156) 2019-10-08 11:12:52 -07:00
Rohit Agrawal - MSFT 535a47f409
For databricks by default disable autoscale (#152) 2019-10-03 12:17:51 -07:00
Kwangje Cho 35c1c4f3ed
- Do a better job of handling the case where there is no batch job to deploy (#144)
- Clean up job names for the samples as for a job which hasn't been deployed, it should be null
2019-09-18 11:00:21 -07:00
Kwangje Cho f99abc8ca0
ARM: for the sample deployment script, pass the servicefabric cluster name to the utility module (#142) 2019-09-12 16:04:27 -07:00
Rohit Agrawal - MSFT 6713833ebd
Fix batch job for databricks (#141)
* Fix batch job for databricks

* move createBlobStorageKeyBroadcastVariable to its own class, refactor resolve storage account key retrieval, add new default storage account environment variable

* rename file

* PR feedback
2019-09-11 16:47:46 -07:00
Rohit Agrawal - MSFT 75e0faa45d
Databricks fix output to blobs (#138)
* Databricks fix output to blobs

* PR feedback

* create function to create broadcast variable and change return type of resolveStorageAccount method to option[string]

* create a method to set Storage Account Key On Hadoop Conf
2019-09-05 15:42:37 -07:00
Rohit Agrawal - MSFT 07aa2e69cb
fix live query involving udf in databricks and add dependency jars for various outputs (#133)
* fix live query involving udf in databricks and add dependency jars for various outputs

* PR feedback
2019-08-27 13:43:52 -07:00
Rohit Agrawal - MSFT 4923294d29
ARM support for databricks (#108)
* ARM support for databricks

* PR feedback

* create databricks in existing vnet, update hdinsight kafka zookeeper vmsize to standard_a4_v2, add port 443 rule for hdinsight
2019-08-12 12:01:11 -07:00
Kwangje Cho 3767ec090b
ARM: assign the writer role to the service AAD app (#109)
* ARM: assign the writer role to the service AAD app

* Refactor Set-AzureAADApiPermission function not to pass the roleId as a param
2019-08-06 10:19:54 -07:00
Kwangje Cho 013aa64dc8
Flow service: Add Blob support (#89)
* The flow service: Add Blob support

* The flow service: Add Blob support

* move the kv secret resolution from the FlattenJobConfig to GenerateJobConfigBatching

* add tests for batch and update based on feedback

* enable sign for the new projects

* merge w/ master

* updated based on feedback

* update based on feedback

* fixing tests

* fixing tests

* fixing tests

* clean up code
2019-07-29 02:29:13 -07:00
Kwangje Cho b53e650010
bug fixes: spark nuspec, iothub sku, simulator service num events (#102) 2019-07-17 10:54:10 -07:00
Vijay Upadya d0631a2728
SQL Output: UI and config gen service updates (#86)
* SQL Output: UI and config gen service updates

* Minor updates to UI based on review feedback.

* Flatterner template update and minor UI tweaks

* Update package version.

* Fix a typo
2019-06-19 17:20:17 -07:00
Anupama Tuli 828c5ab023 Making a few tweaks: updating all files to be LF instead of CRLF. Adding quotes for servicedllname and adding a new line for each of the Dockerfile and finalrun.sh 2019-06-18 19:38:22 -07:00
Anupama Tuli b11b7f428c Refactored the Dockerfile and finalrun.sh such that CICD can be enabled easily by passing in the service name as parameter. Also adding the yaml files for each service which will need the parameters to be passed in prior to deploying the service to the Kubernetes cluster (just as we do for when deploying to the service fabric cluster.) 2019-06-17 22:05:44 -07:00
Kwangje Cho 1db0840859
ARM: remove snapshot from the templates to unblock ARM deployment from master branch (#82) 2019-06-17 15:43:01 -07:00
Kwangje Cho 1688bc9bbf
ARM deployment: add kafka support (#70)
* ARM deployment: add kafka support

* update based on feedback

* update proton version for kafkaDataXDirect template which is missing from a previous commit
2019-05-29 14:35:20 -07:00
Vijay Upadya 89f1e98d42
Update proton-j and jackson lib versions. (#71) 2019-05-29 14:15:44 -07:00
Anupama Tuli 1331ca0699 Updating the line endings to be LF and not CRLF for the DockerFile and the yaml 2019-05-17 09:00:00 -07:00
Anupama Tuli 1412b64391 Adding the required files for creating and deploying the Pod Identity Service. This code also adds in the required DockerFile and finalrun.sh for creating the docker container for Kubernetes scneario 2019-05-17 06:40:53 -07:00
Anupama Tuli 97ed3407b7 Merge branch 'master' of https://github.com/Microsoft/data-accelerator into kubernetes/Integrate 2019-05-17 06:22:38 -07:00
Rohit Agrawal - MSFT 2a6a5bddb6
rev maven package version (#56)
* rev maven package version

* update few more files

* remove filter to include only jar
2019-05-16 10:17:15 -07:00
Anupama Tuli 877d29345c Adding contents for creating docker container for Floe.ManagementService 2019-05-09 11:28:51 -07:00
Kwangje Cho 40803e1273
Add multiple credentials to the AAD app (#39) 2019-04-30 10:06:47 -07:00
Kwangje Cho b449d76a49
expose a new param for installing modules and use the minimumVersion for checking (#34) 2019-04-22 15:05:55 -07:00
Carl Brochu cd46a0bf0c
Sync json files to latest nuget content 2019-04-16 11:09:55 -07:00
Carl Brochu 0f039194b4
Update configgenConfigs.json 2019-04-16 11:09:09 -07:00
kwangjecho 9ffafc73bd
update to use the new sfpkg names (#4) 2019-04-16 03:50:54 -07:00
Dinesh Chandnani 1bf96e340c
Update README.md 2019-04-16 00:28:05 -07:00
Rohit Agrawal c8104966a1 Initial Checkin 2019-04-15 23:57:37 -07:00