AZTK powered by Azure Batch: On-demand, Dockerized, Spark Jobs on Azure

azure-batch azure-storage docker spark spark-jobs

Перейти к файлу

JS Tan 6ea20a76ba Merged PR 2: Merge ssh-update to master ## ssh update: enabled jupyter, attached user to the cluster lvl (as opposed to job level) ## features: * new cli cmd: spark-cluster-create-user (create user is needed because we cannot automatically create a user on pool creation since we don't necessarily wait for cluster creation to finish) * new cli cmd: spark-cluster-jupyter * new cli cmd: spark-cluster-webui * spark-cluster-ssh does not tunnel any ports by default		2017-04-17 05:10:41 +00:00
bin	Merged PR 2: Merge ssh-update to master	2017-04-17 05:10:41 +00:00
example-jobs	configured bin dir for scripts + setup.py	2017-04-12 08:14:31 +00:00
redbull	Merged PR 2: Merge ssh-update to master	2017-04-17 05:10:41 +00:00
.gitignore	configured bin dir for scripts + setup.py	2017-04-12 08:14:31 +00:00
README.md	readme.md	2017-04-13 01:47:44 +00:00
requirements.txt	az_spark_start + az_spark_submit	2017-04-07 20:55:05 +00:00
setup.py	Merged PR 2: Merge ssh-update to master	2017-04-17 05:10:41 +00:00

README.md

redbull

Run Spark on Azure Batch

Develop

Create a virtual environment (either virtualenv or with conda)
Use setuptools:
```
python3 setup.py develop
```