f51f613fe7
Enable docker authentication for private images ( #92 )
...
* Docker login working
* fix docker image ignored
* Fix job need to wait for pool to be created
* Fix
* fix
* Fix
* Fix
* Update docs
* Fix cr
* Fix merge issue
2017-10-03 13:30:12 -07:00
37c5d873a0
Feature: Support for multiple custom scripts and master only scrips ( #93 )
...
* added IS_MASTER environment variable, moved custom script execution to python
* allow for custom_scripts to check if executing on spark master
* allow for multiple custom scripts
* updated docs, fixed overwrite bug, more precise typing
* added support for script ordering, consolidated file uploading
* rename custom script on upload to prevent conflicts
* removed unnecessary import
* added environment variable IS_MASTER back
* white space
* updated docs
* updated docs
* fix pylint errors
* undo previous commit error
* clearer docs
* removed cli custom script functionality, fixed no custom scirpts bug
* fixed pylint errors
* changed location to runOn, removed unused parameters, add error checking
* removed unused parameter
* added information about storage account
* remove unused functions to upload scripts
* chagned dtde to aztk
2017-10-02 13:20:11 -07:00
d3a4f9c559
Feature/update docker repo ( #130 )
...
* Update cluster.yaml
* update dockerfile, node_scripts, and constants
* update docker-image/README.md
2017-09-30 22:29:53 -07:00
eb2a221cca
Bug: rename dtde to aztk, rename thunderbolt to aztk, rename azb to aztk ( #125 )
...
* rename dtde to aztk, rename thunderbolt to aztk
* rename azb to aztk, fix bug with --wait
* changed exception to AztkError
* Fix issue with wait flag if not set
2017-09-29 17:13:22 -07:00
bd2878ecdb
Feature: upload custom spark.conf ( #72 )
...
* added load and remove spark configuration files functions to config.py
* added setup_conf method to spark.py to move spark conf files to proper location
* changed expected location for spark conf files to .thunderbolt/
* fixed typo
2017-09-18 14:34:37 -07:00
03baff6a26
Feature: custom docker images ( #68 )
...
* symlink /home/spark-version to /home/spark-current
* working checkpoint
* working checkpoint
* docker readme update
* removed deprecated dockerfile
* docker cli + docs for docker
* docs update
* docs update
* remove unneeded comments
* docker version and docs updates, misc fixes
2017-09-15 20:57:16 -07:00
fdd87c0a99
Feature/samples ( #50 )
...
* added snapshot of examples from spark-2.2.0
* jupyter samples
* fixed bug
* rename 'pyspark' to 'jupyter'
* curated samples
* Readme update for samples
2017-08-22 18:36:25 -04:00
577ff296b5
Feature/wasb ( #47 )
...
* update configuration.cfg to secrets.cfg
* fix node count pretty print
* add more unit tests
* configure pytest to only run tests from the 'tests' directory
* add missing space
* remove static call to loading secrets file
* initial commit for automatic wasb support
* wasb native support
* fix for wasb
* bug fix in azure_api
* clean up setup_wasb
* README for wasb
* README updates for wasb
* configure wasb creds in start task environment variables
* add missing merged files
* revert back to working state
* added storage account suffix
* removed zip to secrets
2017-08-22 11:57:45 -04:00
213f9226b0
removed dsvm code ( #48 )
2017-08-21 02:08:40 -04:00
d94c43ed82
Feature: docker
...
Feature: docker
2017-08-17 09:21:00 -07:00
ca1517acfc
Feature: logging instead of prints and more sdk like ( #34 )
...
* Getting started on logging
* new logger
* Cluster list and get api like
* SSH sdk like
* Job submit log
* util
* Cluster create log
* fix cr
2017-08-02 08:43:26 -07:00
9e45f78600
Simplify the commands to all be under azb and make windows experience better ( #29 )
...
* Added some commands
* More
* Missing file
* Refactor
* update
* Constant exe name
* List and get command
* Added user
* update cluster create, and job submit
* Remove legacy scripts
* Fix cr comments
* remove leftover print
2017-07-19 10:48:23 -07:00
4a1d9adf25
Revert "Revert "Feature: Setup the cluster with the start task." ( #26 )" ( #28 )
...
This reverts commit dc1618ae94
2017-07-14 12:00:47 -04:00
dc1618ae94
Revert "Feature: Setup the cluster with the start task." ( #26 )
2017-07-13 20:57:04 -07:00
6a2a54c2f1
Typo
2017-07-13 15:23:49 -07:00
9a4eccda1f
fix spark master ui port
2017-07-13 15:13:34 -07:00
27b14e6adc
Finish reviewing
2017-07-13 13:23:19 -07:00
aeecacc194
Some cr fixes
2017-07-13 12:44:10 -07:00
b72a9bc935
Move juypter wd to be empty
2017-07-12 15:12:25 -07:00
f7b22be1dc
fix
2017-07-12 13:57:24 -07:00
3404579b98
Pick master
2017-07-12 13:24:40 -07:00
9dc24bd7f6
Pep8 config file
2017-07-12 13:23:16 -07:00
1ef325056a
Starting master as a worker too
2017-07-12 12:22:08 -07:00
96f94e7714
job submit is working
2017-07-12 11:33:42 -07:00
8b45359ff6
Jupyter working
2017-07-12 10:52:16 -07:00
dc8f397494
Fix juypiter?
2017-07-11 14:52:40 -07:00
e9c7f74e79
more
2017-07-11 14:25:17 -07:00
222821cdb2
Use master node id
2017-07-11 09:05:22 -07:00
d60f647540
more fixes
2017-07-10 17:03:04 -07:00
ef3d84853e
Use popen
2017-07-10 16:25:14 -07:00
a4026fa702
Setup spark from the start task
2017-07-10 16:04:17 -07:00
0385df991f
Working
2017-07-10 14:20:50 -07:00
58201d1b0d
Refactor
2017-07-10 12:33:49 -07:00
Timothee Guerin
Jacob Freck
JS
Timothee Guerin