Граф коммитов

324 Коммитов

Автор SHA1 Сообщение Дата
Jacob Freck 90c8438700 Bug: fix same cluster id error (#145)
* add-user accept ssh-key path and prompt for password

* remove passive failure for create_pool_if_not_exists

* raise AztkError to overwrite service error message

* raise errors other than pool exists
2017-10-09 14:48:52 -07:00
Jacob Freck cadc2855f3 Bug: reset opposite size param if other is set (#153)
Fix #127
2017-10-09 09:17:26 -07:00
Jacob Freck a2755e3574 Bug: service url error message (#155)
* fixed error message

* fix typo
2017-10-06 14:47:55 -07:00
Jacob Freck 1f73125c04 Bug: fix ssh fail on non-windows (#157) 2017-10-06 14:44:06 -07:00
Timothee Guerin 8cd5e0f38d Don't use absolute path when submitting app (#152) 2017-10-06 09:59:18 -07:00
JS f15d18fd5d Update docs (#148)
* Update 11-custom-scripts.md

* Update 11-custom-scripts.md

* Update 12-docker-image.md

* Update 13-configuration.md

* Update 20-spark-submit.md
2017-10-05 10:13:26 -07:00
JS 5c3b4d67f3 Update README.md (#146) 2017-10-04 16:38:04 -07:00
Jacob Freck dd5a5c938a Feature: ssh directly into container (#142)
* add-user accept ssh-key path and prompt for password

* change ssh command to connect directly to container

* add --host flag for ssh

* update docs

* cleaner checking of config options
2017-10-04 15:51:22 -07:00
Jacob Freck 6bc4dc31aa Bug: fix typo (#137) 2017-10-04 15:06:39 -07:00
Jacob Freck 45a058c26d Feature: add-user accept ssh-key path and prompt for password (#138) 2017-10-04 15:04:36 -07:00
JS 1217d1c489 Feature/yaml space note (#134)
* Update secrets.yaml.template

* added spaces
2017-10-03 22:55:57 -07:00
JS 4fd8772869 Update README.md (#136) 2017-10-03 17:30:09 -07:00
Jacob Freck 9b88913529 Bug: Updated confusing cluster create success message (#80)
* updated confusing success message

* updated message and whitespace

* remove old file
2017-10-03 16:43:22 -07:00
Jacob Freck ef2910ceef Bug: simplify secrets loading (#133)
* create aztklib object, move clusterlib and joblib to objects, update cli endpoints

* fix errors in joblib

* refactor SecretsConfig loading, refactor batch and blob client loading

* fix tests

* fix test and pylint errors

* rename clusterlib to cluster and rename joblib to job

* whitespace

* rename aztklib to aztk and fix whitespace

* fix get command

* fixed issues with docker_auth and tests
2017-10-03 16:22:48 -07:00
Timothee Guerin f51f613fe7 Enable docker authentication for private images (#92)
* Docker login working

* fix docker image ignored

* Fix job need to wait for pool to be created

* Fix

* fix

* Fix

* Fix

* Update docs

* Fix cr

* Fix merge issue
2017-10-03 13:30:12 -07:00
Jacob Freck 952d8c85a9 Bug: prompt for confirmation of cluster id on delete (#109)
* prompt for confirmation of cluster id on delete

* add force flag
2017-10-03 09:37:47 -07:00
Jacob Freck 37c5d873a0 Feature: Support for multiple custom scripts and master only scrips (#93)
* added IS_MASTER environment variable, moved custom script execution to python

* allow for custom_scripts to check if executing on spark master

* allow for multiple custom scripts

* updated docs, fixed overwrite bug, more precise typing

* added support for script ordering, consolidated file uploading

* rename custom script on upload to prevent conflicts

* removed unnecessary import

* added environment variable IS_MASTER back

* white space

* updated docs

* updated docs

* fix pylint errors

* undo previous commit error

* clearer docs

* removed cli custom script functionality, fixed no custom scirpts bug

* fixed pylint errors

* changed location to runOn, removed unused parameters, add error checking

* removed unused parameter

* added information about storage account

* remove unused functions to upload scripts

* chagned dtde to aztk
2017-10-02 13:20:11 -07:00
JS d3a4f9c559 Feature/update docker repo (#130)
* Update cluster.yaml

* update dockerfile, node_scripts, and constants

* update docker-image/README.md
2017-09-30 22:29:53 -07:00
JS e76c5fbf7e more informative error (#129)
* more informative error

Make error more informative when --no-wait is True and username is set

* Update clusterlib.py
2017-09-30 19:33:48 -07:00
JS f158ceafc5 Docs/update (#128)
* Update README.md

* Update 10-clusters.md

* Update README.md

* Update README.md

* Update README.md
2017-09-30 05:02:25 -04:00
JS 57cfa9dd1d Docs/update (#106)
* Updates to README.md

* typos

* Update 00-getting-started.md

* Update README.md

* Update 30-cloud-storage.md

* faq in readme

* Update 13-configuration.md

* switch to use underscore for consistency

* Update 10-clusters.md

* Update README.md

* Update README.md

* fixes

* fix broken link

* Update 20-spark-submit.md

* update to aztk on README

* aztk fix

* Update 00-getting-started.md

* Update 10-clusters.md

* Update 12-docker-image.md

* Update 13-configuration.md

* Update 20-spark-submit.md

* Update 30-cloud-storage.md

* Update README.md

* Update 10-clusters.md

* vm size link

* dummy commit1

* dummy commit 2

* Update 00-getting-started.md

* Update 13-configuration.md
2017-09-29 22:23:14 -04:00
Jacob Freck eb2a221cca Bug: rename dtde to aztk, rename thunderbolt to aztk, rename azb to aztk (#125)
* rename dtde to aztk, rename thunderbolt to aztk

* rename azb to aztk, fix bug with --wait

* changed exception to AztkError

* Fix issue with wait flag if not set
2017-09-29 17:13:22 -07:00
Jacob Freck dabcda0ca1 Bug: fixed --no-wait not being respected (#108)
* fixed bug where --no-wait was not respected

* dummy commit

* undo dummy commit
2017-09-29 14:52:07 -07:00
Jacob Freck c2172b5f64 Bug: add spark conf defaults (#115)
* added default spark conf files to config/

* refactored loading and cleanup of spark conf files

* curated spark conf files and added docs
2017-09-29 13:03:11 -07:00
Jacob Freck 5f12ff66dc Feature: prompt for password (#116)
* prompt for password on cluster create, add secrets validation, add custom ssh priv key

* whitespace

* fixed error

* remove ssh_priv_key from secrets template, add docs

* updated error message, fixed typo

* fixed innacurate message
2017-09-29 11:16:54 -07:00
Jacob Freck 5c96fc6bf0 Bug: fixed issue with submit paths (#118) 2017-09-29 11:02:25 -07:00
Jacob Freck 750b0848aa Bug: change default cluster create experience to require --id cli parameter (#112)
* change default cluster create experience to require --id cli parameter

* updated error message to reflect intended workflow
2017-09-29 10:26:43 -07:00
Jacob Freck fe9bb7be83 Bug: better error message for yaml syntax issues (#113) 2017-09-29 10:23:43 -07:00
Jacob Freck 1119a8b0be Bug: clearer error message for bad vm size (#121)
* clearer error message for bad vm size

* leverage util print_batch_exception_error for better printing
2017-09-29 10:19:43 -07:00
Jacob Freck b7e41739b1 Bug: size parameters don't override if 0 (#123)
* fix issue with size parameter not respected if 0

* better error message description
2017-09-29 10:17:43 -07:00
Jacob Freck 9199e1e5ca Bug: fixed incorrect spark configuration destination path constant (#100) 2017-09-26 09:46:56 -07:00
Jacob Freck d40a4c577c Bug: ssh key required in secrets.yaml (#95)
* allow ssh_key to remain unspecified in secrets.yaml

* delete unecessary print

* removed extra white space
2017-09-25 08:45:27 -07:00
Jacob Freck 0742f438fc Bug: removed printed errors when spark conf files don't exist (#97) 2017-09-25 08:41:49 -07:00
Jacob Freck 6543d3c94c fix typo (#96) 2017-09-25 08:21:03 -07:00
Jacob Freck 859c6fa6c8 Feature: ssh configuration file support (#77)
* added ssh.yaml configuration file

* added jupyter support, better format for ssh.yaml instructions

* refactored read_conf_file method

* rename master ui to web ui

* fixed typo in comments

* added documentation for ssh.yaml

* refactored merge and _merge_dict methods

* changed default ssh experience to require --id cli parameter

* renamed conflicting documentation

* improved docs, changed default port forwarding to standard ports

* fix typo
2017-09-21 17:45:57 -07:00
Jacob Freck 4ae20caa14 Feature: added azb spark init command (#79)
* added command azb spark init

* azb spark init no longer overwrites existing files

* added documentation for azb spark init

* init renames secrets template file automatically

* updated docs to reflect changed workflow

* fixed bug where file rename would fail if file exists
2017-09-20 15:36:01 -07:00
Jacob Freck 5370bd5761 Bug: Clean up error messages (#85)
* improved error handling, added exception types

* AzureApiInitError inherrits from ThunderboltError
2017-09-20 15:26:23 -07:00
Jacob Freck 611cc584cf Feature: config files (#65)
* moved secrets config template to config directory

* updated new default file location for secrets.cfg

* added ConfigObject class

* added template for cluster config file

* updated .gitignore to include cluster.yaml

* updated cluster.yaml.template to include more fields

* added read_config_file method to ConfigObject, modified cluster_create.py to use cluster.yaml configuration by default

* added pyyaml to requirements.txt

* added documentation for cluster.yaml config file

* added constant for default cluster.yaml location

* changed secrets.cfg to secrets.yaml, added SecretsConfig object

* removed secrets.cfg.template

* improved error messages for bad secrets.yaml config

* changed config/ directory to .thunderbolt/

* remove old unused code

* removed unncessary ignored file

* renamed configuration class to ClusterConfig and added merge method with error checking

* collapsed cluster.yaml file

* updated cluster.yaml.template to new collapsed format

* better descriptions of cluster config settings

* added error checking to cluster config settings

* added back accidentally removed code

* removed template for cluster.yaml, added cluster.yaml

* renamed .thunderbolt/ to config/, added defaults for cluster.yaml

* fixed typo in secrets.yaml.template

* added .thunderbolt/ directory to .gitignore

* fixed bug when adding user with ssh_key

* code cleanup

* updated docs

* removed redundant docs, added instructions to secrets.yaml.template

* fixed typo

* refactored read_config_file method and changed default cluster id

* removed unnecessary ignored file

* changed format for cluster.yaml instructions

* added back docker-repo support

* refactored merge and merge_dict methods
2017-09-19 14:08:26 -07:00
Jacob Freck 43dd8e57f1 Bug: Move commands under app to cluster (#82)
* moved azb spark app commands to azb spark cluster

* fixed docs to reflect new cli structure

* renamed commands for clarity
2017-09-19 08:29:23 -07:00
Jacob Freck bd2878ecdb Feature: upload custom spark.conf (#72)
* added load and remove spark configuration files functions to config.py

* added setup_conf method to spark.py to move spark conf files to proper location

* changed expected location for spark conf files to .thunderbolt/

* fixed typo
2017-09-18 14:34:37 -07:00
JS 03baff6a26 Feature: custom docker images (#68)
* symlink /home/spark-version to /home/spark-current

* working checkpoint

* working checkpoint

* docker readme update

* removed deprecated dockerfile

* docker cli + docs for docker

* docs update

* docs update

* remove unneeded comments

* docker version and docs updates, misc fixes
2017-09-15 20:57:16 -07:00
Amol Thacker 551d8a1a45 Fix documentation links and typos (#56) 2017-09-11 21:34:49 -07:00
JS 636cd2e1d6 Update README.md (#54)
* Update README.md

update readme for a more descriptive introduction

* upper case S for spark

* fix
2017-09-11 13:41:33 -07:00
JS 5edd120c52 fix broken links in docs/README (#55)
* fix broken links in docs/README

* typo
2017-09-11 13:17:46 -07:00
Pablo Selem 219cf412c0 update docs with new name 'Thunderbolt' (#53)
* update docs with new name 'Thunderbolt'

* update instructions around pip for clarity
2017-08-23 11:37:13 -07:00
JS cf6ceff6f7 removed deprecated wasb-dsvm custom script & removed depcredated wasb docker image (#49) 2017-08-23 11:29:27 -04:00
JS fdd87c0a99 Feature/samples (#50)
* added snapshot of examples from spark-2.2.0

* jupyter samples

* fixed bug

* rename 'pyspark' to 'jupyter'

* curated samples

* Readme update for samples
2017-08-22 18:36:25 -04:00
Timothee Guerin 5149eacb22 Fix showing password in plain text (#52) 2017-08-22 09:54:18 -07:00
Pablo Selem ae26453ba5 Feature/docs (#51)
* update configuration.cfg to secrets.cfg

* fix node count pretty print

* add more unit tests

* configure pytest to only run tests from the 'tests' directory

* add missing space

* remove static call to loading secrets file

* initial docs layout

* rename files and add images to getting started

* fix typos

* Add next steps to clusters docs

* Add custom scripts doc

* Initial text for spark submit

* fix merge conflict

* finish spark submit docs

* cloud storage docs

* fix typo in storage docs

* refactor some docs

* pr feedback

* add link to Getting Started in creating account details in main readme
2017-08-22 09:13:10 -07:00
JS 577ff296b5 Feature/wasb (#47)
* update configuration.cfg to secrets.cfg

* fix node count pretty print

* add more unit tests

* configure pytest to only run tests from the 'tests' directory

* add missing space

* remove static call to loading secrets file

* initial commit for automatic wasb support

* wasb native support

* fix for wasb

* bug fix in azure_api

* clean up setup_wasb

* README for wasb

* README updates for wasb

* configure wasb creds in start task environment variables

* add missing merged files

* revert back to working state

* added storage account suffix

* removed zip to secrets
2017-08-22 11:57:45 -04:00