Граф коммитов

146 Коммитов

Автор SHA1 Сообщение Дата
Timothee Guerin 97ac1fc76f udate cluster 2017-09-22 11:39:18 -07:00
Timothee Guerin 07bd12d797 Provide a list of ports to open on cluster create 2017-09-22 11:13:09 -07:00
Timothee Guerin c31ab84fb5 Merge master 2017-09-22 10:51:43 -07:00
Jacob Freck 859c6fa6c8 Feature: ssh configuration file support (#77)
* added ssh.yaml configuration file

* added jupyter support, better format for ssh.yaml instructions

* refactored read_conf_file method

* rename master ui to web ui

* fixed typo in comments

* added documentation for ssh.yaml

* refactored merge and _merge_dict methods

* changed default ssh experience to require --id cli parameter

* renamed conflicting documentation

* improved docs, changed default port forwarding to standard ports

* fix typo
2017-09-21 17:45:57 -07:00
Jacob Freck 4ae20caa14 Feature: added azb spark init command (#79)
* added command azb spark init

* azb spark init no longer overwrites existing files

* added documentation for azb spark init

* init renames secrets template file automatically

* updated docs to reflect changed workflow

* fixed bug where file rename would fail if file exists
2017-09-20 15:36:01 -07:00
Jacob Freck 5370bd5761 Bug: Clean up error messages (#85)
* improved error handling, added exception types

* AzureApiInitError inherrits from ThunderboltError
2017-09-20 15:26:23 -07:00
Jacob Freck 611cc584cf Feature: config files (#65)
* moved secrets config template to config directory

* updated new default file location for secrets.cfg

* added ConfigObject class

* added template for cluster config file

* updated .gitignore to include cluster.yaml

* updated cluster.yaml.template to include more fields

* added read_config_file method to ConfigObject, modified cluster_create.py to use cluster.yaml configuration by default

* added pyyaml to requirements.txt

* added documentation for cluster.yaml config file

* added constant for default cluster.yaml location

* changed secrets.cfg to secrets.yaml, added SecretsConfig object

* removed secrets.cfg.template

* improved error messages for bad secrets.yaml config

* changed config/ directory to .thunderbolt/

* remove old unused code

* removed unncessary ignored file

* renamed configuration class to ClusterConfig and added merge method with error checking

* collapsed cluster.yaml file

* updated cluster.yaml.template to new collapsed format

* better descriptions of cluster config settings

* added error checking to cluster config settings

* added back accidentally removed code

* removed template for cluster.yaml, added cluster.yaml

* renamed .thunderbolt/ to config/, added defaults for cluster.yaml

* fixed typo in secrets.yaml.template

* added .thunderbolt/ directory to .gitignore

* fixed bug when adding user with ssh_key

* code cleanup

* updated docs

* removed redundant docs, added instructions to secrets.yaml.template

* fixed typo

* refactored read_config_file method and changed default cluster id

* removed unnecessary ignored file

* changed format for cluster.yaml instructions

* added back docker-repo support

* refactored merge and merge_dict methods
2017-09-19 14:08:26 -07:00
Jacob Freck 43dd8e57f1 Bug: Move commands under app to cluster (#82)
* moved azb spark app commands to azb spark cluster

* fixed docs to reflect new cli structure

* renamed commands for clarity
2017-09-19 08:29:23 -07:00
Jacob Freck bd2878ecdb Feature: upload custom spark.conf (#72)
* added load and remove spark configuration files functions to config.py

* added setup_conf method to spark.py to move spark conf files to proper location

* changed expected location for spark conf files to .thunderbolt/

* fixed typo
2017-09-18 14:34:37 -07:00
JS 03baff6a26 Feature: custom docker images (#68)
* symlink /home/spark-version to /home/spark-current

* working checkpoint

* working checkpoint

* docker readme update

* removed deprecated dockerfile

* docker cli + docs for docker

* docs update

* docs update

* remove unneeded comments

* docker version and docs updates, misc fixes
2017-09-15 20:57:16 -07:00
Amol Thacker 551d8a1a45 Fix documentation links and typos (#56) 2017-09-11 21:34:49 -07:00
Timothee Guerin b6491a0ea9 Added doc for port forwarding 2017-09-11 18:00:19 -07:00
Timothee Guerin 62dbf4e203 Multiple 2017-09-11 16:53:33 -07:00
Timothee Guerin 5ca4b83f28 Forward additional ports 2017-09-11 16:48:07 -07:00
JS 636cd2e1d6 Update README.md (#54)
* Update README.md

update readme for a more descriptive introduction

* upper case S for spark

* fix
2017-09-11 13:41:33 -07:00
JS 5edd120c52 fix broken links in docs/README (#55)
* fix broken links in docs/README

* typo
2017-09-11 13:17:46 -07:00
Pablo Selem 219cf412c0 update docs with new name 'Thunderbolt' (#53)
* update docs with new name 'Thunderbolt'

* update instructions around pip for clarity
2017-08-23 11:37:13 -07:00
JS cf6ceff6f7 removed deprecated wasb-dsvm custom script & removed depcredated wasb docker image (#49) 2017-08-23 11:29:27 -04:00
JS fdd87c0a99 Feature/samples (#50)
* added snapshot of examples from spark-2.2.0

* jupyter samples

* fixed bug

* rename 'pyspark' to 'jupyter'

* curated samples

* Readme update for samples
2017-08-22 18:36:25 -04:00
Timothee Guerin 5149eacb22 Fix showing password in plain text (#52) 2017-08-22 09:54:18 -07:00
Pablo Selem ae26453ba5 Feature/docs (#51)
* update configuration.cfg to secrets.cfg

* fix node count pretty print

* add more unit tests

* configure pytest to only run tests from the 'tests' directory

* add missing space

* remove static call to loading secrets file

* initial docs layout

* rename files and add images to getting started

* fix typos

* Add next steps to clusters docs

* Add custom scripts doc

* Initial text for spark submit

* fix merge conflict

* finish spark submit docs

* cloud storage docs

* fix typo in storage docs

* refactor some docs

* pr feedback

* add link to Getting Started in creating account details in main readme
2017-08-22 09:13:10 -07:00
JS 577ff296b5 Feature/wasb (#47)
* update configuration.cfg to secrets.cfg

* fix node count pretty print

* add more unit tests

* configure pytest to only run tests from the 'tests' directory

* add missing space

* remove static call to loading secrets file

* initial commit for automatic wasb support

* wasb native support

* fix for wasb

* bug fix in azure_api

* clean up setup_wasb

* README for wasb

* README updates for wasb

* configure wasb creds in start task environment variables

* add missing merged files

* revert back to working state

* added storage account suffix

* removed zip to secrets
2017-08-22 11:57:45 -04:00
Pablo Selem 4822172b6a Feature/secrets.cfg (#43)
* update configuration.cfg to secrets.cfg

* fix node count pretty print

* add more unit tests

* configure pytest to only run tests from the 'tests' directory

* add missing space

* remove static call to loading secrets file
2017-08-21 12:36:45 -07:00
JS 213f9226b0 removed dsvm code (#48) 2017-08-21 02:08:40 -04:00
Timothee Guerin b21cae0cc2 Fix: windows path given in the command line (#41)
* Fix windows path

* Upload custom-script

* Custom script works with windows path

* Fix util

* 24 Hours
2017-08-19 13:47:44 -07:00
Timothee Guerin d552624705 Feature: read/tail spark submit logs (#40)
* Tail log command

* tail log is working

* Cleanup code

* fix cr

* Fix cr

* Submit

* Comment

* fix tail

* Sleep
2017-08-18 12:35:04 -07:00
JS d94c43ed82 Feature: docker
Feature: docker
2017-08-17 09:21:00 -07:00
JS fbfdaf489d remove deprecated instructions (#39)
* remove deprecated instructions

* Update README.md
2017-08-10 11:00:43 -07:00
Timothee Guerin ca1517acfc Feature: logging instead of prints and more sdk like (#34)
* Getting started on logging

* new logger

* Cluster list and get api like

* SSH sdk like

* Job submit log

* util

* Cluster create log

* fix cr
2017-08-02 08:43:26 -07:00
Timothee Guerin aea5e36690 Feature: Setup user with ssh public key instead of password (#32)
* Login with ssh public key

* Update to have ssh-key command

* Fix readme

* Fix Cr

* remove unused files

* Change head to master
2017-07-27 10:45:31 -07:00
Timothee Guerin 9236bc883b Fix software circular (#35) 2017-07-24 13:02:46 -07:00
Timothee Guerin 4fe6c66190 Only list spark clusters (#31)
* Only list spark clusters

* reformat

* reformat

* reformat

* fix cr

* Spark use enum

* Version

* Fix
2017-07-24 09:17:42 -07:00
Timothee Guerin 17d09ee659 Force convert to bool (#33) 2017-07-21 08:52:28 -07:00
Timothee Guerin 9e45f78600 Simplify the commands to all be under azb and make windows experience better (#29)
* Added some commands

* More

* Missing file

* Refactor

* update

* Constant exe name

* List and get command

* Added user

* update cluster create, and job submit

* Remove legacy scripts

* Fix cr comments

* remove leftover print
2017-07-19 10:48:23 -07:00
Timothee Guerin d92df3c0be Initial structure for test framework and CI (#30)
* Command builder tests

* Setup travis

* Update to travis

* Update to travis

* Update to travis

* Error only

* builder to array
2017-07-18 14:07:11 -07:00
Timothee Guerin 4a1d9adf25 Revert "Revert "Feature: Setup the cluster with the start task." (#26)" (#28)
This reverts commit dc1618ae94.
2017-07-14 12:00:47 -04:00
JS dc1618ae94 Revert "Feature: Setup the cluster with the start task." (#26) 2017-07-13 20:57:04 -07:00
Timothee Guerin d943237c8e Merge pull request #24 from Azure/feature/start-task-master
Feature: Setup the cluster with the start task.
2017-07-13 17:16:59 -07:00
Timothee Guerin 6a2a54c2f1 Typo 2017-07-13 15:23:49 -07:00
Timothee Guerin 9a4eccda1f fix spark master ui port 2017-07-13 15:13:34 -07:00
Timothee Guerin 8f5757cea3 Fix broken code after CR 2017-07-13 13:56:52 -07:00
Timothee Guerin 27b14e6adc Finish reviewing 2017-07-13 13:23:19 -07:00
Timothee Guerin aeecacc194 Some cr fixes 2017-07-13 12:44:10 -07:00
Timothee Guerin b72a9bc935 Move juypter wd to be empty 2017-07-12 15:12:25 -07:00
Timothee Guerin f7b22be1dc fix 2017-07-12 13:57:24 -07:00
Timothee Guerin 3404579b98 Pick master 2017-07-12 13:24:40 -07:00
Timothee Guerin 9dc24bd7f6 Pep8 config file 2017-07-12 13:23:16 -07:00
Timothee Guerin 583eefec79 Command builder 2017-07-12 13:10:13 -07:00
Timothee Guerin de32bf6590 Added editorconfig and settings 2017-07-12 12:41:15 -07:00
Timothee Guerin f334de60c7 Cleanup cluster lib 2017-07-12 12:33:18 -07:00