4b2acc8491
Fix: add toolkit to sdk docs and example ( #602 )
2018-06-11 15:34:29 -07:00
7d7a814c50
Fix: fix typos ( #595 )
2018-06-07 09:57:43 -07:00
88d04195ec
Feature: add cluster list quiet flag, ability to compose with delete ( #581 )
...
* add quiet flag, ability to compose with delete
* log.print instead of print
* add some docs
2018-06-06 16:03:34 -07:00
f16aac091e
Feature: pure python ssh ( #577 )
...
* forward multiple ports
* plumb through cli
* continue cli implementation
* fixes
* pylint ignore
* spacing
* remove debug stuff, fix bug
* add --internal support
* add to init
* add comment
* remove nesting
* add logging
* add some docs
2018-06-04 17:16:51 -07:00
af449dc194
Feature: add node run command ( #572 )
...
* add node run command
* whitespace
* add node-run doc
* add host flag
* refactor, print->log
* generated username
* more secure random
* better handling of find node, type conversion
* add generate_user_on_node
* docs update
* fix docs
* remove duplicate import, sort
2018-06-04 13:58:33 -07:00
49a890a5df
Fix: switch create user to pool wide ( #574 )
...
* switch create user to pool wide
* fix password bug
* update doc
2018-05-30 13:36:59 -07:00
8fea9ce092
Feature: Disable scheduling on group of nodes ( #540 )
2018-05-30 13:02:48 -07:00
02f336b0a0
Feature: New Models design with auto validation, default and merging ( #543 )
2018-05-30 09:07:09 -07:00
f6735cc6dd
Feature: Support passing of remote executables via aztk spark cluster submit ( #549 )
2018-05-24 10:29:23 -07:00
1527929e30
Feature: TensorflowOnSpark python plugin ( #525 )
...
* initial commit
* update
* update
* add gpu support
* remove comment
* change class to function
* fix merge issue
* add some docs
2018-05-21 13:22:55 -07:00
4d4916e349
Release: v0.7.0 ( #535 )
...
* changelog and migration guide
* update version, links, and doc
* update links
* add docs version
* update changelog
2018-05-01 18:44:26 -07:00
7a7e63c54f
Feature: New Toolkit configuration ( #507 )
2018-05-01 16:36:44 -07:00
9bc76396bc
Docs: Added worker on master docs ( #531 )
2018-05-01 14:40:31 -07:00
779bffb2da
Feature: refactor docker images ( #510 )
...
* add spark2.3.0 hadoop2.8.3 dockerfile
* start update to docker image
* add SPARK_DIST_CLASSPATH to bashrc, source .bashrc in docker run
* add maven install for jars
* docker image update and code fix
* add libthrift (still broken)
* start image refactor, build from source,
* add refactor to r base image
* finish refactor r image
* add storage jars and deps
* exclude netty to get rid of dependency conflict
* add miniconda image
* update 2.2.0 base, anaconda image
* remove unused cuda-8.0 image
* start pipenv implementation
* miniconda version arg
* update anaconda and miniconda image
* style
* pivot to virtualenv
* remove virtualenv from path when submitting apps
* flatten layers
* explicit calls to aztk python instead of activating virtualenv
* update base, miniconda, anaconda
* add compatibility version for base aztk images
* typo fix
* update pom
* update environment variable name
* update environment variables
* add anaconda images base & gpu
* update gpu and miniconda base images
* create venv in cluster create
* update base docker files, remove virtualenv
* fix path
* add exclusion to base images
* update r images
* delete python images (in favor of anaconda and miniconda)
* add miniconda gpu images
* update comment
* update aztk_version_compatibility to dokcer image version
* add a build script
* virutalenv->pipenv, add pipfile & pipfile.lock remove secretstorage
* aztk/staging->aztk/spark
* remove jars, add .null to keep directory
* update pipfile, update jupyter and jupyterlab
* update default images
* update base images to fix hdfs
* update build script with correct path
* add spark1.6.3 anaconda, miniconda, r base and gpu images
* update build script to include spark1.6.3
* mkdir out
* exclude commons lang and slf4j dependencies
* mkdir out
* no fail if dir exists
* update node_scripts
* update env var name
* update env var name
* fix the docker_repo docs
* master->0.7.0
2018-04-30 17:19:01 -07:00
c98df7d1df
Feature: Added custom scripts functionality for plugins with the cli(Deprecate custom scripts) ( #517 )
2018-04-27 10:31:24 -07:00
e361c3b0b3
Feature: Readthedocs support ( #497 )
2018-04-26 14:03:45 -07:00
b8a3fccaf0
Fix: AZTK_IS_MASTER not set on worker and failing ( #506 )
...
* Fix: AZTK_IS_MASTER_NOT_SET
* Update jupyter lab too
* update jupyterlab target role
* True false doc
2018-04-24 12:14:47 -07:00
de7898334c
Feature: Plugin V2: Running plugin on host ( #461 )
2018-04-23 17:20:43 -07:00
2e995b4899
Feature: spark ui proxy plugin ( #467 )
...
* initial commit
* add args
* add docs
* change default plugins
* update ssh cli ui, remove plugin name
* change conditional
* update docs to include jupyterlab
* remove spark_ui_proxy as default plugin
2018-04-23 12:12:31 -07:00
7ef721f0c1
Feature: getting started script ( #475 )
...
* initial changes for getting started scripts
* add temp error handling
* rename file - fix typo
* add debug strings
* add handling for existing user
* WIP: wait for subprocess to complete to get exit code
* WIP: handle existing user and refactor code
* WIP: add missing return statements
* WIP: fix typo
* start sdk refactor
* mostly working create
* working happy create path
* handle errors for vnet, aad application
* make account setup interactive
* add prompt
* add docs
* rename account_setup_refac to account_setup
* add some logging
* pip install msrest, azure-cli-core, import issues
* remove in script pip, add shell wrapper program
* ellipsis to period
* update branch name for account_setup.sh
* docstring
* retry resource group creation
* fix typo, update retry
* explicitly set output location
* wget overwrite flag, docs update
* add prompt for multi tenants
* fix bug with batch account creation
* add spinner, print statements, fix formatting bug
* fix param bug
2018-04-11 13:27:55 -07:00
44a07654aa
Feature: spark debug tool ( #455 )
...
* start implementation of cluster debug utility
* update debug program
* update debug
* fix output directory structure
* cleanup output, add error checking
* sort imports
* start untar
* extract tar
* add debug.py to pylintc ignore, line too long
* crlf->lf
* add app logs
* call get_spark_app_logs, typos
* add docs
* remove debug.py from pylintrc ignore
* added debug.py back to pylint ignore
* change pylint ignore
* remove commented log
* update cluster_run
* refactor cluster_copy
* update debug, add spinner for run and copy
* make new sdk cluster_download endpoint
2018-04-09 15:02:43 -07:00
1eaa1b6e42
Feature: add internal flag to node commands ( #482 )
...
* add internal ssh flag
* add --internal flag to cluster get
* cluster run internal flag
* fix add command back
* cluster copy internal
* fix method params
* fix method params
* add debug statement
* fix params
* remove debug statement
* fixes
* add debug statement
* remove debug statement
* add hostname to /etc/hosts
* remove hostname from /etc/hosts
* add sdk docs for internal switch in cluster run and copy
2018-04-06 15:59:13 -07:00
4ef3dd09df
Bug: add spark.history.fs.logDirectory to required keys ( #456 )
...
* add spark.history.fs.logDirectory to requried keys
* add spark_event_log_enabled_key to required_keys
* docs, add history server config to spark-defaults.conf
* fix bad logic
* crlf->lf
2018-04-05 14:11:35 -07:00
82ad0296af
Bug: add gitattributes file ( #470 )
...
Bug: line endings, add gitattributes file
2018-04-04 13:44:26 -07:00
8aa1843f23
Feature: managed storage for clusters and jobs ( #443 )
...
* add in storage management for clusters, jobs
* add warning logs on cli delete
* whitespace
* add keep-logs flag
* add docs on storage lifetime
2018-03-20 10:45:49 -07:00
4be5ac2f44
Fix job configuration option for `aztk spark job submit` command ( #435 )
...
`--job-conf` option mentioned in the docs wasn't working.
CLI help was showing that option is named `--configuration-c`
which seems to be a result of a missing comma in option definition.
2018-03-13 11:07:48 -07:00
17755e0ffa
Feature: Basic Cluster and Job Submission SDK Tests ( #344 )
...
* add initial cluster tests
* add cluster tests, add simple job submission test scenario
* sort imports
* fix job tests
* fix job tests
* remove pytest from travis build
* cluster per test, parallel pytest plugin
* delete cluster after tests, wait until deleted
* fix bugs
* catch right error, change cluster_id to base_cluster_id
* fix test name
* fixes
* move tests to intregration_tests dir
* update travis to run non-integration tests
* directory structure, decoupled job tests
* fix job tests, issue with submit_job
* fix bug
* add test docs
* add cluster and job delete to finally clause
2018-02-22 14:06:16 -08:00
6b1c86195d
Feature: SDK support for file-like configuration objects ( #373 )
...
* add support for filelike objects for conifguration files
* fix custom scripts
* remove os.pathlike
* merge error
2018-02-21 16:55:26 -08:00
85e444ce34
Feature: Cluster Run and Copy ( #304 )
...
* start implementation of cluster run
* fix cluster_run
* start debug sequential user add and delete
* parallelize user creation and deletion, start implementation of cluster scp
* continue cluster_scp implementation
* debug statements, disconnect error: permission denied
* untesteed parakimo implementation of clus_run
* continue debugging user creation bug
* fix bug with pool user creation, start concurrent implementation
* start fix of paramiko cluster_run and cluster_copy
* working paramiko cluster_run implementation, start cluster_scp
* fix cluster_scp command
* update requirements, rename cluster_run function
* remove unused shell functions
* parallelize run and scp, add container_name, create logs wrapper
* change scp to copy, clean up
* sort imports
* remove asyncssh from node requirements
* remove old import
* remove bad error handling
* make cluster user management methods private
* remove comment
* remove accidental commit
* fix merge, move delete to finally clause
* add docs
* formatting
2018-02-07 16:19:33 -08:00
1a15245eb4
Feature: spark init docker repo customization ( #358 )
...
* customize docker_repo based on init args
* whitespace
* add some docs
* r-base to r
* case insensitive r flag, typo fix
2018-02-07 10:56:20 -08:00
748a1269fa
Feature: Spark mixed mode support ( #350 )
...
* add support for aad creds for storage on node
* add mixed mode support
* add docs
* switch error order
* add dedicated to get_cluster
* remove mixed mode in print_cluster_conf
2018-02-07 10:41:51 -08:00
63fb81b2b6
Allow submitting jobs into a VNET ( #365 )
...
* Add subnet_id to job submission cluster config
* add some docs
2018-02-05 15:43:13 -08:00
f490f9643f
Feature: AAD tutorial docs ( #353 )
...
* aad tutorial docs
* typo, specify application type
2018-01-25 13:58:59 -08:00
1aae3b5a8b
Feature: expose exit_code ( #320 )
...
* expose the batch task exit_code in ApplicationLog
* add exit_code to job list-apps output
* add execution info to applicationlog object
* whitespace
2018-01-19 16:53:45 -05:00
882418a928
Feature: Spark job submission ( #278 )
...
* refactor submit, initial job-submission commit
* fix merge conflicts
* comment out gpu_enabled
* update schedule, user job manager task, wait until container ready
* wait until docker container running
* add get_job_log
* add job id
* start multitask job submission
* start multitask implementation
* wait for node setup to complete before completing start task
* fix incorrect logic
* add environment variable, fix loading task definition, fix waiting for container
* fix early job completion, autokill pool, cleanup code
* include debug print error
* define job submission sdk stubs
* Job and JobConfiguration models
* remove timestamp, implement job function stubs , add Application model
* add delete_job, bug fixes
* add wait_until_job_finished, wait_until_all_jobs_finished
* fix storage output logs location, function names
* whitespace
* add cli file stubs
* start job cli implementation
* better output for job cli, added job.yaml configuration file
* error catch for missing blob, add sdk docs, models updates
* start Jobs tutorial doc
* template for rest of job tutorial doc
* rename app_id to application_name, add job sdk docs
* docs fix
* add support for low pri nodes
* better formatting for job.yaml
* remove unecessary comments
* add defaults and commnets for job.yaml, fix spark_configuration file loading
* fix spark_configuration file paths
* fix app_arg issue
* clean up code, address comments
* rename to cluster_submit_helper
* update get job print
* add list-apps, update help text for job id
* add application metadata to job, update job get print
* better error for get_application_log, add working example to job.yaml, add print_application
* add warning about master selection
* update list_applications return value
* whitespace
* Add link to docs in job.yaml, validation for job.yaml
* convert to commandbuilder, remove print
* fix submit exit code, fix no app_args bug, set autoscale interval
* wait until custom scripts are completed
* add missed import in get_app_logs, whitespace
* use correct python version for on node app submit
* update get output format
2018-01-19 13:37:58 -05:00
e193ed30dd
Feature: VNet support ( #324 )
...
* VNet support
* Azure active directory authentication
2018-01-17 11:14:56 -08:00
7e9bf9c383
Feature: Spark retry job ( #318 )
2018-01-08 12:31:47 -08:00
6091b1d390
Docs: update ( #263 )
...
* Update README.md
streamline and update main readme.md
* Update README.md
* Update README.md
* Update 13-configuration.md
* Update 12-docker-image.md
* Update 12-docker-image.md
* Update README.md
* Create README.md
* Update README.md
* Update 10-clusters.md
2017-12-11 17:02:51 -08:00
40bd2d62f3
Bug: fix wrong path for global secrets ( #265 )
...
* fix wrong path for global secrets
* load spark_conf files correctly
* docker-image docs fix
* docker-image docs fix
* move load_aztk_spark_config function to config.py
2017-12-11 15:06:08 -05:00
c12ecebad2
Update 60-gpu.md ( #253 )
...
* Update 60-gpu.md
make sure is available in region
* Update 60-gpu.md
2017-12-07 14:11:53 -08:00
6c26943819
Feature: update docker image doc ( #251 )
...
* update docker-image readme with new images
* update docs
2017-12-06 14:04:07 -08:00
8a060a2f78
Feature: Spark GPU ( #206 )
...
* conditionally install and use nvidia-docker
* status statements, and -y flag for install
* add example, remove unnecessary ppa
* rename custom script, remove print statement, update example
* add Dockerfile
* fix path in Dockerfile
* update Docker images to use service account
* updated docs, changed default docker repo for gpu skus
* make timing statements more verbose
* remove unnecessary script
* added gpu docs
* fix up docs and numba example
2017-12-04 13:28:05 -08:00
d74ceee3f5
Feature: Rename SDK ( #231 )
...
* initial refactor
* rename cli_fe to cli
* add docs for sdk client
* typo
* remove conflict
* fix zip node scripts bug, add sdk_example program
* start models docs
* add ClusterConfiguration docs, fix merge bug
* Application docs update
* added Application and SparkConfiguration docs
* whitespace
* rename cli.py and spark/cli
* add docstring for load_spark_client
2017-12-01 13:42:55 -08:00
cabcc29b3c
Feature: Azure Files ( #241 )
...
* initial take on installing azure files
* fix cluster.yaml parsing of files shares
* remove test code
* add docs for Azure Files
2017-11-30 14:16:53 -08:00
b983d12419
update 10-clusters.md - rm jupyter ref (for now) ( #222 )
2017-11-27 09:37:15 -08:00
7c59567bec
Fix a typo in link ( #235 )
2017-11-27 08:56:54 -08:00
e50cf8c52c
Spellchecking ( #233 )
...
aztk was misspelled as aztb, amended to correct spelling.
2017-11-27 08:35:02 -08:00
ef0acbabbe
Amended documentation to correctly obtain logs ( #234 )
...
Documentation amended: Use of aztk spark app logs results in an error. Correct usage is aztk spark cluster app-logs. Document amended to reflect this.
2017-11-27 08:31:17 -08:00
60cae3b8dd
Feature: HDFS plugin ( #215 )
...
* hdfs plugin initial
* fix passwordless ssh key, allow raw ip for datanodes
* remoe debug statement
* description of forwarded ports
* add prcryptodome to requirements.txt
* fix file copy bug
* add namenode ui to ssh command, add docs
2017-11-22 14:51:11 -08:00
65a56dd657
Feature/python container ( #210 )
...
* added python container, jupyter install script, vanilla container
* Update README.md
* Create README.md
* Create README.md
* Update README.md
* Update README.md
* Update README.md
* Update README.md
* Update README.md
* Update jupyter.sh
* Update README.md
* Update README.md
* python
* readme update
* docker updates
* Update README.md
* Update 12-docker-image.md
* Update constants.py
* add image files for wiki
* update imageS
* .
* dockerfile typo
* dockerfiles
* Removed r
* readme
* update constants.py
* update readme
* readme updates
* readme updates
2017-11-09 00:43:32 -08:00
Jacob Freck
mmduyzend
Timothee Guerin
lachiemurray
Dmitry Stratiychuk
Emlyn Corrin
JS
Pablo Selem
Ian McDonald
Matt Scanlon