1cc71c7a59
Fix: allow cluster config to be printed when no username has been set ( #597 )
2018-06-11 10:42:24 -07:00
98c601ceb8
Fix: Deprecation messages cause TypeError in non-verbose mode ( #596 )
...
* Stop deprecate() from throwing when not in verbose mode
* Improve deprecation warning messages
2018-06-08 10:59:13 -07:00
7d7a814c50
Fix: fix typos ( #595 )
2018-06-07 09:57:43 -07:00
88d04195ec
Feature: add cluster list quiet flag, ability to compose with delete ( #581 )
...
* add quiet flag, ability to compose with delete
* log.print instead of print
* add some docs
2018-06-06 16:03:34 -07:00
fbf1bab704
Conda, Apt-Get and Pip Install Plugins ( #594 )
...
* Added install plugins
* Moved packages to directory
* Removed channel from conda install
* changed default to none
* Added line
* fixed template
* Fixed naming of apt get
2018-06-06 15:16:27 -07:00
fa3ac0eb3b
Fix: --size-low-pri being ignored ( #593 )
2018-06-05 10:54:02 -07:00
3f0c8f9bfc
Fix: set logger to stdout ( #588 )
...
* set logger to stdout
* typo
* add log.print level
2018-06-04 17:39:24 -07:00
f16aac091e
Feature: pure python ssh ( #577 )
...
* forward multiple ports
* plumb through cli
* continue cli implementation
* fixes
* pylint ignore
* spacing
* remove debug stuff, fix bug
* add --internal support
* add to init
* add comment
* remove nesting
* add logging
* add some docs
2018-06-04 17:16:51 -07:00
af449dc194
Feature: add node run command ( #572 )
...
* add node run command
* whitespace
* add node-run doc
* add host flag
* refactor, print->log
* generated username
* more secure random
* better handling of find node, type conversion
* add generate_user_on_node
* docs update
* fix docs
* remove duplicate import, sort
2018-06-04 13:58:33 -07:00
b9a863b2f5
Warnings show stacktrace on verbose ( #587 )
2018-06-04 08:10:00 -07:00
8b8cd6260f
Fix: Remove old spark-defaults.conf jars ( #567 )
2018-05-30 13:05:55 -07:00
8fea9ce092
Feature: Disable scheduling on group of nodes ( #540 )
2018-05-30 13:02:48 -07:00
02f336b0a0
Feature: New Models design with auto validation, default and merging ( #543 )
2018-05-30 09:07:09 -07:00
f6735cc6dd
Feature: Support passing of remote executables via aztk spark cluster submit ( #549 )
2018-05-24 10:29:23 -07:00
1527929e30
Feature: TensorflowOnSpark python plugin ( #525 )
...
* initial commit
* update
* update
* add gpu support
* remove comment
* change class to function
* fix merge issue
* add some docs
2018-05-21 13:22:55 -07:00
603a413d12
Feature: nvBLAS and OpenBLAS plugin ( #539 )
...
* add openblas plugin, update gpu docker images with netlib-lgpl
* update images and plugins
* add nvblas plugin
* revert gpu docker image change, add -Pnetlib-lgpl to base images
* change configuraitons to functions, add pugins to cluster.yaml
2018-05-15 17:47:41 -07:00
a99bbe19e6
Fix pass docker repo command back to the cluster config ( #538 )
2018-05-03 08:48:38 -07:00
7a7e63c54f
Feature: New Toolkit configuration ( #507 )
2018-05-01 16:36:44 -07:00
9bc76396bc
Docs: Added worker on master docs ( #531 )
2018-05-01 14:40:31 -07:00
779bffb2da
Feature: refactor docker images ( #510 )
...
* add spark2.3.0 hadoop2.8.3 dockerfile
* start update to docker image
* add SPARK_DIST_CLASSPATH to bashrc, source .bashrc in docker run
* add maven install for jars
* docker image update and code fix
* add libthrift (still broken)
* start image refactor, build from source,
* add refactor to r base image
* finish refactor r image
* add storage jars and deps
* exclude netty to get rid of dependency conflict
* add miniconda image
* update 2.2.0 base, anaconda image
* remove unused cuda-8.0 image
* start pipenv implementation
* miniconda version arg
* update anaconda and miniconda image
* style
* pivot to virtualenv
* remove virtualenv from path when submitting apps
* flatten layers
* explicit calls to aztk python instead of activating virtualenv
* update base, miniconda, anaconda
* add compatibility version for base aztk images
* typo fix
* update pom
* update environment variable name
* update environment variables
* add anaconda images base & gpu
* update gpu and miniconda base images
* create venv in cluster create
* update base docker files, remove virtualenv
* fix path
* add exclusion to base images
* update r images
* delete python images (in favor of anaconda and miniconda)
* add miniconda gpu images
* update comment
* update aztk_version_compatibility to dokcer image version
* add a build script
* virutalenv->pipenv, add pipfile & pipfile.lock remove secretstorage
* aztk/staging->aztk/spark
* remove jars, add .null to keep directory
* update pipfile, update jupyter and jupyterlab
* update default images
* update base images to fix hdfs
* update build script with correct path
* add spark1.6.3 anaconda, miniconda, r base and gpu images
* update build script to include spark1.6.3
* mkdir out
* exclude commons lang and slf4j dependencies
* mkdir out
* no fail if dir exists
* update node_scripts
* update env var name
* update env var name
* fix the docker_repo docs
* master->0.7.0
2018-04-30 17:19:01 -07:00
47000a5c7d
Bug: add timeout handling to cluster_run and copy ( #524 )
...
* update cluster_run and copy to handle timeouts
* fix
* move timeout default to connect function
2018-04-30 16:49:58 -07:00
c98df7d1df
Feature: Added custom scripts functionality for plugins with the cli(Deprecate custom scripts) ( #517 )
2018-04-27 10:31:24 -07:00
2e995b4899
Feature: spark ui proxy plugin ( #467 )
...
* initial commit
* add args
* add docs
* change default plugins
* update ssh cli ui, remove plugin name
* change conditional
* update docs to include jupyterlab
* remove spark_ui_proxy as default plugin
2018-04-23 12:12:31 -07:00
44a07654aa
Feature: spark debug tool ( #455 )
...
* start implementation of cluster debug utility
* update debug program
* update debug
* fix output directory structure
* cleanup output, add error checking
* sort imports
* start untar
* extract tar
* add debug.py to pylintc ignore, line too long
* crlf->lf
* add app logs
* call get_spark_app_logs, typos
* add docs
* remove debug.py from pylintrc ignore
* added debug.py back to pylint ignore
* change pylint ignore
* remove commented log
* update cluster_run
* refactor cluster_copy
* update debug, add spinner for run and copy
* make new sdk cluster_download endpoint
2018-04-09 15:02:43 -07:00
1eaa1b6e42
Feature: add internal flag to node commands ( #482 )
...
* add internal ssh flag
* add --internal flag to cluster get
* cluster run internal flag
* fix add command back
* cluster copy internal
* fix method params
* fix method params
* add debug statement
* fix params
* remove debug statement
* fixes
* add debug statement
* remove debug statement
* add hostname to /etc/hosts
* remove hostname from /etc/hosts
* add sdk docs for internal switch in cluster run and copy
2018-04-06 15:59:13 -07:00
be8cd2a490
Bug: Remove unused ssh plugin flags ( #488 )
2018-04-06 14:55:47 -07:00
a33bdbc5a9
Bug: fix broken spark init command ( #486 )
2018-04-06 14:10:40 -07:00
4ef3dd09df
Bug: add spark.history.fs.logDirectory to required keys ( #456 )
...
* add spark.history.fs.logDirectory to requried keys
* add spark_event_log_enabled_key to required_keys
* docs, add history server config to spark-defaults.conf
* fix bad logic
* crlf->lf
2018-04-05 14:11:35 -07:00
32de752d53
Feature: Spark add output logs flag ( #468 )
...
* add output flag to cluster submit
* add output flag to cluster app-logs
* add output flag to job get-app-logs
* sort imports
* make spinner context
2018-04-05 12:21:56 -07:00
8889059aad
Feature: match cluster submit exit code in cli ( #478 )
2018-04-05 11:54:25 -07:00
ee1e61bb9d
Bug: fix spark job submit path ( #474 )
...
* fix job submit path, fix raise error, remove print
* source bashrc before executing
2018-04-03 11:19:35 -07:00
2dd7891499
Bug: add support for jars, pyfiles, files in Jobs ( #408 )
...
* add support for jars, pyfiles, files, refactor JobConfig
* set encoding explicitly
* fix typerror bug in mixed_mode()
2018-03-26 11:38:05 -07:00
5761a3663a
Bug: set explicit file open encoding ( #448 )
...
* explicit file encoding
* crlf->lf
2018-03-23 13:42:30 -07:00
8aa1843f23
Feature: managed storage for clusters and jobs ( #443 )
...
* add in storage management for clusters, jobs
* add warning logs on cli delete
* whitespace
* add keep-logs flag
* add docs on storage lifetime
2018-03-20 10:45:49 -07:00
9253aac0ea
Fix: VNet required error now showing if using mixed mode without it ( #440 )
2018-03-14 10:27:48 -07:00
4be5ac2f44
Fix job configuration option for `aztk spark job submit` command ( #435 )
...
`--job-conf` option mentioned in the docs wasn't working.
CLI help was showing that option is named `--configuration-c`
which seems to be a result of a missing comma in option definition.
2018-03-13 11:07:48 -07:00
2bed496c39
Internal: Cluster data helpers and upload_node_script into cluster_data module ( #401 )
2018-03-08 10:34:19 -08:00
c237501a9f
Feature: Pypi auto deployement ( #428 )
2018-03-05 17:18:47 -08:00
mmduyzend
Jacob Freck
Brian
Timothee Guerin
lachiemurray
Dmitry Stratiychuk