8f8e7b268b
Update README.md ( #728 )
2021-02-02 09:38:45 -08:00
6e04372d19
Release v0.10.3 ( #714 )
...
* Build(deps): Bump requests from 2.19.1 to 2.20.0 in /aztk/node_scripts
Bumps [requests](https://github.com/requests/requests ) from 2.19.1 to 2.20.0.
- [Release notes](https://github.com/requests/requests/releases )
- [Changelog](https://github.com/psf/requests/blob/master/HISTORY.md )
- [Commits](https://github.com/requests/requests/compare/v2.19.1...v2.20.0 )
Signed-off-by: dependabot[bot] <support@github.com>
* Build(deps): Bump pyyaml from 3.13 to 5.1 in /aztk/node_scripts
Bumps [pyyaml](https://github.com/yaml/pyyaml ) from 3.13 to 5.1.
- [Release notes](https://github.com/yaml/pyyaml/releases )
- [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES )
- [Commits](https://github.com/yaml/pyyaml/compare/3.13...5.1 )
Signed-off-by: dependabot[bot] <support@github.com>
* update changelog for 0.10.3 release
* update requirements and pipfile
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-02-14 15:27:19 -08:00
5be9e87c70
Pin pipenv dependencies in setup_host.sh ( #709 )
2020-02-13 10:01:23 -08:00
72c7833fbf
Fix: list_applications throwing error ( #692 )
...
* Fixed Client.cluster.list_applications function to return [aztk.models.Task] since it was previously throwing errors.
* Updated documentation for the base _list_applications function.
* Updated to return a list of aztk.spark.models.Application instances instead of the raw tasks as would be expected by the name. Also updated documentation to reflect this change.
2019-01-23 18:03:41 -05:00
1f6fc8b8e1
Release: v0.10.2 ( #691 )
...
* update changelog
* update version
* update version in docs links
2018-12-07 12:58:32 -08:00
723995cf2f
Fix: Storage table never deleted ( #690 )
...
* fix bad detection of table existence, delete if exist
* remove unused import
* fix double retry logic
2018-12-07 12:22:41 -08:00
a91690ba5b
Feature: Migrate off preview vsts queues ( #685 )
...
* update syntax, switch to non-preview pools
* debug condition
* remove debug trigger
* typo
2018-11-19 13:37:37 -08:00
5379b00b73
Release: 0.10.1 ( #682 )
...
* update version
* update changelog for 0.10.1 release
* update relevent links
2018-11-02 11:45:53 -07:00
a0bc2f0f09
Feature: Include cluster creation time in cluster list and cluster get output ( #678 )
...
* Include cluster creation time in cluster list and info
* Make helper function for formatting
* Make yapf happy
2018-11-02 11:11:21 -07:00
385040d7b1
Fix: Verify Batch task has node_info ( #681 )
2018-11-01 13:56:51 -07:00
6d2e6c5f7b
Release: v0.10.0 ( #676 )
...
* fix function name
* update version
* changelog updates
* set minor version to correct number
* update changelog
* changelog formatting
* typos
* whitespace
* remove print debug statement
* whitespace
* whitespace
2018-10-29 12:28:26 -07:00
e486536919
Feature: ClusterState enum ( #677 )
...
* add cluster state enum, update usage
* fix docstrings
* stopping->stopping_resize
2018-10-29 11:05:36 -07:00
8c2bf0c1a6
Feature: spark submit scheduling internal ( #674 )
...
* add internal support for scheduling_target cluster submit
* add internal support for scheduling target job submission
* add cli flag
2018-10-26 16:58:38 -07:00
18b74e47d5
Feature: Spark retry docker pull ( #672 )
...
* retry docker pulls
* change order of pool, job, storage creation to reduce conflicts
* add error message on docker-compose curl failure
2018-10-24 16:49:02 -07:00
9e32b4b57b
Feature: remove custom scripts ( #673 )
2018-10-24 15:18:56 -07:00
fc5053654c
Feature: 0.10.0 remove deprecated code ( #671 )
...
* remove deprecated code
* remove unused imports
* fix field name
* remove unused imports
* clean up comments, linting warnings
2018-10-24 10:43:47 -07:00
4408c4fc41
Feature: Spark scheduling target ( #661 )
...
* initial
* update pipfile and pipfile.lock
* uncomment scheduling target, start ssh_submit impl
* get rid of debug code
* finish ssh_submit implementation
* serialize object instead of properties
* fix upload log bug, temp workaround for get logs
* remove unused function
* clean up node_scripts submit, remove debug code
* ensure warns on deprecated test
* remove commented timeout
* start scheduling_target for job_submission
* continue job scheduling target implementation
* update pipefile.lock
* update Pipfile deps, pin pynacl to fix build failure
* fix syntax
* fix pipfile with latest azure-nspkg
* update path for scheduling scripts
* update config.py import
* add nohup dependency
* use nohup and exit immediately
* remove bad dep
* remove nohup
* remove commented code
* add block to ssh, get retcode from node_exec
* fix typo
* fix some imports, add test stubs
* fixes
* start implementation of task table service
* add scheduling_target support for get_application_log
* todos
* remove useless statement
* move get_application_status to core, add scheduling_target support
* update deps in requirements.txt
* fix false positive pylint import error
* remove bad import
* bad local variable
* add batch task abstraction, add datetime field
* mediate table insertion with task abstraction
* fix issues with task abstraction usage
* fix pylint import error
* fix update task on run
* update job submission test
* make test package, update pylint
* update job submission with scheduling_target
* add job support for scheduling_target
* fix taskstate serialization to storage
* fix job submission job manager task, catch table storage errors
* fix import
* fix imports for batch sdk 5.0+
* fix test model module
* fix node election exception catch
* start fix job tests
* move get_task_status to base
* fix job tests
* fix get_application, add abstraction to batch task gets
* fix some bugs, remove some debug statements
* fix test
* use jobstate and application state
* add start_task retries
* make jobstate an enum
* fix import
* fixes
* fixes
* revert settings.json
* fixes for application state in cli
* conditionally create storage table
* remove commented code
* conditionally create storage table
* remove commented code
* fix test
* respond to comments
* fix debug statement, fix starttask issue
* remove debug test print
* formatting
* update doc string with correct return value
* revert settings.json
* more robust starget test, fix get_application for starget
* whitespace
2018-10-23 15:47:54 -07:00
b7da355618
Release: 0.9.1 ( #669 )
...
* update changelog and version
* update docs links to new version number
2018-10-05 13:07:15 -07:00
cb62207917
Fix: vsts mutliline secrets ( #668 )
...
* backticks to quotes
* debug integration tests
* typo
* remove debug condition
2018-10-05 12:45:59 -07:00
0606598399
Fix: pin all node dependencies not in Pipfile ( #667 )
2018-10-05 11:34:08 -07:00
4a60c8ae1f
Fix: vsts integration tests block ( #657 )
...
* only run integration tests in integration tests block
* update vsts build
* whitespace
* whitespace
* syntax
* syntax
* fix condition variable reference
* fix
* update name and trigger
* change from preview queue to ga
* upgrade pip
* add setuptools
* add pip upgrade step
* debug statement
* debug
* revert to preview
* change how env vars are referenced
* debug statement
* add missing key
* support multiline env vars
* change process value to 70
* only run integration on master or release
2018-09-20 12:49:09 -07:00
0b675f9b42
Fix: 0.9.0 changelog ( #656 )
...
* changelog and version update
* update date in changelog
* add features section to changelog, add additional breaking changes
2018-08-30 11:14:54 -07:00
cfcfada285
Release: 0.9.0 ( #655 )
...
* changelog and version update
* update date in changelog
2018-08-30 10:47:11 -07:00
93615d9a43
Fix: spark roll back scheduling disable ( #653 )
...
* disable offlining on node
* disable scheduling_target in config, cli, and sdk
* remove schedluing target function
* formatting
* remove alway none return value
2018-08-29 15:40:09 -07:00
7c37b06bca
Fix: typo in vsts build ( #654 )
2018-08-27 10:32:54 -07:00
828162ef10
Internal: fix pylint warnings ( #651 )
...
* inital, remove unused imports
* run yapf
* remove unused imports and variables, fix declaration outside init
* fix some pylint warnings, add ssh_into_master
* remove unused imports
* unused variables
* string and function normalization
* stop using list comprehension for side effects, make method function
* stop using protected member
* various pylint fixes
* formatting
* formatting
* add retry decorator with tests
* start adding retry decorator, retry docker compose download
* update pip and tests
* logic fix
* change no delete if
* factor out reused functions
* fix wait_for_all_nodes
* fix download return type bug
* test vsts ci update
* temporarily disable integration tests
* syntax fix
* update vsts build
* add back integration tests, remove debug branch
* remove parallel unit tests
* more verbose clis
* update pylint
* typo
* fix imports
* function returns nothing, don't return
* make iterator list
* change debug value
2018-08-24 17:21:22 -07:00
0a9ce94104
Fix: remove bad node scripts import ( #652 )
2018-08-17 21:47:13 -04:00
442228a30f
Deprecate: remove custom scripts ( #650 )
2018-08-17 20:36:11 -04:00
293f297541
Fix: diagnostics function write error result bug ( #649 )
...
* fix output writing for diagnostics function
* log on ssh connect
2018-08-17 17:36:51 -04:00
7c14648005
Fix: expose get cluster configuration API ( #648 )
...
* fix get and ssh cli calls, add get_configuration api
* update builds to lint aztk_cli in parallel
* remove unnecessary get_configuration calls
2018-08-17 16:12:08 -04:00
9098533969
Feature: first run docs update ( #644 )
...
* update getting started page, remove custom scripts doc
* recommend venv
2018-08-17 15:58:22 -04:00
b7bdd8c268
Feature: add brief flag to debug tool ( #634 )
...
* add brief flag
* add some docs
* fix requirements
2018-08-16 15:18:43 -04:00
eef36dc062
Feature: 0.9.0 deprecated code removal ( #645 )
...
* remove deprecated code
* remove deprecated tests, yapf test directory
* add import
* remove unused test
* remove deprecated field name in tests
* update test parameter to non deprecated name
2018-08-16 15:11:59 -04:00
9d554c3255
Feature: Add ability to specify docker run options in toolkit config ( #613 )
...
* Feature: Add ability to specify docker run options in cluster config
* update function calls to match new sdk refactor
* fix empty docker_run_options failure
* formatting
* fix formatting (#3 )
2018-08-13 09:03:58 -07:00
f98d0373f0
Fix: update incompatible dependencies in setup.py ( #639 )
2018-08-08 10:48:57 -07:00
6eda21eecf
Internal: update vsts build ( #635 )
...
* explicit style configuration, add pylint to vsts
* print yapf version
* typo
* make vsts use python 3.6
* remove debug statements
* separate unit and integration tests in vsts
* remove dev trigger
* whitespace
* up the number of parallel processes
* fix diagnostic tool return value
* reenable dev builds
* remove dev builds
2018-08-07 17:05:10 -07:00
7730c46ee4
Internal: verify code formatting in build ( #633 )
...
* format all files, enforce formatting in travis build
* add yapf to vsts build
* update vsts build
* fix
* fix
* fix
* change queue to ubuntu
* revert
* temporarily enable builds on pushes to this branch
* change to non preview
* revert
* update yapf version, rerun
* update pytest parallelism
* add retry to arm call to avoid failures
* remove non-master trigger
* update builds, formatting style
2018-08-06 15:29:06 -07:00
b18eb695a1
Feature: SDK refactor ( #622 )
...
* start refactor
* continue refactor for cluster and job functions
* fix imports
* fixes
* fixes
* refactor integration test secrets management
* fix cluster create, add new test
* add tests for new sdk api and fix bugs
* fix naming and bugs
* update job operations naming, bug fixes
* fix cluster tests
* fix joboperations and tests
* update cli and fix some bugs
* start fixes
* fix pylint errors, bugs
* add deprecated warning checks, rename tests
* add docstrings for baseoperations
* add docstrings
* docstrings, add back compat for coreclient, fix init for spark client
* whitespace
* docstrings, whitespace
* docstrings, fixes
* docstrings, fixes
* fix the sdk documentation, bugs
* fix method call
* pool_id->id
* rename ids
* cluster_id->id
* cluster_id->id
* add todo
* fixes
* add some todos
* rename pool to cluster, add todo for nodes params
* add todos for nodes param removal
* update functions names
* remove deprecated fucntion calls
* update docs and docstrings
* update docstrings
* get rid of TODOs, fix docstrings
* remove unused setting
* inheritance -> composition
* fix models bugs
* fix create_user bug
* update sdk_example.py
* fix create user argument issue
* update sdk_example.py
* update doc
* use Software model instead of string
* add job wait flag, add cluster application wait functions
* add docs for wait, update tests
* fix bug
* add clientrequesterror catch to fix tests
2018-08-03 15:20:05 -07:00
c9fd8bbfeb
Release: v0.8.1 ( #616 )
...
* changelog update
* version update
* change release date of 0.8.0 to accurate date
2018-06-21 12:30:53 -07:00
a8f8e92629
Fix: docs links version ( #614 )
...
* update changelog
* update versions
2018-06-20 14:55:13 -07:00
4e0b1ecd0f
Fix: spark debug tool filter out .venv, make debug tool testable ( #612 )
...
* filter out .venv
* add NodeOutput model
* add debug tool integration test
* add test for debug tool
* split condition
* revert style change
* remove debug print
* whitespace
* remove other model implementation
* fix cluster copy
* fix cluster run and cluster copy
2018-06-20 14:26:35 -07:00
883980d345
Suppress msrest warnings ( #611 )
2018-06-18 12:59:28 -07:00
5306a2ad24
Fix: set defaults for SparkConfiguration, add tests ( #606 )
...
* add SparkConfiguration tests
* change test directory structure
2018-06-15 14:05:40 -07:00
34b25855d5
Fix: release v0.8.0 ( #600 )
...
* update changelog and version
* update changelog
* add deprecation version and tests
* fix tests
* update changelog with deprecations
* update changelog
2018-06-11 15:48:08 -07:00
4b2acc8491
Fix: add toolkit to sdk docs and example ( #602 )
2018-06-11 15:34:29 -07:00
2cb56f8cfd
Release: v0.8.0 ( #599 )
...
* update changelog and version
* update changelog
2018-06-11 12:15:45 -07:00
1cc71c7a59
Fix: allow cluster config to be printed when no username has been set ( #597 )
2018-06-11 10:42:24 -07:00
98c601ceb8
Fix: Deprecation messages cause TypeError in non-verbose mode ( #596 )
...
* Stop deprecate() from throwing when not in verbose mode
* Improve deprecation warning messages
2018-06-08 10:59:13 -07:00
7d7a814c50
Fix: fix typos ( #595 )
2018-06-07 09:57:43 -07:00
88d04195ec
Feature: add cluster list quiet flag, ability to compose with delete ( #581 )
...
* add quiet flag, ability to compose with delete
* log.print instead of print
* add some docs
2018-06-06 16:03:34 -07:00
Jacob Freck
Brett Simons
Emlyn Corrin
mmduyzend
Timothee Guerin