fc5053654c
Feature: 0.10.0 remove deprecated code ( #671 )
...
* remove deprecated code
* remove unused imports
* fix field name
* remove unused imports
* clean up comments, linting warnings
2018-10-24 10:43:47 -07:00
4408c4fc41
Feature: Spark scheduling target ( #661 )
...
* initial
* update pipfile and pipfile.lock
* uncomment scheduling target, start ssh_submit impl
* get rid of debug code
* finish ssh_submit implementation
* serialize object instead of properties
* fix upload log bug, temp workaround for get logs
* remove unused function
* clean up node_scripts submit, remove debug code
* ensure warns on deprecated test
* remove commented timeout
* start scheduling_target for job_submission
* continue job scheduling target implementation
* update pipefile.lock
* update Pipfile deps, pin pynacl to fix build failure
* fix syntax
* fix pipfile with latest azure-nspkg
* update path for scheduling scripts
* update config.py import
* add nohup dependency
* use nohup and exit immediately
* remove bad dep
* remove nohup
* remove commented code
* add block to ssh, get retcode from node_exec
* fix typo
* fix some imports, add test stubs
* fixes
* start implementation of task table service
* add scheduling_target support for get_application_log
* todos
* remove useless statement
* move get_application_status to core, add scheduling_target support
* update deps in requirements.txt
* fix false positive pylint import error
* remove bad import
* bad local variable
* add batch task abstraction, add datetime field
* mediate table insertion with task abstraction
* fix issues with task abstraction usage
* fix pylint import error
* fix update task on run
* update job submission test
* make test package, update pylint
* update job submission with scheduling_target
* add job support for scheduling_target
* fix taskstate serialization to storage
* fix job submission job manager task, catch table storage errors
* fix import
* fix imports for batch sdk 5.0+
* fix test model module
* fix node election exception catch
* start fix job tests
* move get_task_status to base
* fix job tests
* fix get_application, add abstraction to batch task gets
* fix some bugs, remove some debug statements
* fix test
* use jobstate and application state
* add start_task retries
* make jobstate an enum
* fix import
* fixes
* fixes
* revert settings.json
* fixes for application state in cli
* conditionally create storage table
* remove commented code
* conditionally create storage table
* remove commented code
* fix test
* respond to comments
* fix debug statement, fix starttask issue
* remove debug test print
* formatting
* update doc string with correct return value
* revert settings.json
* more robust starget test, fix get_application for starget
* whitespace
2018-10-23 15:47:54 -07:00
93615d9a43
Fix: spark roll back scheduling disable ( #653 )
...
* disable offlining on node
* disable scheduling_target in config, cli, and sdk
* remove schedluing target function
* formatting
* remove alway none return value
2018-08-29 15:40:09 -07:00
828162ef10
Internal: fix pylint warnings ( #651 )
...
* inital, remove unused imports
* run yapf
* remove unused imports and variables, fix declaration outside init
* fix some pylint warnings, add ssh_into_master
* remove unused imports
* unused variables
* string and function normalization
* stop using list comprehension for side effects, make method function
* stop using protected member
* various pylint fixes
* formatting
* formatting
* add retry decorator with tests
* start adding retry decorator, retry docker compose download
* update pip and tests
* logic fix
* change no delete if
* factor out reused functions
* fix wait_for_all_nodes
* fix download return type bug
* test vsts ci update
* temporarily disable integration tests
* syntax fix
* update vsts build
* add back integration tests, remove debug branch
* remove parallel unit tests
* more verbose clis
* update pylint
* typo
* fix imports
* function returns nothing, don't return
* make iterator list
* change debug value
2018-08-24 17:21:22 -07:00
442228a30f
Deprecate: remove custom scripts ( #650 )
2018-08-17 20:36:11 -04:00
eef36dc062
Feature: 0.9.0 deprecated code removal ( #645 )
...
* remove deprecated code
* remove deprecated tests, yapf test directory
* add import
* remove unused test
* remove deprecated field name in tests
* update test parameter to non deprecated name
2018-08-16 15:11:59 -04:00
b18eb695a1
Feature: SDK refactor ( #622 )
...
* start refactor
* continue refactor for cluster and job functions
* fix imports
* fixes
* fixes
* refactor integration test secrets management
* fix cluster create, add new test
* add tests for new sdk api and fix bugs
* fix naming and bugs
* update job operations naming, bug fixes
* fix cluster tests
* fix joboperations and tests
* update cli and fix some bugs
* start fixes
* fix pylint errors, bugs
* add deprecated warning checks, rename tests
* add docstrings for baseoperations
* add docstrings
* docstrings, add back compat for coreclient, fix init for spark client
* whitespace
* docstrings, whitespace
* docstrings, fixes
* docstrings, fixes
* fix the sdk documentation, bugs
* fix method call
* pool_id->id
* rename ids
* cluster_id->id
* cluster_id->id
* add todo
* fixes
* add some todos
* rename pool to cluster, add todo for nodes params
* add todos for nodes param removal
* update functions names
* remove deprecated fucntion calls
* update docs and docstrings
* update docstrings
* get rid of TODOs, fix docstrings
* remove unused setting
* inheritance -> composition
* fix models bugs
* fix create_user bug
* update sdk_example.py
* fix create user argument issue
* update sdk_example.py
* update doc
* use Software model instead of string
* add job wait flag, add cluster application wait functions
* add docs for wait, update tests
* fix bug
* add clientrequesterror catch to fix tests
2018-08-03 15:20:05 -07:00
4e0b1ecd0f
Fix: spark debug tool filter out .venv, make debug tool testable ( #612 )
...
* filter out .venv
* add NodeOutput model
* add debug tool integration test
* add test for debug tool
* split condition
* revert style change
* remove debug print
* whitespace
* remove other model implementation
* fix cluster copy
* fix cluster run and cluster copy
2018-06-20 14:26:35 -07:00
5306a2ad24
Fix: set defaults for SparkConfiguration, add tests ( #606 )
...
* add SparkConfiguration tests
* change test directory structure
2018-06-15 14:05:40 -07:00
34b25855d5
Fix: release v0.8.0 ( #600 )
...
* update changelog and version
* update changelog
* add deprecation version and tests
* fix tests
* update changelog with deprecations
* update changelog
2018-06-11 15:48:08 -07:00
7d7a814c50
Fix: fix typos ( #595 )
2018-06-07 09:57:43 -07:00
1eeff23d06
Fix: models v2 deserialization ( #584 )
...
* fix model deserialization
* whitespace
2018-05-30 16:42:42 -07:00
8fea9ce092
Feature: Disable scheduling on group of nodes ( #540 )
2018-05-30 13:02:48 -07:00
02f336b0a0
Feature: New Models design with auto validation, default and merging ( #543 )
2018-05-30 09:07:09 -07:00
66037fd5cb
Feature: Add VSTS CI ( #561 )
...
* fix job submission bug and integration tests
* merge
* update tests, add vsts-ci.yml
* add python step
* debug statements
* update build
* update build add print
* build update
* fix bug
* debug
* undo
* parallelize build
* add trigger
* typo
* remove env
* whitespace
* whitespace
* whitespace
* remove debug branch
2018-05-22 15:17:02 -07:00
7a7e63c54f
Feature: New Toolkit configuration ( #507 )
2018-05-01 16:36:44 -07:00
c98df7d1df
Feature: Added custom scripts functionality for plugins with the cli(Deprecate custom scripts) ( #517 )
2018-04-27 10:31:24 -07:00
5579d95b41
Fix: Worker on master flag ignored and standardize boolean environment ( #514 )
2018-04-26 09:27:37 -07:00
de7898334c
Feature: Plugin V2: Running plugin on host ( #461 )
2018-04-23 17:20:43 -07:00
c237501a9f
Feature: Pypi auto deployement ( #428 )
2018-03-05 17:18:47 -08:00
6827181a34
Fix typo load_aztk_screts -> load_aztk_secrets ( #421 )
2018-02-27 04:34:46 -08:00
c724d9403f
Feature: Plugins ( #387 )
2018-02-26 16:36:31 -08:00
17755e0ffa
Feature: Basic Cluster and Job Submission SDK Tests ( #344 )
...
* add initial cluster tests
* add cluster tests, add simple job submission test scenario
* sort imports
* fix job tests
* fix job tests
* remove pytest from travis build
* cluster per test, parallel pytest plugin
* delete cluster after tests, wait until deleted
* fix bugs
* catch right error, change cluster_id to base_cluster_id
* fix test name
* fixes
* move tests to intregration_tests dir
* update travis to run non-integration tests
* directory structure, decoupled job tests
* fix job tests, issue with submit_job
* fix bug
* add test docs
* add cluster and job delete to finally clause
2018-02-22 14:06:16 -08:00
e31d5e73ae
Feature: SDK ( #180 )
...
* initial sdk commit
* added submit, wait_until_cluster_ready, wait_until_jobs_done, async options
* remove incorrect public method
* initial error checking
* factored helper commands out of spark client file
* remove unnecessary print statement
* add get_cluster and list_cluster, fix imports
* add create_user
* remove appmodel from base class, create app_logs_model
* fix imports and models call bug
* change asynchronous to wait, add get_logs(), add wait_until_app_done()
* add get_application_status(), add_create_cluster_in_parallel(), add submit_all_applications(), add wait_until_all_clusters_are_ready, create_user() accepts cluster_id, rename app to application
* add try catches for all public methods, raise AztkErrors
* add Custom Script model
* added custom script support
* added ssh conf model
* added ssh conf subclass, fixed typing issue
* add support for spark configuration files, move upload_node_scripts to spark
* changed submit to require cluster_id
* whitespace
* initial integration commit
* create_user takes ssh key or path to key
* fix get_user_public_key
* add name for parameter
* integrate cluster_create and cluster_add_user with sdk
* expose pool in Cluster model
* add bool return value to delete_cluster
* integrate cluster_delete
* integrate cluster_get and cluster_list with sdk
* integrate cluster_submit and cluster_app_logs with sdk
* integrate ssh with sdk
* change master_ui to web_ui and web_ui to job_ui
* fix cluster_create, cluster_get, and cluster_ssh, aztklib
* add home_directory_path constant
* remove unnecessary files in cli
* remove unnecessary files
* fix setup.py constants
* redo #167
* fix constants and setup.py
* remove old tests, fix constants
* fix get_log typo
* refactor cluster_create for readability
* decouple cli from sdk, and batch functions from software functions
* update version, fix in setup.py
* whitespace
* fix init source path
* change import
* move error.py to root sdk directory
* fix cluster_ssh error call
* fix bug if no app_args are present
* remove default value for docker_repo in constructor
2017-10-31 12:34:23 -07:00
ef2910ceef
Bug: simplify secrets loading ( #133 )
...
* create aztklib object, move clusterlib and joblib to objects, update cli endpoints
* fix errors in joblib
* refactor SecretsConfig loading, refactor batch and blob client loading
* fix tests
* fix test and pylint errors
* rename clusterlib to cluster and rename joblib to job
* whitespace
* rename aztklib to aztk and fix whitespace
* fix get command
* fixed issues with docker_auth and tests
2017-10-03 16:22:48 -07:00
eb2a221cca
Bug: rename dtde to aztk, rename thunderbolt to aztk, rename azb to aztk ( #125 )
...
* rename dtde to aztk, rename thunderbolt to aztk
* rename azb to aztk, fix bug with --wait
* changed exception to AztkError
* Fix issue with wait flag if not set
2017-09-29 17:13:22 -07:00
4822172b6a
Feature/secrets.cfg ( #43 )
...
* update configuration.cfg to secrets.cfg
* fix node count pretty print
* add more unit tests
* configure pytest to only run tests from the 'tests' directory
* add missing space
* remove static call to loading secrets file
2017-08-21 12:36:45 -07:00
d92df3c0be
Initial structure for test framework and CI ( #30 )
...
* Command builder tests
* Setup travis
* Update to travis
* Update to travis
* Update to travis
* Error only
* builder to array
2017-07-18 14:07:11 -07:00
Jacob Freck
mmduyzend
Timothee Guerin
Emlyn Corrin
Pablo Selem