aztk/requirements.txt

# Distribution
azure-batch==3.0.0
azure-mgmt-batch==5.0.0
azure-mgmt-storage==1.5.0
azure-storage==0.33.0
pyyaml==3.12
pycryptodome==3.4.7
paramiko==2.4.0

# Development
yapf==0.20.1
pylint==1.7.2
pytest==3.1.3
pytest-xdist==1.22.0
Update stable to v0.6.0 (#418) * Feature: on node user creation (#303) * client side on node user creation * start create user on node implementation * fix on node user creation * remove debug statements * remove commented code * line too long * fix spinner password prompt ui bug * set wait to false by default, formatting * encrypt password on client, decrypt on node * update docs, log warning if password used * Fix list-apps crash (#364) * Allow submitting jobs into a VNET (#365) * Add subnet_id to job submission cluster config * add some docs * Feature: Spark mixed mode support (#350) * add support for aad creds for storage on node * add mixed mode support * add docs * switch error order * add dedicated to get_cluster * remove mixed mode in print_cluster_conf * Feature: spark init docker repo customization (#358) * customize docker_repo based on init args * whitespace * add some docs * r-base to r * case insensitive r flag, typo fix * Bug: Load default Jars for job submission CLI (#367) * load jars in .aztk/ by default * rewrite loading config files * Feature: Cluster Run and Copy (#304) * start implementation of cluster run * fix cluster_run * start debug sequential user add and delete * parallelize user creation and deletion, start implementation of cluster scp * continue cluster_scp implementation * debug statements, disconnect error: permission denied * untesteed parakimo implementation of clus_run * continue debugging user creation bug * fix bug with pool user creation, start concurrent implementation * start fix of paramiko cluster_run and cluster_copy * working paramiko cluster_run implementation, start cluster_scp * fix cluster_scp command * update requirements, rename cluster_run function * remove unused shell functions * parallelize run and scp, add container_name, create logs wrapper * change scp to copy, clean up * sort imports * remove asyncssh from node requirements * remove old import * remove bad error handling * make cluster user management methods private * remove comment * remove accidental commit * fix merge, move delete to finally clause * add docs * formatting * Feature: Refactor cluster config to use ClusterConfiguration model (#343) * Bug: fix core-site.xml typo (#378) * fix typo * crlf->lf * Bug: fix regex for is_gpu_enabled (#380) * fix regex for is_gpu_enabled * crlf->lf * Bug: spark SDK example fix (#383) * start fix sdk * fix sdk example * crlf->lf * Fix: Custom scripts not read from cluster.yaml (#388) * Feature: spark shuffle service (#374) * start shuffle service by default * whitespace, delete misplaced file * crlf->lf * crlf->lf * move spark scratch space off os drive * Feature: enable dynamic allocation by default (#386) * Bug: stop using mutable default parameters (#392) * Bug: always upload spark job logs errors (#395) * Bug: spark submit upload error log type error (#397) * Bug: Spark Job list apps exit code 0 (#396) * Bug: fix spark-submit cores args (#399) * Fix: Trying to add user before master is ready show better error (#402) * Bug: move spark.local.dir to location usable by rstudioserver (#407) * Feature: SDK support for file-like configuration objects (#373) * add support for filelike objects for conifguration files * fix custom scripts * remove os.pathlike * merge error * Feature: Basic Cluster and Job Submission SDK Tests (#344) * add initial cluster tests * add cluster tests, add simple job submission test scenario * sort imports * fix job tests * fix job tests * remove pytest from travis build * cluster per test, parallel pytest plugin * delete cluster after tests, wait until deleted * fix bugs * catch right error, change cluster_id to base_cluster_id * fix test name * fixes * move tests to intregration_tests dir * update travis to run non-integration tests * directory structure, decoupled job tests * fix job tests, issue with submit_job * fix bug * add test docs * add cluster and job delete to finally clause * Feature: Spark add worker on master option (#415) * Add worker_on_master to ClusterConfiguration * add worker_on_master to JobConfiguration * Feature: task affinity to master node (#413) * Release: v0.6.0 (#416) * update changelog and version * underscores to stars 2018-02-24 02:18:03 +03:00			`# Distribution`
Feature/low pri (#4) * initial working version * update requirements.txt and commands to use low-pri * cluster commands should use --id instead of --cluster-id * make --size and --size-low-pri mutually exclusive * update readme to include low-pri flag 2017-06-20 07:58:56 +03:00			`azure-batch==3.0.0`
Feature: VNet support (#324) * VNet support * Azure active directory authentication 2018-01-17 22:14:56 +03:00			`azure-mgmt-batch==5.0.0`
			`azure-mgmt-storage==1.5.0`
az_spark_start + az_spark_submit 2017-04-07 23:55:05 +03:00			`azure-storage==0.33.0`
Feature: config files (#65) * moved secrets config template to config directory * updated new default file location for secrets.cfg * added ConfigObject class * added template for cluster config file * updated .gitignore to include cluster.yaml * updated cluster.yaml.template to include more fields * added read_config_file method to ConfigObject, modified cluster_create.py to use cluster.yaml configuration by default * added pyyaml to requirements.txt * added documentation for cluster.yaml config file * added constant for default cluster.yaml location * changed secrets.cfg to secrets.yaml, added SecretsConfig object * removed secrets.cfg.template * improved error messages for bad secrets.yaml config * changed config/ directory to .thunderbolt/ * remove old unused code * removed unncessary ignored file * renamed configuration class to ClusterConfig and added merge method with error checking * collapsed cluster.yaml file * updated cluster.yaml.template to new collapsed format * better descriptions of cluster config settings * added error checking to cluster config settings * added back accidentally removed code * removed template for cluster.yaml, added cluster.yaml * renamed .thunderbolt/ to config/, added defaults for cluster.yaml * fixed typo in secrets.yaml.template * added .thunderbolt/ directory to .gitignore * fixed bug when adding user with ssh_key * code cleanup * updated docs * removed redundant docs, added instructions to secrets.yaml.template * fixed typo * refactored read_config_file method and changed default cluster id * removed unnecessary ignored file * changed format for cluster.yaml instructions * added back docker-repo support * refactored merge and merge_dict methods 2017-09-20 00:08:26 +03:00			`pyyaml==3.12`
remove redundant setting in non-master code section and use non-os drive to mount HDFS (#242) 2017-11-30 23:17:50 +03:00			`pycryptodome==3.4.7`
Update stable to v0.6.0 (#418) * Feature: on node user creation (#303) * client side on node user creation * start create user on node implementation * fix on node user creation * remove debug statements * remove commented code * line too long * fix spinner password prompt ui bug * set wait to false by default, formatting * encrypt password on client, decrypt on node * update docs, log warning if password used * Fix list-apps crash (#364) * Allow submitting jobs into a VNET (#365) * Add subnet_id to job submission cluster config * add some docs * Feature: Spark mixed mode support (#350) * add support for aad creds for storage on node * add mixed mode support * add docs * switch error order * add dedicated to get_cluster * remove mixed mode in print_cluster_conf * Feature: spark init docker repo customization (#358) * customize docker_repo based on init args * whitespace * add some docs * r-base to r * case insensitive r flag, typo fix * Bug: Load default Jars for job submission CLI (#367) * load jars in .aztk/ by default * rewrite loading config files * Feature: Cluster Run and Copy (#304) * start implementation of cluster run * fix cluster_run * start debug sequential user add and delete * parallelize user creation and deletion, start implementation of cluster scp * continue cluster_scp implementation * debug statements, disconnect error: permission denied * untesteed parakimo implementation of clus_run * continue debugging user creation bug * fix bug with pool user creation, start concurrent implementation * start fix of paramiko cluster_run and cluster_copy * working paramiko cluster_run implementation, start cluster_scp * fix cluster_scp command * update requirements, rename cluster_run function * remove unused shell functions * parallelize run and scp, add container_name, create logs wrapper * change scp to copy, clean up * sort imports * remove asyncssh from node requirements * remove old import * remove bad error handling * make cluster user management methods private * remove comment * remove accidental commit * fix merge, move delete to finally clause * add docs * formatting * Feature: Refactor cluster config to use ClusterConfiguration model (#343) * Bug: fix core-site.xml typo (#378) * fix typo * crlf->lf * Bug: fix regex for is_gpu_enabled (#380) * fix regex for is_gpu_enabled * crlf->lf * Bug: spark SDK example fix (#383) * start fix sdk * fix sdk example * crlf->lf * Fix: Custom scripts not read from cluster.yaml (#388) * Feature: spark shuffle service (#374) * start shuffle service by default * whitespace, delete misplaced file * crlf->lf * crlf->lf * move spark scratch space off os drive * Feature: enable dynamic allocation by default (#386) * Bug: stop using mutable default parameters (#392) * Bug: always upload spark job logs errors (#395) * Bug: spark submit upload error log type error (#397) * Bug: Spark Job list apps exit code 0 (#396) * Bug: fix spark-submit cores args (#399) * Fix: Trying to add user before master is ready show better error (#402) * Bug: move spark.local.dir to location usable by rstudioserver (#407) * Feature: SDK support for file-like configuration objects (#373) * add support for filelike objects for conifguration files * fix custom scripts * remove os.pathlike * merge error * Feature: Basic Cluster and Job Submission SDK Tests (#344) * add initial cluster tests * add cluster tests, add simple job submission test scenario * sort imports * fix job tests * fix job tests * remove pytest from travis build * cluster per test, parallel pytest plugin * delete cluster after tests, wait until deleted * fix bugs * catch right error, change cluster_id to base_cluster_id * fix test name * fixes * move tests to intregration_tests dir * update travis to run non-integration tests * directory structure, decoupled job tests * fix job tests, issue with submit_job * fix bug * add test docs * add cluster and job delete to finally clause * Feature: Spark add worker on master option (#415) * Add worker_on_master to ClusterConfiguration * add worker_on_master to JobConfiguration * Feature: task affinity to master node (#413) * Release: v0.6.0 (#416) * update changelog and version * underscores to stars 2018-02-24 02:18:03 +03:00			`paramiko==2.4.0`

			`# Development`
			`yapf==0.20.1`
			`pylint==1.7.2`
			`pytest==3.1.3`
			`pytest-xdist==1.22.0`