8f8e7b268b
Update README.md ( #728 )
2021-02-02 09:38:45 -08:00
6e04372d19
Release v0.10.3 ( #714 )
...
* Build(deps): Bump requests from 2.19.1 to 2.20.0 in /aztk/node_scripts
Bumps [requests](https://github.com/requests/requests ) from 2.19.1 to 2.20.0.
- [Release notes](https://github.com/requests/requests/releases )
- [Changelog](https://github.com/psf/requests/blob/master/HISTORY.md )
- [Commits](https://github.com/requests/requests/compare/v2.19.1...v2.20.0 )
Signed-off-by: dependabot[bot] <support@github.com>
* Build(deps): Bump pyyaml from 3.13 to 5.1 in /aztk/node_scripts
Bumps [pyyaml](https://github.com/yaml/pyyaml ) from 3.13 to 5.1.
- [Release notes](https://github.com/yaml/pyyaml/releases )
- [Changelog](https://github.com/yaml/pyyaml/blob/master/CHANGES )
- [Commits](https://github.com/yaml/pyyaml/compare/3.13...5.1 )
Signed-off-by: dependabot[bot] <support@github.com>
* update changelog for 0.10.3 release
* update requirements and pipfile
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2020-02-14 15:27:19 -08:00
1f6fc8b8e1
Release: v0.10.2 ( #691 )
...
* update changelog
* update version
* update version in docs links
2018-12-07 12:58:32 -08:00
5379b00b73
Release: 0.10.1 ( #682 )
...
* update version
* update changelog for 0.10.1 release
* update relevent links
2018-11-02 11:45:53 -07:00
b7da355618
Release: 0.9.1 ( #669 )
...
* update changelog and version
* update docs links to new version number
2018-10-05 13:07:15 -07:00
442228a30f
Deprecate: remove custom scripts ( #650 )
2018-08-17 20:36:11 -04:00
a8f8e92629
Fix: docs links version ( #614 )
...
* update changelog
* update versions
2018-06-20 14:55:13 -07:00
7d7a814c50
Fix: fix typos ( #595 )
2018-06-07 09:57:43 -07:00
779bffb2da
Feature: refactor docker images ( #510 )
...
* add spark2.3.0 hadoop2.8.3 dockerfile
* start update to docker image
* add SPARK_DIST_CLASSPATH to bashrc, source .bashrc in docker run
* add maven install for jars
* docker image update and code fix
* add libthrift (still broken)
* start image refactor, build from source,
* add refactor to r base image
* finish refactor r image
* add storage jars and deps
* exclude netty to get rid of dependency conflict
* add miniconda image
* update 2.2.0 base, anaconda image
* remove unused cuda-8.0 image
* start pipenv implementation
* miniconda version arg
* update anaconda and miniconda image
* style
* pivot to virtualenv
* remove virtualenv from path when submitting apps
* flatten layers
* explicit calls to aztk python instead of activating virtualenv
* update base, miniconda, anaconda
* add compatibility version for base aztk images
* typo fix
* update pom
* update environment variable name
* update environment variables
* add anaconda images base & gpu
* update gpu and miniconda base images
* create venv in cluster create
* update base docker files, remove virtualenv
* fix path
* add exclusion to base images
* update r images
* delete python images (in favor of anaconda and miniconda)
* add miniconda gpu images
* update comment
* update aztk_version_compatibility to dokcer image version
* add a build script
* virutalenv->pipenv, add pipfile & pipfile.lock remove secretstorage
* aztk/staging->aztk/spark
* remove jars, add .null to keep directory
* update pipfile, update jupyter and jupyterlab
* update default images
* update base images to fix hdfs
* update build script with correct path
* add spark1.6.3 anaconda, miniconda, r base and gpu images
* update build script to include spark1.6.3
* mkdir out
* exclude commons lang and slf4j dependencies
* mkdir out
* no fail if dir exists
* update node_scripts
* update env var name
* update env var name
* fix the docker_repo docs
* master->0.7.0
2018-04-30 17:19:01 -07:00
7ef721f0c1
Feature: getting started script ( #475 )
...
* initial changes for getting started scripts
* add temp error handling
* rename file - fix typo
* add debug strings
* add handling for existing user
* WIP: wait for subprocess to complete to get exit code
* WIP: handle existing user and refactor code
* WIP: add missing return statements
* WIP: fix typo
* start sdk refactor
* mostly working create
* working happy create path
* handle errors for vnet, aad application
* make account setup interactive
* add prompt
* add docs
* rename account_setup_refac to account_setup
* add some logging
* pip install msrest, azure-cli-core, import issues
* remove in script pip, add shell wrapper program
* ellipsis to period
* update branch name for account_setup.sh
* docstring
* retry resource group creation
* fix typo, update retry
* explicitly set output location
* wget overwrite flag, docs update
* add prompt for multi tenants
* fix bug with batch account creation
* add spinner, print statements, fix formatting bug
* fix param bug
2018-04-11 13:27:55 -07:00
748a1269fa
Feature: Spark mixed mode support ( #350 )
...
* add support for aad creds for storage on node
* add mixed mode support
* add docs
* switch error order
* add dedicated to get_cluster
* remove mixed mode in print_cluster_conf
2018-02-07 10:41:51 -08:00
85a472c591
Feature: on node user creation ( #303 )
...
* client side on node user creation
* start create user on node implementation
* fix on node user creation
* remove debug statements
* remove commented code
* line too long
* fix spinner password prompt ui bug
* set wait to false by default, formatting
* encrypt password on client, decrypt on node
* update docs, log warning if password used
2018-01-26 16:14:15 -08:00
6091b1d390
Docs: update ( #263 )
...
* Update README.md
streamline and update main readme.md
* Update README.md
* Update README.md
* Update 13-configuration.md
* Update 12-docker-image.md
* Update 12-docker-image.md
* Update README.md
* Create README.md
* Update README.md
* Update 10-clusters.md
2017-12-11 17:02:51 -08:00
ea7e17aec4
Update README.md ( #225 )
...
If you set both --size and --size-low-pri you receive an error...
dciborow@6433739-0524:/mnt/c/GIT/ABN/spark-recommender$ aztk spark cluster create --id testCluster1 --size 0 --size-low-pri 2 --vm-size standard_d2_v2
usage: aztk spark cluster create [-h] [--id CLUSTER_ID]
[--size SIZE | --size-low-pri SIZE_LOW_PRI]
[--vm-size VM_SIZE] [--username USERNAME]
[--password PASSWORD] [--ssh-key SSH_KEY]
[--docker-repo DOCKER_REPO] [--no-wait]
[--wait]
aztk spark cluster create: error: argument --size-low-pri: not allowed with argument --size
2017-11-21 10:09:15 -08:00
65a56dd657
Feature/python container ( #210 )
...
* added python container, jupyter install script, vanilla container
* Update README.md
* Create README.md
* Create README.md
* Update README.md
* Update README.md
* Update README.md
* Update README.md
* Update README.md
* Update jupyter.sh
* Update README.md
* Update README.md
* python
* readme update
* docker updates
* Update README.md
* Update 12-docker-image.md
* Update constants.py
* add image files for wiki
* update imageS
* .
* dockerfile typo
* dockerfiles
* Removed r
* readme
* update constants.py
* update readme
* readme updates
* readme updates
2017-11-09 00:43:32 -08:00
dbc1adddcd
Feature: Azure Data Lake Store support ( #170 )
...
* initial adls work
* initial native support for ADLS connector
* add core-site.xml and azure storage/adls jars to project
* enable custom jars and core-site.xml changes for custom storage connectors
* remove unused ADL crendentials from environment
* documentation feedback updates
* documentation feedback
* PR feedback
2017-10-19 09:16:14 -07:00
5c3b4d67f3
Update README.md ( #146 )
2017-10-04 16:38:04 -07:00
4fd8772869
Update README.md ( #136 )
2017-10-03 17:30:09 -07:00
f158ceafc5
Docs/update ( #128 )
...
* Update README.md
* Update 10-clusters.md
* Update README.md
* Update README.md
* Update README.md
2017-09-30 05:02:25 -04:00
57cfa9dd1d
Docs/update ( #106 )
...
* Updates to README.md
* typos
* Update 00-getting-started.md
* Update README.md
* Update 30-cloud-storage.md
* faq in readme
* Update 13-configuration.md
* switch to use underscore for consistency
* Update 10-clusters.md
* Update README.md
* Update README.md
* fixes
* fix broken link
* Update 20-spark-submit.md
* update to aztk on README
* aztk fix
* Update 00-getting-started.md
* Update 10-clusters.md
* Update 12-docker-image.md
* Update 13-configuration.md
* Update 20-spark-submit.md
* Update 30-cloud-storage.md
* Update README.md
* Update 10-clusters.md
* vm size link
* dummy commit1
* dummy commit 2
* Update 00-getting-started.md
* Update 13-configuration.md
2017-09-29 22:23:14 -04:00
43dd8e57f1
Bug: Move commands under app to cluster ( #82 )
...
* moved azb spark app commands to azb spark cluster
* fixed docs to reflect new cli structure
* renamed commands for clarity
2017-09-19 08:29:23 -07:00
03baff6a26
Feature: custom docker images ( #68 )
...
* symlink /home/spark-version to /home/spark-current
* working checkpoint
* working checkpoint
* docker readme update
* removed deprecated dockerfile
* docker cli + docs for docker
* docs update
* docs update
* remove unneeded comments
* docker version and docs updates, misc fixes
2017-09-15 20:57:16 -07:00
551d8a1a45
Fix documentation links and typos ( #56 )
2017-09-11 21:34:49 -07:00
636cd2e1d6
Update README.md ( #54 )
...
* Update README.md
update readme for a more descriptive introduction
* upper case S for spark
* fix
2017-09-11 13:41:33 -07:00
5edd120c52
fix broken links in docs/README ( #55 )
...
* fix broken links in docs/README
* typo
2017-09-11 13:17:46 -07:00
219cf412c0
update docs with new name 'Thunderbolt' ( #53 )
...
* update docs with new name 'Thunderbolt'
* update instructions around pip for clarity
2017-08-23 11:37:13 -07:00
fdd87c0a99
Feature/samples ( #50 )
...
* added snapshot of examples from spark-2.2.0
* jupyter samples
* fixed bug
* rename 'pyspark' to 'jupyter'
* curated samples
* Readme update for samples
2017-08-22 18:36:25 -04:00
ae26453ba5
Feature/docs ( #51 )
...
* update configuration.cfg to secrets.cfg
* fix node count pretty print
* add more unit tests
* configure pytest to only run tests from the 'tests' directory
* add missing space
* remove static call to loading secrets file
* initial docs layout
* rename files and add images to getting started
* fix typos
* Add next steps to clusters docs
* Add custom scripts doc
* Initial text for spark submit
* fix merge conflict
* finish spark submit docs
* cloud storage docs
* fix typo in storage docs
* refactor some docs
* pr feedback
* add link to Getting Started in creating account details in main readme
2017-08-22 09:13:10 -07:00
577ff296b5
Feature/wasb ( #47 )
...
* update configuration.cfg to secrets.cfg
* fix node count pretty print
* add more unit tests
* configure pytest to only run tests from the 'tests' directory
* add missing space
* remove static call to loading secrets file
* initial commit for automatic wasb support
* wasb native support
* fix for wasb
* bug fix in azure_api
* clean up setup_wasb
* README for wasb
* README updates for wasb
* configure wasb creds in start task environment variables
* add missing merged files
* revert back to working state
* added storage account suffix
* removed zip to secrets
2017-08-22 11:57:45 -04:00
4822172b6a
Feature/secrets.cfg ( #43 )
...
* update configuration.cfg to secrets.cfg
* fix node count pretty print
* add more unit tests
* configure pytest to only run tests from the 'tests' directory
* add missing space
* remove static call to loading secrets file
2017-08-21 12:36:45 -07:00
d552624705
Feature: read/tail spark submit logs ( #40 )
...
* Tail log command
* tail log is working
* Cleanup code
* fix cr
* Fix cr
* Submit
* Comment
* fix tail
* Sleep
2017-08-18 12:35:04 -07:00
d94c43ed82
Feature: docker
...
Feature: docker
2017-08-17 09:21:00 -07:00
fbfdaf489d
remove deprecated instructions ( #39 )
...
* remove deprecated instructions
* Update README.md
2017-08-10 11:00:43 -07:00
aea5e36690
Feature: Setup user with ssh public key instead of password ( #32 )
...
* Login with ssh public key
* Update to have ssh-key command
* Fix readme
* Fix Cr
* remove unused files
* Change head to master
2017-07-27 10:45:31 -07:00
9e45f78600
Simplify the commands to all be under azb and make windows experience better ( #29 )
...
* Added some commands
* More
* Missing file
* Refactor
* update
* Constant exe name
* List and get command
* Added user
* update cluster create, and job submit
* Remove legacy scripts
* Fix cr comments
* remove leftover print
2017-07-19 10:48:23 -07:00
0b99e86307
added naming condition to spark jobs & clusters ( #14 )
...
* added naming condition to spark jobs
* Update README.md
2017-07-06 11:33:09 -07:00
b98ae85726
Merge pull request #7 from Azure/feature/create-user-with-cluster
...
create user option on cluster create
2017-07-05 10:04:36 -07:00
1fe33c3d55
Corrections to README ( #8 )
2017-07-04 21:27:46 -07:00
61dc9d32b2
Update README.md
2017-06-26 18:49:40 -04:00
0e9da1cfba
Update README.md
2017-06-20 01:03:11 -04:00
2f324fca9c
Feature/low pri ( #4 )
...
* initial working version
* update requirements.txt and commands to use low-pri
* cluster commands should use --id instead of --cluster-id
* make --size and --size-low-pri mutually exclusive
* update readme to include low-pri flag
2017-06-19 21:58:56 -07:00
5abeaceb66
Update README.md
2017-06-15 22:46:01 -07:00
6cd8c8ce62
Merged PR 5: Merge feature/batch2.0.0 to master
2017-05-12 20:25:47 +00:00
85c1d27ab0
Merged PR 4: Merge readme-update to master
2017-04-20 23:54:36 +00:00
50073cf2c9
readme.md
2017-04-13 01:47:44 +00:00
ace37b0fea
first commit
2017-04-06 21:02:15 +00:00
Jacob Freck
mmduyzend
JS
Daniel Ciborowski
Pablo Selem
Amol Thacker
Timothee Guerin
paselem