* update changelog

* update versions
This commit is contained in:
Jacob Freck 2018-06-20 14:55:13 -07:00 коммит произвёл GitHub
Родитель 4e0b1ecd0f
Коммит a8f8e92629
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
8 изменённых файлов: 37 добавлений и 37 удалений

Просмотреть файл

@ -9,39 +9,39 @@
* Spark version 1.6 has been deprecated
**Added Features**
* add cluster list quiet flag, ability to compose with delete (#581) ([88d0419](https://github.com/jafreck/aztk/commit/88d0419)), closes [#581](https://github.com/jafreck/aztk/issues/581)
* add node run command (#572) ([af449dc](https://github.com/jafreck/aztk/commit/af449dc)), closes [#572](https://github.com/jafreck/aztk/issues/572)
* Add VSTS CI (#561) ([66037fd](https://github.com/jafreck/aztk/commit/66037fd)), closes [#561](https://github.com/jafreck/aztk/issues/561)
* Disable scheduling on group of nodes (#540) ([8fea9ce](https://github.com/jafreck/aztk/commit/8fea9ce)), closes [#540](https://github.com/jafreck/aztk/issues/540)
* New Models design with auto validation, default and merging (#543) ([02f336b](https://github.com/jafreck/aztk/commit/02f336b)), closes [#543](https://github.com/jafreck/aztk/issues/543)
* nvBLAS and OpenBLAS plugin (#539) ([603a413](https://github.com/jafreck/aztk/commit/603a413)), closes [#539](https://github.com/jafreck/aztk/issues/539)
* pure python ssh (#577) ([f16aac0](https://github.com/jafreck/aztk/commit/f16aac0)), closes [#577](https://github.com/jafreck/aztk/issues/577)
* Support passing of remote executables via aztk spark cluster submit (#549) ([f6735cc](https://github.com/jafreck/aztk/commit/f6735cc)), closes [#549](https://github.com/jafreck/aztk/issues/549)
* TensorflowOnSpark python plugin (#525) ([1527929](https://github.com/jafreck/aztk/commit/1527929)), closes [#525](https://github.com/jafreck/aztk/issues/525)
* Conda, Apt-Get and Pip Install Plugins (#594) ([fbf1bab](https://github.com/jafreck/aztk/commit/fbf1bab)), closes [#594](https://github.com/jafreck/aztk/issues/594)
* Warnings show stacktrace on verbose (#587) ([b9a863b](https://github.com/jafreck/aztk/commit/b9a863b)), closes [#587](https://github.com/jafreck/aztk/issues/587)
* add cluster list quiet flag, ability to compose with delete (#581) ([88d0419](https://github.com/Azure/aztk/commit/88d0419)), closes [#581](https://github.com/Azure/aztk/issues/581)
* add node run command (#572) ([af449dc](https://github.com/Azure/aztk/commit/af449dc)), closes [#572](https://github.com/Azure/aztk/issues/572)
* Add VSTS CI (#561) ([66037fd](https://github.com/Azure/aztk/commit/66037fd)), closes [#561](https://github.com/Azure/aztk/issues/561)
* Disable scheduling on group of nodes (#540) ([8fea9ce](https://github.com/Azure/aztk/commit/8fea9ce)), closes [#540](https://github.com/Azure/aztk/issues/540)
* New Models design with auto validation, default and merging (#543) ([02f336b](https://github.com/Azure/aztk/commit/02f336b)), closes [#543](https://github.com/Azure/aztk/issues/543)
* nvBLAS and OpenBLAS plugin (#539) ([603a413](https://github.com/Azure/aztk/commit/603a413)), closes [#539](https://github.com/Azure/aztk/issues/539)
* pure python ssh (#577) ([f16aac0](https://github.com/Azure/aztk/commit/f16aac0)), closes [#577](https://github.com/Azure/aztk/issues/577)
* Support passing of remote executables via aztk spark cluster submit (#549) ([f6735cc](https://github.com/Azure/aztk/commit/f6735cc)), closes [#549](https://github.com/Azure/aztk/issues/549)
* TensorflowOnSpark python plugin (#525) ([1527929](https://github.com/Azure/aztk/commit/1527929)), closes [#525](https://github.com/Azure/aztk/issues/525)
* Conda, Apt-Get and Pip Install Plugins (#594) ([fbf1bab](https://github.com/Azure/aztk/commit/fbf1bab)), closes [#594](https://github.com/Azure/aztk/issues/594)
* Warnings show stacktrace on verbose (#587) ([b9a863b](https://github.com/Azure/aztk/commit/b9a863b)), closes [#587](https://github.com/Azure/aztk/issues/587)
**Bug Fixes**
* add toolkit to sdk docs and example ([d688c9c](https://github.com/jafreck/aztk/commit/d688c9c))
* --size-low-pri being ignored (#593) ([fa3ac0e](https://github.com/jafreck/aztk/commit/fa3ac0e)), closes [#593](https://github.com/jafreck/aztk/issues/593)
* fix typos (#595) ([7d7a814](https://github.com/jafreck/aztk/commit/7d7a814)), closes [#595](https://github.com/jafreck/aztk/issues/595)
* getting started script reuse aad application (#569) ([3d16cf3](https://github.com/jafreck/aztk/commit/3d16cf3)), closes [#569](https://github.com/jafreck/aztk/issues/569)
* models v2 deserialization (#584) ([1eeff23](https://github.com/jafreck/aztk/commit/1eeff23)), closes [#584](https://github.com/jafreck/aztk/issues/584)
* optimize start task (#582) ([e5e529a](https://github.com/jafreck/aztk/commit/e5e529a)), closes [#582](https://github.com/jafreck/aztk/issues/582)
* remove deprecated vm_count call (#586) ([dbde8bc](https://github.com/jafreck/aztk/commit/dbde8bc)), closes [#586](https://github.com/jafreck/aztk/issues/586)
* Remove old spark-defaults.conf jars (#567) ([8b8cd62](https://github.com/jafreck/aztk/commit/8b8cd62)), closes [#567](https://github.com/jafreck/aztk/issues/567)
* set logger to stdout (#588) ([3f0c8f9](https://github.com/jafreck/aztk/commit/3f0c8f9)), closes [#588](https://github.com/jafreck/aztk/issues/588)
* switch create user to pool wide (#574) ([49a890a](https://github.com/jafreck/aztk/commit/49a890a)), closes [#574](https://github.com/jafreck/aztk/issues/574)
* switch from pycryptodome to pycryptodomex (#564) ([19dde42](https://github.com/jafreck/aztk/commit/19dde42)), closes [#564](https://github.com/jafreck/aztk/issues/564)
* allow cluster config to be printed when no username has been set (#597) ([1cc71c7](https://github.com/jafreck/aztk/commit/1cc71c7)), closes [#597](https://github.com/jafreck/aztk/issues/597)
* add toolkit to sdk docs and example ([d688c9c](https://github.com/Azure/aztk/commit/d688c9c))
* --size-low-pri being ignored (#593) ([fa3ac0e](https://github.com/Azure/aztk/commit/fa3ac0e)), closes [#593](https://github.com/Azure/aztk/issues/593)
* fix typos (#595) ([7d7a814](https://github.com/Azure/aztk/commit/7d7a814)), closes [#595](https://github.com/Azure/aztk/issues/595)
* getting started script reuse aad application (#569) ([3d16cf3](https://github.com/Azure/aztk/commit/3d16cf3)), closes [#569](https://github.com/Azure/aztk/issues/569)
* models v2 deserialization (#584) ([1eeff23](https://github.com/Azure/aztk/commit/1eeff23)), closes [#584](https://github.com/Azure/aztk/issues/584)
* optimize start task (#582) ([e5e529a](https://github.com/Azure/aztk/commit/e5e529a)), closes [#582](https://github.com/Azure/aztk/issues/582)
* remove deprecated vm_count call (#586) ([dbde8bc](https://github.com/Azure/aztk/commit/dbde8bc)), closes [#586](https://github.com/Azure/aztk/issues/586)
* Remove old spark-defaults.conf jars (#567) ([8b8cd62](https://github.com/Azure/aztk/commit/8b8cd62)), closes [#567](https://github.com/Azure/aztk/issues/567)
* set logger to stdout (#588) ([3f0c8f9](https://github.com/Azure/aztk/commit/3f0c8f9)), closes [#588](https://github.com/Azure/aztk/issues/588)
* switch create user to pool wide (#574) ([49a890a](https://github.com/Azure/aztk/commit/49a890a)), closes [#574](https://github.com/Azure/aztk/issues/574)
* switch from pycryptodome to pycryptodomex (#564) ([19dde42](https://github.com/Azure/aztk/commit/19dde42)), closes [#564](https://github.com/Azure/aztk/issues/564)
* allow cluster config to be printed when no username has been set (#597) ([1cc71c7](https://github.com/Azure/aztk/commit/1cc71c7)), closes [#597](https://github.com/Azure/aztk/issues/597)
## 0.7.1 (2018-05-11)
**Bug Fixes**
* Fix: create virtual environment even if container exists ([2db7b00](https://github.com/jafreck/aztk/commit/2db7b00))
* Fix: gitattributes for jar files (#548) ([a18660b](https://github.com/jafreck/aztk/commit/a18660b)), closes [#548](https://github.com/jafreck/aztk/issues/548)
* Fix: pass docker repo command back to the cluster config (#538) ([a99bbe1](https://github.com/jafreck/aztk/commit/a99bbe1)), closes [#538](https://github.com/jafreck/aztk/issues/538)
* Fix: create virtual environment even if container exists ([2db7b00](https://github.com/Azure/aztk/commit/2db7b00))
* Fix: gitattributes for jar files (#548) ([a18660b](https://github.com/Azure/aztk/commit/a18660b)), closes [#548](https://github.com/Azure/aztk/issues/548)
* Fix: pass docker repo command back to the cluster config (#538) ([a99bbe1](https://github.com/Azure/aztk/commit/a99bbe1)), closes [#538](https://github.com/Azure/aztk/issues/538)
## 0.7.0 (2018-05-01)

Просмотреть файл

@ -27,7 +27,7 @@ This toolkit is built on top of Azure Batch but does not require any Azure Batch
```
3. Login or register for an [Azure Account](https://azure.microsoft.com), navigate to [Azure Cloud Shell](https://shell.azure.com), and run:
```sh
wget -q https://raw.githubusercontent.com/Azure/aztk/v0.7.0/account_setup.sh -O account_setup.sh &&
wget -q https://raw.githubusercontent.com/Azure/aztk/v0.8.1/account_setup.sh -O account_setup.sh &&
chmod 755 account_setup.sh &&
/bin/bash account_setup.sh
```

Просмотреть файл

@ -4,7 +4,7 @@ echo "Installing dependencies..." &&
pip install --force-reinstall --upgrade --user pyyaml==3.12 azure==3.0.0 azure-cli-core==2.0.30 msrestazure==0.4.25 > /dev/null 2>&1 &&
echo "Finished installing dependencies." &&
echo "Getting account setup script..." &&
wget -q https://raw.githubusercontent.com/Azure/aztk/v0.7.0/account_setup.py -O account_setup.py &&
wget -q https://raw.githubusercontent.com/Azure/aztk/v0.8.1/account_setup.py -O account_setup.py &&
chmod 755 account_setup.py &&
echo "Finished getting account setup script." &&
echo "Running account setup script..." &&

Просмотреть файл

@ -1,7 +1,7 @@
# Job Configuration
# An Aztk Job is a cluster and an array of Spark applications to run on that cluster
# AZTK Spark Jobs will automatically manage the lifecycle of the cluster
# For more information see the documentation at: https://github.com/Azure/aztk/blob/v0.7.0/docs/70-jobs.md
# For more information see the documentation at: http://aztk.readthedocs.io/en/latest/70-jobs.html
job:
id:

Просмотреть файл

@ -1,5 +1,5 @@
# For instructions on creating a Batch and Storage account, see
# Getting Started (https://github.com/Azure/aztk/blob/v0.7.0/docs/00-getting-started.md)
# Getting Started (http://aztk.readthedocs.io/en/latest/00-getting-started.html)
# NOTE - YAML requires a space after the colon. Ex: "batchaccountname: mybatchaccount"
service_principal:

Просмотреть файл

@ -14,7 +14,7 @@ The script outputs all of the necessary information to use `aztk`, just copy the
## Usage
Copy and paste the following into an [Azure Cloud Shell](https://shell.azure.com):
```sh
wget -q https://raw.githubusercontent.com/Azure/aztk/v0.7.0/account_setup.sh &&
wget -q https://raw.githubusercontent.com/Azure/aztk/v0.8.1/account_setup.sh &&
chmod 755 account_setup.sh &&
/bin/bash account_setup.sh
```

Просмотреть файл

@ -17,7 +17,7 @@ aztk spark cluster create ... --docker-repo aztk/base:spark1.6.3
```
## Using a custom Docker Image
You can build your own Docker image on top or beneath one of our supported base images _OR_ you can modify the [supported Dockerfiles](https://github.com/Azure/aztk/tree/v0.7.0/docker-image) and build your own image that way.
You can build your own Docker image on top or beneath one of our supported base images _OR_ you can modify the [supported Dockerfiles](https://github.com/Azure/aztk/tree/v0.8.1/docker-image) and build your own image that way.
Once you have your Docker image built and hosted publicly, you can then use the **--docker-repo** parameter in your **aztk spark cluster create** command to point to it.
@ -92,4 +92,4 @@ If you are using your own version of Spark, make that it is symlinked by "/home/
## Hosting your Docker Image
By default, aztk assumes that your Docker images are publicly hosted on Docker Hub. However, we also support hosting your images privately.
See [here](https://github.com/Azure/aztk/blob/v0.7.0/docs/12-docker-image.md#using-a-custom-docker-image-that-is-privately-hosted) to learn more about using privately hosted Docker Images.
See [here](12-docker-image.html#using-a-custom-docker-image-that-is-privately-hosted) to learn more about using privately hosted Docker Images.

Просмотреть файл

@ -33,16 +33,16 @@ A number of changes have been made that affect previously init'ed aztk environme
7. Create your new 0.7.0 cluster!
### cluster.yaml
In cluster.yaml, the `toolkit` key has been added. It is used to select the default, supported Docker images. Please refer to [the configuration file documentation.](https://aztk.readthedocs.io/en/v0.7.0/13-configuration.html#cluster-yaml)
In cluster.yaml, the `toolkit` key has been added. It is used to select the default, supported Docker images. Please refer to [the configuration file documentation.](13-configuration.html#cluster-yaml)
## Docker images
A major backwards-incompatible refactor of the Docker images has occurred. Previous Docker images will no longer work with 0.7.0. To update to a new supported docker image, you will need to update your `.aztk/cluster.yaml` configuration file with the `toolkit` block in place of `docker_repo`. If you do not do so, cluster creation will fail!
Please refer to the [the configuration file documentation](https://aztk.readthedocs.io/en/v0.7.0/13-configuration.html#cluster-yaml) for more information on the `toolkit` in `cluster.yaml`.
Please refer to the [the configuration file documentation](13-configuration.html#cluster-yaml) for more information on the `toolkit` in `cluster.yaml`.
## Custom scripts depreciation and Plugins
Custom scripts have been depreciated in favor of Plugins. Plugins have a number of advantages, including the ability to execute on the host (and not in the Spark Docker container). A number of supported plugins are shipped with aztk, please refer to [the plugin documentation to learn more.](https://aztk.readthedocs.io/en/v0.7.0/15-plugins.html)
Custom scripts have been depreciated in favor of Plugins. Plugins have a number of advantages, including the ability to execute on the host (and not in the Spark Docker container). A number of supported plugins are shipped with aztk, please refer to [the plugin documentation to learn more.](15-plugins.html)
## Read the Docs
[Documentation has migrated to readthedocs](https://aztk.readthedocs.io/en/v0.7.0).
[Documentation has migrated to readthedocs](https://aztk.readthedocs.io/en/latest).