diff --git a/CHANGELOG.md b/CHANGELOG.md index 0f912420..a136df75 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,39 +9,39 @@ * Spark version 1.6 has been deprecated **Added Features** -* add cluster list quiet flag, ability to compose with delete (#581) ([88d0419](https://github.com/jafreck/aztk/commit/88d0419)), closes [#581](https://github.com/jafreck/aztk/issues/581) -* add node run command (#572) ([af449dc](https://github.com/jafreck/aztk/commit/af449dc)), closes [#572](https://github.com/jafreck/aztk/issues/572) -* Add VSTS CI (#561) ([66037fd](https://github.com/jafreck/aztk/commit/66037fd)), closes [#561](https://github.com/jafreck/aztk/issues/561) -* Disable scheduling on group of nodes (#540) ([8fea9ce](https://github.com/jafreck/aztk/commit/8fea9ce)), closes [#540](https://github.com/jafreck/aztk/issues/540) -* New Models design with auto validation, default and merging (#543) ([02f336b](https://github.com/jafreck/aztk/commit/02f336b)), closes [#543](https://github.com/jafreck/aztk/issues/543) -* nvBLAS and OpenBLAS plugin (#539) ([603a413](https://github.com/jafreck/aztk/commit/603a413)), closes [#539](https://github.com/jafreck/aztk/issues/539) -* pure python ssh (#577) ([f16aac0](https://github.com/jafreck/aztk/commit/f16aac0)), closes [#577](https://github.com/jafreck/aztk/issues/577) -* Support passing of remote executables via aztk spark cluster submit (#549) ([f6735cc](https://github.com/jafreck/aztk/commit/f6735cc)), closes [#549](https://github.com/jafreck/aztk/issues/549) -* TensorflowOnSpark python plugin (#525) ([1527929](https://github.com/jafreck/aztk/commit/1527929)), closes [#525](https://github.com/jafreck/aztk/issues/525) -* Conda, Apt-Get and Pip Install Plugins (#594) ([fbf1bab](https://github.com/jafreck/aztk/commit/fbf1bab)), closes [#594](https://github.com/jafreck/aztk/issues/594) -* Warnings show stacktrace on verbose (#587) ([b9a863b](https://github.com/jafreck/aztk/commit/b9a863b)), closes [#587](https://github.com/jafreck/aztk/issues/587) +* add cluster list quiet flag, ability to compose with delete (#581) ([88d0419](https://github.com/Azure/aztk/commit/88d0419)), closes [#581](https://github.com/Azure/aztk/issues/581) +* add node run command (#572) ([af449dc](https://github.com/Azure/aztk/commit/af449dc)), closes [#572](https://github.com/Azure/aztk/issues/572) +* Add VSTS CI (#561) ([66037fd](https://github.com/Azure/aztk/commit/66037fd)), closes [#561](https://github.com/Azure/aztk/issues/561) +* Disable scheduling on group of nodes (#540) ([8fea9ce](https://github.com/Azure/aztk/commit/8fea9ce)), closes [#540](https://github.com/Azure/aztk/issues/540) +* New Models design with auto validation, default and merging (#543) ([02f336b](https://github.com/Azure/aztk/commit/02f336b)), closes [#543](https://github.com/Azure/aztk/issues/543) +* nvBLAS and OpenBLAS plugin (#539) ([603a413](https://github.com/Azure/aztk/commit/603a413)), closes [#539](https://github.com/Azure/aztk/issues/539) +* pure python ssh (#577) ([f16aac0](https://github.com/Azure/aztk/commit/f16aac0)), closes [#577](https://github.com/Azure/aztk/issues/577) +* Support passing of remote executables via aztk spark cluster submit (#549) ([f6735cc](https://github.com/Azure/aztk/commit/f6735cc)), closes [#549](https://github.com/Azure/aztk/issues/549) +* TensorflowOnSpark python plugin (#525) ([1527929](https://github.com/Azure/aztk/commit/1527929)), closes [#525](https://github.com/Azure/aztk/issues/525) +* Conda, Apt-Get and Pip Install Plugins (#594) ([fbf1bab](https://github.com/Azure/aztk/commit/fbf1bab)), closes [#594](https://github.com/Azure/aztk/issues/594) +* Warnings show stacktrace on verbose (#587) ([b9a863b](https://github.com/Azure/aztk/commit/b9a863b)), closes [#587](https://github.com/Azure/aztk/issues/587) **Bug Fixes** -* add toolkit to sdk docs and example ([d688c9c](https://github.com/jafreck/aztk/commit/d688c9c)) -* --size-low-pri being ignored (#593) ([fa3ac0e](https://github.com/jafreck/aztk/commit/fa3ac0e)), closes [#593](https://github.com/jafreck/aztk/issues/593) -* fix typos (#595) ([7d7a814](https://github.com/jafreck/aztk/commit/7d7a814)), closes [#595](https://github.com/jafreck/aztk/issues/595) -* getting started script reuse aad application (#569) ([3d16cf3](https://github.com/jafreck/aztk/commit/3d16cf3)), closes [#569](https://github.com/jafreck/aztk/issues/569) -* models v2 deserialization (#584) ([1eeff23](https://github.com/jafreck/aztk/commit/1eeff23)), closes [#584](https://github.com/jafreck/aztk/issues/584) -* optimize start task (#582) ([e5e529a](https://github.com/jafreck/aztk/commit/e5e529a)), closes [#582](https://github.com/jafreck/aztk/issues/582) -* remove deprecated vm_count call (#586) ([dbde8bc](https://github.com/jafreck/aztk/commit/dbde8bc)), closes [#586](https://github.com/jafreck/aztk/issues/586) -* Remove old spark-defaults.conf jars (#567) ([8b8cd62](https://github.com/jafreck/aztk/commit/8b8cd62)), closes [#567](https://github.com/jafreck/aztk/issues/567) -* set logger to stdout (#588) ([3f0c8f9](https://github.com/jafreck/aztk/commit/3f0c8f9)), closes [#588](https://github.com/jafreck/aztk/issues/588) -* switch create user to pool wide (#574) ([49a890a](https://github.com/jafreck/aztk/commit/49a890a)), closes [#574](https://github.com/jafreck/aztk/issues/574) -* switch from pycryptodome to pycryptodomex (#564) ([19dde42](https://github.com/jafreck/aztk/commit/19dde42)), closes [#564](https://github.com/jafreck/aztk/issues/564) -* allow cluster config to be printed when no username has been set (#597) ([1cc71c7](https://github.com/jafreck/aztk/commit/1cc71c7)), closes [#597](https://github.com/jafreck/aztk/issues/597) +* add toolkit to sdk docs and example ([d688c9c](https://github.com/Azure/aztk/commit/d688c9c)) +* --size-low-pri being ignored (#593) ([fa3ac0e](https://github.com/Azure/aztk/commit/fa3ac0e)), closes [#593](https://github.com/Azure/aztk/issues/593) +* fix typos (#595) ([7d7a814](https://github.com/Azure/aztk/commit/7d7a814)), closes [#595](https://github.com/Azure/aztk/issues/595) +* getting started script reuse aad application (#569) ([3d16cf3](https://github.com/Azure/aztk/commit/3d16cf3)), closes [#569](https://github.com/Azure/aztk/issues/569) +* models v2 deserialization (#584) ([1eeff23](https://github.com/Azure/aztk/commit/1eeff23)), closes [#584](https://github.com/Azure/aztk/issues/584) +* optimize start task (#582) ([e5e529a](https://github.com/Azure/aztk/commit/e5e529a)), closes [#582](https://github.com/Azure/aztk/issues/582) +* remove deprecated vm_count call (#586) ([dbde8bc](https://github.com/Azure/aztk/commit/dbde8bc)), closes [#586](https://github.com/Azure/aztk/issues/586) +* Remove old spark-defaults.conf jars (#567) ([8b8cd62](https://github.com/Azure/aztk/commit/8b8cd62)), closes [#567](https://github.com/Azure/aztk/issues/567) +* set logger to stdout (#588) ([3f0c8f9](https://github.com/Azure/aztk/commit/3f0c8f9)), closes [#588](https://github.com/Azure/aztk/issues/588) +* switch create user to pool wide (#574) ([49a890a](https://github.com/Azure/aztk/commit/49a890a)), closes [#574](https://github.com/Azure/aztk/issues/574) +* switch from pycryptodome to pycryptodomex (#564) ([19dde42](https://github.com/Azure/aztk/commit/19dde42)), closes [#564](https://github.com/Azure/aztk/issues/564) +* allow cluster config to be printed when no username has been set (#597) ([1cc71c7](https://github.com/Azure/aztk/commit/1cc71c7)), closes [#597](https://github.com/Azure/aztk/issues/597) ## 0.7.1 (2018-05-11) **Bug Fixes** -* Fix: create virtual environment even if container exists ([2db7b00](https://github.com/jafreck/aztk/commit/2db7b00)) -* Fix: gitattributes for jar files (#548) ([a18660b](https://github.com/jafreck/aztk/commit/a18660b)), closes [#548](https://github.com/jafreck/aztk/issues/548) -* Fix: pass docker repo command back to the cluster config (#538) ([a99bbe1](https://github.com/jafreck/aztk/commit/a99bbe1)), closes [#538](https://github.com/jafreck/aztk/issues/538) +* Fix: create virtual environment even if container exists ([2db7b00](https://github.com/Azure/aztk/commit/2db7b00)) +* Fix: gitattributes for jar files (#548) ([a18660b](https://github.com/Azure/aztk/commit/a18660b)), closes [#548](https://github.com/Azure/aztk/issues/548) +* Fix: pass docker repo command back to the cluster config (#538) ([a99bbe1](https://github.com/Azure/aztk/commit/a99bbe1)), closes [#538](https://github.com/Azure/aztk/issues/538) ## 0.7.0 (2018-05-01) diff --git a/README.md b/README.md index a97cbc7e..73fddf8b 100644 --- a/README.md +++ b/README.md @@ -27,7 +27,7 @@ This toolkit is built on top of Azure Batch but does not require any Azure Batch ``` 3. Login or register for an [Azure Account](https://azure.microsoft.com), navigate to [Azure Cloud Shell](https://shell.azure.com), and run: ```sh -wget -q https://raw.githubusercontent.com/Azure/aztk/v0.7.0/account_setup.sh -O account_setup.sh && +wget -q https://raw.githubusercontent.com/Azure/aztk/v0.8.1/account_setup.sh -O account_setup.sh && chmod 755 account_setup.sh && /bin/bash account_setup.sh ``` diff --git a/account_setup.sh b/account_setup.sh index 01cde2c9..6a098b59 100644 --- a/account_setup.sh +++ b/account_setup.sh @@ -4,7 +4,7 @@ echo "Installing dependencies..." && pip install --force-reinstall --upgrade --user pyyaml==3.12 azure==3.0.0 azure-cli-core==2.0.30 msrestazure==0.4.25 > /dev/null 2>&1 && echo "Finished installing dependencies." && echo "Getting account setup script..." && -wget -q https://raw.githubusercontent.com/Azure/aztk/v0.7.0/account_setup.py -O account_setup.py && +wget -q https://raw.githubusercontent.com/Azure/aztk/v0.8.1/account_setup.py -O account_setup.py && chmod 755 account_setup.py && echo "Finished getting account setup script." && echo "Running account setup script..." && diff --git a/aztk_cli/config/job.yaml b/aztk_cli/config/job.yaml index f9900ff6..6a311213 100644 --- a/aztk_cli/config/job.yaml +++ b/aztk_cli/config/job.yaml @@ -1,7 +1,7 @@ # Job Configuration # An Aztk Job is a cluster and an array of Spark applications to run on that cluster # AZTK Spark Jobs will automatically manage the lifecycle of the cluster -# For more information see the documentation at: https://github.com/Azure/aztk/blob/v0.7.0/docs/70-jobs.md +# For more information see the documentation at: http://aztk.readthedocs.io/en/latest/70-jobs.html job: id: diff --git a/aztk_cli/config/secrets.yaml.template b/aztk_cli/config/secrets.yaml.template index a8eaca85..d2f7e2a5 100644 --- a/aztk_cli/config/secrets.yaml.template +++ b/aztk_cli/config/secrets.yaml.template @@ -1,5 +1,5 @@ # For instructions on creating a Batch and Storage account, see -# Getting Started (https://github.com/Azure/aztk/blob/v0.7.0/docs/00-getting-started.md) +# Getting Started (http://aztk.readthedocs.io/en/latest/00-getting-started.html) # NOTE - YAML requires a space after the colon. Ex: "batchaccountname: mybatchaccount" service_principal: diff --git a/docs/01-getting-started-script.md b/docs/01-getting-started-script.md index 023c0603..4bdd8338 100644 --- a/docs/01-getting-started-script.md +++ b/docs/01-getting-started-script.md @@ -14,7 +14,7 @@ The script outputs all of the necessary information to use `aztk`, just copy the ## Usage Copy and paste the following into an [Azure Cloud Shell](https://shell.azure.com): ```sh -wget -q https://raw.githubusercontent.com/Azure/aztk/v0.7.0/account_setup.sh && +wget -q https://raw.githubusercontent.com/Azure/aztk/v0.8.1/account_setup.sh && chmod 755 account_setup.sh && /bin/bash account_setup.sh ``` diff --git a/docs/12-docker-image.md b/docs/12-docker-image.md index 2cadfea1..bad8d5ed 100644 --- a/docs/12-docker-image.md +++ b/docs/12-docker-image.md @@ -17,7 +17,7 @@ aztk spark cluster create ... --docker-repo aztk/base:spark1.6.3 ``` ## Using a custom Docker Image -You can build your own Docker image on top or beneath one of our supported base images _OR_ you can modify the [supported Dockerfiles](https://github.com/Azure/aztk/tree/v0.7.0/docker-image) and build your own image that way. +You can build your own Docker image on top or beneath one of our supported base images _OR_ you can modify the [supported Dockerfiles](https://github.com/Azure/aztk/tree/v0.8.1/docker-image) and build your own image that way. Once you have your Docker image built and hosted publicly, you can then use the **--docker-repo** parameter in your **aztk spark cluster create** command to point to it. @@ -92,4 +92,4 @@ If you are using your own version of Spark, make that it is symlinked by "/home/ ## Hosting your Docker Image By default, aztk assumes that your Docker images are publicly hosted on Docker Hub. However, we also support hosting your images privately. -See [here](https://github.com/Azure/aztk/blob/v0.7.0/docs/12-docker-image.md#using-a-custom-docker-image-that-is-privately-hosted) to learn more about using privately hosted Docker Images. +See [here](12-docker-image.html#using-a-custom-docker-image-that-is-privately-hosted) to learn more about using privately hosted Docker Images. diff --git a/docs/80-migration.md b/docs/80-migration.md index a779fa12..a2e3be06 100644 --- a/docs/80-migration.md +++ b/docs/80-migration.md @@ -33,16 +33,16 @@ A number of changes have been made that affect previously init'ed aztk environme 7. Create your new 0.7.0 cluster! ### cluster.yaml -In cluster.yaml, the `toolkit` key has been added. It is used to select the default, supported Docker images. Please refer to [the configuration file documentation.](https://aztk.readthedocs.io/en/v0.7.0/13-configuration.html#cluster-yaml) +In cluster.yaml, the `toolkit` key has been added. It is used to select the default, supported Docker images. Please refer to [the configuration file documentation.](13-configuration.html#cluster-yaml) ## Docker images A major backwards-incompatible refactor of the Docker images has occurred. Previous Docker images will no longer work with 0.7.0. To update to a new supported docker image, you will need to update your `.aztk/cluster.yaml` configuration file with the `toolkit` block in place of `docker_repo`. If you do not do so, cluster creation will fail! -Please refer to the [the configuration file documentation](https://aztk.readthedocs.io/en/v0.7.0/13-configuration.html#cluster-yaml) for more information on the `toolkit` in `cluster.yaml`. +Please refer to the [the configuration file documentation](13-configuration.html#cluster-yaml) for more information on the `toolkit` in `cluster.yaml`. ## Custom scripts depreciation and Plugins -Custom scripts have been depreciated in favor of Plugins. Plugins have a number of advantages, including the ability to execute on the host (and not in the Spark Docker container). A number of supported plugins are shipped with aztk, please refer to [the plugin documentation to learn more.](https://aztk.readthedocs.io/en/v0.7.0/15-plugins.html) +Custom scripts have been depreciated in favor of Plugins. Plugins have a number of advantages, including the ability to execute on the host (and not in the Spark Docker container). A number of supported plugins are shipped with aztk, please refer to [the plugin documentation to learn more.](15-plugins.html) ## Read the Docs -[Documentation has migrated to readthedocs](https://aztk.readthedocs.io/en/v0.7.0). \ No newline at end of file +[Documentation has migrated to readthedocs](https://aztk.readthedocs.io/en/latest). \ No newline at end of file