HPO PyTorch Tutorial (#4653)

.gitignore

@@ -13,6 +13,9 @@
 /test/ut/retiarii/_debug_graph_data.json
 /test/ut/retiarii/out.tmp
 
+# example generated files
+/nni_assets/**/data/
+
 # Logs
 logs
 *.log

docs/source/hpo/advanced_toctree.rst

@@ -5,7 +5,6 @@ Hyperparameter Optimization
 ..  toctree::
     :maxdepth: 2
 
-    TensorBoard Integration <tensorboard>
     Implement Custom Tuners and Assessors <custom_algorithm>
     Install Custom or 3rd-party Tuners and Assessors <custom_algorithm_installation>
     Tuner Benchmark <hpo_benchmark>

docs/source/hpo/index.rst

@@ -2,6 +2,8 @@
 Hyperparameter Optimization
 ###########################
 
+Auto hyperparameter optimization (HPO), or auto tuning, is one of the key features of NNI.
+
 ..  raw:: html
 
     <script>
@@ -15,7 +17,9 @@ Hyperparameter Optimization
     :maxdepth: 2
 
     Overview <overview>
+    Tutorial </tutorials/hpo_quickstart_pytorch/main>
     Search Space <search_space>
     Tuners <tuners>
     Assessors <assessors>
+    TensorBoard Integration <tensorboard>
     Advanced Usage <advanced_toctree.rst>

docs/source/hpo/overview.rst

@@ -87,10 +87,10 @@ to manually customize hyperparameters, and to manage multiple HPO experiments.
 Tutorials
 ---------
 
-To start using NNI HPO, choose the tutorial of your favorite framework:
+To start using NNI HPO, choose the quickstart tutorial of your favorite framework:
 
-* PyTorch MNIST tutorial
-* :doc:`TensorFlow MNIST tutorial </tutorials/hpo_quickstart_tensorflow/main>`
+* :doc:`PyTorch tutorial </tutorials/hpo_quickstart_pytorch/main>`
+* :doc:`TensorFlow tutorial </tutorials/hpo_quickstart_tensorflow/main>`
 
 Extra Features
 --------------

docs/source/index.rst

@@ -96,7 +96,7 @@ Then, please read :doc:`Quick start <Tutorial/QuickStart>` and :doc:`Tutorials <
 .. codesnippetcard::
    :icon: ../img/thumbnails/hpo-icon-small.png
    :title: Hyper-parameter Tuning
-   :link: tutorials/hpo_quickstart_tensorflow/main
+   :link: tutorials/hpo_quickstart_pytorch/main
 
    .. code-block::
 

docs/source/index_zh.rst

@@ -1,4 +1,4 @@
-.. 737612334d31e3c0ee8db7f53dc2944f
+.. c20f29083a3327d6ecad5f73901e7f99
 
 ###########################
 Neural Network Intelligence

docs/source/tutorials.rst

@@ -24,6 +24,13 @@ Tutorials
    :image: ../img/thumbnails/overview-31.png
    :tags: Experiment/HPO
 
+.. cardlinkitem::
+   :header: HPO Quickstart with PyTorch
+   :description: Use HPO to tune a PyTorch FashionMNIST model
+   :link: tutorials/hpo_quickstart_pytorch/main.html
+   :image: ../img/thumbnails/overview-33.png
+   :tags: HPO
+
 .. cardlinkitem::
    :header: HPO Quickstart with TensorFlow
    :description: Use HPO to tune a TensorFlow MNIST model

docs/source/tutorials/hpo_quickstart_pytorch/main.ipynb

@@ -0,0 +1,176 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "%matplotlib inline"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "\n# NNI HPO Quickstart with PyTorch\nThis tutorial optimizes the model in `official PyTorch quickstart`_ with auto-tuning.\n\nThe tutorial consists of 4 steps: \n\n 1. Modify the model for auto-tuning.\n 2. Define hyperparameters' search space.\n 3. Configure the experiment.\n 4. Run the experiment.\n\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 1: Prepare the model\nIn first step, you need to prepare the model to be tuned.\n\nThe model should be put in a separate script.\nIt will be evaluated many times concurrently,\nand possibly will be trained on distributed platforms.\n\nIn this tutorial, the model is defined in :doc:`model.py <model>`.\n\nPlease understand the model code before continue to next step.\n\n"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 2: Define search space\nIn model code, we have prepared 3 hyperparameters to be tuned:\n*features*, *lr*, and *momentum*.\n\nHere we need to define their *search space* so the tuning algorithm can sample them in desired range.\n\nAssuming we have following prior knowledge for these hyperparameters:\n\n 1. *features* should be one of 128, 256, 512, 1024.\n 2. *lr* should be a float between 0.0001 and 0.1, and it follows exponential distribution.\n 3. *momentum* should be a float between 0 and 1.\n\nIn NNI, the space of *features* is called ``choice``;\nthe space of *lr* is called ``loguniform``;\nand the space of *momentum* is called ``uniform``.\nYou may have noticed, these names are derived from ``numpy.random``.\n\nFor full specification of search space, check :doc:`the reference </hpo/search_space>`.\n\nNow we can define the search space as follow:\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "search_space = {\n    'features': {'_type': 'choice', '_value': [128, 256, 512, 1024]},\n    'lr': {'_type': 'loguniform', '_value': [0.0001, 0.1]},\n    'momentum': {'_type': 'uniform', '_value': [0, 1]},\n}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 3: Configure the experiment\nNNI uses an *experiment* to manage the HPO process.\nThe *experiment config* defines how to train the models and how to explore the search space.\n\nIn this tutorial we use a *local* mode experiment,\nwhich means models will be trained on local machine, without using any special training platform.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "from nni.experiment import Experiment\nexperiment = Experiment('local')"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Now we start to configure the experiment.\n\nFirstly, specify the model code.\nIn NNI evaluation of each hyperparameter set is called a *trial*.\nSo the model script is called *trial code*.\n\nIf you are using Linux system without Conda, you many need to change ``python`` to ``python3``.\n\nWhen ``trial_code_directory`` is a relative path, it relates to current working directory.\nTo run ``main.py`` from a different path, you can set trial code directory to ``Path(__file__).parent``.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "experiment.config.trial_command = 'python model.py'\nexperiment.config.trial_code_directory = '.'"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Then specify the search space we defined above:\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "experiment.config.search_space = search_space"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Choose a tuning algorithm.\nHere we use :doc:`TPE tuner </hpo/tuners>`.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "experiment.config.tuner.name = 'TPE'\nexperiment.config.tuner.class_args['optimize_mode'] = 'maximize'"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "Specify how many trials to run.\nHere we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 4 sets at a time.\n\nPlease note that ``max_trial_number`` here is merely for a quick example.\nWith default config TPE tuner requires 20 trials to warm up.\nIn real world max trial number is commonly set to 100+.\n\nYou can also set ``max_experiment_duration = '1h'`` to limit running time.\n\nAnd alternatively, you can skip this part and set no limit at all.\nThe experiment will run forever until you press Ctrl-C.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "experiment.config.max_trial_number = 10\nexperiment.config.trial_concurrency = 4"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Step 4: Run the experiment\nNow the experiment is ready. Choose a port and launch it.\n\nYou can use the web portal to view experiment status: http://localhost:8080.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "experiment.run(8080)"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.2"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

docs/source/tutorials/hpo_quickstart_pytorch/main.py

@@ -0,0 +1,114 @@
+"""
+NNI HPO Quickstart with PyTorch
+===============================
+This tutorial optimizes the model in `official PyTorch quickstart`_ with auto-tuning.
+
+The tutorial consists of 4 steps: 
+
+ 1. Modify the model for auto-tuning.
+ 2. Define hyperparameters' search space.
+ 3. Configure the experiment.
+ 4. Run the experiment.
+
+.. _official PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html
+"""
+
+# %%
+# Step 1: Prepare the model
+# -------------------------
+# In first step, you need to prepare the model to be tuned.
+#
+# The model should be put in a separate script.
+# It will be evaluated many times concurrently,
+# and possibly will be trained on distributed platforms.
+#
+# In this tutorial, the model is defined in :doc:`model.py <model>`.
+#
+# Please understand the model code before continue to next step.
+
+# %%
+# Step 2: Define search space
+# ---------------------------
+# In model code, we have prepared 3 hyperparameters to be tuned:
+# *features*, *lr*, and *momentum*.
+#
+# Here we need to define their *search space* so the tuning algorithm can sample them in desired range.
+#
+# Assuming we have following prior knowledge for these hyperparameters:
+#
+#  1. *features* should be one of 128, 256, 512, 1024.
+#  2. *lr* should be a float between 0.0001 and 0.1, and it follows exponential distribution.
+#  3. *momentum* should be a float between 0 and 1.
+#
+# In NNI, the space of *features* is called ``choice``;
+# the space of *lr* is called ``loguniform``;
+# and the space of *momentum* is called ``uniform``.
+# You may have noticed, these names are derived from ``numpy.random``.
+#
+# For full specification of search space, check :doc:`the reference </hpo/search_space>`.
+#
+# Now we can define the search space as follow:
+
+search_space = {
+    'features': {'_type': 'choice', '_value': [128, 256, 512, 1024]},
+    'lr': {'_type': 'loguniform', '_value': [0.0001, 0.1]},
+    'momentum': {'_type': 'uniform', '_value': [0, 1]},
+}
+
+# %%
+# Step 3: Configure the experiment
+# --------------------------------
+# NNI uses an *experiment* to manage the HPO process.
+# The *experiment config* defines how to train the models and how to explore the search space.
+# 
+# In this tutorial we use a *local* mode experiment,
+# which means models will be trained on local machine, without using any special training platform.
+from nni.experiment import Experiment
+experiment = Experiment('local')
+
+# %%
+# Now we start to configure the experiment.
+#
+# Firstly, specify the model code.
+# In NNI evaluation of each hyperparameter set is called a *trial*.
+# So the model script is called *trial code*.
+#
+# If you are using Linux system without Conda, you many need to change ``python`` to ``python3``.
+#
+# When ``trial_code_directory`` is a relative path, it relates to current working directory.
+# To run ``main.py`` from a different path, you can set trial code directory to ``Path(__file__).parent``.
+experiment.config.trial_command = 'python model.py'
+experiment.config.trial_code_directory = '.'
+
+# %%
+# Then specify the search space we defined above:
+experiment.config.search_space = search_space
+
+# %%
+# Choose a tuning algorithm.
+# Here we use :doc:`TPE tuner </hpo/tuners>`.
+experiment.config.tuner.name = 'TPE'
+experiment.config.tuner.class_args['optimize_mode'] = 'maximize'
+
+# %%
+# Specify how many trials to run.
+# Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 4 sets at a time.
+#
+# Please note that ``max_trial_number`` here is merely for a quick example.
+# With default config TPE tuner requires 20 trials to warm up.
+# In real world max trial number is commonly set to 100+.
+#
+# You can also set ``max_experiment_duration = '1h'`` to limit running time.
+#
+# And alternatively, you can skip this part and set no limit at all.
+# The experiment will run forever until you press Ctrl-C.
+experiment.config.max_trial_number = 10
+experiment.config.trial_concurrency = 4
+
+# %%
+# Step 4: Run the experiment
+# --------------------------
+# Now the experiment is ready. Choose a port and launch it.
+#
+# You can use the web portal to view experiment status: http://localhost:8080.
+experiment.run(8080)

docs/source/tutorials/hpo_quickstart_pytorch/main.py.md5

@@ -0,0 +1 @@
+264c7f7dffbb756f8a6aa675d4e792e4

docs/source/tutorials/hpo_quickstart_pytorch/main.rst

@@ -0,0 +1,271 @@
+:orphan:
+
+.. DO NOT EDIT.
+.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
+.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
+.. "tutorials/hpo_quickstart_pytorch/main.py"
+.. LINE NUMBERS ARE GIVEN BELOW.
+
+.. only:: html
+
+    .. note::
+        :class: sphx-glr-download-link-note
+
+        Click :ref:`here <sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py>`
+        to download the full example code
+
+.. rst-class:: sphx-glr-example-title
+
+.. _sphx_glr_tutorials_hpo_quickstart_pytorch_main.py:
+
+
+NNI HPO Quickstart with PyTorch
+===============================
+This tutorial optimizes the model in `official PyTorch quickstart`_ with auto-tuning.
+
+The tutorial consists of 4 steps: 
+
+ 1. Modify the model for auto-tuning.
+ 2. Define hyperparameters' search space.
+ 3. Configure the experiment.
+ 4. Run the experiment.
+
+.. _official PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html
+
+.. GENERATED FROM PYTHON SOURCE LINES 17-28
+
+Step 1: Prepare the model
+-------------------------
+In first step, you need to prepare the model to be tuned.
+
+The model should be put in a separate script.
+It will be evaluated many times concurrently,
+and possibly will be trained on distributed platforms.
+
+In this tutorial, the model is defined in :doc:`model.py <model>`.
+
+Please understand the model code before continue to next step.
+
+.. GENERATED FROM PYTHON SOURCE LINES 30-51
+
+Step 2: Define search space
+---------------------------
+In model code, we have prepared 3 hyperparameters to be tuned:
+*features*, *lr*, and *momentum*.
+
+Here we need to define their *search space* so the tuning algorithm can sample them in desired range.
+
+Assuming we have following prior knowledge for these hyperparameters:
+
+ 1. *features* should be one of 128, 256, 512, 1024.
+ 2. *lr* should be a float between 0.0001 and 0.1, and it follows exponential distribution.
+ 3. *momentum* should be a float between 0 and 1.
+
+In NNI, the space of *features* is called ``choice``;
+the space of *lr* is called ``loguniform``;
+and the space of *momentum* is called ``uniform``.
+You may have noticed, these names are derived from ``numpy.random``.
+
+For full specification of search space, check :doc:`the reference </hpo/search_space>`.
+
+Now we can define the search space as follow:
+
+.. GENERATED FROM PYTHON SOURCE LINES 51-58
+
+.. code-block:: default
+
+
+    search_space = {
+        'features': {'_type': 'choice', '_value': [128, 256, 512, 1024]},
+        'lr': {'_type': 'loguniform', '_value': [0.0001, 0.1]},
+        'momentum': {'_type': 'uniform', '_value': [0, 1]},
+    }
+
+
+
+
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 59-66
+
+Step 3: Configure the experiment
+--------------------------------
+NNI uses an *experiment* to manage the HPO process.
+The *experiment config* defines how to train the models and how to explore the search space.
+
+In this tutorial we use a *local* mode experiment,
+which means models will be trained on local machine, without using any special training platform.
+
+.. GENERATED FROM PYTHON SOURCE LINES 66-69
+
+.. code-block:: default
+
+    from nni.experiment import Experiment
+    experiment = Experiment('local')
+
+
+
+
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 70-80
+
+Now we start to configure the experiment.
+
+Firstly, specify the model code.
+In NNI evaluation of each hyperparameter set is called a *trial*.
+So the model script is called *trial code*.
+
+If you are using Linux system without Conda, you many need to change ``python`` to ``python3``.
+
+When ``trial_code_directory`` is a relative path, it relates to current working directory.
+To run ``main.py`` from a different path, you can set trial code directory to ``Path(__file__).parent``.
+
+.. GENERATED FROM PYTHON SOURCE LINES 80-83
+
+.. code-block:: default
+
+    experiment.config.trial_command = 'python model.py'
+    experiment.config.trial_code_directory = '.'
+
+
+
+
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 84-85
+
+Then specify the search space we defined above:
+
+.. GENERATED FROM PYTHON SOURCE LINES 85-87
+
+.. code-block:: default
+
+    experiment.config.search_space = search_space
+
+
+
+
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 88-90
+
+Choose a tuning algorithm.
+Here we use :doc:`TPE tuner </hpo/tuners>`.
+
+.. GENERATED FROM PYTHON SOURCE LINES 90-93
+
+.. code-block:: default
+
+    experiment.config.tuner.name = 'TPE'
+    experiment.config.tuner.class_args['optimize_mode'] = 'maximize'
+
+
+
+
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 94-105
+
+Specify how many trials to run.
+Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 4 sets at a time.
+
+Please note that ``max_trial_number`` here is merely for a quick example.
+With default config TPE tuner requires 20 trials to warm up.
+In real world max trial number is commonly set to 100+.
+
+You can also set ``max_experiment_duration = '1h'`` to limit running time.
+
+And alternatively, you can skip this part and set no limit at all.
+The experiment will run forever until you press Ctrl-C.
+
+.. GENERATED FROM PYTHON SOURCE LINES 105-108
+
+.. code-block:: default
+
+    experiment.config.max_trial_number = 10
+    experiment.config.trial_concurrency = 4
+
+
+
+
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 109-114
+
+Step 4: Run the experiment
+--------------------------
+Now the experiment is ready. Choose a port and launch it.
+
+You can use the web portal to view experiment status: http://localhost:8080.
+
+.. GENERATED FROM PYTHON SOURCE LINES 114-115
+
+.. code-block:: default
+
+    experiment.run(8080)
+
+
+
+
+.. rst-class:: sphx-glr-script-out
+
+ Out:
+
+ .. code-block:: none
+
+    [2022-03-18 12:58:19] Creating experiment, Experiment ID: 2ijcdvau
+    [2022-03-18 12:58:19] Starting web server...
+    [2022-03-18 12:58:21] Setting up...
+    [2022-03-18 12:58:21] Web UI URLs: http://127.0.0.1:8080 http://172.22.104.196:8080
+    [2022-03-18 13:57:44] Stopping experiment, please wait...
+    [2022-03-18 13:57:47] Experiment stopped
+
+    True
+
+
+
+
+.. rst-class:: sphx-glr-timing
+
+   **Total running time of the script:** ( 59 minutes  28.055 seconds)
+
+
+.. _sphx_glr_download_tutorials_hpo_quickstart_pytorch_main.py:
+
+
+.. only :: html
+
+ .. container:: sphx-glr-footer
+    :class: sphx-glr-footer-example
+
+
+
+  .. container:: sphx-glr-download sphx-glr-download-python
+
+     :download:`Download Python source code: main.py <main.py>`
+
+
+
+  .. container:: sphx-glr-download sphx-glr-download-jupyter
+
+     :download:`Download Jupyter notebook: main.ipynb <main.ipynb>`
+
+
+.. only:: html
+
+ .. rst-class:: sphx-glr-signature
+
+    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_

docs/source/tutorials/hpo_quickstart_pytorch/model.ipynb

@@ -0,0 +1,162 @@
+{
+  "cells": [
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "%matplotlib inline"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "\n# Port PyTorch Quickstart to NNI\nThis is a modified version of `PyTorch quickstart`_.\n\nIt can be run directly and will have the exact same result as original version.\n\nFurthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later.\n\nFor now, we recommend to run this script directly to verify the environment.\n\nThere are only 2 key differences from the original version:\n\n1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters.\n2. In `Train the model and report accuracy`_ part, it reports accuracy metrics for tuner to generate next hyperparameter set.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "import nni\nimport torch\nfrom torch import nn\nfrom torch.utils.data import DataLoader\nfrom torchvision import datasets\nfrom torchvision.transforms import ToTensor"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Hyperparameters to be tuned\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "params = {\n    'features': 512,\n    'lr': 0.001,\n    'momentum': 0,\n}"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Get optimized hyperparameters\nIf run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict.\nBut with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "optimized_params = nni.get_next_parameter()\nparams.update(optimized_params)\nprint(params)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Load dataset\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "training_data = datasets.FashionMNIST(root=\"data\", train=True, download=True, transform=ToTensor())\ntest_data = datasets.FashionMNIST(root=\"data\", train=False, download=True, transform=ToTensor())\n\nbatch_size = 64\n\ntrain_dataloader = DataLoader(training_data, batch_size=batch_size)\ntest_dataloader = DataLoader(test_data, batch_size=batch_size)"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Build model with hyperparameters\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "device = \"cuda\" if torch.cuda.is_available() else \"cpu\"\nprint(f\"Using {device} device\")\n\nclass NeuralNetwork(nn.Module):\n    def __init__(self):\n        super(NeuralNetwork, self).__init__()\n        self.flatten = nn.Flatten()\n        self.linear_relu_stack = nn.Sequential(\n            nn.Linear(28*28, params['features']),\n            nn.ReLU(),\n            nn.Linear(params['features'], params['features']),\n            nn.ReLU(),\n            nn.Linear(params['features'], 10)\n        )\n\n    def forward(self, x):\n        x = self.flatten(x)\n        logits = self.linear_relu_stack(x)\n        return logits\n\nmodel = NeuralNetwork().to(device)\n\nloss_fn = nn.CrossEntropyLoss()\noptimizer = torch.optim.SGD(model.parameters(), lr=params['lr'], momentum=params['momentum'])"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Define train() and test()\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "def train(dataloader, model, loss_fn, optimizer):\n    size = len(dataloader.dataset)\n    model.train()\n    for batch, (X, y) in enumerate(dataloader):\n        X, y = X.to(device), y.to(device)\n        pred = model(X)\n        loss = loss_fn(pred, y)\n        optimizer.zero_grad()\n        loss.backward()\n        optimizer.step()\n\ndef test(dataloader, model, loss_fn):\n    size = len(dataloader.dataset)\n    num_batches = len(dataloader)\n    model.eval()\n    test_loss, correct = 0, 0\n    with torch.no_grad():\n        for X, y in dataloader:\n            X, y = X.to(device), y.to(device)\n            pred = model(X)\n            test_loss += loss_fn(pred, y).item()\n            correct += (pred.argmax(1) == y).type(torch.float).sum().item()\n    test_loss /= num_batches\n    correct /= size\n    return correct"
+      ]
+    },
+    {
+      "cell_type": "markdown",
+      "metadata": {},
+      "source": [
+        "## Train the model and report accuracy\nReport accuracy to NNI so the tuning algorithm can predict best hyperparameters.\n\n"
+      ]
+    },
+    {
+      "cell_type": "code",
+      "execution_count": null,
+      "metadata": {
+        "collapsed": false
+      },
+      "outputs": [],
+      "source": [
+        "epochs = 5\nfor t in range(epochs):\n    print(f\"Epoch {t+1}\\n-------------------------------\")\n    train(train_dataloader, model, loss_fn, optimizer)\n    accuracy = test(test_dataloader, model, loss_fn)\n    nni.report_intermediate_result(accuracy)\nnni.report_final_result(accuracy)"
+      ]
+    }
+  ],
+  "metadata": {
+    "kernelspec": {
+      "display_name": "Python 3",
+      "language": "python",
+      "name": "python3"
+    },
+    "language_info": {
+      "codemirror_mode": {
+        "name": "ipython",
+        "version": 3
+      },
+      "file_extension": ".py",
+      "mimetype": "text/x-python",
+      "name": "python",
+      "nbconvert_exporter": "python",
+      "pygments_lexer": "ipython3",
+      "version": "3.10.2"
+    }
+  },
+  "nbformat": 4,
+  "nbformat_minor": 0
+}

docs/source/tutorials/hpo_quickstart_pytorch/model.py

@@ -0,0 +1,124 @@
+"""
+Port PyTorch Quickstart to NNI
+==============================
+This is a modified version of `PyTorch quickstart`_.
+
+It can be run directly and will have the exact same result as original version.
+
+Furthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later.
+
+For now, we recommend to run this script directly to verify the environment.
+
+There are only 2 key differences from the original version:
+
+1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters.
+2. In `Train the model and report accuracy`_ part, it reports accuracy metrics for tuner to generate next hyperparameter set.
+
+.. _PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html
+"""
+
+# %%
+import nni
+import torch
+from torch import nn
+from torch.utils.data import DataLoader
+from torchvision import datasets
+from torchvision.transforms import ToTensor
+
+# %%
+# Hyperparameters to be tuned
+# ---------------------------
+params = {
+    'features': 512,
+    'lr': 0.001,
+    'momentum': 0,
+}
+
+# %%
+# Get optimized hyperparameters
+# -----------------------------
+# If run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict.
+# But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.
+optimized_params = nni.get_next_parameter()
+params.update(optimized_params)
+print(params)
+
+# %%
+# Load dataset
+# ------------
+training_data = datasets.FashionMNIST(root="data", train=True, download=True, transform=ToTensor())
+test_data = datasets.FashionMNIST(root="data", train=False, download=True, transform=ToTensor())
+
+batch_size = 64
+
+train_dataloader = DataLoader(training_data, batch_size=batch_size)
+test_dataloader = DataLoader(test_data, batch_size=batch_size)
+
+# %%
+# Build model with hyperparameters
+# --------------------------------
+device = "cuda" if torch.cuda.is_available() else "cpu"
+print(f"Using {device} device")
+
+class NeuralNetwork(nn.Module):
+    def __init__(self):
+        super(NeuralNetwork, self).__init__()
+        self.flatten = nn.Flatten()
+        self.linear_relu_stack = nn.Sequential(
+            nn.Linear(28*28, params['features']),
+            nn.ReLU(),
+            nn.Linear(params['features'], params['features']),
+            nn.ReLU(),
+            nn.Linear(params['features'], 10)
+        )
+
+    def forward(self, x):
+        x = self.flatten(x)
+        logits = self.linear_relu_stack(x)
+        return logits
+
+model = NeuralNetwork().to(device)
+
+loss_fn = nn.CrossEntropyLoss()
+optimizer = torch.optim.SGD(model.parameters(), lr=params['lr'], momentum=params['momentum'])
+
+# %%
+# Define train() and test()
+# -------------------------
+def train(dataloader, model, loss_fn, optimizer):
+    size = len(dataloader.dataset)
+    model.train()
+    for batch, (X, y) in enumerate(dataloader):
+        X, y = X.to(device), y.to(device)
+        pred = model(X)
+        loss = loss_fn(pred, y)
+        optimizer.zero_grad()
+        loss.backward()
+        optimizer.step()
+
+def test(dataloader, model, loss_fn):
+    size = len(dataloader.dataset)
+    num_batches = len(dataloader)
+    model.eval()
+    test_loss, correct = 0, 0
+    with torch.no_grad():
+        for X, y in dataloader:
+            X, y = X.to(device), y.to(device)
+            pred = model(X)
+            test_loss += loss_fn(pred, y).item()
+            correct += (pred.argmax(1) == y).type(torch.float).sum().item()
+    test_loss /= num_batches
+    correct /= size
+    return correct
+
+# %%
+# Train the model and report accuracy
+# -----------------------------------
+# Report accuracy to NNI so the tuning algorithm can predict best hyperparameters.
+epochs = 5
+for t in range(epochs):
+    print(f"Epoch {t+1}\n-------------------------------")
+    train(train_dataloader, model, loss_fn, optimizer)
+    accuracy = test(test_dataloader, model, loss_fn)
+    nni.report_intermediate_result(accuracy)
+nni.report_final_result(accuracy)

docs/source/tutorials/hpo_quickstart_pytorch/model.py.md5

@@ -0,0 +1 @@
+8250278967847abcd7b16ec4e811e825

docs/source/tutorials/hpo_quickstart_pytorch/model.rst

@@ -0,0 +1,304 @@
+:orphan:
+
+.. DO NOT EDIT.
+.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
+.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
+.. "tutorials/hpo_quickstart_pytorch/model.py"
+.. LINE NUMBERS ARE GIVEN BELOW.
+
+.. only:: html
+
+    .. note::
+        :class: sphx-glr-download-link-note
+
+        Click :ref:`here <sphx_glr_download_tutorials_hpo_quickstart_pytorch_model.py>`
+        to download the full example code
+
+.. rst-class:: sphx-glr-example-title
+
+.. _sphx_glr_tutorials_hpo_quickstart_pytorch_model.py:
+
+
+Port PyTorch Quickstart to NNI
+==============================
+This is a modified version of `PyTorch quickstart`_.
+
+It can be run directly and will have the exact same result as original version.
+
+Furthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later.
+
+For now, we recommend to run this script directly to verify the environment.
+
+There are only 2 key differences from the original version:
+
+1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters.
+2. In `Train the model and report accuracy`_ part, it reports accuracy metrics for tuner to generate next hyperparameter set.
+
+.. _PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html
+
+.. GENERATED FROM PYTHON SOURCE LINES 21-28
+
+.. code-block:: default
+
+    import nni
+    import torch
+    from torch import nn
+    from torch.utils.data import DataLoader
+    from torchvision import datasets
+    from torchvision.transforms import ToTensor
+
+
+
+
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 29-31
+
+Hyperparameters to be tuned
+---------------------------
+
+.. GENERATED FROM PYTHON SOURCE LINES 31-37
+
+.. code-block:: default
+
+    params = {
+        'features': 512,
+        'lr': 0.001,
+        'momentum': 0,
+    }
+
+
+
+
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 38-42
+
+Get optimized hyperparameters
+-----------------------------
+If run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict.
+But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.
+
+.. GENERATED FROM PYTHON SOURCE LINES 42-46
+
+.. code-block:: default
+
+    optimized_params = nni.get_next_parameter()
+    params.update(optimized_params)
+    print(params)
+
+
+
+
+
+.. rst-class:: sphx-glr-script-out
+
+ Out:
+
+ .. code-block:: none
+
+    /home/lz/nnisrc/nni/runtime/platform/standalone.py:32: RuntimeWarning: Running NNI code without runtime. Check the following tutorial if you are new to NNI: https://nni.readthedocs.io/en/stable/Tutorial/QuickStart.html#id1
+      warnings.warn(warning_message, RuntimeWarning)
+    {'features': 512, 'lr': 0.001, 'momentum': 0}
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 47-49
+
+Load dataset
+------------
+
+.. GENERATED FROM PYTHON SOURCE LINES 49-57
+
+.. code-block:: default
+
+    training_data = datasets.FashionMNIST(root="data", train=True, download=True, transform=ToTensor())
+    test_data = datasets.FashionMNIST(root="data", train=False, download=True, transform=ToTensor())
+
+    batch_size = 64
+
+    train_dataloader = DataLoader(training_data, batch_size=batch_size)
+    test_dataloader = DataLoader(test_data, batch_size=batch_size)
+
+
+
+
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 58-60
+
+Build model with hyperparameters
+--------------------------------
+
+.. GENERATED FROM PYTHON SOURCE LINES 60-85
+
+.. code-block:: default
+
+    device = "cuda" if torch.cuda.is_available() else "cpu"
+    print(f"Using {device} device")
+
+    class NeuralNetwork(nn.Module):
+        def __init__(self):
+            super(NeuralNetwork, self).__init__()
+            self.flatten = nn.Flatten()
+            self.linear_relu_stack = nn.Sequential(
+                nn.Linear(28*28, params['features']),
+                nn.ReLU(),
+                nn.Linear(params['features'], params['features']),
+                nn.ReLU(),
+                nn.Linear(params['features'], 10)
+            )
+
+        def forward(self, x):
+            x = self.flatten(x)
+            logits = self.linear_relu_stack(x)
+            return logits
+
+    model = NeuralNetwork().to(device)
+
+    loss_fn = nn.CrossEntropyLoss()
+    optimizer = torch.optim.SGD(model.parameters(), lr=params['lr'], momentum=params['momentum'])
+
+
+
+
+
+.. rst-class:: sphx-glr-script-out
+
+ Out:
+
+ .. code-block:: none
+
+    Using cpu device
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 86-88
+
+Define train() and test()
+-------------------------
+
+.. GENERATED FROM PYTHON SOURCE LINES 88-114
+
+.. code-block:: default
+
+    def train(dataloader, model, loss_fn, optimizer):
+        size = len(dataloader.dataset)
+        model.train()
+        for batch, (X, y) in enumerate(dataloader):
+            X, y = X.to(device), y.to(device)
+            pred = model(X)
+            loss = loss_fn(pred, y)
+            optimizer.zero_grad()
+            loss.backward()
+            optimizer.step()
+
+    def test(dataloader, model, loss_fn):
+        size = len(dataloader.dataset)
+        num_batches = len(dataloader)
+        model.eval()
+        test_loss, correct = 0, 0
+        with torch.no_grad():
+            for X, y in dataloader:
+                X, y = X.to(device), y.to(device)
+                pred = model(X)
+                test_loss += loss_fn(pred, y).item()
+                correct += (pred.argmax(1) == y).type(torch.float).sum().item()
+        test_loss /= num_batches
+        correct /= size
+        return correct
+
+
+
+
+
+
+
+
+.. GENERATED FROM PYTHON SOURCE LINES 115-118
+
+Train the model and report accuracy
+-----------------------------------
+Report accuracy to NNI so the tuning algorithm can predict best hyperparameters.
+
+.. GENERATED FROM PYTHON SOURCE LINES 118-125
+
+.. code-block:: default
+
+    epochs = 5
+    for t in range(epochs):
+        print(f"Epoch {t+1}\n-------------------------------")
+        train(train_dataloader, model, loss_fn, optimizer)
+        accuracy = test(test_dataloader, model, loss_fn)
+        nni.report_intermediate_result(accuracy)
+    nni.report_final_result(accuracy)
+
+
+
+
+.. rst-class:: sphx-glr-script-out
+
+ Out:
+
+ .. code-block:: none
+
+    Epoch 1
+    -------------------------------
+    [2022-03-18 14:02:46] INFO (nni/MainThread) Intermediate result: 0.5418  (Index 0)
+    Epoch 2
+    -------------------------------
+    [2022-03-18 14:02:52] INFO (nni/MainThread) Intermediate result: 0.5945  (Index 1)
+    Epoch 3
+    -------------------------------
+    [2022-03-18 14:02:58] INFO (nni/MainThread) Intermediate result: 0.6202  (Index 2)
+    Epoch 4
+    -------------------------------
+    [2022-03-18 14:03:04] INFO (nni/MainThread) Intermediate result: 0.6376  (Index 3)
+    Epoch 5
+    -------------------------------
+    [2022-03-18 14:03:09] INFO (nni/MainThread) Intermediate result: 0.652  (Index 4)
+    [2022-03-18 14:03:09] INFO (nni/MainThread) Final result: 0.652
+
+
+
+
+
+.. rst-class:: sphx-glr-timing
+
+   **Total running time of the script:** ( 0 minutes  29.026 seconds)
+
+
+.. _sphx_glr_download_tutorials_hpo_quickstart_pytorch_model.py:
+
+
+.. only :: html
+
+ .. container:: sphx-glr-footer
+    :class: sphx-glr-footer-example
+
+
+
+  .. container:: sphx-glr-download sphx-glr-download-python
+
+     :download:`Download Python source code: model.py <model.py>`
+
+
+
+  .. container:: sphx-glr-download sphx-glr-download-jupyter
+
+     :download:`Download Jupyter notebook: model.ipynb <model.ipynb>`
+
+
+.. only:: html
+
+ .. rst-class:: sphx-glr-signature
+
+    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_

docs/source/tutorials/hpo_quickstart_pytorch/sg_execution_times.rst

@@ -0,0 +1,14 @@
+
+:orphan:
+
+.. _sphx_glr_tutorials_hpo_quickstart_pytorch_sg_execution_times:
+
+Computation times
+=================
+**00:29.026** total execution time for **tutorials_hpo_quickstart_pytorch** files:
+
++--------------------------------------------------------------------------+-----------+--------+
+| :ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_model.py` (``model.py``) | 00:29.026 | 0.0 MB |
++--------------------------------------------------------------------------+-----------+--------+
+| :ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_main.py` (``main.py``)   | 00:00.000 | 0.0 MB |
++--------------------------------------------------------------------------+-----------+--------+

docs/source/tutorials/index.rst

@@ -203,6 +203,58 @@ Tutorials
 
 
 
+.. _sphx_glr_tutorials_hpo_quickstart_pytorch:
+
+
+
+
+.. raw:: html
+
+    <div class="sphx-glr-thumbcontainer" tooltip="The tutorial consists of 4 steps: ">
+
+.. only:: html
+
+ .. figure:: /tutorials/hpo_quickstart_pytorch/images/thumb/sphx_glr_main_thumb.png
+     :alt: NNI HPO Quickstart with PyTorch
+
+     :ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_main.py`
+
+.. raw:: html
+
+    </div>
+
+
+.. toctree::
+   :hidden:
+
+   /tutorials/hpo_quickstart_pytorch/main
+
+.. raw:: html
+
+    <div class="sphx-glr-thumbcontainer" tooltip="It can be run directly and will have the exact same result as original version.">
+
+.. only:: html
+
+ .. figure:: /tutorials/hpo_quickstart_pytorch/images/thumb/sphx_glr_model_thumb.png
+     :alt: Port PyTorch Quickstart to NNI
+
+     :ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_model.py`
+
+.. raw:: html
+
+    </div>
+
+
+.. toctree::
+   :hidden:
+
+   /tutorials/hpo_quickstart_pytorch/model
+.. raw:: html
+
+    <div class="sphx-glr-clear"></div>
+
+
+
 .. _sphx_glr_tutorials_hpo_quickstart_tensorflow:
 
 

examples/tutorials/hpo_quickstart_pytorch/main.py

@@ -0,0 +1,114 @@
+"""
+NNI HPO Quickstart with PyTorch
+===============================
+This tutorial optimizes the model in `official PyTorch quickstart`_ with auto-tuning.
+
+The tutorial consists of 4 steps: 
+
+ 1. Modify the model for auto-tuning.
+ 2. Define hyperparameters' search space.
+ 3. Configure the experiment.
+ 4. Run the experiment.
+
+.. _official PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html
+"""
+
+# %%
+# Step 1: Prepare the model
+# -------------------------
+# In first step, you need to prepare the model to be tuned.
+#
+# The model should be put in a separate script.
+# It will be evaluated many times concurrently,
+# and possibly will be trained on distributed platforms.
+#
+# In this tutorial, the model is defined in :doc:`model.py <model>`.
+#
+# Please understand the model code before continue to next step.
+
+# %%
+# Step 2: Define search space
+# ---------------------------
+# In model code, we have prepared 3 hyperparameters to be tuned:
+# *features*, *lr*, and *momentum*.
+#
+# Here we need to define their *search space* so the tuning algorithm can sample them in desired range.
+#
+# Assuming we have following prior knowledge for these hyperparameters:
+#
+#  1. *features* should be one of 128, 256, 512, 1024.
+#  2. *lr* should be a float between 0.0001 and 0.1, and it follows exponential distribution.
+#  3. *momentum* should be a float between 0 and 1.
+#
+# In NNI, the space of *features* is called ``choice``;
+# the space of *lr* is called ``loguniform``;
+# and the space of *momentum* is called ``uniform``.
+# You may have noticed, these names are derived from ``numpy.random``.
+#
+# For full specification of search space, check :doc:`the reference </hpo/search_space>`.
+#
+# Now we can define the search space as follow:
+
+search_space = {
+    'features': {'_type': 'choice', '_value': [128, 256, 512, 1024]},
+    'lr': {'_type': 'loguniform', '_value': [0.0001, 0.1]},
+    'momentum': {'_type': 'uniform', '_value': [0, 1]},
+}
+
+# %%
+# Step 3: Configure the experiment
+# --------------------------------
+# NNI uses an *experiment* to manage the HPO process.
+# The *experiment config* defines how to train the models and how to explore the search space.
+# 
+# In this tutorial we use a *local* mode experiment,
+# which means models will be trained on local machine, without using any special training platform.
+from nni.experiment import Experiment
+experiment = Experiment('local')
+
+# %%
+# Now we start to configure the experiment.
+#
+# Firstly, specify the model code.
+# In NNI evaluation of each hyperparameter set is called a *trial*.
+# So the model script is called *trial code*.
+#
+# If you are using Linux system without Conda, you many need to change ``python`` to ``python3``.
+#
+# When ``trial_code_directory`` is a relative path, it relates to current working directory.
+# To run ``main.py`` from a different path, you can set trial code directory to ``Path(__file__).parent``.
+experiment.config.trial_command = 'python model.py'
+experiment.config.trial_code_directory = '.'
+
+# %%
+# Then specify the search space we defined above:
+experiment.config.search_space = search_space
+
+# %%
+# Choose a tuning algorithm.
+# Here we use :doc:`TPE tuner </hpo/tuners>`.
+experiment.config.tuner.name = 'TPE'
+experiment.config.tuner.class_args['optimize_mode'] = 'maximize'
+
+# %%
+# Specify how many trials to run.
+# Here we evaluate 10 sets of hyperparameters in total, and concurrently evaluate 4 sets at a time.
+#
+# Please note that ``max_trial_number`` here is merely for a quick example.
+# With default config TPE tuner requires 20 trials to warm up.
+# In real world max trial number is commonly set to 100+.
+#
+# You can also set ``max_experiment_duration = '1h'`` to limit running time.
+#
+# And alternatively, you can skip this part and set no limit at all.
+# The experiment will run forever until you press Ctrl-C.
+experiment.config.max_trial_number = 10
+experiment.config.trial_concurrency = 4
+
+# %%
+# Step 4: Run the experiment
+# --------------------------
+# Now the experiment is ready. Choose a port and launch it.
+#
+# You can use the web portal to view experiment status: http://localhost:8080.
+experiment.run(8080)

examples/tutorials/hpo_quickstart_pytorch/model.py

@@ -0,0 +1,124 @@
+"""
+Port PyTorch Quickstart to NNI
+==============================
+This is a modified version of `PyTorch quickstart`_.
+
+It can be run directly and will have the exact same result as original version.
+
+Furthermore, it enables the ability of auto-tuning with an NNI *experiment*, which will be discussed later.
+
+For now, we recommend to run this script directly to verify the environment.
+
+There are only 2 key differences from the original version:
+
+1. In `Get optimized hyperparameters`_ part, it receives auto-generated hyperparameters.
+2. In `Train the model and report accuracy`_ part, it reports accuracy metrics for tuner to generate next hyperparameter set.
+
+.. _PyTorch quickstart: https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html
+"""
+
+# %%
+import nni
+import torch
+from torch import nn
+from torch.utils.data import DataLoader
+from torchvision import datasets
+from torchvision.transforms import ToTensor
+
+# %%
+# Hyperparameters to be tuned
+# ---------------------------
+params = {
+    'features': 512,
+    'lr': 0.001,
+    'momentum': 0,
+}
+
+# %%
+# Get optimized hyperparameters
+# -----------------------------
+# If run directly, ``nni.get_next_parameters()`` is a no-op and returns an empty dict.
+# But with an NNI *experiment*, it will receive optimized hyperparameters from tuning algorithm.
+optimized_params = nni.get_next_parameter()
+params.update(optimized_params)
+print(params)
+
+# %%
+# Load dataset
+# ------------
+training_data = datasets.FashionMNIST(root="data", train=True, download=True, transform=ToTensor())
+test_data = datasets.FashionMNIST(root="data", train=False, download=True, transform=ToTensor())
+
+batch_size = 64
+
+train_dataloader = DataLoader(training_data, batch_size=batch_size)
+test_dataloader = DataLoader(test_data, batch_size=batch_size)
+
+# %%
+# Build model with hyperparameters
+# --------------------------------
+device = "cuda" if torch.cuda.is_available() else "cpu"
+print(f"Using {device} device")
+
+class NeuralNetwork(nn.Module):
+    def __init__(self):
+        super(NeuralNetwork, self).__init__()
+        self.flatten = nn.Flatten()
+        self.linear_relu_stack = nn.Sequential(
+            nn.Linear(28*28, params['features']),
+            nn.ReLU(),
+            nn.Linear(params['features'], params['features']),
+            nn.ReLU(),
+            nn.Linear(params['features'], 10)
+        )
+
+    def forward(self, x):
+        x = self.flatten(x)
+        logits = self.linear_relu_stack(x)
+        return logits
+
+model = NeuralNetwork().to(device)
+
+loss_fn = nn.CrossEntropyLoss()
+optimizer = torch.optim.SGD(model.parameters(), lr=params['lr'], momentum=params['momentum'])
+
+# %%
+# Define train() and test()
+# -------------------------
+def train(dataloader, model, loss_fn, optimizer):
+    size = len(dataloader.dataset)
+    model.train()
+    for batch, (X, y) in enumerate(dataloader):
+        X, y = X.to(device), y.to(device)
+        pred = model(X)
+        loss = loss_fn(pred, y)
+        optimizer.zero_grad()
+        loss.backward()
+        optimizer.step()
+
+def test(dataloader, model, loss_fn):
+    size = len(dataloader.dataset)
+    num_batches = len(dataloader)
+    model.eval()
+    test_loss, correct = 0, 0
+    with torch.no_grad():
+        for X, y in dataloader:
+            X, y = X.to(device), y.to(device)
+            pred = model(X)
+            test_loss += loss_fn(pred, y).item()
+            correct += (pred.argmax(1) == y).type(torch.float).sum().item()
+    test_loss /= num_batches
+    correct /= size
+    return correct
+
+# %%
+# Train the model and report accuracy
+# -----------------------------------
+# Report accuracy to NNI so the tuning algorithm can predict best hyperparameters.
+epochs = 5
+for t in range(epochs):
+    print(f"Epoch {t+1}\n-------------------------------")
+    train(train_dataloader, model, loss_fn, optimizer)
+    accuracy = test(test_dataloader, model, loss_fn)
+    nni.report_intermediate_result(accuracy)
+nni.report_final_result(accuracy)

nni/tools/nnictl/hello.py

@@ -0,0 +1,26 @@
+# Copyright (c) Microsoft Corporation.
+# Licensed under the MIT license.
+
+"""
+Provide ``nnictl hello`` command to generate quickstart example.
+"""
+
+from pathlib import Path
+import shutil
+
+from colorama import Fore
+
+import nni_assets
+
+def create_example(_args):
+    example_path = Path(nni_assets.__path__[0], 'hello_hpo')
+    try:
+        shutil.copytree(example_path, 'nni_hello_hpo')
+    except PermissionError:
+        print(Fore.RED + 'Permission denied. Please run the command in a writable directory.' + Fore.RESET)
+        exit(1)
+    except FileExistsError:
+        print('File exists. Please run "python nni_hello_hpo/main.py" to start the example.')
+        exit(1)
+    print('A hyperparameter optimization example has been created at "nni_hello_hpo" directory.')
+    print('Please run "python nni_hello_hpo/main.py" to try it out.')

nni/tools/nnictl/nnictl.py

@@ -16,7 +16,8 @@ from .nnictl_utils import stop_experiment, trial_ls, trial_kill, list_experiment
     save_experiment, load_experiment
 from .algo_management import algo_reg, algo_unreg, algo_show, algo_list
 from .constants import DEFAULT_REST_PORT
-from .import ts_management
+from . import hello
+from . import ts_management
 
 init(autoreset=True)
 
@@ -483,6 +484,10 @@ def get_parser():
     jupyter_uninstall_parser = jupyter_subparsers.add_parser('uninstall', description='Uninstall JupyterLab extension.')
     jupyter_uninstall_parser.set_defaults(func=_jupyter_uninstall)
 
+    # hello command
+    parser_hello = subparsers.add_parser('hello', description='Create "hello nni" example in current directory.')
+    parser_hello.set_defaults(func=hello.create_example)
+
     return parser
 
 

nni_assets/hello_hpo/main.py

@@ -0,0 +1,33 @@
+"""
+NNI hyperparameter optimization example.
+
+Check the online tutorial for details:
+https://nni.readthedocs.io/en/stable/tutorials/hpo_quickstart_pytorch/main.html
+"""
+
+from pathlib import Path
+import signal
+
+from nni.experiment import Experiment
+
+# Define search space
+search_space = {
+    'features': {'_type': 'choice', '_value': [128, 256, 512, 1024]},
+    'lr': {'_type': 'loguniform', '_value': [0.0001, 0.1]},
+    'momentum': {'_type': 'uniform', '_value': [0, 1]},
+}
+
+# Configure experiment
+experiment = Experiment('local')
+experiment.config.trial_command = 'python model.py'
+experiment.config.trial_code_directory = Path(__file__).parent
+experiment.config.search_space = search_space
+experiment.config.tuner.name = 'Random'
+experiment.config.max_trial_number = 10
+experiment.config.trial_concurrency = 2
+
+# Run it!
+experiment.run(port=8080, wait_completion=False)
+
+print('Experiment is running. Press Ctrl-C to quit.')
+signal.pause()

nni_assets/hello_hpo/model.py

@@ -0,0 +1,67 @@
+"""
+Run main.py to start.
+
+This script is modified from PyTorch quickstart:
+https://pytorch.org/tutorials/beginner/basics/quickstart_tutorial.html
+"""
+
+import nni
+import torch
+from torch import nn
+from torch.utils.data import DataLoader
+from torchvision import datasets
+from torchvision.transforms import ToTensor
+
+# Get optimized hyperparameters
+params = {'features': 512, 'lr': 0.001, 'momentum': 0}
+optimized_params = nni.get_next_parameter()
+params.update(optimized_params)
+
+# Load dataset
+training_data = datasets.FashionMNIST(root='data', train=True, download=True, transform=ToTensor())
+test_data = datasets.FashionMNIST(root='data', train=False, download=True, transform=ToTensor())
+train_dataloader = DataLoader(training_data, batch_size=64)
+test_dataloader = DataLoader(test_data, batch_size=64)
+
+# Build model
+device = 'cuda' if torch.cuda.is_available() else 'cpu'
+model = nn.Sequential(
+    nn.Flatten(),
+    nn.Linear(28*28, params['features']),
+    nn.ReLU(),
+    nn.Linear(params['features'], params['features']),
+    nn.ReLU(),
+    nn.Linear(params['features'], 10)
+).to(device)
+
+# Training functions
+loss_fn = nn.CrossEntropyLoss()
+optimizer = torch.optim.SGD(model.parameters(), lr=params['lr'], momentum=params['momentum'])
+
+def train(dataloader, model, loss_fn, optimizer):
+    model.train()
+    for batch, (X, y) in enumerate(dataloader):
+        X, y = X.to(device), y.to(device)
+        pred = model(X)
+        loss = loss_fn(pred, y)
+        optimizer.zero_grad()
+        loss.backward()
+        optimizer.step()
+
+def test(dataloader, model, loss_fn):
+    model.eval()
+    correct = 0
+    with torch.no_grad():
+        for X, y in dataloader:
+            X, y = X.to(device), y.to(device)
+            pred = model(X)
+            correct += (pred.argmax(1) == y).type(torch.float).sum().item()
+    return correct / len(dataloader.dataset)
+
+# Train the model
+epochs = 5
+for t in range(epochs):
+    train(train_dataloader, model, loss_fn, optimizer)
+    accuracy = test(test_dataloader, model, loss_fn)
+    nni.report_intermediate_result(accuracy)
+nni.report_final_result(accuracy)

setup.py

@@ -112,6 +112,7 @@ def _setup():
         packages = _find_python_packages(),
         package_data = {
             'nni': _find_requirements_txt() + _find_default_config(),  # setuptools issue #1806
+            'nni_assets': _find_asset_files(),
             'nni_node': _find_node_files()  # note: this does not work before building
         },
 
@@ -165,6 +166,14 @@ def _find_requirements_txt():
 def _find_default_config():
     return ['runtime/default_config/' + name for name in os.listdir('nni/runtime/default_config')]
 
+def _find_asset_files():
+    files = []
+    for dirpath, dirnames, filenames in os.walk('nni_assets'):
+        for filename in filenames:
+            if os.path.splitext(filename)[1] == '.py':
+                files.append(os.path.join(dirpath[len('nni_assets/'):], filename))
+    return sorted(files)
+
 def _find_node_files():
     if not os.path.exists('nni_node'):
         if release and 'build_ts' not in sys.argv and 'clean' not in sys.argv:
@@ -279,7 +288,10 @@ _temp_files = [
     'test/model_path/',
     'test/temp.json',
     'test/ut/sdk/*.pth',
-    'test/ut/tools/annotation/_generated/'
+    'test/ut/tools/annotation/_generated/',
+
+    # example
+    'nni_assets/**/data/',
 ]