microsoft
/
nni
зеркало из https://github.com/microsoft/nni.git
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

NAS 3.0 docs (quick update) (#5509)

This commit is contained in:
Yuge Zhang 2023-04-17 18:28:29 +08:00 коммит произвёл GitHub
Родитель 332435c0f3
Коммит 0d4ea75a2e
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
53 изменённых файлов: 1523 добавлений и 2223 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

4
docs/source/conf.py
Просмотреть файл

@ -210,8 +210,8 @@ gettext_documents = [
r'^index$',
r'^quickstart$',
r'^installation$',
r'^(nas|hpo|compression)/overview$',
r'^tutorials/(hello_nas|pruning_quick_start_mnist|hpo_quickstart_pytorch/main)$',
r'^(hpo|compression)/overview$',
r'^tutorials/(pruning_quick_start_mnist|hpo_quickstart_pytorch/main)$',
]
# List of patterns, relative to source directory, that match files and

782
docs/source/locales/zh/LC_MESSAGES/tutorials.po
Просмотреть файл

@ -8,7 +8,7 @@ msgid ""
msgstr ""
"Project-Id-Version: NNI \n"
"Report-Msgid-Bugs-To: \n"
"POT-Creation-Date: 2022-10-18 19:27+0800\n"
"POT-Creation-Date: 2023-04-12 10:42+0800\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
@ -17,334 +17,6 @@ msgstr ""
"Content-Transfer-Encoding: 8bit\n"
"Generated-By: Babel 2.9.1\n"
#: ../../source/tutorials/hello_nas.rst:13
msgid ""
"Click :ref:`here <sphx_glr_download_tutorials_hello_nas.py>` to download "
"the full example code"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:22
msgid "Hello, NAS!"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:24
msgid ""
"This is the 101 tutorial of Neural Architecture Search (NAS) on NNI. In "
"this tutorial, we will search for a neural architecture on MNIST dataset "
"with the help of NAS framework of NNI, i.e., *Retiarii*. We use multi-"
"trial NAS as an example to show how to construct and explore a model "
"space."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:28
msgid ""
"There are mainly three crucial components for a neural architecture "
"search task, namely,"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:30
msgid "Model search space that defines a set of models to explore."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:31
msgid "A proper strategy as the method to explore this model space."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:32
msgid ""
"A model evaluator that reports the performance of every model in the "
"space."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:34
msgid ""
"Currently, PyTorch is the only supported framework by Retiarii, and we "
"have only tested **PyTorch 1.7 to 1.10**. This tutorial assumes PyTorch "
"context but it should also apply to other frameworks, which is in our "
"future plan."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:38
msgid "Define your Model Space"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:40
msgid ""
"Model space is defined by users to express a set of models that users "
"want to explore, which contains potentially good-performing models. In "
"this framework, a model space is defined with two parts: a base model and"
" possible mutations on the base model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:46
msgid "Define Base Model"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:48
msgid ""
"Defining a base model is almost the same as defining a PyTorch (or "
"TensorFlow) model. Usually, you only need to replace the code ``import "
"torch.nn as nn`` with ``import nni.retiarii.nn.pytorch as nn`` to use our"
" wrapped PyTorch modules."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:52
msgid "Below is a very simple example of defining a base model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:93
msgid ""
"Always keep in mind that you should use ``import nni.retiarii.nn.pytorch "
"as nn`` and :meth:`nni.retiarii.model_wrapper`. Many mistakes are a "
"result of forgetting one of those. Also, please use ``torch.nn`` for "
"submodules of ``nn.init``, e.g., ``torch.nn.init`` instead of "
"``nn.init``."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:98
msgid "Define Model Mutations"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:100
msgid ""
"A base model is only one concrete model not a model space. We provide "
":doc:`API and Primitives </nas/construct_space>` for users to express how"
" the base model can be mutated. That is, to build a model space which "
"includes many models."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:103
msgid "Based on the above base model, we can define a model space as below."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:134
msgid "This results in the following code:"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:189
#: ../../source/tutorials/hello_nas.rst:259
#: ../../source/tutorials/hello_nas.rst:471
#: ../../source/tutorials/hello_nas.rst:564
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:244
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:281
msgid "Out:"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:210
msgid ""
"This example uses two mutation APIs, :class:`nn.LayerChoice "
"<nni.retiarii.nn.pytorch.LayerChoice>` and :class:`nn.InputChoice "
"<nni.retiarii.nn.pytorch.ValueChoice>`. :class:`nn.LayerChoice "
"<nni.retiarii.nn.pytorch.LayerChoice>` takes a list of candidate modules "
"(two in this example), one will be chosen for each sampled model. It can "
"be used like normal PyTorch module. :class:`nn.InputChoice "
"<nni.retiarii.nn.pytorch.ValueChoice>` takes a list of candidate values, "
"one will be chosen to take effect for each sampled model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:219
msgid ""
"More detailed API description and usage can be found :doc:`here "
"</nas/construct_space>`."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:223
msgid ""
"We are actively enriching the mutation APIs, to facilitate easy "
"construction of model space. If the currently supported mutation APIs "
"cannot express your model space, please refer to :doc:`this doc "
"</nas/mutator>` for customizing mutators."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:228
msgid "Explore the Defined Model Space"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:230
msgid ""
"There are basically two exploration approaches: (1) search by evaluating "
"each sampled model independently, which is the search approach in :ref"
":`multi-trial NAS <multi-trial-nas>` and (2) one-shot weight-sharing "
"based search, which is used in one-shot NAS. We demonstrate the first "
"approach in this tutorial. Users can refer to :ref:`here <one-shot-nas>` "
"for the second approach."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:235
msgid ""
"First, users need to pick a proper exploration strategy to explore the "
"defined model space. Second, users need to pick or customize a model "
"evaluator to evaluate the performance of each explored model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:239
msgid "Pick an exploration strategy"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:241
msgid ""
"Retiarii supports many :doc:`exploration strategies "
"</nas/exploration_strategy>`."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:243
msgid "Simply choosing (i.e., instantiate) an exploration strategy as below."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:273
msgid "Pick or customize a model evaluator"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:275
msgid ""
"In the exploration process, the exploration strategy repeatedly generates"
" new models. A model evaluator is for training and validating each "
"generated model to obtain the model's performance. The performance is "
"sent to the exploration strategy for the strategy to generate better "
"models."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:279
msgid ""
"Retiarii has provided :doc:`built-in model evaluators </nas/evaluator>`, "
"but to start with, it is recommended to use :class:`FunctionalEvaluator "
"<nni.retiarii.evaluator.FunctionalEvaluator>`, that is, to wrap your own "
"training and evaluation code with one single function. This function "
"should receive one single model class and uses "
":func:`nni.report_final_result` to report the final score of this model."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:284
msgid ""
"An example here creates a simple evaluator that runs on MNIST dataset, "
"trains for 2 epochs, and reports its validation accuracy."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:367
msgid "Create the evaluator"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:386
msgid ""
"The ``train_epoch`` and ``test_epoch`` here can be any customized "
"function, where users can write their own training recipe."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:389
msgid ""
"It is recommended that the ``evaluate_model`` here accepts no additional "
"arguments other than ``model_cls``. However, in the :doc:`advanced "
"tutorial </nas/evaluator>`, we will show how to use additional arguments "
"in case you actually need those. In future, we will support mutation on "
"the arguments of evaluators, which is commonly called \"Hyper-parmeter "
"tuning\"."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:394
msgid "Launch an Experiment"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:396
msgid ""
"After all the above are prepared, it is time to start an experiment to do"
" the model search. An example is shown below."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:417
msgid ""
"The following configurations are useful to control how many trials to run"
" at most / at the same time."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:436
msgid ""
"Remember to set the following config if you want to GPU. "
"``use_active_gpu`` should be set true if you wish to use an occupied GPU "
"(possibly running a GUI)."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:456
msgid ""
"Launch the experiment. The experiment should take several minutes to "
"finish on a workstation with 2 GPUs."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:495
msgid ""
"Users can also run Retiarii Experiment with :doc:`different training "
"services </experiment/training_service/overview>` besides ``local`` "
"training service."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:499
msgid "Visualize the Experiment"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:501
msgid ""
"Users can visualize their experiment in the same way as visualizing a "
"normal hyper-parameter tuning experiment. For example, open "
"``localhost:8081`` in your browser, 8081 is the port that you set in "
"``exp.run``. Please refer to :doc:`here "
"</experiment/web_portal/web_portal>` for details."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:505
msgid ""
"We support visualizing models with 3rd-party visualization engines (like "
"`Netron <https://netron.app/>`__). This can be used by clicking "
"``Visualization`` in detail panel for each trial. Note that current "
"visualization is based on `onnx <https://onnx.ai/>`__ , thus "
"visualization is not feasible if the model cannot be exported into onnx."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:510
msgid ""
"Built-in evaluators (e.g., Classification) will automatically export the "
"model into a file. For your own evaluator, you need to save your file "
"into ``$NNI_OUTPUT_DIR/model.onnx`` to make this work. For instance,"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:541
msgid "Relaunch the experiment, and a button is shown on Web portal."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:546
msgid "Export Top Models"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:548
msgid ""
"Users can export top models after the exploration is done using "
"``export_top_models``."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:575
msgid ""
"The output is ``json`` object which records the mutation actions of the "
"top model. If users want to output source code of the top model, they can"
" use :ref:`graph-based execution engine <graph-based-execution-engine>` "
"for the experiment, by simply adding the following two lines."
msgstr ""
#: ../../source/tutorials/hello_nas.rst:597
msgid "**Total running time of the script:** ( 2 minutes 4.499 seconds)"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:612
msgid ":download:`Download Python source code: hello_nas.py <hello_nas.py>`"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:618
msgid ":download:`Download Jupyter notebook: hello_nas.ipynb <hello_nas.ipynb>`"
msgstr ""
#: ../../source/tutorials/hello_nas.rst:625
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:335
#: ../../source/tutorials/pruning_quick_start_mnist.rst:343
msgid "`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:13
msgid ""
"Click :ref:`here "
@ -579,6 +251,11 @@ msgid ""
"http://localhost:8080."
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:244
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:281
msgid "Out:"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:260
msgid "After the experiment is done"
msgstr ""
@ -630,6 +307,11 @@ msgstr ""
msgid ":download:`Download Jupyter notebook: main.ipynb <main.ipynb>`"
msgstr ""
#: ../../source/tutorials/hpo_quickstart_pytorch/main.rst:335
#: ../../source/tutorials/pruning_quick_start_mnist.rst:343
msgid "`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_"
msgstr ""
#: ../../source/tutorials/pruning_quick_start_mnist.rst:13
msgid ""
"Click :ref:`here "
@ -775,3 +457,445 @@ msgstr ""
#~ msgid "**Total running time of the script:** ( 1 minutes 30.730 seconds)"
#~ msgstr ""
#~ msgid ""
#~ "Click :ref:`here "
#~ "<sphx_glr_download_tutorials_hello_nas.py>` to download"
#~ " the full example code"
#~ msgstr ""
#~ msgid ""
#~ "Currently, PyTorch is the only supported"
#~ " framework by Retiarii, and we have"
#~ " only tested **PyTorch 1.7 to 1.10**."
#~ " This tutorial assumes PyTorch context "
#~ "but it should also apply to other"
#~ " frameworks, which is in our future"
#~ " plan."
#~ msgstr ""
#~ msgid ""
#~ "Defining a base model is almost "
#~ "the same as defining a PyTorch (or"
#~ " TensorFlow) model. Usually, you only "
#~ "need to replace the code ``import "
#~ "torch.nn as nn`` with ``import "
#~ "nni.retiarii.nn.pytorch as nn`` to use "
#~ "our wrapped PyTorch modules."
#~ msgstr ""
#~ msgid ""
#~ "Always keep in mind that you "
#~ "should use ``import nni.retiarii.nn.pytorch as"
#~ " nn`` and :meth:`nni.retiarii.model_wrapper`. "
#~ "Many mistakes are a result of "
#~ "forgetting one of those. Also, please"
#~ " use ``torch.nn`` for submodules of "
#~ "``nn.init``, e.g., ``torch.nn.init`` instead "
#~ "of ``nn.init``."
#~ msgstr ""
#~ msgid "Define Model Mutations"
#~ msgstr ""
#~ msgid ""
#~ "This example uses two mutation APIs, "
#~ ":class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`"
#~ " and :class:`nn.InputChoice "
#~ "<nni.retiarii.nn.pytorch.ValueChoice>`. :class:`nn.LayerChoice"
#~ " <nni.retiarii.nn.pytorch.LayerChoice>` takes a "
#~ "list of candidate modules (two in "
#~ "this example), one will be chosen "
#~ "for each sampled model. It can be"
#~ " used like normal PyTorch module. "
#~ ":class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>`"
#~ " takes a list of candidate values,"
#~ " one will be chosen to take "
#~ "effect for each sampled model."
#~ msgstr ""
#~ msgid ""
#~ "Retiarii supports many :doc:`exploration "
#~ "strategies </nas/exploration_strategy>`."
#~ msgstr ""
#~ msgid ""
#~ "Retiarii has provided :doc:`built-in "
#~ "model evaluators </nas/evaluator>`, but to "
#~ "start with, it is recommended to "
#~ "use :class:`FunctionalEvaluator "
#~ "<nni.retiarii.evaluator.FunctionalEvaluator>`, that is,"
#~ " to wrap your own training and "
#~ "evaluation code with one single "
#~ "function. This function should receive "
#~ "one single model class and uses "
#~ ":func:`nni.report_final_result` to report the "
#~ "final score of this model."
#~ msgstr ""
#~ msgid ""
#~ "It is recommended that the "
#~ "``evaluate_model`` here accepts no additional"
#~ " arguments other than ``model_cls``. "
#~ "However, in the :doc:`advanced tutorial "
#~ "</nas/evaluator>`, we will show how to"
#~ " use additional arguments in case you"
#~ " actually need those. In future, we"
#~ " will support mutation on the "
#~ "arguments of evaluators, which is "
#~ "commonly called \"Hyper-parmeter tuning\"."
#~ msgstr ""
#~ msgid ""
#~ "The following configurations are useful "
#~ "to control how many trials to run"
#~ " at most / at the same time."
#~ msgstr ""
#~ msgid ""
#~ "Remember to set the following config "
#~ "if you want to GPU. ``use_active_gpu``"
#~ " should be set true if you wish"
#~ " to use an occupied GPU (possibly "
#~ "running a GUI)."
#~ msgstr ""
#~ msgid ""
#~ "Users can also run Retiarii Experiment"
#~ " with :doc:`different training services "
#~ "</experiment/training_service/overview>` besides ``local``"
#~ " training service."
#~ msgstr ""
#~ msgid ""
#~ "The output is ``json`` object which "
#~ "records the mutation actions of the "
#~ "top model. If users want to output"
#~ " source code of the top model, "
#~ "they can use :ref:`graph-based execution"
#~ " engine <graph-based-execution-engine>` "
#~ "for the experiment, by simply adding "
#~ "the following two lines."
#~ msgstr ""
#~ msgid "**Total running time of the script:** ( 2 minutes 4.499 seconds)"
#~ msgstr ""
#~ msgid ""
#~ ":ref:`Go to the end "
#~ "<sphx_glr_download_tutorials_hello_nas.py>` to download"
#~ " the full example code"
#~ msgstr ""
#~ msgid "Hello, NAS!"
#~ msgstr ""
#~ msgid ""
#~ "This is the 101 tutorial of Neural"
#~ " Architecture Search (NAS) on NNI. In"
#~ " this tutorial, we will search for"
#~ " a neural architecture on MNIST "
#~ "dataset with the help of NAS "
#~ "framework of NNI, i.e., *Retiarii*. We"
#~ " use multi-trial NAS as an "
#~ "example to show how to construct "
#~ "and explore a model space."
#~ msgstr ""
#~ msgid ""
#~ "There are mainly three crucial "
#~ "components for a neural architecture "
#~ "search task, namely,"
#~ msgstr ""
#~ msgid "Model search space that defines a set of models to explore."
#~ msgstr ""
#~ msgid "A proper strategy as the method to explore this model space."
#~ msgstr ""
#~ msgid ""
#~ "A model evaluator that reports the "
#~ "performance of every model in the "
#~ "space."
#~ msgstr ""
#~ msgid ""
#~ "Currently, PyTorch is the only supported"
#~ " framework by Retiarii, and we have"
#~ " only tested **PyTorch 1.9 to 1.13**."
#~ " This tutorial assumes PyTorch context "
#~ "but it should also apply to other"
#~ " frameworks, which is in our future"
#~ " plan."
#~ msgstr ""
#~ msgid "Define your Model Space"
#~ msgstr ""
#~ msgid ""
#~ "Model space is defined by users to"
#~ " express a set of models that "
#~ "users want to explore, which contains"
#~ " potentially good-performing models. In "
#~ "this framework, a model space is "
#~ "defined with two parts: a base "
#~ "model and possible mutations on the "
#~ "base model."
#~ msgstr ""
#~ msgid "Define Base Model"
#~ msgstr ""
#~ msgid ""
#~ "Defining a base model is almost "
#~ "the same as defining a PyTorch (or"
#~ " TensorFlow) model."
#~ msgstr ""
#~ msgid "Below is a very simple example of defining a base model."
#~ msgstr ""
#~ msgid "Define Model Variations"
#~ msgstr ""
#~ msgid ""
#~ "A base model is only one concrete"
#~ " model not a model space. We "
#~ "provide :doc:`API and Primitives "
#~ "</nas/construct_space>` for users to express"
#~ " how the base model can be "
#~ "mutated. That is, to build a model"
#~ " space which includes many models."
#~ msgstr ""
#~ msgid "Based on the above base model, we can define a model space as below."
#~ msgstr ""
#~ msgid "This results in the following code:"
#~ msgstr ""
#~ msgid ""
#~ "This example uses two mutation APIs, "
#~ ":class:`nn.LayerChoice <nni.nas.nn.pytorch.LayerChoice>` "
#~ "and :func:`nni.choice`. :class:`nn.LayerChoice "
#~ "<nni.nas.nn.pytorch.LayerChoice>` takes a list "
#~ "of candidate modules (two in this "
#~ "example), one will be chosen for "
#~ "each sampled model. It can be used"
#~ " like normal PyTorch module. "
#~ ":func:`nni.choice` is used as parameter "
#~ "of `MutableDropout`, which then takes "
#~ "the result as dropout rate."
#~ msgstr ""
#~ msgid ""
#~ "More detailed API description and usage"
#~ " can be found :doc:`here "
#~ "</nas/construct_space>`."
#~ msgstr ""
#~ msgid ""
#~ "We are actively enriching the mutation"
#~ " APIs, to facilitate easy construction "
#~ "of model space. If the currently "
#~ "supported mutation APIs cannot express "
#~ "your model space, please refer to "
#~ ":doc:`this doc </nas/mutator>` for customizing"
#~ " mutators."
#~ msgstr ""
#~ msgid "Explore the Defined Model Space"
#~ msgstr ""
#~ msgid ""
#~ "There are basically two exploration "
#~ "approaches: (1) search by evaluating "
#~ "each sampled model independently, which "
#~ "is the search approach in :ref"
#~ ":`multi-trial NAS <multi-trial-nas>` "
#~ "and (2) one-shot weight-sharing "
#~ "based search, which is used in "
#~ "one-shot NAS. We demonstrate the "
#~ "first approach in this tutorial. Users"
#~ " can refer to :ref:`here <one-"
#~ "shot-nas>` for the second approach."
#~ msgstr ""
#~ msgid ""
#~ "First, users need to pick a proper"
#~ " exploration strategy to explore the "
#~ "defined model space. Second, users need"
#~ " to pick or customize a model "
#~ "evaluator to evaluate the performance of"
#~ " each explored model."
#~ msgstr ""
#~ msgid "Pick an exploration strategy"
#~ msgstr ""
#~ msgid ""
#~ "NNI NAS supports many :doc:`exploration "
#~ "strategies </nas/exploration_strategy>`."
#~ msgstr ""
#~ msgid "Simply choosing (i.e., instantiate) an exploration strategy as below."
#~ msgstr ""
#~ msgid "Pick or customize a model evaluator"
#~ msgstr ""
#~ msgid ""
#~ "In the exploration process, the "
#~ "exploration strategy repeatedly generates new"
#~ " models. A model evaluator is for "
#~ "training and validating each generated "
#~ "model to obtain the model's performance."
#~ " The performance is sent to the "
#~ "exploration strategy for the strategy to"
#~ " generate better models."
#~ msgstr ""
#~ msgid ""
#~ "NNI NAS has provided :doc:`built-in "
#~ "model evaluators </nas/evaluator>`, but to "
#~ "start with, it is recommended to "
#~ "use :class:`FunctionalEvaluator "
#~ "<nni.nas.evaluator.FunctionalEvaluator>`, that is, "
#~ "to wrap your own training and "
#~ "evaluation code with one single "
#~ "function. This function should receive "
#~ "one single model class and uses "
#~ ":func:`nni.report_final_result` to report the "
#~ "final score of this model."
#~ msgstr ""
#~ msgid ""
#~ "An example here creates a simple "
#~ "evaluator that runs on MNIST dataset,"
#~ " trains for 2 epochs, and reports "
#~ "its validation accuracy."
#~ msgstr ""
#~ msgid "Create the evaluator"
#~ msgstr ""
#~ msgid ""
#~ "The ``train_epoch`` and ``test_epoch`` here"
#~ " can be any customized function, "
#~ "where users can write their own "
#~ "training recipe."
#~ msgstr ""
#~ msgid ""
#~ "It is recommended that the "
#~ "``evaluate_model`` here accepts no additional"
#~ " arguments other than ``model``. However,"
#~ " in the :doc:`advanced tutorial "
#~ "</nas/evaluator>`, we will show how to"
#~ " use additional arguments in case you"
#~ " actually need those. In future, we"
#~ " will support mutation on the "
#~ "arguments of evaluators, which is "
#~ "commonly called \"Hyper-parameter tuning\"."
#~ msgstr ""
#~ msgid "Launch an Experiment"
#~ msgstr ""
#~ msgid ""
#~ "After all the above are prepared, "
#~ "it is time to start an experiment"
#~ " to do the model search. An "
#~ "example is shown below."
#~ msgstr ""
#~ msgid ""
#~ "Different from HPO experiment, NAS "
#~ "experiment will generate an experiment "
#~ "config automatically. It should work for"
#~ " most cases. For example, when using"
#~ " multi-trial strategies, local training "
#~ "service with concurrency 1 will be "
#~ "used by default. Users can customize "
#~ "the config. For example,"
#~ msgstr ""
#~ msgid ""
#~ "Remember to set the following config "
#~ "if you want to GPU. ``use_active_gpu``"
#~ " should be set true if you wish"
#~ " to use an occupied GPU (possibly "
#~ "running a GUI)::"
#~ msgstr ""
#~ msgid ""
#~ "Launch the experiment. The experiment "
#~ "should take several minutes to finish"
#~ " on a workstation with 2 GPUs."
#~ msgstr ""
#~ msgid ""
#~ "Users can also run NAS Experiment "
#~ "with :doc:`different training services "
#~ "</experiment/training_service/overview>` besides ``local``"
#~ " training service."
#~ msgstr ""
#~ msgid "Visualize the Experiment"
#~ msgstr ""
#~ msgid ""
#~ "Users can visualize their experiment in"
#~ " the same way as visualizing a "
#~ "normal hyper-parameter tuning experiment. "
#~ "For example, open ``localhost:8081`` in "
#~ "your browser, 8081 is the port "
#~ "that you set in ``exp.run``. Please "
#~ "refer to :doc:`here "
#~ "</experiment/web_portal/web_portal>` for details."
#~ msgstr ""
#~ msgid ""
#~ "We support visualizing models with "
#~ "3rd-party visualization engines (like "
#~ "`Netron <https://netron.app/>`__). This can be"
#~ " used by clicking ``Visualization`` in "
#~ "detail panel for each trial. Note "
#~ "that current visualization is based on"
#~ " `onnx <https://onnx.ai/>`__ , thus "
#~ "visualization is not feasible if the "
#~ "model cannot be exported into onnx."
#~ msgstr ""
#~ msgid ""
#~ "Built-in evaluators (e.g., Classification) "
#~ "will automatically export the model into"
#~ " a file. For your own evaluator, "
#~ "you need to save your file into"
#~ " ``$NNI_OUTPUT_DIR/model.onnx`` to make this "
#~ "work. For instance,"
#~ msgstr ""
#~ msgid "Relaunch the experiment, and a button is shown on Web portal."
#~ msgstr ""
#~ msgid "Export Top Models"
#~ msgstr ""
#~ msgid ""
#~ "Users can export top models after "
#~ "the exploration is done using "
#~ "``export_top_models``."
#~ msgstr ""
#~ msgid "**Total running time of the script:** ( 13 minutes 8.330 seconds)"
#~ msgstr ""
#~ msgid ":download:`Download Python source code: hello_nas.py <hello_nas.py>`"
#~ msgstr ""
#~ msgid ""
#~ ":download:`Download Jupyter notebook: "
#~ "hello_nas.ipynb <hello_nas.ipynb>`"
#~ msgstr ""

1
docs/source/nas/advanced_usage.rst
Просмотреть файл

@ -7,6 +7,5 @@ Advanced Usage
execution_engine
hardware_aware_nas
mutator
customize_strategy
serialization
benchmarks_toctree

42
docs/source/nas/benchmarks.rst
Просмотреть файл

@ -14,38 +14,18 @@ Prerequisites
Data Preparation
----------------
Option 1 (Recommended)
^^^^^^^^^^^^^^^^^^^^^^
You can download the preprocessed benchmark files via ``python -m nni.nas.benchmark.download <benchmark_name>``, where ``<benchmark_name>`` can be ``nasbench101``, ``nasbench201``, and etc. Add ``--help`` to the command for supported command line arguments.
Option 2
^^^^^^^^
.. note:: If you have files that are processed before v2.5, it is recommended that you delete them and try option 1.
#. Clone NNI to your machine and enter ``examples/nas/benchmarks`` directory.
.. code-block:: bash
git clone -b ${NNI_VERSION} https://github.com/microsoft/nni
cd nni/examples/nas/benchmarks
Replace ``${NNI_VERSION}`` with a released version name or branch name, e.g., ``v2.4``.
#. Install dependencies via ``pip3 install -r xxx.requirements.txt``. ``xxx`` can be ``nasbench101``, ``nasbench201`` or ``nds``.
#. Generate the database via ``./xxx.sh``. The directory that stores the benchmark file can be configured with ``NASBENCHMARK_DIR`` environment variable, which defaults to ``~/.nni/nasbenchmark``. Note that the NAS-Bench-201 dataset will be downloaded from a google drive.
Please make sure there is at least 10GB free disk space and note that the conversion process can take up to hours to complete.
Example Usages
--------------
Please refer to :doc:`examples usages of Benchmarks API </tutorials/nasbench_as_dataset>`.
Please refer to :githublink:`test/algo/nas/benchmark/test_algo.py` on how to test algorithms on NAS benchmarks.
Supported benchmarks
--------------------
NAS-Bench-101
-------------
^^^^^^^^^^^^^
* `Paper link <https://arxiv.org/abs/1902.09635>`__
* `Open-source <https://github.com/google-research/nasbench>`__
@ -54,10 +34,8 @@ NAS-Bench-101 contains 423,624 unique neural networks, combined with 4 variation
Notably, NAS-Bench-101 eliminates invalid cells (e.g., there is no path from input to output, or there is redundant computation). Furthermore, isomorphic cells are de-duplicated, i.e., all the remaining cells are computationally unique.
See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nas-bench-101-reference>`.
NAS-Bench-201
-------------
^^^^^^^^^^^^^
* `Paper link <https://arxiv.org/abs/2001.00326>`__
* `Open-source API <https://github.com/D-X-Y/NAS-Bench-201>`__
@ -65,10 +43,8 @@ NAS-Bench-201
NAS-Bench-201 is a cell-wise search space that views nodes as tensors and edges as operators. The search space contains all possible densely-connected DAGs with 4 nodes, resulting in 15,625 candidates in total. Each operator (i.e., edge) is selected from a pre-defined operator set (\ ``NONE``, ``SKIP_CONNECT``, ``CONV_1X1``, ``CONV_3X3`` and ``AVG_POOL_3X3``\ ). Training appraoches vary in the dataset used (CIFAR-10, CIFAR-100, ImageNet) and number of epochs scheduled (12 and 200). Each combination of architecture and training approach is repeated 1 - 3 times with different random seeds.
See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nas-bench-201-reference>`.
NDS
---
^^^
* `Paper link <https://arxiv.org/abs/1905.13214>`__
* `Open-source <https://github.com/facebookresearch/nds>`__
@ -82,4 +58,6 @@ Here is a list of available operators used in NDS.
.. automodule:: nni.nas.benchmark.nds.constants
:noindex:
See :doc:`example usages </tutorials/nasbench_as_dataset>` and :ref:`API references <nds-reference>`.
.. warning::
NDS benchmark doesn't support benchmarking with existing algorithms because the authors only released a subset of data points in the search space. Benchmarking on NDS is a work in progress.

41
docs/source/nas/construct_space.rst
Просмотреть файл

@ -3,7 +3,7 @@ Construct Model Space
NNI provides powerful (and multi-level) APIs for users to easily express model space (or search space).
* *Mutation Primitives*: high-level APIs (e.g., ValueChoice, LayerChoice) that are utilities to build blocks in search space. In most cases, mutation pritimives should be straightforward yet expressive enough. **We strongly recommend users to try them first,** and report issues if those APIs are not satisfying.
* *Mutation Primitives*: high-level APIs (e.g., LayerChoice) that are utilities to build blocks in search space. In most cases, mutation pritimives should be straightforward yet expressive enough. **We strongly recommend users to try them first,** and report issues if those APIs are not satisfying.
* *Hyper-module Library*: plug-and-play modules that are proved useful. They are usually well studied in research, and comes with pre-searched results. (For example, the optimal activation function in `AutoActivation <https://arxiv.org/abs/1710.05941>`__ is reported to be `Swish <https://pytorch.org/docs/stable/generated/torch.nn.SiLU.html>`__).
* *Mutator*: for advanced users only. NNI provides interface to customize new mutators for expressing more complicated model spaces.
@ -16,30 +16,33 @@ The following table summarizes all the APIs we have provided for constructing se
* - Name
- Category
- Brief Description
* - :class:`LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`
- :ref:`Mutation Primitives <mutation-primitives>`
* - :class:`~nni.nas.nn.pytorch.ModelSpace`
- Mutation Primitives
- All model spaces should inherit this class
* - :class:`~nni.nas.nn.pytorch.ParametrizedModule`
- Mutation Primitives
- Modules with mutable parameters should inherit this class
* - :class:`LayerChoice <nni.nas.nn.pytorch.LayerChoice>`
- Mutation Primitives
- Select from some PyTorch modules
* - :class:`InputChoice <nni.retiarii.nn.pytorch.InputChoice>`
- :ref:`Mutation Primitives <mutation-primitives>`
* - :class:`InputChoice <nni.nas.nn.pytorch.InputChoice>`
- Mutation Primitives
- Select from some inputs (tensors)
* - :class:`ValueChoice <nni.retiarii.nn.pytorch.ValueChoice>`
- :ref:`Mutation Primitives <mutation-primitives>`
- Select from some candidate values
* - :class:`Repeat <nni.retiarii.nn.pytorch.Repeat>`
- :ref:`Mutation Primitives <mutation-primitives>`
* - :class:`Repeat <nni.nas.nn.pytorch.Repeat>`
- Mutation Primitives
- Repeat a block by a variable number of times
* - :class:`Cell <nni.retiarii.nn.pytorch.Cell>`
- :ref:`Mutation Primitives <mutation-primitives>`
* - :class:`Cell <nni.nas.nn.pytorch.Cell>`
- Mutation Primitives
- Cell structure popularly used in literature
* - :class:`NasBench101Cell <nni.retiarii.nn.pytorch.NasBench101Cell>`
- :ref:`Mutation Primitives <mutation-primitives>`
* - :class:`NasBench101Cell <nni.nas.hub.pytorch.modules.NasBench101Cell>`
- Mutation Primitives
- Cell structure (variant) proposed by NAS-Bench-101
* - :class:`NasBench201Cell <nni.retiarii.nn.pytorch.NasBench201Cell>`
- :ref:`Mutation Primitives <mutation-primitives>`
* - :class:`NasBench201Cell <nni.nas.hub.pytorch.modules.NasBench201Cell>`
- Mutation Primitives
- Cell structure (variant) proposed by NAS-Bench-201
* - :class:`AutoActivation <nni.retiarii.nn.pytorch.AutoActivation>`
- :ref:`Hyper-modules Library <hyper-modules>`
* - :class:`AutoActivation <nni.nas.hub.pytorch.modules.AutoActivation>`
- Hyper-modules library
- Searching for activation functions
* - :class:`Mutator <nni.retiarii.Mutator>`
* - :class:`Mutator <nni.nas.space.Mutator>`
- :doc:`Mutator <mutator>`
- Flexible mutations on graphs. :doc:`See tutorial here <mutator>`

98
docs/source/nas/customize_strategy.rst
Просмотреть файл

@ -1,98 +0,0 @@
Customize Exploration Strategy
==============================
Customize Multi-trial Strategy
------------------------------
If users want to innovate a new exploration strategy, they can easily customize a new one following the interface provided by NNI. Specifically, users should inherit the base strategy class :class:`nni.retiarii.strategy.BaseStrategy`, then implement the member function ``run``. This member function takes ``base_model`` and ``applied_mutators`` as its input arguments. It can simply apply the user specified mutators in ``applied_mutators`` onto ``base_model`` to generate a new model. When a mutator is applied, it should be bound with a sampler (e.g., ``RandomSampler``). Every sampler implements the ``choice`` function which chooses value(s) from candidate values. The ``choice`` functions invoked in mutators are executed with the sampler.
Below is a very simple random strategy, which makes the choices completely random.
.. code-block:: python
from nni.retiarii import Sampler
class RandomSampler(Sampler):
def choice(self, candidates, mutator, model, index):
return random.choice(candidates)
class RandomStrategy(BaseStrategy):
def __init__(self):
self.random_sampler = RandomSampler()
def run(self, base_model, applied_mutators):
_logger.info('stargety start...')
while True:
avail_resource = query_available_resources()
if avail_resource > 0:
model = base_model
_logger.info('apply mutators...')
_logger.info('mutators: %s', str(applied_mutators))
for mutator in applied_mutators:
mutator.bind_sampler(self.random_sampler)
model = mutator.apply(model)
# run models
submit_models(model)
else:
time.sleep(2)
You can find that this strategy does not know the search space beforehand, it passively makes decisions every time ``choice`` is invoked from mutators. If a strategy wants to know the whole search space before making any decision (e.g., TPE, SMAC), it can use ``dry_run`` function provided by ``Mutator`` to obtain the space. An example strategy can be found :githublink:`here <nni/retiarii/strategy/tpe_strategy.py>`.
After generating a new model, the strategy can use our provided APIs (e.g., :func:`nni.retiarii.execution.submit_models`, :func:`nni.retiarii.execution.is_stopped_exec`) to submit the model and get its reported results.
Customize a New One-shot Trainer (legacy)
-----------------------------------------
One-shot trainers should inherit :class:`nni.retiarii.oneshot.BaseOneShotTrainer`, and need to implement ``fit()`` (used to conduct the fitting and searching process) and ``export()`` method (used to return the searched best architecture).
Writing a one-shot trainer is very different to single-arch evaluator. First of all, there are no more restrictions on init method arguments, any Python arguments are acceptable. Secondly, the model fed into one-shot trainers might be a model with Retiarii-specific modules, such as LayerChoice and InputChoice. Such model cannot directly forward-propagate and trainers need to decide how to handle those modules.
A typical example is DartsTrainer, where learnable-parameters are used to combine multiple choices in LayerChoice. Retiarii provides ease-to-use utility functions for module-replace purposes, namely :meth:`nni.retiarii.oneshot.pytorch.utils.replace_layer_choice`, :meth:`nni.retiarii.oneshot.pytorch.utils.replace_input_choice`. A simplified example is as follows:
.. code-block:: python
from nni.retiarii.oneshot import BaseOneShotTrainer
from nni.retiarii.oneshot.pytorch.utils import replace_layer_choice, replace_input_choice
class DartsLayerChoice(nn.Module):
def __init__(self, layer_choice):
super(DartsLayerChoice, self).__init__()
self.name = layer_choice.label
self.op_choices = nn.ModuleDict(layer_choice.named_children())
self.alpha = nn.Parameter(torch.randn(len(self.op_choices)) * 1e-3)
def forward(self, *args, **kwargs):
op_results = torch.stack([op(*args, **kwargs) for op in self.op_choices.values()])
alpha_shape = [-1] + [1] * (len(op_results.size()) - 1)
return torch.sum(op_results * F.softmax(self.alpha, -1).view(*alpha_shape), 0)
class DartsTrainer(BaseOneShotTrainer):
def __init__(self, model, loss, metrics, optimizer):
self.model = model
self.loss = loss
self.metrics = metrics
self.num_epochs = 10
self.nas_modules = []
replace_layer_choice(self.model, DartsLayerChoice, self.nas_modules)
... # init dataloaders and optimizers
def fit(self):
for i in range(self.num_epochs):
for (trn_X, trn_y), (val_X, val_y) in zip(self.train_loader, self.valid_loader):
self.train_architecture(val_X, val_y)
self.train_model_weight(trn_X, trn_y)
@torch.no_grad()
def export(self):
result = dict()
for name, module in self.nas_modules:
if name not in result:
result[name] = select_best_of_module(module)
return result
The full code of DartsTrainer is available to Retiarii source code. Please have a check at :githublink:`DartsTrainer <nni/retiarii/oneshot/pytorch/darts.py>`.

77
docs/source/nas/evaluator.rst
Просмотреть файл

@ -8,15 +8,14 @@ A model evaluator is for training and validating each generated model. They are
Customize Evaluator with Any Function
-------------------------------------
The simplest way to customize a new evaluator is with :class:`~nni.retiarii.evaluator.FunctionalEvaluator`, which is very easy when training code is already available. Users only need to write a fit function that wraps everything, which usually includes training, validating and testing of a single model. This function takes one positional arguments (``model_cls``) and possible keyword arguments. The keyword arguments (other than ``model_cls``) are fed to :class:`~nni.retiarii.evaluator.FunctionalEvaluator` as its initialization parameters (note that they will be :doc:`serialized <./serialization>`). In this way, users get everything under their control, but expose less information to the framework and as a result, further optimizations like :ref:`CGO <cgo-execution-engine>` might be not feasible. An example is as belows:
The simplest way to customize a new evaluator is with :class:`~nni.nas.evaluator.FunctionalEvaluator`, which is very easy when training code is already available. Users only need to write a fit function that wraps everything, which usually includes training, validating and testing of a single model. This function takes one positional arguments (``model``) and possible keyword arguments. The keyword arguments (other than ``model``) are fed to :class:`~nni.nas.evaluator.FunctionalEvaluator` as its initialization parameters (note that they will be :doc:`serialized <./serialization>`). In this way, users get everything under their control, but expose less information to the framework and as a result, further optimizations like :ref:`CGO <cgo-execution-engine>` might be not feasible. An example is as belows:
.. code-block:: python
from nni.retiarii.evaluator import FunctionalEvaluator
from nni.retiarii.experiment.pytorch import RetiariiExperiment
from nni.nas.evaluator import FunctionalEvaluator
from nni.nas.experiment import NasExperiment
def fit(model_cls, dataloader):
model = model_cls()
def fit(model, dataloader):
train(model, dataloader)
acc = test(model, dataloader)
nni.report_final_result(acc)
@ -24,7 +23,11 @@ The simplest way to customize a new evaluator is with :class:`~nni.retiarii.eval
# The dataloader will be serialized, thus ``nni.trace`` is needed here.
# See serialization tutorial for more details.
evaluator = FunctionalEvaluator(fit, dataloader=nni.trace(DataLoader)(foo, bar))
experiment = RetiariiExperiment(base_model, evaluator, mutators, strategy)
experiment = NasExperiment(base_model, lightning, strategy)
.. note::
Different from the legacy Retiarii FunctionEvaluator, the new FunctionalEvaluator now accepts model instance as the first argument, rather than ``model_cls``. This makes it more intuitive and easier to use.
.. tip::
@ -32,8 +35,7 @@ The simplest way to customize a new evaluator is with :class:`~nni.retiarii.eval
.. code-block:: python
def fit(model_cls):
model = model_cls()
def fit(model):
onnx_path = Path(os.environ.get('NNI_OUTPUT_DIR', '.')) / 'model.onnx'
onnx_path.parent.mkdir(exist_ok=True)
dummy_input = torch.randn(10, 3, 224, 224)
@ -54,28 +56,35 @@ The usage is shown below:
.. code-block:: python
# Class definition of single model, for example, ResNet.
class SingleModel(nn.Module):
def __init__(): # Can't have init parameters here.
...
# Class definition of a model space, for example, ResNet.
class MyModelSpace(ModelSpace):
...
# Use a callable returning a model
evaluator.evaluate(SingleModel)
# Or initialize the model beforehand
evaluator.evaluate(SingleModel())
# Mock a model instance
from nni.nas.space import RawFormatModelSpace
model_container = RawFormatModelSpace.from_model(MyModelSpace())
The underlying implementation of :meth:`~nni.retiarii.Evaluator.evaluate` depends on concrete evaluator that you used.
For example, if :class:`~nni.retiarii.evaluator.FunctionalEvaluator` is used, it will run your customized fit function.
If lightning evaluators like :class:`nni.retiarii.evaluator.pytorch.Classification` are used, it will invoke the ``trainer.fit()`` of Lightning.
# Randomly sample a model
model = model_container.random()
To evaluate an architecture that is exported from experiment (i.e., from :meth:`~nni.retiarii.experiment.pytorch.RetiariiExperiment.export_top_models`), use :func:`nni.retiarii.fixed_arch` to instantiate the exported model::
# Mock a runtime so that `nni.get_next_parameter` and `nni.report_xxx_result` will work.
with evaluator.mock_runtime(model):
evaluator.evaluate(model.executable_model())
with fixed_arch(exported_model):
model = ModelSpace()
The underlying implementation of :meth:`~nni.nas.Evaluator.evaluate` depends on concrete evaluator that you used.
For example, if :class:`~nni.nas.evaluator.FunctionalEvaluator` is used, it will run your customized fit function.
If lightning evaluators like :class:`nni.nas.evaluator.pytorch.Classification` are used, it will invoke the ``trainer.fit()`` of Lightning.
To evaluate an architecture that is exported from experiment (i.e., from :meth:`~nni.nas.experiment.NasExperiment.export_top_models`), use :func:`nni.nas.space.model_context` to instantiate the exported model::
with model_context(exported_model_dict):
model = MyModelSpace()
# Then use evaluator.evaluate
evaluator.evaluate(model)
.. tip:: There is a way to port the trained checkpoint of super-net produced by one-shot strategies, to the concrete chosen architecture, thanks to :func:`nni.retiarii.utils.original_state_dict_hooks`. This is helpful in implementing recent multi-stage NAS algorithms like `SPOS <https://arxiv.org/abs/1904.00420>`__.
Another way of doing this is probably using ``freeze`` API. It will also preserve the weights at best effort if the model space has been mutated by one-shot strategies::
MyModelSpace().freeze(exported_model_dict)
.. _lightning-evaluator:
@ -85,21 +94,21 @@ Evaluators with PyTorch-Lightning
Use Built-in Evaluators
^^^^^^^^^^^^^^^^^^^^^^^
NNI provides some commonly used model evaluators for users' convenience. These evaluators are built upon the awesome library PyTorch-Lightning. Read the :doc:`reference </reference/nas/evaluator>` for their detailed usages.
NNI provides some commonly used model evaluators for users' convenience. These evaluators are built upon the awesome library PyTorch-Lightning. Read the :doc:`reference </reference/nas>` for their detailed usages.
* :class:`nni.retiarii.evaluator.pytorch.Classification`: for classification tasks.
* :class:`nni.retiarii.evaluator.pytorch.Regression`: for regression tasks.
* :class:`nni.nas.evaluator.pytorch.Classification`: for classification tasks.
* :class:`nni.nas.evaluator.pytorch.Regression`: for regression tasks.
We recommend to read the :doc:`serialization tutorial <serialization>` before using these evaluators. A few notes to summarize the tutorial:
1. :class:`nni.retiarii.evaluator.pytorch.DataLoader` should be used in place of ``torch.utils.data.DataLoader``.
1. :class:`nni.nas.evaluator.pytorch.DataLoader` should be used in place of ``torch.utils.data.DataLoader``.
2. The datasets used in data-loader should be decorated with :meth:`nni.trace` recursively.
For example,
.. code-block:: python
import nni.retiarii.evaluator.pytorch.lightning as pl
import nni.nas.evaluator.pytorch.lightning as pl
from torchvision import transforms
transform = nni.trace(transforms.Compose, [nni.trace(transforms.ToTensor()), nni.trace(transforms.Normalize, (0.1307,), (0.3081,))])
@ -116,13 +125,13 @@ Customize Evaluator with PyTorch-Lightning
Another approach is to write training code in PyTorch-Lightning style, that is, to write a LightningModule that defines all elements needed for training (e.g., loss function, optimizer) and to define a trainer that takes (optional) dataloaders to execute the training. Before that, please read the `document of PyTorch-lightning <https://pytorch-lightning.readthedocs.io/>`__ to learn the basic concepts and components provided by PyTorch-lightning.
In practice, writing a new training module in Retiarii should inherit :class:`nni.retiarii.evaluator.pytorch.LightningModule`, which has a ``set_model`` that will be called after ``__init__`` to save the candidate model (generated by strategy) as ``self.model``. The rest of the process (like ``training_step``) should be the same as writing any other lightning module. Evaluators should also communicate with strategies via two API calls (:meth:`nni.report_intermediate_result` for periodical metrics and :meth:`nni.report_final_result` for final metrics), added in ``on_validation_epoch_end`` and ``teardown`` respectively.
In practice, writing a new training module in nas should inherit :class:`nni.nas.evaluator.pytorch.LightningModule`, which has a ``set_model`` that will be called after ``__init__`` to save the candidate model (generated by strategy) as ``self.model``. The rest of the process (like ``training_step``) should be the same as writing any other lightning module. Evaluators should also communicate with strategies via two API calls (:meth:`nni.report_intermediate_result` for periodical metrics and :meth:`nni.report_final_result` for final metrics), added in ``on_validation_epoch_end`` and ``teardown`` respectively.
An example is as follows:
.. code-block:: python
from nni.retiarii.evaluator.pytorch.lightning import LightningModule # please import this one
from nni.nas.evaluator.pytorch.lightning import LightningModule # please import this one
@nni.trace
class AutoEncoder(LightningModule):
@ -173,15 +182,15 @@ An example is as follows:
If you are trying to use your customized evaluator with one-shot strategy, bear in mind that your defined methods will be reassembled into another LightningModule, which might result in extra constraints when writing the LightningModule. For example, your validation step could appear else where (e.g., in ``training_step``). This prohibits you from returning arbitrary object in ``validation_step``.
Then, users need to wrap everything (including LightningModule, trainer and dataloaders) into a :class:`nni.retiarii.evaluator.pytorch.Lightning` object, and pass this object into a Retiarii experiment.
Then, users need to wrap everything (including LightningModule, trainer and dataloaders) into a :class:`nni.nas.evaluator.pytorch.Lightning` object, and pass this object into a nas experiment.
.. code-block:: python
import nni.retiarii.evaluator.pytorch.lightning as pl
from nni.retiarii.experiment.pytorch import RetiariiExperiment
import nni.nas.evaluator.pytorch.lightning as pl
from nni.nas.experiment import NasExperiment
lightning = pl.Lightning(AutoEncoder(),
pl.Trainer(max_epochs=10),
train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
val_dataloaders=pl.DataLoader(test_dataset, batch_size=100))
experiment = RetiariiExperiment(base_model, lightning, mutators, strategy)
experiment = NasExperiment(base_model, lightning, strategy)

95
docs/source/nas/execution_engine.rst
Просмотреть файл

@ -1,110 +1,83 @@
Execution Engines
=================
Execution Engine and Model Format
=================================
Execution engine is for running Retiarii Experiment. NNI supports three execution engines, users can choose a specific engine according to the type of their model mutation definition and their requirements for cross-model optimizations.
After a model space and strategy has been prepared, NNI will be responsible for spawning trials and exploring the model space. Specifically, NNI will first convert the model space into a special **model format** so that it becomes easier to mutate / explore. Then the mutated models will be sent to **execution engines** for running.
* **Pure-python execution engine** is the default engine, it supports the model space expressed by :doc:`mutation primitives <construct_space>`.
We list the model formats currently supported.
* **Graph-based execution engine** supports the use of :doc:`mutation primitives <construct_space>` and model spaces represented by :doc:`mutators <mutator>`. It requires the user's model to be parsed by `TorchScript <https://pytorch.org/docs/stable/jit.html>`__.
* **Raw format**: It's default for one-shot strategy. It directly operates on the raw model space. This should be used when the model space is attached to a set of weights and the weights are intended to be kept / inherited.
* **Simplified format**: It's default for multi-trial strategy. It converts the model space to a dict of mutables, so that strategy will not need to mutate the dict when creating new models. This will be extremely memory efficient, but all weights and states on the model will be lost when instanting the new model.
* **Graph format**: Converting the model to a graph representation (called graph IR) using `TorchScript <https://pytorch.org/docs/stable/jit.html>`__. Each mutation on the model space will be reflected as an node/edge operation on the graph. See :doc:`mutation primitives <construct_space>` and :doc:`mutators <mutator>` for details.
* **CGO execution engine** has the same requirements and capabilities as the **Graph-based execution engine**. But further enables cross-model optimizations, which makes model space exploration faster.
NNI supports two execution engines, along with one execution engine middleware:
.. _pure-python-execution-engine:
* **Training service execution engine** is the default engine for multi-trial strategy. It will spawn the trials concurrently and the trials will run by :doc:`NNI training service </experiment/training_service/overview>`.
* **Sequential execution engine** is the default engine for one-shot strategy. It will run the trials in the current process. The trials will run sequentially without parallelism. It's also good for debugging multi-trial strategies.
* **Cross-graph optimization middleware** experimentally supports cross-model optimizations, which makes model space exploration faster. It is only compatible with graph format above.
Pure-python Execution Engine
----------------------------
Execution engine and model format are configurable via :class:`~nni.nas.experiment.NasExperimentConfig`. If not configured, it will be automatically inferred based on the settings of your model space and choice of exploration strategy. In most cases, the default setting should work well.
Pure-python Execution Engine is the default engine, we recommend users to keep using this execution engine, if they are new to NNI NAS. Pure-python execution engine plays magic within the scope of inline mutation APIs, while does not touch the rest of user model. Thus, it has minimal requirement on user model.
.. tip::
Rememeber to add :meth:`nni.retiarii.model_wrapper` decorator outside the whole PyTorch model before using this engine.
The legacy graph-based execution engine is identical to training service engine + graph format now.
.. note:: You should always use ``super().__init__()`` instead of ``super(MyNetwork, self).__init__()`` in the PyTorch model, because the latter one has issues with model wrapper.
Advanced Usages
---------------
.. _graph-based-execution-engine:
Graph-based Model Format
""""""""""""""""""""""""
Graph-based Execution Engine
----------------------------
Graph model format converts user-defined model to a graph representation (called graph IR) using `TorchScript <https://pytorch.org/docs/stable/jit.html>`__, each instantiated module in the model is converted to a subgraph. Then mutations are applied to the graph to generate new graphs. Each new graph is then converted back to PyTorch code and executed on the user specified training service.
For graph-based execution engine, it converts user-defined model to a graph representation (called graph IR) using `TorchScript <https://pytorch.org/docs/stable/jit.html>`__, each instantiated module in the model is converted to a subgraph. Then mutations are applied to the graph to generate new graphs. Each new graph is then converted back to PyTorch code and executed on the user specified training service.
Users may find ``@basic_unit`` helpful in some cases. :meth:`nni.retiarii.basic_unit` here means the module will not be converted to a subgraph, instead, it is converted to a single graph node as a basic unit.
``@basic_unit`` is usually used in the following cases:
* When users want to tune initialization parameters of a module using :class:`nni.retiarii.nn.pytorch.ValueChoice`, then decorate the module with ``@basic_unit``. For example, ``self.conv = MyConv(kernel_size=nn.ValueChoice([1, 3, 5]))``, here ``MyConv`` should be decorated.
* When a module cannot be successfully parsed to a subgraph, decorate the module with ``@basic_unit``. The parse failure could be due to complex control flow. Currently Retiarii does not support adhoc loop, if there is adhoc loop in a module's forward, this class should be decorated as serializable module. For example, the following ``MyModule`` should be decorated.
Users may find ``_nni_basic_unit`` of :class:`~nni.nas.nn.pytorch.ParameterizedModule` helpful in some cases. ``_nni_basic_unit`` here means the module will not be converted to a subgraph, instead, it is converted to a single graph node as a basic unit. When a module cannot be successfully parsed to a subgraph, please inherit :class:`~nni.nas.nn.pytorch.ParameterizedModule`, which will automatically enable ``_nni_basic_unit``. The parse failure could be due to complex control flow. Currently Retiarii does not support adhoc loop, if there is adhoc loop in a module's forward, this class should be decorated as serializable module. For example, the following ``MyModule`` should be a basic unit.
.. code-block:: python
@basic_unit
class MyModule(nn.Module):
class MyModule(ParameterizedModule):
def __init__(self):
...
def forward(self, x):
for i in range(10): # <- adhoc loop
...
* Some inline mutation APIs require their handled module to be decorated with ``@basic_unit``. For example, user-defined module that is provided to :class:`nni.retiarii.nn.pytorch.LayerChoice` as a candidate op should be decorated.
Three steps are need to use graph-based execution engine.
1. Remove ``@nni.retiarii.model_wrapper`` if there is any in your model.
2. Add ``config.execution_engine = 'base'`` to :class:`nni.retiarii.experiment.pytorch.RetiariiExeConfig`. The default value of ``execution_engine`` is 'py', which means pure-python execution engine.
3. Add ``@basic_unit`` when necessary following the above guidelines.
* Some inline mutation APIs require their handled module to be a basic unit. For example, user-defined module that is provided to :class:`~nni.nas.nn.pytorch.LayerChoice` as a candidate op should be basic units.
For exporting top models, graph-based execution engine supports exporting source code for top models by running ``exp.export_top_models(formatter='code')``.
.. _cgo-execution-engine:
CGO Execution Engine (experimental)
-----------------------------------
Cross Graph Optimization (experimental)
"""""""""""""""""""""""""""""""""""""""
CGO (Cross-Graph Optimization) execution engine does cross-model optimizations based on the graph-based execution engine. In CGO execution engine, multiple models could be merged and trained together in one trial.
Currently, it only supports ``DedupInputOptimizer`` that can merge graphs sharing the same dataset to only loading and pre-processing each batch of data once, which can avoid bottleneck on data loading.
CGO (Cross-Graph Optimization) middleware does cross-model optimizations based on the graph-based model format. With CGO, multiple models could be merged and trained together in one trial. Currently, it only supports ``DedupInputOptimizer`` that can merge graphs sharing the same dataset to only loading and pre-processing each batch of data once, which can avoid bottleneck on data loading.
.. note :: To use CGO engine, PyTorch Lightning >= 1.6.1 is required.
To enable CGO execution engine, you need to follow these steps:
1. Create RetiariiExeConfig with remote training service. CGO execution engine currently only supports remote training service.
2. Add configurations for remote training service
3. Add configurations for CGO engine
1. Use training service engine.
2. Set training service to remote training service. CGO middleware currently only supports remote training service.
.. code-block:: python
exp = RetiariiExperiment(base_model, trainer, mutators, strategy)
config = RetiariiExeConfig('remote')
exp = NasExperiment(base_model, evaluator, strategy, config=NasExperimentConfig('cgo', 'graph', 'remote'))
# ...
# other configurations of RetiariiExeConfig
# other configurations of NasExperimentConfig
config.execution_engine = 'cgo' # set execution engine to CGO
config.max_concurrency_cgo = 3 # the maximum number of concurrent models to merge
config.batch_waiting_time = 10 # how many seconds CGO execution engine should wait before optimizing a new batch of models
config.batch_waiting_time = 10 # how many seconds CGO should wait before optimizing a new batch of models
rm_conf = RemoteMachineConfig()
# ...
# server configuration in rm_conf
rm_conf.gpu_indices = [0, 1, 2, 3] # gpu_indices must be set in RemoteMachineConfig for CGO execution engine
rm_conf.gpu_indices = [0, 1, 2, 3] # gpu_indices must be set in RemoteMachineConfig for CGO
config.training_service.machine_list = [rm_conf]
exp.run(config, 8099)
CGO Execution Engine only supports pytorch-lightning trainer that inherits :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.MultiModelSupervisedLearningModule`.
For a trial running multiple models, the trainers inheriting :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.MultiModelSupervisedLearningModule` can handle the multiple outputs from the merged model for training, test and validation.
We have already implemented two trainers: :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.Classification` and :class:`nni.retiarii.evaluator.pytorch.cgo.evaluator.Regression`.
.. code-block:: python
from nni.retiarii.evaluator.pytorch.cgo.evaluator import Classification
trainer = Classification(train_dataloaders=pl.DataLoader(train_dataset, batch_size=100),
val_dataloaders=pl.DataLoader(test_dataset, batch_size=100),
max_epochs=1, limit_train_batches=0.2)
Advanced users can also implement their own trainers by inheriting ``MultiModelSupervisedLearningModule``.
CGO middleware only supports pytorch-lightning trainer that inherits :class:`~nni.nas.execution.cgo.MultiModelLightningModule`.
For a trial running multiple models, the trainers inheriting :class:`~nni.nas.execution.cgo.MultiModelTrainer` can handle the multiple outputs from the merged model for training, test and validation.
Sometimes, a mutated model cannot be executed (e.g., due to shape mismatch). When a trial running multiple models contains
a bad model, CGO execution engine will re-run each model independently in separate trials without cross-model optimizations.
a bad model, CGO will re-run each model independently in separate trials without cross-model optimizations.

4
docs/source/nas/exploration_strategy.rst
Просмотреть файл

@ -88,8 +88,8 @@ Starting from v2.8, the usage of one-shot strategies are much alike to multi-tri
exp_config.execution_engine = 'oneshot'
One-shot strategies only support a limited set of :ref:`mutation-primitives`, and does not support :doc:`customizing mutators <mutator>` at all. See the :ref:`reference <one-shot-strategy-reference>` for the detailed support list of each algorithm.
One-shot strategies only support a limited set of mutation primitives, and does not support :doc:`customizing mutators <mutator>` at all. See the :doc:`reference </reference/nas>` for the detailed support list of each algorithm.
.. versionadded:: 2.8
One-shot strategy is now compatible with `Lightning accelerators <https://pytorch-lightning.readthedocs.io/en/stable/accelerators/gpu.html>`__. It means that, you can accelerate one-shot strategies on hardwares like multiple GPUs. To enable this feature, you only need to pass the keyword arguments which used to be set in ``pytorch_lightning.Trainer``, to your evaluator. See :doc:`this reference </reference/nas/evaluator>` for more details.
One-shot strategy is now compatible with `Lightning accelerators <https://pytorch-lightning.readthedocs.io/en/stable/accelerators/gpu.html>`__. It means that, you can accelerate one-shot strategies on hardwares like multiple GPUs. To enable this feature, you only need to pass the keyword arguments which used to be set in ``pytorch_lightning.Trainer``, to your evaluator. See :doc:`this reference </reference/nas>` for more details.

139
docs/source/nas/hardware_aware_nas.rst
Просмотреть файл

@ -1,83 +1,88 @@
Hardware-aware NAS
==================
.. This file should be rewritten as a tutorial
Hardware-aware NAS is a technique to search for architectures under the constraints of a given hardware platform. Deploying a neural network on a specific hardware is challenging because different hardwares have different preferences for operations, combinations of operations. Different use scenarios might also pose different levels of requirements. Some strict scenarios might require the networks to be smaller and faster. Our hardware-aware NAS supports searching for architectures under the constraints of latency, model size or FLOPs. This document demonstrates how to use multi-trial strategy or one-shot strategy combining those constraints.
End-to-end Multi-trial SPOS Demo
Profiler
--------
:class:`~nni.nas.profiler.Profiler` is designed to efficiently compute metrics like latency for models within the same model space. Specifically, it is first initialized with a model space, in which it precomputes some data, and for any sample in the model space, it can quickly give out a metric::
class MyModelSpace(ModelSpace):
...
from nni.nas.profiler.pytorch.flops import FlopsProfiler
# initialization
profiler = FlopsProfiler(net, torch.randn(3)) # randn(3) is a dummy input. It could be a tensor or a tuple of tensors.
# compute flops for a sample
flops = profiler.profile({'layer1': 'conv'})
NNI currently supports the following types of profilers:
.. list-table::
:header-rows: 1
:widths: auto
* - Name
- Brief Description
* - :class:`~nni.nas.profiler.pytorch.flops.FlopsProfiler`
- Profile the FLOPs of a model
* - :class:`~nni.nas.profiler.pytorch.flops.NumParamsProfiler`
- Profile the number of parameters of a model
* - :class:`~nni.nas.profiler.pytorch.nn_meter.NnMeterProfiler`
- Profile the estimated latency of a model with nn-meter
Hardware-aware multi-trial search
---------------------------------
When using multi-trial strategies, the most intuitive approach to combine a hardware-aware constraint is to filter out those outside the constraints. This can be done via strategy middleware :class:`~nni.nas.strategy.middleware.Filter`. An example is as follows::
from nni.nas.strategy import Random
from nni.nas.strategy.middleware import Filter, Chain
from nni.nas.profiler.pytorch.flops import FlopsProfiler
profiler = FlopsProfiler(model_space, dummy_input)
strategy = Chain(
Random(),
Filter(lambda sample: profiler.profile(sample) < 300e6)
)
The example here uses a random strategy to randomly generate models, but the FLOPs is restricted to less than 300M. Models over the limitation will be directly discarded without training. :class:`~nni.nas.strategy.middleware.Filter` can be also set to a mode to return a bad metric for models out of constraint, so that the strategy can learn to avoid sampling such models. The profiler can also be replaced with any profiler listed above or customized.
Hardware-aware one-shot strategy
--------------------------------
To empower affordable DNN on the edge and mobile devices, hardware-aware NAS searches both high accuracy and low latency models. In particular, the search algorithm only considers the models within the target latency constraints during the search process.
One-shot strategy can be also combined with metrics on hardware. There are usually two approaches.
To run this demo, first install nn-Meter by running:
The first approach is to add a special regularization term to the loss within one-shot strategy, penalizing the sampling of models that does not fit the constraints. To do this, a penalty term (either :class:`~nni.nas.oneshot.pytorch.profiler.ExpectationProfilerPenalty` for differentiable algorithms, or :class:`~nni.nas.oneshot.pytorch.profiler.SampleProfilerPenalty` for sampling-based algorithms). Example below::
.. code-block:: bash
from nni.nas.strategy import ProxylessNAS
from nni.nas.oneshot.pytorch.profiler import ExpectationProfilerPenalty
# For sampling-based algorithms like ENAS, use `SampleProfilerPenalty` here.
# Please see the document for each algorithm for the type of penalties they have supported.
pip install nn-meter
profiler = FlopsProfiler(model_space, dummy_input)
penalty = ExpectationProfilerPenalty(profiler, 300e6) # 300M is the expected profiler here. Exceeding it will be penalized.
strategy = ProxylessNAS(penalty=penalty)
Then run multi-trail SPOS demo:
Another approach is similar to what we've done for multi-trial strategies: to directly prevent models out of constraints from being sampled. To do this, use :class:`~nni.nas.oneshot.pytorch.profiler.RangeProfilerFilter`. Example::
.. code-block:: bash
from nni.nas.strategy import ENAS
from nni.nas.oneshot.pytorch.profiler import RangeProfilerFilter
cd ${NNI_ROOT}/examples/nas/oneshot/spos/
python search.py --latency-filter cortexA76cpu_tflite21
profiler = FlopsProfiler(model_space, dummy_input)
penalty = RangeProfilerFilter(profiler, 200e6, 300e6) # Only flops between 200M and 300M are considered legal.
strategy = ENAS(filter=filter)
.. tip:: The penalty and filter here are specialized for one-shot strategies, please do not use them in multi-trial strategies.
How the demo works
^^^^^^^^^^^^^^^^^^
Best practices of hardware-aware NAS
------------------------------------
To support hardware-aware NAS, you first need a ``Strategy`` that supports filtering the models by latency. We provide such a filter named ``LatencyFilter`` in NNI and initialize a ``RegularizedEvolution`` strategy with the filter:
The hardware-aware part in NAS is probably the most complex component within the whole NNI NAS framework. It's expected that users might encounter technical issues when using hardware-aware NAS. A full troubleshooting guide is still under preparation. For now, we recommend the following practices, briefly.
.. code-block:: python
evolution_strategy = strategy.RegularizedEvolution(
model_filter=latency_filter,
sample_size=args.evolution_sample_size, population_size=args.evolution_population_size, cycles=args.evolution_cycles
)
``LatencyFilter`` will predict the models\' latency by using nn-Meter and filter out the models whose latency are larger than the threshold (i.e., ``100`` in this example).
You can also build your own strategies and filters to support more flexible NAS such as sorting the models according to latency.
Then, pass this strategy to ``RetiariiExperiment``:
.. code-block:: python
exp = RetiariiExperiment(base_model, evaluator, strategy=evolution_strategy)
exp_config = RetiariiExeConfig('local')
...
exp_config.dummy_input = [1, 3, 224, 224]
exp.run(exp_config, args.port)
In ``exp_config``, ``dummy_input`` is required for tracing shape info in latency predictor.
End-to-end ProxylessNAS with Latency Constraints
------------------------------------------------
`ProxylessNAS <https://arxiv.org/abs/1812.00332>`__ is a hardware-aware one-shot NAS algorithm. ProxylessNAS applies the expected latency of the model to build a differentiable metric and design efficient neural network architectures for hardware. The latency loss is added as a regularization term for architecture parameter optimization. In this example, nn-Meter provides a latency estimator to predict expected latency for the mixed operation on other types of mobile and edge hardware.
To run the one-shot ProxylessNAS demo, first install nn-Meter by running:
.. code-block:: bash
pip install nn-meter
Then run one-shot ProxylessNAS demo:
.. code-block:: bash
python ${NNI_ROOT}/examples/nas/oneshot/proxylessnas/main.py --applied_hardware HARDWARE --reference_latency REFERENCE_LATENCY_MS
How the demo works
^^^^^^^^^^^^^^^^^^
In the implementation of ProxylessNAS ``trainer``, we provide a ``HardwareLatencyEstimator`` which currently builds a lookup table, that stores the measured latency of each candidate building block in the search space. The latency sum of all building blocks in a candidate model will be treated as the model inference latency. The latency prediction is obtained by ``nn-Meter``. ``HardwareLatencyEstimator`` predicts expected latency for the mixed operation based on the path weight of ``ProxylessLayerChoice``. With leveraging ``nn-Meter`` in NNI, users can apply ProxylessNAS to search efficient DNN models on more types of edge devices.
Despite of ``applied_hardware`` and ``reference_latency``, There are some other parameters related to hardware-aware ProxylessNAS training in this :githublink:`example <examples/nas/oneshot/proxylessnas/main.py>`:
* ``grad_reg_loss_type``: Regularization type to add hardware related loss. Allowed types include ``"mul#log"`` and ``"add#linear"``. Type of ``mul#log`` is calculate by ``(torch.log(expected_latency) / math.log(reference_latency)) ** beta``. Type of ``"add#linear"`` is calculate by ``reg_lambda * (expected_latency - reference_latency) / reference_latency``.
* ``grad_reg_loss_lambda``: Regularization params, is set to ``0.1`` by default.
* ``grad_reg_loss_alpha``: Regularization params, is set to ``0.2`` by default.
* ``grad_reg_loss_beta``: Regularization params, is set to ``0.3`` by default.
* ``dummy_input``: The dummy input shape when applied to the target hardware. This parameter is set as (1, 3, 224, 224) by default.
1. Make sure shape inference succeeds. In order to make profiler to work, we will dry run the model space and infer a symbolic shape for inputs and outputs of every submodule. Built-in implementations only support a limited set of operations when inferencing shapes. If errors like ``no shape formula``, please register the shape formula following the prompt, or decorate the whole module as a leaf module that doesn't need to be opened. Note that if the shape inference doesn't open a module, its FLOPs and latency might also need to compute as a whole. You might also need to write FLOPs / latency formula for the module.
2. Try with FLOPs first. In our experience, complex profilers like nn-Meter might make it harder to debug when something goes wrong. Remember to examine whether the FLOPs profiler returns a reasonable result. This can be done by manually invoking ``profiler.profile(sample)``.
3. :class:`~nni.nas.profiler.pytorch.nn_meter.NnMeterProfiler` will expand all the possible modules when it considers a module space as a leaf module (note that nn-meter has its own leaf module settings and do not follow what has been set for shape inference). If the submodule contains too many combinations. The profiler might hang when preprocessing. Try using ``logging.getLogger('nni').setLevel(logging.DEBUG)`` to print debug logs, so as to identify the cause of the issue.
4. For a specific model space and a specific hardware, you can also build your own profiler with :class:`~nni.nas.profiler.Profiler`. As long as they follow the interface of :class:`~nni.nas.profiler.Profiler`, the inner implementation doesn't matter. Users can use lookup tables, build predictors, or even connecting to the real device for profiling. If the interface is compatible, it's possible to use it combining our built-in strategies. It's usually the recommended method when your model space is too complex for the general shape inference to work, or you are targetting at the specialized hardware we do not yet support.

6
docs/source/nas/mutator.rst
Просмотреть файл

@ -1,5 +1,7 @@
Construct Space with Mutator
============================
Construct Space with Mutator (legacy)
=====================================
.. attention:: This is a legacy document for NNI v2.x. This is now currently no longer maintained.
Besides the mutation primitives demonstrated in the :doc:`basic tutorial <construct_space>`, NNI provides a more general approach to express a model space, i.e., *Mutator*, to cover more complex model spaces. The high-level APIs are also implemented with mutator in the underlying system, which can be seen as a special case of model mutation.

89
docs/source/nas/overview_zh.rst
Просмотреть файл

@ -1,89 +0,0 @@
.. 48c39585a539a877461aadef63078c48
神经架构搜索
===========================
.. toctree::
:hidden:
快速入门 </tutorials/hello_nas>
构建搜索空间 <construct_space>
探索策略 <exploration_strategy>
评估器 <evaluator>
高级用法 <advanced_usage>
.. attention:: NNI 最新的架构搜索支持都是基于 Retiarii 框架，还在使用 `NNI 架构搜索的早期版本 <https://nni.readthedocs.io/en/v2.2/nas.html>`__ 的用户应尽快将您的工作迁移到 Retiarii。我们计划在接下来的几个版本中删除旧的架构搜索框架。
.. attention:: PyTorch 是 **Retiarii 唯一支持的框架**。有关 Tensorflow 上架构搜索支持的需求在 `此讨论 <https://github.com/microsoft/nni/discussions/4605>`__ 中。另外，如果您打算使用 PyTorch 和 Tensorflow 以外的 DL 框架运行 NAS，请 `创建新 issue <https://github.com/microsoft/nni/issues>`__ 让我们知道。
概述
------
自动神经架构搜索 (Neural Architecture Search, NAS）在寻找更好的模型方面发挥着越来越重要的作用。最近的研究证明了自动架构搜索的可行性，并导致模型击败了许多手动设计和调整的模型。其中具有代表性的有 `NASNet <https://arxiv.org/abs/1707.07012>`__、 `ENAS <https://arxiv.org/abs/1802.03268>`__、 `DARTS <https://arxiv.org/ abs/1806.09055>`__、 `Network Morphism <https://arxiv.org/abs/1806.10282>`__ 和 `进化算法 <https://arxiv.org/abs/1703.01041>`__。此外，新的创新正不断涌现。
总的来说，使用神经架构搜索解决任何特定任务通常需要：搜索空间设计、搜索策略选择和性能评估。这三个组件形成如下的循环（图来自于 `架构搜索综述 <https://arxiv.org/abs/1808.05377>`__）：
.. image:: ../../img/nas_abstract_illustration.png
:align: center
:width: 700
在这个图中：
* *模型搜索空间* 是指一组模型，从中探索/搜索最佳模型，简称为 *搜索空间* 或 *模型空间*。
* *探索策略* 是用于探索模型搜索空间的算法。有时我们也称它为 *搜索策略*。
* *模型评估者* 负责训练模型并评估其性能。
该过程类似于 :doc:`超参数优化 </hpo/overview>`，只不过目标是最佳网络结构而不是最优超参数。具体来说，探索策略从预定义的搜索空间中选择架构。该架构被传递给性能评估以获得评分，该评分表示这个网络结构在特定任务上的表现。重复此过程，直到搜索过程能够找到最优的网络结构。
主要特点
------------
NNI 中当前的架构搜索框架由 `Retiarii: A Deep Learning Exploratory-Training Framework <https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf>`__ 的研究支撑，具有以下特点：
* :doc:`简单的 API，让您轻松构建搜索空间 <construct_space>`
* :doc:`SOTA 架构搜索算法，以高效探索搜索空间 <exploration_strategy>`
* :doc:`后端支持，在大规模 AI 平台上运行实验 </experiment/overview>`
为什么使用 NNI 的架构搜索
-------------------------------
若没有 NNI，实现架构搜索将极具挑战性，主要包含以下三个方面。当用户想在自己的场景中尝试架构搜索技术时，NNI 提供的解决方案可以极大程度上减轻用户的工作量。
搜索空间设计
^^^^^^^^^^^^^^^^^^^
搜索空间定义了架构的可行域集合。为了简化搜索，我们通常需要结合任务相关的先验知识，减小搜索空间的规模。然而，这也引入了人类的偏见，在某种程度上可能会丧失突破人类认知的可能性。无论如何，对于初学者来说，搜索空间设计是一个极具挑战性的任务，因为他们可能无法在简单的空间和丰富的想象力之间取得平衡。
在 NNI 中，我们提供了不同层级的 API 来构建搜索空间。有 :doc:`高层 API <construct_space>`，引入大量先验，帮助用户迅速了解什么是好的架构或搜索空间；也有 :doc:`底层 API <mutator>`，提供了最底层的算子和图变换原语。
探索策略
^^^^^^^^^^^^^^^^^^^^
探索策略定义了如何探索搜索空间（通常是指数级规模的）。它包含经典的探索-利用权衡。一方面，我们希望快速找到性能良好的架构；而另一方面，我们也应避免过早收敛到次优架构的区域。我们往往需要通常通过反复试验找到特定场景的“最佳”探索策略。由于许多近期发表的探索策略都是使用自己的代码库实现的，因此从一个切换到另一个变得非常麻烦。
在 NNI 中，我们还提供了 :doc:`一系列的探索策略 <exploration_strategy>`。其中一些功能强大但耗时，而另一些可能不能找到最优架构但非常高效。鉴于所有策略都使用统一的用户接口实现，用户可以轻松找到符合他们需求的策略。
性能评估
^^^^^^^^^^^^^^^^^^^^^^
架构搜索的目标通常是找到能够在测试数据集表现理想的网络结构。性能评估的作用便是量化每个网络的好坏。其主要难点在于可扩展性，即如何在大规模训练平台上同时运行和管理多个试验。
在 NNI 中，我们使用 :doc:`evaluator <evaluator>` 来标准化性能评估流程。它负责估计模型的性能。NNI 内建了不少性能评估器，从最简单的交叉验证，到复杂的自定义配置。评估器在 *试验 (trials)* 中运行，可以通过我们强大的 :doc:`训练平台 </experiment/training_service/overview>` 将试验分发到大规模训练平台上。
教程
---------
要开始使用 NNI 架构搜索框架，我们建议至少阅读以下教程：
* :doc:`快速入门 </tutorials/hello_nas>`
* :doc:`构建搜索空间 <construct_space>`
* :doc:`探索策略 <exploration_strategy>`
* :doc:`评估器 <evaluator>`
资源
---------
以下文章将有助于更好地了解 NAS 的最新发展：
* `神经架构搜索：综述 <https://arxiv.org/abs/1808.05377>`__
* `神经架构搜索的综述：挑战和解决方案 <https://arxiv.org/abs/2006.02903>`__

36
docs/source/nas/space_hub.rst
Просмотреть файл

@ -19,27 +19,27 @@ The model spaces provided so far are all built for image classification tasks, t
* - Name
- Brief Description
* - :class:`~nni.retiarii.hub.pytorch.NasBench101`
* - :class:`~nni.nas.hub.pytorch.NasBench101`
- Search space benchmarked by `NAS-Bench-101 <http://proceedings.mlr.press/v97/ying19a/ying19a.pdf>`__
* - :class:`~nni.retiarii.hub.pytorch.NasBench201`
* - :class:`~nni.nas.hub.pytorch.NasBench201`
- Search space benchmarked by `NAS-Bench-201 <https://arxiv.org/abs/2001.00326>`__
* - :class:`~nni.retiarii.hub.pytorch.NASNet`
* - :class:`~nni.nas.hub.pytorch.NASNet`
- Proposed by `Learning Transferable Architectures for Scalable Image Recognition <https://arxiv.org/abs/1707.07012>`__
* - :class:`~nni.retiarii.hub.pytorch.ENAS`
* - :class:`~nni.nas.hub.pytorch.ENAS`
- Proposed by `Efficient neural architecture search via parameter sharing <https://arxiv.org/abs/1802.03268>`__, subtly different from NASNet
* - :class:`~nni.retiarii.hub.pytorch.AmoebaNet`
* - :class:`~nni.nas.hub.pytorch.AmoebaNet`
- Proposed by `Regularized evolution for image classifier architecture search <https://arxiv.org/abs/1802.01548>`__, subtly different from NASNet
* - :class:`~nni.retiarii.hub.pytorch.PNAS`
* - :class:`~nni.nas.hub.pytorch.PNAS`
- Proposed by `Progressive neural architecture search <https://arxiv.org/abs/1712.00559>`__, subtly different from NASNet
* - :class:`~nni.retiarii.hub.pytorch.DARTS`
* - :class:`~nni.nas.hub.pytorch.DARTS`
- Proposed by `Darts: Differentiable architecture search <https://arxiv.org/abs/1806.09055>`__, most popularly used in evaluating one-shot algorithms
* - :class:`~nni.retiarii.hub.pytorch.ProxylessNAS`
* - :class:`~nni.nas.hub.pytorch.ProxylessNAS`
- Proposed by `ProxylessNAS <https://arxiv.org/abs/1812.00332>`__, based on MobileNetV2.
* - :class:`~nni.retiarii.hub.pytorch.MobileNetV3Space`
* - :class:`~nni.nas.hub.pytorch.MobileNetV3Space`
- The largest space in `TuNAS <https://arxiv.org/abs/2008.06120>`__.
* - :class:`~nni.retiarii.hub.pytorch.ShuffleNetSpace`
* - :class:`~nni.nas.hub.pytorch.ShuffleNetSpace`
- Based on ShuffleNetV2, proposed by `Single Path One-shot <https://www.ecva.net/papers/eccv_2020/papers_ECCV/papers/123610528.pdf>`__
* - :class:`~nni.retiarii.hub.pytorch.AutoformerSpace`
* - :class:`~nni.nas.hub.pytorch.AutoFormer`
- Based on ViT, proposed by `Autoformer <https://arxiv.org/abs/2107.00651>`__
.. note::
@ -59,7 +59,7 @@ One way to use the model space is to directly leverage the searched results. Not
.. code-block:: python
import torch
from nni.retiarii.hub.pytorch import MobileNetV3Space
from nni.nas.hub.pytorch import MobileNetV3Space
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.datasets import ImageNet
@ -150,20 +150,20 @@ Here is a short sample code snippet for reference.
.. code-block:: python
# Create the model space
from nni.retiarii.hub.pytorch import MobileNetV3Space
from nni.nas.hub.pytorch import MobileNetV3Space
model_space = MobileNetV3Space()
# Pick a search strategy
from nni.retiarii.strategy import Evolution
strategy = Evolution() # It can be any strategy, including one-shot strategies.
from nni.nas.strategy import RegularizedEvolution
strategy = RegularizedEvolution() # It can be any strategy, including one-shot strategies.
# Define an evaluator
from nni.retiarii.evaluator.pytorch import Classification
from nni.nas.evaluator.pytorch import Classification
evaluator = Classification(train_dataloaders=DataLoader(train_dataset, batch_size=batch_size),
val_dataloaders=DataLoader(test_dataset, batch_size=batch_size))
# Launch the experiment, start the search process
experiment = RetiariiExperiment(model_space, evaluator, [], strategy)
experiment.run(experiment_config)
experiment = NasExperiment(model_space, evaluator, strategy)
experiment.run()
.. todo: search reproduction results

341
docs/source/reference/nas.rst Normal file
Просмотреть файл

@ -0,0 +1,341 @@
NAS API Reference
=================
Model space
-----------
.. autoclass:: nni.nas.nn.pytorch.LayerChoice
:members:
.. autoclass:: nni.nas.nn.pytorch.InputChoice
:members:
.. autoclass:: nni.nas.nn.pytorch.Repeat
:members:
.. autoclass:: nni.nas.nn.pytorch.Cell
:members:
.. autoclass:: nni.nas.nn.pytorch.ModelSpace
:members:
.. autoclass:: nni.nas.nn.pytorch.ParametrizedModule
:members:
.. autoclass:: nni.nas.nn.pytorch.MutableModule
:members:
Model Space Hub
---------------
NasBench101
^^^^^^^^^^^
.. autoclass:: nni.nas.hub.pytorch.NasBench101
:members:
NasBench201
^^^^^^^^^^^
.. autoclass:: nni.nas.hub.pytorch.NasBench201
:members:
NASNet
^^^^^^
.. autoclass:: nni.nas.hub.pytorch.NASNet
:members:
.. autoclass:: nni.nas.hub.pytorch.nasnet.NDS
:members:
.. autoclass:: nni.nas.hub.pytorch.nasnet.NDSStage
:members:
.. autoclass:: nni.nas.hub.pytorch.nasnet.NDSStagePathSampling
:members:
.. autoclass:: nni.nas.hub.pytorch.nasnet.NDSStageDifferentiable
:members:
ENAS
^^^^
.. autoclass:: nni.nas.hub.pytorch.ENAS
:members:
AmoebaNet
^^^^^^^^^
.. autoclass:: nni.nas.hub.pytorch.AmoebaNet
:members:
PNAS
^^^^
.. autoclass:: nni.nas.hub.pytorch.PNAS
:members:
DARTS
^^^^^
.. autoclass:: nni.nas.hub.pytorch.DARTS
:members:
ProxylessNAS
^^^^^^^^^^^^
.. autoclass:: nni.nas.hub.pytorch.ProxylessNAS
:members:
.. autoclass:: nni.nas.hub.pytorch.proxylessnas.InvertedResidual
:members:
MobileNetV3Space
^^^^^^^^^^^^^^^^
.. autoclass:: nni.nas.hub.pytorch.MobileNetV3Space
:members:
ShuffleNetSpace
^^^^^^^^^^^^^^^
.. autoclass:: nni.nas.hub.pytorch.ShuffleNetSpace
:members:
AutoFormer
^^^^^^^^^^
.. autoclass:: nni.nas.hub.pytorch.AutoFormer
:members:
Module Components
^^^^^^^^^^^^^^^^^
.. automodule:: nni.nas.hub.pytorch.modules
:members:
:imported-members:
Evaluator
---------
.. autoclass:: nni.nas.evaluator.FunctionalEvaluator
:members:
.. autoclass:: nni.nas.evaluator.Evaluator
:members:
.. autoclass:: nni.nas.evaluator.MutableEvaluator
:members:
.. autoclass:: nni.nas.evaluator.pytorch.Classification
:members:
.. autoclass:: nni.nas.evaluator.pytorch.ClassificationModule
:members:
.. autoclass:: nni.nas.evaluator.pytorch.Regression
:members:
.. autoclass:: nni.nas.evaluator.pytorch.RegressionModule
:members:
.. autoclass:: nni.nas.evaluator.pytorch.Trainer
.. autoclass:: nni.nas.evaluator.pytorch.DataLoader
.. autoclass:: nni.nas.evaluator.pytorch.Lightning
:members:
.. autoclass:: nni.nas.evaluator.pytorch.LightningModule
:members:
Multi-trial strategy
--------------------
.. autoclass:: nni.nas.strategy.GridSearch
:members:
.. autoclass:: nni.nas.strategy.Random
:members:
.. autoclass:: nni.nas.strategy.RegularizedEvolution
:members:
.. autoclass:: nni.nas.strategy.PolicyBasedRL
:members:
.. autoclass:: nni.nas.strategy.TPE
:members:
Advanced APIs
^^^^^^^^^^^^^
Base
""""
.. automodule:: nni.nas.strategy.base
:members:
Middleware
""""""""""
.. automodule:: nni.nas.strategy.middleware
:members:
Utilities
"""""""""
.. automodule:: nni.nas.strategy.utils
:members:
One-shot strategies
-------------------
.. autoclass:: nni.nas.strategy.RandomOneShot
:members:
.. autoclass:: nni.nas.strategy.ENAS
:members:
.. autoclass:: nni.nas.strategy.DARTS
:members:
.. autoclass:: nni.nas.strategy.GumbelDARTS
:members:
.. autoclass:: nni.nas.strategy.Proxyless
:members:
Advanced APIs
^^^^^^^^^^^^^
.. autoclass:: nni.nas.oneshot.pytorch.strategy.OneShotStrategy
:members:
base_lightning
""""""""""""""
.. automodule:: nni.nas.oneshot.pytorch.base_lightning
:members:
:imported-members:
supermodule.differentiable
""""""""""""""""""""""""""
.. automodule:: nni.nas.oneshot.pytorch.supermodule.differentiable
:members:
:imported-members:
supermodule.sampling
""""""""""""""""""""
.. automodule:: nni.nas.oneshot.pytorch.supermodule.sampling
:members:
:imported-members:
supermodule.proxyless
"""""""""""""""""""""
.. automodule:: nni.nas.oneshot.pytorch.supermodule.proxyless
:members:
:imported-members:
supermodule.operation
"""""""""""""""""""""
.. automodule:: nni.nas.oneshot.pytorch.supermodule.operation
:members:
:imported-members:
Profiler Utilities
""""""""""""""""""
.. automodule:: nni.nas.oneshot.pytorch.profiler
:members:
Experiment
----------
.. autoclass:: nni.nas.experiment.NasExperiment
:members:
.. automodule:: nni.nas.experiment.config
:members:
:imported-members:
Profiler
--------
.. autoclass:: nni.nas.profiler.Profiler
:members:
.. autoclass:: nni.nas.profiler.ExpressionProfiler
:members:
FLOPs
^^^^^
.. automodule:: nni.nas.profiler.pytorch.flops
:members:
nn-Meter
^^^^^^^^
.. automodule:: nni.nas.profiler.pytorch.nn_meter
:members:
Model format
------------
.. automodule:: nni.nas.space
:members:
:imported-members:
Execution engine
----------------
.. automodule:: nni.nas.execution
:members:
:imported-members:
Cross-graph optimization
^^^^^^^^^^^^^^^^^^^^^^^^
.. automodule:: nni.nas.execution.cgo
:members:
:imported-members:
NAS Benchmarks
--------------
.. automodule:: nni.nas.benchmark
:members:
:imported-members:
NAS-Bench-101
^^^^^^^^^^^^^
.. automodule:: nni.nas.benchmark.nasbench101
:members:
:imported-members:
NAS-Bench-201
^^^^^^^^^^^^^
.. automodule:: nni.nas.benchmark.nasbench201
:members:
:imported-members:
NDS
^^^
.. automodule:: nni.nas.benchmark.nds
:members:
:imported-members:
Miscellaneous Utilities
-----------------------
.. automodule:: nni.nas.utils.serializer
:members:

48
docs/source/reference/nas/evaluator.rst
Просмотреть файл

@ -1,48 +0,0 @@
Evaluator
=========
FunctionalEvaluator
-------------------
.. autoclass:: nni.nas.evaluator.FunctionalEvaluator
:members:
Classification
--------------
.. autoclass:: nni.nas.evaluator.pytorch.Classification
:members:
.. autoclass:: nni.nas.evaluator.pytorch.ClassificationModule
:members:
Regression
----------
.. autoclass:: nni.nas.evaluator.pytorch.Regression
:members:
.. autoclass:: nni.nas.evaluator.pytorch.RegressionModule
:members:
Utilities
---------
.. autoclass:: nni.nas.evaluator.pytorch.Trainer
.. autoclass:: nni.nas.evaluator.pytorch.DataLoader
Customization
-------------
.. autoclass:: nni.nas.evaluator.Evaluator
:members:
.. autoclass:: nni.nas.evaluator.pytorch.Lightning
:members:
.. autoclass:: nni.nas.evaluator.pytorch.LightningModule
:members:
Cross-graph Optimization (experimental)
---------------------------------------

29
docs/source/reference/nas/others.rst
Просмотреть файл

@ -1,29 +0,0 @@
Uncategorized Modules
=====================
Experiment
----------
NAS Benchmarks
--------------
.. _nas-bench-101-reference:
NAS-Bench-101
^^^^^^^^^^^^^
.. _nas-bench-201-reference:
NAS-Bench-201
^^^^^^^^^^^^^
.. _nds-reference:
NDS
^^^
Retrain (Architecture Evaluation)
---------------------------------
Utilities
---------

24
docs/source/reference/nas/search_space.rst
Просмотреть файл

@ -1,24 +0,0 @@
Search Space
============
.. _mutation-primitives:
Mutation Primitives
-------------------
.. _hyper-modules:
Hyper-module Library (experimental)
-----------------------------------
Mutators (advanced)
-------------------
Mutator
^^^^^^^
Placeholder
^^^^^^^^^^^
Graph
^^^^^

21
docs/source/reference/nas/strategy.rst
Просмотреть файл

@ -1,21 +0,0 @@
Strategy
========
.. _multi-trial-nas-reference:
Multi-trial Strategy
--------------------
.. _one-shot-strategy-reference:
One-shot Strategy
-----------------
Customization
-------------
Multi-trial
^^^^^^^^^^^
One-shot
^^^^^^^^

10
docs/source/reference/nas/toctree.rst
Просмотреть файл

@ -1,10 +0,0 @@
NAS API Reference
=================
.. toctree::
:maxdepth: 2
search_space
strategy
evaluator
Others <others>

2
docs/source/reference/python_api.rst
Просмотреть файл

@ -5,7 +5,7 @@ Python API Reference
:maxdepth: 1
Hyperparameter Optimization <hpo>
Neural Architecture Search <nas/toctree>
Neural Architecture Search <nas>
Model Compression <compression/toctree>
Experiment <experiment>
Mutable <mutable>

46
docs/source/tutorials/darts.ipynb сгенерированный
Просмотреть файл

@ -15,7 +15,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"\n# Searching in DARTS search space\n\nIn this tutorial, we demonstrate how to search in the famous model space proposed in `DARTS`_.\n\nThrough this process, you will learn:\n\n* How to use the built-in model spaces from NNI's model space hub.\n* How to use one-shot exploration strategies to explore a model space.\n* How to customize evaluators to achieve the best performance.\n\nIn the end, we get a strong-performing model on CIFAR-10 dataset, which achieves up to 97.28% accuracy.\n\n.. attention::\n\n Running this tutorial requires a GPU.\n If you don't have one, you can set ``gpus`` in :class:`~nni.retiarii.evaluator.pytorch.Classification` to be 0,\n but do note that it will be much slower.\n\n\n## Use a pre-searched DARTS model\n\nSimilar to [the beginner tutorial of PyTorch](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html)_,\nwe begin with CIFAR-10 dataset, which is a image classification dataset of 10 categories.\nThe images in CIFAR-10 are of size 3x32x32, i.e., RGB-colored images of 32x32 pixels in size.\n\nWe first load the CIFAR-10 dataset with torchvision.\n"
"\n# Searching in DARTS search space\n\nIn this tutorial, we demonstrate how to search in the famous model space proposed in `DARTS`_.\n\nThrough this process, you will learn:\n\n* How to use the built-in model spaces from NNI's model space hub.\n* How to use one-shot exploration strategies to explore a model space.\n* How to customize evaluators to achieve the best performance.\n\nIn the end, we get a strong-performing model on CIFAR-10 dataset, which achieves up to 97.28% accuracy.\n\n.. attention::\n\n Running this tutorial requires a GPU.\n If you don't have one, you can set ``gpus`` in :class:`~nni.nas.evaluator.pytorch.Classification` to be 0,\n but do note that it will be much slower.\n\n\n## Use a pre-searched DARTS model\n\nSimilar to [the beginner tutorial of PyTorch](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html)_,\nwe begin with CIFAR-10 dataset, which is a image classification dataset of 10 categories.\nThe images in CIFAR-10 are of size 3x32x32, i.e., RGB-colored images of 32x32 pixels in size.\n\nWe first load the CIFAR-10 dataset with torchvision.\n"
]
},
{
@ -26,14 +26,14 @@
},
"outputs": [],
"source": [
"import nni\nimport torch\nfrom torchvision import transforms\nfrom torchvision.datasets import CIFAR10\nfrom nni.retiarii.evaluator.pytorch import DataLoader\n\nCIFAR_MEAN = [0.49139968, 0.48215827, 0.44653124]\nCIFAR_STD = [0.24703233, 0.24348505, 0.26158768]\n\ntransform_valid = transforms.Compose([\n transforms.ToTensor(),\n transforms.Normalize(CIFAR_MEAN, CIFAR_STD),\n])\nvalid_data = nni.trace(CIFAR10)(root='./data', train=False, download=True, transform=transform_valid)\nvalid_loader = DataLoader(valid_data, batch_size=256, num_workers=6)"
"import nni\nimport torch\nfrom torchvision import transforms\nfrom torchvision.datasets import CIFAR10\nfrom nni.nas.evaluator.pytorch import DataLoader\n\nCIFAR_MEAN = [0.49139968, 0.48215827, 0.44653124]\nCIFAR_STD = [0.24703233, 0.24348505, 0.26158768]\n\ntransform_valid = transforms.Compose([\n transforms.ToTensor(),\n transforms.Normalize(CIFAR_MEAN, CIFAR_STD),\n])\nvalid_data = nni.trace(CIFAR10)(root='./data', train=False, download=True, transform=transform_valid)\nvalid_loader = DataLoader(valid_data, batch_size=256, num_workers=6)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\"><h4>Note</h4><p>If you are to use multi-trial strategies, wrapping CIFAR10 with :func:`nni.trace` and\n use DataLoader from ``nni.retiarii.evaluator.pytorch`` (instead of ``torch.utils.data``) are mandatory.\n Otherwise, it's optional.</p></div>\n\nNNI presents many built-in model spaces, along with many *pre-searched models* in :doc:`model space hub </nas/space_hub>`,\nwhich are produced by most popular NAS literatures.\nA pre-trained model is a saved network that was previously trained on a large dataset like CIFAR-10 or ImageNet.\nYou can easily load these models as a starting point, validate their performances, and finetune them if you need.\n\nIn this tutorial, we choose one from `DARTS`_ search space, which is natively trained on our target dataset, CIFAR-10,\nso as to save the tedious steps of finetuning.\n\n.. tip::\n\n Finetuning a pre-searched model on other datasets is no different from finetuning *any model*.\n We recommend reading\n [this tutorial of object detection finetuning](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html)_\n if you want to know how finetuning is generally done in PyTorch.\n\n"
"<div class=\"alert alert-info\"><h4>Note</h4><p>If you are to use multi-trial strategies, wrapping CIFAR10 with :func:`nni.trace` and\n use DataLoader from ``nni.nas.evaluator.pytorch`` (instead of ``torch.utils.data``) are mandatory.\n Otherwise, it's optional.</p></div>\n\nNNI presents many built-in model spaces, along with many *pre-searched models* in :doc:`model space hub </nas/space_hub>`,\nwhich are produced by most popular NAS literatures.\nA pre-trained model is a saved network that was previously trained on a large dataset like CIFAR-10 or ImageNet.\nYou can easily load these models as a starting point, validate their performances, and finetune them if you need.\n\nIn this tutorial, we choose one from `DARTS`_ search space, which is natively trained on our target dataset, CIFAR-10,\nso as to save the tedious steps of finetuning.\n\n.. tip::\n\n Finetuning a pre-searched model on other datasets is no different from finetuning *any model*.\n We recommend reading\n [this tutorial of object detection finetuning](https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html)_\n if you want to know how finetuning is generally done in PyTorch.\n\n"
]
},
{
@ -44,14 +44,14 @@
},
"outputs": [],
"source": [
"from nni.retiarii.hub.pytorch import DARTS as DartsSpace\n\ndarts_v2_model = DartsSpace.load_searched_model('darts-v2', pretrained=True, download=True)\n\ndef evaluate_model(model, cuda=False):\n device = torch.device('cuda' if cuda else 'cpu')\n model.to(device)\n model.eval()\n with torch.no_grad():\n correct = total = 0\n for inputs, targets in valid_loader:\n inputs, targets = inputs.to(device), targets.to(device)\n logits = model(inputs)\n _, predict = torch.max(logits, 1)\n correct += (predict == targets).sum().cpu().item()\n total += targets.size(0)\n print('Accuracy:', correct / total)\n return correct / total\n\nevaluate_model(darts_v2_model, cuda=True) # Set this to false if there's no GPU."
"from nni.nas.hub.pytorch import DARTS as DartsSpace\n\ndarts_v2_model = DartsSpace.load_searched_model('darts-v2', pretrained=True, download=True)\n\ndef evaluate_model(model, cuda=False):\n device = torch.device('cuda' if cuda else 'cpu')\n model.to(device)\n model.eval()\n with torch.no_grad():\n correct = total = 0\n for inputs, targets in valid_loader:\n inputs, targets = inputs.to(device), targets.to(device)\n logits = model(inputs)\n _, predict = torch.max(logits, 1)\n correct += (predict == targets).sum().cpu().item()\n total += targets.size(0)\n print('Accuracy:', correct / total)\n return correct / total\n\nevaluate_model(darts_v2_model, cuda=True) # Set this to false if there's no GPU."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The journey of using a pre-searched model could end here. Or you are interested,\nwe can go a step further to search a model within :class:`~nni.retiarii.hub.pytorch.DARTS` space on our own.\n\n## Use the DARTS model space\n\nThe model space provided in `DARTS`_ originated from [NASNet](https://arxiv.org/abs/1707.07012)_,\nwhere the full model is constructed by repeatedly stacking a single computational unit (called a **cell**).\nThere are two types of cells within a network. The first type is called *normal cell*, and the second type is called *reduction cell*.\nThe key difference between normal and reduction cell is that the reduction cell will downsample the input feature map,\nand decrease its resolution. Normal and reduction cells are stacked alternately, as shown in the following figure.\n\n<img src=\"file://../../img/nasnet_cell_stack.png\">\n\nA cell takes outputs from two previous cells as inputs and contains a collection of *nodes*.\nEach node takes two previous nodes within the same cell (or the two cell inputs),\nand applies an *operator* (e.g., convolution, or max-pooling) to each input,\nand sums the outputs of operators as the output of the node.\nThe output of cell is the concatenation of all the nodes that are never used as inputs of another node.\nUsers could read [NDS](https://arxiv.org/pdf/1905.13214.pdf)_ or [ENAS](https://arxiv.org/abs/1802.03268)_ for more details.\n\nWe illustrate an example of cells in the following figure.\n\n<img src=\"file://../../img/nasnet_cell.png\">\n\nThe search space proposed in `DARTS`_ paper introduced two modifications to the original space\nin [NASNet](https://arxiv.org/abs/1707.07012)_.\n\nFirstly, the operator candidates have been narrowed down to seven:\n\n- Max pooling 3x3\n- Average pooling 3x3\n- Skip connect (Identity)\n- Separable convolution 3x3\n- Separable convolution 5x5\n- Dilated convolution 3x3\n- Dilated convolution 5x5\n\nSecondly, the output of cell is the concatenate of **all the nodes within the cell**.\n\nAs the search space is based on cell, once the normal and reduction cell has been fixed, we can stack them for indefinite times.\nTo save the search cost, the common practice is to reduce the number of filters (i.e., channels) and number of stacked cells\nduring the search phase, and increase them back when training the final searched architecture.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>`DARTS`_ is one of those papers that innovate both in search space and search strategy.\n In this tutorial, we will search on **model space** provided by DARTS with **search strategy** proposed by DARTS.\n We refer to them as *DARTS model space* (``DartsSpace``) and *DARTS strategy* (``DartsStrategy``), respectively.\n We did NOT imply that the :class:`~nni.retiarii.hub.pytorch.DARTS` space and\n :class:`~nni.retiarii.strategy.DARTS` strategy has to used together.\n You can always explore the DARTS space with another search strategy, or use your own strategy to search a different model space.</p></div>\n\nIn the following example, we initialize a :class:`~nni.retiarii.hub.pytorch.DARTS`\nmodel space, with 16 initial filters and 8 stacked cells.\nThe network is specialized for CIFAR-10 dataset with 32x32 input resolution.\n\nThe :class:`~nni.retiarii.hub.pytorch.DARTS` model space here is provided by :doc:`model space hub </nas/space_hub>`,\nwhere we have supported multiple popular model spaces for plug-and-play.\n\n.. tip::\n\n The model space here can be replaced with any space provided in the hub,\n or even customized spaces built from scratch.\n\n"
"The journey of using a pre-searched model could end here. Or you are interested,\nwe can go a step further to search a model within :class:`~nni.nas.hub.pytorch.DARTS` space on our own.\n\n## Use the DARTS model space\n\nThe model space provided in `DARTS`_ originated from [NASNet](https://arxiv.org/abs/1707.07012)_,\nwhere the full model is constructed by repeatedly stacking a single computational unit (called a **cell**).\nThere are two types of cells within a network. The first type is called *normal cell*, and the second type is called *reduction cell*.\nThe key difference between normal and reduction cell is that the reduction cell will downsample the input feature map,\nand decrease its resolution. Normal and reduction cells are stacked alternately, as shown in the following figure.\n\n<img src=\"file://../../img/nasnet_cell_stack.png\">\n\nA cell takes outputs from two previous cells as inputs and contains a collection of *nodes*.\nEach node takes two previous nodes within the same cell (or the two cell inputs),\nand applies an *operator* (e.g., convolution, or max-pooling) to each input,\nand sums the outputs of operators as the output of the node.\nThe output of cell is the concatenation of all the nodes that are never used as inputs of another node.\nUsers could read [NDS](https://arxiv.org/pdf/1905.13214.pdf)_ or [ENAS](https://arxiv.org/abs/1802.03268)_ for more details.\n\nWe illustrate an example of cells in the following figure.\n\n<img src=\"file://../../img/nasnet_cell.png\">\n\nThe search space proposed in `DARTS`_ paper introduced two modifications to the original space\nin [NASNet](https://arxiv.org/abs/1707.07012)_.\n\nFirstly, the operator candidates have been narrowed down to seven:\n\n- Max pooling 3x3\n- Average pooling 3x3\n- Skip connect (Identity)\n- Separable convolution 3x3\n- Separable convolution 5x5\n- Dilated convolution 3x3\n- Dilated convolution 5x5\n\nSecondly, the output of cell is the concatenate of **all the nodes within the cell**.\n\nAs the search space is based on cell, once the normal and reduction cell has been fixed, we can stack them for indefinite times.\nTo save the search cost, the common practice is to reduce the number of filters (i.e., channels) and number of stacked cells\nduring the search phase, and increase them back when training the final searched architecture.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>`DARTS`_ is one of those papers that innovate both in search space and search strategy.\n In this tutorial, we will search on **model space** provided by DARTS with **search strategy** proposed by DARTS.\n We refer to them as *DARTS model space* (``DartsSpace``) and *DARTS strategy* (``DartsStrategy``), respectively.\n We did NOT imply that the :class:`~nni.nas.hub.pytorch.DARTS` space and\n :class:`~nni.nas.strategy.DARTS` strategy has to used together.\n You can always explore the DARTS space with another search strategy, or use your own strategy to search a different model space.</p></div>\n\nIn the following example, we initialize a :class:`~nni.nas.hub.pytorch.DARTS`\nmodel space, with 16 initial filters and 8 stacked cells.\nThe network is specialized for CIFAR-10 dataset with 32x32 input resolution.\n\nThe :class:`~nni.nas.hub.pytorch.DARTS` model space here is provided by :doc:`model space hub </nas/space_hub>`,\nwhere we have supported multiple popular model spaces for plug-and-play.\n\n.. tip::\n\n The model space here can be replaced with any space provided in the hub,\n or even customized spaces built from scratch.\n\n"
]
},
{
@ -87,7 +87,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"### Evaluator\n\nTo begin exploring the model space, one firstly need to have an evaluator to provide the criterion of a \"good model\".\nAs we are searching on CIFAR-10 dataset, one can easily use the :class:`~nni.retiarii.evaluator.pytorch.Classification`\nas a starting point.\n\nNote that for a typical setup of NAS, the model search should be on validation set, and the evaluation of the final searched model\nshould be on test set. However, as CIFAR-10 dataset doesn't have a test dataset (only 50k train + 10k valid),\nwe have to split the original training set into a training set and a validation set.\nThe recommended train/val split by `DARTS`_ strategy is 1:1.\n\n"
"### Evaluator\n\nTo begin exploring the model space, one firstly need to have an evaluator to provide the criterion of a \"good model\".\nAs we are searching on CIFAR-10 dataset, one can easily use the :class:`~nni.nas.evaluator.pytorch.Classification`\nas a starting point.\n\nNote that for a typical setup of NAS, the model search should be on validation set, and the evaluation of the final searched model\nshould be on test set. However, as CIFAR-10 dataset doesn't have a test dataset (only 50k train + 10k valid),\nwe have to split the original training set into a training set and a validation set.\nThe recommended train/val split by `DARTS`_ strategy is 1:1.\n\n"
]
},
{
@ -98,14 +98,14 @@
},
"outputs": [],
"source": [
"import numpy as np\nfrom nni.retiarii.evaluator.pytorch import Classification\nfrom torch.utils.data import SubsetRandomSampler\n\ntransform = transforms.Compose([\n transforms.RandomCrop(32, padding=4),\n transforms.RandomHorizontalFlip(),\n transforms.ToTensor(),\n transforms.Normalize(CIFAR_MEAN, CIFAR_STD),\n])\n\ntrain_data = nni.trace(CIFAR10)(root='./data', train=True, download=True, transform=transform)\n\nnum_samples = len(train_data)\nindices = np.random.permutation(num_samples)\nsplit = num_samples // 2\n\nsearch_train_loader = DataLoader(\n train_data, batch_size=64, num_workers=6,\n sampler=SubsetRandomSampler(indices[:split]),\n)\n\nsearch_valid_loader = DataLoader(\n train_data, batch_size=64, num_workers=6,\n sampler=SubsetRandomSampler(indices[split:]),\n)\n\nevaluator = Classification(\n learning_rate=1e-3,\n weight_decay=1e-4,\n train_dataloaders=search_train_loader,\n val_dataloaders=search_valid_loader,\n max_epochs=10,\n gpus=1,\n fast_dev_run=fast_dev_run,\n)"
"import numpy as np\nfrom nni.nas.evaluator.pytorch import Classification\nfrom torch.utils.data import SubsetRandomSampler\n\ntransform = transforms.Compose([\n transforms.RandomCrop(32, padding=4),\n transforms.RandomHorizontalFlip(),\n transforms.ToTensor(),\n transforms.Normalize(CIFAR_MEAN, CIFAR_STD),\n])\n\ntrain_data = nni.trace(CIFAR10)(root='./data', train=True, download=True, transform=transform)\n\nnum_samples = len(train_data)\nindices = np.random.permutation(num_samples)\nsplit = num_samples // 2\n\nsearch_train_loader = DataLoader(\n train_data, batch_size=64, num_workers=6,\n sampler=SubsetRandomSampler(indices[:split]),\n)\n\nsearch_valid_loader = DataLoader(\n train_data, batch_size=64, num_workers=6,\n sampler=SubsetRandomSampler(indices[split:]),\n)\n\nevaluator = Classification(\n learning_rate=1e-3,\n weight_decay=1e-4,\n train_dataloaders=search_train_loader,\n val_dataloaders=search_valid_loader,\n max_epochs=10,\n gpus=1,\n fast_dev_run=fast_dev_run,\n)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Strategy\n\nWe will use `DARTS`_ (Differentiable ARchiTecture Search) as the search strategy to explore the model space.\n:class:`~nni.retiarii.strategy.DARTS` strategy belongs to the category of `one-shot strategy <one-shot-nas>`.\nThe fundamental differences between One-shot strategies and `multi-trial strategies <multi-trial-nas>` is that,\none-shot strategy combines search with model training into a single run.\nCompared to multi-trial strategies, one-shot NAS doesn't need to iteratively spawn new trials (i.e., models),\nand thus saves the excessive cost of model training.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>It's worth mentioning that one-shot NAS also suffers from multiple drawbacks despite its computational efficiency.\n We recommend\n [Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap](https://arxiv.org/abs/2008.01475)_\n and\n [How Does Supernet Help in Neural Architecture Search?](https://arxiv.org/abs/2010.08219)_ for interested readers.</p></div>\n\n:class:`~nni.retiarii.strategy.DARTS` strategy is provided as one of NNI's :doc:`built-in search strategies </nas/exploration_strategy>`.\nUsing it can be as simple as one line of code.\n\n"
"### Strategy\n\nWe will use `DARTS`_ (Differentiable ARchiTecture Search) as the search strategy to explore the model space.\n:class:`~nni.nas.strategy.DARTS` strategy belongs to the category of `one-shot strategy <one-shot-nas>`.\nThe fundamental differences between One-shot strategies and `multi-trial strategies <multi-trial-nas>` is that,\none-shot strategy combines search with model training into a single run.\nCompared to multi-trial strategies, one-shot NAS doesn't need to iteratively spawn new trials (i.e., models),\nand thus saves the excessive cost of model training.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>It's worth mentioning that one-shot NAS also suffers from multiple drawbacks despite its computational efficiency.\n We recommend\n [Weight-Sharing Neural Architecture Search: A Battle to Shrink the Optimization Gap](https://arxiv.org/abs/2008.01475)_\n and\n [How Does Supernet Help in Neural Architecture Search?](https://arxiv.org/abs/2010.08219)_ for interested readers.</p></div>\n\n:class:`~nni.nas.strategy.DARTS` strategy is provided as one of NNI's :doc:`built-in search strategies </nas/exploration_strategy>`.\nUsing it can be as simple as one line of code.\n\n"
]
},
{
@ -116,14 +116,14 @@
},
"outputs": [],
"source": [
"from nni.retiarii.strategy import DARTS as DartsStrategy\n\nstrategy = DartsStrategy()"
"from nni.nas.strategy import DARTS as DartsStrategy\n\nstrategy = DartsStrategy()"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
".. tip:: The ``DartsStrategy`` here can be replaced by any search strategies, even multi-trial strategies.\n\nIf you want to know how DARTS strategy works, here is a brief version.\nUnder the hood, DARTS converts the cell into a densely connected graph, and put operators on edges (see the following figure).\nSince the operators are not decided yet, every edge is a weighted mixture of multiple operators (multiple color in the figure).\nDARTS then learns to assign the optimal \"color\" for each edge during the network training.\nIt finally selects one \"color\" for each edge, and drops redundant edges.\nThe weights on the edges are called *architecture weights*.\n\n<img src=\"file://../../img/darts_illustration.png\">\n\n.. tip:: It's NOT reflected in the figure that, for DARTS model space, exactly two inputs are kept for every node.\n\n### Launch experiment\n\nWe then come to the step of launching the experiment.\nThis step is similar to what we have done in the :doc:`beginner tutorial <hello_nas>`,\nexcept that the ``execution_engine`` argument should be set to ``oneshot``.\n\n"
".. tip:: The ``DartsStrategy`` here can be replaced by any search strategies, even multi-trial strategies.\n\nIf you want to know how DARTS strategy works, here is a brief version.\nUnder the hood, DARTS converts the cell into a densely connected graph, and put operators on edges (see the following figure).\nSince the operators are not decided yet, every edge is a weighted mixture of multiple operators (multiple color in the figure).\nDARTS then learns to assign the optimal \"color\" for each edge during the network training.\nIt finally selects one \"color\" for each edge, and drops redundant edges.\nThe weights on the edges are called *architecture weights*.\n\n<img src=\"file://../../img/darts_illustration.png\">\n\n.. tip:: It's NOT reflected in the figure that, for DARTS model space, exactly two inputs are kept for every node.\n\n### Launch experiment\n\nWe then come to the step of launching the experiment.\nThis step is similar to what we have done in the :doc:`beginner tutorial <hello_nas>`.\n\n"
]
},
{
@ -134,7 +134,7 @@
},
"outputs": [],
"source": [
"from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig\n\nconfig = RetiariiExeConfig(execution_engine='oneshot')\nexperiment = RetiariiExperiment(model_space, evaluator=evaluator, strategy=strategy)\nexperiment.run(config)"
"from nni.nas.experiment import NasExperiment\n\nexperiment = NasExperiment(model_space, evaluator, strategy)\nexperiment.run()"
]
},
{
@ -152,7 +152,7 @@
},
"outputs": [],
"source": [
"exported_arch = experiment.export_top_models()[0]\n\nexported_arch"
"exported_arch = experiment.export_top_models(formatter='dict')[0]\n\nexported_arch"
]
},
{
@ -195,7 +195,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"## Retrain the searched model\n\nWhat we have got in the last step, is only a cell structure.\nTo get a final usable model with trained weights, we need to construct a real model based on this structure,\nand then fully train it.\n\nTo construct a fixed model based on the architecture dict exported from the experiment,\nwe can use :func:`nni.retiarii.fixed_arch`. Under the with-context, we will creating a fixed model based on ``exported_arch``,\ninstead of creating a space.\n\n"
"## Retrain the searched model\n\nWhat we have got in the last step, is only a cell structure.\nTo get a final usable model with trained weights, we need to construct a real model based on this structure,\nand then fully train it.\n\nTo construct a fixed model based on the architecture dict exported from the experiment,\nwe can use :func:`nni.nas.space.model_context`. Under the with-context, we will creating a fixed model based on ``exported_arch``,\ninstead of creating a space.\n\n"
]
},
{
@ -206,7 +206,7 @@
},
"outputs": [],
"source": [
"from nni.retiarii import fixed_arch\n\nwith fixed_arch(exported_arch):\n final_model = DartsSpace(width=16, num_cells=8, dataset='cifar')"
"from nni.nas.space import model_context\n\nwith model_context(exported_arch):\n final_model = DartsSpace(width=16, num_cells=8, dataset='cifar')"
]
},
{
@ -249,7 +249,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We must create a new evaluator here because a different data split is used.\nAlso, we should avoid the underlying pytorch-lightning implementation of :class:`~nni.retiarii.evaluator.pytorch.Classification`\nevaluator from loading the wrong checkpoint.\n\n"
"We must create a new evaluator here because a different data split is used.\nAlso, we should avoid the underlying pytorch-lightning implementation of :class:`~nni.nas.evaluator.pytorch.Classification`\nevaluator from loading the wrong checkpoint.\n\n"
]
},
{
@ -267,7 +267,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"<div class=\"alert alert-info\"><h4>Note</h4><p>When ``fast_dev_run`` is turned off, we achieve a validation accuracy of 89.69% after training for 100 epochs.</p></div>\n\n## Reproduce results in DARTS paper\n\nAfter a brief walkthrough of search + retrain process with one-shot strategy,\nwe then fill the gap between our results (89.69%) and the results in the `DARTS` paper.\nThis is because we didn't introduce some extra training tricks, including [DropPath](https://arxiv.org/pdf/1605.07648v4.pdf)_,\nAuxiliary loss, gradient clipping and augmentations like [Cutout](https://arxiv.org/pdf/1708.04552v2.pdf)_.\nThey also train the deeper (20 cells) and wider (36 filters) networks for longer time (600 epochs).\nHere we reproduce these tricks to get comparable results with DARTS paper.\n\n\n### Evaluator\n\nTo implement these tricks, we first need to rewrite a few parts of evaluator.\n\nWorking with one-shot strategies, evaluators need to be implemented in the style of `PyTorch-Lightning <lightning-evaluator>`,\nThe full tutorial can be found in :doc:`/nas/evaluator`.\nPutting it briefly, the core part of writing a new evaluator is to write a new LightningModule.\n[LightingModule](https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html)_ is a concept in\nPyTorch-Lightning, which organizes the model training process into a list of functions, such as,\n``training_step``, ``validation_step``, ``configure_optimizers``, etc.\nSince we are merely adding a few ingredients to :class:`~nni.retiarii.evaluator.pytorch.Classification`,\nwe can simply inherit :class:`~nni.retiarii.evaluator.pytorch.ClassificationModule`, which is the underlying LightningModule\nbehind :class:`~nni.retiarii.evaluator.pytorch.Classification`.\nThis could look intimidating at first, but most of them are just plug-and-play tricks which you don't need to know details about.\n\n"
"<div class=\"alert alert-info\"><h4>Note</h4><p>When ``fast_dev_run`` is turned off, we achieve a validation accuracy of 89.69% after training for 100 epochs.</p></div>\n\n## Reproduce results in DARTS paper\n\nAfter a brief walkthrough of search + retrain process with one-shot strategy,\nwe then fill the gap between our results (89.69%) and the results in the `DARTS` paper.\nThis is because we didn't introduce some extra training tricks, including [DropPath](https://arxiv.org/pdf/1605.07648v4.pdf)_,\nAuxiliary loss, gradient clipping and augmentations like [Cutout](https://arxiv.org/pdf/1708.04552v2.pdf)_.\nThey also train the deeper (20 cells) and wider (36 filters) networks for longer time (600 epochs).\nHere we reproduce these tricks to get comparable results with DARTS paper.\n\n\n### Evaluator\n\nTo implement these tricks, we first need to rewrite a few parts of evaluator.\n\nWorking with one-shot strategies, evaluators need to be implemented in the style of `PyTorch-Lightning <lightning-evaluator>`,\nThe full tutorial can be found in :doc:`/nas/evaluator`.\nPutting it briefly, the core part of writing a new evaluator is to write a new LightningModule.\n[LightingModule](https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html)_ is a concept in\nPyTorch-Lightning, which organizes the model training process into a list of functions, such as,\n``training_step``, ``validation_step``, ``configure_optimizers``, etc.\nSince we are merely adding a few ingredients to :class:`~nni.nas.evaluator.pytorch.Classification`,\nwe can simply inherit :class:`~nni.nas.evaluator.pytorch.ClassificationModule`, which is the underlying LightningModule\nbehind :class:`~nni.nas.evaluator.pytorch.Classification`.\nThis could look intimidating at first, but most of them are just plug-and-play tricks which you don't need to know details about.\n\n"
]
},
{
@ -278,14 +278,14 @@
},
"outputs": [],
"source": [
"import torch\nfrom nni.retiarii.evaluator.pytorch import ClassificationModule\n\nclass DartsClassificationModule(ClassificationModule):\n def __init__(\n self,\n learning_rate: float = 0.001,\n weight_decay: float = 0.,\n auxiliary_loss_weight: float = 0.4,\n max_epochs: int = 600\n ):\n self.auxiliary_loss_weight = auxiliary_loss_weight\n # Training length will be used in LR scheduler\n self.max_epochs = max_epochs\n super().__init__(learning_rate=learning_rate, weight_decay=weight_decay, export_onnx=False)\n\n def configure_optimizers(self):\n \"\"\"Customized optimizer with momentum, as well as a scheduler.\"\"\"\n optimizer = torch.optim.SGD(\n self.parameters(),\n momentum=0.9,\n lr=self.hparams.learning_rate,\n weight_decay=self.hparams.weight_decay\n )\n return {\n 'optimizer': optimizer,\n 'lr_scheduler': torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, self.max_epochs, eta_min=1e-3)\n }\n\n def training_step(self, batch, batch_idx):\n \"\"\"Training step, customized with auxiliary loss.\"\"\"\n x, y = batch\n if self.auxiliary_loss_weight:\n y_hat, y_aux = self(x)\n loss_main = self.criterion(y_hat, y)\n loss_aux = self.criterion(y_aux, y)\n self.log('train_loss_main', loss_main)\n self.log('train_loss_aux', loss_aux)\n loss = loss_main + self.auxiliary_loss_weight * loss_aux\n else:\n y_hat = self(x)\n loss = self.criterion(y_hat, y)\n self.log('train_loss', loss, prog_bar=True)\n for name, metric in self.metrics.items():\n self.log('train_' + name, metric(y_hat, y), prog_bar=True)\n return loss\n\n def on_train_epoch_start(self):\n # Set drop path probability before every epoch. This has no effect if drop path is not enabled in model.\n self.model.set_drop_path_prob(self.model.drop_path_prob * self.current_epoch / self.max_epochs)\n\n # Logging learning rate at the beginning of every epoch\n self.log('lr', self.trainer.optimizers[0].param_groups[0]['lr'])"
"import torch\nfrom nni.nas.evaluator.pytorch import ClassificationModule\n\nclass DartsClassificationModule(ClassificationModule):\n def __init__(\n self,\n learning_rate: float = 0.001,\n weight_decay: float = 0.,\n auxiliary_loss_weight: float = 0.4,\n max_epochs: int = 600\n ):\n self.auxiliary_loss_weight = auxiliary_loss_weight\n # Training length will be used in LR scheduler\n self.max_epochs = max_epochs\n super().__init__(learning_rate=learning_rate, weight_decay=weight_decay, export_onnx=False)\n\n def configure_optimizers(self):\n \"\"\"Customized optimizer with momentum, as well as a scheduler.\"\"\"\n optimizer = torch.optim.SGD(\n self.parameters(),\n momentum=0.9,\n lr=self.hparams.learning_rate,\n weight_decay=self.hparams.weight_decay\n )\n return {\n 'optimizer': optimizer,\n 'lr_scheduler': torch.optim.lr_scheduler.CosineAnnealingLR(optimizer, self.max_epochs, eta_min=1e-3)\n }\n\n def training_step(self, batch, batch_idx):\n \"\"\"Training step, customized with auxiliary loss.\"\"\"\n x, y = batch\n if self.auxiliary_loss_weight:\n y_hat, y_aux = self(x)\n loss_main = self.criterion(y_hat, y)\n loss_aux = self.criterion(y_aux, y)\n self.log('train_loss_main', loss_main)\n self.log('train_loss_aux', loss_aux)\n loss = loss_main + self.auxiliary_loss_weight * loss_aux\n else:\n y_hat = self(x)\n loss = self.criterion(y_hat, y)\n self.log('train_loss', loss, prog_bar=True)\n for name, metric in self.metrics.items():\n self.log('train_' + name, metric(y_hat, y), prog_bar=True)\n return loss\n\n def on_train_epoch_start(self):\n # Set drop path probability before every epoch. This has no effect if drop path is not enabled in model.\n self.model.set_drop_path_prob(self.model.drop_path_prob * self.current_epoch / self.max_epochs)\n\n # Logging learning rate at the beginning of every epoch\n self.log('lr', self.trainer.optimizers[0].param_groups[0]['lr'])"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The full evaluator is written as follows,\nwhich simply wraps everything (except model space and search strategy of course), in a single object.\n:class:`~nni.retiarii.evaluator.pytorch.Lightning` here is a special type of evaluator.\nDon't forget to use the train/val data split specialized for search (1:1) here.\n\n"
"The full evaluator is written as follows,\nwhich simply wraps everything (except model space and search strategy of course), in a single object.\n:class:`~nni.nas.evaluator.pytorch.Lightning` here is a special type of evaluator.\nDon't forget to use the train/val data split specialized for search (1:1) here.\n\n"
]
},
{
@ -296,14 +296,14 @@
},
"outputs": [],
"source": [
"from nni.retiarii.evaluator.pytorch import Lightning, Trainer\n\nmax_epochs = 50\n\nevaluator = Lightning(\n DartsClassificationModule(0.025, 3e-4, 0., max_epochs),\n Trainer(\n gpus=1,\n max_epochs=max_epochs,\n fast_dev_run=fast_dev_run,\n ),\n train_dataloaders=search_train_loader,\n val_dataloaders=search_valid_loader\n)"
"from nni.nas.evaluator.pytorch import Lightning, Trainer\n\nmax_epochs = 50\n\nevaluator = Lightning(\n DartsClassificationModule(0.025, 3e-4, 0., max_epochs),\n Trainer(\n gpus=1,\n max_epochs=max_epochs,\n fast_dev_run=fast_dev_run,\n ),\n train_dataloaders=search_train_loader,\n val_dataloaders=search_valid_loader\n)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Strategy\n\n:class:`~nni.retiarii.strategy.DARTS` strategy is created with gradient clip turned on.\nIf you are familiar with PyTorch-Lightning, you might aware that gradient clipping can be enabled in Lightning trainer.\nHowever, enabling gradient clip in the trainer above won't work, because the underlying\nimplementation of :class:`~nni.retiarii.strategy.DARTS` strategy is based on\n[manual optimization](https://pytorch-lightning.readthedocs.io/en/stable/common/optimization.html)_.\n\n"
"### Strategy\n\n:class:`~nni.nas.strategy.DARTS` strategy is created with gradient clip turned on.\nIf you are familiar with PyTorch-Lightning, you might aware that gradient clipping can be enabled in Lightning trainer.\nHowever, enabling gradient clip in the trainer above won't work, because the underlying\nimplementation of :class:`~nni.nas.strategy.DARTS` strategy is based on\n[manual optimization](https://pytorch-lightning.readthedocs.io/en/stable/common/optimization.html)_.\n\n"
]
},
{
@ -332,7 +332,7 @@
},
"outputs": [],
"source": [
"model_space = DartsSpace(width=16, num_cells=8, dataset='cifar')\n\nconfig = RetiariiExeConfig(execution_engine='oneshot')\nexperiment = RetiariiExperiment(model_space, evaluator=evaluator, strategy=strategy)\nexperiment.run(config)\n\nexported_arch = experiment.export_top_models()[0]\n\nexported_arch"
"model_space = DartsSpace(width=16, num_cells=8, dataset='cifar')\n\nexperiment = NasExperiment(model_space, evaluator, strategy)\nexperiment.run()\n\nexported_arch = experiment.export_top_models(formatter='dict')[0]\n\nexported_arch"
]
},
{
@ -404,7 +404,7 @@
},
"outputs": [],
"source": [
"with fixed_arch(exported_arch):\n final_model = DartsSpace(width=36, num_cells=20, dataset='cifar', auxiliary_loss=True, drop_path_prob=0.2)"
"with model_context(exported_arch):\n final_model = DartsSpace(width=36, num_cells=20, dataset='cifar', auxiliary_loss=True, drop_path_prob=0.2)"
]
},
{
@ -449,7 +449,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.13"
"version": "3.8.8"
}
},
"nbformat": 4,

73
docs/source/tutorials/darts.py сгенерированный
Просмотреть файл

@ -15,7 +15,7 @@ In the end, we get a strong-performing model on CIFAR-10 dataset, which achieves
.. attention::
Running this tutorial requires a GPU.
If you don't have one, you can set ``gpus`` in :class:`~nni.retiarii.evaluator.pytorch.Classification` to be 0,
If you don't have one, you can set ``gpus`` in :class:`~nni.nas.evaluator.pytorch.Classification` to be 0,
but do note that it will be much slower.
.. _DARTS: https://arxiv.org/abs/1806.09055
@ -34,7 +34,7 @@ import nni
import torch
from torchvision import transforms
from torchvision.datasets import CIFAR10
from nni.retiarii.evaluator.pytorch import DataLoader
from nni.nas.evaluator.pytorch import DataLoader
CIFAR_MEAN = [0.49139968, 0.48215827, 0.44653124]
CIFAR_STD = [0.24703233, 0.24348505, 0.26158768]
@ -51,7 +51,7 @@ valid_loader = DataLoader(valid_data, batch_size=256, num_workers=6)
# .. note::
#
# If you are to use multi-trial strategies, wrapping CIFAR10 with :func:`nni.trace` and
# use DataLoader from ``nni.retiarii.evaluator.pytorch`` (instead of ``torch.utils.data``) are mandatory.
# use DataLoader from ``nni.nas.evaluator.pytorch`` (instead of ``torch.utils.data``) are mandatory.
# Otherwise, it's optional.
#
# NNI presents many built-in model spaces, along with many *pre-searched models* in :doc:`model space hub </nas/space_hub>`,
@ -69,7 +69,7 @@ valid_loader = DataLoader(valid_data, batch_size=256, num_workers=6)
# `this tutorial of object detection finetuning <https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html>`__
# if you want to know how finetuning is generally done in PyTorch.
from nni.retiarii.hub.pytorch import DARTS as DartsSpace
from nni.nas.hub.pytorch import DARTS as DartsSpace
darts_v2_model = DartsSpace.load_searched_model('darts-v2', pretrained=True, download=True)
@ -93,7 +93,7 @@ evaluate_model(darts_v2_model, cuda=True) # Set this to false if there's no GPU
# %%
#
# The journey of using a pre-searched model could end here. Or you are interested,
# we can go a step further to search a model within :class:`~nni.retiarii.hub.pytorch.DARTS` space on our own.
# we can go a step further to search a model within :class:`~nni.nas.hub.pytorch.DARTS` space on our own.
#
# Use the DARTS model space
# -------------------------
@ -141,15 +141,15 @@ evaluate_model(darts_v2_model, cuda=True) # Set this to false if there's no GPU
# `DARTS`_ is one of those papers that innovate both in search space and search strategy.
# In this tutorial, we will search on **model space** provided by DARTS with **search strategy** proposed by DARTS.
# We refer to them as *DARTS model space* (``DartsSpace``) and *DARTS strategy* (``DartsStrategy``), respectively.
# We did NOT imply that the :class:`~nni.retiarii.hub.pytorch.DARTS` space and
# :class:`~nni.retiarii.strategy.DARTS` strategy has to used together.
# We did NOT imply that the :class:`~nni.nas.hub.pytorch.DARTS` space and
# :class:`~nni.nas.strategy.DARTS` strategy has to used together.
# You can always explore the DARTS space with another search strategy, or use your own strategy to search a different model space.
#
# In the following example, we initialize a :class:`~nni.retiarii.hub.pytorch.DARTS`
# In the following example, we initialize a :class:`~nni.nas.hub.pytorch.DARTS`
# model space, with 16 initial filters and 8 stacked cells.
# The network is specialized for CIFAR-10 dataset with 32x32 input resolution.
#
# The :class:`~nni.retiarii.hub.pytorch.DARTS` model space here is provided by :doc:`model space hub </nas/space_hub>`,
# The :class:`~nni.nas.hub.pytorch.DARTS` model space here is provided by :doc:`model space hub </nas/space_hub>`,
# where we have supported multiple popular model spaces for plug-and-play.
#
# .. tip::
@ -181,7 +181,7 @@ fast_dev_run = True
# ^^^^^^^^^
#
# To begin exploring the model space, one firstly need to have an evaluator to provide the criterion of a "good model".
# As we are searching on CIFAR-10 dataset, one can easily use the :class:`~nni.retiarii.evaluator.pytorch.Classification`
# As we are searching on CIFAR-10 dataset, one can easily use the :class:`~nni.nas.evaluator.pytorch.Classification`
# as a starting point.
#
# Note that for a typical setup of NAS, the model search should be on validation set, and the evaluation of the final searched model
@ -190,7 +190,7 @@ fast_dev_run = True
# The recommended train/val split by `DARTS`_ strategy is 1:1.
import numpy as np
from nni.retiarii.evaluator.pytorch import Classification
from nni.nas.evaluator.pytorch import Classification
from torch.utils.data import SubsetRandomSampler
transform = transforms.Compose([
@ -232,7 +232,7 @@ evaluator = Classification(
# ^^^^^^^^
#
# We will use `DARTS`_ (Differentiable ARchiTecture Search) as the search strategy to explore the model space.
# :class:`~nni.retiarii.strategy.DARTS` strategy belongs to the category of :ref:`one-shot strategy <one-shot-nas>`.
# :class:`~nni.nas.strategy.DARTS` strategy belongs to the category of :ref:`one-shot strategy <one-shot-nas>`.
# The fundamental differences between One-shot strategies and :ref:`multi-trial strategies <multi-trial-nas>` is that,
# one-shot strategy combines search with model training into a single run.
# Compared to multi-trial strategies, one-shot NAS doesn't need to iteratively spawn new trials (i.e., models),
@ -246,10 +246,10 @@ evaluator = Classification(
# and
# `How Does Supernet Help in Neural Architecture Search? <https://arxiv.org/abs/2010.08219>`__ for interested readers.
#
# :class:`~nni.retiarii.strategy.DARTS` strategy is provided as one of NNI's :doc:`built-in search strategies </nas/exploration_strategy>`.
# :class:`~nni.nas.strategy.DARTS` strategy is provided as one of NNI's :doc:`built-in search strategies </nas/exploration_strategy>`.
# Using it can be as simple as one line of code.
from nni.retiarii.strategy import DARTS as DartsStrategy
from nni.nas.strategy import DARTS as DartsStrategy
strategy = DartsStrategy()
@ -272,14 +272,12 @@ strategy = DartsStrategy()
# ^^^^^^^^^^^^^^^^^
#
# We then come to the step of launching the experiment.
# This step is similar to what we have done in the :doc:`beginner tutorial <hello_nas>`,
# except that the ``execution_engine`` argument should be set to ``oneshot``.
# This step is similar to what we have done in the :doc:`beginner tutorial <hello_nas>`.
from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig
from nni.nas.experiment import NasExperiment
config = RetiariiExeConfig(execution_engine='oneshot')
experiment = RetiariiExperiment(model_space, evaluator=evaluator, strategy=strategy)
experiment.run(config)
experiment = NasExperiment(model_space, evaluator, strategy)
experiment.run()
# %%
#
@ -296,7 +294,7 @@ experiment.run(config)
# We can then retrieve the best model found by the strategy with ``export_top_models``.
# Here, the retrieved model is a dict (called *architecture dict*) describing the selected normal cell and reduction cell.
exported_arch = experiment.export_top_models()[0]
exported_arch = experiment.export_top_models(formatter='dict')[0]
exported_arch
@ -411,12 +409,12 @@ plot_double_cells({
# and then fully train it.
#
# To construct a fixed model based on the architecture dict exported from the experiment,
# we can use :func:`nni.retiarii.fixed_arch`. Under the with-context, we will creating a fixed model based on ``exported_arch``,
# we can use :func:`nni.nas.space.model_context`. Under the with-context, we will creating a fixed model based on ``exported_arch``,
# instead of creating a space.
from nni.retiarii import fixed_arch
from nni.nas.space import model_context
with fixed_arch(exported_arch):
with model_context(exported_arch):
final_model = DartsSpace(width=16, num_cells=8, dataset='cifar')
# %%
@ -434,7 +432,7 @@ valid_loader
# %%
#
# We must create a new evaluator here because a different data split is used.
# Also, we should avoid the underlying pytorch-lightning implementation of :class:`~nni.retiarii.evaluator.pytorch.Classification`
# Also, we should avoid the underlying pytorch-lightning implementation of :class:`~nni.nas.evaluator.pytorch.Classification`
# evaluator from loading the wrong checkpoint.
max_epochs = 100
@ -478,13 +476,13 @@ evaluator.fit(final_model)
# `LightingModule <https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html>`__ is a concept in
# PyTorch-Lightning, which organizes the model training process into a list of functions, such as,
# ``training_step``, ``validation_step``, ``configure_optimizers``, etc.
# Since we are merely adding a few ingredients to :class:`~nni.retiarii.evaluator.pytorch.Classification`,
# we can simply inherit :class:`~nni.retiarii.evaluator.pytorch.ClassificationModule`, which is the underlying LightningModule
# behind :class:`~nni.retiarii.evaluator.pytorch.Classification`.
# Since we are merely adding a few ingredients to :class:`~nni.nas.evaluator.pytorch.Classification`,
# we can simply inherit :class:`~nni.nas.evaluator.pytorch.ClassificationModule`, which is the underlying LightningModule
# behind :class:`~nni.nas.evaluator.pytorch.Classification`.
# This could look intimidating at first, but most of them are just plug-and-play tricks which you don't need to know details about.
import torch
from nni.retiarii.evaluator.pytorch import ClassificationModule
from nni.nas.evaluator.pytorch import ClassificationModule
class DartsClassificationModule(ClassificationModule):
def __init__(
@ -541,10 +539,10 @@ class DartsClassificationModule(ClassificationModule):
#
# The full evaluator is written as follows,
# which simply wraps everything (except model space and search strategy of course), in a single object.
# :class:`~nni.retiarii.evaluator.pytorch.Lightning` here is a special type of evaluator.
# :class:`~nni.nas.evaluator.pytorch.Lightning` here is a special type of evaluator.
# Don't forget to use the train/val data split specialized for search (1:1) here.
from nni.retiarii.evaluator.pytorch import Lightning, Trainer
from nni.nas.evaluator.pytorch import Lightning, Trainer
max_epochs = 50
@ -564,10 +562,10 @@ evaluator = Lightning(
# Strategy
# ^^^^^^^^
#
# :class:`~nni.retiarii.strategy.DARTS` strategy is created with gradient clip turned on.
# :class:`~nni.nas.strategy.DARTS` strategy is created with gradient clip turned on.
# If you are familiar with PyTorch-Lightning, you might aware that gradient clipping can be enabled in Lightning trainer.
# However, enabling gradient clip in the trainer above won't work, because the underlying
# implementation of :class:`~nni.retiarii.strategy.DARTS` strategy is based on
# implementation of :class:`~nni.nas.strategy.DARTS` strategy is based on
# `manual optimization <https://pytorch-lightning.readthedocs.io/en/stable/common/optimization.html>`__.
strategy = DartsStrategy(gradient_clip_val=5.)
@ -586,11 +584,10 @@ strategy = DartsStrategy(gradient_clip_val=5.)
model_space = DartsSpace(width=16, num_cells=8, dataset='cifar')
config = RetiariiExeConfig(execution_engine='oneshot')
experiment = RetiariiExperiment(model_space, evaluator=evaluator, strategy=strategy)
experiment.run(config)
experiment = NasExperiment(model_space, evaluator, strategy)
experiment.run()
exported_arch = experiment.export_top_models()[0]
exported_arch = experiment.export_top_models(formatter='dict')[0]
exported_arch
@ -684,7 +681,7 @@ train_loader_cutout = DataLoader(train_data_cutout, batch_size=96)
# Following the same procedure as paper, we also increase the number of filters to 36, and number of cells to 20,
# so as to reasonably increase the model size and boost the performance.
with fixed_arch(exported_arch):
with model_context(exported_arch):
final_model = DartsSpace(width=36, num_cells=20, dataset='cifar', auxiliary_loss=True, drop_path_prob=0.2)
# %%

2
docs/source/tutorials/darts.py.md5 сгенерированный
Просмотреть файл

@ -1 +1 @@
f314677f825241fdc926f4d01c55680d
9d3fcdd210dc926847438a0320b075cf

300
docs/source/tutorials/darts.rst сгенерированный
Просмотреть файл

@ -10,7 +10,7 @@
.. note::
:class: sphx-glr-download-link-note
Click :ref:`here <sphx_glr_download_tutorials_darts.py>`
:ref:`Go to the end <sphx_glr_download_tutorials_darts.py>`
to download the full example code
.. rst-class:: sphx-glr-example-title
@ -34,7 +34,7 @@ In the end, we get a strong-performing model on CIFAR-10 dataset, which achieves
.. attention::
Running this tutorial requires a GPU.
If you don't have one, you can set ``gpus`` in :class:`~nni.retiarii.evaluator.pytorch.Classification` to be 0,
If you don't have one, you can set ``gpus`` in :class:`~nni.nas.evaluator.pytorch.Classification` to be 0,
but do note that it will be much slower.
.. _DARTS: https://arxiv.org/abs/1806.09055
@ -57,7 +57,7 @@ We first load the CIFAR-10 dataset with torchvision.
import torch
from torchvision import transforms
from torchvision.datasets import CIFAR10
from nni.retiarii.evaluator.pytorch import DataLoader
from nni.nas.evaluator.pytorch import DataLoader
CIFAR_MEAN = [0.49139968, 0.48215827, 0.44653124]
CIFAR_STD = [0.24703233, 0.24348505, 0.26158768]
@ -87,7 +87,7 @@ We first load the CIFAR-10 dataset with torchvision.
.. note::
If you are to use multi-trial strategies, wrapping CIFAR10 with :func:`nni.trace` and
use DataLoader from ``nni.retiarii.evaluator.pytorch`` (instead of ``torch.utils.data``) are mandatory.
use DataLoader from ``nni.nas.evaluator.pytorch`` (instead of ``torch.utils.data``) are mandatory.
Otherwise, it's optional.
NNI presents many built-in model spaces, along with many *pre-searched models* in :doc:`model space hub </nas/space_hub>`,
@ -110,7 +110,7 @@ so as to save the tedious steps of finetuning.
.. code-block:: default
from nni.retiarii.hub.pytorch import DARTS as DartsSpace
from nni.nas.hub.pytorch import DARTS as DartsSpace
darts_v2_model = DartsSpace.load_searched_model('darts-v2', pretrained=True, download=True)
@ -139,6 +139,43 @@ so as to save the tedious steps of finetuning.
.. code-block:: none
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/ray/autoscaler/_private/cli_logger.py:57: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via `pip install 'ray[default]'`. Please update your install command.
warnings.warn(
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method relu of PyCapsule object at 0x7fa2d74d25a0> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method gelu of PyCapsule object at 0x7fa2d74d25d0> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method hardswish of PyCapsule object at 0x7fa2d74d2720> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method hardsigmoid of PyCapsule object at 0x7fa2d74d2870> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method relu_ of PyCapsule object at 0x7fa2d74d2960> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method hardswish_ of PyCapsule object at 0x7fa2d74d2900> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method hardsigmoid_ of PyCapsule object at 0x7fa2d74d2930> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method hardtanh_ of PyCapsule object at 0x7fa2d74d28d0> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method permute of PyCapsule object at 0x7fa2d74d28a0> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a int in torch.ops.aten because <built-in method select of PyCapsule object at 0x7fa2d74d2990> has no attribute int. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method cat of PyCapsule object at 0x7fa2d74d26f0> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a dim in torch.ops.aten because <built-in method mean of PyCapsule object at 0x7fa2d74d29f0> has no attribute dim. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a default in torch.ops.aten because <built-in method _log_softmax of PyCapsule object at 0x7fa2d74d2a20> has no attribute default. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:112: UserWarning: Fail to register shape inference formula for aten operator _reshape_alias because: No such operator aten::_reshape_alias
warnings.warn(f'Fail to register shape inference formula for aten operator {name} because: {e}')
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a Tensor in torch.ops.aten because <built-in method add of PyCapsule object at 0x7fa2d74d2ab0> has no attribute Tensor. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a Tensor in torch.ops.aten because <built-in method mul of PyCapsule object at 0x7fa2d74d2a80> has no attribute Tensor. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
/home/yugzhan/nni/nni/nas/profiler/pytorch/utils/shape_formula.py:107: UserWarning: Cannot find a Tensor in torch.ops.aten because <built-in method slice of PyCapsule object at 0x7fa2d74d2a50> has no attribute Tensor. Skip registering the shape inference formula.
warnings.warn(f'Cannot find a {name} in torch.ops.aten because {object} has no attribute {name}. '
Accuracy: 0.9737
0.9737
@ -148,7 +185,7 @@ so as to save the tedious steps of finetuning.
.. GENERATED FROM PYTHON SOURCE LINES 94-158
The journey of using a pre-searched model could end here. Or you are interested,
we can go a step further to search a model within :class:`~nni.retiarii.hub.pytorch.DARTS` space on our own.
we can go a step further to search a model within :class:`~nni.nas.hub.pytorch.DARTS` space on our own.
Use the DARTS model space
-------------------------
@ -196,15 +233,15 @@ during the search phase, and increase them back when training the final searched
`DARTS`_ is one of those papers that innovate both in search space and search strategy.
In this tutorial, we will search on **model space** provided by DARTS with **search strategy** proposed by DARTS.
We refer to them as *DARTS model space* (``DartsSpace``) and *DARTS strategy* (``DartsStrategy``), respectively.
We did NOT imply that the :class:`~nni.retiarii.hub.pytorch.DARTS` space and
:class:`~nni.retiarii.strategy.DARTS` strategy has to used together.
We did NOT imply that the :class:`~nni.nas.hub.pytorch.DARTS` space and
:class:`~nni.nas.strategy.DARTS` strategy has to used together.
You can always explore the DARTS space with another search strategy, or use your own strategy to search a different model space.
In the following example, we initialize a :class:`~nni.retiarii.hub.pytorch.DARTS`
In the following example, we initialize a :class:`~nni.nas.hub.pytorch.DARTS`
model space, with 16 initial filters and 8 stacked cells.
The network is specialized for CIFAR-10 dataset with 32x32 input resolution.
The :class:`~nni.retiarii.hub.pytorch.DARTS` model space here is provided by :doc:`model space hub </nas/space_hub>`,
The :class:`~nni.nas.hub.pytorch.DARTS` model space here is provided by :doc:`model space hub </nas/space_hub>`,
where we have supported multiple popular model spaces for plug-and-play.
.. tip::
@ -260,7 +297,7 @@ Evaluator
^^^^^^^^^
To begin exploring the model space, one firstly need to have an evaluator to provide the criterion of a "good model".
As we are searching on CIFAR-10 dataset, one can easily use the :class:`~nni.retiarii.evaluator.pytorch.Classification`
As we are searching on CIFAR-10 dataset, one can easily use the :class:`~nni.nas.evaluator.pytorch.Classification`
as a starting point.
Note that for a typical setup of NAS, the model search should be on validation set, and the evaluation of the final searched model
@ -274,7 +311,7 @@ The recommended train/val split by `DARTS`_ strategy is 1:1.
import numpy as np
from nni.retiarii.evaluator.pytorch import Classification
from nni.nas.evaluator.pytorch import Classification
from torch.utils.data import SubsetRandomSampler
transform = transforms.Compose([
@ -319,13 +356,8 @@ The recommended train/val split by `DARTS`_ strategy is 1:1.
.. code-block:: none
Files already downloaded and verified
/data/data0/jiahang/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:447: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:446: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
rank_zero_deprecation(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Running in `fast_dev_run` mode: will run the requested loop using 1 batch(es). Logging and checkpointing is suppressed.
@ -336,7 +368,7 @@ Strategy
^^^^^^^^
We will use `DARTS`_ (Differentiable ARchiTecture Search) as the search strategy to explore the model space.
:class:`~nni.retiarii.strategy.DARTS` strategy belongs to the category of :ref:`one-shot strategy <one-shot-nas>`.
:class:`~nni.nas.strategy.DARTS` strategy belongs to the category of :ref:`one-shot strategy <one-shot-nas>`.
The fundamental differences between One-shot strategies and :ref:`multi-trial strategies <multi-trial-nas>` is that,
one-shot strategy combines search with model training into a single run.
Compared to multi-trial strategies, one-shot NAS doesn't need to iteratively spawn new trials (i.e., models),
@ -350,7 +382,7 @@ and thus saves the excessive cost of model training.
and
`How Does Supernet Help in Neural Architecture Search? <https://arxiv.org/abs/2010.08219>`__ for interested readers.
:class:`~nni.retiarii.strategy.DARTS` strategy is provided as one of NNI's :doc:`built-in search strategies </nas/exploration_strategy>`.
:class:`~nni.nas.strategy.DARTS` strategy is provided as one of NNI's :doc:`built-in search strategies </nas/exploration_strategy>`.
Using it can be as simple as one line of code.
.. GENERATED FROM PYTHON SOURCE LINES 251-256
@ -358,7 +390,7 @@ Using it can be as simple as one line of code.
.. code-block:: default
from nni.retiarii.strategy import DARTS as DartsStrategy
from nni.nas.strategy import DARTS as DartsStrategy
strategy = DartsStrategy()
@ -369,7 +401,7 @@ Using it can be as simple as one line of code.
.. GENERATED FROM PYTHON SOURCE LINES 257-276
.. GENERATED FROM PYTHON SOURCE LINES 257-275
.. tip:: The ``DartsStrategy`` here can be replaced by any search strategies, even multi-trial strategies.
@ -388,19 +420,17 @@ Launch experiment
^^^^^^^^^^^^^^^^^
We then come to the step of launching the experiment.
This step is similar to what we have done in the :doc:`beginner tutorial <hello_nas>`,
except that the ``execution_engine`` argument should be set to ``oneshot``.
This step is similar to what we have done in the :doc:`beginner tutorial <hello_nas>`.
.. GENERATED FROM PYTHON SOURCE LINES 277-284
.. GENERATED FROM PYTHON SOURCE LINES 276-282
.. code-block:: default
from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig
from nni.nas.experiment import NasExperiment
config = RetiariiExeConfig(execution_engine='oneshot')
experiment = RetiariiExperiment(model_space, evaluator=evaluator, strategy=strategy)
experiment.run(config)
experiment = NasExperiment(model_space, evaluator, strategy)
experiment.run()
@ -410,25 +440,16 @@ except that the ``execution_engine`` argument should be set to ``oneshot``.
.. code-block:: none
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [3]
| Name | Type | Params
-----------------------------------------------
0 | model | ClassificationModule | 3.0 M
-----------------------------------------------
3.0 M Trainable params
0 Non-trainable params
3.0 M Total params
12.164 Total estimated model params size (MB)
/data/data0/jiahang/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1892: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
`training_service` will be ignored for sequential execution engine.
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1555: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
rank_zero_warn(
Training: 0it [00:00, ?it/s] Training: 0%| | 0/1 [00:00<?, ?it/s] Epoch 0: 0%| | 0/1 [00:00<?, ?it/s] Epoch 0: 100%|##########| 1/1 [00:03<00:00, 3.75s/it] Epoch 0: 100%|##########| 1/1 [00:03<00:00, 3.75s/it, v_num=, train_loss=2.310, train_acc=0.0781] Epoch 0: 100%|##########| 1/1 [00:03<00:00, 3.76s/it, v_num=, train_loss=2.310, train_acc=0.0781]`Trainer.fit` stopped: `max_epochs=1` reached.
Epoch 0: 100%|##########| 1/1 [00:03<00:00, 3.77s/it, v_num=, train_loss=2.310, train_acc=0.0781]
Training: 0it [00:00, ?it/s] Training: 0%| | 0/1 [00:00<?, ?it/s] Epoch 0: 0%| | 0/1 [00:00<?, ?it/s] Epoch 0: 100%|##########| 1/1 [00:03<00:00, 3.29s/it] Epoch 0: 100%|##########| 1/1 [00:03<00:00, 3.30s/it, v_num=, train_loss=2.380, train_acc=0.156] Epoch 0: 100%|##########| 1/1 [00:03<00:00, 3.30s/it, v_num=, train_loss=2.380, train_acc=0.156] Epoch 0: 100%|##########| 1/1 [00:03<00:00, 3.31s/it, v_num=, train_loss=2.380, train_acc=0.156]
True
.. GENERATED FROM PYTHON SOURCE LINES 285-297
.. GENERATED FROM PYTHON SOURCE LINES 283-295
.. tip::
@ -443,12 +464,12 @@ except that the ``execution_engine`` argument should be set to ``oneshot``.
We can then retrieve the best model found by the strategy with ``export_top_models``.
Here, the retrieved model is a dict (called *architecture dict*) describing the selected normal cell and reduction cell.
.. GENERATED FROM PYTHON SOURCE LINES 298-303
.. GENERATED FROM PYTHON SOURCE LINES 296-301
.. code-block:: default
exported_arch = experiment.export_top_models()[0]
exported_arch = experiment.export_top_models(formatter='dict')[0]
exported_arch
@ -461,16 +482,16 @@ Here, the retrieved model is a dict (called *architecture dict*) describing the
.. code-block:: none
{'normal/op_2_0': 'sep_conv_5x5', 'normal/input_2_0': 1, 'normal/op_2_1': 'max_pool_3x3', 'normal/input_2_1': 0, 'normal/op_3_0': 'dil_conv_5x5', 'normal/input_3_0': 0, 'normal/op_3_1': 'sep_conv_3x3', 'normal/input_3_1': 2, 'normal/op_4_0': 'dil_conv_5x5', 'normal/input_4_0': 3, 'normal/op_4_1': 'sep_conv_3x3', 'normal/input_4_1': 1, 'normal/op_5_0': 'sep_conv_5x5', 'normal/input_5_0': 1, 'normal/op_5_1': 'dil_conv_5x5', 'normal/input_5_1': 3, 'reduce/op_2_0': 'dil_conv_5x5', 'reduce/input_2_0': 0, 'reduce/op_2_1': 'sep_conv_5x5', 'reduce/input_2_1': 1, 'reduce/op_3_0': 'sep_conv_5x5', 'reduce/input_3_0': 1, 'reduce/op_3_1': 'max_pool_3x3', 'reduce/input_3_1': 2, 'reduce/op_4_0': 'avg_pool_3x3', 'reduce/input_4_0': 1, 'reduce/op_4_1': 'dil_conv_5x5', 'reduce/input_4_1': 3, 'reduce/op_5_0': 'sep_conv_3x3', 'reduce/input_5_0': 1, 'reduce/op_5_1': 'sep_conv_5x5', 'reduce/input_5_1': 3}
{'normal/op_2_0': 'sep_conv_3x3', 'normal/input_2_0': [0], 'normal/op_2_1': 'dil_conv_5x5', 'normal/input_2_1': [1], 'normal/op_3_0': 'sep_conv_3x3', 'normal/input_3_0': [2], 'normal/op_3_1': 'sep_conv_5x5', 'normal/input_3_1': [1], 'normal/op_4_0': 'skip_connect', 'normal/input_4_0': [1], 'normal/op_4_1': 'sep_conv_3x3', 'normal/input_4_1': [2], 'normal/op_5_0': 'sep_conv_3x3', 'normal/input_5_0': [0], 'normal/op_5_1': 'sep_conv_3x3', 'normal/input_5_1': [3], 'reduce/op_2_0': 'dil_conv_3x3', 'reduce/input_2_0': [0], 'reduce/op_2_1': 'sep_conv_3x3', 'reduce/input_2_1': [1], 'reduce/op_3_0': 'sep_conv_5x5', 'reduce/input_3_0': [2], 'reduce/op_3_1': 'sep_conv_5x5', 'reduce/input_3_1': [1], 'reduce/op_4_0': 'dil_conv_5x5', 'reduce/input_4_0': [2], 'reduce/op_4_1': 'avg_pool_3x3', 'reduce/input_4_1': [1], 'reduce/op_5_0': 'sep_conv_5x5', 'reduce/input_5_0': [3], 'reduce/op_5_1': 'dil_conv_5x5', 'reduce/input_5_1': [4]}
.. GENERATED FROM PYTHON SOURCE LINES 304-306
.. GENERATED FROM PYTHON SOURCE LINES 302-304
The cell can be visualized with the following code snippet
(copied and modified from `DARTS visualization <https://github.com/quark0/darts/blob/master/cnn/visualize.py>`__).
.. GENERATED FROM PYTHON SOURCE LINES 307-362
.. GENERATED FROM PYTHON SOURCE LINES 305-360
.. code-block:: default
@ -541,14 +562,14 @@ The cell can be visualized with the following code snippet
.. GENERATED FROM PYTHON SOURCE LINES 363-367
.. GENERATED FROM PYTHON SOURCE LINES 361-365
.. warning:: The cell above is obtained via ``fast_dev_run`` (i.e., running only 1 mini-batch).
When ``fast_dev_run`` is turned off, we get a model with the following architecture,
where you might notice an interesting fact that around half the operations have selected ``sep_conv_3x3``.
.. GENERATED FROM PYTHON SOURCE LINES 368-404
.. GENERATED FROM PYTHON SOURCE LINES 366-402
.. code-block:: default
@ -600,7 +621,7 @@ where you might notice an interesting fact that around half the operations have
.. GENERATED FROM PYTHON SOURCE LINES 405-415
.. GENERATED FROM PYTHON SOURCE LINES 403-413
Retrain the searched model
--------------------------
@ -610,17 +631,17 @@ To get a final usable model with trained weights, we need to construct a real mo
and then fully train it.
To construct a fixed model based on the architecture dict exported from the experiment,
we can use :func:`nni.retiarii.fixed_arch`. Under the with-context, we will creating a fixed model based on ``exported_arch``,
we can use :func:`nni.nas.space.model_context`. Under the with-context, we will creating a fixed model based on ``exported_arch``,
instead of creating a space.
.. GENERATED FROM PYTHON SOURCE LINES 416-422
.. GENERATED FROM PYTHON SOURCE LINES 414-420
.. code-block:: default
from nni.retiarii import fixed_arch
from nni.nas.space import model_context
with fixed_arch(exported_arch):
with model_context(exported_arch):
final_model = DartsSpace(width=16, num_cells=8, dataset='cifar')
@ -630,11 +651,11 @@ instead of creating a space.
.. GENERATED FROM PYTHON SOURCE LINES 423-424
.. GENERATED FROM PYTHON SOURCE LINES 421-422
We then train the model on full CIFAR-10 training dataset, and evaluate it on the original CIFAR-10 validation dataset.
.. GENERATED FROM PYTHON SOURCE LINES 425-428
.. GENERATED FROM PYTHON SOURCE LINES 423-426
.. code-block:: default
@ -648,11 +669,11 @@ We then train the model on full CIFAR-10 training dataset, and evaluate it on th
.. GENERATED FROM PYTHON SOURCE LINES 429-430
.. GENERATED FROM PYTHON SOURCE LINES 427-428
The validation data loader can be reused.
.. GENERATED FROM PYTHON SOURCE LINES 431-434
.. GENERATED FROM PYTHON SOURCE LINES 429-432
.. code-block:: default
@ -668,17 +689,17 @@ The validation data loader can be reused.
.. code-block:: none
<torch.utils.data.dataloader.DataLoader object at 0x7f5e187c0430>
<torch.utils.data.dataloader.DataLoader object at 0x7fa2eb2f21c0>
.. GENERATED FROM PYTHON SOURCE LINES 435-438
.. GENERATED FROM PYTHON SOURCE LINES 433-436
We must create a new evaluator here because a different data split is used.
Also, we should avoid the underlying pytorch-lightning implementation of :class:`~nni.retiarii.evaluator.pytorch.Classification`
Also, we should avoid the underlying pytorch-lightning implementation of :class:`~nni.nas.evaluator.pytorch.Classification`
evaluator from loading the wrong checkpoint.
.. GENERATED FROM PYTHON SOURCE LINES 439-455
.. GENERATED FROM PYTHON SOURCE LINES 437-453
.. code-block:: default
@ -706,39 +727,21 @@ evaluator from loading the wrong checkpoint.
.. code-block:: none
/data/data0/jiahang/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:447: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:446: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
rank_zero_deprecation(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Running in `fast_dev_run` mode: will run the requested loop using 1 batch(es). Logging and checkpointing is suppressed.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [3]
| Name | Type | Params
-----------------------------------------------
0 | criterion | CrossEntropyLoss | 0
1 | metrics | ModuleDict | 0
2 | model | DARTS | 345 K
-----------------------------------------------
345 K Trainable params
0 Non-trainable params
345 K Total params
1.381 Total estimated model params size (MB)
/data/data0/jiahang/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1892: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1555: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
rank_zero_warn(
Training: 0it [00:00, ?it/s] Training: 0%| | 0/2 [00:00<?, ?it/s] Epoch 0: 0%| | 0/2 [00:00<?, ?it/s] Epoch 0: 50%|##### | 1/2 [00:00<00:00, 1.02it/s] Epoch 0: 50%|##### | 1/2 [00:00<00:00, 1.02it/s, loss=2.46, v_num=, train_loss=2.460, train_acc=0.0729]
Training: 0it [00:00, ?it/s] Training: 0%| | 0/2 [00:00<?, ?it/s] Epoch 0: 0%| | 0/2 [00:00<?, ?it/s] Epoch 0: 50%|##### | 1/2 [00:00<00:00, 1.77it/s] Epoch 0: 50%|##### | 1/2 [00:00<00:00, 1.77it/s, loss=2.33, v_num=, train_loss=2.330, train_acc=0.104]
Validation: 0it [00:00, ?it/s]
Validation: 0%| | 0/1 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/1 [00:00<?, ?it/s]
Validation DataLoader 0: 100%|##########| 1/1 [00:00<00:00, 11.12it/s] Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.15it/s, loss=2.46, v_num=, train_loss=2.460, train_acc=0.0729] Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.15it/s, loss=2.46, v_num=, train_loss=2.460, train_acc=0.0729, val_loss=2.300, val_acc=0.117]
Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.15it/s, loss=2.46, v_num=, train_loss=2.460, train_acc=0.0729, val_loss=2.300, val_acc=0.117]`Trainer.fit` stopped: `max_steps=1` reached.
Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.15it/s, loss=2.46, v_num=, train_loss=2.460, train_acc=0.0729, val_loss=2.300, val_acc=0.117]
Validation DataLoader 0: 100%|##########| 1/1 [00:00<00:00, 13.64it/s] Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.93it/s, loss=2.33, v_num=, train_loss=2.330, train_acc=0.104] Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.93it/s, loss=2.33, v_num=, train_loss=2.330, train_acc=0.104, val_loss=2.300, val_acc=0.0898]
Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.92it/s, loss=2.33, v_num=, train_loss=2.330, train_acc=0.104, val_loss=2.300, val_acc=0.0898] Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.92it/s, loss=2.33, v_num=, train_loss=2.330, train_acc=0.104, val_loss=2.300, val_acc=0.0898]
.. GENERATED FROM PYTHON SOURCE LINES 456-484
.. GENERATED FROM PYTHON SOURCE LINES 454-482
.. note:: When ``fast_dev_run`` is turned off, we achieve a validation accuracy of 89.69% after training for 100 epochs.
@ -764,18 +767,18 @@ Putting it briefly, the core part of writing a new evaluator is to write a new L
`LightingModule <https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html>`__ is a concept in
PyTorch-Lightning, which organizes the model training process into a list of functions, such as,
``training_step``, ``validation_step``, ``configure_optimizers``, etc.
Since we are merely adding a few ingredients to :class:`~nni.retiarii.evaluator.pytorch.Classification`,
we can simply inherit :class:`~nni.retiarii.evaluator.pytorch.ClassificationModule`, which is the underlying LightningModule
behind :class:`~nni.retiarii.evaluator.pytorch.Classification`.
Since we are merely adding a few ingredients to :class:`~nni.nas.evaluator.pytorch.Classification`,
we can simply inherit :class:`~nni.nas.evaluator.pytorch.ClassificationModule`, which is the underlying LightningModule
behind :class:`~nni.nas.evaluator.pytorch.Classification`.
This could look intimidating at first, but most of them are just plug-and-play tricks which you don't need to know details about.
.. GENERATED FROM PYTHON SOURCE LINES 485-540
.. GENERATED FROM PYTHON SOURCE LINES 483-538
.. code-block:: default
import torch
from nni.retiarii.evaluator.pytorch import ClassificationModule
from nni.nas.evaluator.pytorch import ClassificationModule
class DartsClassificationModule(ClassificationModule):
def __init__(
@ -835,19 +838,19 @@ This could look intimidating at first, but most of them are just plug-and-play t
.. GENERATED FROM PYTHON SOURCE LINES 541-545
.. GENERATED FROM PYTHON SOURCE LINES 539-543
The full evaluator is written as follows,
which simply wraps everything (except model space and search strategy of course), in a single object.
:class:`~nni.retiarii.evaluator.pytorch.Lightning` here is a special type of evaluator.
:class:`~nni.nas.evaluator.pytorch.Lightning` here is a special type of evaluator.
Don't forget to use the train/val data split specialized for search (1:1) here.
.. GENERATED FROM PYTHON SOURCE LINES 546-562
.. GENERATED FROM PYTHON SOURCE LINES 544-560
.. code-block:: default
from nni.retiarii.evaluator.pytorch import Lightning, Trainer
from nni.nas.evaluator.pytorch import Lightning, Trainer
max_epochs = 50
@ -870,29 +873,24 @@ Don't forget to use the train/val data split specialized for search (1:1) here.
.. code-block:: none
/data/data0/jiahang/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:447: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:446: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
rank_zero_deprecation(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Running in `fast_dev_run` mode: will run the requested loop using 1 batch(es). Logging and checkpointing is suppressed.
.. GENERATED FROM PYTHON SOURCE LINES 563-571
.. GENERATED FROM PYTHON SOURCE LINES 561-569
Strategy
^^^^^^^^
:class:`~nni.retiarii.strategy.DARTS` strategy is created with gradient clip turned on.
:class:`~nni.nas.strategy.DARTS` strategy is created with gradient clip turned on.
If you are familiar with PyTorch-Lightning, you might aware that gradient clipping can be enabled in Lightning trainer.
However, enabling gradient clip in the trainer above won't work, because the underlying
implementation of :class:`~nni.retiarii.strategy.DARTS` strategy is based on
implementation of :class:`~nni.nas.strategy.DARTS` strategy is based on
`manual optimization <https://pytorch-lightning.readthedocs.io/en/stable/common/optimization.html>`__.
.. GENERATED FROM PYTHON SOURCE LINES 572-575
.. GENERATED FROM PYTHON SOURCE LINES 570-573
.. code-block:: default
@ -906,7 +904,7 @@ implementation of :class:`~nni.retiarii.strategy.DARTS` strategy is based on
.. GENERATED FROM PYTHON SOURCE LINES 576-585
.. GENERATED FROM PYTHON SOURCE LINES 574-583
Launch experiment
^^^^^^^^^^^^^^^^^
@ -918,18 +916,17 @@ Then we use the newly created evaluator and strategy to launch the experiment ag
``model_space`` has to be re-instantiated because a known limitation,
i.e., one model space instance can't be reused across multiple experiments.
.. GENERATED FROM PYTHON SOURCE LINES 586-597
.. GENERATED FROM PYTHON SOURCE LINES 584-594
.. code-block:: default
model_space = DartsSpace(width=16, num_cells=8, dataset='cifar')
config = RetiariiExeConfig(execution_engine='oneshot')
experiment = RetiariiExperiment(model_space, evaluator=evaluator, strategy=strategy)
experiment.run(config)
experiment = NasExperiment(model_space, evaluator, strategy)
experiment.run()
exported_arch = experiment.export_top_models()[0]
exported_arch = experiment.export_top_models(formatter='dict')[0]
exported_arch
@ -941,30 +938,19 @@ Then we use the newly created evaluator and strategy to launch the experiment ag
.. code-block:: none
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [3]
| Name | Type | Params
----------------------------------------------------
0 | model | DartsClassificationModule | 3.0 M
----------------------------------------------------
3.0 M Trainable params
0 Non-trainable params
3.0 M Total params
12.164 Total estimated model params size (MB)
/data/data0/jiahang/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1892: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1555: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
rank_zero_warn(
Training: 0it [00:00, ?it/s] Training: 0%| | 0/1 [00:00<?, ?it/s] Epoch 0: 0%| | 0/1 [00:00<?, ?it/s] Epoch 0: 100%|##########| 1/1 [01:04<00:00, 64.95s/it] Epoch 0: 100%|##########| 1/1 [01:04<00:00, 64.95s/it, v_num=, train_loss=2.450, train_acc=0.0625] Epoch 0: 100%|##########| 1/1 [01:04<00:00, 64.96s/it, v_num=, train_loss=2.450, train_acc=0.0625]`Trainer.fit` stopped: `max_epochs=1` reached.
Epoch 0: 100%|##########| 1/1 [01:04<00:00, 64.97s/it, v_num=, train_loss=2.450, train_acc=0.0625]
Training: 0it [00:00, ?it/s] Training: 0%| | 0/1 [00:00<?, ?it/s] Epoch 0: 0%| | 0/1 [00:00<?, ?it/s] Epoch 0: 100%|##########| 1/1 [00:02<00:00, 2.84s/it] Epoch 0: 100%|##########| 1/1 [00:02<00:00, 2.84s/it, v_num=, train_loss=2.410, train_acc=0.0781] Epoch 0: 100%|##########| 1/1 [00:02<00:00, 2.85s/it, v_num=, train_loss=2.410, train_acc=0.0781] Epoch 0: 100%|##########| 1/1 [00:02<00:00, 2.86s/it, v_num=, train_loss=2.410, train_acc=0.0781]
{'normal/op_2_0': 'avg_pool_3x3', 'normal/input_2_0': 0, 'normal/op_2_1': 'avg_pool_3x3', 'normal/input_2_1': 1, 'normal/op_3_0': 'sep_conv_5x5', 'normal/input_3_0': 2, 'normal/op_3_1': 'avg_pool_3x3', 'normal/input_3_1': 0, 'normal/op_4_0': 'dil_conv_3x3', 'normal/input_4_0': 2, 'normal/op_4_1': 'sep_conv_3x3', 'normal/input_4_1': 0, 'normal/op_5_0': 'avg_pool_3x3', 'normal/input_5_0': 2, 'normal/op_5_1': 'dil_conv_5x5', 'normal/input_5_1': 4, 'reduce/op_2_0': 'sep_conv_3x3', 'reduce/input_2_0': 1, 'reduce/op_2_1': 'sep_conv_5x5', 'reduce/input_2_1': 0, 'reduce/op_3_0': 'avg_pool_3x3', 'reduce/input_3_0': 2, 'reduce/op_3_1': 'sep_conv_3x3', 'reduce/input_3_1': 0, 'reduce/op_4_0': 'max_pool_3x3', 'reduce/input_4_0': 1, 'reduce/op_4_1': 'dil_conv_5x5', 'reduce/input_4_1': 2, 'reduce/op_5_0': 'dil_conv_3x3', 'reduce/input_5_0': 3, 'reduce/op_5_1': 'max_pool_3x3', 'reduce/input_5_1': 4}
{'normal/op_2_0': 'dil_conv_5x5', 'normal/input_2_0': [0], 'normal/op_2_1': 'dil_conv_5x5', 'normal/input_2_1': [1], 'normal/op_3_0': 'sep_conv_3x3', 'normal/input_3_0': [0], 'normal/op_3_1': 'sep_conv_5x5', 'normal/input_3_1': [2], 'normal/op_4_0': 'avg_pool_3x3', 'normal/input_4_0': [3], 'normal/op_4_1': 'dil_conv_3x3', 'normal/input_4_1': [2], 'normal/op_5_0': 'avg_pool_3x3', 'normal/input_5_0': [0], 'normal/op_5_1': 'sep_conv_5x5', 'normal/input_5_1': [2], 'reduce/op_2_0': 'dil_conv_3x3', 'reduce/input_2_0': [1], 'reduce/op_2_1': 'dil_conv_5x5', 'reduce/input_2_1': [0], 'reduce/op_3_0': 'skip_connect', 'reduce/input_3_0': [0], 'reduce/op_3_1': 'avg_pool_3x3', 'reduce/input_3_1': [1], 'reduce/op_4_0': 'max_pool_3x3', 'reduce/input_4_0': [2], 'reduce/op_4_1': 'max_pool_3x3', 'reduce/input_4_1': [3], 'reduce/op_5_0': 'dil_conv_3x3', 'reduce/input_5_0': [4], 'reduce/op_5_1': 'dil_conv_5x5', 'reduce/input_5_1': [3]}
.. GENERATED FROM PYTHON SOURCE LINES 598-599
.. GENERATED FROM PYTHON SOURCE LINES 595-596
We get the following architecture when ``fast_dev_run`` is set to False. It takes around 8 hours on a P100 GPU.
.. GENERATED FROM PYTHON SOURCE LINES 600-636
.. GENERATED FROM PYTHON SOURCE LINES 597-633
.. code-block:: default
@ -1016,7 +1002,7 @@ We get the following architecture when ``fast_dev_run`` is set to False. It take
.. GENERATED FROM PYTHON SOURCE LINES 637-644
.. GENERATED FROM PYTHON SOURCE LINES 634-641
Retrain
^^^^^^^
@ -1026,7 +1012,7 @@ we extend the original dataloader to introduce another trick called `Cutout <htt
Cutout is a data augmentation technique that randomly masks out rectangular regions in images.
In CIFAR-10, the typical masked size is 16x16 (the image sizes are 32x32 in the dataset).
.. GENERATED FROM PYTHON SOURCE LINES 645-671
.. GENERATED FROM PYTHON SOURCE LINES 642-668
.. code-block:: default
@ -1063,12 +1049,12 @@ In CIFAR-10, the typical masked size is 16x16 (the image sizes are 32x32 in the
.. GENERATED FROM PYTHON SOURCE LINES 672-674
.. GENERATED FROM PYTHON SOURCE LINES 669-671
The train dataloader needs to be reinstantiated with the new transform.
The validation dataloader is not affected, and thus can be reused.
.. GENERATED FROM PYTHON SOURCE LINES 675-679
.. GENERATED FROM PYTHON SOURCE LINES 672-676
.. code-block:: default
@ -1089,7 +1075,7 @@ The validation dataloader is not affected, and thus can be reused.
.. GENERATED FROM PYTHON SOURCE LINES 680-685
.. GENERATED FROM PYTHON SOURCE LINES 677-682
We then create the final model based on the new exported architecture.
This time, auxiliary loss and drop path probability is enabled.
@ -1097,12 +1083,12 @@ This time, auxiliary loss and drop path probability is enabled.
Following the same procedure as paper, we also increase the number of filters to 36, and number of cells to 20,
so as to reasonably increase the model size and boost the performance.
.. GENERATED FROM PYTHON SOURCE LINES 686-690
.. GENERATED FROM PYTHON SOURCE LINES 683-687
.. code-block:: default
with fixed_arch(exported_arch):
with model_context(exported_arch):
final_model = DartsSpace(width=36, num_cells=20, dataset='cifar', auxiliary_loss=True, drop_path_prob=0.2)
@ -1112,11 +1098,11 @@ so as to reasonably increase the model size and boost the performance.
.. GENERATED FROM PYTHON SOURCE LINES 691-692
.. GENERATED FROM PYTHON SOURCE LINES 688-689
We create a new evaluator for the retraining process, where the gradient clipping is put into the keyword arguments of trainer.
.. GENERATED FROM PYTHON SOURCE LINES 693-710
.. GENERATED FROM PYTHON SOURCE LINES 690-707
.. code-block:: default
@ -1145,43 +1131,23 @@ We create a new evaluator for the retraining process, where the gradient clippin
.. code-block:: none
/data/data0/jiahang/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:447: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/accelerator_connector.py:446: LightningDeprecationWarning: Setting `Trainer(gpus=1)` is deprecated in v1.7 and will be removed in v2.0. Please use `Trainer(accelerator='gpu', devices=1)` instead.
rank_zero_deprecation(
GPU available: True (cuda), used: True
TPU available: False, using: 0 TPU cores
IPU available: False, using: 0 IPUs
HPU available: False, using: 0 HPUs
Running in `fast_dev_run` mode: will run the requested loop using 1 batch(es). Logging and checkpointing is suppressed.
LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [3]
| Name | Type | Params
-----------------------------------------------
0 | criterion | CrossEntropyLoss | 0
1 | metrics | ModuleDict | 0
2 | model | DARTS | 3.2 M
-----------------------------------------------
3.2 M Trainable params
0 Non-trainable params
3.2 M Total params
12.942 Total estimated model params size (MB)
/data/data0/jiahang/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:225: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 56 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:224: PossibleUserWarning: The dataloader, train_dataloader, does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` (try 16 which is the number of cpus on this machine) in the `DataLoader` init to improve performance.
rank_zero_warn(
/data/data0/jiahang/miniconda3/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1892: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py:1555: PossibleUserWarning: The number of training batches (1) is smaller than the logging interval Trainer(log_every_n_steps=50). Set a lower value for log_every_n_steps if you want to see logs for the training epoch.
rank_zero_warn(
Training: 0it [00:00, ?it/s] Training: 0%| | 0/2 [00:00<?, ?it/s] Epoch 0: 0%| | 0/2 [00:00<?, ?it/s] /data/data0/jiahang/miniconda3/lib/python3.8/site-packages/torchvision/transforms/functional_pil.py:41: DeprecationWarning: FLIP_LEFT_RIGHT is deprecated and will be removed in Pillow 10 (2023-07-01). Use Transpose.FLIP_LEFT_RIGHT instead.
return img.transpose(Image.FLIP_LEFT_RIGHT)
Epoch 0: 50%|##### | 1/2 [00:00<00:00, 1.33it/s] Epoch 0: 50%|##### | 1/2 [00:00<00:00, 1.33it/s, loss=3.47, v_num=, train_loss=3.470, train_acc=0.0625]
Training: 0it [00:00, ?it/s] Training: 0%| | 0/2 [00:00<?, ?it/s] Epoch 0: 0%| | 0/2 [00:00<?, ?it/s] Epoch 0: 50%|##### | 1/2 [00:00<00:00, 1.77it/s] Epoch 0: 50%|##### | 1/2 [00:00<00:00, 1.77it/s, loss=3.36, v_num=, train_loss=3.360, train_acc=0.0729]
Validation: 0it [00:00, ?it/s]
Validation: 0%| | 0/1 [00:00<?, ?it/s]
Validation DataLoader 0: 0%| | 0/1 [00:00<?, ?it/s]
Validation DataLoader 0: 100%|##########| 1/1 [00:00<00:00, 3.13it/s] Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.20it/s, loss=3.47, v_num=, train_loss=3.470, train_acc=0.0625] Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.20it/s, loss=3.47, v_num=, train_loss=3.470, train_acc=0.0625, val_loss=2.300, val_acc=0.0938]
Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.19it/s, loss=3.47, v_num=, train_loss=3.470, train_acc=0.0625, val_loss=2.300, val_acc=0.0938]`Trainer.fit` stopped: `max_steps=1` reached.
Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.19it/s, loss=3.47, v_num=, train_loss=3.470, train_acc=0.0625, val_loss=2.300, val_acc=0.0938]
Validation DataLoader 0: 100%|##########| 1/1 [00:00<00:00, 1.55it/s] Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.26it/s, loss=3.36, v_num=, train_loss=3.360, train_acc=0.0729] Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.26it/s, loss=3.36, v_num=, train_loss=3.360, train_acc=0.0729, val_loss=2.300, val_acc=0.0938]
Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.25it/s, loss=3.36, v_num=, train_loss=3.360, train_acc=0.0729, val_loss=2.300, val_acc=0.0938] Epoch 0: 100%|##########| 2/2 [00:01<00:00, 1.25it/s, loss=3.36, v_num=, train_loss=3.360, train_acc=0.0729, val_loss=2.300, val_acc=0.0938]
.. GENERATED FROM PYTHON SOURCE LINES 711-725
.. GENERATED FROM PYTHON SOURCE LINES 708-722
When ``fast_dev_run`` is turned off, after retraining, the architecture yields a top-1 accuracy of 97.12%.
If we take the best snapshot throughout the retrain process,
@ -1201,7 +1167,7 @@ The implementation of second order DARTS is in our future plan, and we also welc
.. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 1 minutes 53.716 seconds)
**Total running time of the script:** ( 0 minutes 40.472 seconds)
.. _sphx_glr_download_tutorials_darts.py:
@ -1211,6 +1177,8 @@ The implementation of second order DARTS is in our future plan, and we also welc
.. container:: sphx-glr-footer sphx-glr-footer-example
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: darts.py <darts.py>`

Двоичные данные
docs/source/tutorials/darts_codeobj.pickle сгенерированный
Просмотреть файл

Двоичный файл не отображается.

74
docs/source/tutorials/hello_nas.ipynb сгенерированный
Просмотреть файл

@ -15,14 +15,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"\n# Hello, NAS!\n\nThis is the 101 tutorial of Neural Architecture Search (NAS) on NNI.\nIn this tutorial, we will search for a neural architecture on MNIST dataset with the help of NAS framework of NNI, i.e., *Retiarii*.\nWe use multi-trial NAS as an example to show how to construct and explore a model space.\n\nThere are mainly three crucial components for a neural architecture search task, namely,\n\n* Model search space that defines a set of models to explore.\n* A proper strategy as the method to explore this model space.\n* A model evaluator that reports the performance of every model in the space.\n\nCurrently, PyTorch is the only supported framework by Retiarii, and we have only tested **PyTorch 1.7 to 1.10**.\nThis tutorial assumes PyTorch context but it should also apply to other frameworks, which is in our future plan.\n\n## Define your Model Space\n\nModel space is defined by users to express a set of models that users want to explore, which contains potentially good-performing models.\nIn this framework, a model space is defined with two parts: a base model and possible mutations on the base model.\n"
"\n# Hello, NAS!\n\nThis is the 101 tutorial of Neural Architecture Search (NAS) on NNI.\nIn this tutorial, we will search for a neural architecture on MNIST dataset with the help of NAS framework of NNI, i.e., *Retiarii*.\nWe use multi-trial NAS as an example to show how to construct and explore a model space.\n\nThere are mainly three crucial components for a neural architecture search task, namely,\n\n* Model search space that defines a set of models to explore.\n* A proper strategy as the method to explore this model space.\n* A model evaluator that reports the performance of every model in the space.\n\nCurrently, PyTorch is the only supported framework by Retiarii, and we have only tested **PyTorch 1.9 to 1.13**.\nThis tutorial assumes PyTorch context but it should also apply to other frameworks, which is in our future plan.\n\n## Define your Model Space\n\nModel space is defined by users to express a set of models that users want to explore, which contains potentially good-performing models.\nIn this framework, a model space is defined with two parts: a base model and possible mutations on the base model.\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Define Base Model\n\nDefining a base model is almost the same as defining a PyTorch (or TensorFlow) model.\nUsually, you only need to replace the code ``import torch.nn as nn`` with\n``import nni.retiarii.nn.pytorch as nn`` to use our wrapped PyTorch modules.\n\nBelow is a very simple example of defining a base model.\n\n"
"### Define Base Model\n\nDefining a base model is almost the same as defining a PyTorch (or TensorFlow) model.\n\nBelow is a very simple example of defining a base model.\n\n"
]
},
{
@ -33,14 +33,14 @@
},
"outputs": [],
"source": [
"import torch\nimport torch.nn.functional as F\nimport nni.retiarii.nn.pytorch as nn\nfrom nni.retiarii import model_wrapper\n\n\n@model_wrapper # this decorator should be put on the out most\nclass Net(nn.Module):\n def __init__(self):\n super().__init__()\n self.conv1 = nn.Conv2d(1, 32, 3, 1)\n self.conv2 = nn.Conv2d(32, 64, 3, 1)\n self.dropout1 = nn.Dropout(0.25)\n self.dropout2 = nn.Dropout(0.5)\n self.fc1 = nn.Linear(9216, 128)\n self.fc2 = nn.Linear(128, 10)\n\n def forward(self, x):\n x = F.relu(self.conv1(x))\n x = F.max_pool2d(self.conv2(x), 2)\n x = torch.flatten(self.dropout1(x), 1)\n x = self.fc2(self.dropout2(F.relu(self.fc1(x))))\n output = F.log_softmax(x, dim=1)\n return output"
"import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nimport nni\nfrom nni.nas.nn.pytorch import LayerChoice, ModelSpace, MutableDropout, MutableLinear\n\n\nclass Net(ModelSpace): # should inherit ModelSpace rather than nn.Module\n def __init__(self):\n super().__init__()\n self.conv1 = nn.Conv2d(1, 32, 3, 1)\n self.conv2 = nn.Conv2d(32, 64, 3, 1)\n self.dropout1 = nn.Dropout(0.25)\n self.dropout2 = nn.Dropout(0.5)\n self.fc1 = nn.Linear(9216, 128)\n self.fc2 = nn.Linear(128, 10)\n\n def forward(self, x):\n x = F.relu(self.conv1(x))\n x = F.max_pool2d(self.conv2(x), 2)\n x = torch.flatten(self.dropout1(x), 1)\n x = self.fc2(self.dropout2(F.relu(self.fc1(x))))\n output = F.log_softmax(x, dim=1)\n return output"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
".. tip:: Always keep in mind that you should use ``import nni.retiarii.nn.pytorch as nn`` and :meth:`nni.retiarii.model_wrapper`.\n Many mistakes are a result of forgetting one of those.\n Also, please use ``torch.nn`` for submodules of ``nn.init``, e.g., ``torch.nn.init`` instead of ``nn.init``.\n\n### Define Model Mutations\n\nA base model is only one concrete model not a model space. We provide :doc:`API and Primitives </nas/construct_space>`\nfor users to express how the base model can be mutated. That is, to build a model space which includes many models.\n\nBased on the above base model, we can define a model space as below.\n\n.. code-block:: diff\n\n @model_wrapper\n class Net(nn.Module):\n def __init__(self):\n super().__init__()\n self.conv1 = nn.Conv2d(1, 32, 3, 1)\n - self.conv2 = nn.Conv2d(32, 64, 3, 1)\n + self.conv2 = nn.LayerChoice([\n + nn.Conv2d(32, 64, 3, 1),\n + DepthwiseSeparableConv(32, 64)\n + ])\n - self.dropout1 = nn.Dropout(0.25)\n + self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75]))\n self.dropout2 = nn.Dropout(0.5)\n - self.fc1 = nn.Linear(9216, 128)\n - self.fc2 = nn.Linear(128, 10)\n + feature = nn.ValueChoice([64, 128, 256])\n + self.fc1 = nn.Linear(9216, feature)\n + self.fc2 = nn.Linear(feature, 10)\n\n def forward(self, x):\n x = F.relu(self.conv1(x))\n x = F.max_pool2d(self.conv2(x), 2)\n x = torch.flatten(self.dropout1(x), 1)\n x = self.fc2(self.dropout2(F.relu(self.fc1(x))))\n output = F.log_softmax(x, dim=1)\n return output\n\nThis results in the following code:\n\n"
"### Define Model Variations\n\nA base model is only one concrete model not a model space. We provide :doc:`API and Primitives </nas/construct_space>`\nfor users to express how the base model can be mutated. That is, to build a model space which includes many models.\n\nBased on the above base model, we can define a model space as below.\n\n```diff\nclass Net(ModelSpace):\n def __init__(self):\n super().__init__()\n self.conv1 = nn.Conv2d(1, 32, 3, 1)\n- self.conv2 = nn.Conv2d(32, 64, 3, 1)\n+ self.conv2 = LayerChoice([\n+ nn.Conv2d(32, 64, 3, 1),\n+ DepthwiseSeparableConv(32, 64)\n+ ], label='conv2)\n- self.dropout1 = nn.Dropout(0.25)\n+ self.dropout1 = MutableDropout(nni.choice('dropout', [0.25, 0.5, 0.75]))\n self.dropout2 = nn.Dropout(0.5)\n- self.fc1 = nn.Linear(9216, 128)\n- self.fc2 = nn.Linear(128, 10)\n+ feature = nni.choice('feature', [64, 128, 256])\n+ self.fc1 = MutableLinear(9216, feature)\n+ self.fc2 = MutableLinear(feature, 10)\n\n def forward(self, x):\n x = F.relu(self.conv1(x))\n x = F.max_pool2d(self.conv2(x), 2)\n x = torch.flatten(self.dropout1(x), 1)\n x = self.fc2(self.dropout2(F.relu(self.fc1(x))))\n output = F.log_softmax(x, dim=1)\n return output\n```\nThis results in the following code:\n\n"
]
},
{
@ -51,14 +51,14 @@
},
"outputs": [],
"source": [
"class DepthwiseSeparableConv(nn.Module):\n def __init__(self, in_ch, out_ch):\n super().__init__()\n self.depthwise = nn.Conv2d(in_ch, in_ch, kernel_size=3, groups=in_ch)\n self.pointwise = nn.Conv2d(in_ch, out_ch, kernel_size=1)\n\n def forward(self, x):\n return self.pointwise(self.depthwise(x))\n\n\n@model_wrapper\nclass ModelSpace(nn.Module):\n def __init__(self):\n super().__init__()\n self.conv1 = nn.Conv2d(1, 32, 3, 1)\n # LayerChoice is used to select a layer between Conv2d and DwConv.\n self.conv2 = nn.LayerChoice([\n nn.Conv2d(32, 64, 3, 1),\n DepthwiseSeparableConv(32, 64)\n ])\n # ValueChoice is used to select a dropout rate.\n # ValueChoice can be used as parameter of modules wrapped in `nni.retiarii.nn.pytorch`\n # or customized modules wrapped with `@basic_unit`.\n self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75])) # choose dropout rate from 0.25, 0.5 and 0.75\n self.dropout2 = nn.Dropout(0.5)\n feature = nn.ValueChoice([64, 128, 256])\n self.fc1 = nn.Linear(9216, feature)\n self.fc2 = nn.Linear(feature, 10)\n\n def forward(self, x):\n x = F.relu(self.conv1(x))\n x = F.max_pool2d(self.conv2(x), 2)\n x = torch.flatten(self.dropout1(x), 1)\n x = self.fc2(self.dropout2(F.relu(self.fc1(x))))\n output = F.log_softmax(x, dim=1)\n return output\n\n\nmodel_space = ModelSpace()\nmodel_space"
"class DepthwiseSeparableConv(nn.Module):\n def __init__(self, in_ch, out_ch):\n super().__init__()\n self.depthwise = nn.Conv2d(in_ch, in_ch, kernel_size=3, groups=in_ch)\n self.pointwise = nn.Conv2d(in_ch, out_ch, kernel_size=1)\n\n def forward(self, x):\n return self.pointwise(self.depthwise(x))\n\n\nclass MyModelSpace(ModelSpace):\n def __init__(self):\n super().__init__()\n self.conv1 = nn.Conv2d(1, 32, 3, 1)\n # LayerChoice is used to select a layer between Conv2d and DwConv.\n self.conv2 = LayerChoice([\n nn.Conv2d(32, 64, 3, 1),\n DepthwiseSeparableConv(32, 64)\n ], label='conv2')\n # nni.choice is used to select a dropout rate.\n # The result can be used as parameters of `MutableXXX`.\n self.dropout1 = MutableDropout(nni.choice('dropout', [0.25, 0.5, 0.75])) # choose dropout rate from 0.25, 0.5 and 0.75\n self.dropout2 = nn.Dropout(0.5)\n feature = nni.choice('feature', [64, 128, 256])\n self.fc1 = MutableLinear(9216, feature)\n self.fc2 = MutableLinear(feature, 10)\n\n def forward(self, x):\n x = F.relu(self.conv1(x))\n x = F.max_pool2d(self.conv2(x), 2)\n x = torch.flatten(self.dropout1(x), 1)\n x = self.fc2(self.dropout2(F.relu(self.fc1(x))))\n output = F.log_softmax(x, dim=1)\n return output\n\n\nmodel_space = MyModelSpace()\nmodel_space"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"This example uses two mutation APIs,\n:class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>` and\n:class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>`.\n:class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`\ntakes a list of candidate modules (two in this example), one will be chosen for each sampled model.\nIt can be used like normal PyTorch module.\n:class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>` takes a list of candidate values,\none will be chosen to take effect for each sampled model.\n\nMore detailed API description and usage can be found :doc:`here </nas/construct_space>`.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>We are actively enriching the mutation APIs, to facilitate easy construction of model space.\n If the currently supported mutation APIs cannot express your model space,\n please refer to :doc:`this doc </nas/mutator>` for customizing mutators.</p></div>\n\n## Explore the Defined Model Space\n\nThere are basically two exploration approaches: (1) search by evaluating each sampled model independently,\nwhich is the search approach in `multi-trial NAS <multi-trial-nas>`\nand (2) one-shot weight-sharing based search, which is used in one-shot NAS.\nWe demonstrate the first approach in this tutorial. Users can refer to `here <one-shot-nas>` for the second approach.\n\nFirst, users need to pick a proper exploration strategy to explore the defined model space.\nSecond, users need to pick or customize a model evaluator to evaluate the performance of each explored model.\n\n### Pick an exploration strategy\n\nRetiarii supports many :doc:`exploration strategies </nas/exploration_strategy>`.\n\nSimply choosing (i.e., instantiate) an exploration strategy as below.\n\n"
"This example uses two mutation APIs,\n:class:`nn.LayerChoice <nni.nas.nn.pytorch.LayerChoice>` and\n:func:`nni.choice`.\n:class:`nn.LayerChoice <nni.nas.nn.pytorch.LayerChoice>`\ntakes a list of candidate modules (two in this example), one will be chosen for each sampled model.\nIt can be used like normal PyTorch module.\n:func:`nni.choice` is used as parameter of `MutableDropout`, which then takes the result as dropout rate.\n\nMore detailed API description and usage can be found :doc:`here </nas/construct_space>`.\n\n<div class=\"alert alert-info\"><h4>Note</h4><p>We are actively enriching the mutation APIs, to facilitate easy construction of model space.\n If the currently supported mutation APIs cannot express your model space,\n please refer to :doc:`this doc </nas/mutator>` for customizing mutators.</p></div>\n\n## Explore the Defined Model Space\n\nThere are basically two exploration approaches: (1) search by evaluating each sampled model independently,\nwhich is the search approach in `multi-trial NAS <multi-trial-nas>`\nand (2) one-shot weight-sharing based search, which is used in one-shot NAS.\nWe demonstrate the first approach in this tutorial. Users can refer to `here <one-shot-nas>` for the second approach.\n\nFirst, users need to pick a proper exploration strategy to explore the defined model space.\nSecond, users need to pick or customize a model evaluator to evaluate the performance of each explored model.\n\n### Pick an exploration strategy\n\nNNI NAS supports many :doc:`exploration strategies </nas/exploration_strategy>`.\n\nSimply choosing (i.e., instantiate) an exploration strategy as below.\n\n"
]
},
{
@ -69,14 +69,14 @@
},
"outputs": [],
"source": [
"import nni.retiarii.strategy as strategy\nsearch_strategy = strategy.Random(dedup=True) # dedup=False if deduplication is not wanted"
"import nni.nas.strategy as strategy\nsearch_strategy = strategy.Random() # dedup=False if deduplication is not wanted"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Pick or customize a model evaluator\n\nIn the exploration process, the exploration strategy repeatedly generates new models. A model evaluator is for training\nand validating each generated model to obtain the model's performance.\nThe performance is sent to the exploration strategy for the strategy to generate better models.\n\nRetiarii has provided :doc:`built-in model evaluators </nas/evaluator>`, but to start with,\nit is recommended to use :class:`FunctionalEvaluator <nni.retiarii.evaluator.FunctionalEvaluator>`,\nthat is, to wrap your own training and evaluation code with one single function.\nThis function should receive one single model class and uses :func:`nni.report_final_result` to report the final score of this model.\n\nAn example here creates a simple evaluator that runs on MNIST dataset, trains for 2 epochs, and reports its validation accuracy.\n\n"
"### Pick or customize a model evaluator\n\nIn the exploration process, the exploration strategy repeatedly generates new models. A model evaluator is for training\nand validating each generated model to obtain the model's performance.\nThe performance is sent to the exploration strategy for the strategy to generate better models.\n\nNNI NAS has provided :doc:`built-in model evaluators </nas/evaluator>`, but to start with,\nit is recommended to use :class:`FunctionalEvaluator <nni.nas.evaluator.FunctionalEvaluator>`,\nthat is, to wrap your own training and evaluation code with one single function.\nThis function should receive one single model class and uses :func:`nni.report_final_result` to report the final score of this model.\n\nAn example here creates a simple evaluator that runs on MNIST dataset, trains for 2 epochs, and reports its validation accuracy.\n\n"
]
},
{
@ -87,7 +87,7 @@
},
"outputs": [],
"source": [
"import nni\n\nfrom torchvision import transforms\nfrom torchvision.datasets import MNIST\nfrom torch.utils.data import DataLoader\n\n\ndef train_epoch(model, device, train_loader, optimizer, epoch):\n loss_fn = torch.nn.CrossEntropyLoss()\n model.train()\n for batch_idx, (data, target) in enumerate(train_loader):\n data, target = data.to(device), target.to(device)\n optimizer.zero_grad()\n output = model(data)\n loss = loss_fn(output, target)\n loss.backward()\n optimizer.step()\n if batch_idx % 10 == 0:\n print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n epoch, batch_idx * len(data), len(train_loader.dataset),\n 100. * batch_idx / len(train_loader), loss.item()))\n\n\ndef test_epoch(model, device, test_loader):\n model.eval()\n test_loss = 0\n correct = 0\n with torch.no_grad():\n for data, target in test_loader:\n data, target = data.to(device), target.to(device)\n output = model(data)\n pred = output.argmax(dim=1, keepdim=True)\n correct += pred.eq(target.view_as(pred)).sum().item()\n\n test_loss /= len(test_loader.dataset)\n accuracy = 100. * correct / len(test_loader.dataset)\n\n print('\\nTest set: Accuracy: {}/{} ({:.0f}%)\\n'.format(\n correct, len(test_loader.dataset), accuracy))\n\n return accuracy\n\n\ndef evaluate_model(model_cls):\n # \"model_cls\" is a class, need to instantiate\n model = model_cls()\n\n device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n model.to(device)\n\n optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)\n transf = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])\n train_loader = DataLoader(MNIST('data/mnist', download=True, transform=transf), batch_size=64, shuffle=True)\n test_loader = DataLoader(MNIST('data/mnist', download=True, train=False, transform=transf), batch_size=64)\n\n for epoch in range(3):\n # train the model for one epoch\n train_epoch(model, device, train_loader, optimizer, epoch)\n # test the model for one epoch\n accuracy = test_epoch(model, device, test_loader)\n # call report intermediate result. Result can be float or dict\n nni.report_intermediate_result(accuracy)\n\n # report final test result\n nni.report_final_result(accuracy)"
"import nni\n\nfrom torchvision import transforms\nfrom torchvision.datasets import MNIST\nfrom torch.utils.data import DataLoader\n\n\ndef train_epoch(model, device, train_loader, optimizer, epoch):\n loss_fn = torch.nn.CrossEntropyLoss()\n model.train()\n for batch_idx, (data, target) in enumerate(train_loader):\n data, target = data.to(device), target.to(device)\n optimizer.zero_grad()\n output = model(data)\n loss = loss_fn(output, target)\n loss.backward()\n optimizer.step()\n if batch_idx % 10 == 0:\n print('Train Epoch: {} [{}/{} ({:.0f}%)]\\tLoss: {:.6f}'.format(\n epoch, batch_idx * len(data), len(train_loader.dataset),\n 100. * batch_idx / len(train_loader), loss.item()))\n\n\ndef test_epoch(model, device, test_loader):\n model.eval()\n test_loss = 0\n correct = 0\n with torch.no_grad():\n for data, target in test_loader:\n data, target = data.to(device), target.to(device)\n output = model(data)\n pred = output.argmax(dim=1, keepdim=True)\n correct += pred.eq(target.view_as(pred)).sum().item()\n\n test_loss /= len(test_loader.dataset)\n accuracy = 100. * correct / len(test_loader.dataset)\n\n print('\\nTest set: Accuracy: {}/{} ({:.0f}%)\\n'.format(\n correct, len(test_loader.dataset), accuracy))\n\n return accuracy\n\n\ndef evaluate_model(model):\n # By v3.0, the model will be instantiated by default.\n device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')\n model.to(device)\n\n optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)\n transf = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])\n train_loader = DataLoader(MNIST('data/mnist', download=True, transform=transf), batch_size=64, shuffle=True)\n test_loader = DataLoader(MNIST('data/mnist', download=True, train=False, transform=transf), batch_size=64)\n\n for epoch in range(3):\n # train the model for one epoch\n train_epoch(model, device, train_loader, optimizer, epoch)\n # test the model for one epoch\n accuracy = test_epoch(model, device, test_loader)\n # call report intermediate result. Result can be float or dict\n nni.report_intermediate_result(accuracy)\n\n # report final test result\n nni.report_final_result(accuracy)"
]
},
{
@ -105,14 +105,14 @@
},
"outputs": [],
"source": [
"from nni.retiarii.evaluator import FunctionalEvaluator\nevaluator = FunctionalEvaluator(evaluate_model)"
"from nni.nas.evaluator import FunctionalEvaluator\nevaluator = FunctionalEvaluator(evaluate_model)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The ``train_epoch`` and ``test_epoch`` here can be any customized function,\nwhere users can write their own training recipe.\n\nIt is recommended that the ``evaluate_model`` here accepts no additional arguments other than ``model_cls``.\nHowever, in the :doc:`advanced tutorial </nas/evaluator>`, we will show how to use additional arguments in case you actually need those.\nIn future, we will support mutation on the arguments of evaluators, which is commonly called \"Hyper-parmeter tuning\".\n\n## Launch an Experiment\n\nAfter all the above are prepared, it is time to start an experiment to do the model search. An example is shown below.\n\n"
"The ``train_epoch`` and ``test_epoch`` here can be any customized function,\nwhere users can write their own training recipe.\n\nIt is recommended that the ``evaluate_model`` here accepts no additional arguments other than ``model``.\nHowever, in the :doc:`advanced tutorial </nas/evaluator>`, we will show how to use additional arguments in case you actually need those.\nIn future, we will support mutation on the arguments of evaluators, which is commonly called \"Hyper-parameter tuning\".\n\n## Launch an Experiment\n\nAfter all the above are prepared, it is time to start an experiment to do the model search. An example is shown below.\n\n"
]
},
{
@ -123,14 +123,14 @@
},
"outputs": [],
"source": [
"from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig\nexp = RetiariiExperiment(model_space, evaluator, [], search_strategy)\nexp_config = RetiariiExeConfig('local')\nexp_config.experiment_name = 'mnist_search'"
"from nni.nas.experiment import NasExperiment\nexp = NasExperiment(model_space, evaluator, search_strategy)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The following configurations are useful to control how many trials to run at most / at the same time.\n\n"
"Different from HPO experiment, NAS experiment will generate an experiment config automatically.\nIt should work for most cases. For example, when using multi-trial strategies,\nlocal training service with concurrency 1 will be used by default.\nUsers can customize the config. For example,\n\n"
]
},
{
@ -141,14 +141,14 @@
},
"outputs": [],
"source": [
"exp_config.max_trial_number = 4 # spawn 4 trials at most\nexp_config.trial_concurrency = 2 # will run two trials concurrently"
"exp.config.max_trial_number = 3 # spawn 3 trials at most\nexp.config.trial_concurrency = 1 # will run 1 trial concurrently\nexp.config.trial_gpu_number = 0 # will not use GPU"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Remember to set the following config if you want to GPU.\n``use_active_gpu`` should be set true if you wish to use an occupied GPU (possibly running a GUI).\n\n"
"Remember to set the following config if you want to GPU.\n``use_active_gpu`` should be set true if you wish to use an occupied GPU (possibly running a GUI)::\n\n exp.config.trial_gpu_number = 1\n exp.config.training_service.use_active_gpu = True\n\nLaunch the experiment. The experiment should take several minutes to finish on a workstation with 2 GPUs.\n\n"
]
},
{
@ -159,14 +159,14 @@
},
"outputs": [],
"source": [
"exp_config.trial_gpu_number = 1\nexp_config.training_service.use_active_gpu = True"
"exp.run(port=8081)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Launch the experiment. The experiment should take several minutes to finish on a workstation with 2 GPUs.\n\n"
"Users can also run NAS Experiment with :doc:`different training services </experiment/training_service/overview>`\nbesides ``local`` training service.\n\n## Visualize the Experiment\n\nUsers can visualize their experiment in the same way as visualizing a normal hyper-parameter tuning experiment.\nFor example, open ``localhost:8081`` in your browser, 8081 is the port that you set in ``exp.run``.\nPlease refer to :doc:`here </experiment/web_portal/web_portal>` for details.\n\nWe support visualizing models with 3rd-party visualization engines (like [Netron](https://netron.app/)_).\nThis can be used by clicking ``Visualization`` in detail panel for each trial.\nNote that current visualization is based on [onnx](https://onnx.ai/)_ ,\nthus visualization is not feasible if the model cannot be exported into onnx.\n\nBuilt-in evaluators (e.g., Classification) will automatically export the model into a file.\nFor your own evaluator, you need to save your file into ``$NNI_OUTPUT_DIR/model.onnx`` to make this work.\nFor instance,\n\n"
]
},
{
@ -177,25 +177,7 @@
},
"outputs": [],
"source": [
"exp.run(exp_config, 8081)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"Users can also run Retiarii Experiment with :doc:`different training services </experiment/training_service/overview>`\nbesides ``local`` training service.\n\n## Visualize the Experiment\n\nUsers can visualize their experiment in the same way as visualizing a normal hyper-parameter tuning experiment.\nFor example, open ``localhost:8081`` in your browser, 8081 is the port that you set in ``exp.run``.\nPlease refer to :doc:`here </experiment/web_portal/web_portal>` for details.\n\nWe support visualizing models with 3rd-party visualization engines (like `Netron <https://netron.app/>`__).\nThis can be used by clicking ``Visualization`` in detail panel for each trial.\nNote that current visualization is based on `onnx <https://onnx.ai/>`__ ,\nthus visualization is not feasible if the model cannot be exported into onnx.\n\nBuilt-in evaluators (e.g., Classification) will automatically export the model into a file.\nFor your own evaluator, you need to save your file into ``$NNI_OUTPUT_DIR/model.onnx`` to make this work.\nFor instance,\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"import os\nfrom pathlib import Path\n\n\ndef evaluate_model_with_visualization(model_cls):\n model = model_cls()\n # dump the model into an onnx\n if 'NNI_OUTPUT_DIR' in os.environ:\n dummy_input = torch.zeros(1, 3, 32, 32)\n torch.onnx.export(model, (dummy_input, ),\n Path(os.environ['NNI_OUTPUT_DIR']) / 'model.onnx')\n evaluate_model(model_cls)"
"import os\nfrom pathlib import Path\n\n\ndef evaluate_model_with_visualization(model):\n # dump the model into an onnx\n if 'NNI_OUTPUT_DIR' in os.environ:\n dummy_input = torch.zeros(1, 3, 32, 32)\n torch.onnx.export(model, (dummy_input, ),\n Path(os.environ['NNI_OUTPUT_DIR']) / 'model.onnx')\n evaluate_model(model)"
]
},
{
@ -215,24 +197,6 @@
"source": [
"for model_dict in exp.export_top_models(formatter='dict'):\n print(model_dict)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"The output is ``json`` object which records the mutation actions of the top model.\nIf users want to output source code of the top model,\nthey can use `graph-based execution engine <graph-based-execution-engine>` for the experiment,\nby simply adding the following two lines.\n\n"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"collapsed": false
},
"outputs": [],
"source": [
"exp_config.execution_engine = 'base'\nexport_formatter = 'code'"
]
}
],
"metadata": {
@ -251,7 +215,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.8"
"version": "3.8.12"
}
},
"nbformat": 4,

132
docs/source/tutorials/hello_nas.py сгенерированный
Просмотреть файл

@ -12,7 +12,7 @@ There are mainly three crucial components for a neural architecture search task,
* A proper strategy as the method to explore this model space.
* A model evaluator that reports the performance of every model in the space.
Currently, PyTorch is the only supported framework by Retiarii, and we have only tested **PyTorch 1.7 to 1.10**.
Currently, PyTorch is the only supported framework by Retiarii, and we have only tested **PyTorch 1.9 to 1.13**.
This tutorial assumes PyTorch context but it should also apply to other frameworks, which is in our future plan.
Define your Model Space
@ -28,19 +28,17 @@ In this framework, a model space is defined with two parts: a base model and pos
# ^^^^^^^^^^^^^^^^^
#
# Defining a base model is almost the same as defining a PyTorch (or TensorFlow) model.
# Usually, you only need to replace the code ``import torch.nn as nn`` with
# ``import nni.retiarii.nn.pytorch as nn`` to use our wrapped PyTorch modules.
#
# Below is a very simple example of defining a base model.
import torch
import torch.nn as nn
import torch.nn.functional as F
import nni.retiarii.nn.pytorch as nn
from nni.retiarii import model_wrapper
import nni
from nni.nas.nn.pytorch import LayerChoice, ModelSpace, MutableDropout, MutableLinear
@model_wrapper # this decorator should be put on the out most
class Net(nn.Module):
class Net(ModelSpace): # should inherit ModelSpace rather than nn.Module
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, 3, 1)
@ -59,12 +57,9 @@ class Net(nn.Module):
return output
# %%
# .. tip:: Always keep in mind that you should use ``import nni.retiarii.nn.pytorch as nn`` and :meth:`nni.retiarii.model_wrapper`.
# Many mistakes are a result of forgetting one of those.
# Also, please use ``torch.nn`` for submodules of ``nn.init``, e.g., ``torch.nn.init`` instead of ``nn.init``.
#
# Define Model Mutations
# ^^^^^^^^^^^^^^^^^^^^^^
# Define Model Variations
# ^^^^^^^^^^^^^^^^^^^^^^^
#
# A base model is only one concrete model not a model space. We provide :doc:`API and Primitives </nas/construct_space>`
# for users to express how the base model can be mutated. That is, to build a model space which includes many models.
@ -73,24 +68,23 @@ class Net(nn.Module):
#
# .. code-block:: diff
#
# @model_wrapper
# class Net(nn.Module):
# class Net(ModelSpace):
# def __init__(self):
# super().__init__()
# self.conv1 = nn.Conv2d(1, 32, 3, 1)
# - self.conv2 = nn.Conv2d(32, 64, 3, 1)
# + self.conv2 = nn.LayerChoice([
# + self.conv2 = LayerChoice([
# + nn.Conv2d(32, 64, 3, 1),
# + DepthwiseSeparableConv(32, 64)
# + ])
# + ], label='conv2)
# - self.dropout1 = nn.Dropout(0.25)
# + self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75]))
# + self.dropout1 = MutableDropout(nni.choice('dropout', [0.25, 0.5, 0.75]))
# self.dropout2 = nn.Dropout(0.5)
# - self.fc1 = nn.Linear(9216, 128)
# - self.fc2 = nn.Linear(128, 10)
# + feature = nn.ValueChoice([64, 128, 256])
# + self.fc1 = nn.Linear(9216, feature)
# + self.fc2 = nn.Linear(feature, 10)
# + feature = nni.choice('feature', [64, 128, 256])
# + self.fc1 = MutableLinear(9216, feature)
# + self.fc2 = MutableLinear(feature, 10)
#
# def forward(self, x):
# x = F.relu(self.conv1(x))
@ -113,24 +107,22 @@ class DepthwiseSeparableConv(nn.Module):
return self.pointwise(self.depthwise(x))
@model_wrapper
class ModelSpace(nn.Module):
class MyModelSpace(ModelSpace):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, 3, 1)
# LayerChoice is used to select a layer between Conv2d and DwConv.
self.conv2 = nn.LayerChoice([
self.conv2 = LayerChoice([
nn.Conv2d(32, 64, 3, 1),
DepthwiseSeparableConv(32, 64)
])
# ValueChoice is used to select a dropout rate.
# ValueChoice can be used as parameter of modules wrapped in `nni.retiarii.nn.pytorch`
# or customized modules wrapped with `@basic_unit`.
self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75])) # choose dropout rate from 0.25, 0.5 and 0.75
], label='conv2')
# nni.choice is used to select a dropout rate.
# The result can be used as parameters of `MutableXXX`.
self.dropout1 = MutableDropout(nni.choice('dropout', [0.25, 0.5, 0.75])) # choose dropout rate from 0.25, 0.5 and 0.75
self.dropout2 = nn.Dropout(0.5)
feature = nn.ValueChoice([64, 128, 256])
self.fc1 = nn.Linear(9216, feature)
self.fc2 = nn.Linear(feature, 10)
feature = nni.choice('feature', [64, 128, 256])
self.fc1 = MutableLinear(9216, feature)
self.fc2 = MutableLinear(feature, 10)
def forward(self, x):
x = F.relu(self.conv1(x))
@ -141,18 +133,17 @@ class ModelSpace(nn.Module):
return output
model_space = ModelSpace()
model_space = MyModelSpace()
model_space
# %%
# This example uses two mutation APIs,
# :class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>` and
# :class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>`.
# :class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`
# :class:`nn.LayerChoice <nni.nas.nn.pytorch.LayerChoice>` and
# :func:`nni.choice`.
# :class:`nn.LayerChoice <nni.nas.nn.pytorch.LayerChoice>`
# takes a list of candidate modules (two in this example), one will be chosen for each sampled model.
# It can be used like normal PyTorch module.
# :class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>` takes a list of candidate values,
# one will be chosen to take effect for each sampled model.
# :func:`nni.choice` is used as parameter of `MutableDropout`, which then takes the result as dropout rate.
#
# More detailed API description and usage can be found :doc:`here </nas/construct_space>`.
#
@ -176,12 +167,12 @@ model_space
# Pick an exploration strategy
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#
# Retiarii supports many :doc:`exploration strategies </nas/exploration_strategy>`.
# NNI NAS supports many :doc:`exploration strategies </nas/exploration_strategy>`.
#
# Simply choosing (i.e., instantiate) an exploration strategy as below.
import nni.retiarii.strategy as strategy
search_strategy = strategy.Random(dedup=True) # dedup=False if deduplication is not wanted
import nni.nas.strategy as strategy
search_strategy = strategy.Random() # dedup=False if deduplication is not wanted
# %%
# Pick or customize a model evaluator
@ -191,8 +182,8 @@ search_strategy = strategy.Random(dedup=True) # dedup=False if deduplication is
# and validating each generated model to obtain the model's performance.
# The performance is sent to the exploration strategy for the strategy to generate better models.
#
# Retiarii has provided :doc:`built-in model evaluators </nas/evaluator>`, but to start with,
# it is recommended to use :class:`FunctionalEvaluator <nni.retiarii.evaluator.FunctionalEvaluator>`,
# NNI NAS has provided :doc:`built-in model evaluators </nas/evaluator>`, but to start with,
# it is recommended to use :class:`FunctionalEvaluator <nni.nas.evaluator.FunctionalEvaluator>`,
# that is, to wrap your own training and evaluation code with one single function.
# This function should receive one single model class and uses :func:`nni.report_final_result` to report the final score of this model.
#
@ -241,10 +232,8 @@ def test_epoch(model, device, test_loader):
return accuracy
def evaluate_model(model_cls):
# "model_cls" is a class, need to instantiate
model = model_cls()
def evaluate_model(model):
# By v3.0, the model will be instantiated by default.
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)
@ -268,7 +257,7 @@ def evaluate_model(model_cls):
# %%
# Create the evaluator
from nni.retiarii.evaluator import FunctionalEvaluator
from nni.nas.evaluator import FunctionalEvaluator
evaluator = FunctionalEvaluator(evaluate_model)
# %%
@ -276,40 +265,41 @@ evaluator = FunctionalEvaluator(evaluate_model)
# The ``train_epoch`` and ``test_epoch`` here can be any customized function,
# where users can write their own training recipe.
#
# It is recommended that the ``evaluate_model`` here accepts no additional arguments other than ``model_cls``.
# It is recommended that the ``evaluate_model`` here accepts no additional arguments other than ``model``.
# However, in the :doc:`advanced tutorial </nas/evaluator>`, we will show how to use additional arguments in case you actually need those.
# In future, we will support mutation on the arguments of evaluators, which is commonly called "Hyper-parmeter tuning".
# In future, we will support mutation on the arguments of evaluators, which is commonly called "Hyper-parameter tuning".
#
# Launch an Experiment
# --------------------
#
# After all the above are prepared, it is time to start an experiment to do the model search. An example is shown below.
from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig
exp = RetiariiExperiment(model_space, evaluator, [], search_strategy)
exp_config = RetiariiExeConfig('local')
exp_config.experiment_name = 'mnist_search'
from nni.nas.experiment import NasExperiment
exp = NasExperiment(model_space, evaluator, search_strategy)
# %%
# The following configurations are useful to control how many trials to run at most / at the same time.
# Different from HPO experiment, NAS experiment will generate an experiment config automatically.
# It should work for most cases. For example, when using multi-trial strategies,
# local training service with concurrency 1 will be used by default.
# Users can customize the config. For example,
exp_config.max_trial_number = 4 # spawn 4 trials at most
exp_config.trial_concurrency = 2 # will run two trials concurrently
exp.config.max_trial_number = 3 # spawn 3 trials at most
exp.config.trial_concurrency = 1 # will run 1 trial concurrently
exp.config.trial_gpu_number = 0 # will not use GPU
# %%
# Remember to set the following config if you want to GPU.
# ``use_active_gpu`` should be set true if you wish to use an occupied GPU (possibly running a GUI).
exp_config.trial_gpu_number = 1
exp_config.training_service.use_active_gpu = True
# %%
# ``use_active_gpu`` should be set true if you wish to use an occupied GPU (possibly running a GUI)::
#
# exp.config.trial_gpu_number = 1
# exp.config.training_service.use_active_gpu = True
#
# Launch the experiment. The experiment should take several minutes to finish on a workstation with 2 GPUs.
exp.run(exp_config, 8081)
exp.run(port=8081)
# %%
# Users can also run Retiarii Experiment with :doc:`different training services </experiment/training_service/overview>`
# Users can also run NAS Experiment with :doc:`different training services </experiment/training_service/overview>`
# besides ``local`` training service.
#
# Visualize the Experiment
@ -332,14 +322,13 @@ import os
from pathlib import Path
def evaluate_model_with_visualization(model_cls):
model = model_cls()
def evaluate_model_with_visualization(model):
# dump the model into an onnx
if 'NNI_OUTPUT_DIR' in os.environ:
dummy_input = torch.zeros(1, 3, 32, 32)
torch.onnx.export(model, (dummy_input, ),
Path(os.environ['NNI_OUTPUT_DIR']) / 'model.onnx')
evaluate_model(model_cls)
evaluate_model(model)
# %%
# Relaunch the experiment, and a button is shown on Web portal.
@ -353,12 +342,3 @@ def evaluate_model_with_visualization(model_cls):
for model_dict in exp.export_top_models(formatter='dict'):
print(model_dict)
# %%
# The output is ``json`` object which records the mutation actions of the top model.
# If users want to output source code of the top model,
# they can use :ref:`graph-based execution engine <graph-based-execution-engine>` for the experiment,
# by simply adding the following two lines.
exp_config.execution_engine = 'base'
export_formatter = 'code'

2
docs/source/tutorials/hello_nas.py.md5 сгенерированный
Просмотреть файл

@ -1 +1 @@
0e49e3aef98633744807b814786f6b31
62b11c437f43487c907e8f969736030a

269
docs/source/tutorials/hello_nas.rst сгенерированный
Просмотреть файл

@ -10,7 +10,7 @@
.. note::
:class: sphx-glr-download-link-note
Click :ref:`here <sphx_glr_download_tutorials_hello_nas.py>`
:ref:`Go to the end <sphx_glr_download_tutorials_hello_nas.py>`
to download the full example code
.. rst-class:: sphx-glr-example-title
@ -31,7 +31,7 @@ There are mainly three crucial components for a neural architecture search task,
* A proper strategy as the method to explore this model space.
* A model evaluator that reports the performance of every model in the space.
Currently, PyTorch is the only supported framework by Retiarii, and we have only tested **PyTorch 1.7 to 1.10**.
Currently, PyTorch is the only supported framework by Retiarii, and we have only tested **PyTorch 1.9 to 1.13**.
This tutorial assumes PyTorch context but it should also apply to other frameworks, which is in our future plan.
Define your Model Space
@ -40,30 +40,28 @@ Define your Model Space
Model space is defined by users to express a set of models that users want to explore, which contains potentially good-performing models.
In this framework, a model space is defined with two parts: a base model and possible mutations on the base model.
.. GENERATED FROM PYTHON SOURCE LINES 26-34
.. GENERATED FROM PYTHON SOURCE LINES 26-32
Define Base Model
^^^^^^^^^^^^^^^^^
Defining a base model is almost the same as defining a PyTorch (or TensorFlow) model.
Usually, you only need to replace the code ``import torch.nn as nn`` with
``import nni.retiarii.nn.pytorch as nn`` to use our wrapped PyTorch modules.
Below is a very simple example of defining a base model.
.. GENERATED FROM PYTHON SOURCE LINES 35-61
.. GENERATED FROM PYTHON SOURCE LINES 33-59
.. code-block:: default
import torch
import torch.nn as nn
import torch.nn.functional as F
import nni.retiarii.nn.pytorch as nn
from nni.retiarii import model_wrapper
import nni
from nni.nas.nn.pytorch import LayerChoice, ModelSpace, MutableDropout, MutableLinear
@model_wrapper # this decorator should be put on the out most
class Net(nn.Module):
class Net(ModelSpace): # should inherit ModelSpace rather than nn.Module
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, 3, 1)
@ -88,14 +86,10 @@ Below is a very simple example of defining a base model.
.. GENERATED FROM PYTHON SOURCE LINES 62-104
.. GENERATED FROM PYTHON SOURCE LINES 60-97
.. tip:: Always keep in mind that you should use ``import nni.retiarii.nn.pytorch as nn`` and :meth:`nni.retiarii.model_wrapper`.
Many mistakes are a result of forgetting one of those.
Also, please use ``torch.nn`` for submodules of ``nn.init``, e.g., ``torch.nn.init`` instead of ``nn.init``.
Define Model Mutations
^^^^^^^^^^^^^^^^^^^^^^
Define Model Variations
^^^^^^^^^^^^^^^^^^^^^^^
A base model is only one concrete model not a model space. We provide :doc:`API and Primitives </nas/construct_space>`
for users to express how the base model can be mutated. That is, to build a model space which includes many models.
@ -104,24 +98,23 @@ Based on the above base model, we can define a model space as below.
.. code-block:: diff
@model_wrapper
class Net(nn.Module):
class Net(ModelSpace):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, 3, 1)
- self.conv2 = nn.Conv2d(32, 64, 3, 1)
+ self.conv2 = nn.LayerChoice([
+ self.conv2 = LayerChoice([
+ nn.Conv2d(32, 64, 3, 1),
+ DepthwiseSeparableConv(32, 64)
+ ])
+ ], label='conv2)
- self.dropout1 = nn.Dropout(0.25)
+ self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75]))
+ self.dropout1 = MutableDropout(nni.choice('dropout', [0.25, 0.5, 0.75]))
self.dropout2 = nn.Dropout(0.5)
- self.fc1 = nn.Linear(9216, 128)
- self.fc2 = nn.Linear(128, 10)
+ feature = nn.ValueChoice([64, 128, 256])
+ self.fc1 = nn.Linear(9216, feature)
+ self.fc2 = nn.Linear(feature, 10)
+ feature = nni.choice('feature', [64, 128, 256])
+ self.fc1 = MutableLinear(9216, feature)
+ self.fc2 = MutableLinear(feature, 10)
def forward(self, x):
x = F.relu(self.conv1(x))
@ -133,7 +126,7 @@ Based on the above base model, we can define a model space as below.
This results in the following code:
.. GENERATED FROM PYTHON SOURCE LINES 104-147
.. GENERATED FROM PYTHON SOURCE LINES 98-139
.. code-block:: default
@ -149,24 +142,22 @@ This results in the following code:
return self.pointwise(self.depthwise(x))
@model_wrapper
class ModelSpace(nn.Module):
class MyModelSpace(ModelSpace):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, 3, 1)
# LayerChoice is used to select a layer between Conv2d and DwConv.
self.conv2 = nn.LayerChoice([
self.conv2 = LayerChoice([
nn.Conv2d(32, 64, 3, 1),
DepthwiseSeparableConv(32, 64)
])
# ValueChoice is used to select a dropout rate.
# ValueChoice can be used as parameter of modules wrapped in `nni.retiarii.nn.pytorch`
# or customized modules wrapped with `@basic_unit`.
self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75])) # choose dropout rate from 0.25, 0.5 and 0.75
], label='conv2')
# nni.choice is used to select a dropout rate.
# The result can be used as parameters of `MutableXXX`.
self.dropout1 = MutableDropout(nni.choice('dropout', [0.25, 0.5, 0.75])) # choose dropout rate from 0.25, 0.5 and 0.75
self.dropout2 = nn.Dropout(0.5)
feature = nn.ValueChoice([64, 128, 256])
self.fc1 = nn.Linear(9216, feature)
self.fc2 = nn.Linear(feature, 10)
feature = nni.choice('feature', [64, 128, 256])
self.fc1 = MutableLinear(9216, feature)
self.fc2 = MutableLinear(feature, 10)
def forward(self, x):
x = F.relu(self.conv1(x))
@ -177,7 +168,7 @@ This results in the following code:
return output
model_space = ModelSpace()
model_space = MyModelSpace()
model_space
@ -186,35 +177,36 @@ This results in the following code:
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
ModelSpace(
MyModelSpace(
(conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))
(conv2): LayerChoice([Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1)), DepthwiseSeparableConv(
(depthwise): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), groups=32)
(pointwise): Conv2d(32, 64, kernel_size=(1, 1), stride=(1, 1))
)], label='model_1')
(dropout1): Dropout(p=0.25, inplace=False)
(conv2): LayerChoice(
label='conv2'
(0): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))
(1): DepthwiseSeparableConv(
(depthwise): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), groups=32)
(pointwise): Conv2d(32, 64, kernel_size=(1, 1), stride=(1, 1))
)
)
(dropout1): MutableDropout(p=Categorical([0.25, 0.5, 0.75], label='dropout'))
(dropout2): Dropout(p=0.5, inplace=False)
(fc1): Linear(in_features=9216, out_features=64, bias=True)
(fc2): Linear(in_features=64, out_features=10, bias=True)
(fc1): MutableLinear(in_features=9216, out_features=Categorical([64, 128, 256], label='feature'))
(fc2): MutableLinear(in_features=Categorical([64, 128, 256], label='feature'), out_features=10)
)
.. GENERATED FROM PYTHON SOURCE LINES 148-182
.. GENERATED FROM PYTHON SOURCE LINES 140-173
This example uses two mutation APIs,
:class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>` and
:class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>`.
:class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`
:class:`nn.LayerChoice <nni.nas.nn.pytorch.LayerChoice>` and
:func:`nni.choice`.
:class:`nn.LayerChoice <nni.nas.nn.pytorch.LayerChoice>`
takes a list of candidate modules (two in this example), one will be chosen for each sampled model.
It can be used like normal PyTorch module.
:class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>` takes a list of candidate values,
one will be chosen to take effect for each sampled model.
:func:`nni.choice` is used as parameter of `MutableDropout`, which then takes the result as dropout rate.
More detailed API description and usage can be found :doc:`here </nas/construct_space>`.
@ -238,36 +230,26 @@ Second, users need to pick or customize a model evaluator to evaluate the perfor
Pick an exploration strategy
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Retiarii supports many :doc:`exploration strategies </nas/exploration_strategy>`.
NNI NAS supports many :doc:`exploration strategies </nas/exploration_strategy>`.
Simply choosing (i.e., instantiate) an exploration strategy as below.
.. GENERATED FROM PYTHON SOURCE LINES 182-186
.. GENERATED FROM PYTHON SOURCE LINES 173-177
.. code-block:: default
import nni.retiarii.strategy as strategy
search_strategy = strategy.Random(dedup=True) # dedup=False if deduplication is not wanted
import nni.nas.strategy as strategy
search_strategy = strategy.Random() # dedup=False if deduplication is not wanted
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/ray/autoscaler/_private/cli_logger.py:57: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via `pip install 'ray[default]'`. Please update your install command.
warnings.warn(
.. GENERATED FROM PYTHON SOURCE LINES 187-200
.. GENERATED FROM PYTHON SOURCE LINES 178-191
Pick or customize a model evaluator
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
@ -276,14 +258,14 @@ In the exploration process, the exploration strategy repeatedly generates new mo
and validating each generated model to obtain the model's performance.
The performance is sent to the exploration strategy for the strategy to generate better models.
Retiarii has provided :doc:`built-in model evaluators </nas/evaluator>`, but to start with,
it is recommended to use :class:`FunctionalEvaluator <nni.retiarii.evaluator.FunctionalEvaluator>`,
NNI NAS has provided :doc:`built-in model evaluators </nas/evaluator>`, but to start with,
it is recommended to use :class:`FunctionalEvaluator <nni.nas.evaluator.FunctionalEvaluator>`,
that is, to wrap your own training and evaluation code with one single function.
This function should receive one single model class and uses :func:`nni.report_final_result` to report the final score of this model.
An example here creates a simple evaluator that runs on MNIST dataset, trains for 2 epochs, and reports its validation accuracy.
.. GENERATED FROM PYTHON SOURCE LINES 200-268
.. GENERATED FROM PYTHON SOURCE LINES 191-257
.. code-block:: default
@ -331,10 +313,8 @@ An example here creates a simple evaluator that runs on MNIST dataset, trains fo
return accuracy
def evaluate_model(model_cls):
# "model_cls" is a class, need to instantiate
model = model_cls()
def evaluate_model(model):
# By v3.0, the model will be instantiated by default.
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)
@ -362,16 +342,16 @@ An example here creates a simple evaluator that runs on MNIST dataset, trains fo
.. GENERATED FROM PYTHON SOURCE LINES 269-270
.. GENERATED FROM PYTHON SOURCE LINES 258-259
Create the evaluator
.. GENERATED FROM PYTHON SOURCE LINES 270-274
.. GENERATED FROM PYTHON SOURCE LINES 259-263
.. code-block:: default
from nni.retiarii.evaluator import FunctionalEvaluator
from nni.nas.evaluator import FunctionalEvaluator
evaluator = FunctionalEvaluator(evaluate_model)
@ -381,29 +361,27 @@ Create the evaluator
.. GENERATED FROM PYTHON SOURCE LINES 275-286
.. GENERATED FROM PYTHON SOURCE LINES 264-275
The ``train_epoch`` and ``test_epoch`` here can be any customized function,
where users can write their own training recipe.
It is recommended that the ``evaluate_model`` here accepts no additional arguments other than ``model_cls``.
It is recommended that the ``evaluate_model`` here accepts no additional arguments other than ``model``.
However, in the :doc:`advanced tutorial </nas/evaluator>`, we will show how to use additional arguments in case you actually need those.
In future, we will support mutation on the arguments of evaluators, which is commonly called "Hyper-parmeter tuning".
In future, we will support mutation on the arguments of evaluators, which is commonly called "Hyper-parameter tuning".
Launch an Experiment
--------------------
After all the above are prepared, it is time to start an experiment to do the model search. An example is shown below.
.. GENERATED FROM PYTHON SOURCE LINES 287-293
.. GENERATED FROM PYTHON SOURCE LINES 276-280
.. code-block:: default
from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig
exp = RetiariiExperiment(model_space, evaluator, [], search_strategy)
exp_config = RetiariiExeConfig('local')
exp_config.experiment_name = 'mnist_search'
from nni.nas.experiment import NasExperiment
exp = NasExperiment(model_space, evaluator, search_strategy)
@ -412,17 +390,21 @@ After all the above are prepared, it is time to start an experiment to do the mo
.. GENERATED FROM PYTHON SOURCE LINES 294-295
.. GENERATED FROM PYTHON SOURCE LINES 281-285
The following configurations are useful to control how many trials to run at most / at the same time.
Different from HPO experiment, NAS experiment will generate an experiment config automatically.
It should work for most cases. For example, when using multi-trial strategies,
local training service with concurrency 1 will be used by default.
Users can customize the config. For example,
.. GENERATED FROM PYTHON SOURCE LINES 295-299
.. GENERATED FROM PYTHON SOURCE LINES 285-290
.. code-block:: default
exp_config.max_trial_number = 4 # spawn 4 trials at most
exp_config.trial_concurrency = 2 # will run two trials concurrently
exp.config.max_trial_number = 3 # spawn 3 trials at most
exp.config.trial_concurrency = 1 # will run 1 trial concurrently
exp.config.trial_gpu_number = 0 # will not use GPU
@ -431,36 +413,22 @@ The following configurations are useful to control how many trials to run at mos
.. GENERATED FROM PYTHON SOURCE LINES 300-302
.. GENERATED FROM PYTHON SOURCE LINES 291-298
Remember to set the following config if you want to GPU.
``use_active_gpu`` should be set true if you wish to use an occupied GPU (possibly running a GUI).
``use_active_gpu`` should be set true if you wish to use an occupied GPU (possibly running a GUI)::
.. GENERATED FROM PYTHON SOURCE LINES 302-306
.. code-block:: default
exp_config.trial_gpu_number = 1
exp_config.training_service.use_active_gpu = True
.. GENERATED FROM PYTHON SOURCE LINES 307-308
exp.config.trial_gpu_number = 1
exp.config.training_service.use_active_gpu = True
Launch the experiment. The experiment should take several minutes to finish on a workstation with 2 GPUs.
.. GENERATED FROM PYTHON SOURCE LINES 308-311
.. GENERATED FROM PYTHON SOURCE LINES 298-301
.. code-block:: default
exp.run(exp_config, 8081)
exp.run(port=8081)
@ -468,31 +436,16 @@ Launch the experiment. The experiment should take several minutes to finish on a
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
INFO:nni.experiment:Creating experiment, Experiment ID: z8ns5fv7
INFO:nni.experiment:Connecting IPC pipe...
INFO:nni.experiment:Starting web server...
INFO:nni.experiment:Setting up...
INFO:nni.runtime.msg_dispatcher_base:Dispatcher started
INFO:nni.retiarii.experiment.pytorch:Web UI URLs: http://127.0.0.1:8081 http://10.190.172.35:8081 http://192.168.49.1:8081 http://172.17.0.1:8081
INFO:nni.retiarii.experiment.pytorch:Start strategy...
INFO:root:Successfully update searchSpace.
INFO:nni.retiarii.strategy.bruteforce:Random search running in fixed size mode. Dedup: on.
INFO:nni.retiarii.experiment.pytorch:Stopping experiment, please wait...
INFO:nni.retiarii.experiment.pytorch:Strategy exit
INFO:nni.retiarii.experiment.pytorch:Waiting for experiment to become DONE (you can ctrl+c if there is no running trial jobs)...
INFO:nni.runtime.msg_dispatcher_base:Dispatcher exiting...
INFO:nni.retiarii.experiment.pytorch:Experiment stopped
True
.. GENERATED FROM PYTHON SOURCE LINES 302-320
.. GENERATED FROM PYTHON SOURCE LINES 312-330
Users can also run Retiarii Experiment with :doc:`different training services </experiment/training_service/overview>`
Users can also run NAS Experiment with :doc:`different training services </experiment/training_service/overview>`
besides ``local`` training service.
Visualize the Experiment
@ -511,7 +464,7 @@ Built-in evaluators (e.g., Classification) will automatically export the model i
For your own evaluator, you need to save your file into ``$NNI_OUTPUT_DIR/model.onnx`` to make this work.
For instance,
.. GENERATED FROM PYTHON SOURCE LINES 330-344
.. GENERATED FROM PYTHON SOURCE LINES 320-333
.. code-block:: default
@ -520,14 +473,13 @@ For instance,
from pathlib import Path
def evaluate_model_with_visualization(model_cls):
model = model_cls()
def evaluate_model_with_visualization(model):
# dump the model into an onnx
if 'NNI_OUTPUT_DIR' in os.environ:
dummy_input = torch.zeros(1, 3, 32, 32)
torch.onnx.export(model, (dummy_input, ),
Path(os.environ['NNI_OUTPUT_DIR']) / 'model.onnx')
evaluate_model(model_cls)
evaluate_model(model)
@ -536,7 +488,7 @@ For instance,
.. GENERATED FROM PYTHON SOURCE LINES 345-353
.. GENERATED FROM PYTHON SOURCE LINES 334-342
Relaunch the experiment, and a button is shown on Web portal.
@ -547,7 +499,7 @@ Export Top Models
Users can export top models after the exploration is done using ``export_top_models``.
.. GENERATED FROM PYTHON SOURCE LINES 353-357
.. GENERATED FROM PYTHON SOURCE LINES 342-345
.. code-block:: default
@ -558,35 +510,11 @@ Users can export top models after the exploration is done using ``export_top_mod
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
{'model_1': '0', 'model_2': 0.25, 'model_3': 64}
.. GENERATED FROM PYTHON SOURCE LINES 358-362
The output is ``json`` object which records the mutation actions of the top model.
If users want to output source code of the top model,
they can use :ref:`graph-based execution engine <graph-based-execution-engine>` for the experiment,
by simply adding the following two lines.
.. GENERATED FROM PYTHON SOURCE LINES 362-365
.. code-block:: default
exp_config.execution_engine = 'base'
export_formatter = 'code'
{'conv2': 1, 'dropout': 0.25, 'feature': 256}
@ -594,28 +522,25 @@ by simply adding the following two lines.
.. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 2 minutes 4.499 seconds)
**Total running time of the script:** ( 13 minutes 8.330 seconds)
.. _sphx_glr_download_tutorials_hello_nas.py:
.. only:: html
.. only :: html
.. container:: sphx-glr-footer
:class: sphx-glr-footer-example
.. container:: sphx-glr-footer sphx-glr-footer-example
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: hello_nas.py <hello_nas.py>`
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: hello_nas.py <hello_nas.py>`
.. container:: sphx-glr-download sphx-glr-download-jupyter
.. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: hello_nas.ipynb <hello_nas.ipynb>`
:download:`Download Jupyter notebook: hello_nas.ipynb <hello_nas.ipynb>`
.. only:: html

Двоичные данные
docs/source/tutorials/hello_nas_codeobj.pickle сгенерированный
Просмотреть файл

Двоичный файл не отображается.

617
docs/source/tutorials/hello_nas_zh.rst сгенерированный
Просмотреть файл

@ -1,617 +0,0 @@
.. 8a873f2c9cb0e8e3ed2d66b9d16c330f
.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "tutorials/hello_nas.py"
.. LINE NUMBERS ARE GIVEN BELOW.
.. only:: html
.. note::
:class: sphx-glr-download-link-note
Click :ref:`here <sphx_glr_download_tutorials_hello_nas.py>`
to download the full example code
.. rst-class:: sphx-glr-example-title
.. _sphx_glr_tutorials_hello_nas.py:
架构搜索入门教程
================
这是 NNI 上的神经架构搜索（NAS）的入门教程。
在本教程中，我们将借助 NNI 的 NAS 框架，即 *Retiarii*，在 MNIST 数据集上实现网络结构搜索。
我们以多尝试的架构搜索为例来展示如何构建和探索模型空间。
神经架构搜索任务主要有三个关键组成部分，即
* 模型搜索空间，定义了一个要探索的模型的集合。
* 一个合适的策略作为探索这个模型空间的方法。
* 一个模型评估器，用于为搜索空间中每个模型评估性能。
目前，Retiarii 只支持 PyTorch，并对 **PyTorch 1.7 到 1.10** 进行了测试。
所以本教程假定您使用 PyTorch 作为深度学习框架。未来我们会支持更多框架。
定义您的模型空间
----------------------
模型空间是由用户定义的，用来表达用户想要探索的一组模型，其中包含有潜力的好模型。
在 NNI 的框架中，模型空间由两部分定义：基本模型和基本模型上可能的变化。
.. GENERATED FROM PYTHON SOURCE LINES 26-34
定义基本模型
^^^^^^^^^^^^^^^^^
定义基本模型与定义 PyTorch（或 TensorFlow）模型几乎相同。
通常，您只需将代码 ``import torch.nn as nn`` 替换为
``import nni.retiarii.nn.pytorch as nn`` 以使用我们打包的 PyTorch 模块。
下面是定义基本模型的一个非常简单的示例。
.. GENERATED FROM PYTHON SOURCE LINES 35-61
.. code-block:: default
import torch
import torch.nn.functional as F
import nni.retiarii.nn.pytorch as nn
from nni.retiarii import model_wrapper
@model_wrapper # this decorator should be put on the out most
class Net(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, 3, 1)
self.conv2 = nn.Conv2d(32, 64, 3, 1)
self.dropout1 = nn.Dropout(0.25)
self.dropout2 = nn.Dropout(0.5)
self.fc1 = nn.Linear(9216, 128)
self.fc2 = nn.Linear(128, 10)
def forward(self, x):
x = F.relu(self.conv1(x))
x = F.max_pool2d(self.conv2(x), 2)
x = torch.flatten(self.dropout1(x), 1)
x = self.fc2(self.dropout2(F.relu(self.fc1(x))))
output = F.log_softmax(x, dim=1)
return output
.. GENERATED FROM PYTHON SOURCE LINES 62-104
.. tip:: 记住，您应该使用 ``import nni.retiarii.nn.pytorch as nn`` 和 :meth:`nni.retiarii.model_wrapper`。
许多错误都是因为忘记使用某一个。
另外，要使用 ``nn.init`` 的子模块，可以使用 ``torch.nn``，例如， ``torch.nn.init`` 而不是 ``nn.init``。
定义模型变化
^^^^^^^^^^^^^^^^^^^^^^
基本模型只是一个具体模型，而不是模型空间。 我们提供 :doc:`模型变化的 API </nas/construct_space>`
让用户表达如何改变基本模型。 即构建一个包含许多模型的搜索空间。
基于上述基本模型，我们可以定义如下模型空间。
.. code-block:: diff
@model_wrapper
class Net(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, 3, 1)
- self.conv2 = nn.Conv2d(32, 64, 3, 1)
+ self.conv2 = nn.LayerChoice([
+ nn.Conv2d(32, 64, 3, 1),
+ DepthwiseSeparableConv(32, 64)
+ ])
- self.dropout1 = nn.Dropout(0.25)
+ self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75]))
self.dropout2 = nn.Dropout(0.5)
- self.fc1 = nn.Linear(9216, 128)
- self.fc2 = nn.Linear(128, 10)
+ feature = nn.ValueChoice([64, 128, 256])
+ self.fc1 = nn.Linear(9216, feature)
+ self.fc2 = nn.Linear(feature, 10)
def forward(self, x):
x = F.relu(self.conv1(x))
x = F.max_pool2d(self.conv2(x), 2)
x = torch.flatten(self.dropout1(x), 1)
x = self.fc2(self.dropout2(F.relu(self.fc1(x))))
output = F.log_softmax(x, dim=1)
return output
结果是以下代码：
.. GENERATED FROM PYTHON SOURCE LINES 104-147
.. code-block:: default
class DepthwiseSeparableConv(nn.Module):
def __init__(self, in_ch, out_ch):
super().__init__()
self.depthwise = nn.Conv2d(in_ch, in_ch, kernel_size=3, groups=in_ch)
self.pointwise = nn.Conv2d(in_ch, out_ch, kernel_size=1)
def forward(self, x):
return self.pointwise(self.depthwise(x))
@model_wrapper
class ModelSpace(nn.Module):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, 3, 1)
# LayerChoice is used to select a layer between Conv2d and DwConv.
self.conv2 = nn.LayerChoice([
nn.Conv2d(32, 64, 3, 1),
DepthwiseSeparableConv(32, 64)
])
# ValueChoice is used to select a dropout rate.
# ValueChoice can be used as parameter of modules wrapped in `nni.retiarii.nn.pytorch`
# or customized modules wrapped with `@basic_unit`.
self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75])) # choose dropout rate from 0.25, 0.5 and 0.75
self.dropout2 = nn.Dropout(0.5)
feature = nn.ValueChoice([64, 128, 256])
self.fc1 = nn.Linear(9216, feature)
self.fc2 = nn.Linear(feature, 10)
def forward(self, x):
x = F.relu(self.conv1(x))
x = F.max_pool2d(self.conv2(x), 2)
x = torch.flatten(self.dropout1(x), 1)
x = self.fc2(self.dropout2(F.relu(self.fc1(x))))
output = F.log_softmax(x, dim=1)
return output
model_space = ModelSpace()
model_space
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
ModelSpace(
(conv1): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))
(conv2): LayerChoice([Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1)), DepthwiseSeparableConv(
(depthwise): Conv2d(32, 32, kernel_size=(3, 3), stride=(1, 1), groups=32)
(pointwise): Conv2d(32, 64, kernel_size=(1, 1), stride=(1, 1))
)], label='model_1')
(dropout1): Dropout(p=0.25, inplace=False)
(dropout2): Dropout(p=0.5, inplace=False)
(fc1): Linear(in_features=9216, out_features=64, bias=True)
(fc2): Linear(in_features=64, out_features=10, bias=True)
)
.. GENERATED FROM PYTHON SOURCE LINES 148-182
这个例子使用了两个模型变化的 API， :class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>` 和 :class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>`。
:class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>` 可以从一系列的候选子模块中（在本例中为两个），为每个采样模型选择一个。
它可以像原来的 PyTorch 子模块一样使用。
:class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>` 的参数是一个候选值列表，语义是为每个采样模型选择一个值。
更详细的 API 描述和用法可以在 :doc:`这里 </nas/construct_space>` 找到。
.. note::
我们正在积极丰富模型变化的 API，使得您可以轻松构建模型空间。
如果当前支持的模型变化的 API 不能表达您的模型空间，
请参考 :doc:`这篇文档 </nas/mutator>` 来自定义突变。
探索定义的模型空间
-------------------------------------------
简单来讲，有两种探索方法：
(1) 独立评估每个采样到的模型，这是 :ref:`多尝试 NAS <multi-trial-nas>` 中的搜索方法。
(2) 单尝试共享权重型的搜索，简称单尝试 NAS。
我们在本教程中演示了第一种方法。第二种方法用户可以参考 :ref:`这里 <one-shot-nas>`。
首先，用户需要选择合适的探索策略来探索定义好的模型空间。
其次，用户需要选择或自定义模型性能评估来评估每个探索模型的性能。
选择探索策略
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Retiarii 支持许多 :doc:`探索策略</nas/exploration_strategy>`。
只需选择（即实例化）探索策略，就如下面的代码演示的一样：
.. GENERATED FROM PYTHON SOURCE LINES 182-186
.. code-block:: default
import nni.retiarii.strategy as strategy
search_strategy = strategy.Random(dedup=True) # dedup=False if deduplication is not wanted
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
/home/yugzhan/miniconda3/envs/cu102/lib/python3.8/site-packages/ray/autoscaler/_private/cli_logger.py:57: FutureWarning: Not all Ray CLI dependencies were found. In Ray 1.4+, the Ray CLI, autoscaler, and dashboard will only be usable via `pip install 'ray[default]'`. Please update your install command.
warnings.warn(
.. GENERATED FROM PYTHON SOURCE LINES 187-200
挑选或自定义模型评估器
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
在探索过程中，探索策略反复生成新模型。模型评估器负责训练并验证每个生成的模型以获得模型的性能。
该性能作为模型的得分被发送到探索策略以帮助其生成更好的模型。
Retiarii 提供了 :doc:`内置模型评估器 </nas/evaluator>`，但在此之前，
我们建议使用 :class:`FunctionalEvaluator <nni.retiarii.evaluator.FunctionalEvaluator>`，即用一个函数包装您自己的训练和评估代码。
这个函数应该接收一个单一的模型类并使用 :func:`nni.report_final_result` 报告这个模型的最终分数。
此处的示例创建了一个简单的评估器，该评估器在 MNIST 数据集上运行，训练 2 个 epoch，并报告其在验证集上的准确率。
.. GENERATED FROM PYTHON SOURCE LINES 200-268
.. code-block:: default
import nni
from torchvision import transforms
from torchvision.datasets import MNIST
from torch.utils.data import DataLoader
def train_epoch(model, device, train_loader, optimizer, epoch):
loss_fn = torch.nn.CrossEntropyLoss()
model.train()
for batch_idx, (data, target) in enumerate(train_loader):
data, target = data.to(device), target.to(device)
optimizer.zero_grad()
output = model(data)
loss = loss_fn(output, target)
loss.backward()
optimizer.step()
if batch_idx % 10 == 0:
print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
epoch, batch_idx * len(data), len(train_loader.dataset),
100. * batch_idx / len(train_loader), loss.item()))
def test_epoch(model, device, test_loader):
model.eval()
test_loss = 0
correct = 0
with torch.no_grad():
for data, target in test_loader:
data, target = data.to(device), target.to(device)
output = model(data)
pred = output.argmax(dim=1, keepdim=True)
correct += pred.eq(target.view_as(pred)).sum().item()
test_loss /= len(test_loader.dataset)
accuracy = 100. * correct / len(test_loader.dataset)
print('\nTest set: Accuracy: {}/{} ({:.0f}%)\n'.format(
correct, len(test_loader.dataset), accuracy))
return accuracy
def evaluate_model(model_cls):
# "model_cls" is a class, need to instantiate
model = model_cls()
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)
transf = transforms.Compose([transforms.ToTensor(), transforms.Normalize((0.1307,), (0.3081,))])
train_loader = DataLoader(MNIST('data/mnist', download=True, transform=transf), batch_size=64, shuffle=True)
test_loader = DataLoader(MNIST('data/mnist', download=True, train=False, transform=transf), batch_size=64)
for epoch in range(3):
# train the model for one epoch
train_epoch(model, device, train_loader, optimizer, epoch)
# test the model for one epoch
accuracy = test_epoch(model, device, test_loader)
# call report intermediate result. Result can be float or dict
nni.report_intermediate_result(accuracy)
# report final test result
nni.report_final_result(accuracy)
.. GENERATED FROM PYTHON SOURCE LINES 269-270
创建评估器
.. GENERATED FROM PYTHON SOURCE LINES 270-274
.. code-block:: default
from nni.retiarii.evaluator import FunctionalEvaluator
evaluator = FunctionalEvaluator(evaluate_model)
.. GENERATED FROM PYTHON SOURCE LINES 275-286
这里的 ``train_epoch`` 和 ``test_epoch`` 可以是任何自定义函数，用户可以在其中编写自己的训练逻辑。
建议这里的 ``evaluate_model`` 不接受除 ``model_cls`` 之外的其他参数。
但是，在 `高级教程 </nas/evaluator>` 中，我们将展示如何使用其他参数，以免您确实需要这些参数。
未来，我们将支持对评估器的参数进行变化（通常称为“超参数调优”）。
启动实验
--------------------
一切都已准备就绪，现在就可以开始做模型搜索的实验了。如下所示。
.. GENERATED FROM PYTHON SOURCE LINES 287-293
.. code-block:: default
from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig
exp = RetiariiExperiment(model_space, evaluator, [], search_strategy)
exp_config = RetiariiExeConfig('local')
exp_config.experiment_name = 'mnist_search'
.. GENERATED FROM PYTHON SOURCE LINES 294-295
以下配置可以用于控制最多/同时运行多少试验。
.. GENERATED FROM PYTHON SOURCE LINES 295-299
.. code-block:: default
exp_config.max_trial_number = 4 # 最多运行 4 个实验
exp_config.trial_concurrency = 2 # 最多同时运行 2 个试验
.. GENERATED FROM PYTHON SOURCE LINES 300-302
如果要使用 GPU，请设置以下配置。
如果您希望使用被占用了的 GPU（比如 GPU 上可能正在运行 GUI），则 ``use_active_gpu`` 应设置为 true。
.. GENERATED FROM PYTHON SOURCE LINES 302-306
.. code-block:: default
exp_config.trial_gpu_number = 1
exp_config.training_service.use_active_gpu = True
.. GENERATED FROM PYTHON SOURCE LINES 307-308
启动实验。 在一个有两块 GPU 的工作站上完成整个实验大约需要几分钟时间。
.. GENERATED FROM PYTHON SOURCE LINES 308-311
.. code-block:: default
exp.run(exp_config, 8081)
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
INFO:nni.experiment:Creating experiment, Experiment ID: z8ns5fv7
INFO:nni.experiment:Connecting IPC pipe...
INFO:nni.experiment:Starting web server...
INFO:nni.experiment:Setting up...
INFO:nni.runtime.msg_dispatcher_base:Dispatcher started
INFO:nni.retiarii.experiment.pytorch:Web UI URLs: http://127.0.0.1:8081 http://10.190.172.35:8081 http://192.168.49.1:8081 http://172.17.0.1:8081
INFO:nni.retiarii.experiment.pytorch:Start strategy...
INFO:root:Successfully update searchSpace.
INFO:nni.retiarii.strategy.bruteforce:Random search running in fixed size mode. Dedup: on.
INFO:nni.retiarii.experiment.pytorch:Stopping experiment, please wait...
INFO:nni.retiarii.experiment.pytorch:Strategy exit
INFO:nni.retiarii.experiment.pytorch:Waiting for experiment to become DONE (you can ctrl+c if there is no running trial jobs)...
INFO:nni.runtime.msg_dispatcher_base:Dispatcher exiting...
INFO:nni.retiarii.experiment.pytorch:Experiment stopped
.. GENERATED FROM PYTHON SOURCE LINES 312-330
除了 ``local`` 训练平台，用户还可以使用 :doc:`不同的训练平台 </experiment/training_service/overview>` 来运行 Retiarii 试验。
可视化实验
----------------------
用户可以可视化他们的架构搜索实验，就像可视化超参调优实验一样。
例如，在浏览器中打开 ``localhost:8081``，8081 是您在 ``exp.run`` 中设置的端口。
详情请参考 :doc:`这里</experiment/web_portal/web_portal>`。
我们支持使用第三方可视化引擎（如 `Netron <https://netron.app/>`__）对模型进行可视化。
这可以通过单击每个试验的详细面板中的“可视化”来使用。
请注意，当前的可视化是基于 `onnx <https://onnx.ai/>`__，
因此，如果模型不能导出为 onnx，可视化是不可行的。
内置评估器（例如 Classification）会将模型自动导出到文件中。
对于您自己的评估器，您需要将文件保存到 ``$NNI_OUTPUT_DIR/model.onnx``。
例如，
.. GENERATED FROM PYTHON SOURCE LINES 330-344
.. code-block:: default
import os
from pathlib import Path
def evaluate_model_with_visualization(model_cls):
model = model_cls()
# dump the model into an onnx
if 'NNI_OUTPUT_DIR' in os.environ:
dummy_input = torch.zeros(1, 3, 32, 32)
torch.onnx.export(model, (dummy_input, ),
Path(os.environ['NNI_OUTPUT_DIR']) / 'model.onnx')
evaluate_model(model_cls)
.. GENERATED FROM PYTHON SOURCE LINES 345-353
重新启动实验，Web 界面上会显示一个按钮。
.. image:: ../../img/netron_entrance_webui.png
导出最优模型
-----------------
搜索完成后，用户可以使用 ``export_top_models`` 导出最优模型。
.. GENERATED FROM PYTHON SOURCE LINES 353-357
.. code-block:: default
for model_dict in exp.export_top_models(formatter='dict'):
print(model_dict)
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
{'model_1': '0', 'model_2': 0.25, 'model_3': 64}
.. GENERATED FROM PYTHON SOURCE LINES 358-362
输出是一个 JSON 对象，记录了最好的模型的每一个选择都选了什么。
如果用户想要搜出来的模型的源代码，他们可以使用 :ref:`基于图的引擎 <graph-based-execution-engine>`，只需增加如下两行。
.. GENERATED FROM PYTHON SOURCE LINES 362-365
.. code-block:: default
exp_config.execution_engine = 'base'
export_formatter = 'code'
.. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 2 minutes 4.499 seconds)
.. _sphx_glr_download_tutorials_hello_nas.py:
.. only :: html
.. container:: sphx-glr-footer
:class: sphx-glr-footer-example
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: hello_nas.py <hello_nas.py>`
.. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: hello_nas.ipynb <hello_nas.ipynb>`
.. only:: html
.. rst-class:: sphx-glr-signature
`Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_

4
docs/source/tutorials/hpo_quickstart_pytorch/index.rst
Просмотреть файл

@ -17,7 +17,7 @@
.. only:: html
.. image:: /tutorials/hpo_quickstart_pytorch/images/thumb/sphx_glr_main_thumb.png
:alt: HPO Quickstart with PyTorch
:alt:
:ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_main.py`
@ -34,7 +34,7 @@
.. only:: html
.. image:: /tutorials/hpo_quickstart_pytorch/images/thumb/sphx_glr_model_thumb.png
:alt: Port PyTorch Quickstart to NNI
:alt:
:ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_model.py`

4
docs/source/tutorials/hpo_quickstart_tensorflow/index.rst
Просмотреть файл

@ -17,7 +17,7 @@
.. only:: html
.. image:: /tutorials/hpo_quickstart_tensorflow/images/thumb/sphx_glr_main_thumb.png
:alt: HPO Quickstart with TensorFlow
:alt:
:ref:`sphx_glr_tutorials_hpo_quickstart_tensorflow_main.py`
@ -34,7 +34,7 @@
.. only:: html
.. image:: /tutorials/hpo_quickstart_tensorflow/images/thumb/sphx_glr_model_thumb.png
:alt: Port TensorFlow Quickstart to NNI
:alt:
:ref:`sphx_glr_tutorials_hpo_quickstart_tensorflow_model.py`

Двоичные данные
docs/source/tutorials/images/sphx_glr_darts_001.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 96 KiB

После

Ширина:  |  Высота:  |  Размер: 255 KiB

Двоичные данные
docs/source/tutorials/images/sphx_glr_darts_002.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 118 KiB

После

Ширина:  |  Высота:  |  Размер: 118 KiB

Двоичные данные
docs/source/tutorials/images/sphx_glr_darts_003.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 201 KiB

После

Ширина:  |  Высота:  |  Размер: 200 KiB

Двоичные данные
docs/source/tutorials/images/thumb/sphx_glr_darts_thumb.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 12 KiB

После

Ширина:  |  Высота:  |  Размер: 28 KiB

Двоичные данные
docs/source/tutorials/images/thumb/sphx_glr_hello_nas_thumb.png
Просмотреть файл

Двоичный файл не отображается.
Режим "рядом" Свайп Оверлей

До

Ширина:  |  Высота:  |  Размер: 18 KiB

После

Ширина:  |  Высота:  |  Размер: 35 KiB

31
docs/source/tutorials/index.rst сгенерированный
Просмотреть файл

@ -17,7 +17,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/images/thumb/sphx_glr_pruning_speedup_thumb.png
:alt: Speedup Model with Mask
:alt:
:ref:`sphx_glr_tutorials_pruning_speedup.py`
@ -34,7 +34,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/images/thumb/sphx_glr_quantization_quick_start_mnist_thumb.png
:alt: Quantization Quickstart
:alt:
:ref:`sphx_glr_tutorials_quantization_quick_start_mnist.py`
@ -51,7 +51,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/images/thumb/sphx_glr_pruning_quick_start_mnist_thumb.png
:alt: Pruning Quickstart
:alt:
:ref:`sphx_glr_tutorials_pruning_quick_start_mnist.py`
@ -68,7 +68,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/images/thumb/sphx_glr_quantization_customize_thumb.png
:alt: Customize a new quantization algorithm
:alt:
:ref:`sphx_glr_tutorials_quantization_customize.py`
@ -85,7 +85,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/images/thumb/sphx_glr_nasbench_as_dataset_thumb.png
:alt: Use NAS Benchmarks as Datasets
:alt:
:ref:`sphx_glr_tutorials_nasbench_as_dataset.py`
@ -102,7 +102,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/images/thumb/sphx_glr_quantization_speedup_thumb.png
:alt: Speed Up Quantized Model with TensorRT
:alt:
:ref:`sphx_glr_tutorials_quantization_speedup.py`
@ -119,7 +119,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/images/thumb/sphx_glr_hello_nas_thumb.png
:alt: Hello, NAS!
:alt:
:ref:`sphx_glr_tutorials_hello_nas.py`
@ -136,7 +136,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/images/thumb/sphx_glr_darts_thumb.png
:alt: Searching in DARTS search space
:alt:
:ref:`sphx_glr_tutorials_darts.py`
@ -148,12 +148,12 @@ Tutorials
.. raw:: html
<div class="sphx-glr-thumbcontainer" tooltip="This is a new tutorial on pruning transformer in nni v3.0 (`old tutorial &lt;https://nni.readthedo...">
<div class="sphx-glr-thumbcontainer" tooltip="This is a new tutorial on pruning transformer in nni v3.0 (old tutorial). The main difference b...">
.. only:: html
.. image:: /tutorials/images/thumb/sphx_glr_new_pruning_bert_glue_thumb.png
:alt: Pruning Bert on Task MNLI
:alt:
:ref:`sphx_glr_tutorials_new_pruning_bert_glue.py`
@ -170,7 +170,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/images/thumb/sphx_glr_pruning_bert_glue_thumb.png
:alt: Pruning Bert on Task MNLI
:alt:
:ref:`sphx_glr_tutorials_pruning_bert_glue.py`
@ -214,7 +214,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/hpo_quickstart_pytorch/images/thumb/sphx_glr_main_thumb.png
:alt: HPO Quickstart with PyTorch
:alt:
:ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_main.py`
@ -231,7 +231,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/hpo_quickstart_pytorch/images/thumb/sphx_glr_model_thumb.png
:alt: Port PyTorch Quickstart to NNI
:alt:
:ref:`sphx_glr_tutorials_hpo_quickstart_pytorch_model.py`
@ -260,7 +260,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/hpo_quickstart_tensorflow/images/thumb/sphx_glr_main_thumb.png
:alt: HPO Quickstart with TensorFlow
:alt:
:ref:`sphx_glr_tutorials_hpo_quickstart_tensorflow_main.py`
@ -277,7 +277,7 @@ Tutorials
.. only:: html
.. image:: /tutorials/hpo_quickstart_tensorflow/images/thumb/sphx_glr_model_thumb.png
:alt: Port TensorFlow Quickstart to NNI
:alt:
:ref:`sphx_glr_tutorials_hpo_quickstart_tensorflow_model.py`
@ -296,6 +296,7 @@ Tutorials
:hidden:
:includehidden:
/tutorials/hpo_quickstart_pytorch/index.rst
/tutorials/hpo_quickstart_tensorflow/index.rst

2
docs/source/tutorials/nasbench_as_dataset.ipynb сгенерированный
Просмотреть файл

@ -44,7 +44,7 @@
},
"outputs": [],
"source": [
"import pprint\n\nfrom nni.nas.benchmarks.nasbench101 import query_nb101_trial_stats\nfrom nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats\nfrom nni.nas.benchmarks.nds import query_nds_trial_stats"
"import pprint\n\nfrom nni.nas.benchmark.nasbench101 import query_nb101_trial_stats\nfrom nni.nas.benchmark.nasbench201 import query_nb201_trial_stats\nfrom nni.nas.benchmark.nds import query_nds_trial_stats"
]
},
{

6
docs/source/tutorials/nasbench_as_dataset.py сгенерированный
Просмотреть файл

@ -23,9 +23,9 @@ os.listdir(os.path.expanduser('~/.cache/nni/nasbenchmark'))
# %%
import pprint
from nni.nas.benchmarks.nasbench101 import query_nb101_trial_stats
from nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats
from nni.nas.benchmarks.nds import query_nds_trial_stats
from nni.nas.benchmark.nasbench101 import query_nb101_trial_stats
from nni.nas.benchmark.nasbench201 import query_nb201_trial_stats
from nni.nas.benchmark.nds import query_nds_trial_stats
# %%
# NAS-Bench-101

2
docs/source/tutorials/nasbench_as_dataset.py.md5 сгенерированный
Просмотреть файл

@ -1 +1 @@
715de24d20c57f3639033f6f10376c21
f5165dd74d4385744c53d17ce05fdc2c

50
docs/source/tutorials/nasbench_as_dataset.rst сгенерированный
Просмотреть файл

@ -10,7 +10,7 @@
.. note::
:class: sphx-glr-download-link-note
Click :ref:`here <sphx_glr_download_tutorials_nasbench_as_dataset.py>`
:ref:`Go to the end <sphx_glr_download_tutorials_nasbench_as_dataset.py>`
to download the full example code
.. rst-class:: sphx-glr-example-title
@ -50,12 +50,10 @@ As a result, the directory should look like:
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
['nasbench101-209f5694.db', 'nasbench201-b2b60732.db', 'nds-5745c235.db']
['nds-5745c235.db', 'nasbench201-b2b60732.db', 'nasbench101-209f5694.db']
@ -65,9 +63,9 @@ As a result, the directory should look like:
import pprint
from nni.nas.benchmarks.nasbench101 import query_nb101_trial_stats
from nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats
from nni.nas.benchmarks.nds import query_nds_trial_stats
from nni.nas.benchmark.nasbench101 import query_nb101_trial_stats
from nni.nas.benchmark.nasbench201 import query_nb201_trial_stats
from nni.nas.benchmark.nds import query_nds_trial_stats
@ -112,11 +110,8 @@ Use the following architecture as an example:
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
[2022-02-28 13:48:51] INFO (nni.nas.benchmarks.utils/MainThread) "/home/yugzhan/.cache/nni/nasbenchmark/nasbench101-209f5694.db" already exists. Checking hash.
{'config': {'arch': {'input1': [0],
'input2': [1],
'input3': [2],
@ -256,11 +251,8 @@ Use the following architecture as an example:
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
[2022-02-28 13:49:09] INFO (nni.nas.benchmarks.utils/MainThread) "/home/yugzhan/.cache/nni/nasbenchmark/nasbench201-b2b60732.db" already exists. Checking hash.
{'config': {'arch': {'0_1': 'avg_pool_3x3',
'0_2': 'conv_1x1',
'0_3': 'conv_1x1',
@ -368,8 +360,6 @@ Intermediate results are also available.
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
{'id': 4, 'arch': {'0_1': 'avg_pool_3x3', '0_2': 'conv_1x1', '0_3': 'conv_1x1', '1_2': 'skip_connect', '1_3': 'skip_connect', '2_3': 'skip_connect'}, 'num_epochs': 12, 'num_channels': 16, 'num_cells': 5, 'dataset': 'imagenet16-120'}
@ -432,11 +422,8 @@ Use none as a wildcard.
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
[2022-02-28 13:49:36] INFO (nni.nas.benchmarks.utils/MainThread) "/home/yugzhan/.cache/nni/nasbenchmark/nds-5745c235.db" already exists. Checking hash.
{'best_test_acc': 90.48,
'best_train_acc': 96.356,
'best_train_loss': 0.116,
@ -486,8 +473,6 @@ Use none as a wildcard.
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
[{'current_epoch': 1,
@ -558,8 +543,6 @@ Use none as a wildcard.
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
{'best_test_acc': 93.58,
@ -605,8 +588,6 @@ Get the first one.
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
{'best_test_acc': 84.5,
@ -701,8 +682,6 @@ Count number.
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
{'best_test_acc': 93.37,
@ -791,8 +770,6 @@ Count number.
.. rst-class:: sphx-glr-script-out
Out:
.. code-block:: none
NDS (amoeba) count: 5107
@ -803,28 +780,25 @@ Count number.
.. rst-class:: sphx-glr-timing
**Total running time of the script:** ( 1 minutes 2.214 seconds)
**Total running time of the script:** ( 1 minutes 51.444 seconds)
.. _sphx_glr_download_tutorials_nasbench_as_dataset.py:
.. only:: html
.. only :: html
.. container:: sphx-glr-footer
:class: sphx-glr-footer-example
.. container:: sphx-glr-footer sphx-glr-footer-example
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: nasbench_as_dataset.py <nasbench_as_dataset.py>`
.. container:: sphx-glr-download sphx-glr-download-python
:download:`Download Python source code: nasbench_as_dataset.py <nasbench_as_dataset.py>`
.. container:: sphx-glr-download sphx-glr-download-jupyter
.. container:: sphx-glr-download sphx-glr-download-jupyter
:download:`Download Jupyter notebook: nasbench_as_dataset.ipynb <nasbench_as_dataset.ipynb>`
:download:`Download Jupyter notebook: nasbench_as_dataset.ipynb <nasbench_as_dataset.ipynb>`
.. only:: html

Двоичные данные
docs/source/tutorials/nasbench_as_dataset_codeobj.pickle сгенерированный
Просмотреть файл

Двоичный файл не отображается.

10
docs/source/tutorials/sg_execution_times.rst сгенерированный
Просмотреть файл

@ -6,16 +6,18 @@
Computation times
=================
**00:01.990** total execution time for **tutorials** files:
**02:31.916** total execution time for **tutorials** files:
+-----------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_tutorials_new_pruning_bert_glue.py` (``new_pruning_bert_glue.py``) | 00:01.990 | 0.0 MB |
| :ref:`sphx_glr_tutorials_nasbench_as_dataset.py` (``nasbench_as_dataset.py``) | 01:51.444 | 0.0 MB |
+-----------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_tutorials_darts.py` (``darts.py``) | 00:00.000 | 0.0 MB |
| :ref:`sphx_glr_tutorials_darts.py` (``darts.py``) | 00:40.472 | 0.0 MB |
+-----------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_tutorials_hello_nas.py` (``hello_nas.py``) | 00:00.000 | 0.0 MB |
+-----------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_tutorials_nasbench_as_dataset.py` (``nasbench_as_dataset.py``) | 00:00.000 | 0.0 MB |
| :ref:`sphx_glr_tutorials_new_pruning_bert_glue.py` (``new_pruning_bert_glue.py``) | 00:00.000 | 0.0 MB |
+-----------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_tutorials_pruning_bert_glue.py` (``pruning_bert_glue.py``) | 00:00.000 | 0.0 MB |
+-----------------------------------------------------------------------------------------------------+-----------+--------+
| :ref:`sphx_glr_tutorials_pruning_quick_start_mnist.py` (``pruning_quick_start_mnist.py``) | 00:00.000 | 0.0 MB |
+-----------------------------------------------------------------------------------------------------+-----------+--------+

73
examples/tutorials/darts.py
Просмотреть файл

@ -15,7 +15,7 @@ In the end, we get a strong-performing model on CIFAR-10 dataset, which achieves
.. attention::
Running this tutorial requires a GPU.
If you don't have one, you can set ``gpus`` in :class:`~nni.retiarii.evaluator.pytorch.Classification` to be 0,
If you don't have one, you can set ``gpus`` in :class:`~nni.nas.evaluator.pytorch.Classification` to be 0,
but do note that it will be much slower.
.. _DARTS: https://arxiv.org/abs/1806.09055
@ -34,7 +34,7 @@ import nni
import torch
from torchvision import transforms
from torchvision.datasets import CIFAR10
from nni.retiarii.evaluator.pytorch import DataLoader
from nni.nas.evaluator.pytorch import DataLoader
CIFAR_MEAN = [0.49139968, 0.48215827, 0.44653124]
CIFAR_STD = [0.24703233, 0.24348505, 0.26158768]
@ -51,7 +51,7 @@ valid_loader = DataLoader(valid_data, batch_size=256, num_workers=6)
# .. note::
#
# If you are to use multi-trial strategies, wrapping CIFAR10 with :func:`nni.trace` and
# use DataLoader from ``nni.retiarii.evaluator.pytorch`` (instead of ``torch.utils.data``) are mandatory.
# use DataLoader from ``nni.nas.evaluator.pytorch`` (instead of ``torch.utils.data``) are mandatory.
# Otherwise, it's optional.
#
# NNI presents many built-in model spaces, along with many *pre-searched models* in :doc:`model space hub </nas/space_hub>`,
@ -69,7 +69,7 @@ valid_loader = DataLoader(valid_data, batch_size=256, num_workers=6)
# `this tutorial of object detection finetuning <https://pytorch.org/tutorials/intermediate/torchvision_tutorial.html>`__
# if you want to know how finetuning is generally done in PyTorch.
from nni.retiarii.hub.pytorch import DARTS as DartsSpace
from nni.nas.hub.pytorch import DARTS as DartsSpace
darts_v2_model = DartsSpace.load_searched_model('darts-v2', pretrained=True, download=True)
@ -93,7 +93,7 @@ evaluate_model(darts_v2_model, cuda=True) # Set this to false if there's no GPU
# %%
#
# The journey of using a pre-searched model could end here. Or you are interested,
# we can go a step further to search a model within :class:`~nni.retiarii.hub.pytorch.DARTS` space on our own.
# we can go a step further to search a model within :class:`~nni.nas.hub.pytorch.DARTS` space on our own.
#
# Use the DARTS model space
# -------------------------
@ -141,15 +141,15 @@ evaluate_model(darts_v2_model, cuda=True) # Set this to false if there's no GPU
# `DARTS`_ is one of those papers that innovate both in search space and search strategy.
# In this tutorial, we will search on **model space** provided by DARTS with **search strategy** proposed by DARTS.
# We refer to them as *DARTS model space* (``DartsSpace``) and *DARTS strategy* (``DartsStrategy``), respectively.
# We did NOT imply that the :class:`~nni.retiarii.hub.pytorch.DARTS` space and
# :class:`~nni.retiarii.strategy.DARTS` strategy has to used together.
# We did NOT imply that the :class:`~nni.nas.hub.pytorch.DARTS` space and
# :class:`~nni.nas.strategy.DARTS` strategy has to used together.
# You can always explore the DARTS space with another search strategy, or use your own strategy to search a different model space.
#
# In the following example, we initialize a :class:`~nni.retiarii.hub.pytorch.DARTS`
# In the following example, we initialize a :class:`~nni.nas.hub.pytorch.DARTS`
# model space, with 16 initial filters and 8 stacked cells.
# The network is specialized for CIFAR-10 dataset with 32x32 input resolution.
#
# The :class:`~nni.retiarii.hub.pytorch.DARTS` model space here is provided by :doc:`model space hub </nas/space_hub>`,
# The :class:`~nni.nas.hub.pytorch.DARTS` model space here is provided by :doc:`model space hub </nas/space_hub>`,
# where we have supported multiple popular model spaces for plug-and-play.
#
# .. tip::
@ -181,7 +181,7 @@ fast_dev_run = True
# ^^^^^^^^^
#
# To begin exploring the model space, one firstly need to have an evaluator to provide the criterion of a "good model".
# As we are searching on CIFAR-10 dataset, one can easily use the :class:`~nni.retiarii.evaluator.pytorch.Classification`
# As we are searching on CIFAR-10 dataset, one can easily use the :class:`~nni.nas.evaluator.pytorch.Classification`
# as a starting point.
#
# Note that for a typical setup of NAS, the model search should be on validation set, and the evaluation of the final searched model
@ -190,7 +190,7 @@ fast_dev_run = True
# The recommended train/val split by `DARTS`_ strategy is 1:1.
import numpy as np
from nni.retiarii.evaluator.pytorch import Classification
from nni.nas.evaluator.pytorch import Classification
from torch.utils.data import SubsetRandomSampler
transform = transforms.Compose([
@ -232,7 +232,7 @@ evaluator = Classification(
# ^^^^^^^^
#
# We will use `DARTS`_ (Differentiable ARchiTecture Search) as the search strategy to explore the model space.
# :class:`~nni.retiarii.strategy.DARTS` strategy belongs to the category of :ref:`one-shot strategy <one-shot-nas>`.
# :class:`~nni.nas.strategy.DARTS` strategy belongs to the category of :ref:`one-shot strategy <one-shot-nas>`.
# The fundamental differences between One-shot strategies and :ref:`multi-trial strategies <multi-trial-nas>` is that,
# one-shot strategy combines search with model training into a single run.
# Compared to multi-trial strategies, one-shot NAS doesn't need to iteratively spawn new trials (i.e., models),
@ -246,10 +246,10 @@ evaluator = Classification(
# and
# `How Does Supernet Help in Neural Architecture Search? <https://arxiv.org/abs/2010.08219>`__ for interested readers.
#
# :class:`~nni.retiarii.strategy.DARTS` strategy is provided as one of NNI's :doc:`built-in search strategies </nas/exploration_strategy>`.
# :class:`~nni.nas.strategy.DARTS` strategy is provided as one of NNI's :doc:`built-in search strategies </nas/exploration_strategy>`.
# Using it can be as simple as one line of code.
from nni.retiarii.strategy import DARTS as DartsStrategy
from nni.nas.strategy import DARTS as DartsStrategy
strategy = DartsStrategy()
@ -272,14 +272,12 @@ strategy = DartsStrategy()
# ^^^^^^^^^^^^^^^^^
#
# We then come to the step of launching the experiment.
# This step is similar to what we have done in the :doc:`beginner tutorial <hello_nas>`,
# except that the ``execution_engine`` argument should be set to ``oneshot``.
# This step is similar to what we have done in the :doc:`beginner tutorial <hello_nas>`.
from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig
from nni.nas.experiment import NasExperiment
config = RetiariiExeConfig(execution_engine='oneshot')
experiment = RetiariiExperiment(model_space, evaluator=evaluator, strategy=strategy)
experiment.run(config)
experiment = NasExperiment(model_space, evaluator, strategy)
experiment.run()
# %%
#
@ -296,7 +294,7 @@ experiment.run(config)
# We can then retrieve the best model found by the strategy with ``export_top_models``.
# Here, the retrieved model is a dict (called *architecture dict*) describing the selected normal cell and reduction cell.
exported_arch = experiment.export_top_models()[0]
exported_arch = experiment.export_top_models(formatter='dict')[0]
exported_arch
@ -411,12 +409,12 @@ plot_double_cells({
# and then fully train it.
#
# To construct a fixed model based on the architecture dict exported from the experiment,
# we can use :func:`nni.retiarii.fixed_arch`. Under the with-context, we will creating a fixed model based on ``exported_arch``,
# we can use :func:`nni.nas.space.model_context`. Under the with-context, we will creating a fixed model based on ``exported_arch``,
# instead of creating a space.
from nni.retiarii import fixed_arch
from nni.nas.space import model_context
with fixed_arch(exported_arch):
with model_context(exported_arch):
final_model = DartsSpace(width=16, num_cells=8, dataset='cifar')
# %%
@ -434,7 +432,7 @@ valid_loader
# %%
#
# We must create a new evaluator here because a different data split is used.
# Also, we should avoid the underlying pytorch-lightning implementation of :class:`~nni.retiarii.evaluator.pytorch.Classification`
# Also, we should avoid the underlying pytorch-lightning implementation of :class:`~nni.nas.evaluator.pytorch.Classification`
# evaluator from loading the wrong checkpoint.
max_epochs = 100
@ -478,13 +476,13 @@ evaluator.fit(final_model)
# `LightingModule <https://pytorch-lightning.readthedocs.io/en/stable/common/lightning_module.html>`__ is a concept in
# PyTorch-Lightning, which organizes the model training process into a list of functions, such as,
# ``training_step``, ``validation_step``, ``configure_optimizers``, etc.
# Since we are merely adding a few ingredients to :class:`~nni.retiarii.evaluator.pytorch.Classification`,
# we can simply inherit :class:`~nni.retiarii.evaluator.pytorch.ClassificationModule`, which is the underlying LightningModule
# behind :class:`~nni.retiarii.evaluator.pytorch.Classification`.
# Since we are merely adding a few ingredients to :class:`~nni.nas.evaluator.pytorch.Classification`,
# we can simply inherit :class:`~nni.nas.evaluator.pytorch.ClassificationModule`, which is the underlying LightningModule
# behind :class:`~nni.nas.evaluator.pytorch.Classification`.
# This could look intimidating at first, but most of them are just plug-and-play tricks which you don't need to know details about.
import torch
from nni.retiarii.evaluator.pytorch import ClassificationModule
from nni.nas.evaluator.pytorch import ClassificationModule
class DartsClassificationModule(ClassificationModule):
def __init__(
@ -541,10 +539,10 @@ class DartsClassificationModule(ClassificationModule):
#
# The full evaluator is written as follows,
# which simply wraps everything (except model space and search strategy of course), in a single object.
# :class:`~nni.retiarii.evaluator.pytorch.Lightning` here is a special type of evaluator.
# :class:`~nni.nas.evaluator.pytorch.Lightning` here is a special type of evaluator.
# Don't forget to use the train/val data split specialized for search (1:1) here.
from nni.retiarii.evaluator.pytorch import Lightning, Trainer
from nni.nas.evaluator.pytorch import Lightning, Trainer
max_epochs = 50
@ -564,10 +562,10 @@ evaluator = Lightning(
# Strategy
# ^^^^^^^^
#
# :class:`~nni.retiarii.strategy.DARTS` strategy is created with gradient clip turned on.
# :class:`~nni.nas.strategy.DARTS` strategy is created with gradient clip turned on.
# If you are familiar with PyTorch-Lightning, you might aware that gradient clipping can be enabled in Lightning trainer.
# However, enabling gradient clip in the trainer above won't work, because the underlying
# implementation of :class:`~nni.retiarii.strategy.DARTS` strategy is based on
# implementation of :class:`~nni.nas.strategy.DARTS` strategy is based on
# `manual optimization <https://pytorch-lightning.readthedocs.io/en/stable/common/optimization.html>`__.
strategy = DartsStrategy(gradient_clip_val=5.)
@ -586,11 +584,10 @@ strategy = DartsStrategy(gradient_clip_val=5.)
model_space = DartsSpace(width=16, num_cells=8, dataset='cifar')
config = RetiariiExeConfig(execution_engine='oneshot')
experiment = RetiariiExperiment(model_space, evaluator=evaluator, strategy=strategy)
experiment.run(config)
experiment = NasExperiment(model_space, evaluator, strategy)
experiment.run()
exported_arch = experiment.export_top_models()[0]
exported_arch = experiment.export_top_models(formatter='dict')[0]
exported_arch
@ -684,7 +681,7 @@ train_loader_cutout = DataLoader(train_data_cutout, batch_size=96)
# Following the same procedure as paper, we also increase the number of filters to 36, and number of cells to 20,
# so as to reasonably increase the model size and boost the performance.
with fixed_arch(exported_arch):
with model_context(exported_arch):
final_model = DartsSpace(width=36, num_cells=20, dataset='cifar', auxiliary_loss=True, drop_path_prob=0.2)
# %%

132
examples/tutorials/hello_nas.py
Просмотреть файл

@ -12,7 +12,7 @@ There are mainly three crucial components for a neural architecture search task,
* A proper strategy as the method to explore this model space.
* A model evaluator that reports the performance of every model in the space.
Currently, PyTorch is the only supported framework by Retiarii, and we have only tested **PyTorch 1.7 to 1.10**.
Currently, PyTorch is the only supported framework by Retiarii, and we have only tested **PyTorch 1.9 to 1.13**.
This tutorial assumes PyTorch context but it should also apply to other frameworks, which is in our future plan.
Define your Model Space
@ -28,19 +28,17 @@ In this framework, a model space is defined with two parts: a base model and pos
# ^^^^^^^^^^^^^^^^^
#
# Defining a base model is almost the same as defining a PyTorch (or TensorFlow) model.
# Usually, you only need to replace the code ``import torch.nn as nn`` with
# ``import nni.retiarii.nn.pytorch as nn`` to use our wrapped PyTorch modules.
#
# Below is a very simple example of defining a base model.
import torch
import torch.nn as nn
import torch.nn.functional as F
import nni.retiarii.nn.pytorch as nn
from nni.retiarii import model_wrapper
import nni
from nni.nas.nn.pytorch import LayerChoice, ModelSpace, MutableDropout, MutableLinear
@model_wrapper # this decorator should be put on the out most
class Net(nn.Module):
class Net(ModelSpace): # should inherit ModelSpace rather than nn.Module
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, 3, 1)
@ -59,12 +57,9 @@ class Net(nn.Module):
return output
# %%
# .. tip:: Always keep in mind that you should use ``import nni.retiarii.nn.pytorch as nn`` and :meth:`nni.retiarii.model_wrapper`.
# Many mistakes are a result of forgetting one of those.
# Also, please use ``torch.nn`` for submodules of ``nn.init``, e.g., ``torch.nn.init`` instead of ``nn.init``.
#
# Define Model Mutations
# ^^^^^^^^^^^^^^^^^^^^^^
# Define Model Variations
# ^^^^^^^^^^^^^^^^^^^^^^^
#
# A base model is only one concrete model not a model space. We provide :doc:`API and Primitives </nas/construct_space>`
# for users to express how the base model can be mutated. That is, to build a model space which includes many models.
@ -73,24 +68,23 @@ class Net(nn.Module):
#
# .. code-block:: diff
#
# @model_wrapper
# class Net(nn.Module):
# class Net(ModelSpace):
# def __init__(self):
# super().__init__()
# self.conv1 = nn.Conv2d(1, 32, 3, 1)
# - self.conv2 = nn.Conv2d(32, 64, 3, 1)
# + self.conv2 = nn.LayerChoice([
# + self.conv2 = LayerChoice([
# + nn.Conv2d(32, 64, 3, 1),
# + DepthwiseSeparableConv(32, 64)
# + ])
# + ], label='conv2)
# - self.dropout1 = nn.Dropout(0.25)
# + self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75]))
# + self.dropout1 = MutableDropout(nni.choice('dropout', [0.25, 0.5, 0.75]))
# self.dropout2 = nn.Dropout(0.5)
# - self.fc1 = nn.Linear(9216, 128)
# - self.fc2 = nn.Linear(128, 10)
# + feature = nn.ValueChoice([64, 128, 256])
# + self.fc1 = nn.Linear(9216, feature)
# + self.fc2 = nn.Linear(feature, 10)
# + feature = nni.choice('feature', [64, 128, 256])
# + self.fc1 = MutableLinear(9216, feature)
# + self.fc2 = MutableLinear(feature, 10)
#
# def forward(self, x):
# x = F.relu(self.conv1(x))
@ -113,24 +107,22 @@ class DepthwiseSeparableConv(nn.Module):
return self.pointwise(self.depthwise(x))
@model_wrapper
class ModelSpace(nn.Module):
class MyModelSpace(ModelSpace):
def __init__(self):
super().__init__()
self.conv1 = nn.Conv2d(1, 32, 3, 1)
# LayerChoice is used to select a layer between Conv2d and DwConv.
self.conv2 = nn.LayerChoice([
self.conv2 = LayerChoice([
nn.Conv2d(32, 64, 3, 1),
DepthwiseSeparableConv(32, 64)
])
# ValueChoice is used to select a dropout rate.
# ValueChoice can be used as parameter of modules wrapped in `nni.retiarii.nn.pytorch`
# or customized modules wrapped with `@basic_unit`.
self.dropout1 = nn.Dropout(nn.ValueChoice([0.25, 0.5, 0.75])) # choose dropout rate from 0.25, 0.5 and 0.75
], label='conv2')
# nni.choice is used to select a dropout rate.
# The result can be used as parameters of `MutableXXX`.
self.dropout1 = MutableDropout(nni.choice('dropout', [0.25, 0.5, 0.75])) # choose dropout rate from 0.25, 0.5 and 0.75
self.dropout2 = nn.Dropout(0.5)
feature = nn.ValueChoice([64, 128, 256])
self.fc1 = nn.Linear(9216, feature)
self.fc2 = nn.Linear(feature, 10)
feature = nni.choice('feature', [64, 128, 256])
self.fc1 = MutableLinear(9216, feature)
self.fc2 = MutableLinear(feature, 10)
def forward(self, x):
x = F.relu(self.conv1(x))
@ -141,18 +133,17 @@ class ModelSpace(nn.Module):
return output
model_space = ModelSpace()
model_space = MyModelSpace()
model_space
# %%
# This example uses two mutation APIs,
# :class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>` and
# :class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>`.
# :class:`nn.LayerChoice <nni.retiarii.nn.pytorch.LayerChoice>`
# :class:`nn.LayerChoice <nni.nas.nn.pytorch.LayerChoice>` and
# :func:`nni.choice`.
# :class:`nn.LayerChoice <nni.nas.nn.pytorch.LayerChoice>`
# takes a list of candidate modules (two in this example), one will be chosen for each sampled model.
# It can be used like normal PyTorch module.
# :class:`nn.InputChoice <nni.retiarii.nn.pytorch.ValueChoice>` takes a list of candidate values,
# one will be chosen to take effect for each sampled model.
# :func:`nni.choice` is used as parameter of `MutableDropout`, which then takes the result as dropout rate.
#
# More detailed API description and usage can be found :doc:`here </nas/construct_space>`.
#
@ -176,12 +167,12 @@ model_space
# Pick an exploration strategy
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
#
# Retiarii supports many :doc:`exploration strategies </nas/exploration_strategy>`.
# NNI NAS supports many :doc:`exploration strategies </nas/exploration_strategy>`.
#
# Simply choosing (i.e., instantiate) an exploration strategy as below.
import nni.retiarii.strategy as strategy
search_strategy = strategy.Random(dedup=True) # dedup=False if deduplication is not wanted
import nni.nas.strategy as strategy
search_strategy = strategy.Random() # dedup=False if deduplication is not wanted
# %%
# Pick or customize a model evaluator
@ -191,8 +182,8 @@ search_strategy = strategy.Random(dedup=True) # dedup=False if deduplication is
# and validating each generated model to obtain the model's performance.
# The performance is sent to the exploration strategy for the strategy to generate better models.
#
# Retiarii has provided :doc:`built-in model evaluators </nas/evaluator>`, but to start with,
# it is recommended to use :class:`FunctionalEvaluator <nni.retiarii.evaluator.FunctionalEvaluator>`,
# NNI NAS has provided :doc:`built-in model evaluators </nas/evaluator>`, but to start with,
# it is recommended to use :class:`FunctionalEvaluator <nni.nas.evaluator.FunctionalEvaluator>`,
# that is, to wrap your own training and evaluation code with one single function.
# This function should receive one single model class and uses :func:`nni.report_final_result` to report the final score of this model.
#
@ -241,10 +232,8 @@ def test_epoch(model, device, test_loader):
return accuracy
def evaluate_model(model_cls):
# "model_cls" is a class, need to instantiate
model = model_cls()
def evaluate_model(model):
# By v3.0, the model will be instantiated by default.
device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')
model.to(device)
@ -268,7 +257,7 @@ def evaluate_model(model_cls):
# %%
# Create the evaluator
from nni.retiarii.evaluator import FunctionalEvaluator
from nni.nas.evaluator import FunctionalEvaluator
evaluator = FunctionalEvaluator(evaluate_model)
# %%
@ -276,40 +265,41 @@ evaluator = FunctionalEvaluator(evaluate_model)
# The ``train_epoch`` and ``test_epoch`` here can be any customized function,
# where users can write their own training recipe.
#
# It is recommended that the ``evaluate_model`` here accepts no additional arguments other than ``model_cls``.
# It is recommended that the ``evaluate_model`` here accepts no additional arguments other than ``model``.
# However, in the :doc:`advanced tutorial </nas/evaluator>`, we will show how to use additional arguments in case you actually need those.
# In future, we will support mutation on the arguments of evaluators, which is commonly called "Hyper-parmeter tuning".
# In future, we will support mutation on the arguments of evaluators, which is commonly called "Hyper-parameter tuning".
#
# Launch an Experiment
# --------------------
#
# After all the above are prepared, it is time to start an experiment to do the model search. An example is shown below.
from nni.retiarii.experiment.pytorch import RetiariiExperiment, RetiariiExeConfig
exp = RetiariiExperiment(model_space, evaluator, [], search_strategy)
exp_config = RetiariiExeConfig('local')
exp_config.experiment_name = 'mnist_search'
from nni.nas.experiment import NasExperiment
exp = NasExperiment(model_space, evaluator, search_strategy)
# %%
# The following configurations are useful to control how many trials to run at most / at the same time.
# Different from HPO experiment, NAS experiment will generate an experiment config automatically.
# It should work for most cases. For example, when using multi-trial strategies,
# local training service with concurrency 1 will be used by default.
# Users can customize the config. For example,
exp_config.max_trial_number = 4 # spawn 4 trials at most
exp_config.trial_concurrency = 2 # will run two trials concurrently
exp.config.max_trial_number = 3 # spawn 3 trials at most
exp.config.trial_concurrency = 1 # will run 1 trial concurrently
exp.config.trial_gpu_number = 0 # will not use GPU
# %%
# Remember to set the following config if you want to GPU.
# ``use_active_gpu`` should be set true if you wish to use an occupied GPU (possibly running a GUI).
exp_config.trial_gpu_number = 1
exp_config.training_service.use_active_gpu = True
# %%
# ``use_active_gpu`` should be set true if you wish to use an occupied GPU (possibly running a GUI)::
#
# exp.config.trial_gpu_number = 1
# exp.config.training_service.use_active_gpu = True
#
# Launch the experiment. The experiment should take several minutes to finish on a workstation with 2 GPUs.
exp.run(exp_config, 8081)
exp.run(port=8081)
# %%
# Users can also run Retiarii Experiment with :doc:`different training services </experiment/training_service/overview>`
# Users can also run NAS Experiment with :doc:`different training services </experiment/training_service/overview>`
# besides ``local`` training service.
#
# Visualize the Experiment
@ -332,14 +322,13 @@ import os
from pathlib import Path
def evaluate_model_with_visualization(model_cls):
model = model_cls()
def evaluate_model_with_visualization(model):
# dump the model into an onnx
if 'NNI_OUTPUT_DIR' in os.environ:
dummy_input = torch.zeros(1, 3, 32, 32)
torch.onnx.export(model, (dummy_input, ),
Path(os.environ['NNI_OUTPUT_DIR']) / 'model.onnx')
evaluate_model(model_cls)
evaluate_model(model)
# %%
# Relaunch the experiment, and a button is shown on Web portal.
@ -353,12 +342,3 @@ def evaluate_model_with_visualization(model_cls):
for model_dict in exp.export_top_models(formatter='dict'):
print(model_dict)
# %%
# The output is ``json`` object which records the mutation actions of the top model.
# If users want to output source code of the top model,
# they can use :ref:`graph-based execution engine <graph-based-execution-engine>` for the experiment,
# by simply adding the following two lines.
exp_config.execution_engine = 'base'
export_formatter = 'code'

6
examples/tutorials/nasbench_as_dataset.py
Просмотреть файл

@ -23,9 +23,9 @@ os.listdir(os.path.expanduser('~/.cache/nni/nasbenchmark'))
# %%
import pprint
from nni.nas.benchmarks.nasbench101 import query_nb101_trial_stats
from nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats
from nni.nas.benchmarks.nds import query_nds_trial_stats
from nni.nas.benchmark.nasbench101 import query_nb101_trial_stats
from nni.nas.benchmark.nasbench201 import query_nb201_trial_stats
from nni.nas.benchmark.nds import query_nds_trial_stats
# %%
# NAS-Bench-101

4
nni/nas/benchmark/__init__.py
Просмотреть файл

@ -1,12 +1,14 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.
import warnings
try:
import peewee
except ImportError:
import warnings
warnings.warn('peewee is not installed. Please install it to use NAS benchmarks.')
del peewee
from .evaluator import *
from .space import *
from .utils import load_benchmark, download_benchmark

1
nni/nas/execution/cgo/__init__.py
Просмотреть файл

@ -1,4 +1,5 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT license.
from .evaluator import *
from .middleware import CrossGraphOptimization

2
nni/nas/hub/pytorch/modules/nasbench101.py
Просмотреть файл

@ -295,7 +295,7 @@ class NasBench101Cell(MutableModule):
Warnings
--------
:class:`NasBench101Cell` is not supported in :ref:`graph-based execution engine <graph-based-execution-engine>`.
:class:`NasBench101Cell` is not supported for graph-based model format.
It's also not supported by most one-shot algorithms currently.
"""

4
nni/nas/nn/pytorch/cell.py
Просмотреть файл

@ -108,7 +108,7 @@ class Cell(MutableModule):
`review article <https://sh-tsang.medium.com/review-nasnet-neural-architecture-search-network-image-classification-23139ea0425d>`__
if you are interested in details.
.. image:: ../../../img/nasnet_cell.png
.. image:: ../../img/nasnet_cell.png
:width: 900
:align: center
@ -230,7 +230,7 @@ class Cell(MutableModule):
Warnings
--------
:class:`Cell` is not supported in :ref:`graph-based execution engine <graph-based-execution-engine>`.
:class:`Cell` is not supported in :class:`~nni.nas.space.GraphModelSpace` model format.
Attributes
----------

11
nni/nas/oneshot/pytorch/profiler.py
Просмотреть файл

@ -12,6 +12,11 @@ It might be moved to a more general place in the future.
from __future__ import annotations
__all__ = [
'ProfilerFilter', 'RangeProfilerFilter', 'ProfilerPenalty',
'ExpectationProfilerPenalty', 'SampleProfilerPenalty'
]
import logging
from typing import cast
from typing_extensions import Literal
@ -179,6 +184,9 @@ class ProfilerPenalty(nn.Module):
class ExpectationProfilerPenalty(ProfilerPenalty):
""":class:`ProfilerPenalty` for a sample with distributions.
Value for each label is a a mapping from chosen value to probablity.
"""
def profile(self, sample: Sample) -> float:
"""Profile based on a distribution of samples.
@ -195,6 +203,9 @@ class ExpectationProfilerPenalty(ProfilerPenalty):
class SampleProfilerPenalty(ProfilerPenalty):
""":class:`ProfilerPenalty` for a single sample.
Value for each label is a specifically chosen value.
"""
def profile(self, sample: Sample) -> float:
"""Profile based on a single sample."""