microsoft
/
nn-Meter
зеркало из
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность

Refine readme.md and notebook

This commit is contained in:
jiahangxu 2021-10-08 17:13:45 +08:00
Родитель a06ebabfe6
Коммит 1bb4485493
3 изменённых файлов: 53 добавлений и 50 удалений
Показать статистику Скачать Patch файл Скачать Diff файл Показать все файлы Свернуть все файлы

4
README.md
Просмотреть файл

@ -223,9 +223,9 @@ In `exp_config`, `dummy_input` is required for tracing shape info.
## Bench Dataset
[BRP-NAS](https://arxiv.org/abs/2007.08668v2) proposes an end-to-end latency predictor which consists of a GCN. Their GCN predictor demonstrates significant improvement over the layer-wise predictor on [NAS-Bench-201](https://arxiv.org/abs/2001.00326). While on our bench dataset, the preformance of BRP-NAS is consistently poor. As discussed in our paper, the reason is the model graph difference between training and testing set. GNN learns the representation of model graphs. Although the models in our bench dataset have largely overlapped operator types, the operator configurations, edges, and model latency ranges are different.
To evaluate the effectiveness of a prediction model on an arbitrary DNN model, we need a representative dataset that covers a large prediction scope. nn-Meter collects and generates 26k CNN models. (Please refer the paper for the dataset generation method.)
To better deal with the problems above, we give a GNN example with graph representation improved. We build the data structure of our model in `GNNDataset` and `GNNDataloader` from `nn_meter/dataset/gnn_dataloader.py` to convert the Dataset in `.jsonl` format into GNN required Dataset and Dataloader. And for specific use cases, please refer to `examples/gnn_for_bench_dataset.ipynb`.
We release the dataset, and provide an interface of `nn_meter.dataset` for users to get access to the dataset. Users can also download the data from the [Download Link](https://github.com/microsoft/nn-Meter/releases/download/v1.0-data/datasets.zip) on their own.

1
examples/README.md
Просмотреть файл

@ -22,7 +22,6 @@ Finally, we provide more hardware-ware NAS examples in NNI.
2. [Use nn-Meter with the bench dataset](nn-meter_for_bench_dataset.ipynb)
3. [Use bench dataset for GNN training](gnn_for_bench_dataset.ipynb)
4. Use nn-Meter to construct latency constraint in SPOS NAS (TBD)
- [Use nn-Meter in search part](https://github.com/microsoft/nni/blob/master/examples/nas/oneshot/spos/multi_trial.py)
- [Use nn-Meter in sampling part](https://github.com/microsoft/nni/blob/master/examples/nas/oneshot/spos/supernet.py)
5. [Use nn-Meter to construct latency penalty in Proxyless NAS](https://github.com/microsoft/nni/tree/master/examples/nas/oneshot/proxylessnas)

98
examples/gnn_for_bench_dataset.ipynb
Просмотреть файл

@ -3,10 +3,14 @@
{
"cell_type": "markdown",
"source": [
"# Latency Dataset - GNN Model\r\n",
"\r\n",
"Considering the dataset is encoded in a graph format, here is an example of using GNN to predict the model latency with the bench dataset. `GNNDataset` and `GNNDataloader` in `nn_meter/dataset/gnn_dataloader.py` build the model structure of the Dataset in `.jsonl` format into our required Dataset and Dataloader. We will first build our GNN model, which is constructed based on GraphSAGE, and maxpooling is selected as out pooling method. Next, we will start training after the data is loaded.\r\n",
"\r\n",
"# Latency Dataset - GNN Model\n",
"\n",
"Considering the dataset is encoded in a graph format, here is an example of using GNN to predict the model latency with the bench dataset. \n",
"\n",
"In the previous work of [BRP-NAS](https://arxiv.org/abs/2007.08668v2), the authors propose an end-to-end latency predictor which consists of a GCN. Their GCN predictor demonstrates significant improvement over the layer-wise predictor on [NAS-Bench-201](https://arxiv.org/abs/2001.00326). While on our bench dataset, the preformance of BRP-NAS is consistently poor. As discussed in our paper, the reason is the model graph difference between training and testing set. GNN learns the representation of model graphs. Although the models in our bench dataset have largely overlapped operator types, the operator configurations, edges, and model latency ranges are different.\n",
"\n",
"To better deal with the problems above, we give a GNN example with graph representation improved. We first build our GNN model, which is constructed based on GraphSAGE, and maxpooling is selected as out pooling method. Next, we will start training after the data is loaded. `GNNDataset` and `GNNDataloader` in `nn_meter/dataset/gnn_dataloader.py` build the model structure of the Dataset in `.jsonl` format into our required Dataset and Dataloader. \n",
"\n",
"Let's start our journey!"
],
"metadata": {}
@ -24,49 +28,49 @@
"cell_type": "code",
"execution_count": 1,
"source": [
"import torch\r\n",
"import torch.nn as nn\r\n",
"from torch.nn.modules.module import Module\r\n",
"\r\n",
"from dgl.nn.pytorch.glob import MaxPooling\r\n",
"import dgl.nn as dglnn\r\n",
"from torch.optim.lr_scheduler import CosineAnnealingLR\r\n",
"\r\n",
"\r\n",
"class GNN(Module):\r\n",
" def __init__(self, \r\n",
" num_features=0, \r\n",
" num_layers=2,\r\n",
" num_hidden=32,\r\n",
" dropout_ratio=0):\r\n",
"\r\n",
" super(GNN, self).__init__()\r\n",
" self.nfeat = num_features\r\n",
" self.nlayer = num_layers\r\n",
" self.nhid = num_hidden\r\n",
" self.dropout_ratio = dropout_ratio\r\n",
" self.gc = nn.ModuleList([dglnn.SAGEConv(self.nfeat if i==0 else self.nhid, self.nhid, 'pool') for i in range(self.nlayer)])\r\n",
" self.bn = nn.ModuleList([nn.LayerNorm(self.nhid) for i in range(self.nlayer)])\r\n",
" self.relu = nn.ModuleList([nn.ReLU() for i in range(self.nlayer)])\r\n",
" self.pooling = MaxPooling()\r\n",
" self.fc = nn.Linear(self.nhid, 1)\r\n",
" self.fc1 = nn.Linear(self.nhid, self.nhid)\r\n",
" self.dropout = nn.ModuleList([nn.Dropout(self.dropout_ratio) for i in range(self.nlayer)])\r\n",
"\r\n",
" def forward_single_model(self, g, features):\r\n",
" x = self.relu[0](self.bn[0](self.gc[0](g, features)))\r\n",
" x = self.dropout[0](x)\r\n",
" for i in range(1,self.nlayer):\r\n",
" x = self.relu[i](self.bn[i](self.gc[i](g, x)))\r\n",
" x = self.dropout[i](x)\r\n",
" return x\r\n",
"\r\n",
" def forward(self, g, features):\r\n",
" x = self.forward_single_model(g, features)\r\n",
" with g.local_scope():\r\n",
" g.ndata['h'] = x\r\n",
" x = self.pooling(g, x)\r\n",
" x = self.fc1(x)\r\n",
"import torch\n",
"import torch.nn as nn\n",
"from torch.nn.modules.module import Module\n",
"\n",
"from dgl.nn.pytorch.glob import MaxPooling\n",
"import dgl.nn as dglnn\n",
"from torch.optim.lr_scheduler import CosineAnnealingLR\n",
"\n",
"\n",
"class GNN(Module):\n",
" def __init__(self, \n",
" num_features=0, \n",
" num_layers=2,\n",
" num_hidden=32,\n",
" dropout_ratio=0):\n",
"\n",
" super(GNN, self).__init__()\n",
" self.nfeat = num_features\n",
" self.nlayer = num_layers\n",
" self.nhid = num_hidden\n",
" self.dropout_ratio = dropout_ratio\n",
" self.gc = nn.ModuleList([dglnn.SAGEConv(self.nfeat if i==0 else self.nhid, self.nhid, 'pool') for i in range(self.nlayer)])\n",
" self.bn = nn.ModuleList([nn.LayerNorm(self.nhid) for i in range(self.nlayer)])\n",
" self.relu = nn.ModuleList([nn.ReLU() for i in range(self.nlayer)])\n",
" self.pooling = MaxPooling()\n",
" self.fc = nn.Linear(self.nhid, 1)\n",
" self.fc1 = nn.Linear(self.nhid, self.nhid)\n",
" self.dropout = nn.ModuleList([nn.Dropout(self.dropout_ratio) for i in range(self.nlayer)])\n",
"\n",
" def forward_single_model(self, g, features):\n",
" x = self.relu[0](self.bn[0](self.gc[0](g, features)))\n",
" x = self.dropout[0](x)\n",
" for i in range(1,self.nlayer):\n",
" x = self.relu[i](self.bn[i](self.gc[i](g, x)))\n",
" x = self.dropout[i](x)\n",
" return x\n",
"\n",
" def forward(self, g, features):\n",
" x = self.forward_single_model(g, features)\n",
" with g.local_scope():\n",
" g.ndata['h'] = x\n",
" x = self.pooling(g, x)\n",
" x = self.fc1(x)\n",
" return self.fc(x)"
],
"outputs": [