Refine readme.md and notebook
This commit is contained in:
jiahangxu
Родитель
a06ebabfe6
Коммит
1bb4485493
|
|
@ -223,9 +223,9 @@ In `exp_config`, `dummy_input` is required for tracing shape info.
|
||||||
|
|
||||||
## Bench Dataset
|
## Bench Dataset
|
||||||
|
|
||||||
[BRP-NAS](https://arxiv.org/abs/2007.08668v2) proposes an end-to-end latency predictor which consists of a GCN. Their GCN predictor demonstrates significant improvement over the layer-wise predictor on [NAS-Bench-201](https://arxiv.org/abs/2001.00326). While on our bench dataset, the preformance of BRP-NAS is consistently poor. As discussed in our paper, the reason is the model graph difference between training and testing set. GNN learns the representation of model graphs. Although the models in our bench dataset have largely overlapped operator types, the operator configurations, edges, and model latency ranges are different.
|
To evaluate the effectiveness of a prediction model on an arbitrary DNN model, we need a representative dataset that covers a large prediction scope. nn-Meter collects and generates 26k CNN models. (Please refer the paper for the dataset generation method.)
|
||||||
|
|
||||||
To better deal with the problems above, we give a GNN example with graph representation improved. We build the data structure of our model in `GNNDataset` and `GNNDataloader` from `nn_meter/dataset/gnn_dataloader.py` to convert the Dataset in `.jsonl` format into GNN required Dataset and Dataloader. And for specific use cases, please refer to `examples/gnn_for_bench_dataset.ipynb`.
|
We release the dataset, and provide an interface of `nn_meter.dataset` for users to get access to the dataset. Users can also download the data from the [Download Link](https://github.com/microsoft/nn-Meter/releases/download/v1.0-data/datasets.zip) on their own.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -22,7 +22,6 @@ Finally, we provide more hardware-ware NAS examples in NNI.
|
||||||
2. [Use nn-Meter with the bench dataset](nn-meter_for_bench_dataset.ipynb)
|
2. [Use nn-Meter with the bench dataset](nn-meter_for_bench_dataset.ipynb)
|
||||||
3. [Use bench dataset for GNN training](gnn_for_bench_dataset.ipynb)
|
3. [Use bench dataset for GNN training](gnn_for_bench_dataset.ipynb)
|
||||||
4. Use nn-Meter to construct latency constraint in SPOS NAS (TBD)
|
4. Use nn-Meter to construct latency constraint in SPOS NAS (TBD)
|
||||||
|
|
||||||
- [Use nn-Meter in search part](https://github.com/microsoft/nni/blob/master/examples/nas/oneshot/spos/multi_trial.py)
|
- [Use nn-Meter in search part](https://github.com/microsoft/nni/blob/master/examples/nas/oneshot/spos/multi_trial.py)
|
||||||
- [Use nn-Meter in sampling part](https://github.com/microsoft/nni/blob/master/examples/nas/oneshot/spos/supernet.py)
|
- [Use nn-Meter in sampling part](https://github.com/microsoft/nni/blob/master/examples/nas/oneshot/spos/supernet.py)
|
||||||
5. [Use nn-Meter to construct latency penalty in Proxyless NAS](https://github.com/microsoft/nni/tree/master/examples/nas/oneshot/proxylessnas)
|
5. [Use nn-Meter to construct latency penalty in Proxyless NAS](https://github.com/microsoft/nni/tree/master/examples/nas/oneshot/proxylessnas)
|
||||||
|
|
|
||||||
|
|
@ -3,10 +3,14 @@
|
||||||
{
|
{
|
||||||
"cell_type": "markdown",
|
"cell_type": "markdown",
|
||||||
"source": [
|
"source": [
|
||||||
"# Latency Dataset - GNN Model\r\n",
|
"# Latency Dataset - GNN Model\n",
|
||||||
"\r\n",
|
"\n",
|
||||||
"Considering the dataset is encoded in a graph format, here is an example of using GNN to predict the model latency with the bench dataset. `GNNDataset` and `GNNDataloader` in `nn_meter/dataset/gnn_dataloader.py` build the model structure of the Dataset in `.jsonl` format into our required Dataset and Dataloader. We will first build our GNN model, which is constructed based on GraphSAGE, and maxpooling is selected as out pooling method. Next, we will start training after the data is loaded.\r\n",
|
"Considering the dataset is encoded in a graph format, here is an example of using GNN to predict the model latency with the bench dataset. \n",
|
||||||
"\r\n",
|
"\n",
|
||||||
|
"In the previous work of [BRP-NAS](https://arxiv.org/abs/2007.08668v2), the authors propose an end-to-end latency predictor which consists of a GCN. Their GCN predictor demonstrates significant improvement over the layer-wise predictor on [NAS-Bench-201](https://arxiv.org/abs/2001.00326). While on our bench dataset, the preformance of BRP-NAS is consistently poor. As discussed in our paper, the reason is the model graph difference between training and testing set. GNN learns the representation of model graphs. Although the models in our bench dataset have largely overlapped operator types, the operator configurations, edges, and model latency ranges are different.\n",
|
||||||
|
"\n",
|
||||||
|
"To better deal with the problems above, we give a GNN example with graph representation improved. We first build our GNN model, which is constructed based on GraphSAGE, and maxpooling is selected as out pooling method. Next, we will start training after the data is loaded. `GNNDataset` and `GNNDataloader` in `nn_meter/dataset/gnn_dataloader.py` build the model structure of the Dataset in `.jsonl` format into our required Dataset and Dataloader. \n",
|
||||||
|
"\n",
|
||||||
"Let's start our journey!"
|
"Let's start our journey!"
|
||||||
],
|
],
|
||||||
"metadata": {}
|
"metadata": {}
|
||||||
|
|
@ -24,49 +28,49 @@
|
||||||
"cell_type": "code",
|
"cell_type": "code",
|
||||||
"execution_count": 1,
|
"execution_count": 1,
|
||||||
"source": [
|
"source": [
|
||||||
"import torch\r\n",
|
"import torch\n",
|
||||||
"import torch.nn as nn\r\n",
|
"import torch.nn as nn\n",
|
||||||
"from torch.nn.modules.module import Module\r\n",
|
"from torch.nn.modules.module import Module\n",
|
||||||
"\r\n",
|
"\n",
|
||||||
"from dgl.nn.pytorch.glob import MaxPooling\r\n",
|
"from dgl.nn.pytorch.glob import MaxPooling\n",
|
||||||
"import dgl.nn as dglnn\r\n",
|
"import dgl.nn as dglnn\n",
|
||||||
"from torch.optim.lr_scheduler import CosineAnnealingLR\r\n",
|
"from torch.optim.lr_scheduler import CosineAnnealingLR\n",
|
||||||
"\r\n",
|
"\n",
|
||||||
"\r\n",
|
"\n",
|
||||||
"class GNN(Module):\r\n",
|
"class GNN(Module):\n",
|
||||||
" def __init__(self, \r\n",
|
" def __init__(self, \n",
|
||||||
" num_features=0, \r\n",
|
" num_features=0, \n",
|
||||||
" num_layers=2,\r\n",
|
" num_layers=2,\n",
|
||||||
" num_hidden=32,\r\n",
|
" num_hidden=32,\n",
|
||||||
" dropout_ratio=0):\r\n",
|
" dropout_ratio=0):\n",
|
||||||
"\r\n",
|
"\n",
|
||||||
" super(GNN, self).__init__()\r\n",
|
" super(GNN, self).__init__()\n",
|
||||||
" self.nfeat = num_features\r\n",
|
" self.nfeat = num_features\n",
|
||||||
" self.nlayer = num_layers\r\n",
|
" self.nlayer = num_layers\n",
|
||||||
" self.nhid = num_hidden\r\n",
|
" self.nhid = num_hidden\n",
|
||||||
" self.dropout_ratio = dropout_ratio\r\n",
|
" self.dropout_ratio = dropout_ratio\n",
|
||||||
" self.gc = nn.ModuleList([dglnn.SAGEConv(self.nfeat if i==0 else self.nhid, self.nhid, 'pool') for i in range(self.nlayer)])\r\n",
|
" self.gc = nn.ModuleList([dglnn.SAGEConv(self.nfeat if i==0 else self.nhid, self.nhid, 'pool') for i in range(self.nlayer)])\n",
|
||||||
" self.bn = nn.ModuleList([nn.LayerNorm(self.nhid) for i in range(self.nlayer)])\r\n",
|
" self.bn = nn.ModuleList([nn.LayerNorm(self.nhid) for i in range(self.nlayer)])\n",
|
||||||
" self.relu = nn.ModuleList([nn.ReLU() for i in range(self.nlayer)])\r\n",
|
" self.relu = nn.ModuleList([nn.ReLU() for i in range(self.nlayer)])\n",
|
||||||
" self.pooling = MaxPooling()\r\n",
|
" self.pooling = MaxPooling()\n",
|
||||||
" self.fc = nn.Linear(self.nhid, 1)\r\n",
|
" self.fc = nn.Linear(self.nhid, 1)\n",
|
||||||
" self.fc1 = nn.Linear(self.nhid, self.nhid)\r\n",
|
" self.fc1 = nn.Linear(self.nhid, self.nhid)\n",
|
||||||
" self.dropout = nn.ModuleList([nn.Dropout(self.dropout_ratio) for i in range(self.nlayer)])\r\n",
|
" self.dropout = nn.ModuleList([nn.Dropout(self.dropout_ratio) for i in range(self.nlayer)])\n",
|
||||||
"\r\n",
|
"\n",
|
||||||
" def forward_single_model(self, g, features):\r\n",
|
" def forward_single_model(self, g, features):\n",
|
||||||
" x = self.relu[0](self.bn[0](self.gc[0](g, features)))\r\n",
|
" x = self.relu[0](self.bn[0](self.gc[0](g, features)))\n",
|
||||||
" x = self.dropout[0](x)\r\n",
|
" x = self.dropout[0](x)\n",
|
||||||
" for i in range(1,self.nlayer):\r\n",
|
" for i in range(1,self.nlayer):\n",
|
||||||
" x = self.relu[i](self.bn[i](self.gc[i](g, x)))\r\n",
|
" x = self.relu[i](self.bn[i](self.gc[i](g, x)))\n",
|
||||||
" x = self.dropout[i](x)\r\n",
|
" x = self.dropout[i](x)\n",
|
||||||
" return x\r\n",
|
" return x\n",
|
||||||
"\r\n",
|
"\n",
|
||||||
" def forward(self, g, features):\r\n",
|
" def forward(self, g, features):\n",
|
||||||
" x = self.forward_single_model(g, features)\r\n",
|
" x = self.forward_single_model(g, features)\n",
|
||||||
" with g.local_scope():\r\n",
|
" with g.local_scope():\n",
|
||||||
" g.ndata['h'] = x\r\n",
|
" g.ndata['h'] = x\n",
|
||||||
" x = self.pooling(g, x)\r\n",
|
" x = self.pooling(g, x)\n",
|
||||||
" x = self.fc1(x)\r\n",
|
" x = self.fc1(x)\n",
|
||||||
" return self.fc(x)"
|
" return self.fc(x)"
|
||||||
],
|
],
|
||||||
"outputs": [
|
"outputs": [
|
||||||
|
|
|
||||||
Загрузка…
Ссылка в новой задаче