Update template

design/example-data.md

@@ -1,51 +0,0 @@
-# COVID-19 Open Research Dataset
-
-- Summary
-  
-  Open dataset of COVID19 from AI2, CZI, MSR, Georgetown, NIH & The White House.
-
-- Description
-
-  ## Dataset Description
-
-  In response to the COVID-19 pandemic, the White House and a coalition of leading research groups have prepared the COVID-19 Open Research Dataset (CORD-19). CORD-19 is a resource of over 134,000 scholarly articles, including over 60,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. This freely available dataset is provided to the global research community to apply recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease. There is a growing urgency for these approaches because of the rapid acceleration in new coronavirus literature, making it difficult for the medical research community to keep up.
-
-  ## Call to Action
-
-  We are issuing a call to action to the world's artificial intelligence experts to develop text and data mining tools that can help the medical community develop answers to high priority scientific questions. The CORD-19 dataset represents the most extensive machine-readable coronavirus literature collection available for data mining to date. This allows the worldwide AI research community the opportunity to apply text and data mining approaches to find answers to questions within, and connect insights across, this content in support of the ongoing COVID-19 response efforts worldwide. There is a growing urgency for these approaches because of the rapid increase in coronavirus literature, making it difficult for the medical community to keep up.
-
-  A list of our initial key questions can be found under the Tasks section of this dataset. These key scientific questions are drawn from the NASEM’s SCIED (National Academies of Sciences, Engineering, and Medicine’s Standing Committee on Emerging Infectious Diseases and 21st Century Health Threats) research topics and the World Health Organization’s R&D Blueprint for COVID-19.
-
-  Many of these questions are suitable for text mining, and we encourage researchers to develop text mining tools to provide insights on these questions.
-
-  We are maintaining a summary of the community's contributions. For guidance on how to make your contributions useful, we're maintaining a forum thread with the feedback we're getting from the medical and health policy communities.
-
-  ## Prizes
-
-  Kaggle is sponsoring a \$1,000 per task award to the winner whose submission is identified as best meeting the evaluation criteria. The winner may elect to receive this award as a charitable donation to COVID-19 relief/research efforts or as a monetary payment. More details on the prizes and timeline can be found on the discussion post.
-  Accessing the Dataset
-
-  ## Acknowledgements
-
-  This dataset was created by the Allen Institute for AI in partnership with the Chan Zuckerberg Initiative, Georgetown University’s Center for Security and Emerging Technology, Microsoft Research, and the National Library of Medicine - National Institutes of Health, in coordination with The White House Office of Science and Technology Policy.
-
-- Data Explorer
-
-  - row_right
-    folder
-
-  - arrow_right
-    folder
-
-  - arrow_right
-    folder
-
-  - COVID.DATA.LIC.AGMT.pdf
-
-  - json_schema.txt
-
-  - metadata.csv
-
-  - metadata.readme
-
-- Usage ???

design/example-image.md

@@ -1,40 +0,0 @@
-# marketplace image example - Transformer environment
-
-- Summary
-  
-  Transformer docker environment.
-
-- Description
-
-  ## transformers
-
-   Transformers (formerly known as pytorch-transformers and pytorch-pretrained-bert) provides state-of-the-art general-purpose architectures (BERT, GPT-2, RoBERTa, XLM, DistilBert, XLNet, T5, CTRL...) for Natural Language Understanding (NLU) and Natural Language Generation (NLG) with over thousands of pretrained models in 100+ languages and deep interoperability between PyTorch & TensorFlow 2.0.
-
-- Docker file
-
-FROM ubuntu:18.04
-LABEL maintainer="Hugging Face"
-LABEL repository="transformers"
-
-RUN apt update && \
-    apt install -y bash \
-                   build-essential \
-                   git \
-                   curl \
-                   ca-certificates \
-                   python3 \
-                   python3-pip && \
-    rm -rf /var/lib/apt/lists
-
-RUN python3 -m pip install --no-cache-dir --upgrade pip && \
-    python3 -m pip install --no-cache-dir \
-    jupyter \
-    tensorflow-cpu \
-    torch
-
-WORKDIR /workspace
-COPY . transformers/
-RUN cd transformers/ && \
-    python3 -m pip install --no-cache-dir .
-
-CMD ["/bin/bash"]

design/example_job.md

@@ -1,92 +0,0 @@
-# marketplace job example - Caffe minist example
-
-- Summary
-  
-  This is an [example caffe Docker image on OpenPAI](https://github.com/Microsoft/pai/tree/master/examples/caffe).
-
-- Description
-
-  # Caffe MNIST Example
-
-  This example shows how to train LeNet on MNIST with Caffe on OpenPAI.
-
-  ## Dataset
-
-  The MNIST dataset is downloaded from MNIST website and converted into caffe format.
-
-  ## LeNet
-
-  This example will use the LeNet network, which is known to work well on digit classification tasks.
-  It will use a slightly different version from the original LeNet implementation,
-  replacing the sigmoid activations with Rectified Linear Unit (ReLU) activations for the neurons.
-  The design of LeNet contains the essence of CNNs that are still used in larger models such as the ones in ImageNet.
-  In general, it consists of a convolutional layer followed by a pooling layer, another convolution layer followed by a pooling layer,
-  and then two fully connected layers similar to the conventional multilayer perceptrons.
-  The layers are defined in `$CAFFE_ROOT/examples/mnist/lenet_train_test.prototxt`.
-
-  ## Reference
-
-  http://caffe.berkeleyvision.org/gathered/examples/mnist.html
-
-- Platform resources
-
-   type: openpai int bed platform (https://pai.example.test.intbed.com)
-   cpu: 4
-   gpu: 1
-   memory: 8192mb
-
-- Platform config
-
-protocolVersion: 2
-name: caffe_mnist
-type: job
-version: 1.0
-contributor: OpenPAI
-description: |
-  # Caffe MNIST Example
-  This example shows how to train LeNet on MNIST with Caffe on OpenPAI.
-  ## Dataset
-  The MNIST dataset is downloaded from MNIST website and converted into caffe format.
-  ## LeNet
-  This example will use the LeNet network, which is known to work well on digit classification tasks.
-  It will use a slightly different version from the original LeNet implementation,
-  replacing the sigmoid activations with Rectified Linear Unit (ReLU) activations for the neurons.
-  The design of LeNet contains the essence of CNNs that are still used in larger models such as the ones in ImageNet.
-  In general, it consists of a convolutional layer followed by a pooling layer, another convolution layer followed by a pooling layer,
-  and then two fully connected layers similar to the conventional multilayer perceptrons.
-  The layers are defined in `$CAFFE_ROOT/examples/mnist/lenet_train_test.prototxt`.
-  ## Reference
-  http://caffe.berkeleyvision.org/gathered/examples/mnist.html
-prerequisites:
-  - protocolVersion: 2
-    name: caffe_example
-    type: dockerimage
-    version: 1.0
-    contributor : OpenPAI
-    description: |
-      This is an [example caffe Docker image on OpenPAI](https://github.com/Microsoft/pai/tree/master/examples/caffe).
-    uri : openpai/pai.example.caffe
-
-taskRoles:
-  train:
-    instances: 1
-    completion:
-      minSucceededInstances: 1
-    dockerImage: caffe_example
-    resourcePerInstance:
-      cpu: 4
-      memoryMB: 8192
-      gpu: 1
-    commands:
-      - ./examples/mnist/train_lenet.sh
-
-deployments:
-  - name: caffe_example
-    taskRoles:
-      train:
-        preCommands:
-          - ./data/mnist/get_mnist.sh
-          - ./examples/mnist/create_mnist.sh
-
-defaults:
-  deployment: caffe_example

examples/couplet_dataset.md

@@ -0,0 +1,27 @@
+# Couplet Dataset
+
+This is the dataset of couplet. 
+
+## Data content
+
+This dataset contains processed data based on [Microsoft AI EDU project](https://github.com/microsoft/ai-edu/blob/master/B-%E5%AE%9E%E8%B7%B5%E6%A1%88%E4%BE%8B/B13-AI%E5%AF%B9%E8%81%94%E7%94%9F%E6%88%90%E6%A1%88%E4%BE%8B/docs/fairseq.md).
+
+The original dataset was downloaded from [Public couplet dataset](https://github.com/wb14123/couplet-dataset) and was splited into ```test, train and valid``` with 98:1:1 proportion. The ```.up``` and ```.down``` files contains upper part and down part of a certain couplet seperately.
+
+## The file stucture
+
+```
+.
+|-- test.down // down part of couplet
+|-- test.up  // up part of couplet
+|-- train.down
+|-- train.up
+|-- valid.down
+|-- valid.up
+```
+
+## How to use it
+
+The data will be mounted at ```DATA_DIR``` environment variable. You could use ```$DATA_DIR``` in your command when submit jobs in pai.
+
+

examples/couplet_inference.md

@@ -0,0 +1,17 @@
+# Couplet Training Job Template
+
+This is a model inference process. The input data is the trainning models trained by ```couplet training job```, and the this job will produce a url for user to ask for down part for a upper part of couplet.
+
+## How to use
+
+When use this module, you should set three environment variables:
+
+- ```DATA_DIR```: the training model path in container, by default it uses the output of couplet training job. If you want to use your own models. First make sure mount your models into container, and then change the ```$DATA_DIR``` with the path.
+
+- ```CODE_DIR```: the service code, it will start a server at the given port.
+
+- ```FLASK_RUN_PORT```: the service port container will output.
+
+## How to check the result
+
+After job finished successfully, you could check the job detail to get the container ip and ```flask_port``` number, then go to http://<ip>:<flask_port>/upper=<input> to test the result.

examples/couplet_training.md

@@ -0,0 +1,21 @@
+# Couplet Training Job Template
+
+This is a model training process. After training, this model will give a down part with an upper part of couplet. Please refer to Microsoft AI Edu Project for more details.
+
+## Training Data
+
+You could use Couplet Dataset data component as training data, or any dataset follows fairseq model requirements.
+
+## How to use
+
+When use this module, you should set three environment variables:
+
+- ```DATA_DIR```: the training data path in container, by default it uses Couplet Dataset data component. If you want to use your own datasets. First make sure mount your data into container, and then change the ```$DATA_DIR``` with the path.
+
+- PREPROCESSED_DATA_DIR: the path to store intermediate result, by default it is ./processed_data.
+
+- ```OUTPUT_DIR```: the path to store output result, i.e. the training model files. By default it will mount a nfs storage, and you could change it with other mounted storage.
+
+## How to check the result
+
+After job finished successfully, you could check the output model files in the output storage. The storage server url is in details page.

webportal/src/app/utils/constants.js

@@ -15,7 +15,7 @@ export const MARKET_ITEM_LIST = [
     tags: ['official example'],
     summary: 'Dataset of couplet',
     description:
-      '# Couplet Dataset\n\nThis is the dataset of couplet. \n\n## Data content\n\nThis dataset contains processed data based on [Microsoft AI EDU project](https://github.com/microsoft/ai-edu/blob/master/B-%E5%AE%9E%E8%B7%B5%E6%A1%88%E4%BE%8B/B13-AI%E5%AF%B9%E8%81%94%E7%94%9F%E6%88%90%E6%A1%88%E4%BE%8B/docs/fairseq.md).\n\nThe original dataset was downloaded from [Public couplet dataset](https://github.com/wb14123/couplet-dataset) and was splited into ```test, train and valid``` with 98:1:1 proportion. The ```.up``` and ```.down``` files contains upper part and down part of a certain couplet seperately.\n\n## The file stucture\n\n```\n.\n|-- test.down // down part of couplet\n|-- test.up  // up part of couplet\n|-- train.down\n|-- train.up\n|-- valid.down\n|-- valid.up\n```\n\n## How to use it\n\nThe data was stored in a pai nfs storage. It will be mounted in container when you use the data in pai cluster.\n\n\n',
+      '# Couplet Dataset\n\nThis is the dataset of couplet. \n\n## Data content\n\nThis dataset contains processed data based on [Microsoft AI EDU project](https://github.com/microsoft/ai-edu/blob/master/B-%E5%AE%9E%E8%B7%B5%E6%A1%88%E4%BE%8B/B13-AI%E5%AF%B9%E8%81%94%E7%94%9F%E6%88%90%E6%A1%88%E4%BE%8B/docs/fairseq.md).\n\nThe original dataset was downloaded from [Public couplet dataset](https://github.com/wb14123/couplet-dataset) and was splited into ```test, train and valid``` with 98:1:1 proportion. The ```.up``` and ```.down``` files contains upper part and down part of a certain couplet seperately.\n\n## The file stucture\n\n```\n.\n|-- test.down // down part of couplet\n|-- test.up  // up part of couplet\n|-- train.down\n|-- train.up\n|-- valid.down\n|-- valid.up\n```\n\n## How to use it\n\nThe data will be mounted at ```DATA_DIR``` environment variable. You could use ```$DATA_DIR``` in your command when submit jobs in pai.\n\n\n',
     content: {
       dataStorage: {
         storageType: 'nfs',
@@ -33,14 +33,14 @@ export const MARKET_ITEM_LIST = [
   },
   {
     id: '8e0b3086-0359-4e75-b11c-c5527487626e',
-    name: 'Couplet Training Model',
+    name: 'Couplet Training Job Template',
     author: 'OpenPAI',
     type: 'template',
     categories: 'AI couplet',
     tags: ['official example'],
     summary: 'Dataset of couplet',
     description:
-      '# Couplet Training Model\n\nThis is a model training process. After training, this model will give a down part with an upper part of couplet. Please refer to [Microsoft AI Edu Project](https://github.com/microsoft/ai-edu/blob/master/B-%E5%AE%9E%E8%B7%B5%E6%A1%88%E4%BE%8B/B13-AI%E5%AF%B9%E8%81%94%E7%94%9F%E6%88%90%E6%A1%88%E4%BE%8B/docs/fairseq.md) for more details.\n\n## Training Data\n\nYou could use ```Couplet Dataset``` data component as training data, or any dataset follows ```fairseq``` model requirements.\n\n## How to use\n\nWhen use this module, you should set three environment variables:\n\n- ```RAW_DATA_DIR```: the training data path in container, if you use ```Couplet Dataset``` data component, this value will be auto filled.\n\n- ```PREPROCESSED_DATA_DIR```: the path to store intermediate result\n\n- ```MODEL_SAVE_DIR```: the path to store output result, i.e. the training model. You could use the predefined output storage, then you could get the results outside container.',
+      '# Couplet Training Model\n\nThis is a model training process. After training, this model will give a down part with an upper part of couplet. Please refer to Microsoft AI Edu Project for more details.\n\n## Training Data\n\nYou could use Couplet Dataset data component as training data, or any dataset follows fairseq model requirements.\n\n## How to use\n\nWhen use this module, you should set three environment variables:\n\n- ```DATA_DIR```: the training data path in container, by default it uses Couplet Dataset data component. If you want to use your own datasets. First make sure mount your data into container, and then change the ```$DATA_DIR``` with the path.\n\n- PREPROCESSED_DATA_DIR: the path to store intermediate result, by default it is ./processed_data.\n\n- ```OUTPUT_DIR```: the path to store output result, i.e. the training model files. By default it will mount a nfs storage, and you could change it with other mounted storage.\n\n## How to check the result\n\nAfter job finished successfully, you could check the output model files in the output storage. The storage server url is in details page.\n',
     content: {
       dockerImage: 'openpai/standard:python_3.6-pytorch_1.2.0-gpu',
       dataStorage: {
@@ -88,41 +88,37 @@ export const MARKET_ITEM_LIST = [
   },
   {
     id: 'a493d4cf-a79e-490f-95c9-06900cdcbd98',
-    name: 'Couplet Inference Model',
+    name: 'Couplet Inference Job Template',
     author: 'OpenPAI',
     type: 'template',
     categories: 'AI couplet',
     tags: ['official example'],
     summary: 'Dataset of couplet',
     description:
-      '# Caffe MNIST Example\nThis example shows how to train LeNet on MNIST with Caffe on OpenPAI.\n\n## Dataset\nThe MNIST dataset is downloaded from MNIST website and converted into caffe format.\n\n## LeNet\nThis example will use the LeNet network, which is known to work well on digit classification tasks.\nIt will use a slightly different version from the original LeNet implementation,\nreplacing the sigmoid activations with Rectified Linear Unit (ReLU) activations for the neurons.\n\nThe design of LeNet contains the essence of CNNs that are still used in larger models such as the ones in ImageNet.\nIn general, it consists of a convolutional layer followed by a pooling layer, another convolution layer followed by a pooling layer,\nand then two fully connected layers similar to the conventional multilayer perceptrons.\nThe layers are defined in `$CAFFE_ROOT/examples/mnist/lenet_train_test.prototxt`.\n\n## Reference\nhttp://caffe.berkeleyvision.org/gathered/examples/mnist.html\n',
+      '# Couplet Training Job Template\n\nThis is a model inference process. The input data is the trainning models trained by ```couplet training job```, and the this job will produce a url for user to ask for down part for a upper part of couplet.\n\n## How to use\n\nWhen use this module, you should set three environment variables:\n\n- ```DATA_DIR```: the training model path in container, by default it uses the output of couplet training job. If you want to use your own models. First make sure mount your models into container, and then change the ```$DATA_DIR``` with the path.\n\n- ```CODE_DIR```: the service code, it will start a server at the given port.\n\n- ```FLASK_RUN_PORT```: the service port container will output.\n\n## How to check the result\n\nAfter job finished successfully, you could check the job detail to get the container ip and ```flask_port``` number, then go to http://<ip>:<flask_port>/upper=<input> to test the result.\n',
     content: {
       dockerImage: 'openpai/standard:python_3.6-pytorch_1.2.0-gpu',
+      dataStorage: {
+        storageType: 'nfs',
+        groups: ['default'],
+        storageName: 'confignfs',
+        serverPath: '10.151.40.235:/data/couplet/checkpoints',
+        containerPath: '/mnt/confignfs/couplet/checkpoints',
+      },
       codeStorage: {
         storageType: 'nfs',
         groups: ['default'],
         storageName: 'confignfs',
-        serverPath: '10.151.40.235:/data/',
-        subPaths: 'couplet_service',
-        containerPath: '/mnt/confignfs/',
-      },
-      outputStorage: {
-        storageType: 'nfs',
-        groups: ['default'],
-        storageName: 'confignfs',
-        serverPath: '10.151.40.235:/data/',
-        subPaths: 'output',
-        containerPath: '/mnt/confignfs/',
-      },
-      environmentVariables: {
-        FLASK_RUN_PORT: { type: 'port', value: null },
+        serverPath: '10.151.40.235:/data/couplet',
+        containerPath: '/mnt/confignfs/couplet',
       },
       commands: [
-        ' pip install fairseq',
-        ' pip install flask',
-        ' cd /mnt/confignfs/couplet_service/',
-        ' export FLASK_APP=app.py',
-        ' python -m flask run',
+        'export FLASK_PORT=$PAI_PORT_LIST_taskrole_0_flask_port',
+        'pip install fairseq',
+        'pip install flask',
+        'pip install gunicorn',
+        'cd ${CODE_DIR}',
+        'gunicorn --bind=0.0.0.0:${FLASK_PORT} app:app',
       ],
     },
     useNumber: 0,