![\"Open](\"https://colab.research.google.com/assets/colab-badge.svg\"){kind=icon}
"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "jG-SjOQTskcX",
"colab_type": "text"
},
"source": [
"## **How to benchmark models with Transformers**\n",
"\n",
"With ever-larger language models, it is no longer enough to just \n",
"compare models on their performance on a specific task. One should always be aware of the computational cost that is attached to a specific model. For a given computation environment (*e.g.* type of GPU), the computational cost of training a model or deploying it in inference usually depends only on **the required memory** and **the required time**. \n",
"\n",
"Being able to accurately benchmark language models on both *speed* and *required memory* is therefore very important.\n",
"\n",
"HuggingFace's Transformer library allows users to benchmark models for both TensorFlow 2 and PyTorch using the `PyTorchBenchmark` and `TensorFlowBenchmark` classes.\n",
"\n",
"The currently available features for `PyTorchBenchmark` are summarized in the following table.\n",
"\n",
"\n",
"| | CPU | CPU + torchscript | GPU | GPU + torchscript | GPU + FP16 | TPU |\n",
":-- | :--- | :--- | :--- | :--- | :--- | :--- |\n",
"**Speed - Inference** | ✔ | ✔ | ✔ | ✔ | ✔ | ✔ |\n",
"**Memory - Inference** | ✔ | ✔ | ✔ | ✔ | ✔ | ✘ |\n",
"**Speed - Train** | ✔ | ✘ | ✔ | ✘ | ✔ | ✔ |\n",
"**Memory - Train** | ✔ | ✘ | ✔ | ✘ | ✔ | ✘ |\n",
"\n",
"\n",
"* *FP16* stands for mixed-precision meaning that computations within the model are done using a mixture of 16-bit and 32-bit floating-point operations, see [here](https://pytorch.org/docs/stable/nn.html#torch.nn.Module.half) for more detail.\n",
"\n",
"* *torchscript* corresponds to PyTorch's torchscript format, see [here](https://pytorch.org/docs/stable/jit.html).\n",
"\n",
"The currently available features for `TensorFlowBenchmark` are summarized in the following table.\n",
"\n",
"| | CPU | CPU + eager execution | GPU | GPU + eager execution | GPU + XLA | GPU + FP16 | TPU |\n",
":-- | :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n",
"**Speed - Inference** | ✔ | ✔ | ✔ | ✔ | ✔ | ✘ | ✔ |\n",
"**Memory - Inference** | ✔ | ✔ | ✔ | ✔ | ✔ | ✘ | ✘ |\n",
"**Speed - Train** | ✔ | ✘ | ✔ | ✘ | ✘ | ✘ | ✔ |\n",
"**Memory - Train** | ✔ | ✘ | ✔ | ✘ | ✘ | ✘ | ✘ |\n",
"\n",
"* *eager execution* means that the function is run in the eager execution environment of TensorFlow 2, see [here](https://www.tensorflow.org/guide/eager).\n",
"\n",
"* *XLA* stands for TensorFlow's Accelerated Linear Algebra (XLA) compiler, see [here](https://www.tensorflow.org/xla)\n",
"\n",
"* *FP16* stands for TensorFlow's mixed-precision package and is analogous to PyTorch's FP16 feature, see [here](https://www.tensorflow.org/guide/mixed_precision).\n",
"\n",
"***Note***: Benchmark training in TensorFlow is not included in v3.0.2, but available in master.\n",
"\n",
"\n",
"This notebook will show the user how to use `PyTorchBenchmark` and `TensorFlowBenchmark` for two different scenarios:\n",
"\n",
"1. **Inference - Pre-trained Model Comparison** - *A user wants to implement a pre-trained model in production for inference. She wants to compare different models on speed and required memory.*\n",
"\n",
"2. **Training - Configuration Comparison** - *A user wants to train a specific model and searches that for himself most effective model configuration.*\n"
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "j-jvAvZ1-GIh",
"colab_type": "text"
},
"source": [
"### **Inference - Pre-trained Model Comparison**\n",
"\n",
"Let's say we want to employ a question-answering model in production. The questions are expected to be of the same format as in **SQuAD v2**, so that the model to choose should have been fine-tuned on this dataset. \n",
"\n",
"HuggingFace's new dataset [webpage](https://huggingface.co/datasets) lets the user see all relevant information about a dataset and even links the models that have been fine-tuned on this specific dataset. Let's check out the dataset webpage of SQuAD v2 [here](https://huggingface.co/datasets/squad_v2).\n",
"\n",
"Nice, we can see that there are 7 available models.\n",
"\n",
"\n",
"\n",
"Let's assume that we have decided to restrict our pipeline to \"encoder-only\" models so that we are left with:\n",
"\n",
"- `a-ware/roberta-large-squad-classification`\n",
"- `a-ware/xlmroberta-squadv2`\n",
"- `aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616_squad2`\n",
"- `deepset/roberta-base-squad2`\n",
"- `mrm8488/longformer-base-4096-finetuned-squadv2`\n",
"\n",
"Great! In this notebook, we will now benchmark these models on both peak memory consumption and inference time to decide which model should be employed in production.\n",
"\n",
"***Note***: None of the models has been tested on performance so that we will just assume that all models perform more or less equally well. The purpose of this notebook is not to find the best model for SQuAD v2, but to showcase how Transformers benchmarking tools can be leveraged.\n",
"\n",
"First, we assume to be limited by the available GPU on this google colab, which in this copy amounts to 16 GB of RAM."
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "2l9C7d7K5-G4",
"colab_type": "text"
},
"source": [
"In a first step, we will check which models are the most memory-efficient ones.\n",
"Let's make sure 100% of the GPU is available to us in this notebook."
]
},
{
"cell_type": "code",
"metadata": {
"id": "M7cQmgM5TvlO",
"colab_type": "code",
"cellView": "form",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 67
},
"outputId": "2797c14e-a62d-42cc-97a6-6c61b015d569"
},
"source": [
"#@title Check available memory of GPU\n",
"# Check that we are using 100% of GPU\n",
"# memory footprint support libraries/code\n",
"!ln -sf /opt/bin/nvidia-smi /usr/bin/nvidia-smi\n",
"!pip -q install gputil\n",
"!pip -q install psutil\n",
"!pip -q install humanize\n",
"import psutil\n",
"import humanize\n",
"import os\n",
"import GPUtil as GPU\n",
"GPUs = GPU.getGPUs()\n",
"# XXX: only one GPU on Colab and isn’t guaranteed\n",
"gpu = GPUs[0]\n",
"def printm():\n",
" process = psutil.Process(os.getpid())\n",
" print(\"Gen RAM Free: \" + humanize.naturalsize( psutil.virtual_memory().available ), \" | Proc size: \" + humanize.naturalsize( process.memory_info().rss))\n",
" print(\"GPU RAM Free: {0:.0f}MB | Used: {1:.0f}MB | Util {2:3.0f}% | Total {3:.0f}MB\".format(gpu.memoryFree, gpu.memoryUsed, gpu.memoryUtil*100, gpu.memoryTotal))\n",
"printm()"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
" Building wheel for gputil (setup.py) ... \u001b[?25l\u001b[?25hdone\n",
"Gen RAM Free: 12.8 GB | Proc size: 160.0 MB\n",
"GPU RAM Free: 16280MB | Used: 0MB | Util 0% | Total 16280MB\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "code",
"metadata": {
"id": "NuS2CKuQ4qSk",
"colab_type": "code",
"colab": {}
},
"source": [
"# If GPU RAM Util > 0% => crash notebook on purpose\n",
"# !kill -9 -1"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ikdYDXsj6Nzv",
"colab_type": "text"
},
"source": [
"Looks good! Now we import `transformers` and download the scripts `run_benchmark.py`, `run_benchmark_tf.py`, and `plot_csv_file.py` which can be found under `transformers/examples/benchmarking`.\n",
"\n",
"`run_benchmark_tf.py` and `run_benchmark.py` are very simple scripts leveraging the `PyTorchBenchmark` and `TensorFlowBenchmark` classes, respectively."
]
},
{
"cell_type": "code",
"metadata": {
"id": "Dylftiyd1IG1",
"colab_type": "code",
"cellView": "both",
"colab": {}
},
"source": [
"# install transformes\n",
"!pip uninstall -y transformers\n",
"!pip install -q git+https://github.com/huggingface/transformers.git\n",
"\n",
"# install py3nvml to track GPU memory usage\n",
"!pip install -q py3nvml\n",
"\n",
"!rm -f run_benchmark.py\n",
"!rm -f run_benchmark_tf.py\n",
"!rm -f plot_csv_file.py\n",
"!wget https://raw.githubusercontent.com/huggingface/transformers/master/examples/benchmarking/run_benchmark.py -qq\n",
"!wget https://raw.githubusercontent.com/huggingface/transformers/master/examples/benchmarking/run_benchmark_tf.py -qq\n",
"!wget https://raw.githubusercontent.com/huggingface/transformers/master/examples/benchmarking/plot_csv_file.py -qq\n",
"\n",
"# import pandas to pretty print csv files\n",
"import pandas as pd"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "C4nz5nGFkOrK",
"colab_type": "text"
},
"source": [
"Information about the input arguments to the *run_benchmark* scripts can be accessed by running `!python run_benchmark.py --help` for PyTorch and `!python run_benchmark_tf.py --help` for TensorFlow."
]
},
{
"cell_type": "code",
"metadata": {
"id": "zu7Oufe0jcAj",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 1000
},
"outputId": "bc52dea5-b721-410c-cf3b-8a7b983a558e"
},
"source": [
"!python run_benchmark.py --help"
],
"execution_count": null,
"outputs": [
{
"output_type": "stream",
"text": [
"2020-06-26 11:51:47.129203: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1\n",
"usage: run_benchmark.py [-h] [--models MODELS [MODELS ...]]\n",
" [--batch_sizes BATCH_SIZES [BATCH_SIZES ...]]\n",
" [--sequence_lengths SEQUENCE_LENGTHS [SEQUENCE_LENGTHS ...]]\n",
" [--no_inference] [--no_cuda] [--no_tpu] [--fp16]\n",
" [--training] [--verbose] [--no_speed] [--no_memory]\n",
" [--trace_memory_line_by_line] [--save_to_csv]\n",
" [--log_print] [--no_env_print] [--no_multi_process]\n",
" [--with_lm_head]\n",
" [--inference_time_csv_file INFERENCE_TIME_CSV_FILE]\n",
" [--inference_memory_csv_file INFERENCE_MEMORY_CSV_FILE]\n",
" [--train_time_csv_file TRAIN_TIME_CSV_FILE]\n",
" [--train_memory_csv_file TRAIN_MEMORY_CSV_FILE]\n",
" [--env_info_csv_file ENV_INFO_CSV_FILE]\n",
" [--log_filename LOG_FILENAME] [--repeat REPEAT]\n",
" [--only_pretrain_model] [--torchscript]\n",
" [--torch_xla_tpu_print_metrics]\n",
" [--fp16_opt_level FP16_OPT_LEVEL]\n",
"\n",
"optional arguments:\n",
" -h, --help show this help message and exit\n",
" --models MODELS [MODELS ...]\n",
" Model checkpoints to be provided to the AutoModel\n",
" classes. Leave blank to benchmark the base version of\n",
" all available models\n",
" --batch_sizes BATCH_SIZES [BATCH_SIZES ...]\n",
" List of batch sizes for which memory and time\n",
" performance will be evaluated\n",
" --sequence_lengths SEQUENCE_LENGTHS [SEQUENCE_LENGTHS ...]\n",
" List of sequence lengths for which memory and time\n",
" performance will be evaluated\n",
" --no_inference Don't benchmark inference of model\n",
" --no_cuda Whether to run on available cuda devices\n",
" --no_tpu Whether to run on available tpu devices\n",
" --fp16 Use FP16 to accelerate inference.\n",
" --training Benchmark training of model\n",
" --verbose Verbose memory tracing\n",
" --no_speed Don't perform speed measurments\n",
" --no_memory Don't perform memory measurments\n",
" --trace_memory_line_by_line\n",
" Trace memory line by line\n",
" --save_to_csv Save result to a CSV file\n",
" --log_print Save all print statements in a log file\n",
" --no_env_print Don't print environment information\n",
" --no_multi_process Don't use multiprocessing for memory and speed\n",
" measurement. It is highly recommended to use\n",
" multiprocessing for accurate CPU and GPU memory\n",
" measurements. This option should only be used for\n",
" debugging / testing and on TPU.\n",
" --with_lm_head Use model with its language model head\n",
" (MODEL_WITH_LM_HEAD_MAPPING instead of MODEL_MAPPING)\n",
" --inference_time_csv_file INFERENCE_TIME_CSV_FILE\n",
" CSV filename used if saving time results to csv.\n",
" --inference_memory_csv_file INFERENCE_MEMORY_CSV_FILE\n",
" CSV filename used if saving memory results to csv.\n",
" --train_time_csv_file TRAIN_TIME_CSV_FILE\n",
" CSV filename used if saving time results to csv for\n",
" training.\n",
" --train_memory_csv_file TRAIN_MEMORY_CSV_FILE\n",
" CSV filename used if saving memory results to csv for\n",
" training.\n",
" --env_info_csv_file ENV_INFO_CSV_FILE\n",
" CSV filename used if saving environment information.\n",
" --log_filename LOG_FILENAME\n",
" Log filename used if print statements are saved in\n",
" log.\n",
" --repeat REPEAT Times an experiment will be run.\n",
" --only_pretrain_model\n",
" Instead of loading the model as defined in\n",
" `config.architectures` if exists, just load the\n",
" pretrain model weights.\n",
" --torchscript Trace the models using torchscript\n",
" --torch_xla_tpu_print_metrics\n",
" Print Xla/PyTorch tpu metrics\n",
" --fp16_opt_level FP16_OPT_LEVEL\n",
" For fp16: Apex AMP optimization level selected in\n",
" ['O0', 'O1', 'O2', and 'O3'].See details at\n",
" https://nvidia.github.io/apex/amp.html\n"
],
"name": "stdout"
}
]
},
{
"cell_type": "markdown",
"metadata": {
"id": "Q_3TZshjcrjP",
"colab_type": "text"
},
"source": [
"Great, we are ready to run our first memory benchmark. By default, both the *required memory* and *time* for inference is enabled. To disable benchmarking on *time*, we add `--no_speed`.\n",
"\n",
"The only required parameter is `--models` which expects a list of model identifiers as defined on the [model hub](https://huggingface.co/models). Here we add the five model identifiers listed above.\n",
"\n",
"Next, we define the `sequence_lengths` and `batch_sizes` for which the peak memory is calculated.\n",
"\n",
"Finally, because the results should be stored in a *CSV* file, the option `--save_to_csv` is added and the path to save the results is added via the `--inference_memory_csv_file` argument. \n",
"Whenever a benchmark is run, the environment information, *e.g.* GPU type, library versions, ... can be saved using the `--env_info_csv_file` argument."
]
},
{
"cell_type": "code",
"metadata": {
"id": "ykJqt7MEbHIq",
"colab_type": "code",
"colab": {}
},
"source": [
"# create plots folder in content\n",
"!mkdir -p plots_pt"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "code",
"metadata": {
"id": "TSJgpQxBe-Fj",
"colab_type": "code",
"colab": {}
},
"source": [
"# run benchmark\n",
"!python run_benchmark.py --no_speed --save_to_csv \\\n",
" --models a-ware/roberta-large-squad-classification \\\n",
" a-ware/xlmroberta-squadv2 \\\n",
" aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200616_squad2 \\\n",
" deepset/roberta-base-squad2 \\\n",
" mrm8488/longformer-base-4096-finetuned-squadv2 \\\n",
" --sequence_lengths 32 128 512 1024 \\\n",
" --batch_sizes 32 \\\n",
" --inference_memory_csv_file plots_pt/required_memory.csv \\\n",
" --env_info_csv_file plots_pt/env.csv >/dev/null 2>&1 # redirect all prints"
],
"execution_count": null,
"outputs": []
},
{
"cell_type": "markdown",
"metadata": {
"id": "ESHrlnKik396",
"colab_type": "text"
},
"source": [
"Under `plots_pt`, two files are now created: `required_memory.csv` and `env.csv`. Let's check out `required_memory.csv` first."
]
},
{
"cell_type": "code",
"metadata": {
"id": "rPg_7fPnuDUa",
"colab_type": "code",
"colab": {
"base_uri": "https://localhost:8080/",
"height": 639
},
"outputId": "b6272763-7235-43c6-c457-0a4a13bb02e5"
},
"source": [
"df = pd.read_csv('plots_pt/required_memory.csv')\n",
"df"
],
"execution_count": null,
"outputs": [
{
"output_type": "execute_result",
"data": {
"text/html": [
"| \n", " | model | \n", "batch_size | \n", "sequence_length | \n", "result | \n", "
|---|---|---|---|---|
| 0 | \n", "a-ware/roberta-large-squad-classification | \n", "32 | \n", "32 | \n", "2219.0 | \n", "
| 1 | \n", "a-ware/roberta-large-squad-classification | \n", "32 | \n", "128 | \n", "2455.0 | \n", "
| 2 | \n", "a-ware/roberta-large-squad-classification | \n", "32 | \n", "512 | \n", "3641.0 | \n", "
| 3 | \n", "a-ware/roberta-large-squad-classification | \n", "32 | \n", "1024 | \n", "NaN | \n", "
| 4 | \n", "a-ware/xlmroberta-squadv2 | \n", "32 | \n", "32 | \n", "2999.0 | \n", "
| 5 | \n", "a-ware/xlmroberta-squadv2 | \n", "32 | \n", "128 | \n", "3235.0 | \n", "
| 6 | \n", "a-ware/xlmroberta-squadv2 | \n", "32 | \n", "512 | \n", "4421.0 | \n", "
| 7 | \n", "a-ware/xlmroberta-squadv2 | \n", "32 | \n", "1024 | \n", "NaN | \n", "
| 8 | \n", "aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200... | \n", "32 | \n", "32 | \n", "1025.0 | \n", "
| 9 | \n", "aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200... | \n", "32 | \n", "128 | \n", "1143.0 | \n", "
| 10 | \n", "aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200... | \n", "32 | \n", "512 | \n", "1719.0 | \n", "
| 11 | \n", "aodiniz/bert_uncased_L-10_H-512_A-8_cord19-200... | \n", "32 | \n", "1024 | \n", "NaN | \n", "
| 12 | \n", "deepset/roberta-base-squad2 | \n", "32 | \n", "32 | \n", "1373.0 | \n", "
| 13 | \n", "deepset/roberta-base-squad2 | \n", "32 | \n", "128 | \n", "1533.0 | \n", "
| 14 | \n", "deepset/roberta-base-squad2 | \n", "32 | \n", "512 | \n", "2433.0 | \n", "
| 15 | \n", "deepset/roberta-base-squad2 | \n", "32 | \n", "1024 | \n", "NaN | \n", "
| 16 | \n", "mrm8488/longformer-base-4096-finetuned-squadv2 | \n", "32 | \n", "32 | \n", "3783.0 | \n", "
| 17 | \n", "mrm8488/longformer-base-4096-finetuned-squadv2 | \n", "32 | \n", "128 | \n", "3783.0 | \n", "
| 18 | \n", "mrm8488/longformer-base-4096-finetuned-squadv2 | \n", "32 | \n", "512 | \n", "3783.0 | \n", "
| 19 | \n", "mrm8488/longformer-base-4096-finetuned-squadv2 | \n", "32 | \n", "1024 | \n", "6427.0 | \n", "
| \n", " | transformers_version | \n", "2.11.0 | \n", "
|---|---|---|
| 0 | \n", "framework | \n", "PyTorch | \n", "
| 1 | \n", "use_torchscript | \n", "False | \n", "
| 2 | \n", "framework_version | \n", "1.5.1+cu101 | \n", "
| 3 | \n", "python_version | \n", "3.6.9 | \n", "
| 4 | \n", "system | \n", "Linux | \n", "
| 5 | \n", "cpu | \n", "x86_64 | \n", "
| 6 | \n", "architecture | \n", "64bit | \n", "
| 7 | \n", "date | \n", "2020-06-26 | \n", "
| 8 | \n", "time | \n", "11:56:37.277009 | \n", "
| 9 | \n", "fp16 | \n", "False | \n", "
| 10 | \n", "use_multiprocessing | \n", "True | \n", "
| 11 | \n", "only_pretrain_model | \n", "False | \n", "
| 12 | \n", "cpu_ram_mb | \n", "13021 | \n", "
| 13 | \n", "use_gpu | \n", "True | \n", "
| 14 | \n", "num_gpus | \n", "1 | \n", "
| 15 | \n", "gpu | \n", "Tesla P100-PCIE-16GB | \n", "
| 16 | \n", "gpu_ram_mb | \n", "16280 | \n", "
| 17 | \n", "gpu_power_watts | \n", "250.0 | \n", "
| 18 | \n", "gpu_performance_state | \n", "0 | \n", "
| 19 | \n", "use_tpu | \n", "False | \n", "