Doc styling (#8067)
* Important files * Styling them all * Revert "Styling them all" This reverts commit 7d029395fdae8513b8281cbc2a6c239f8093503e. * Syling them for realsies * Fix syntax error * Fix benchmark_utils * More fixes * Fix modeling auto and script * Remove new line * Fixes * More fixes * Fix more files * Style * Add FSMT * More fixes * More fixes * More fixes * More fixes * Fixes * More fixes * More fixes * Last fixes * Make sphinx happy
This commit is contained in:
Sylvain Gugger
GitHub
Родитель
04a17f8550
Коммит
08f534d2da
|
|
@ -322,6 +322,7 @@ jobs:
|
|||
- run: black --check examples templates tests src utils
|
||||
- run: isort --check-only examples templates tests src utils
|
||||
- run: flake8 examples templates tests src utils
|
||||
- run: python utils/style_doc.py src/transformers docs/source --max_len 119 --check_only
|
||||
- run: python utils/check_copies.py
|
||||
- run: python utils/check_dummies.py
|
||||
- run: python utils/check_repo.py
|
||||
|
|
|
|||
3
Makefile
3
Makefile
|
|
@ -15,6 +15,7 @@ modified_only_fixup:
|
|||
black $(modified_py_files); \
|
||||
isort $(modified_py_files); \
|
||||
flake8 $(modified_py_files); \
|
||||
python utils/style_doc.py $(modified_py_files) --max_len 119; \
|
||||
else \
|
||||
echo "No library .py files were modified"; \
|
||||
fi
|
||||
|
|
@ -31,6 +32,7 @@ quality:
|
|||
black --check $(check_dirs)
|
||||
isort --check-only $(check_dirs)
|
||||
flake8 $(check_dirs)
|
||||
python utils/style_doc.py src/transformers docs/source --max_len 119 --check_only
|
||||
${MAKE} extra_quality_checks
|
||||
|
||||
# Format source code automatically and check is there are any problems left that need manual fixing
|
||||
|
|
@ -38,6 +40,7 @@ quality:
|
|||
style:
|
||||
black $(check_dirs)
|
||||
isort $(check_dirs)
|
||||
python utils/style_doc.py src/transformers docs/source --max_len 119
|
||||
|
||||
# Super fast fix and check target that only works on relevant modified files since the branch was made
|
||||
|
||||
|
|
|
|||
|
|
@ -180,7 +180,7 @@ Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih.
|
|||
1. **[Longformer](https://huggingface.co/transformers/model_doc/longformer.html)** (from AllenAI) released with the paper [Longformer: The Long-Document Transformer](https://arxiv.org/abs/2004.05150) by Iz Beltagy, Matthew E. Peters, Arman Cohan.
|
||||
1. **[LXMERT](https://huggingface.co/transformers/model_doc/lxmert.html)** (from UNC Chapel Hill) released with the paper [LXMERT: Learning Cross-Modality Encoder Representations from Transformers for Open-Domain Question Answering](https://arxiv.org/abs/1908.07490) by Hao Tan and Mohit Bansal.
|
||||
1. **[MarianMT](https://huggingface.co/transformers/model_doc/marian.html)** Machine translation models trained using [OPUS](http://opus.nlpl.eu/) data by Jörg Tiedemann. The [Marian Framework](https://marian-nmt.github.io/) is being developed by the Microsoft Translator Team.
|
||||
1. **[MBart](https://huggingface.co/transformers/model_doc/mbart.html)** (from Facebook) released with the paper [Multilingual Denoising Pre-training for Neural Machine Translation](https://arxiv.org/abs/2001.08210) by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.
|
||||
1. **[MBart](https://huggingface.co/transformers/model_doc/mbart.html)** (from Facebook) released with the paper [Multilingual Denoising Pre-training for Neural Machine Translation](https://arxiv.org/abs/2001.08210) by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.
|
||||
1. **[Pegasus](https://huggingface.co/transformers/model_doc/pegasus.html)** (from Google) released with the paper [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://arxiv.org/abs/1912.08777)> by Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu.
|
||||
1. **[ProphetNet](https://huggingface.co/transformers/model_doc/prophetnet.html)** (from Microsoft Research) released with the paper [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://arxiv.org/abs/2001.04063) by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang and Ming Zhou.
|
||||
1. **[Reformer](https://huggingface.co/transformers/model_doc/reformer.html)** (from Google Research) released with the paper [Reformer: The Efficient Transformer](https://arxiv.org/abs/2001.04451) by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya.
|
||||
|
|
|
|||
|
|
@ -3,21 +3,27 @@ Benchmarks
|
|||
|
||||
Let's take a look at how 🤗 Transformer models can be benchmarked, best practices, and already available benchmarks.
|
||||
|
||||
A notebook explaining in more detail how to benchmark 🤗 Transformer models can be found `here <https://github.com/huggingface/transformers/blob/master/notebooks/05-benchmark.ipynb>`__.
|
||||
A notebook explaining in more detail how to benchmark 🤗 Transformer models can be found `here
|
||||
<https://github.com/huggingface/transformers/blob/master/notebooks/05-benchmark.ipynb>`__.
|
||||
|
||||
How to benchmark 🤗 Transformer models
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The classes :class:`~transformers.PyTorchBenchmark` and :class:`~transformers.TensorFlowBenchmark` allow to flexibly benchmark 🤗 Transformer models.
|
||||
The benchmark classes allow us to measure the `peak memory usage` and `required time` for both
|
||||
`inference` and `training`.
|
||||
The classes :class:`~transformers.PyTorchBenchmark` and :class:`~transformers.TensorFlowBenchmark` allow to flexibly
|
||||
benchmark 🤗 Transformer models. The benchmark classes allow us to measure the `peak memory usage` and `required time`
|
||||
for both `inference` and `training`.
|
||||
|
||||
.. note::
|
||||
|
||||
Hereby, `inference` is defined by a single forward pass, and `training` is defined by a single forward pass and backward pass.
|
||||
Hereby, `inference` is defined by a single forward pass, and `training` is defined by a single forward pass and
|
||||
backward pass.
|
||||
|
||||
The benchmark classes :class:`~transformers.PyTorchBenchmark` and :class:`~transformers.TensorFlowBenchmark` expect an object of type :class:`~transformers.PyTorchBenchmarkArguments` and :class:`~transformers.TensorFlowBenchmarkArguments`, respectively, for instantiation. :class:`~transformers.PyTorchBenchmarkArguments` and :class:`~transformers.TensorFlowBenchmarkArguments` are data classes and contain all relevant configurations for their corresponding benchmark class.
|
||||
In the following example, it is shown how a BERT model of type `bert-base-cased` can be benchmarked.
|
||||
The benchmark classes :class:`~transformers.PyTorchBenchmark` and :class:`~transformers.TensorFlowBenchmark` expect an
|
||||
object of type :class:`~transformers.PyTorchBenchmarkArguments` and
|
||||
:class:`~transformers.TensorFlowBenchmarkArguments`, respectively, for instantiation.
|
||||
:class:`~transformers.PyTorchBenchmarkArguments` and :class:`~transformers.TensorFlowBenchmarkArguments` are data
|
||||
classes and contain all relevant configurations for their corresponding benchmark class. In the following example, it
|
||||
is shown how a BERT model of type `bert-base-cased` can be benchmarked.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -34,11 +40,15 @@ In the following example, it is shown how a BERT model of type `bert-base-cased`
|
|||
>>> benchmark = TensorFlowBenchmark(args)
|
||||
|
||||
|
||||
Here, three arguments are given to the benchmark argument data classes, namely ``models``, ``batch_sizes``, and ``sequence_lengths``. The argument ``models`` is required and expects a :obj:`list` of model identifiers from the `model hub <https://huggingface.co/models>`__
|
||||
The :obj:`list` arguments ``batch_sizes`` and ``sequence_lengths`` define the size of the ``input_ids`` on which the model is benchmarked.
|
||||
There are many more parameters that can be configured via the benchmark argument data classes. For more detail on these one can either directly consult the files
|
||||
``src/transformers/benchmark/benchmark_args_utils.py``, ``src/transformers/benchmark/benchmark_args.py`` (for PyTorch) and ``src/transformers/benchmark/benchmark_args_tf.py`` (for Tensorflow).
|
||||
Alternatively, running the following shell commands from root will print out a descriptive list of all configurable parameters for PyTorch and Tensorflow respectively.
|
||||
Here, three arguments are given to the benchmark argument data classes, namely ``models``, ``batch_sizes``, and
|
||||
``sequence_lengths``. The argument ``models`` is required and expects a :obj:`list` of model identifiers from the
|
||||
`model hub <https://huggingface.co/models>`__ The :obj:`list` arguments ``batch_sizes`` and ``sequence_lengths`` define
|
||||
the size of the ``input_ids`` on which the model is benchmarked. There are many more parameters that can be configured
|
||||
via the benchmark argument data classes. For more detail on these one can either directly consult the files
|
||||
``src/transformers/benchmark/benchmark_args_utils.py``, ``src/transformers/benchmark/benchmark_args.py`` (for PyTorch)
|
||||
and ``src/transformers/benchmark/benchmark_args_tf.py`` (for Tensorflow). Alternatively, running the following shell
|
||||
commands from root will print out a descriptive list of all configurable parameters for PyTorch and Tensorflow
|
||||
respectively.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -65,7 +75,7 @@ An instantiated benchmark object can then simply be run by calling ``benchmark.r
|
|||
bert-base-uncased 8 128 0.018
|
||||
bert-base-uncased 8 512 0.088
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
==================== INFERENCE - MEMORY - RESULT ====================
|
||||
--------------------------------------------------------------------------------
|
||||
Model Name Batch Size Seq Length Memory in MB
|
||||
|
|
@ -75,7 +85,7 @@ An instantiated benchmark object can then simply be run by calling ``benchmark.r
|
|||
bert-base-uncased 8 128 1307
|
||||
bert-base-uncased 8 512 1539
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
==================== ENVIRONMENT INFORMATION ====================
|
||||
- transformers_version: 2.11.0
|
||||
- framework: PyTorch
|
||||
|
|
@ -98,7 +108,7 @@ An instantiated benchmark object can then simply be run by calling ``benchmark.r
|
|||
- gpu_power_watts: 280.0
|
||||
- gpu_performance_state: 2
|
||||
- use_tpu: False
|
||||
|
||||
|
||||
>>> ## TENSORFLOW CODE
|
||||
>>> results = benchmark.run()
|
||||
>>> print(results)
|
||||
|
|
@ -111,7 +121,7 @@ An instantiated benchmark object can then simply be run by calling ``benchmark.r
|
|||
bert-base-uncased 8 128 0.022
|
||||
bert-base-uncased 8 512 0.105
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
==================== INFERENCE - MEMORY - RESULT ====================
|
||||
--------------------------------------------------------------------------------
|
||||
Model Name Batch Size Seq Length Memory in MB
|
||||
|
|
@ -121,7 +131,7 @@ An instantiated benchmark object can then simply be run by calling ``benchmark.r
|
|||
bert-base-uncased 8 128 1330
|
||||
bert-base-uncased 8 512 1770
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
==================== ENVIRONMENT INFORMATION ====================
|
||||
- transformers_version: 2.11.0
|
||||
- framework: Tensorflow
|
||||
|
|
@ -145,14 +155,17 @@ An instantiated benchmark object can then simply be run by calling ``benchmark.r
|
|||
- gpu_performance_state: 2
|
||||
- use_tpu: False
|
||||
|
||||
By default, the `time` and the `required memory` for `inference` are benchmarked.
|
||||
In the example output above the first two sections show the result corresponding to `inference time` and `inference memory`.
|
||||
In addition, all relevant information about the computing environment, `e.g.` the GPU type, the system, the library versions, etc... are printed out in the third section under `ENVIRONMENT INFORMATION`.
|
||||
This information can optionally be saved in a `.csv` file when adding the argument :obj:`save_to_csv=True` to :class:`~transformers.PyTorchBenchmarkArguments` and :class:`~transformers.TensorFlowBenchmarkArguments` respectively.
|
||||
In this case, every section is saved in a separate `.csv` file. The path to each `.csv` file can optionally be defined via the argument data classes.
|
||||
By default, the `time` and the `required memory` for `inference` are benchmarked. In the example output above the first
|
||||
two sections show the result corresponding to `inference time` and `inference memory`. In addition, all relevant
|
||||
information about the computing environment, `e.g.` the GPU type, the system, the library versions, etc... are printed
|
||||
out in the third section under `ENVIRONMENT INFORMATION`. This information can optionally be saved in a `.csv` file
|
||||
when adding the argument :obj:`save_to_csv=True` to :class:`~transformers.PyTorchBenchmarkArguments` and
|
||||
:class:`~transformers.TensorFlowBenchmarkArguments` respectively. In this case, every section is saved in a separate
|
||||
`.csv` file. The path to each `.csv` file can optionally be defined via the argument data classes.
|
||||
|
||||
Instead of benchmarking pre-trained models via their model identifier, `e.g.` `bert-base-uncased`, the user can alternatively benchmark an arbitrary configuration of any available model class.
|
||||
In this case, a :obj:`list` of configurations must be inserted with the benchmark args as follows.
|
||||
Instead of benchmarking pre-trained models via their model identifier, `e.g.` `bert-base-uncased`, the user can
|
||||
alternatively benchmark an arbitrary configuration of any available model class. In this case, a :obj:`list` of
|
||||
configurations must be inserted with the benchmark args as follows.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -183,7 +196,7 @@ In this case, a :obj:`list` of configurations must be inserted with the benchmar
|
|||
bert-6-lay 8 128 0.009
|
||||
bert-6-lay 8 512 0.044
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
==================== INFERENCE - MEMORY - RESULT ====================
|
||||
--------------------------------------------------------------------------------
|
||||
Model Name Batch Size Seq Length Memory in MB
|
||||
|
|
@ -201,7 +214,7 @@ In this case, a :obj:`list` of configurations must be inserted with the benchmar
|
|||
bert-6-lay 8 128 1127
|
||||
bert-6-lay 8 512 1359
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
==================== ENVIRONMENT INFORMATION ====================
|
||||
- transformers_version: 2.11.0
|
||||
- framework: PyTorch
|
||||
|
|
@ -252,7 +265,7 @@ In this case, a :obj:`list` of configurations must be inserted with the benchmar
|
|||
bert-6-lay 8 128 0.0011
|
||||
bert-6-lay 8 512 0.074
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
==================== INFERENCE - MEMORY - RESULT ====================
|
||||
--------------------------------------------------------------------------------
|
||||
Model Name Batch Size Seq Length Memory in MB
|
||||
|
|
@ -270,7 +283,7 @@ In this case, a :obj:`list` of configurations must be inserted with the benchmar
|
|||
bert-6-lay 8 128 1330
|
||||
bert-6-lay 8 512 1540
|
||||
--------------------------------------------------------------------------------
|
||||
|
||||
|
||||
==================== ENVIRONMENT INFORMATION ====================
|
||||
- transformers_version: 2.11.0
|
||||
- framework: Tensorflow
|
||||
|
|
@ -295,8 +308,9 @@ In this case, a :obj:`list` of configurations must be inserted with the benchmar
|
|||
- use_tpu: False
|
||||
|
||||
|
||||
Again, `inference time` and `required memory` for `inference` are measured, but this time for customized configurations of the :obj:`BertModel` class. This feature can especially be helpful when
|
||||
deciding for which configuration the model should be trained.
|
||||
Again, `inference time` and `required memory` for `inference` are measured, but this time for customized configurations
|
||||
of the :obj:`BertModel` class. This feature can especially be helpful when deciding for which configuration the model
|
||||
should be trained.
|
||||
|
||||
|
||||
Benchmark best practices
|
||||
|
|
@ -304,19 +318,28 @@ Benchmark best practices
|
|||
|
||||
This section lists a couple of best practices one should be aware of when benchmarking a model.
|
||||
|
||||
- Currently, only single device benchmarking is supported. When benchmarking on GPU, it is recommended that the user
|
||||
specifies on which device the code should be run by setting the ``CUDA_VISIBLE_DEVICES`` environment variable in the shell, `e.g.` ``export CUDA_VISIBLE_DEVICES=0`` before running the code.
|
||||
- The option :obj:`no_multi_processing` should only be set to :obj:`True` for testing and debugging. To ensure accurate memory measurement it is recommended to run each memory benchmark in a separate process by making sure :obj:`no_multi_processing` is set to :obj:`True`.
|
||||
- One should always state the environment information when sharing the results of a model benchmark. Results can vary heavily between different GPU devices, library versions, etc., so that benchmark results on their own are not very useful for the community.
|
||||
- Currently, only single device benchmarking is supported. When benchmarking on GPU, it is recommended that the user
|
||||
specifies on which device the code should be run by setting the ``CUDA_VISIBLE_DEVICES`` environment variable in the
|
||||
shell, `e.g.` ``export CUDA_VISIBLE_DEVICES=0`` before running the code.
|
||||
- The option :obj:`no_multi_processing` should only be set to :obj:`True` for testing and debugging. To ensure accurate
|
||||
memory measurement it is recommended to run each memory benchmark in a separate process by making sure
|
||||
:obj:`no_multi_processing` is set to :obj:`True`.
|
||||
- One should always state the environment information when sharing the results of a model benchmark. Results can vary
|
||||
heavily between different GPU devices, library versions, etc., so that benchmark results on their own are not very
|
||||
useful for the community.
|
||||
|
||||
|
||||
Sharing your benchmark
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Previously all available core models (10 at the time) have been benchmarked for `inference time`, across many different settings: using PyTorch, with
|
||||
and without TorchScript, using TensorFlow, with and without XLA. All of those tests were done across CPUs (except for
|
||||
TensorFlow XLA) and GPUs.
|
||||
Previously all available core models (10 at the time) have been benchmarked for `inference time`, across many different
|
||||
settings: using PyTorch, with and without TorchScript, using TensorFlow, with and without XLA. All of those tests were
|
||||
done across CPUs (except for TensorFlow XLA) and GPUs.
|
||||
|
||||
The approach is detailed in the `following blogpost <https://medium.com/huggingface/benchmarking-transformers-pytorch-and-tensorflow-e2917fb891c2>`__ and the results are available `here <https://docs.google.com/spreadsheets/d/1sryqufw2D0XlUH4sq3e9Wnxu5EAQkaohzrJbd5HdQ_w/edit?usp=sharing>`__.
|
||||
The approach is detailed in the `following blogpost
|
||||
<https://medium.com/huggingface/benchmarking-transformers-pytorch-and-tensorflow-e2917fb891c2>`__ and the results are
|
||||
available `here
|
||||
<https://docs.google.com/spreadsheets/d/1sryqufw2D0XlUH4sq3e9Wnxu5EAQkaohzrJbd5HdQ_w/edit?usp=sharing>`__.
|
||||
|
||||
With the new `benchmark` tools, it is easier than ever to share your benchmark results with the community `here <https://github.com/huggingface/transformers/blob/master/examples/benchmarking/README.md>`__.
|
||||
With the new `benchmark` tools, it is easier than ever to share your benchmark results with the community `here
|
||||
<https://github.com/huggingface/transformers/blob/master/examples/benchmarking/README.md>`__.
|
||||
|
|
|
|||
|
|
@ -1,18 +1,26 @@
|
|||
BERTology
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
There is a growing field of study concerned with investigating the inner working of large-scale transformers like BERT (that some call "BERTology"). Some good examples of this field are:
|
||||
There is a growing field of study concerned with investigating the inner working of large-scale transformers like BERT
|
||||
(that some call "BERTology"). Some good examples of this field are:
|
||||
|
||||
|
||||
* BERT Rediscovers the Classical NLP Pipeline by Ian Tenney, Dipanjan Das, Ellie Pavlick: https://arxiv.org/abs/1905.05950
|
||||
* BERT Rediscovers the Classical NLP Pipeline by Ian Tenney, Dipanjan Das, Ellie Pavlick:
|
||||
https://arxiv.org/abs/1905.05950
|
||||
* Are Sixteen Heads Really Better than One? by Paul Michel, Omer Levy, Graham Neubig: https://arxiv.org/abs/1905.10650
|
||||
* What Does BERT Look At? An Analysis of BERT's Attention by Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D. Manning: https://arxiv.org/abs/1906.04341
|
||||
* What Does BERT Look At? An Analysis of BERT's Attention by Kevin Clark, Urvashi Khandelwal, Omer Levy, Christopher D.
|
||||
Manning: https://arxiv.org/abs/1906.04341
|
||||
|
||||
In order to help this new field develop, we have included a few additional features in the BERT/GPT/GPT-2 models to help people access the inner representations, mainly adapted from the great work of Paul Michel (https://arxiv.org/abs/1905.10650):
|
||||
In order to help this new field develop, we have included a few additional features in the BERT/GPT/GPT-2 models to
|
||||
help people access the inner representations, mainly adapted from the great work of Paul Michel
|
||||
(https://arxiv.org/abs/1905.10650):
|
||||
|
||||
|
||||
* accessing all the hidden-states of BERT/GPT/GPT-2,
|
||||
* accessing all the attention weights for each head of BERT/GPT/GPT-2,
|
||||
* retrieving heads output values and gradients to be able to compute head importance score and prune head as explained in https://arxiv.org/abs/1905.10650.
|
||||
* retrieving heads output values and gradients to be able to compute head importance score and prune head as explained
|
||||
in https://arxiv.org/abs/1905.10650.
|
||||
|
||||
To help you understand and use these features, we have added a specific example script: `bertology.py <https://github.com/huggingface/transformers/blob/master/examples/bertology/run_bertology.py>`_ while extract information and prune a model pre-trained on GLUE.
|
||||
To help you understand and use these features, we have added a specific example script: `bertology.py
|
||||
<https://github.com/huggingface/transformers/blob/master/examples/bertology/run_bertology.py>`_ while extract
|
||||
information and prune a model pre-trained on GLUE.
|
||||
|
|
|
|||
|
|
@ -1,24 +1,40 @@
|
|||
Converting Tensorflow Checkpoints
|
||||
=======================================================================================================================
|
||||
|
||||
A command-line interface is provided to convert original Bert/GPT/GPT-2/Transformer-XL/XLNet/XLM checkpoints in models than be loaded using the ``from_pretrained`` methods of the library.
|
||||
A command-line interface is provided to convert original Bert/GPT/GPT-2/Transformer-XL/XLNet/XLM checkpoints in models
|
||||
than be loaded using the ``from_pretrained`` methods of the library.
|
||||
|
||||
.. note::
|
||||
Since 2.3.0 the conversion script is now part of the transformers CLI (**transformers-cli**)
|
||||
available in any transformers >= 2.3.0 installation.
|
||||
Since 2.3.0 the conversion script is now part of the transformers CLI (**transformers-cli**) available in any
|
||||
transformers >= 2.3.0 installation.
|
||||
|
||||
The documentation below reflects the **transformers-cli convert** command format.
|
||||
|
||||
BERT
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
You can convert any TensorFlow checkpoint for BERT (in particular `the pre-trained models released by Google <https://github.com/google-research/bert#pre-trained-models>`_\ ) in a PyTorch save file by using the `convert_bert_original_tf_checkpoint_to_pytorch.py <https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py>`_ script.
|
||||
You can convert any TensorFlow checkpoint for BERT (in particular `the pre-trained models released by Google
|
||||
<https://github.com/google-research/bert#pre-trained-models>`_\ ) in a PyTorch save file by using the
|
||||
`convert_bert_original_tf_checkpoint_to_pytorch.py
|
||||
<https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py>`_
|
||||
script.
|
||||
|
||||
This CLI takes as input a TensorFlow checkpoint (three files starting with ``bert_model.ckpt``\ ) and the associated configuration file (\ ``bert_config.json``\ ), and creates a PyTorch model for this configuration, loads the weights from the TensorFlow checkpoint in the PyTorch model and saves the resulting model in a standard PyTorch save file that can be imported using ``torch.load()`` (see examples in `run_bert_extract_features.py <https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/run_bert_extract_features.py>`_\ , `run_bert_classifier.py <https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/run_bert_classifier.py>`_ and `run_bert_squad.py <https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/run_bert_squad.py>`_\ ).
|
||||
This CLI takes as input a TensorFlow checkpoint (three files starting with ``bert_model.ckpt``\ ) and the associated
|
||||
configuration file (\ ``bert_config.json``\ ), and creates a PyTorch model for this configuration, loads the weights
|
||||
from the TensorFlow checkpoint in the PyTorch model and saves the resulting model in a standard PyTorch save file that
|
||||
can be imported using ``torch.load()`` (see examples in `run_bert_extract_features.py
|
||||
<https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/run_bert_extract_features.py>`_\ ,
|
||||
`run_bert_classifier.py
|
||||
<https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/run_bert_classifier.py>`_ and
|
||||
`run_bert_squad.py <https://github.com/huggingface/pytorch-pretrained-BERT/tree/master/examples/run_bert_squad.py>`_\
|
||||
).
|
||||
|
||||
You only need to run this conversion script **once** to get a PyTorch model. You can then disregard the TensorFlow checkpoint (the three files starting with ``bert_model.ckpt``\ ) but be sure to keep the configuration file (\ ``bert_config.json``\ ) and the vocabulary file (\ ``vocab.txt``\ ) as these are needed for the PyTorch model too.
|
||||
You only need to run this conversion script **once** to get a PyTorch model. You can then disregard the TensorFlow
|
||||
checkpoint (the three files starting with ``bert_model.ckpt``\ ) but be sure to keep the configuration file (\
|
||||
``bert_config.json``\ ) and the vocabulary file (\ ``vocab.txt``\ ) as these are needed for the PyTorch model too.
|
||||
|
||||
To run this specific conversion script you will need to have TensorFlow and PyTorch installed (\ ``pip install tensorflow``\ ). The rest of the repository only requires PyTorch.
|
||||
To run this specific conversion script you will need to have TensorFlow and PyTorch installed (\ ``pip install
|
||||
tensorflow``\ ). The rest of the repository only requires PyTorch.
|
||||
|
||||
Here is an example of the conversion process for a pre-trained ``BERT-Base Uncased`` model:
|
||||
|
||||
|
|
@ -31,14 +47,20 @@ Here is an example of the conversion process for a pre-trained ``BERT-Base Uncas
|
|||
--config $BERT_BASE_DIR/bert_config.json \
|
||||
--pytorch_dump_output $BERT_BASE_DIR/pytorch_model.bin
|
||||
|
||||
You can download Google's pre-trained models for the conversion `here <https://github.com/google-research/bert#pre-trained-models>`__.
|
||||
You can download Google's pre-trained models for the conversion `here
|
||||
<https://github.com/google-research/bert#pre-trained-models>`__.
|
||||
|
||||
ALBERT
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Convert TensorFlow model checkpoints of ALBERT to PyTorch using the `convert_albert_original_tf_checkpoint_to_pytorch.py <https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py>`_ script.
|
||||
Convert TensorFlow model checkpoints of ALBERT to PyTorch using the
|
||||
`convert_albert_original_tf_checkpoint_to_pytorch.py
|
||||
<https://github.com/huggingface/transformers/blob/master/src/transformers/convert_bert_original_tf_checkpoint_to_pytorch.py>`_
|
||||
script.
|
||||
|
||||
The CLI takes as input a TensorFlow checkpoint (three files starting with ``model.ckpt-best``\ ) and the accompanying configuration file (\ ``albert_config.json``\ ), then creates and saves a PyTorch model. To run this conversion you will need to have TensorFlow and PyTorch installed.
|
||||
The CLI takes as input a TensorFlow checkpoint (three files starting with ``model.ckpt-best``\ ) and the accompanying
|
||||
configuration file (\ ``albert_config.json``\ ), then creates and saves a PyTorch model. To run this conversion you
|
||||
will need to have TensorFlow and PyTorch installed.
|
||||
|
||||
Here is an example of the conversion process for the pre-trained ``ALBERT Base`` model:
|
||||
|
||||
|
|
@ -51,12 +73,15 @@ Here is an example of the conversion process for the pre-trained ``ALBERT Base``
|
|||
--config $ALBERT_BASE_DIR/albert_config.json \
|
||||
--pytorch_dump_output $ALBERT_BASE_DIR/pytorch_model.bin
|
||||
|
||||
You can download Google's pre-trained models for the conversion `here <https://github.com/google-research/albert#pre-trained-models>`__.
|
||||
You can download Google's pre-trained models for the conversion `here
|
||||
<https://github.com/google-research/albert#pre-trained-models>`__.
|
||||
|
||||
OpenAI GPT
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Here is an example of the conversion process for a pre-trained OpenAI GPT model, assuming that your NumPy checkpoint save as the same format than OpenAI pretrained model (see `here <https://github.com/openai/finetune-transformer-lm>`__\ )
|
||||
Here is an example of the conversion process for a pre-trained OpenAI GPT model, assuming that your NumPy checkpoint
|
||||
save as the same format than OpenAI pretrained model (see `here <https://github.com/openai/finetune-transformer-lm>`__\
|
||||
)
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
|
|
@ -72,7 +97,8 @@ Here is an example of the conversion process for a pre-trained OpenAI GPT model,
|
|||
OpenAI GPT-2
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Here is an example of the conversion process for a pre-trained OpenAI GPT-2 model (see `here <https://github.com/openai/gpt-2>`__\ )
|
||||
Here is an example of the conversion process for a pre-trained OpenAI GPT-2 model (see `here
|
||||
<https://github.com/openai/gpt-2>`__\ )
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
|
|
@ -87,7 +113,8 @@ Here is an example of the conversion process for a pre-trained OpenAI GPT-2 mode
|
|||
Transformer-XL
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Here is an example of the conversion process for a pre-trained Transformer-XL model (see `here <https://github.com/kimiyoung/transformer-xl/tree/master/tf#obtain-and-evaluate-pretrained-sota-models>`__\ )
|
||||
Here is an example of the conversion process for a pre-trained Transformer-XL model (see `here
|
||||
<https://github.com/kimiyoung/transformer-xl/tree/master/tf#obtain-and-evaluate-pretrained-sota-models>`__\ )
|
||||
|
||||
.. code-block:: shell
|
||||
|
||||
|
|
@ -130,4 +157,4 @@ Here is an example of the conversion process for a pre-trained XLM model:
|
|||
--tf_checkpoint $XLM_CHECKPOINT_PATH \
|
||||
--pytorch_dump_output $PYTORCH_DUMP_OUTPUT
|
||||
[--config XML_CONFIG] \
|
||||
[--finetuning_task_name XML_FINETUNED_TASK]
|
||||
[--finetuning_task_name XML_FINETUNED_TASK]
|
||||
|
|
|
|||
|
|
@ -3,15 +3,15 @@ Fine-tuning with custom datasets
|
|||
|
||||
.. note::
|
||||
|
||||
The datasets used in this tutorial are available and can be more easily accessed using the
|
||||
`🤗 NLP library <https://github.com/huggingface/nlp>`_. We do not use this library to access the datasets here
|
||||
since this tutorial meant to illustrate how to work with your own data. A brief of introduction can be found
|
||||
at the end of the tutorial in the section ":ref:`nlplib`".
|
||||
The datasets used in this tutorial are available and can be more easily accessed using the `🤗 NLP library
|
||||
<https://github.com/huggingface/nlp>`_. We do not use this library to access the datasets here since this tutorial
|
||||
meant to illustrate how to work with your own data. A brief of introduction can be found at the end of the tutorial
|
||||
in the section ":ref:`nlplib`".
|
||||
|
||||
This tutorial will take you through several examples of using 🤗 Transformers models with your own datasets. The
|
||||
guide shows one of many valid workflows for using these models and is meant to be illustrative rather than
|
||||
definitive. We show examples of reading in several data formats, preprocessing the data for several types of tasks,
|
||||
and then preparing the data into PyTorch/TensorFlow ``Dataset`` objects which can easily be used either with
|
||||
This tutorial will take you through several examples of using 🤗 Transformers models with your own datasets. The guide
|
||||
shows one of many valid workflows for using these models and is meant to be illustrative rather than definitive. We
|
||||
show examples of reading in several data formats, preprocessing the data for several types of tasks, and then preparing
|
||||
the data into PyTorch/TensorFlow ``Dataset`` objects which can easily be used either with
|
||||
:class:`~transformers.Trainer`/:class:`~transformers.TFTrainer` or with native PyTorch/TensorFlow.
|
||||
|
||||
We include several examples, each of which demonstrates a different type of common downstream task:
|
||||
|
|
@ -28,13 +28,13 @@ Sequence Classification with IMDb Reviews
|
|||
|
||||
.. note::
|
||||
|
||||
This dataset can be explored in the Hugging Face model hub (`IMDb <https://huggingface.co/datasets/imdb>`_), and can
|
||||
be alternatively downloaded with the 🤗 NLP library with ``load_dataset("imdb")``.
|
||||
This dataset can be explored in the Hugging Face model hub (`IMDb <https://huggingface.co/datasets/imdb>`_), and
|
||||
can be alternatively downloaded with the 🤗 NLP library with ``load_dataset("imdb")``.
|
||||
|
||||
In this example, we'll show how to download, tokenize, and train a model on the IMDb reviews dataset. This task
|
||||
takes the text of a review and requires the model to predict whether the sentiment of the review is positive or
|
||||
negative. Let's start by downloading the dataset from the
|
||||
`Large Movie Review Dataset <http://ai.stanford.edu/~amaas/data/sentiment/>`_ webpage.
|
||||
In this example, we'll show how to download, tokenize, and train a model on the IMDb reviews dataset. This task takes
|
||||
the text of a review and requires the model to predict whether the sentiment of the review is positive or negative.
|
||||
Let's start by downloading the dataset from the `Large Movie Review Dataset
|
||||
<http://ai.stanford.edu/~amaas/data/sentiment/>`_ webpage.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -62,9 +62,8 @@ read this in.
|
|||
train_texts, train_labels = read_imdb_split('aclImdb/train')
|
||||
test_texts, test_labels = read_imdb_split('aclImdb/test')
|
||||
|
||||
We now have a train and test dataset, but let's also also create a validation set which we can use for for
|
||||
evaluation and tuning without training our test set results. Sklearn has a convenient utility for creating such
|
||||
splits:
|
||||
We now have a train and test dataset, but let's also also create a validation set which we can use for for evaluation
|
||||
and tuning without training our test set results. Sklearn has a convenient utility for creating such splits:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -80,8 +79,8 @@ pre-trained DistilBert, so let's use the DistilBert tokenizer.
|
|||
tokenizer = DistilBertTokenizerFast.from_pretrained('distilbert-base-uncased')
|
||||
|
||||
Now we can simply pass our texts to the tokenizer. We'll pass ``truncation=True`` and ``padding=True``, which will
|
||||
ensure that all of our sequences are padded to the same length and are truncated to be no longer model's maximum
|
||||
input length. This will allow us to feed batches of sequences into the model at the same time.
|
||||
ensure that all of our sequences are padded to the same length and are truncated to be no longer model's maximum input
|
||||
length. This will allow us to feed batches of sequences into the model at the same time.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -90,9 +89,9 @@ input length. This will allow us to feed batches of sequences into the model at
|
|||
test_encodings = tokenizer(test_texts, truncation=True, padding=True)
|
||||
|
||||
Now, let's turn our labels and encodings into a Dataset object. In PyTorch, this is done by subclassing a
|
||||
``torch.utils.data.Dataset`` object and implementing ``__len__`` and ``__getitem__``. In TensorFlow, we pass our input encodings and
|
||||
labels to the ``from_tensor_slices`` constructor method. We put the data in this format so that the data can be
|
||||
easily batched such that each key in the batch encoding corresponds to a named parameter of the
|
||||
``torch.utils.data.Dataset`` object and implementing ``__len__`` and ``__getitem__``. In TensorFlow, we pass our input
|
||||
encodings and labels to the ``from_tensor_slices`` constructor method. We put the data in this format so that the data
|
||||
can be easily batched such that each key in the batch encoding corresponds to a named parameter of the
|
||||
:meth:`~transformers.DistilBertForSequenceClassification.forward` method of the model we will train.
|
||||
|
||||
.. code-block:: python
|
||||
|
|
@ -133,17 +132,17 @@ easily batched such that each key in the batch encoding corresponds to a named p
|
|||
))
|
||||
|
||||
Now that our datasets our ready, we can fine-tune a model either with the 🤗
|
||||
:class:`~transformers.Trainer`/:class:`~transformers.TFTrainer` or with native PyTorch/TensorFlow. See
|
||||
:doc:`training <training>`.
|
||||
:class:`~transformers.Trainer`/:class:`~transformers.TFTrainer` or with native PyTorch/TensorFlow. See :doc:`training
|
||||
<training>`.
|
||||
|
||||
.. _ft_trainer:
|
||||
|
||||
Fine-tuning with Trainer
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The steps above prepared the datasets in the way that the trainer is expected. Now all we need to do is create a
|
||||
model to fine-tune, define the :class:`~transformers.TrainingArguments`/:class:`~transformers.TFTrainingArguments`
|
||||
and instantiate a :class:`~transformers.Trainer`/:class:`~transformers.TFTrainer`.
|
||||
The steps above prepared the datasets in the way that the trainer is expected. Now all we need to do is create a model
|
||||
to fine-tune, define the :class:`~transformers.TrainingArguments`/:class:`~transformers.TFTrainingArguments` and
|
||||
instantiate a :class:`~transformers.Trainer`/:class:`~transformers.TFTrainer`.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -248,15 +247,15 @@ Token Classification with W-NUT Emerging Entities
|
|||
|
||||
.. note::
|
||||
|
||||
This dataset can be explored in the Hugging Face model hub (`WNUT-17 <https://huggingface.co/datasets/wnut_17>`_), and can
|
||||
be alternatively downloaded with the 🤗 NLP library with ``load_dataset("wnut_17")``.
|
||||
This dataset can be explored in the Hugging Face model hub (`WNUT-17 <https://huggingface.co/datasets/wnut_17>`_),
|
||||
and can be alternatively downloaded with the 🤗 NLP library with ``load_dataset("wnut_17")``.
|
||||
|
||||
Next we will look at token classification. Rather than classifying an entire sequence, this task classifies token by
|
||||
token. We'll demonstrate how to do this with
|
||||
`Named Entity Recognition <http://nlpprogress.com/english/named_entity_recognition.html>`_, which involves
|
||||
identifying tokens which correspond to a predefined set of "entities". Specifically, we'll use the
|
||||
`W-NUT Emerging and Rare entities <http://noisy-text.github.io/2017/emerging-rare-entities.html>`_ corpus. The data
|
||||
is given as a collection of pre-tokenized documents where each token is assigned a tag.
|
||||
token. We'll demonstrate how to do this with `Named Entity Recognition
|
||||
<http://nlpprogress.com/english/named_entity_recognition.html>`_, which involves identifying tokens which correspond to
|
||||
a predefined set of "entities". Specifically, we'll use the `W-NUT Emerging and Rare entities
|
||||
<http://noisy-text.github.io/2017/emerging-rare-entities.html>`_ corpus. The data is given as a collection of
|
||||
pre-tokenized documents where each token is assigned a tag.
|
||||
|
||||
Let's start by downloading the data.
|
||||
|
||||
|
|
@ -264,10 +263,10 @@ Let's start by downloading the data.
|
|||
|
||||
wget http://noisy-text.github.io/2017/files/wnut17train.conll
|
||||
|
||||
In this case, we'll just download the train set, which is a single text file. Each line of the file contains either
|
||||
(1) a word and tag separated by a tab, or (2) a blank line indicating the end of a document. Let's write a
|
||||
function to read this in. We'll take in the file path and return ``token_docs`` which is a list of lists of token
|
||||
strings, and ``token_tags`` which is a list of lists of tag strings.
|
||||
In this case, we'll just download the train set, which is a single text file. Each line of the file contains either (1)
|
||||
a word and tag separated by a tab, or (2) a blank line indicating the end of a document. Let's write a function to read
|
||||
this in. We'll take in the file path and return ``token_docs`` which is a list of lists of token strings, and
|
||||
``token_tags`` which is a list of lists of tag strings.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -290,11 +289,11 @@ strings, and ``token_tags`` which is a list of lists of tag strings.
|
|||
tags.append(tag)
|
||||
token_docs.append(tokens)
|
||||
tag_docs.append(tags)
|
||||
|
||||
|
||||
return token_docs, tag_docs
|
||||
|
||||
|
||||
texts, tags = read_wnut('wnut17train.conll')
|
||||
|
||||
|
||||
Just to see what this data looks like, let's take a look at a segment of the first document.
|
||||
|
||||
.. code-block:: python
|
||||
|
|
@ -303,8 +302,8 @@ Just to see what this data looks like, let's take a look at a segment of the fir
|
|||
['for', 'two', 'weeks', '.', 'Empire', 'State', 'Building']
|
||||
['O', 'O', 'O', 'O', 'B-location', 'I-location', 'I-location']
|
||||
|
||||
``location`` is an entity type, ``B-`` indicates the beginning of an entity, and ``I-`` indicates consecutive positions of
|
||||
the same entity ("Empire State Building" is considered one entity). ``O`` indicates the token does not correspond to
|
||||
``location`` is an entity type, ``B-`` indicates the beginning of an entity, and ``I-`` indicates consecutive positions
|
||||
of the same entity ("Empire State Building" is considered one entity). ``O`` indicates the token does not correspond to
|
||||
any entity.
|
||||
|
||||
Now that we've read the data in, let's create a train/validation split:
|
||||
|
|
@ -314,8 +313,8 @@ Now that we've read the data in, let's create a train/validation split:
|
|||
from sklearn.model_selection import train_test_split
|
||||
train_texts, val_texts, train_tags, val_tags = train_test_split(texts, tags, test_size=.2)
|
||||
|
||||
Next, let's create encodings for our tokens and tags. For the tags, we can start by just create a simple mapping
|
||||
which we'll use in a moment:
|
||||
Next, let's create encodings for our tokens and tags. For the tags, we can start by just create a simple mapping which
|
||||
we'll use in a moment:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -323,11 +322,11 @@ which we'll use in a moment:
|
|||
tag2id = {tag: id for id, tag in enumerate(unique_tags)}
|
||||
id2tag = {id: tag for tag, id in tag2id.items()}
|
||||
|
||||
To encode the tokens, we'll use a pre-trained DistilBert tokenizer. We can tell the tokenizer that we're dealing
|
||||
with ready-split tokens rather than full sentence strings by passing ``is_split_into_words=True``. We'll also pass
|
||||
``padding=True`` and ``truncation=True`` to pad the sequences to be the same length. Lastly, we can tell the model
|
||||
to return information about the tokens which are split by the wordpiece tokenization process, which we will need in
|
||||
a moment.
|
||||
To encode the tokens, we'll use a pre-trained DistilBert tokenizer. We can tell the tokenizer that we're dealing with
|
||||
ready-split tokens rather than full sentence strings by passing ``is_split_into_words=True``. We'll also pass
|
||||
``padding=True`` and ``truncation=True`` to pad the sequences to be the same length. Lastly, we can tell the model to
|
||||
return information about the tokens which are split by the wordpiece tokenization process, which we will need in a
|
||||
moment.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -339,26 +338,26 @@ a moment.
|
|||
Great, so now our tokens are nicely encoded in the format that they need to be in to feed them into our DistilBert
|
||||
model below.
|
||||
|
||||
Now we arrive at a common obstacle with using pre-trained models for token-level classification: many of the tokens
|
||||
in the W-NUT corpus are not in DistilBert's vocabulary. Bert and many models like it use a method called WordPiece
|
||||
Tokenization, meaning that single words are split into multiple tokens such that each token is likely to be in
|
||||
the vocabulary. For example, DistilBert's tokenizer would split the Twitter handle ``@huggingface`` into the tokens
|
||||
``['@', 'hugging', '##face']``. This is a problem for us because we have exactly one tag per token. If the tokenizer
|
||||
splits a token into multiple sub-tokens, then we will end up with a mismatch between our tokens and our labels.
|
||||
Now we arrive at a common obstacle with using pre-trained models for token-level classification: many of the tokens in
|
||||
the W-NUT corpus are not in DistilBert's vocabulary. Bert and many models like it use a method called WordPiece
|
||||
Tokenization, meaning that single words are split into multiple tokens such that each token is likely to be in the
|
||||
vocabulary. For example, DistilBert's tokenizer would split the Twitter handle ``@huggingface`` into the tokens ``['@',
|
||||
'hugging', '##face']``. This is a problem for us because we have exactly one tag per token. If the tokenizer splits a
|
||||
token into multiple sub-tokens, then we will end up with a mismatch between our tokens and our labels.
|
||||
|
||||
One way to handle this is to only train on the tag labels for the first subtoken of a split token. We can do this in
|
||||
🤗 Transformers by setting the labels we wish to ignore to ``-100``. In the example above, if the label for
|
||||
One way to handle this is to only train on the tag labels for the first subtoken of a split token. We can do this in 🤗
|
||||
Transformers by setting the labels we wish to ignore to ``-100``. In the example above, if the label for
|
||||
``@HuggingFace`` is ``3`` (indexing ``B-corporation``), we would set the labels of ``['@', 'hugging', '##face']`` to
|
||||
``[3, -100, -100]``.
|
||||
|
||||
Let's write a function to do this. This is where we will use the ``offset_mapping`` from the tokenizer as mentioned
|
||||
above. For each sub-token returned by the tokenizer, the offset mapping gives us a tuple indicating the sub-token's
|
||||
start position and end position relative to the original token it was split from. That means that if the first
|
||||
position in the tuple is anything other than ``0``, we will set its corresponding label to ``-100``. While we're at
|
||||
it, we can also set labels to ``-100`` if the second position of the offset mapping is ``0``, since this means it must
|
||||
be a special token like ``[PAD]`` or ``[CLS]``.
|
||||
start position and end position relative to the original token it was split from. That means that if the first position
|
||||
in the tuple is anything other than ``0``, we will set its corresponding label to ``-100``. While we're at it, we can
|
||||
also set labels to ``-100`` if the second position of the offset mapping is ``0``, since this means it must be a
|
||||
special token like ``[PAD]`` or ``[CLS]``.
|
||||
|
||||
.. note::
|
||||
.. note::
|
||||
|
||||
Due to a recently fixed bug, -1 must be used instead of -100 when using TensorFlow in 🤗 Transformers <= 3.02.
|
||||
|
||||
|
|
@ -379,7 +378,7 @@ be a special token like ``[PAD]`` or ``[CLS]``.
|
|||
encoded_labels.append(doc_enc_labels.tolist())
|
||||
|
||||
return encoded_labels
|
||||
|
||||
|
||||
train_labels = encode_tags(train_tags, train_encodings)
|
||||
val_labels = encode_tags(val_tags, val_encodings)
|
||||
|
||||
|
|
@ -447,8 +446,9 @@ Question Answering with SQuAD 2.0
|
|||
|
||||
.. note::
|
||||
|
||||
This dataset can be explored in the Hugging Face model hub (`SQuAD V2 <https://huggingface.co/datasets/squad_v2>`_), and can
|
||||
be alternatively downloaded with the 🤗 NLP library with ``load_dataset("squad_v2")``.
|
||||
This dataset can be explored in the Hugging Face model hub (`SQuAD V2
|
||||
<https://huggingface.co/datasets/squad_v2>`_), and can be alternatively downloaded with the 🤗 NLP library with
|
||||
``load_dataset("squad_v2")``.
|
||||
|
||||
Question answering comes in many forms. In this example, we'll look at the particular type of extractive QA that
|
||||
involves answering a question about a passage by highlighting the segment of the passage that answers the question.
|
||||
|
|
@ -464,8 +464,8 @@ We will start by downloading the data:
|
|||
wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json -O squad/dev-v2.0.json
|
||||
|
||||
Each split is in a structured json file with a number of questions and answers for each passage (or context). We'll
|
||||
take this apart into parallel lists of contexts, questions, and answers (note that the contexts here are repeated
|
||||
since there are multiple questions per context):
|
||||
take this apart into parallel lists of contexts, questions, and answers (note that the contexts here are repeated since
|
||||
there are multiple questions per context):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -491,17 +491,17 @@ since there are multiple questions per context):
|
|||
answers.append(answer)
|
||||
|
||||
return contexts, questions, answers
|
||||
|
||||
|
||||
train_contexts, train_questions, train_answers = read_squad('squad/train-v2.0.json')
|
||||
val_contexts, val_questions, val_answers = read_squad('squad/dev-v2.0.json')
|
||||
|
||||
The contexts and questions are just strings. The answers are dicts containing the subsequence of the passage with
|
||||
the correct answer as well as an integer indicating the character at which the answer begins. In order to train a
|
||||
model on this data we need (1) the tokenized context/question pairs, and (2) integers indicating at which *token*
|
||||
positions the answer begins and ends.
|
||||
The contexts and questions are just strings. The answers are dicts containing the subsequence of the passage with the
|
||||
correct answer as well as an integer indicating the character at which the answer begins. In order to train a model on
|
||||
this data we need (1) the tokenized context/question pairs, and (2) integers indicating at which *token* positions the
|
||||
answer begins and ends.
|
||||
|
||||
First, let's get the *character* position at which the answer ends in the passage (we are given the starting
|
||||
position). Sometimes SQuAD answers are off by one or two characters, so we will also adjust for that.
|
||||
First, let's get the *character* position at which the answer ends in the passage (we are given the starting position).
|
||||
Sometimes SQuAD answers are off by one or two characters, so we will also adjust for that.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -510,7 +510,7 @@ position). Sometimes SQuAD answers are off by one or two characters, so we will
|
|||
gold_text = answer['text']
|
||||
start_idx = answer['answer_start']
|
||||
end_idx = start_idx + len(gold_text)
|
||||
|
||||
|
||||
# sometimes squad answers are off by a character or two – fix this
|
||||
if context[start_idx:end_idx] == gold_text:
|
||||
answer['answer_end'] = end_idx
|
||||
|
|
@ -524,9 +524,9 @@ position). Sometimes SQuAD answers are off by one or two characters, so we will
|
|||
add_end_idx(train_answers, train_contexts)
|
||||
add_end_idx(val_answers, val_contexts)
|
||||
|
||||
Now ``train_answers`` and ``val_answers`` include the character end positions and the corrected start positions.
|
||||
Next, let's tokenize our context/question pairs. 🤗 Tokenizers can accept parallel lists of sequences and encode
|
||||
them together as sequence pairs.
|
||||
Now ``train_answers`` and ``val_answers`` include the character end positions and the corrected start positions. Next,
|
||||
let's tokenize our context/question pairs. 🤗 Tokenizers can accept parallel lists of sequences and encode them together
|
||||
as sequence pairs.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -536,8 +536,8 @@ them together as sequence pairs.
|
|||
train_encodings = tokenizer(train_contexts, train_questions, truncation=True, padding=True)
|
||||
val_encodings = tokenizer(val_contexts, val_questions, truncation=True, padding=True)
|
||||
|
||||
Next we need to convert our character start/end positions to token start/end positions. When using 🤗 Fast
|
||||
Tokenizers, we can use the built in :func:`~transformers.BatchEncoding.char_to_token` method.
|
||||
Next we need to convert our character start/end positions to token start/end positions. When using 🤗 Fast Tokenizers,
|
||||
we can use the built in :func:`~transformers.BatchEncoding.char_to_token` method.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -557,9 +557,9 @@ Tokenizers, we can use the built in :func:`~transformers.BatchEncoding.char_to_t
|
|||
add_token_positions(train_encodings, train_answers)
|
||||
add_token_positions(val_encodings, val_answers)
|
||||
|
||||
Our data is ready. Let's just put it in a PyTorch/TensorFlow dataset so that we can easily use it for
|
||||
training. In PyTorch, we define a custom ``Dataset`` class. In TensorFlow, we pass a tuple of
|
||||
``(inputs_dict, labels_dict)`` to the ``from_tensor_slices`` method.
|
||||
Our data is ready. Let's just put it in a PyTorch/TensorFlow dataset so that we can easily use it for training. In
|
||||
PyTorch, we define a custom ``Dataset`` class. In TensorFlow, we pass a tuple of ``(inputs_dict, labels_dict)`` to the
|
||||
``from_tensor_slices`` method.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -575,7 +575,7 @@ training. In PyTorch, we define a custom ``Dataset`` class. In TensorFlow, we pa
|
|||
|
||||
def __len__(self):
|
||||
return len(self.encodings.input_ids)
|
||||
|
||||
|
||||
train_dataset = SquadDataset(train_encodings)
|
||||
val_dataset = SquadDataset(val_encodings)
|
||||
## TENSORFLOW CODE
|
||||
|
|
@ -668,12 +668,11 @@ Additional Resources
|
|||
Using the 🤗 NLP Datasets & Metrics library
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This tutorial demonstrates how to read in datasets from various raw text formats and prepare them for training with
|
||||
🤗 Transformers so that you can do the same thing with your own custom datasets. However, we recommend users use the
|
||||
`🤗 NLP library <https://github.com/huggingface/nlp>`_ for working with the 150+ datasets included in the
|
||||
`hub <https://huggingface.co/datasets>`_, including the three datasets used in this tutorial. As a very brief overview,
|
||||
we will show how to use the NLP library to download and prepare the IMDb dataset from the first example,
|
||||
:ref:`seq_imdb`.
|
||||
This tutorial demonstrates how to read in datasets from various raw text formats and prepare them for training with 🤗
|
||||
Transformers so that you can do the same thing with your own custom datasets. However, we recommend users use the `🤗
|
||||
NLP library <https://github.com/huggingface/nlp>`_ for working with the 150+ datasets included in the `hub
|
||||
<https://huggingface.co/datasets>`_, including the three datasets used in this tutorial. As a very brief overview, we
|
||||
will show how to use the NLP library to download and prepare the IMDb dataset from the first example, :ref:`seq_imdb`.
|
||||
|
||||
Start by downloading the dataset:
|
||||
|
||||
|
|
@ -689,8 +688,8 @@ Each dataset has multiple columns corresponding to different features. Let's see
|
|||
>>> print(train.column_names)
|
||||
['label', 'text']
|
||||
|
||||
Great. Now let's tokenize the text. We can do this using the ``map`` method. We'll also rename the ``label`` column
|
||||
to ``labels`` to match the model's input arguments.
|
||||
Great. Now let's tokenize the text. We can do this using the ``map`` method. We'll also rename the ``label`` column to
|
||||
``labels`` to match the model's input arguments.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -711,5 +710,5 @@ dataset elements.
|
|||
>>> {key: val.shape for key, val in train[0].items()})
|
||||
{'labels': TensorShape([]), 'input_ids': TensorShape([512]), 'attention_mask': TensorShape([512])}
|
||||
|
||||
We now have a fully-prepared dataset. Check out `the 🤗 NLP docs <https://huggingface.co/nlp/processing.html>`_ for
|
||||
a more thorough introduction.
|
||||
We now have a fully-prepared dataset. Check out `the 🤗 NLP docs <https://huggingface.co/nlp/processing.html>`_ for a
|
||||
more thorough introduction.
|
||||
|
|
|
|||
|
|
@ -57,8 +57,8 @@ The tokenizer takes care of splitting the sequence into tokens available in the
|
|||
>>> tokenized_sequence = tokenizer.tokenize(sequence)
|
||||
|
||||
The tokens are either words or subwords. Here for instance, "VRAM" wasn't in the model vocabulary, so it's been split
|
||||
in "V", "RA" and "M". To indicate those tokens are not separate words but parts of the same word, a double-hash prefix is
|
||||
added for "RA" and "M":
|
||||
in "V", "RA" and "M". To indicate those tokens are not separate words but parts of the same word, a double-hash prefix
|
||||
is added for "RA" and "M":
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -66,8 +66,8 @@ added for "RA" and "M":
|
|||
['A', 'Titan', 'R', '##T', '##X', 'has', '24', '##GB', 'of', 'V', '##RA', '##M']
|
||||
|
||||
These tokens can then be converted into IDs which are understandable by the model. This can be done by directly feeding
|
||||
the sentence to the tokenizer, which leverages the Rust implementation of
|
||||
`huggingface/tokenizers <https://github.com/huggingface/tokenizers>`__ for peak performance.
|
||||
the sentence to the tokenizer, which leverages the Rust implementation of `huggingface/tokenizers
|
||||
<https://github.com/huggingface/tokenizers>`__ for peak performance.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -105,8 +105,8 @@ because this is the way a :class:`~transformers.BertModel` is going to expect it
|
|||
Attention mask
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The attention mask is an optional argument used when batching sequences together. This argument indicates to the
|
||||
model which tokens should be attended to, and which should not.
|
||||
The attention mask is an optional argument used when batching sequences together. This argument indicates to the model
|
||||
which tokens should be attended to, and which should not.
|
||||
|
||||
For example, consider these two sequences:
|
||||
|
||||
|
|
@ -145,10 +145,10 @@ We can see that 0s have been added on the right of the first sentence to make it
|
|||
>>> padded_sequences["input_ids"]
|
||||
[[101, 1188, 1110, 170, 1603, 4954, 119, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], [101, 1188, 1110, 170, 1897, 1263, 4954, 119, 1135, 1110, 1120, 1655, 2039, 1190, 1103, 4954, 138, 119, 102]]
|
||||
|
||||
This can then be converted into a tensor in PyTorch or TensorFlow. The attention mask is a binary tensor indicating
|
||||
the position of the padded indices so that the model does not attend to them. For the
|
||||
:class:`~transformers.BertTokenizer`, :obj:`1` indicates a value that should be attended to, while :obj:`0` indicates
|
||||
a padded value. This attention mask is in the dictionary returned by the tokenizer under the key "attention_mask":
|
||||
This can then be converted into a tensor in PyTorch or TensorFlow. The attention mask is a binary tensor indicating the
|
||||
position of the padded indices so that the model does not attend to them. For the :class:`~transformers.BertTokenizer`,
|
||||
:obj:`1` indicates a value that should be attended to, while :obj:`0` indicates a padded value. This attention mask is
|
||||
in the dictionary returned by the tokenizer under the key "attention_mask":
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -161,15 +161,16 @@ Token Type IDs
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Some models' purpose is to do sequence classification or question answering. These require two different sequences to
|
||||
be joined in a single "input_ids" entry, which usually is performed with the help of special tokens, such as the classifier (``[CLS]``) and separator (``[SEP]``)
|
||||
tokens. For example, the BERT model builds its two sequence input as such:
|
||||
be joined in a single "input_ids" entry, which usually is performed with the help of special tokens, such as the
|
||||
classifier (``[CLS]``) and separator (``[SEP]``) tokens. For example, the BERT model builds its two sequence input as
|
||||
such:
|
||||
|
||||
.. code-block::
|
||||
|
||||
>>> # [CLS] SEQUENCE_A [SEP] SEQUENCE_B [SEP]
|
||||
|
||||
We can use our tokenizer to automatically generate such a sentence by passing the two sequences to ``tokenizer`` as two arguments (and
|
||||
not a list, like before) like this:
|
||||
We can use our tokenizer to automatically generate such a sentence by passing the two sequences to ``tokenizer`` as two
|
||||
arguments (and not a list, like before) like this:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -189,8 +190,8 @@ which will return:
|
|||
[CLS] HuggingFace is based in NYC [SEP] Where is HuggingFace based? [SEP]
|
||||
|
||||
This is enough for some models to understand where one sequence ends and where another begins. However, other models,
|
||||
such as BERT, also deploy token type IDs (also called segment IDs). They are represented as a binary
|
||||
mask identifying the two types of sequence in the model.
|
||||
such as BERT, also deploy token type IDs (also called segment IDs). They are represented as a binary mask identifying
|
||||
the two types of sequence in the model.
|
||||
|
||||
The tokenizer returns this mask as the "token_type_ids" entry:
|
||||
|
||||
|
|
@ -209,14 +210,15 @@ Some models, like :class:`~transformers.XLNetModel` use an additional token repr
|
|||
Position IDs
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Contrary to RNNs that have the position of each token embedded within them,
|
||||
transformers are unaware of the position of each token. Therefore, the position IDs (``position_ids``) are used by the model to identify each token's position in the list of tokens.
|
||||
Contrary to RNNs that have the position of each token embedded within them, transformers are unaware of the position of
|
||||
each token. Therefore, the position IDs (``position_ids``) are used by the model to identify each token's position in
|
||||
the list of tokens.
|
||||
|
||||
They are an optional parameter. If no ``position_ids`` is passed to the model, the IDs are automatically created as absolute
|
||||
positional embeddings.
|
||||
They are an optional parameter. If no ``position_ids`` is passed to the model, the IDs are automatically created as
|
||||
absolute positional embeddings.
|
||||
|
||||
Absolute positional embeddings are selected in the range ``[0, config.max_position_embeddings - 1]``. Some models
|
||||
use other types of positional embeddings, such as sinusoidal position embeddings or relative position embeddings.
|
||||
Absolute positional embeddings are selected in the range ``[0, config.max_position_embeddings - 1]``. Some models use
|
||||
other types of positional embeddings, such as sinusoidal position embeddings or relative position embeddings.
|
||||
|
||||
.. _labels:
|
||||
|
||||
|
|
@ -224,43 +226,41 @@ Labels
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The labels are an optional argument which can be passed in order for the model to compute the loss itself. These labels
|
||||
should be the expected prediction of the model: it will use the standard loss in order to compute the loss between
|
||||
its predictions and the expected value (the label).
|
||||
should be the expected prediction of the model: it will use the standard loss in order to compute the loss between its
|
||||
predictions and the expected value (the label).
|
||||
|
||||
These labels are different according to the model head, for example:
|
||||
|
||||
- For sequence classification models (e.g., :class:`~transformers.BertForSequenceClassification`), the model expects
|
||||
a tensor of dimension :obj:`(batch_size)` with each value of the batch corresponding to the expected label of the
|
||||
- For sequence classification models (e.g., :class:`~transformers.BertForSequenceClassification`), the model expects a
|
||||
tensor of dimension :obj:`(batch_size)` with each value of the batch corresponding to the expected label of the
|
||||
entire sequence.
|
||||
- For token classification models (e.g., :class:`~transformers.BertForTokenClassification`), the model expects
|
||||
a tensor of dimension :obj:`(batch_size, seq_length)` with each value corresponding to the expected label of each
|
||||
individual token.
|
||||
- For masked language modeling (e.g., :class:`~transformers.BertForMaskedLM`), the model expects
|
||||
a tensor of dimension :obj:`(batch_size, seq_length)` with each value corresponding to the expected label of each
|
||||
individual token: the labels being the token ID for the masked token, and values to be ignored for the rest (usually
|
||||
-100).
|
||||
- For token classification models (e.g., :class:`~transformers.BertForTokenClassification`), the model expects a tensor
|
||||
of dimension :obj:`(batch_size, seq_length)` with each value corresponding to the expected label of each individual
|
||||
token.
|
||||
- For masked language modeling (e.g., :class:`~transformers.BertForMaskedLM`), the model expects a tensor of dimension
|
||||
:obj:`(batch_size, seq_length)` with each value corresponding to the expected label of each individual token: the
|
||||
labels being the token ID for the masked token, and values to be ignored for the rest (usually -100).
|
||||
- For sequence to sequence tasks,(e.g., :class:`~transformers.BartForConditionalGeneration`,
|
||||
:class:`~transformers.MBartForConditionalGeneration`), the model expects a tensor of dimension
|
||||
:obj:`(batch_size, tgt_seq_length)` with each value corresponding to the target sequences associated with each
|
||||
input sequence. During training, both `BART` and `T5` will make the appropriate `decoder_input_ids` and decoder
|
||||
attention masks internally. They usually do not need to be supplied. This does not apply to models leveraging the
|
||||
Encoder-Decoder framework.
|
||||
See the documentation of each model for more information on each specific model's labels.
|
||||
:class:`~transformers.MBartForConditionalGeneration`), the model expects a tensor of dimension :obj:`(batch_size,
|
||||
tgt_seq_length)` with each value corresponding to the target sequences associated with each input sequence. During
|
||||
training, both `BART` and `T5` will make the appropriate `decoder_input_ids` and decoder attention masks internally.
|
||||
They usually do not need to be supplied. This does not apply to models leveraging the Encoder-Decoder framework. See
|
||||
the documentation of each model for more information on each specific model's labels.
|
||||
|
||||
The base models (e.g., :class:`~transformers.BertModel`) do not accept labels, as these are the base transformer models,
|
||||
simply outputting features.
|
||||
The base models (e.g., :class:`~transformers.BertModel`) do not accept labels, as these are the base transformer
|
||||
models, simply outputting features.
|
||||
|
||||
.. _decoder-input-ids:
|
||||
|
||||
Decoder input IDs
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This input is specific to encoder-decoder models, and contains the input IDs that will be fed to the decoder.
|
||||
These inputs should be used for sequence to sequence tasks, such as translation or summarization, and are usually
|
||||
built in a way specific to each model.
|
||||
This input is specific to encoder-decoder models, and contains the input IDs that will be fed to the decoder. These
|
||||
inputs should be used for sequence to sequence tasks, such as translation or summarization, and are usually built in a
|
||||
way specific to each model.
|
||||
|
||||
Most encoder-decoder models (BART, T5) create their :obj:`decoder_input_ids` on their own from the :obj:`labels`.
|
||||
In such models, passing the :obj:`labels` is the preferred way to handle training.
|
||||
Most encoder-decoder models (BART, T5) create their :obj:`decoder_input_ids` on their own from the :obj:`labels`. In
|
||||
such models, passing the :obj:`labels` is the preferred way to handle training.
|
||||
|
||||
Please check each model's docs to see how they handle these input IDs for sequence to sequence training.
|
||||
|
||||
|
|
@ -270,18 +270,18 @@ Feed Forward Chunking
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In each residual attention block in transformers the self-attention layer is usually followed by 2 feed forward layers.
|
||||
The intermediate embedding size of the feed forward layers is often bigger than the hidden size of the model (e.g.,
|
||||
for ``bert-base-uncased``).
|
||||
The intermediate embedding size of the feed forward layers is often bigger than the hidden size of the model (e.g., for
|
||||
``bert-base-uncased``).
|
||||
|
||||
For an input of size ``[batch_size, sequence_length]``, the memory required to store the intermediate feed forward
|
||||
embeddings ``[batch_size, sequence_length, config.intermediate_size]`` can account for a large fraction of the memory
|
||||
use. The authors of `Reformer: The Efficient Transformer <https://arxiv.org/abs/2001.04451>`_ noticed that since the
|
||||
computation is independent of the ``sequence_length`` dimension, it is mathematically equivalent to compute the output
|
||||
embeddings of both feed forward layers ``[batch_size, config.hidden_size]_0, ..., [batch_size, config.hidden_size]_n``
|
||||
individually and concat them afterward to ``[batch_size, sequence_length, config.hidden_size]`` with
|
||||
``n = sequence_length``, which trades increased computation time against reduced memory use, but yields a
|
||||
mathematically **equivalent** result.
|
||||
individually and concat them afterward to ``[batch_size, sequence_length, config.hidden_size]`` with ``n =
|
||||
sequence_length``, which trades increased computation time against reduced memory use, but yields a mathematically
|
||||
**equivalent** result.
|
||||
|
||||
For models employing the function :func:`~.transformers.apply_chunking_to_forward`, the ``chunk_size`` defines the
|
||||
number of output embeddings that are computed in parallel and thus defines the trade-off between memory and time
|
||||
complexity. If ``chunk_size`` is set to 0, no feed forward chunking is done.
|
||||
complexity. If ``chunk_size`` is set to 0, no feed forward chunking is done.
|
||||
|
|
|
|||
|
|
@ -47,6 +47,7 @@ The documentation is organized in five parts:
|
|||
- **RESEARCH** focuses on tutorials that have less to do with how to use the library but more about general resarch in
|
||||
transformers model
|
||||
- The three last section contain the documentation of each public class and function, grouped in:
|
||||
|
||||
- **MAIN CLASSES** for the main classes exposing the important APIs of the library.
|
||||
- **MODELS** for the classes and functions related to each model implemented in the library.
|
||||
- **INTERNAL HELPERS** for the classes and functions we use internally.
|
||||
|
|
@ -122,7 +123,7 @@ conversion utilities for the following models:
|
|||
20. :doc:`MarianMT <model_doc/marian>` Machine translation models trained using `OPUS <http://opus.nlpl.eu/>`__ data by
|
||||
Jörg Tiedemann. The `Marian Framework <https://marian-nmt.github.io/>`__ is being developed by the Microsoft
|
||||
Translator Team.
|
||||
21. :doc:`MBart <model_doc/mbart>` (from Facebook) released with the paper `Multilingual Denoising Pre-training for
|
||||
21. :doc:`MBart <model_doc/mbart>` (from Facebook) released with the paper `Multilingual Denoising Pre-training for
|
||||
Neural Machine Translation <https://arxiv.org/abs/2001.08210>`__ by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li,
|
||||
Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.
|
||||
22. :doc:`Pegasus <model_doc/pegasus>` (from Google) released with the paper `PEGASUS: Pre-training with Extracted
|
||||
|
|
|
|||
|
|
@ -85,4 +85,4 @@ TensorFlow Helper Functions
|
|||
|
||||
.. autofunction:: transformers.modeling_tf_utils.keras_serializable
|
||||
|
||||
.. autofunction:: transformers.modeling_tf_utils.shape_list
|
||||
.. autofunction:: transformers.modeling_tf_utils.shape_list
|
||||
|
|
|
|||
|
|
@ -25,6 +25,7 @@ SpecialTokensMixin
|
|||
|
||||
Enums and namedtuples
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.tokenization_utils_base.ExplicitEnum
|
||||
|
||||
.. autoclass:: transformers.tokenization_utils_base.PaddingStrategy
|
||||
|
|
|
|||
|
|
@ -24,4 +24,4 @@ Distributed Evaluation
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.trainer_pt_utils.DistributedTensorGatherer
|
||||
:members:
|
||||
:members:
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ You can also use the environment variable ``TRANSFORMERS_VERBOSITY`` to override
|
|||
to one of the following: ``debug``, ``info``, ``warning``, ``error``, ``critical``. For example:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
||||
TRANSFORMERS_VERBOSITY=error ./myprogram.py
|
||||
|
||||
All the methods of this logging module are documented below, the main ones are
|
||||
|
|
@ -55,4 +55,4 @@ Other functions
|
|||
|
||||
.. autofunction:: transformers.logging.enable_explicit_format
|
||||
|
||||
.. autofunction:: transformers.logging.reset_format
|
||||
.. autofunction:: transformers.logging.reset_format
|
||||
|
|
|
|||
|
|
@ -52,4 +52,4 @@ Generative models
|
|||
:members:
|
||||
|
||||
.. autoclass:: transformers.generation_tf_utils.TFGenerationMixin
|
||||
:members:
|
||||
:members:
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
Pipelines
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
The pipelines are a great and easy way to use models for inference. These pipelines are objects that abstract most
|
||||
of the complex code from the library, offering a simple API dedicated to several tasks, including Named Entity
|
||||
The pipelines are a great and easy way to use models for inference. These pipelines are objects that abstract most of
|
||||
the complex code from the library, offering a simple API dedicated to several tasks, including Named Entity
|
||||
Recognition, Masked Language Modeling, Sentiment Analysis, Feature Extraction and Question Answering. See the
|
||||
:doc:`task summary <../task_summary>` for examples of use.
|
||||
|
||||
|
|
@ -26,8 +26,8 @@ There are two categories of pipeline abstractions to be aware about:
|
|||
The pipeline abstraction
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The `pipeline` abstraction is a wrapper around all the other available pipelines. It is instantiated as any
|
||||
other pipeline but requires an additional argument which is the `task`.
|
||||
The `pipeline` abstraction is a wrapper around all the other available pipelines. It is instantiated as any other
|
||||
pipeline but requires an additional argument which is the `task`.
|
||||
|
||||
.. autofunction:: transformers.pipeline
|
||||
|
||||
|
|
|
|||
|
|
@ -8,8 +8,8 @@ Processors
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
All processors follow the same architecture which is that of the
|
||||
:class:`~transformers.data.processors.utils.DataProcessor`. The processor returns a list
|
||||
of :class:`~transformers.data.processors.utils.InputExample`. These
|
||||
:class:`~transformers.data.processors.utils.DataProcessor`. The processor returns a list of
|
||||
:class:`~transformers.data.processors.utils.InputExample`. These
|
||||
:class:`~transformers.data.processors.utils.InputExample` can be converted to
|
||||
:class:`~transformers.data.processors.utils.InputFeatures` in order to be fed to the model.
|
||||
|
||||
|
|
@ -28,14 +28,16 @@ of :class:`~transformers.data.processors.utils.InputExample`. These
|
|||
GLUE
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
`General Language Understanding Evaluation (GLUE) <https://gluebenchmark.com/>`__ is a benchmark that evaluates
|
||||
the performance of models across a diverse set of existing NLU tasks. It was released together with the paper
|
||||
`GLUE: A multi-task benchmark and analysis platform for natural language understanding <https://openreview.net/pdf?id=rJ4km2R5t7>`__
|
||||
`General Language Understanding Evaluation (GLUE) <https://gluebenchmark.com/>`__ is a benchmark that evaluates the
|
||||
performance of models across a diverse set of existing NLU tasks. It was released together with the paper `GLUE: A
|
||||
multi-task benchmark and analysis platform for natural language understanding
|
||||
<https://openreview.net/pdf?id=rJ4km2R5t7>`__
|
||||
|
||||
This library hosts a total of 10 processors for the following tasks: MRPC, MNLI, MNLI (mismatched),
|
||||
CoLA, SST2, STSB, QQP, QNLI, RTE and WNLI.
|
||||
This library hosts a total of 10 processors for the following tasks: MRPC, MNLI, MNLI (mismatched), CoLA, SST2, STSB,
|
||||
QQP, QNLI, RTE and WNLI.
|
||||
|
||||
Those processors are:
|
||||
|
||||
- :class:`~transformers.data.processors.utils.MrpcProcessor`
|
||||
- :class:`~transformers.data.processors.utils.MnliProcessor`
|
||||
- :class:`~transformers.data.processors.utils.MnliMismatchedProcessor`
|
||||
|
|
@ -46,7 +48,7 @@ Those processors are:
|
|||
- :class:`~transformers.data.processors.utils.RteProcessor`
|
||||
- :class:`~transformers.data.processors.utils.WnliProcessor`
|
||||
|
||||
Additionally, the following method can be used to load values from a data file and convert them to a list of
|
||||
Additionally, the following method can be used to load values from a data file and convert them to a list of
|
||||
:class:`~transformers.data.processors.utils.InputExample`.
|
||||
|
||||
.. automethod:: transformers.data.processors.glue.glue_convert_examples_to_features
|
||||
|
|
@ -54,36 +56,39 @@ Additionally, the following method can be used to load values from a data file
|
|||
Example usage
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
An example using these processors is given in the `run_glue.py <https://github.com/huggingface/pytorch-transformers/blob/master/examples/text-classification/run_glue.py>`__ script.
|
||||
An example using these processors is given in the `run_glue.py
|
||||
<https://github.com/huggingface/pytorch-transformers/blob/master/examples/text-classification/run_glue.py>`__ script.
|
||||
|
||||
|
||||
XNLI
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
`The Cross-Lingual NLI Corpus (XNLI) <https://www.nyu.edu/projects/bowman/xnli/>`__ is a benchmark that evaluates
|
||||
the quality of cross-lingual text representations.
|
||||
XNLI is crowd-sourced dataset based on `MultiNLI <http://www.nyu.edu/projects/bowman/multinli/>`: pairs of text are labeled with textual entailment
|
||||
annotations for 15 different languages (including both high-resource language such as English and low-resource languages such as Swahili).
|
||||
`The Cross-Lingual NLI Corpus (XNLI) <https://www.nyu.edu/projects/bowman/xnli/>`__ is a benchmark that evaluates the
|
||||
quality of cross-lingual text representations. XNLI is crowd-sourced dataset based on `MultiNLI
|
||||
<http://www.nyu.edu/projects/bowman/multinli/>`: pairs of text are labeled with textual entailment annotations for 15
|
||||
different languages (including both high-resource language such as English and low-resource languages such as Swahili).
|
||||
|
||||
It was released together with the paper
|
||||
`XNLI: Evaluating Cross-lingual Sentence Representations <https://arxiv.org/abs/1809.05053>`__
|
||||
It was released together with the paper `XNLI: Evaluating Cross-lingual Sentence Representations
|
||||
<https://arxiv.org/abs/1809.05053>`__
|
||||
|
||||
This library hosts the processor to load the XNLI data:
|
||||
|
||||
- :class:`~transformers.data.processors.utils.XnliProcessor`
|
||||
|
||||
Please note that since the gold labels are available on the test set, evaluation is performed on the test set.
|
||||
|
||||
An example using these processors is given in the
|
||||
`run_xnli.py <https://github.com/huggingface/pytorch-transformers/blob/master/examples/text-classification/run_xnli.py>`__ script.
|
||||
An example using these processors is given in the `run_xnli.py
|
||||
<https://github.com/huggingface/pytorch-transformers/blob/master/examples/text-classification/run_xnli.py>`__ script.
|
||||
|
||||
|
||||
SQuAD
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
`The Stanford Question Answering Dataset (SQuAD) <https://rajpurkar.github.io/SQuAD-explorer//>`__ is a benchmark that evaluates
|
||||
the performance of models on question answering. Two versions are available, v1.1 and v2.0. The first version (v1.1) was released together with the paper
|
||||
`SQuAD: 100,000+ Questions for Machine Comprehension of Text <https://arxiv.org/abs/1606.05250>`__. The second version (v2.0) was released alongside
|
||||
the paper `Know What You Don't Know: Unanswerable Questions for SQuAD <https://arxiv.org/abs/1806.03822>`__.
|
||||
`The Stanford Question Answering Dataset (SQuAD) <https://rajpurkar.github.io/SQuAD-explorer//>`__ is a benchmark that
|
||||
evaluates the performance of models on question answering. Two versions are available, v1.1 and v2.0. The first version
|
||||
(v1.1) was released together with the paper `SQuAD: 100,000+ Questions for Machine Comprehension of Text
|
||||
<https://arxiv.org/abs/1606.05250>`__. The second version (v2.0) was released alongside the paper `Know What You Don't
|
||||
Know: Unanswerable Questions for SQuAD <https://arxiv.org/abs/1806.03822>`__.
|
||||
|
||||
This library hosts a processor for each of the two versions:
|
||||
|
||||
|
|
@ -91,6 +96,7 @@ Processors
|
|||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Those processors are:
|
||||
|
||||
- :class:`~transformers.data.processors.utils.SquadV1Processor`
|
||||
- :class:`~transformers.data.processors.utils.SquadV2Processor`
|
||||
|
||||
|
|
@ -99,17 +105,18 @@ They both inherit from the abstract class :class:`~transformers.data.processors.
|
|||
.. autoclass:: transformers.data.processors.squad.SquadProcessor
|
||||
:members:
|
||||
|
||||
Additionally, the following method can be used to convert SQuAD examples into :class:`~transformers.data.processors.utils.SquadFeatures`
|
||||
that can be used as model inputs.
|
||||
Additionally, the following method can be used to convert SQuAD examples into
|
||||
:class:`~transformers.data.processors.utils.SquadFeatures` that can be used as model inputs.
|
||||
|
||||
.. automethod:: transformers.data.processors.squad.squad_convert_examples_to_features
|
||||
|
||||
These processors as well as the aforementionned method can be used with files containing the data as well as with the `tensorflow_datasets` package.
|
||||
Examples are given below.
|
||||
These processors as well as the aforementionned method can be used with files containing the data as well as with the
|
||||
`tensorflow_datasets` package. Examples are given below.
|
||||
|
||||
|
||||
Example usage
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Here is an example using the processors as well as the conversion method using data files:
|
||||
|
||||
.. code-block::
|
||||
|
|
@ -149,5 +156,5 @@ Using `tensorflow_datasets` is as easy as using a data file:
|
|||
)
|
||||
|
||||
|
||||
Another example using these processors is given in the
|
||||
`run_squad.py <https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py>`__ script.
|
||||
Another example using these processors is given in the `run_squad.py
|
||||
<https://github.com/huggingface/transformers/blob/master/examples/question-answering/run_squad.py>`__ script.
|
||||
|
|
|
|||
|
|
@ -29,11 +29,12 @@ methods for using all the tokenizers:
|
|||
|
||||
:class:`~transformers.BatchEncoding` holds the output of the tokenizer's encoding methods (``__call__``,
|
||||
``encode_plus`` and ``batch_encode_plus``) and is derived from a Python dictionary. When the tokenizer is a pure python
|
||||
tokenizer, this class behaves just like a standard python dictionary and holds the various model inputs computed by these
|
||||
methods (``input_ids``, ``attention_mask``...). When the tokenizer is a "Fast" tokenizer (i.e., backed by HuggingFace
|
||||
`tokenizers library <https://github.com/huggingface/tokenizers>`__), this class provides in addition several advanced
|
||||
alignment methods which can be used to map between the original string (character and words) and the token space (e.g.,
|
||||
getting the index of the token comprising a given character or the span of characters corresponding to a given token).
|
||||
tokenizer, this class behaves just like a standard python dictionary and holds the various model inputs computed by
|
||||
these methods (``input_ids``, ``attention_mask``...). When the tokenizer is a "Fast" tokenizer (i.e., backed by
|
||||
HuggingFace `tokenizers library <https://github.com/huggingface/tokenizers>`__), this class provides in addition
|
||||
several advanced alignment methods which can be used to map between the original string (character and words) and the
|
||||
token space (e.g., getting the index of the token comprising a given character or the span of characters corresponding
|
||||
to a given token).
|
||||
|
||||
|
||||
PreTrainedTokenizer
|
||||
|
|
|
|||
|
|
@ -4,7 +4,7 @@ Trainer
|
|||
The :class:`~transformers.Trainer` and :class:`~transformers.TFTrainer` classes provide an API for feature-complete
|
||||
training in most standard use cases. It's used in most of the :doc:`example scripts <../examples>`.
|
||||
|
||||
Before instantiating your :class:`~transformers.Trainer`/:class:`~transformers.TFTrainer`, create a
|
||||
Before instantiating your :class:`~transformers.Trainer`/:class:`~transformers.TFTrainer`, create a
|
||||
:class:`~transformers.TrainingArguments`/:class:`~transformers.TFTrainingArguments` to access all the points of
|
||||
customization during training.
|
||||
|
||||
|
|
|
|||
|
|
@ -19,14 +19,14 @@ downstream tasks. However, at some point further model increases become harder d
|
|||
longer training times, and unexpected model degradation. To address these problems, we present two parameter-reduction
|
||||
techniques to lower memory consumption and increase the training speed of BERT. Comprehensive empirical evidence shows
|
||||
that our proposed methods lead to models that scale much better compared to the original BERT. We also use a
|
||||
self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream
|
||||
tasks with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE,
|
||||
RACE, and SQuAD benchmarks while having fewer parameters compared to BERT-large.*
|
||||
self-supervised loss that focuses on modeling inter-sentence coherence, and show it consistently helps downstream tasks
|
||||
with multi-sentence inputs. As a result, our best model establishes new state-of-the-art results on the GLUE, RACE, and
|
||||
SQuAD benchmarks while having fewer parameters compared to BERT-large.*
|
||||
|
||||
Tips:
|
||||
|
||||
- ALBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on
|
||||
the right rather than the left.
|
||||
- ALBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather
|
||||
than the left.
|
||||
- ALBERT uses repeating layers which results in a small memory footprint, however the computational cost remains
|
||||
similar to a BERT-like architecture with the same number of hidden layers as it has to iterate through the same
|
||||
number of (repeating) layers.
|
||||
|
|
|
|||
|
|
@ -2,9 +2,8 @@ AutoClasses
|
|||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
In many cases, the architecture you want to use can be guessed from the name or the path of the pretrained model you
|
||||
are supplying to the :obj:`from_pretrained()` method.
|
||||
AutoClasses are here to do this job for you so that you automatically retrieve the relevant model given the name/path
|
||||
to the pretrained weights/config/vocabulary.
|
||||
are supplying to the :obj:`from_pretrained()` method. AutoClasses are here to do this job for you so that you
|
||||
automatically retrieve the relevant model given the name/path to the pretrained weights/config/vocabulary.
|
||||
|
||||
Instantiating one of :class:`~transformers.AutoConfig`, :class:`~transformers.AutoModel`, and
|
||||
:class:`~transformers.AutoTokenizer` will directly create a class of the relevant architecture. For instance
|
||||
|
|
|
|||
|
|
@ -29,10 +29,10 @@ The Authors' code can be found `here <https://github.com/pytorch/fairseq/tree/ma
|
|||
Implementation Notes
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- Bart doesn't use :obj:`token_type_ids` for sequence classification. Use :class:`~transformers.BartTokenizer`
|
||||
or :meth:`~transformers.BartTokenizer.encode` to get the proper splitting.
|
||||
- Bart doesn't use :obj:`token_type_ids` for sequence classification. Use :class:`~transformers.BartTokenizer` or
|
||||
:meth:`~transformers.BartTokenizer.encode` to get the proper splitting.
|
||||
- The forward pass of :class:`~transformers.BartModel` will create decoder inputs (using the helper function
|
||||
:func:`transformers.modeling_bart._prepare_bart_decoder_inputs`) if they are not passed. This is different than some
|
||||
:func:`transformers.modeling_bart._prepare_bart_decoder_inputs`) if they are not passed. This is different than some
|
||||
other modeling APIs.
|
||||
- Model predictions are intended to be identical to the original implementation. This only works, however, if the
|
||||
string you pass to :func:`fairseq.encode` starts with a space.
|
||||
|
|
|
|||
|
|
@ -25,8 +25,8 @@ improvement) and SQuAD v2.0 Test F1 to 83.1 (5.1 point absolute improvement).*
|
|||
|
||||
Tips:
|
||||
|
||||
- BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on
|
||||
the right rather than the left.
|
||||
- BERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
|
||||
the left.
|
||||
- BERT was trained with the masked language modeling (MLM) and next sentence prediction (NSP) objectives. It is
|
||||
efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation.
|
||||
|
||||
|
|
|
|||
|
|
@ -24,15 +24,15 @@ Usage:
|
|||
- The model can be used in combination with the :class:`~transformers.EncoderDecoderModel` to leverage two pretrained
|
||||
BERT checkpoints for subsequent fine-tuning.
|
||||
|
||||
:: code-block
|
||||
|
||||
.. code-block::
|
||||
|
||||
# leverage checkpoints for Bert2Bert model...
|
||||
# use BERT's cls token as BOS token and sep token as EOS token
|
||||
encoder = BertGenerationEncoder.from_pretrained("bert-large-uncased", bos_token_id=101, eos_token_id=102)
|
||||
# add cross attention layers and use BERT's cls token as BOS token and sep token as EOS token
|
||||
decoder = BertGenerationDecoder.from_pretrained("bert-large-uncased", add_cross_attention=True, is_decoder=True, bos_token_id=101, eos_token_id=102)
|
||||
bert2bert = EncoderDecoderModel(encoder=encoder, decoder=decoder)
|
||||
|
||||
|
||||
# create tokenizer...
|
||||
tokenizer = BertTokenizer.from_pretrained("bert-large-uncased")
|
||||
|
||||
|
|
|
|||
|
|
@ -1,16 +1,28 @@
|
|||
Blenderbot
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
**DISCLAIMER:** If you see something strange,
|
||||
file a `Github Issue <https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title>`__ .
|
||||
|
||||
**DISCLAIMER:** If you see something strange, file a `Github Issue
|
||||
<https://github.com/huggingface/transformers/issues/new?assignees=&labels=&template=bug-report.md&title>`__ .
|
||||
|
||||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The Blender chatbot model was proposed in `Recipes for building an open-domain chatbot <https://arxiv.org/pdf/2004.13637.pdf>`__ Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston on 30 Apr 2020.
|
||||
The Blender chatbot model was proposed in `Recipes for building an open-domain chatbot
|
||||
<https://arxiv.org/pdf/2004.13637.pdf>`__ Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu,
|
||||
Jing Xu, Myle Ott, Kurt Shuster, Eric M. Smith, Y-Lan Boureau, Jason Weston on 30 Apr 2020.
|
||||
|
||||
The abstract of the paper is the following:
|
||||
|
||||
*Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that scaling neural models in the number of parameters and the size of the data they are trained on gives improved results, we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to their partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent persona. We show that large scale models can learn these skills when given appropriate training data and choice of generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models and code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing failure cases of our models.*
|
||||
*Building open-domain chatbots is a challenging area for machine learning research. While prior work has shown that
|
||||
scaling neural models in the number of parameters and the size of the data they are trained on gives improved results,
|
||||
we show that other ingredients are important for a high-performing chatbot. Good conversation requires a number of
|
||||
skills that an expert conversationalist blends in a seamless way: providing engaging talking points and listening to
|
||||
their partners, and displaying knowledge, empathy and personality appropriately, while maintaining a consistent
|
||||
persona. We show that large scale models can learn these skills when given appropriate training data and choice of
|
||||
generation strategy. We build variants of these recipes with 90M, 2.7B and 9.4B parameter models, and make our models
|
||||
and code publicly available. Human evaluations show our best models are superior to existing approaches in multi-turn
|
||||
dialogue in terms of engagingness and humanness measurements. We then discuss the limitations of this work by analyzing
|
||||
failure cases of our models.*
|
||||
|
||||
The authors' code can be found `here <https://github.com/facebookresearch/ParlAI>`__ .
|
||||
|
||||
|
|
@ -20,8 +32,11 @@ Implementation Notes
|
|||
|
||||
- Blenderbot uses a standard `seq2seq model transformer <https://arxiv.org/pdf/1706.03762.pdf>`__ based architecture.
|
||||
- It inherits completely from :class:`~transformers.BartForConditionalGeneration`
|
||||
- Even though blenderbot is one model, it uses two tokenizers :class:`~transformers.BlenderbotSmallTokenizer` for 90M checkpoint and :class:`~transformers.BlenderbotTokenizer` for all other checkpoints.
|
||||
- :class:`~transformers.BlenderbotSmallTokenizer` will always return :class:`~transformers.BlenderbotSmallTokenizer`, regardless of checkpoint. To use the 3B parameter checkpoint, you must call :class:`~transformers.BlenderbotTokenizer` directly.
|
||||
- Even though blenderbot is one model, it uses two tokenizers :class:`~transformers.BlenderbotSmallTokenizer` for 90M
|
||||
checkpoint and :class:`~transformers.BlenderbotTokenizer` for all other checkpoints.
|
||||
- :class:`~transformers.BlenderbotSmallTokenizer` will always return :class:`~transformers.BlenderbotSmallTokenizer`,
|
||||
regardless of checkpoint. To use the 3B parameter checkpoint, you must call
|
||||
:class:`~transformers.BlenderbotTokenizer` directly.
|
||||
- Available checkpoints can be found in the `model hub <https://huggingface.co/models?search=blenderbot>`__.
|
||||
|
||||
|
||||
|
|
@ -56,6 +71,7 @@ Here is how you can check out config values:
|
|||
|
||||
BlenderbotConfig
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.BlenderbotConfig
|
||||
:members:
|
||||
|
||||
|
|
@ -74,6 +90,7 @@ BlenderbotSmallTokenizer
|
|||
|
||||
BlenderbotForConditionalGeneration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
See :obj:`transformers.BartForConditionalGeneration` for arguments to `forward` and `generate`
|
||||
|
||||
.. autoclass:: transformers.BlenderbotForConditionalGeneration
|
||||
|
|
|
|||
|
|
@ -4,26 +4,26 @@ CamemBERT
|
|||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The CamemBERT model was proposed in `CamemBERT: a Tasty French Language Model <https://arxiv.org/abs/1911.03894>`__
|
||||
by Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la
|
||||
The CamemBERT model was proposed in `CamemBERT: a Tasty French Language Model <https://arxiv.org/abs/1911.03894>`__ by
|
||||
Louis Martin, Benjamin Muller, Pedro Javier Ortiz Suárez, Yoann Dupont, Laurent Romary, Éric Villemonte de la
|
||||
Clergerie, Djamé Seddah, and Benoît Sagot. It is based on Facebook's RoBERTa model released in 2019. It is a model
|
||||
trained on 138GB of French text.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success,
|
||||
most available models have either been trained on English data or on the concatenation of data in multiple
|
||||
languages. This makes practical use of such models --in all languages except English-- very limited. Aiming
|
||||
to address this issue for French, we release CamemBERT, a French version of the Bi-directional Encoders for
|
||||
Transformers (BERT). We measure the performance of CamemBERT compared to multilingual models in multiple
|
||||
downstream tasks, namely part-of-speech tagging, dependency parsing, named-entity recognition, and natural
|
||||
language inference. CamemBERT improves the state of the art for most of the tasks considered. We release the
|
||||
pretrained model for CamemBERT hoping to foster research and downstream applications for French NLP.*
|
||||
*Pretrained language models are now ubiquitous in Natural Language Processing. Despite their success, most available
|
||||
models have either been trained on English data or on the concatenation of data in multiple languages. This makes
|
||||
practical use of such models --in all languages except English-- very limited. Aiming to address this issue for French,
|
||||
we release CamemBERT, a French version of the Bi-directional Encoders for Transformers (BERT). We measure the
|
||||
performance of CamemBERT compared to multilingual models in multiple downstream tasks, namely part-of-speech tagging,
|
||||
dependency parsing, named-entity recognition, and natural language inference. CamemBERT improves the state of the art
|
||||
for most of the tasks considered. We release the pretrained model for CamemBERT hoping to foster research and
|
||||
downstream applications for French NLP.*
|
||||
|
||||
Tips:
|
||||
|
||||
- This implementation is the same as RoBERTa. Refer to the :doc:`documentation of RoBERTa <roberta>` for usage
|
||||
examples as well as the information relative to the inputs and outputs.
|
||||
- This implementation is the same as RoBERTa. Refer to the :doc:`documentation of RoBERTa <roberta>` for usage examples
|
||||
as well as the information relative to the inputs and outputs.
|
||||
|
||||
The original code can be found `here <https://camembert-model.fr/>`__.
|
||||
|
||||
|
|
@ -130,4 +130,4 @@ TFCamembertForQuestionAnswering
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.TFCamembertForQuestionAnswering
|
||||
:members:
|
||||
:members:
|
||||
|
|
|
|||
|
|
@ -6,33 +6,33 @@ Overview
|
|||
|
||||
CTRL model was proposed in `CTRL: A Conditional Transformer Language Model for Controllable Generation
|
||||
<https://arxiv.org/abs/1909.05858>`_ by Nitish Shirish Keskar*, Bryan McCann*, Lav R. Varshney, Caiming Xiong and
|
||||
Richard Socher. It's a causal (unidirectional) transformer pre-trained using language modeling on a very large
|
||||
corpus of ~140 GB of text data with the first token reserved as a control code (such as Links, Books, Wikipedia etc.).
|
||||
Richard Socher. It's a causal (unidirectional) transformer pre-trained using language modeling on a very large corpus
|
||||
of ~140 GB of text data with the first token reserved as a control code (such as Links, Books, Wikipedia etc.).
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Large-scale language models show promising text generation capabilities, but users cannot easily control particular
|
||||
aspects of the generated text. We release CTRL, a 1.63 billion-parameter conditional transformer language model,
|
||||
trained to condition on control codes that govern style, content, and task-specific behavior. Control codes were
|
||||
derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning
|
||||
while providing more explicit control over text generation. These codes also allow CTRL to predict which parts of
|
||||
the training data are most likely given a sequence. This provides a potential method for analyzing large amounts
|
||||
of data via model-based source attribution.*
|
||||
derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while
|
||||
providing more explicit control over text generation. These codes also allow CTRL to predict which parts of the
|
||||
training data are most likely given a sequence. This provides a potential method for analyzing large amounts of data
|
||||
via model-based source attribution.*
|
||||
|
||||
Tips:
|
||||
|
||||
- CTRL makes use of control codes to generate text: it requires generations to be started by certain words, sentences
|
||||
or links to generate coherent text. Refer to the `original implementation <https://github.com/salesforce/ctrl>`__
|
||||
for more information.
|
||||
- CTRL is a model with absolute position embeddings so it's usually advised to pad the inputs on
|
||||
the right rather than the left.
|
||||
or links to generate coherent text. Refer to the `original implementation <https://github.com/salesforce/ctrl>`__ for
|
||||
more information.
|
||||
- CTRL is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
|
||||
the left.
|
||||
- CTRL was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
|
||||
token in a sequence. Leveraging this feature allows CTRL to generate syntactically coherent text as
|
||||
it can be observed in the `run_generation.py` example script.
|
||||
token in a sequence. Leveraging this feature allows CTRL to generate syntactically coherent text as it can be
|
||||
observed in the `run_generation.py` example script.
|
||||
- The PyTorch models can take the `past` as input, which is the previously computed key/value attention pairs. Using
|
||||
this `past` value prevents the model from re-computing pre-computed values in the context of text generation.
|
||||
See `reusing the past in generative models <../quickstart.html#using-the-past>`__ for more information on the usage
|
||||
of this argument.
|
||||
this `past` value prevents the model from re-computing pre-computed values in the context of text generation. See
|
||||
`reusing the past in generative models <../quickstart.html#using-the-past>`__ for more information on the usage of
|
||||
this argument.
|
||||
|
||||
The original code can be found `here <https://github.com/salesforce/ctrl>`__.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,40 +1,43 @@
|
|||
DeBERTa
|
||||
----------------------------------------------------
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The DeBERTa model was proposed in `DeBERTa: Decoding-enhanced BERT with Disentangled Attention <https://arxiv.org/abs/2006.03654>`__
|
||||
by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen
|
||||
It is based on Google's BERT model released in 2018 and Facebook's RoBERTa model released in 2019.
|
||||
The DeBERTa model was proposed in `DeBERTa: Decoding-enhanced BERT with Disentangled Attention
|
||||
<https://arxiv.org/abs/2006.03654>`__ by Pengcheng He, Xiaodong Liu, Jianfeng Gao, Weizhu Chen It is based on Google's
|
||||
BERT model released in 2018 and Facebook's RoBERTa model released in 2019.
|
||||
|
||||
It builds on RoBERTa with disentangled attention and enhanced mask decoder training with half of the data used in RoBERTa.
|
||||
It builds on RoBERTa with disentangled attention and enhanced mask decoder training with half of the data used in
|
||||
RoBERTa.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Recent progress in pre-trained neural language models has significantly improved the performance of many natural language processing (NLP) tasks.
|
||||
In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with disentangled attention) that improves the BERT and RoBERTa
|
||||
models using two novel techniques. The first is the disentangled attention mechanism, where each word is represented using two vectors that encode
|
||||
its content and position, respectively, and the attention weights among words are computed using disentangled matrices on their contents and
|
||||
relative positions. Second, an enhanced mask decoder is used to replace the output softmax layer to predict the masked tokens for model pretraining.
|
||||
We show that these two techniques significantly improve the efficiency of model pre-training and performance of downstream tasks. Compared to
|
||||
RoBERTa-Large, a DeBERTa model trained on half of the training data performs consistently better on a wide range of NLP tasks, achieving improvements
|
||||
on MNLI by +0.9% (90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). The DeBERTa code and pre-trained
|
||||
models will be made publicly available at https://github.com/microsoft/DeBERTa.*
|
||||
*Recent progress in pre-trained neural language models has significantly improved the performance of many natural
|
||||
language processing (NLP) tasks. In this paper we propose a new model architecture DeBERTa (Decoding-enhanced BERT with
|
||||
disentangled attention) that improves the BERT and RoBERTa models using two novel techniques. The first is the
|
||||
disentangled attention mechanism, where each word is represented using two vectors that encode its content and
|
||||
position, respectively, and the attention weights among words are computed using disentangled matrices on their
|
||||
contents and relative positions. Second, an enhanced mask decoder is used to replace the output softmax layer to
|
||||
predict the masked tokens for model pretraining. We show that these two techniques significantly improve the efficiency
|
||||
of model pre-training and performance of downstream tasks. Compared to RoBERTa-Large, a DeBERTa model trained on half
|
||||
of the training data performs consistently better on a wide range of NLP tasks, achieving improvements on MNLI by +0.9%
|
||||
(90.2% vs. 91.1%), on SQuAD v2.0 by +2.3% (88.4% vs. 90.7%) and RACE by +3.6% (83.2% vs. 86.8%). The DeBERTa code and
|
||||
pre-trained models will be made publicly available at https://github.com/microsoft/DeBERTa.*
|
||||
|
||||
|
||||
The original code can be found `here <https://github.com/microsoft/DeBERTa>`__.
|
||||
|
||||
|
||||
DebertaConfig
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.DebertaConfig
|
||||
:members:
|
||||
|
||||
|
||||
DebertaTokenizer
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.DebertaTokenizer
|
||||
:members: build_inputs_with_special_tokens, get_special_tokens_mask,
|
||||
|
|
@ -42,21 +45,21 @@ DebertaTokenizer
|
|||
|
||||
|
||||
DebertaModel
|
||||
~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.DebertaModel
|
||||
:members:
|
||||
|
||||
|
||||
DebertaPreTrainedModel
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.DebertaPreTrainedModel
|
||||
:members:
|
||||
|
||||
|
||||
DebertaForSequenceClassification
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.DebertaForSequenceClassification
|
||||
:members:
|
||||
|
|
|
|||
|
|
@ -4,36 +4,39 @@ DialoGPT
|
|||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
DialoGPT was proposed in
|
||||
`DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation <https://arxiv.org/abs/1911.00536>`_
|
||||
by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, Bill Dolan.
|
||||
It's a GPT2 Model trained on 147M conversation-like exchanges extracted from Reddit.
|
||||
DialoGPT was proposed in `DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generation
|
||||
<https://arxiv.org/abs/1911.00536>`_ by Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao,
|
||||
Jianfeng Gao, Jingjing Liu, Bill Dolan. It's a GPT2 Model trained on 147M conversation-like exchanges extracted from
|
||||
Reddit.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained transformer).
|
||||
Trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning from 2005 through 2017, DialoGPT extends the Hugging Face PyTorch transformer to attain a performance close to human both in terms of automatic and human evaluation in single-turn dialogue settings.
|
||||
We show that conversational systems that leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline systems.
|
||||
The pre-trained model and training pipeline are publicly released to facilitate research into neural response generation and the development of more intelligent open-domain dialogue systems.*
|
||||
*We present a large, tunable neural conversational response generation model, DialoGPT (dialogue generative pre-trained
|
||||
transformer). Trained on 147M conversation-like exchanges extracted from Reddit comment chains over a period spanning
|
||||
from 2005 through 2017, DialoGPT extends the Hugging Face PyTorch transformer to attain a performance close to human
|
||||
both in terms of automatic and human evaluation in single-turn dialogue settings. We show that conversational systems
|
||||
that leverage DialoGPT generate more relevant, contentful and context-consistent responses than strong baseline
|
||||
systems. The pre-trained model and training pipeline are publicly released to facilitate research into neural response
|
||||
generation and the development of more intelligent open-domain dialogue systems.*
|
||||
|
||||
Tips:
|
||||
|
||||
- DialoGPT is a model with absolute position embeddings so it's usually advised to pad the inputs on
|
||||
the right rather than the left.
|
||||
- DialoGPT was trained with a causal language modeling (CLM) objective on conversational data and is therefore powerful at response generation in open-domain dialogue systems.
|
||||
- DialoGPT enables the user to create a chat bot in just 10 lines of code as shown on `DialoGPT's model card <https://huggingface.co/microsoft/DialoGPT-medium>`_.
|
||||
- DialoGPT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather
|
||||
than the left.
|
||||
- DialoGPT was trained with a causal language modeling (CLM) objective on conversational data and is therefore powerful
|
||||
at response generation in open-domain dialogue systems.
|
||||
- DialoGPT enables the user to create a chat bot in just 10 lines of code as shown on `DialoGPT's model card
|
||||
<https://huggingface.co/microsoft/DialoGPT-medium>`_.
|
||||
|
||||
Training:
|
||||
|
||||
In order to train or fine-tune DialoGPT, one can use causal language modeling training.
|
||||
To cite the official paper:
|
||||
*We follow the OpenAI GPT-2 to model a multiturn dialogue session
|
||||
as a long text and frame the generation task as language modeling. We first
|
||||
concatenate all dialog turns within a dialogue session into a long text
|
||||
x_1,..., x_N (N is the sequence length), ended by the end-of-text token.*
|
||||
For more information please confer to the original paper.
|
||||
|
||||
In order to train or fine-tune DialoGPT, one can use causal language modeling training. To cite the official paper: *We
|
||||
follow the OpenAI GPT-2 to model a multiturn dialogue session as a long text and frame the generation task as language
|
||||
modeling. We first concatenate all dialog turns within a dialogue session into a long text x_1,..., x_N (N is the
|
||||
sequence length), ended by the end-of-text token.* For more information please confer to the original paper.
|
||||
|
||||
DialoGPT's architecture is based on the GPT2 model, so one can refer to GPT2's `docstring <https://huggingface.co/transformers/model_doc/gpt2.html>`_.
|
||||
|
||||
DialoGPT's architecture is based on the GPT2 model, so one can refer to GPT2's `docstring
|
||||
<https://huggingface.co/transformers/model_doc/gpt2.html>`_.
|
||||
|
||||
The original code can be found `here <https://github.com/microsoft/DialoGPT>`_.
|
||||
|
|
|
|||
|
|
@ -4,13 +4,12 @@ DistilBERT
|
|||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The DistilBERT model was proposed in the blog post
|
||||
`Smaller, faster, cheaper, lighter: Introducing DistilBERT, a distilled version of BERT
|
||||
<https://medium.com/huggingface/distilbert-8cf3380435b5>`__, and the paper `DistilBERT, a distilled version of BERT:
|
||||
smaller, faster, cheaper and lighter <https://arxiv.org/abs/1910.01108>`__.
|
||||
DistilBERT is a small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less
|
||||
parameters than `bert-base-uncased`, runs 60% faster while preserving over 95% of BERT's performances as measured on
|
||||
the GLUE language understanding benchmark.
|
||||
The DistilBERT model was proposed in the blog post `Smaller, faster, cheaper, lighter: Introducing DistilBERT, a
|
||||
distilled version of BERT <https://medium.com/huggingface/distilbert-8cf3380435b5>`__, and the paper `DistilBERT, a
|
||||
distilled version of BERT: smaller, faster, cheaper and lighter <https://arxiv.org/abs/1910.01108>`__. DistilBERT is a
|
||||
small, fast, cheap and light Transformer model trained by distilling BERT base. It has 40% less parameters than
|
||||
`bert-base-uncased`, runs 60% faster while preserving over 95% of BERT's performances as measured on the GLUE language
|
||||
understanding benchmark.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
|
|
@ -18,13 +17,13 @@ The abstract from the paper is the following:
|
|||
operating these large models in on-the-edge and/or under constrained computational training or inference budgets
|
||||
remains challenging. In this work, we propose a method to pre-train a smaller general-purpose language representation
|
||||
model, called DistilBERT, which can then be fine-tuned with good performances on a wide range of tasks like its larger
|
||||
counterparts. While most prior work investigated the use of distillation for building task-specific models, we
|
||||
leverage knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a
|
||||
BERT model by 40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage
|
||||
the inductive biases learned by larger models during pre-training, we introduce a triple loss combining language
|
||||
modeling, distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train
|
||||
and we demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative
|
||||
on-device study.*
|
||||
counterparts. While most prior work investigated the use of distillation for building task-specific models, we leverage
|
||||
knowledge distillation during the pre-training phase and show that it is possible to reduce the size of a BERT model by
|
||||
40%, while retaining 97% of its language understanding capabilities and being 60% faster. To leverage the inductive
|
||||
biases learned by larger models during pre-training, we introduce a triple loss combining language modeling,
|
||||
distillation and cosine-distance losses. Our smaller, faster and lighter model is cheaper to pre-train and we
|
||||
demonstrate its capabilities for on-device computations in a proof-of-concept experiment and a comparative on-device
|
||||
study.*
|
||||
|
||||
Tips:
|
||||
|
||||
|
|
@ -33,7 +32,8 @@ Tips:
|
|||
- DistilBERT doesn't have options to select the input positions (:obj:`position_ids` input). This could be added if
|
||||
necessary though, just let us know if you need this option.
|
||||
|
||||
The original code can be found `here <https://github.com/huggingface/transformers/tree/master/examples/distillation>`__.
|
||||
The original code can be found `here
|
||||
<https://github.com/huggingface/transformers/tree/master/examples/distillation>`__.
|
||||
|
||||
|
||||
DistilBertConfig
|
||||
|
|
|
|||
|
|
@ -4,9 +4,9 @@ DPR
|
|||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Dense Passage Retrieval (DPR) is a set of tools and models for state-of-the-art open-domain Q&A research.
|
||||
It was intorduced in `Dense Passage Retrieval for Open-Domain Question Answering <https://arxiv.org/abs/2004.04906>`__
|
||||
by Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih.
|
||||
Dense Passage Retrieval (DPR) is a set of tools and models for state-of-the-art open-domain Q&A research. It was
|
||||
intorduced in `Dense Passage Retrieval for Open-Domain Question Answering <https://arxiv.org/abs/2004.04906>`__ by
|
||||
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, Wen-tau Yih.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
|
|
|
|||
|
|
@ -12,34 +12,28 @@ identify which tokens were replaced by the generator in the sequence.
|
|||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Masked language modeling (MLM) pre-training methods such as BERT corrupt
|
||||
the input by replacing some tokens with [MASK] and then train a model to
|
||||
reconstruct the original tokens. While they produce good results when transferred
|
||||
to downstream NLP tasks, they generally require large amounts of compute to be
|
||||
effective. As an alternative, we propose a more sample-efficient pre-training task
|
||||
called replaced token detection. Instead of masking the input, our approach
|
||||
corrupts it by replacing some tokens with plausible alternatives sampled from a small
|
||||
generator network. Then, instead of training a model that predicts the original
|
||||
identities of the corrupted tokens, we train a discriminative model that predicts
|
||||
whether each token in the corrupted input was replaced by a generator sample
|
||||
or not. Thorough experiments demonstrate this new pre-training task is more
|
||||
efficient than MLM because the task is defined over all input tokens rather than
|
||||
just the small subset that was masked out. As a result, the contextual representations
|
||||
learned by our approach substantially outperform the ones learned by BERT
|
||||
given the same model size, data, and compute. The gains are particularly strong
|
||||
for small models; for example, we train a model on one GPU for 4 days that
|
||||
outperforms GPT (trained using 30x more compute) on the GLUE natural language
|
||||
understanding benchmark. Our approach also works well at scale, where it
|
||||
performs comparably to RoBERTa and XLNet while using less than 1/4 of their
|
||||
compute and outperforms them when using the same amount of compute.*
|
||||
*Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with
|
||||
[MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to
|
||||
downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a
|
||||
more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach
|
||||
corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead
|
||||
of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that
|
||||
predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments
|
||||
demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens
|
||||
rather than just the small subset that was masked out. As a result, the contextual representations learned by our
|
||||
approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are
|
||||
particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained
|
||||
using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale,
|
||||
where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when
|
||||
using the same amount of compute.*
|
||||
|
||||
Tips:
|
||||
|
||||
- ELECTRA is the pretraining approach, therefore there is nearly no changes done to the underlying model: BERT. The
|
||||
only change is the separation of the embedding size and the hidden size: the embedding size is generally smaller,
|
||||
while the hidden size is larger. An additional projection layer (linear) is used to project the embeddings from
|
||||
their embedding size to the hidden size. In the case where the embedding size is the same as the hidden size, no
|
||||
projection layer is used.
|
||||
while the hidden size is larger. An additional projection layer (linear) is used to project the embeddings from their
|
||||
embedding size to the hidden size. In the case where the embedding size is the same as the hidden size, no projection
|
||||
layer is used.
|
||||
- The ELECTRA checkpoints saved using `Google Research's implementation <https://github.com/google-research/electra>`__
|
||||
contain both the generator and discriminator. The conversion script requires the user to name which model to export
|
||||
into the correct architecture. Once converted to the HuggingFace format, these checkpoints may be loaded into all
|
||||
|
|
|
|||
|
|
@ -13,7 +13,7 @@ any other models (see the examples for more information).
|
|||
|
||||
An application of this architecture could be to leverage two pretrained :class:`~transformers.BertModel` as the encoder
|
||||
and decoder for a summarization model as was shown in: `Text Summarization with Pretrained Encoders
|
||||
<https://arxiv.org/abs/1908.08345>`__ by Yang Liu and Mirella Lapata.
|
||||
<https://arxiv.org/abs/1908.08345>`__ by Yang Liu and Mirella Lapata.
|
||||
|
||||
|
||||
EncoderDecoderConfig
|
||||
|
|
|
|||
|
|
@ -11,17 +11,17 @@ modeling (MLM) objective (like BERT).
|
|||
The abstract from the paper is the following:
|
||||
|
||||
*Language models have become a key step to achieve state-of-the art results in many different Natural Language
|
||||
Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient
|
||||
way to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their
|
||||
Processing (NLP) tasks. Leveraging the huge amount of unlabeled texts nowadays available, they provide an efficient way
|
||||
to pre-train continuous word representations that can be fine-tuned for a downstream task, along with their
|
||||
contextualization at the sentence level. This has been widely demonstrated for English using contextualized
|
||||
representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et
|
||||
al., 2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large
|
||||
and heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre
|
||||
for Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text
|
||||
classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most
|
||||
of the time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified
|
||||
evaluation protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared
|
||||
to the research community for further reproducible experiments in French NLP.*
|
||||
representations (Dai and Le, 2015; Peters et al., 2018; Howard and Ruder, 2018; Radford et al., 2018; Devlin et al.,
|
||||
2019; Yang et al., 2019b). In this paper, we introduce and share FlauBERT, a model learned on a very large and
|
||||
heterogeneous French corpus. Models of different sizes are trained using the new CNRS (French National Centre for
|
||||
Scientific Research) Jean Zay supercomputer. We apply our French language models to diverse NLP tasks (text
|
||||
classification, paraphrasing, natural language inference, parsing, word sense disambiguation) and show that most of the
|
||||
time they outperform other pre-training approaches. Different versions of FlauBERT as well as a unified evaluation
|
||||
protocol for the downstream tasks, called FLUE (French Language Understanding Evaluation), are shared to the research
|
||||
community for further reproducible experiments in French NLP.*
|
||||
|
||||
The original code can be found `here <https://github.com/getalp/Flaubert>`__.
|
||||
|
||||
|
|
|
|||
|
|
@ -58,4 +58,4 @@ FSMTForConditionalGeneration
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.FSMTForConditionalGeneration
|
||||
:members: forward
|
||||
:members: forward
|
||||
|
|
|
|||
|
|
@ -30,8 +30,8 @@ Tips:
|
|||
directly for tasks that just require a sentence summary (like sequence classification or multiple choice). For other
|
||||
tasks, the full model is used; this full model has a decoder that upsamples the final hidden states to the same
|
||||
sequence length as the input.
|
||||
- The Funnel Transformer checkpoints are all available with a full version and a base version. The first ones should
|
||||
be used for :class:`~transformers.FunnelModel`, :class:`~transformers.FunnelForPreTraining`,
|
||||
- The Funnel Transformer checkpoints are all available with a full version and a base version. The first ones should be
|
||||
used for :class:`~transformers.FunnelModel`, :class:`~transformers.FunnelForPreTraining`,
|
||||
:class:`~transformers.FunnelForMaskedLM`, :class:`~transformers.FunnelForTokenClassification` and
|
||||
class:`~transformers.FunnelForQuestionAnswering`. The second ones should be used for
|
||||
:class:`~transformers.FunnelBaseModel`, :class:`~transformers.FunnelForSequenceClassification` and
|
||||
|
|
|
|||
|
|
@ -6,44 +6,39 @@ Overview
|
|||
|
||||
OpenAI GPT model was proposed in `Improving Language Understanding by Generative Pre-Training
|
||||
<https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf>`__
|
||||
by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. It's a causal (unidirectional)
|
||||
transformer pre-trained using language modeling on a large corpus will long range dependencies, the Toronto Book
|
||||
Corpus.
|
||||
by Alec Radford, Karthik Narasimhan, Tim Salimans and Ilya Sutskever. It's a causal (unidirectional) transformer
|
||||
pre-trained using language modeling on a large corpus will long range dependencies, the Toronto Book Corpus.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Natural language understanding comprises a wide range of diverse tasks such
|
||||
as textual entailment, question answering, semantic similarity assessment, and
|
||||
document classification. Although large unlabeled text corpora are abundant,
|
||||
labeled data for learning these specific tasks is scarce, making it challenging for
|
||||
discriminatively trained models to perform adequately. We demonstrate that large
|
||||
gains on these tasks can be realized by generative pre-training of a language model
|
||||
on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each
|
||||
specific task. In contrast to previous approaches, we make use of task-aware input
|
||||
transformations during fine-tuning to achieve effective transfer while requiring
|
||||
minimal changes to the model architecture. We demonstrate the effectiveness of
|
||||
our approach on a wide range of benchmarks for natural language understanding.
|
||||
Our general task-agnostic model outperforms discriminatively trained models that
|
||||
use architectures specifically crafted for each task, significantly improving upon the
|
||||
state of the art in 9 out of the 12 tasks studied.*
|
||||
*Natural language understanding comprises a wide range of diverse tasks such as textual entailment, question answering,
|
||||
semantic similarity assessment, and document classification. Although large unlabeled text corpora are abundant,
|
||||
labeled data for learning these specific tasks is scarce, making it challenging for discriminatively trained models to
|
||||
perform adequately. We demonstrate that large gains on these tasks can be realized by generative pre-training of a
|
||||
language model on a diverse corpus of unlabeled text, followed by discriminative fine-tuning on each specific task. In
|
||||
contrast to previous approaches, we make use of task-aware input transformations during fine-tuning to achieve
|
||||
effective transfer while requiring minimal changes to the model architecture. We demonstrate the effectiveness of our
|
||||
approach on a wide range of benchmarks for natural language understanding. Our general task-agnostic model outperforms
|
||||
discriminatively trained models that use architectures specifically crafted for each task, significantly improving upon
|
||||
the state of the art in 9 out of the 12 tasks studied.*
|
||||
|
||||
Tips:
|
||||
|
||||
- GPT is a model with absolute position embeddings so it's usually advised to pad the inputs on
|
||||
the right rather than the left.
|
||||
- GPT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
|
||||
the left.
|
||||
- GPT was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
|
||||
token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as
|
||||
it can be observed in the `run_generation.py` example script.
|
||||
token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be
|
||||
observed in the `run_generation.py` example script.
|
||||
|
||||
`Write With Transformer <https://transformer.huggingface.co/doc/gpt>`__ is a webapp created and hosted by
|
||||
Hugging Face showcasing the generative capabilities of several models. GPT is one of them.
|
||||
`Write With Transformer <https://transformer.huggingface.co/doc/gpt>`__ is a webapp created and hosted by Hugging Face
|
||||
showcasing the generative capabilities of several models. GPT is one of them.
|
||||
|
||||
The original code can be found `here <https://github.com/openai/finetune-transformer-lm>`__.
|
||||
|
||||
Note:
|
||||
|
||||
If you want to reproduce the original tokenization process of the `OpenAI GPT` paper, you will need to install
|
||||
``ftfy`` and ``SpaCy``::
|
||||
If you want to reproduce the original tokenization process of the `OpenAI GPT` paper, you will need to install ``ftfy``
|
||||
and ``SpaCy``::
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -51,8 +46,7 @@ If you want to reproduce the original tokenization process of the `OpenAI GPT` p
|
|||
python -m spacy download en
|
||||
|
||||
If you don't install ``ftfy`` and ``SpaCy``, the :class:`~transformers.OpenAIGPTTokenizer` will default to tokenize
|
||||
using BERT's :obj:`BasicTokenizer` followed by Byte-Pair Encoding (which should be fine for most usage, don't
|
||||
worry).
|
||||
using BERT's :obj:`BasicTokenizer` followed by Byte-Pair Encoding (which should be fine for most usage, don't worry).
|
||||
|
||||
OpenAIGPTConfig
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
|
|||
|
|
@ -5,29 +5,29 @@ Overview
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
OpenAI GPT-2 model was proposed in `Language Models are Unsupervised Multitask Learners
|
||||
<https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf>`_
|
||||
by Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. It's a causal (unidirectional)
|
||||
transformer pretrained using language modeling on a very large corpus of ~40 GB of text data.
|
||||
<https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf>`_ by Alec
|
||||
Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei and Ilya Sutskever. It's a causal (unidirectional)
|
||||
transformer pretrained using language modeling on a very large corpus of ~40 GB of text data.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset[1]
|
||||
of 8 million web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous
|
||||
words within some text. The diversity of the dataset causes this simple goal to contain naturally occurring
|
||||
demonstrations of many tasks across diverse domains. GPT-2 is a direct scale-up of GPT, with more than 10X
|
||||
the parameters and trained on more than 10X the amount of data.*
|
||||
*GPT-2 is a large transformer-based language model with 1.5 billion parameters, trained on a dataset[1] of 8 million
|
||||
web pages. GPT-2 is trained with a simple objective: predict the next word, given all of the previous words within some
|
||||
text. The diversity of the dataset causes this simple goal to contain naturally occurring demonstrations of many tasks
|
||||
across diverse domains. GPT-2 is a direct scale-up of GPT, with more than 10X the parameters and trained on more than
|
||||
10X the amount of data.*
|
||||
|
||||
Tips:
|
||||
|
||||
- GPT-2 is a model with absolute position embeddings so it's usually advised to pad the inputs on
|
||||
the right rather than the left.
|
||||
- GPT-2 is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
|
||||
the left.
|
||||
- GPT-2 was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
|
||||
token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as
|
||||
it can be observed in the `run_generation.py` example script.
|
||||
token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be
|
||||
observed in the `run_generation.py` example script.
|
||||
- The PyTorch models can take the `past` as input, which is the previously computed key/value attention pairs. Using
|
||||
this `past` value prevents the model from re-computing pre-computed values in the context of text generation.
|
||||
See `reusing the past in generative models <../quickstart.html#using-the-past>`__ for more information on the usage
|
||||
of this argument.
|
||||
this `past` value prevents the model from re-computing pre-computed values in the context of text generation. See
|
||||
`reusing the past in generative models <../quickstart.html#using-the-past>`__ for more information on the usage of
|
||||
this argument.
|
||||
|
||||
`Write With Transformer <https://transformer.huggingface.co/doc/gpt2-large>`__ is a webapp created and hosted by
|
||||
Hugging Face showcasing the generative capabilities of several models. GPT-2 is one of them and is available in five
|
||||
|
|
|
|||
|
|
@ -1,55 +1,66 @@
|
|||
LayoutLM
|
||||
----------------------------------------------------
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The LayoutLM model was proposed in the paper `LayoutLM: Pre-training of Text and Layout for Document Image Understanding <https://arxiv.org/abs/1912.13318>`__
|
||||
by Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. It's a simple but effective pre-training method
|
||||
of text and layout for document image understanding and information extraction tasks, such as form understanding and receipt understanding.
|
||||
The LayoutLM model was proposed in the paper `LayoutLM: Pre-training of Text and Layout for Document Image
|
||||
Understanding <https://arxiv.org/abs/1912.13318>`__ by Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and
|
||||
Ming Zhou. It's a simple but effective pre-training method of text and layout for document image understanding and
|
||||
information extraction tasks, such as form understanding and receipt understanding.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Pre-training techniques have been verified successfully in a variety of NLP tasks in recent years. Despite the widespread use of pre-training models for NLP applications, they almost exclusively focus on text-level manipulation, while neglecting layout and style information that is vital for document image understanding. In this paper, we propose the \textbf{LayoutLM} to jointly model interactions between text and layout information across scanned document images, which is beneficial for a great number of real-world document image understanding tasks such as information extraction from scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into LayoutLM. To the best of our knowledge, this is the first time that text and layout are jointly learned in a single framework for document-level pre-training. It achieves new state-of-the-art results in several downstream tasks, including form understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image classification (from 93.07 to 94.42).*
|
||||
*Pre-training techniques have been verified successfully in a variety of NLP tasks in recent years. Despite the
|
||||
widespread use of pre-training models for NLP applications, they almost exclusively focus on text-level manipulation,
|
||||
while neglecting layout and style information that is vital for document image understanding. In this paper, we propose
|
||||
the \textbf{LayoutLM} to jointly model interactions between text and layout information across scanned document images,
|
||||
which is beneficial for a great number of real-world document image understanding tasks such as information extraction
|
||||
from scanned documents. Furthermore, we also leverage image features to incorporate words' visual information into
|
||||
LayoutLM. To the best of our knowledge, this is the first time that text and layout are jointly learned in a single
|
||||
framework for document-level pre-training. It achieves new state-of-the-art results in several downstream tasks,
|
||||
including form understanding (from 70.72 to 79.27), receipt understanding (from 94.02 to 95.24) and document image
|
||||
classification (from 93.07 to 94.42).*
|
||||
|
||||
Tips:
|
||||
|
||||
- LayoutLM has an extra input called :obj:`bbox`, which is the bounding boxes of the input tokens.
|
||||
- The :obj:`bbox` requires the data that on 0-1000 scale, which means you should normalize the bounding box before passing them into model.
|
||||
- The :obj:`bbox` requires the data that on 0-1000 scale, which means you should normalize the bounding box before
|
||||
passing them into model.
|
||||
|
||||
The original code can be found `here <https://github.com/microsoft/unilm/tree/master/layoutlm>`_.
|
||||
|
||||
|
||||
LayoutLMConfig
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.LayoutLMConfig
|
||||
:members:
|
||||
|
||||
|
||||
LayoutLMTokenizer
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.LayoutLMTokenizer
|
||||
:members:
|
||||
|
||||
|
||||
LayoutLMModel
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.LayoutLMModel
|
||||
:members:
|
||||
|
||||
|
||||
LayoutLMForMaskedLM
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.LayoutLMForMaskedLM
|
||||
:members:
|
||||
|
||||
|
||||
LayoutLMForTokenClassification
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.LayoutLMForTokenClassification
|
||||
:members:
|
||||
|
|
|
|||
|
|
@ -27,20 +27,20 @@ The Authors' code can be found `here <https://github.com/allenai/longformer>`__.
|
|||
Longformer Self Attention
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Longformer self attention employs self attention on both a "local" context and a "global" context.
|
||||
Most tokens only attend "locally" to each other meaning that each token attends to its :math:`\frac{1}{2} w` previous
|
||||
tokens and :math:`\frac{1}{2} w` succeding tokens with :math:`w` being the window length as defined in
|
||||
Longformer self attention employs self attention on both a "local" context and a "global" context. Most tokens only
|
||||
attend "locally" to each other meaning that each token attends to its :math:`\frac{1}{2} w` previous tokens and
|
||||
:math:`\frac{1}{2} w` succeding tokens with :math:`w` being the window length as defined in
|
||||
:obj:`config.attention_window`. Note that :obj:`config.attention_window` can be of type :obj:`List` to define a
|
||||
different :math:`w` for each layer. A selected few tokens attend "globally" to all other tokens, as it is
|
||||
conventionally done for all tokens in :obj:`BertSelfAttention`.
|
||||
|
||||
Note that "locally" and "globally" attending tokens are projected by different query, key and value matrices.
|
||||
Also note that every "locally" attending token not only attends to tokens within its window :math:`w`, but also to all
|
||||
"globally" attending tokens so that global attention is *symmetric*.
|
||||
Note that "locally" and "globally" attending tokens are projected by different query, key and value matrices. Also note
|
||||
that every "locally" attending token not only attends to tokens within its window :math:`w`, but also to all "globally"
|
||||
attending tokens so that global attention is *symmetric*.
|
||||
|
||||
The user can define which tokens attend "locally" and which tokens attend "globally" by setting the tensor
|
||||
:obj:`global_attention_mask` at run-time appropriately. All Longformer models employ the following logic for
|
||||
:obj:`global_attention_mask`:
|
||||
:obj:`global_attention_mask`:
|
||||
|
||||
- 0: the token attends "locally",
|
||||
- 1: the token attends "globally".
|
||||
|
|
|
|||
|
|
@ -8,9 +8,8 @@ The LXMERT model was proposed in `LXMERT: Learning Cross-Modality Encoder Repres
|
|||
<https://arxiv.org/abs/1908.07490>`__ by Hao Tan & Mohit Bansal. It is a series of bidirectional transformer encoders
|
||||
(one for the vision modality, one for the language modality, and then one to fuse both modalities) pretrained using a
|
||||
combination of masked language modeling, visual-language text alignment, ROI-feature regression, masked
|
||||
visual-attribute modeling, masked visual-object modeling, and visual-question answering objectives.
|
||||
The pretraining consists of multiple multi-modal datasets: MSCOCO, Visual-Genome + Visual-Genome Question Answering,
|
||||
VQA 2.0, and GQA.
|
||||
visual-attribute modeling, masked visual-object modeling, and visual-question answering objectives. The pretraining
|
||||
consists of multiple multi-modal datasets: MSCOCO, Visual-Genome + Visual-Genome Question Answering, VQA 2.0, and GQA.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
|
|
|
|||
|
|
@ -3,7 +3,7 @@ MarianMT
|
|||
|
||||
**Bugs:** If you see something strange, file a `Github Issue
|
||||
<https://github.com/huggingface/transformers/issues/new?assignees=sshleifer&labels=&template=bug-report.md&title>`__
|
||||
and assign @sshleifer.
|
||||
and assign @sshleifer.
|
||||
|
||||
Translations should be similar, but not identical to, output in the test set linked to in each model card.
|
||||
|
||||
|
|
@ -12,13 +12,14 @@ Implementation Notes
|
|||
|
||||
- Each model is about 298 MB on disk, there are more than 1,000 models.
|
||||
- The list of supported language pairs can be found `here <https://huggingface.co/Helsinki-NLP>`__.
|
||||
- Models were originally trained by
|
||||
`Jörg Tiedemann <https://researchportal.helsinki.fi/en/persons/j%C3%B6rg-tiedemann>`__ using the
|
||||
`Marian <https://marian-nmt.github.io/>`__ C++ library, which supports fast training and translation.
|
||||
- Models were originally trained by `Jörg Tiedemann
|
||||
<https://researchportal.helsinki.fi/en/persons/j%C3%B6rg-tiedemann>`__ using the `Marian
|
||||
<https://marian-nmt.github.io/>`__ C++ library, which supports fast training and translation.
|
||||
- All models are transformer encoder-decoders with 6 layers in each component. Each model's performance is documented
|
||||
in a model card.
|
||||
- The 80 opus models that require BPE preprocessing are not supported.
|
||||
- The modeling code is the same as :class:`~transformers.BartForConditionalGeneration` with a few minor modifications:
|
||||
|
||||
- static (sinusoid) positional embeddings (:obj:`MarianConfig.static_position_embeddings=True`)
|
||||
- a new final_logits_bias (:obj:`MarianConfig.add_bias_logits=True`)
|
||||
- no layernorm_embedding (:obj:`MarianConfig.normalize_embedding=False`)
|
||||
|
|
@ -29,17 +30,17 @@ Implementation Notes
|
|||
Naming
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
- All model names use the following format: :obj:`Helsinki-NLP/opus-mt-{src}-{tgt}`
|
||||
- All model names use the following format: :obj:`Helsinki-NLP/opus-mt-{src}-{tgt}`
|
||||
- The language codes used to name models are inconsistent. Two digit codes can usually be found `here
|
||||
<https://developers.google.com/admin-sdk/directory/v1/languages>`__, three digit codes require googling
|
||||
"language code {code}".
|
||||
<https://developers.google.com/admin-sdk/directory/v1/languages>`__, three digit codes require googling "language
|
||||
code {code}".
|
||||
- Codes formatted like :obj:`es_AR` are usually :obj:`code_{region}`. That one is Spanish from Argentina.
|
||||
|
||||
|
||||
Multilingual Models
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
All model names use the following format: :obj:`Helsinki-NLP/opus-mt-{src}-{tgt}`:
|
||||
All model names use the following format: :obj:`Helsinki-NLP/opus-mt-{src}-{tgt}`:
|
||||
|
||||
- If :obj:`src` is in all caps, the model supports multiple input languages, you can figure out which ones by
|
||||
looking at the model card, or the Group Members `mapping
|
||||
|
|
@ -112,6 +113,7 @@ Code to see available pretrained models:
|
|||
|
||||
MarianConfig
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.MarianConfig
|
||||
:members:
|
||||
|
||||
|
|
|
|||
|
|
@ -7,9 +7,10 @@ MBart
|
|||
|
||||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The MBart model was presented in `Multilingual Denoising Pre-training for Neural Machine Translation
|
||||
<https://arxiv.org/abs/2001.08210>`_ by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov
|
||||
Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.
|
||||
<https://arxiv.org/abs/2001.08210>`_ by Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov Marjan
|
||||
Ghazvininejad, Mike Lewis, Luke Zettlemoyer.
|
||||
|
||||
According to the abstract, MBART is a sequence-to-sequence denoising auto-encoder pretrained on large-scale monolingual
|
||||
corpora in many languages using the BART objective. mBART is one of the first methods for pre-training a complete
|
||||
|
|
@ -21,12 +22,13 @@ The Authors' code can be found `here <https://github.com/pytorch/fairseq/tree/ma
|
|||
|
||||
Training
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
MBart is a multilingual encoder-decoder (seq-to-seq) model primarily intended for translation task.
|
||||
As the model is multilingual it expects the sequences in a different format. A special language id token
|
||||
is added in both the source and target text. The source text format is :obj:`X [eos, src_lang_code]`
|
||||
where :obj:`X` is the source text. The target text format is :obj:`[tgt_lang_code] X [eos]`. :obj:`bos` is never used.
|
||||
|
||||
The :meth:`~transformers.MBartTokenizer.prepare_seq2seq_batch` handles this automatically and should be used to encode
|
||||
MBart is a multilingual encoder-decoder (seq-to-seq) model primarily intended for translation task. As the model is
|
||||
multilingual it expects the sequences in a different format. A special language id token is added in both the source
|
||||
and target text. The source text format is :obj:`X [eos, src_lang_code]` where :obj:`X` is the source text. The target
|
||||
text format is :obj:`[tgt_lang_code] X [eos]`. :obj:`bos` is never used.
|
||||
|
||||
The :meth:`~transformers.MBartTokenizer.prepare_seq2seq_batch` handles this automatically and should be used to encode
|
||||
the sequences for sequence-to-sequence fine-tuning.
|
||||
|
||||
- Supervised training
|
||||
|
|
@ -44,8 +46,8 @@ the sequences for sequence-to-sequence fine-tuning.
|
|||
|
||||
- Generation
|
||||
|
||||
While generating the target text set the :obj:`decoder_start_token_id` to the target language id.
|
||||
The following example shows how to translate English to Romanian using the `facebook/mbart-large-en-ro` model.
|
||||
While generating the target text set the :obj:`decoder_start_token_id` to the target language id. The following
|
||||
example shows how to translate English to Romanian using the `facebook/mbart-large-en-ro` model.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
|
|||
|
|
@ -14,23 +14,23 @@ The abstract from the paper is the following:
|
|||
*Natural Language Processing (NLP) has recently achieved great success by using huge pre-trained models with hundreds
|
||||
of millions of parameters. However, these models suffer from heavy model sizes and high latency such that they cannot
|
||||
be deployed to resource-limited mobile devices. In this paper, we propose MobileBERT for compressing and accelerating
|
||||
the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied
|
||||
to various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while
|
||||
equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward
|
||||
networks. To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated
|
||||
BERT_LARGE model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that
|
||||
MobileBERT is 4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known
|
||||
benchmarks. On the natural language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7
|
||||
(0.6 lower than BERT_BASE), and 62 ms latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task,
|
||||
MobileBERT achieves a dev F1 score of 90.0/79.2 (1.5/2.1 higher than BERT_BASE).*
|
||||
the popular BERT model. Like the original BERT, MobileBERT is task-agnostic, that is, it can be generically applied to
|
||||
various downstream NLP tasks via simple fine-tuning. Basically, MobileBERT is a thin version of BERT_LARGE, while
|
||||
equipped with bottleneck structures and a carefully designed balance between self-attentions and feed-forward networks.
|
||||
To train MobileBERT, we first train a specially designed teacher model, an inverted-bottleneck incorporated BERT_LARGE
|
||||
model. Then, we conduct knowledge transfer from this teacher to MobileBERT. Empirical studies show that MobileBERT is
|
||||
4.3x smaller and 5.5x faster than BERT_BASE while achieving competitive results on well-known benchmarks. On the
|
||||
natural language inference tasks of GLUE, MobileBERT achieves a GLUEscore o 77.7 (0.6 lower than BERT_BASE), and 62 ms
|
||||
latency on a Pixel 4 phone. On the SQuAD v1.1/v2.0 question answering task, MobileBERT achieves a dev F1 score of
|
||||
90.0/79.2 (1.5/2.1 higher than BERT_BASE).*
|
||||
|
||||
Tips:
|
||||
|
||||
- MobileBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on
|
||||
the right rather than the left.
|
||||
- MobileBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective.
|
||||
It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for
|
||||
text generation. Models trained with a causal language modeling (CLM) objective are better in that regard.
|
||||
- MobileBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather
|
||||
than the left.
|
||||
- MobileBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore
|
||||
efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained
|
||||
with a causal language modeling (CLM) objective are better in that regard.
|
||||
|
||||
The original code can be found `here <https://github.com/google-research/mobilebert>`__.
|
||||
|
||||
|
|
|
|||
|
|
@ -9,9 +9,8 @@ and assign @sshleifer.
|
|||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The Pegasus model was proposed in `PEGASUS: Pre-training with Extracted Gap-sentences for
|
||||
Abstractive Summarization <https://arxiv.org/pdf/1912.08777.pdf>`__ by Jingqing Zhang, Yao Zhao, Mohammad Saleh and
|
||||
Peter J. Liu on Dec 18, 2019.
|
||||
The Pegasus model was proposed in `PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization
|
||||
<https://arxiv.org/pdf/1912.08777.pdf>`__ by Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019.
|
||||
|
||||
According to the abstract,
|
||||
|
||||
|
|
@ -26,7 +25,7 @@ The Authors' code can be found `here <https://github.com/google-research/pegasus
|
|||
Checkpoints
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
All the `checkpoints <https://huggingface.co/models?search=pegasus>`__ are fine-tuned for summarization, besides
|
||||
All the `checkpoints <https://huggingface.co/models?search=pegasus>`__ are fine-tuned for summarization, besides
|
||||
`pegasus-large`, whence the other checkpoints are fine-tuned:
|
||||
|
||||
- Each checkpoint is 2.2 GB on disk and 568M parameters.
|
||||
|
|
@ -44,6 +43,7 @@ Implementation Notes
|
|||
- All models are transformer encoder-decoders with 16 layers in each component.
|
||||
- The implementation is completely inherited from :class:`~transformers.BartForConditionalGeneration`
|
||||
- Some key configuration differences:
|
||||
|
||||
- static, sinusoidal position embeddings
|
||||
- no :obj:`layernorm_embedding` (:obj`PegasusConfig.normalize_embedding=False`)
|
||||
- the model starts generating with pad_token_id (which has 0 token_embedding) as the prefix.
|
||||
|
|
@ -84,6 +84,7 @@ PegasusConfig
|
|||
|
||||
PegasusTokenizer
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
warning: ``add_tokens`` does not work at the moment.
|
||||
|
||||
.. autoclass:: transformers.PegasusTokenizer
|
||||
|
|
|
|||
|
|
@ -8,13 +8,24 @@ ProphetNet
|
|||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The ProphetNet model was proposed in `ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training, <https://arxiv.org/abs/2001.04063>`__ by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou on 13 Jan, 2020.
|
||||
The ProphetNet model was proposed in `ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,
|
||||
<https://arxiv.org/abs/2001.04063>`__ by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei
|
||||
Zhang, Ming Zhou on 13 Jan, 2020.
|
||||
|
||||
ProphetNet is an encoder-decoder model and can predict n-future tokens for "ngram" language modeling instead of just the next token.
|
||||
ProphetNet is an encoder-decoder model and can predict n-future tokens for "ngram" language modeling instead of just
|
||||
the next token.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*In this paper, we present a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of the optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale dataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.*
|
||||
*In this paper, we present a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel
|
||||
self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of
|
||||
the optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by
|
||||
n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time
|
||||
step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent
|
||||
overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale
|
||||
dataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for
|
||||
abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new
|
||||
state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.*
|
||||
|
||||
The Authors' code can be found `here <https://github.com/microsoft/ProphetNet>`__.
|
||||
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
RAG
|
||||
----------------------------------------------------
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and
|
||||
sequence-to-sequence models. RAG models retrieve documents, pass them to a seq2seq model, then marginalize to generate
|
||||
|
|
@ -15,46 +15,40 @@ Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäs
|
|||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Large pre-trained language models have been shown to store factual knowledge
|
||||
in their parameters, and achieve state-of-the-art results when fine-tuned on
|
||||
downstream NLP tasks. However, their ability to access and precisely manipulate
|
||||
knowledge is still limited, and hence on knowledge-intensive tasks, their
|
||||
performance lags behind task-specific architectures. Additionally, providing
|
||||
provenance for their decisions and updating their world knowledge remain open
|
||||
research problems. Pre-trained models with a differentiable access mechanism to
|
||||
explicit nonparametric memory can overcome this issue, but have so far been only
|
||||
investigated for extractive downstream tasks. We explore a general-purpose
|
||||
fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine
|
||||
pre-trained parametric and non-parametric memory for language generation. We
|
||||
introduce RAG models where the parametric memory is a pre-trained seq2seq model and
|
||||
the non-parametric memory is a dense vector index of Wikipedia, accessed with
|
||||
a pre-trained neural retriever. We compare two RAG formulations, one which
|
||||
conditions on the same retrieved passages across the whole generated sequence, the
|
||||
other can use different passages per token. We fine-tune and evaluate our models
|
||||
on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art
|
||||
on three open domain QA tasks, outperforming parametric seq2seq models and
|
||||
task-specific retrieve-and-extract architectures. For language generation tasks, we
|
||||
find that RAG models generate more specific, diverse and factual language than a
|
||||
state-of-the-art parametric-only seq2seq baseline.*
|
||||
*Large pre-trained language models have been shown to store factual knowledge in their parameters, and achieve
|
||||
state-of-the-art results when fine-tuned on downstream NLP tasks. However, their ability to access and precisely
|
||||
manipulate knowledge is still limited, and hence on knowledge-intensive tasks, their performance lags behind
|
||||
task-specific architectures. Additionally, providing provenance for their decisions and updating their world knowledge
|
||||
remain open research problems. Pre-trained models with a differentiable access mechanism to explicit nonparametric
|
||||
memory can overcome this issue, but have so far been only investigated for extractive downstream tasks. We explore a
|
||||
general-purpose fine-tuning recipe for retrieval-augmented generation (RAG) — models which combine pre-trained
|
||||
parametric and non-parametric memory for language generation. We introduce RAG models where the parametric memory is a
|
||||
pre-trained seq2seq model and the non-parametric memory is a dense vector index of Wikipedia, accessed with a
|
||||
pre-trained neural retriever. We compare two RAG formulations, one which conditions on the same retrieved passages
|
||||
across the whole generated sequence, the other can use different passages per token. We fine-tune and evaluate our
|
||||
models on a wide range of knowledge-intensive NLP tasks and set the state-of-the-art on three open domain QA tasks,
|
||||
outperforming parametric seq2seq models and task-specific retrieve-and-extract architectures. For language generation
|
||||
tasks, we find that RAG models generate more specific, diverse and factual language than a state-of-the-art
|
||||
parametric-only seq2seq baseline.*
|
||||
|
||||
|
||||
|
||||
RagConfig
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagConfig
|
||||
:members:
|
||||
|
||||
|
||||
RagTokenizer
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagTokenizer
|
||||
:members: prepare_seq2seq_batch
|
||||
|
||||
|
||||
Rag specific outputs
|
||||
~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.modeling_rag.RetrievAugLMMarginOutput
|
||||
:members:
|
||||
|
|
@ -63,28 +57,28 @@ Rag specific outputs
|
|||
:members:
|
||||
|
||||
RagRetriever
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagRetriever
|
||||
:members:
|
||||
|
||||
|
||||
RagModel
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagModel
|
||||
:members: forward
|
||||
|
||||
|
||||
RagSequenceForGeneration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagSequenceForGeneration
|
||||
:members: forward, generate
|
||||
|
||||
|
||||
RagTokenForGeneration
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
.. autoclass:: transformers.RagTokenForGeneration
|
||||
:members: forward, generate
|
||||
|
|
|
|||
|
|
@ -10,7 +10,7 @@ Overview
|
|||
The Reformer model was proposed in the paper `Reformer: The Efficient Transformer
|
||||
<https://arxiv.org/abs/2001.04451.pdf>`__ by Nikita Kitaev, Łukasz Kaiser, Anselm Levskaya.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can
|
||||
be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of
|
||||
|
|
@ -36,12 +36,12 @@ would result in a position encoding matrix:
|
|||
.. math::
|
||||
X_{i,j}, \text{ with } i \in \left[1,\ldots, d\right] \text{ and } j \in \left[1,\ldots, n_s\right]
|
||||
|
||||
which alone has over 500M parameters to store. Axial positional encodings factorize :math:`X_{i,j}` into two matrices:
|
||||
which alone has over 500M parameters to store. Axial positional encodings factorize :math:`X_{i,j}` into two matrices:
|
||||
|
||||
.. math::
|
||||
X^{1}_{i,j}, \text{ with } i \in \left[1,\ldots, d^1\right] \text{ and } j \in \left[1,\ldots, n_s^1\right]
|
||||
|
||||
and
|
||||
and
|
||||
|
||||
.. math::
|
||||
X^{2}_{i,j}, \text{ with } i \in \left[1,\ldots, d^2\right] \text{ and } j \in \left[1,\ldots, n_s^2\right]
|
||||
|
|
@ -67,22 +67,23 @@ factorized embedding vectors: :math:`x^1_{k, l} + x^2_{l, k}`, where as the :obj
|
|||
Using the above example again, axial position encoding with :math:`d^1 = 2^5, d^2 = 2^5, n_s^1 = 2^9, n_s^2 = 2^{10}`
|
||||
can drastically reduced the number of parameters to :math:`2^{14} + 2^{15} \approx 49000` parameters.
|
||||
|
||||
In practice, the parameter :obj:`config.axial_pos_embds_dim` is set to a tuple :math:`(d^1, d^2)` which sum has to
|
||||
be equal to :obj:`config.hidden_size` and :obj:`config.axial_pos_shape` is set to a tuple :math:`(n_s^1, n_s^2)` which
|
||||
product has to be equal to :obj:`config.max_embedding_size`, which during training has to be equal to the
|
||||
`sequence length` of the :obj:`input_ids`.
|
||||
In practice, the parameter :obj:`config.axial_pos_embds_dim` is set to a tuple :math:`(d^1, d^2)` which sum has to be
|
||||
equal to :obj:`config.hidden_size` and :obj:`config.axial_pos_shape` is set to a tuple :math:`(n_s^1, n_s^2)` which
|
||||
product has to be equal to :obj:`config.max_embedding_size`, which during training has to be equal to the `sequence
|
||||
length` of the :obj:`input_ids`.
|
||||
|
||||
|
||||
LSH Self Attention
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In Locality sensitive hashing (LSH) self attention the key and query projection weights are tied. Therefore, the key
|
||||
query embedding vectors are also tied. LSH self attention uses the locality sensitive hashing mechanism proposed in
|
||||
`Practical and Optimal LSH for Angular Distance <https://arxiv.org/abs/1509.02897>`__ to assign each of the tied key
|
||||
query embedding vectors to one of :obj:`config.num_buckets` possible buckets. The premise is that the more "similar"
|
||||
key query embedding vectors (in terms of *cosine similarity*) are to each other, the more likely they are assigned to
|
||||
the same bucket.
|
||||
the same bucket.
|
||||
|
||||
The accuracy of the LSH mechanism can be improved by increasing :obj:`config.num_hashes` or directly the argument
|
||||
The accuracy of the LSH mechanism can be improved by increasing :obj:`config.num_hashes` or directly the argument
|
||||
:obj:`num_hashes` of the forward function so that the output of the LSH self attention better approximates the output
|
||||
of the "normal" full self attention. The buckets are then sorted and chunked into query key embedding vector chunks
|
||||
each of length :obj:`config.lsh_chunk_length`. For each chunk, the query embedding vectors attend to its key vectors
|
||||
|
|
@ -92,11 +93,11 @@ neighboring chunks and :obj:`config.lsh_num_chunks_after` following neighboring
|
|||
For more information, see the `original Paper <https://arxiv.org/abs/2001.04451>`__ or this great `blog post
|
||||
<https://www.pragmatic.ml/reformer-deep-dive/>`__.
|
||||
|
||||
Note that :obj:`config.num_buckets` can also be factorized into a list
|
||||
:math:`(n_{\text{buckets}}^1, n_{\text{buckets}}^2)`. This way instead of assigning the query key embedding vectors to
|
||||
one of :math:`(1,\ldots, n_{\text{buckets}})` they are assigned to one of
|
||||
:math:`(1-1,\ldots, n_{\text{buckets}}^1-1, \ldots, 1-n_{\text{buckets}}^2, \ldots, n_{\text{buckets}}^1-n_{\text{buckets}}^2)`.
|
||||
This is crucial for very long sequences to save memory.
|
||||
Note that :obj:`config.num_buckets` can also be factorized into a list :math:`(n_{\text{buckets}}^1,
|
||||
n_{\text{buckets}}^2)`. This way instead of assigning the query key embedding vectors to one of :math:`(1,\ldots,
|
||||
n_{\text{buckets}})` they are assigned to one of :math:`(1-1,\ldots, n_{\text{buckets}}^1-1, \ldots,
|
||||
1-n_{\text{buckets}}^2, \ldots, n_{\text{buckets}}^1-n_{\text{buckets}}^2)`. This is crucial for very long sequences to
|
||||
save memory.
|
||||
|
||||
When training a model from scratch, it is recommended to leave :obj:`config.num_buckets=None`, so that depending on the
|
||||
sequence length a good value for :obj:`num_buckets` is calculated on the fly. This value will then automatically be
|
||||
|
|
@ -128,7 +129,7 @@ multiple of :obj:`config.lsh_chunk_length` and :obj:`config.local_chunk_length`
|
|||
Positional Encodings are correctly set as described above. Reformer is very memory efficient so that the model can
|
||||
easily be trained on sequences as long as 64000 tokens.
|
||||
|
||||
For training, the :class:`~transformers.ReformerModelWithLMHead` should be used as follows:
|
||||
For training, the :class:`~transformers.ReformerModelWithLMHead` should be used as follows:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
|
|||
|
|
@ -8,8 +8,8 @@ The RoBERTa model was proposed in `RoBERTa: A Robustly Optimized BERT Pretrainin
|
|||
<https://arxiv.org/abs/1907.11692>`_ by Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer
|
||||
Levy, Mike Lewis, Luke Zettlemoyer, Veselin Stoyanov. It is based on Google's BERT model released in 2018.
|
||||
|
||||
It builds on BERT and modifies key hyperparameters, removing the next-sentence pretraining
|
||||
objective and training with much larger mini-batches and learning rates.
|
||||
It builds on BERT and modifies key hyperparameters, removing the next-sentence pretraining objective and training with
|
||||
much larger mini-batches and learning rates.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
|
|
@ -17,15 +17,15 @@ The abstract from the paper is the following:
|
|||
approaches is challenging. Training is computationally expensive, often done on private datasets of different sizes,
|
||||
and, as we will show, hyperparameter choices have significant impact on the final results. We present a replication
|
||||
study of BERT pretraining (Devlin et al., 2019) that carefully measures the impact of many key hyperparameters and
|
||||
training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of
|
||||
every model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These
|
||||
results highlight the importance of previously overlooked design choices, and raise questions about the source
|
||||
of recently reported improvements. We release our models and code.*
|
||||
training data size. We find that BERT was significantly undertrained, and can match or exceed the performance of every
|
||||
model published after it. Our best model achieves state-of-the-art results on GLUE, RACE and SQuAD. These results
|
||||
highlight the importance of previously overlooked design choices, and raise questions about the source of recently
|
||||
reported improvements. We release our models and code.*
|
||||
|
||||
Tips:
|
||||
|
||||
- This implementation is the same as :class:`~transformers.BertModel` with a tiny embeddings tweak as well as a
|
||||
setup for Roberta pretrained models.
|
||||
- This implementation is the same as :class:`~transformers.BertModel` with a tiny embeddings tweak as well as a setup
|
||||
for Roberta pretrained models.
|
||||
- RoBERTa has the same architecture as BERT, but uses a byte-level BPE as a tokenizer (same as GPT-2) and uses a
|
||||
different pretraining scheme.
|
||||
- RoBERTa doesn't have :obj:`token_type_ids`, you don't need to indicate which token belongs to which segment. Just
|
||||
|
|
|
|||
|
|
@ -4,38 +4,34 @@ SqueezeBERT
|
|||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The SqueezeBERT model was proposed in
|
||||
`SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
|
||||
<https://arxiv.org/abs/2006.11316>`__
|
||||
by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, Kurt W. Keutzer.
|
||||
It's a bidirectional transformer similar to the BERT model.
|
||||
The key difference between the BERT architecture and the SqueezeBERT architecture
|
||||
is that SqueezeBERT uses `grouped convolutions <https://blog.yani.io/filter-group-tutorial>`__
|
||||
The SqueezeBERT model was proposed in `SqueezeBERT: What can computer vision teach NLP about efficient neural networks?
|
||||
<https://arxiv.org/abs/2006.11316>`__ by Forrest N. Iandola, Albert E. Shaw, Ravi Krishna, Kurt W. Keutzer. It's a
|
||||
bidirectional transformer similar to the BERT model. The key difference between the BERT architecture and the
|
||||
SqueezeBERT architecture is that SqueezeBERT uses `grouped convolutions <https://blog.yani.io/filter-group-tutorial>`__
|
||||
instead of fully-connected layers for the Q, K, V and FFN layers.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*Humans read and write hundreds of billions of messages every day. Further, due to the availability of
|
||||
large datasets, large computing systems, and better neural network models, natural language processing (NLP)
|
||||
technology has made significant strides in understanding, proofreading, and organizing these messages.
|
||||
Thus, there is a significant opportunity to deploy NLP in myriad applications to help web users,
|
||||
social networks, and businesses. In particular, we consider smartphones and other mobile devices as
|
||||
crucial platforms for deploying NLP models at scale. However, today's highly-accurate NLP neural network
|
||||
models such as BERT and RoBERTa are extremely computationally expensive, with BERT-base taking 1.7 seconds
|
||||
to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods such as grouped
|
||||
convolutions have yielded significant speedups for computer vision networks, but many of these techniques
|
||||
have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in
|
||||
self-attention layers with grouped convolutions, and we use this technique in a novel network architecture
|
||||
called SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive
|
||||
accuracy on the GLUE test set. The SqueezeBERT code will be released.*
|
||||
*Humans read and write hundreds of billions of messages every day. Further, due to the availability of large datasets,
|
||||
large computing systems, and better neural network models, natural language processing (NLP) technology has made
|
||||
significant strides in understanding, proofreading, and organizing these messages. Thus, there is a significant
|
||||
opportunity to deploy NLP in myriad applications to help web users, social networks, and businesses. In particular, we
|
||||
consider smartphones and other mobile devices as crucial platforms for deploying NLP models at scale. However, today's
|
||||
highly-accurate NLP neural network models such as BERT and RoBERTa are extremely computationally expensive, with
|
||||
BERT-base taking 1.7 seconds to classify a text snippet on a Pixel 3 smartphone. In this work, we observe that methods
|
||||
such as grouped convolutions have yielded significant speedups for computer vision networks, but many of these
|
||||
techniques have not been adopted by NLP neural network designers. We demonstrate how to replace several operations in
|
||||
self-attention layers with grouped convolutions, and we use this technique in a novel network architecture called
|
||||
SqueezeBERT, which runs 4.3x faster than BERT-base on the Pixel 3 while achieving competitive accuracy on the GLUE test
|
||||
set. The SqueezeBERT code will be released.*
|
||||
|
||||
Tips:
|
||||
|
||||
- SqueezeBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on
|
||||
the right rather than the left.
|
||||
- SqueezeBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective.
|
||||
It is therefore efficient at predicting masked tokens and at NLU in general, but is not optimal for
|
||||
text generation. Models trained with a causal language modeling (CLM) objective are better in that regard.
|
||||
- SqueezeBERT is a model with absolute position embeddings so it's usually advised to pad the inputs on the right
|
||||
rather than the left.
|
||||
- SqueezeBERT is similar to BERT and therefore relies on the masked language modeling (MLM) objective. It is therefore
|
||||
efficient at predicting masked tokens and at NLU in general, but is not optimal for text generation. Models trained
|
||||
with a causal language modeling (CLM) objective are better in that regard.
|
||||
- For best results when finetuning on sequence classification tasks, it is recommended to start with the
|
||||
`squeezebert/squeezebert-mnli-headless` checkpoint.
|
||||
|
||||
|
|
|
|||
|
|
@ -29,13 +29,12 @@ Tips:
|
|||
each task is converted into a text-to-text format. T5 works well on a variety of tasks out-of-the-box by prepending a
|
||||
different prefix to the input corresponding to each task, e.g., for translation: *translate English to German: ...*,
|
||||
for summarization: *summarize: ...*.
|
||||
|
||||
|
||||
For more information about which prefix to use, it is easiest to look into Appendix D of the `paper
|
||||
<https://arxiv.org/pdf/1910.10683.pdf>`__.
|
||||
- For sequence-to-sequence generation, it is recommended to use :obj:`T5ForConditionalGeneration.generate()``. This
|
||||
method takes care of feeding the encoded input via cross-attention layers to the decoder and auto-regressively
|
||||
generates the decoder output.
|
||||
- T5 uses relative scalar embeddings. Encoder input padding can be done on the left and on the right.
|
||||
<https://arxiv.org/pdf/1910.10683.pdf>`__. - For sequence-to-sequence generation, it is recommended to use
|
||||
:obj:`T5ForConditionalGeneration.generate()``. This method takes care of feeding the encoded input via
|
||||
cross-attention layers to the decoder and auto-regressively generates the decoder output. - T5 uses relative scalar
|
||||
embeddings. Encoder input padding can be done on the left and on the right.
|
||||
|
||||
The original code can be found `here <https://github.com/google-research/text-to-text-transfer-transformer>`__.
|
||||
|
||||
|
|
@ -51,14 +50,14 @@ token. T5 can be trained / fine-tuned both in a supervised and unsupervised fash
|
|||
|
||||
- Unsupervised denoising training
|
||||
|
||||
In this setup spans of the input sequence are masked by so-called sentinel tokens (*a.k.a* unique mask tokens)
|
||||
and the output sequence is formed as a concatenation of the same sentinel tokens and the *real* masked tokens.
|
||||
Each sentinel token represents a unique mask token for this sentence and should start with :obj:`<extra_id_0>`,
|
||||
In this setup spans of the input sequence are masked by so-called sentinel tokens (*a.k.a* unique mask tokens) and
|
||||
the output sequence is formed as a concatenation of the same sentinel tokens and the *real* masked tokens. Each
|
||||
sentinel token represents a unique mask token for this sentence and should start with :obj:`<extra_id_0>`,
|
||||
:obj:`<extra_id_1>`, ... up to :obj:`<extra_id_99>`. As a default, 100 sentinel tokens are available in
|
||||
:class:`~transformers.T5Tokenizer`.
|
||||
|
||||
|
||||
For instance, the sentence "The cute dog walks in the park" with the masks put on "cute dog" and "the" should be
|
||||
processed as follows:
|
||||
processed as follows:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -69,10 +68,10 @@ token. T5 can be trained / fine-tuned both in a supervised and unsupervised fash
|
|||
|
||||
- Supervised training
|
||||
|
||||
In this setup the input sequence and output sequence are standard sequence-to-sequence input output mapping.
|
||||
In translation, for instance with the input sequence "The house is wonderful." and output sequence "Das Haus ist
|
||||
In this setup the input sequence and output sequence are standard sequence-to-sequence input output mapping. In
|
||||
translation, for instance with the input sequence "The house is wonderful." and output sequence "Das Haus ist
|
||||
wunderbar.", the sentences should be processed as follows:
|
||||
|
||||
|
||||
.. code-block::
|
||||
|
||||
input_ids = tokenizer('translate English to German: The house is wonderful.', return_tensors='pt').input_ids
|
||||
|
|
|
|||
|
|
@ -14,19 +14,19 @@ The abstract from the paper is the following:
|
|||
|
||||
*Transformers have a potential of learning longer-term dependency, but are limited by a fixed-length context in the
|
||||
setting of language modeling. We propose a novel neural architecture Transformer-XL that enables learning dependency
|
||||
beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and
|
||||
a novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves
|
||||
the context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and
|
||||
450% longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up
|
||||
to 1,800+ times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results
|
||||
of bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on
|
||||
Penn Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably
|
||||
beyond a fixed length without disrupting temporal coherence. It consists of a segment-level recurrence mechanism and a
|
||||
novel positional encoding scheme. Our method not only enables capturing longer-term dependency, but also resolves the
|
||||
context fragmentation problem. As a result, Transformer-XL learns dependency that is 80% longer than RNNs and 450%
|
||||
longer than vanilla Transformers, achieves better performance on both short and long sequences, and is up to 1,800+
|
||||
times faster than vanilla Transformers during evaluation. Notably, we improve the state-of-the-art results of
|
||||
bpc/perplexity to 0.99 on enwiki8, 1.08 on text8, 18.3 on WikiText-103, 21.8 on One Billion Word, and 54.5 on Penn
|
||||
Treebank (without finetuning). When trained only on WikiText-103, Transformer-XL manages to generate reasonably
|
||||
coherent, novel text articles with thousands of tokens.*
|
||||
|
||||
Tips:
|
||||
|
||||
- Transformer-XL uses relative sinusoidal positional embeddings. Padding can be done on the left or on the right.
|
||||
The original implementation trains on SQuAD with padding on the left, therefore the padding defaults are set to left.
|
||||
- Transformer-XL uses relative sinusoidal positional embeddings. Padding can be done on the left or on the right. The
|
||||
original implementation trains on SQuAD with padding on the left, therefore the padding defaults are set to left.
|
||||
- Transformer-XL is one of the few models that has no sequence length limit.
|
||||
|
||||
The original code can be found `here <https://github.com/kimiyoung/transformer-xl>`__.
|
||||
|
|
|
|||
|
|
@ -14,21 +14,21 @@ Guillaume Lample, Alexis Conneau. It's a transformer pretrained using one of the
|
|||
The abstract from the paper is the following:
|
||||
|
||||
*Recent studies have demonstrated the efficiency of generative pretraining for English natural language understanding.
|
||||
In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining.
|
||||
We propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual
|
||||
In this work, we extend this approach to multiple languages and show the effectiveness of cross-lingual pretraining. We
|
||||
propose two methods to learn cross-lingual language models (XLMs): one unsupervised that only relies on monolingual
|
||||
data, and one supervised that leverages parallel data with a new cross-lingual language model objective. We obtain
|
||||
state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI,
|
||||
our approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation,
|
||||
we obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On
|
||||
supervised machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming
|
||||
the previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.*
|
||||
state-of-the-art results on cross-lingual classification, unsupervised and supervised machine translation. On XNLI, our
|
||||
approach pushes the state of the art by an absolute gain of 4.9% accuracy. On unsupervised machine translation, we
|
||||
obtain 34.3 BLEU on WMT'16 German-English, improving the previous state of the art by more than 9 BLEU. On supervised
|
||||
machine translation, we obtain a new state of the art of 38.5 BLEU on WMT'16 Romanian-English, outperforming the
|
||||
previous best approach by more than 4 BLEU. Our code and pretrained models will be made publicly available.*
|
||||
|
||||
Tips:
|
||||
|
||||
- XLM has many different checkpoints, which were trained using different objectives: CLM, MLM or TLM. Make sure to
|
||||
select the correct objective for your task (e.g. MLM checkpoints are not suitable for generation).
|
||||
- XLM has multilingual checkpoints which leverage a specific :obj:`lang` parameter. Check out the
|
||||
:doc:`multi-lingual <../multilingual>` page for more information.
|
||||
- XLM has multilingual checkpoints which leverage a specific :obj:`lang` parameter. Check out the :doc:`multi-lingual
|
||||
<../multilingual>` page for more information.
|
||||
|
||||
The original code can be found `here <https://github.com/facebookresearch/XLM/>`__.
|
||||
|
||||
|
|
|
|||
|
|
@ -9,13 +9,25 @@ XLM-ProphetNet
|
|||
Overview
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The XLM-ProphetNet model was proposed in `ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training, <https://arxiv.org/abs/2001.04063>`__ by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou on 13 Jan, 2020.
|
||||
The XLM-ProphetNet model was proposed in `ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training,
|
||||
<https://arxiv.org/abs/2001.04063>`__ by Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei
|
||||
Zhang, Ming Zhou on 13 Jan, 2020.
|
||||
|
||||
XLM-ProphetNet is an encoder-decoder model and can predict n-future tokens for "ngram" language modeling instead of just the next token. Its architecture is identical to ProhpetNet, but the model was trained on the multi-lingual "wiki100" Wikipedia dump.
|
||||
XLM-ProphetNet is an encoder-decoder model and can predict n-future tokens for "ngram" language modeling instead of
|
||||
just the next token. Its architecture is identical to ProhpetNet, but the model was trained on the multi-lingual
|
||||
"wiki100" Wikipedia dump.
|
||||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*In this paper, we present a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of the optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale dataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.*
|
||||
*In this paper, we present a new sequence-to-sequence pre-training model called ProphetNet, which introduces a novel
|
||||
self-supervised objective named future n-gram prediction and the proposed n-stream self-attention mechanism. Instead of
|
||||
the optimization of one-step ahead prediction in traditional sequence-to-sequence model, the ProphetNet is optimized by
|
||||
n-step ahead prediction which predicts the next n tokens simultaneously based on previous context tokens at each time
|
||||
step. The future n-gram prediction explicitly encourages the model to plan for the future tokens and prevent
|
||||
overfitting on strong local correlations. We pre-train ProphetNet using a base scale dataset (16GB) and a large scale
|
||||
dataset (160GB) respectively. Then we conduct experiments on CNN/DailyMail, Gigaword, and SQuAD 1.1 benchmarks for
|
||||
abstractive summarization and question generation tasks. Experimental results show that ProphetNet achieves new
|
||||
state-of-the-art results on all these datasets compared to the models using the same scale pre-training corpus.*
|
||||
|
||||
The Authors' code can be found `here <https://github.com/microsoft/ProphetNet>`__.
|
||||
|
||||
|
|
|
|||
|
|
@ -12,25 +12,25 @@ data.
|
|||
|
||||
The abstract from the paper is the following:
|
||||
|
||||
*This paper shows that pretraining multilingual language models at scale leads to significant performance gains for
|
||||
a wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred
|
||||
*This paper shows that pretraining multilingual language models at scale leads to significant performance gains for a
|
||||
wide range of cross-lingual transfer tasks. We train a Transformer-based masked language model on one hundred
|
||||
languages, using more than two terabytes of filtered CommonCrawl data. Our model, dubbed XLM-R, significantly
|
||||
outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +13.8% average accuracy
|
||||
on XNLI, +12.3% average F1 score on MLQA, and +2.1% average F1 score on NER. XLM-R performs particularly well on
|
||||
low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model.
|
||||
We also present a detailed empirical evaluation of the key factors that are required to achieve these gains,
|
||||
including the trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and
|
||||
low resource languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling
|
||||
without sacrificing per-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE
|
||||
and XNLI benchmarks. We will make XLM-R code, data, and models publicly available.*
|
||||
outperforms multilingual BERT (mBERT) on a variety of cross-lingual benchmarks, including +13.8% average accuracy on
|
||||
XNLI, +12.3% average F1 score on MLQA, and +2.1% average F1 score on NER. XLM-R performs particularly well on
|
||||
low-resource languages, improving 11.8% in XNLI accuracy for Swahili and 9.2% for Urdu over the previous XLM model. We
|
||||
also present a detailed empirical evaluation of the key factors that are required to achieve these gains, including the
|
||||
trade-offs between (1) positive transfer and capacity dilution and (2) the performance of high and low resource
|
||||
languages at scale. Finally, we show, for the first time, the possibility of multilingual modeling without sacrificing
|
||||
per-language performance; XLM-Ris very competitive with strong monolingual models on the GLUE and XNLI benchmarks. We
|
||||
will make XLM-R code, data, and models publicly available.*
|
||||
|
||||
Tips:
|
||||
|
||||
- XLM-RoBERTa is a multilingual model trained on 100 different languages. Unlike some XLM multilingual models, it does
|
||||
not require :obj:`lang` tensors to understand which language is used, and should be able to determine the correct
|
||||
language from the input ids.
|
||||
- This implementation is the same as RoBERTa. Refer to the :doc:`documentation of RoBERTa <roberta>` for usage
|
||||
examples as well as the information relative to the inputs and outputs.
|
||||
- This implementation is the same as RoBERTa. Refer to the :doc:`documentation of RoBERTa <roberta>` for usage examples
|
||||
as well as the information relative to the inputs and outputs.
|
||||
|
||||
The original code can be found `here <https://github.com/pytorch/fairseq/tree/master/examples/xlmr>`__.
|
||||
|
||||
|
|
|
|||
|
|
@ -16,11 +16,11 @@ The abstract from the paper is the following:
|
|||
better performance than pretraining approaches based on autoregressive language modeling. However, relying on
|
||||
corrupting the input with masks, BERT neglects dependency between the masked positions and suffers from a
|
||||
pretrain-finetune discrepancy. In light of these pros and cons, we propose XLNet, a generalized autoregressive
|
||||
pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over
|
||||
all permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive
|
||||
formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model,
|
||||
into pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by
|
||||
a large margin, including question answering, natural language inference, sentiment analysis, and document ranking.*
|
||||
pretraining method that (1) enables learning bidirectional contexts by maximizing the expected likelihood over all
|
||||
permutations of the factorization order and (2) overcomes the limitations of BERT thanks to its autoregressive
|
||||
formulation. Furthermore, XLNet integrates ideas from Transformer-XL, the state-of-the-art autoregressive model, into
|
||||
pretraining. Empirically, under comparable experiment settings, XLNet outperforms BERT on 20 tasks, often by a large
|
||||
margin, including question answering, natural language inference, sentiment analysis, and document ranking.*
|
||||
|
||||
Tips:
|
||||
|
||||
|
|
|
|||
|
|
@ -15,8 +15,8 @@ Prepare your model for uploading
|
|||
|
||||
We have seen in the :doc:`training tutorial <training>`: how to fine-tune a model on a given task. You have probably
|
||||
done something similar on your task, either using the model directly in your own training loop or using the
|
||||
:class:`~.transformers.Trainer`/:class:`~.transformers.TFTrainer` class. Let's see how you can share the result on
|
||||
the `model hub <https://huggingface.co/models>`__.
|
||||
:class:`~.transformers.Trainer`/:class:`~.transformers.TFTrainer` class. Let's see how you can share the result on the
|
||||
`model hub <https://huggingface.co/models>`__.
|
||||
|
||||
Basic steps
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
|
@ -60,22 +60,20 @@ Make your model work on all frameworks
|
|||
|
||||
You probably have your favorite framework, but so will other users! That's why it's best to upload your model with both
|
||||
PyTorch `and` TensorFlow checkpoints to make it easier to use (if you skip this step, users will still be able to load
|
||||
your model in another framework, but it will be slower, as it will have to be converted on the fly). Don't worry, it's super easy to do (and in a future version,
|
||||
it will all be automatic). You will need to install both PyTorch and TensorFlow for this step, but you don't need to
|
||||
worry about the GPU, so it should be very easy. Check the
|
||||
`TensorFlow installation page <https://www.tensorflow.org/install/pip#tensorflow-2.0-rc-is-available>`__
|
||||
and/or the `PyTorch installation page <https://pytorch.org/get-started/locally/#start-locally>`__ to see how.
|
||||
your model in another framework, but it will be slower, as it will have to be converted on the fly). Don't worry, it's
|
||||
super easy to do (and in a future version, it will all be automatic). You will need to install both PyTorch and
|
||||
TensorFlow for this step, but you don't need to worry about the GPU, so it should be very easy. Check the `TensorFlow
|
||||
installation page <https://www.tensorflow.org/install/pip#tensorflow-2.0-rc-is-available>`__ and/or the `PyTorch
|
||||
installation page <https://pytorch.org/get-started/locally/#start-locally>`__ to see how.
|
||||
|
||||
First check that your model class exists in the other framework, that is try to import the same model by either adding
|
||||
or removing TF. For instance, if you trained a :class:`~transformers.DistilBertForSequenceClassification`, try to
|
||||
type
|
||||
or removing TF. For instance, if you trained a :class:`~transformers.DistilBertForSequenceClassification`, try to type
|
||||
|
||||
.. code-block::
|
||||
|
||||
from transformers import TFDistilBertForSequenceClassification
|
||||
|
||||
and if you trained a :class:`~transformers.TFDistilBertForSequenceClassification`, try to
|
||||
type
|
||||
and if you trained a :class:`~transformers.TFDistilBertForSequenceClassification`, try to type
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -112,7 +110,8 @@ Make sure there are no garbage files in the directory you'll upload. It should o
|
|||
- a `tf_model.h5` file, which is the TensorFlow checkpoint (unless you can't have it for some reason) ;
|
||||
- a `special_tokens_map.json`, which is part of your :doc:`tokenizer <main_classes/tokenizer>` save;
|
||||
- a `tokenizer_config.json`, which is part of your :doc:`tokenizer <main_classes/tokenizer>` save;
|
||||
- files named `vocab.json`, `vocab.txt`, `merges.txt`, or similar, which contain the vocabulary of your tokenizer, part of your :doc:`tokenizer <main_classes/tokenizer>` save;
|
||||
- files named `vocab.json`, `vocab.txt`, `merges.txt`, or similar, which contain the vocabulary of your tokenizer, part
|
||||
of your :doc:`tokenizer <main_classes/tokenizer>` save;
|
||||
- maybe a `added_tokens.json`, which is part of your :doc:`tokenizer <main_classes/tokenizer>` save.
|
||||
|
||||
Other files can safely be deleted.
|
||||
|
|
@ -135,7 +134,8 @@ Then log in using the same credentials as on huggingface.co. To upload your mode
|
|||
|
||||
This will upload the folder containing the weights, tokenizer and configuration we prepared in the previous section.
|
||||
|
||||
By default you will be prompted to confirm that you want these files to be uploaded. If you are uploading multiple models and need to script that process, you can add `-y` to bypass the prompt. For example:
|
||||
By default you will be prompted to confirm that you want these files to be uploaded. If you are uploading multiple
|
||||
models and need to script that process, you can add `-y` to bypass the prompt. For example:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -179,15 +179,15 @@ Add a model card
|
|||
To make sure everyone knows what your model can do, what its limitations and potential bias or ethetical
|
||||
considerations, please add a README.md model card to the 🤗 Transformers repo under `model_cards/`. It should then be
|
||||
placed in a subfolder with your username or organization, then another subfolder named like your model
|
||||
(`awesome-name-you-picked`). Or just click on the "Create a model card on GitHub" button on the model page, it will
|
||||
get you directly to the right location. If you need one, `here <https://github.com/huggingface/model_card>`__ is a
|
||||
model card template (meta-suggestions are welcome).
|
||||
(`awesome-name-you-picked`). Or just click on the "Create a model card on GitHub" button on the model page, it will get
|
||||
you directly to the right location. If you need one, `here <https://github.com/huggingface/model_card>`__ is a model
|
||||
card template (meta-suggestions are welcome).
|
||||
|
||||
If your model is fine-tuned from another model coming from the model hub (all 🤗 Transformers pretrained models do),
|
||||
don't forget to link to its model card so that people can fully trace how your model was built.
|
||||
|
||||
If you have never made a pull request to the 🤗 Transformers repo, look at the
|
||||
:doc:`contributing guide <contributing>` to see the steps to follow.
|
||||
If you have never made a pull request to the 🤗 Transformers repo, look at the :doc:`contributing guide <contributing>`
|
||||
to see the steps to follow.
|
||||
|
||||
.. Note::
|
||||
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
Summary of the models
|
||||
=======================================================================================================================
|
||||
|
||||
This is a summary of the models available in 🤗 Transformers. It assumes you’re familiar with the original
|
||||
`transformer model <https://arxiv.org/abs/1706.03762>`_. For a gentle introduction check the `annotated transformer
|
||||
This is a summary of the models available in 🤗 Transformers. It assumes you’re familiar with the original `transformer
|
||||
model <https://arxiv.org/abs/1706.03762>`_. For a gentle introduction check the `annotated transformer
|
||||
<http://nlp.seas.harvard.edu/2018/04/03/attention.html>`_. Here we focus on the high-level differences between the
|
||||
models. You can check them more in detail in their respective documentation. Also checkout the
|
||||
:doc:`pretrained model page </pretrained_models>` to see the checkpoints available for each type of model and all `the
|
||||
community models <https://huggingface.co/models>`_.
|
||||
models. You can check them more in detail in their respective documentation. Also checkout the :doc:`pretrained model
|
||||
page </pretrained_models>` to see the checkpoints available for each type of model and all `the community models
|
||||
<https://huggingface.co/models>`_.
|
||||
|
||||
Each one of the models in the library falls into one of the following categories:
|
||||
|
||||
|
|
@ -19,8 +19,8 @@ Each one of the models in the library falls into one of the following categories
|
|||
Autoregressive models are pretrained on the classic language modeling task: guess the next token having read all the
|
||||
previous ones. They correspond to the decoder of the original transformer model, and a mask is used on top of the full
|
||||
sentence so that the attention heads can only see what was before in the next, and not what’s after. Although those
|
||||
models can be fine-tuned and achieve great results on many tasks, the most natural application is text generation.
|
||||
A typical example of such models is GPT.
|
||||
models can be fine-tuned and achieve great results on many tasks, the most natural application is text generation. A
|
||||
typical example of such models is GPT.
|
||||
|
||||
Autoencoding models are pretrained by corrupting the input tokens in some way and trying to reconstruct the original
|
||||
sentence. They correspond to the encoder of the original transformer model in the sense that they get access to the
|
||||
|
|
@ -30,8 +30,8 @@ sentence classification or token classification. A typical example of such model
|
|||
|
||||
Note that the only difference between autoregressive models and autoencoding models is in the way the model is
|
||||
pretrained. Therefore, the same architecture can be used for both autoregressive and autoencoding models. When a given
|
||||
model has been used for both types of pretraining, we have put it in the category corresponding to the article where it was first
|
||||
introduced.
|
||||
model has been used for both types of pretraining, we have put it in the category corresponding to the article where it
|
||||
was first introduced.
|
||||
|
||||
Sequence-to-sequence models use both the encoder and the decoder of the original transformer, either for translation
|
||||
tasks or by transforming other tasks to sequence-to-sequence problems. They can be fine-tuned to many tasks but their
|
||||
|
|
@ -60,8 +60,8 @@ Original GPT
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-openai--gpt-blueviolet">
|
||||
</a>
|
||||
|
||||
`Improving Language Understanding by Generative Pre-Training <https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf>`_,
|
||||
Alec Radford et al.
|
||||
`Improving Language Understanding by Generative Pre-Training
|
||||
<https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf>`_, Alec Radford et al.
|
||||
|
||||
The first autoregressive model based on the transformer architecture, pretrained on the Book Corpus dataset.
|
||||
|
||||
|
|
@ -80,7 +80,8 @@ GPT-2
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-gpt2-blueviolet">
|
||||
</a>
|
||||
|
||||
`Language Models are Unsupervised Multitask Learners <https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf>`_,
|
||||
`Language Models are Unsupervised Multitask Learners
|
||||
<https://d4mucfpksywv.cloudfront.net/better-language-models/language_models_are_unsupervised_multitask_learners.pdf>`_,
|
||||
Alec Radford et al.
|
||||
|
||||
A bigger and better version of GPT, pretrained on WebText (web pages from outgoing links in Reddit with 3 karmas or
|
||||
|
|
@ -122,8 +123,8 @@ Transformer-XL
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-transfo--xl-blueviolet">
|
||||
</a>
|
||||
|
||||
`Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context <https://arxiv.org/abs/1901.02860>`_,
|
||||
Zihang Dai et al.
|
||||
`Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context <https://arxiv.org/abs/1901.02860>`_, Zihang
|
||||
Dai et al.
|
||||
|
||||
Same as a regular GPT model, but introduces a recurrence mechanism for two consecutive segments (similar to a regular
|
||||
RNNs with two consecutive inputs). In this context, a segment is a number of consecutive tokens (for instance 512) that
|
||||
|
|
@ -153,8 +154,7 @@ Reformer
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-reformer-blueviolet">
|
||||
</a>
|
||||
|
||||
`Reformer: The Efficient Transformer <https://arxiv.org/abs/2001.04451>`_,
|
||||
Nikita Kitaev et al .
|
||||
`Reformer: The Efficient Transformer <https://arxiv.org/abs/2001.04451>`_, Nikita Kitaev et al .
|
||||
|
||||
An autoregressive transformer model with lots of tricks to reduce memory footprint and compute time. Those tricks
|
||||
include:
|
||||
|
|
@ -188,8 +188,8 @@ XLNet
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-xlnet-blueviolet">
|
||||
</a>
|
||||
|
||||
`XLNet: Generalized Autoregressive Pretraining for Language Understanding <https://arxiv.org/abs/1906.08237>`_,
|
||||
Zhilin Yang et al.
|
||||
`XLNet: Generalized Autoregressive Pretraining for Language Understanding <https://arxiv.org/abs/1906.08237>`_, Zhilin
|
||||
Yang et al.
|
||||
|
||||
XLNet is not a traditional autoregressive model but uses a training strategy that builds on that. It permutes the
|
||||
tokens in the sentence, then allows the model to use the last n tokens to predict the token n+1. Since this is all done
|
||||
|
|
@ -207,7 +207,8 @@ Autoencoding models
|
|||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
As mentioned before, these models rely on the encoder part of the original transformer and use no mask so the model can
|
||||
look at all the tokens in the attention heads. For pretraining, targets are the original sentences and inputs are their corrupted versions.
|
||||
look at all the tokens in the attention heads. For pretraining, targets are the original sentences and inputs are their
|
||||
corrupted versions.
|
||||
|
||||
BERT
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
|
@ -260,8 +261,8 @@ Same as BERT but with a few tweaks:
|
|||
sequence of tokens) so it's more logical to have H >> E. Also, the embedding matrix is large since it's V x E (V
|
||||
being the vocab size). If E < H, it has less parameters.
|
||||
* Layers are split in groups that share parameters (to save memory).
|
||||
* Next sentence prediction is replaced by a sentence ordering prediction: in the inputs, we have two sentences A and B
|
||||
(that are consecutive) and we either feed A followed by B or B followed by A. The model must predict if they have
|
||||
* Next sentence prediction is replaced by a sentence ordering prediction: in the inputs, we have two sentences A and
|
||||
B (that are consecutive) and we either feed A followed by B or B followed by A. The model must predict if they have
|
||||
been swapped or not.
|
||||
|
||||
The library provides a version of the model for masked language modeling, token classification, sentence
|
||||
|
|
@ -279,8 +280,7 @@ RoBERTa
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-roberta-blueviolet">
|
||||
</a>
|
||||
|
||||
`RoBERTa: A Robustly Optimized BERT Pretraining Approach <https://arxiv.org/abs/1907.11692>`_,
|
||||
Yinhan Liu et al.
|
||||
`RoBERTa: A Robustly Optimized BERT Pretraining Approach <https://arxiv.org/abs/1907.11692>`_, Yinhan Liu et al.
|
||||
|
||||
Same as BERT with better pretraining tricks:
|
||||
|
||||
|
|
@ -339,8 +339,8 @@ library provides checkpoints for all of them:
|
|||
previous section as well). One of the languages is selected for each training sample, and the model input is a
|
||||
sentence of 256 tokens, that may span over several documents in one of those languages.
|
||||
* Masked language modeling (MLM) which is like RoBERTa. One of the languages is selected for each training sample,
|
||||
and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages, with
|
||||
dynamic masking of the tokens.
|
||||
and the model input is a sentence of 256 tokens, that may span over several documents in one of those languages,
|
||||
with dynamic masking of the tokens.
|
||||
* A combination of MLM and translation language modeling (TLM). This consists of concatenating a sentence in two
|
||||
different languages, with random masking. To predict one of the masked tokens, the model can use both, the
|
||||
surrounding context in language 1 and the context given by language 2.
|
||||
|
|
@ -523,20 +523,21 @@ Pegasus
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-pegasus-blueviolet">
|
||||
</a>
|
||||
|
||||
`PEGASUS: Pre-training with Extracted Gap-sentences forAbstractive Summarization
|
||||
`PEGASUS: Pre-training with Extracted Gap-sentences forAbstractive Summarization
|
||||
<https://arxiv.org/pdf/1912.08777.pdf>`_, Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019.
|
||||
|
||||
Sequence-to-sequence model with the same encoder-decoder model architecture as BART. Pegasus is pre-trained jointly on
|
||||
two self-supervised objective functions: Masked Language Modeling (MLM) and a novel summarization specific pre-training
|
||||
objective, called Gap Sentence Generation (GSG).
|
||||
|
||||
* MLM: encoder input tokens are randomely replaced by a mask tokens and have to be predicted by the encoder (like
|
||||
in BERT)
|
||||
* MLM: encoder input tokens are randomely replaced by a mask tokens and have to be predicted by the encoder (like in
|
||||
BERT)
|
||||
* GSG: whole encoder input sentences are replaced by a second mask token and fed to the decoder, but which has a
|
||||
causal mask to hide the future words like a regular auto-regressive transformer decoder.
|
||||
|
||||
In contrast to BART, Pegasus' pretraining task is intentionally similar to summarization: important sentences are
|
||||
masked and are generated together as one output sequence from the remaining sentences, similar to an extractive summary.
|
||||
masked and are generated together as one output sequence from the remaining sentences, similar to an extractive
|
||||
summary.
|
||||
|
||||
The library provides a version of this model for conditional generation, which should be used for summarization.
|
||||
|
||||
|
|
@ -571,20 +572,20 @@ T5
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-t5-blueviolet">
|
||||
</a>
|
||||
|
||||
`Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer <https://arxiv.org/abs/1910.10683>`_,
|
||||
Colin Raffel et al.
|
||||
`Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
|
||||
<https://arxiv.org/abs/1910.10683>`_, Colin Raffel et al.
|
||||
|
||||
Uses the traditional transformer model (with a slight change in the positional embeddings, which are learned at
|
||||
each layer). To be able to operate on all NLP tasks, it transforms them into text-to-text problems by using specific
|
||||
Uses the traditional transformer model (with a slight change in the positional embeddings, which are learned at each
|
||||
layer). To be able to operate on all NLP tasks, it transforms them into text-to-text problems by using specific
|
||||
prefixes: “summarize: ”, “question: ”, “translate English to German: ” and so forth.
|
||||
|
||||
The pretraining includes both supervised and self-supervised training. Supervised training is conducted on downstream
|
||||
tasks provided by the GLUE and SuperGLUE benchmarks (converting them into text-to-text tasks as explained above).
|
||||
|
||||
Self-supervised training uses corrupted tokens, by randomly removing 15% of the tokens and
|
||||
replacing them with individual sentinel tokens (if several consecutive tokens are marked for removal, the whole group
|
||||
is replaced with a single sentinel token). The input of the encoder is the corrupted sentence, the input of the decoder
|
||||
is the original sentence and the target is then the dropped out tokens delimited by their sentinel tokens.
|
||||
Self-supervised training uses corrupted tokens, by randomly removing 15% of the tokens and replacing them with
|
||||
individual sentinel tokens (if several consecutive tokens are marked for removal, the whole group is replaced with a
|
||||
single sentinel token). The input of the encoder is the corrupted sentence, the input of the decoder is the original
|
||||
sentence and the target is then the dropped out tokens delimited by their sentinel tokens.
|
||||
|
||||
For instance, if we have the sentence “My dog is very cute .”, and we decide to remove the tokens: "dog", "is" and
|
||||
"cute", the encoder input becomes “My <x> very <y> .” and the target input becomes “<x> dog is <y> cute .<z>”
|
||||
|
|
@ -603,13 +604,12 @@ MBart
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-mbart-blueviolet">
|
||||
</a>
|
||||
|
||||
`Multilingual Denoising Pre-training for Neural Machine Translation <https://arxiv.org/abs/2001.08210>`_ by Yinhan
|
||||
Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov
|
||||
Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.
|
||||
`Multilingual Denoising Pre-training for Neural Machine Translation <https://arxiv.org/abs/2001.08210>`_ by Yinhan Liu,
|
||||
Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov Marjan Ghazvininejad, Mike Lewis, Luke Zettlemoyer.
|
||||
|
||||
The model architecture and pre-training objective is same as BART, but MBart is trained on 25 languages
|
||||
and is intended for supervised and unsupervised machine translation. MBart is one of the first methods
|
||||
for pre-training a complete sequence-to-sequence model by denoising full texts in multiple languages,
|
||||
The model architecture and pre-training objective is same as BART, but MBart is trained on 25 languages and is intended
|
||||
for supervised and unsupervised machine translation. MBart is one of the first methods for pre-training a complete
|
||||
sequence-to-sequence model by denoising full texts in multiple languages,
|
||||
|
||||
The library provides a version of this model for conditional generation.
|
||||
|
||||
|
|
@ -636,11 +636,11 @@ ProphetNet
|
|||
Yu Yan, Weizhen Qi, Yeyun Gong, Dayiheng Liu, Nan Duan, Jiusheng Chen, Ruofei Zhang, Ming Zhou.
|
||||
|
||||
ProphetNet introduces a novel *sequence-to-sequence* pre-training objective, called *future n-gram prediction*. In
|
||||
future n-gram prediction, the model predicts the next n tokens simultaneously based on previous context tokens at
|
||||
each time step instead instead of just the single next token. The future n-gram prediction explicitly encourages
|
||||
the model to plan for the future tokens and prevent overfitting on strong local correlations.
|
||||
The model architecture is based on the original Transformer, but replaces the "standard" self-attention mechanism
|
||||
in the decoder by a a main self-attention mechanism and a self and n-stream (predict) self-attention mechanism.
|
||||
future n-gram prediction, the model predicts the next n tokens simultaneously based on previous context tokens at each
|
||||
time step instead instead of just the single next token. The future n-gram prediction explicitly encourages the model
|
||||
to plan for the future tokens and prevent overfitting on strong local correlations. The model architecture is based on
|
||||
the original Transformer, but replaces the "standard" self-attention mechanism in the decoder by a a main
|
||||
self-attention mechanism and a self and n-stream (predict) self-attention mechanism.
|
||||
|
||||
The library provides a pre-trained version of this model for conditional generation and a fine-tuned version for
|
||||
summarization.
|
||||
|
|
@ -682,8 +682,8 @@ et al.
|
|||
|
||||
A transformers model used in multimodal settings, combining a text and an image to make predictions. The transformer
|
||||
model takes as inputs the embeddings of the tokenized text and the final activations of a pretrained on images resnet
|
||||
(after the pooling layer) that goes through a linear layer (to go from number of features at the end of the
|
||||
resnet to the hidden state dimension of the transformer).
|
||||
(after the pooling layer) that goes through a linear layer (to go from number of features at the end of the resnet to
|
||||
the hidden state dimension of the transformer).
|
||||
|
||||
The different inputs are concatenated, and on top of the positional embeddings, a segment embedding is added to let the
|
||||
model know which part of the input vector corresponds to the text and which to the image.
|
||||
|
|
@ -691,8 +691,7 @@ model know which part of the input vector corresponds to the text and which to t
|
|||
The pretrained model only works for classification.
|
||||
|
||||
..
|
||||
More information in this :doc:`model documentation </model_doc/mmbt.html>`.
|
||||
TODO: write this page
|
||||
More information in this :doc:`model documentation </model_doc/mmbt.html>`. TODO: write this page
|
||||
|
||||
.. _retrieval-based-models:
|
||||
|
||||
|
|
@ -714,19 +713,22 @@ DPR
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-dpr-blueviolet">
|
||||
</a>
|
||||
|
||||
`Dense Passage Retrieval for Open-Domain Question Answering <https://arxiv.org/abs/2004.04906>`_,
|
||||
Vladimir Karpukhin et al.
|
||||
`Dense Passage Retrieval for Open-Domain Question Answering <https://arxiv.org/abs/2004.04906>`_, Vladimir Karpukhin et
|
||||
al.
|
||||
|
||||
Dense Passage Retrieval (DPR) - is a set of tools and models for state-of-the-art open-domain question-answering research.
|
||||
Dense Passage Retrieval (DPR) - is a set of tools and models for state-of-the-art open-domain question-answering
|
||||
research.
|
||||
|
||||
|
||||
DPR consists in three models:
|
||||
|
||||
* Question encoder: encode questions as vectors
|
||||
* Context encoder: encode contexts as vectors
|
||||
* Reader: extract the answer of the questions inside retrieved contexts, along with a relevance score (high if the inferred span actually answers the question).
|
||||
* Reader: extract the answer of the questions inside retrieved contexts, along with a relevance score (high if the
|
||||
inferred span actually answers the question).
|
||||
|
||||
DPR's pipeline (not implemented yet) uses a retrieval step to find the top k contexts given a certain question, and then it calls the reader with the question and the retrieved documents to get the answer.
|
||||
DPR's pipeline (not implemented yet) uses a retrieval step to find the top k contexts given a certain question, and
|
||||
then it calls the reader with the question and the retrieved documents to get the answer.
|
||||
|
||||
RAG
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
|
@ -740,12 +742,14 @@ RAG
|
|||
<img alt="Doc" src="https://img.shields.io/badge/Model_documentation-rag-blueviolet">
|
||||
</a>
|
||||
|
||||
`Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks <https://arxiv.org/abs/2005.11401>`_,
|
||||
Patrick Lewis, Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela
|
||||
`Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks <https://arxiv.org/abs/2005.11401>`_, Patrick Lewis,
|
||||
Ethan Perez, Aleksandara Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau
|
||||
Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela
|
||||
|
||||
Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq models.
|
||||
RAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs.
|
||||
The retriever and seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation to adapt to downstream tasks.
|
||||
Retrieval-augmented generation ("RAG") models combine the powers of pretrained dense retrieval (DPR) and Seq2Seq
|
||||
models. RAG models retrieve docs, pass them to a seq2seq model, then marginalize to generate outputs. The retriever and
|
||||
seq2seq modules are initialized from pretrained models, and fine-tuned jointly, allowing both retrieval and generation
|
||||
to adapt to downstream tasks.
|
||||
|
||||
The two models RAG-Token and RAG-Sequence are available for generation.
|
||||
|
||||
|
|
@ -764,19 +768,19 @@ use a sparse version of the attention matrix to speed up training.
|
|||
**LSH attention**
|
||||
|
||||
:ref:`Reformer <reformer>` uses LSH attention. In the softmax(QK^t), only the biggest elements (in the softmax
|
||||
dimension) of the matrix QK^t are going to give useful contributions. So for each query q in Q, we can consider only
|
||||
dimension) of the matrix QK^t are going to give useful contributions. So for each query q in Q, we can consider only
|
||||
the keys k in K that are close to q. A hash function is used to determine if q and k are close. The attention mask is
|
||||
modified to mask the current token (except at the first position), because it will give a query and a key equal (so very
|
||||
similar to each other). Since the hash can be a bit random, several hash functions are used in practice (determined by
|
||||
a n_rounds parameter) and then are averaged together.
|
||||
modified to mask the current token (except at the first position), because it will give a query and a key equal (so
|
||||
very similar to each other). Since the hash can be a bit random, several hash functions are used in practice
|
||||
(determined by a n_rounds parameter) and then are averaged together.
|
||||
|
||||
.. _local-attention:
|
||||
|
||||
**Local attention**
|
||||
|
||||
:ref:`Longformer <longformer>` uses local attention: often, the local context (e.g., what are the two tokens to the left and
|
||||
right?) is enough to take action for a given token. Also, by stacking attention layers that have a small window, the
|
||||
last layer will have a receptive field of more than just the tokens in the window, allowing them to build a
|
||||
:ref:`Longformer <longformer>` uses local attention: often, the local context (e.g., what are the two tokens to the
|
||||
left and right?) is enough to take action for a given token. Also, by stacking attention layers that have a small
|
||||
window, the last layer will have a receptive field of more than just the tokens in the window, allowing them to build a
|
||||
representation of the whole sentence.
|
||||
|
||||
Some preselected input tokens are also given global attention: for those few tokens, the attention matrix can access
|
||||
|
|
@ -799,8 +803,9 @@ Other tricks
|
|||
|
||||
:ref:`Reformer <reformer>` uses axial positional encodings: in traditional transformer models, the positional encoding
|
||||
E is a matrix of size :math:`l` by :math:`d`, :math:`l` being the sequence length and :math:`d` the dimension of the
|
||||
hidden state. If you have very long texts, this matrix can be huge and take way too much space on the GPU. To alleviate that, axial positional encodings consist of factorizing that big matrix E in two smaller matrices E1 and
|
||||
E2, with dimensions :math:`l_{1} \times d_{1}` and :math:`l_{2} \times d_{2}`, such that :math:`l_{1} \times l_{2} = l`
|
||||
and :math:`d_{1} + d_{2} = d` (with the product for the lengths, this ends up being way smaller). The embedding for
|
||||
time step :math:`j` in E is obtained by concatenating the embeddings for timestep :math:`j \% l1` in E1 and
|
||||
:math:`j // l1` in E2.
|
||||
hidden state. If you have very long texts, this matrix can be huge and take way too much space on the GPU. To alleviate
|
||||
that, axial positional encodings consist of factorizing that big matrix E in two smaller matrices E1 and E2, with
|
||||
dimensions :math:`l_{1} \times d_{1}` and :math:`l_{2} \times d_{2}`, such that :math:`l_{1} \times l_{2} = l` and
|
||||
:math:`d_{1} + d_{2} = d` (with the product for the lengths, this ends up being way smaller). The embedding for time
|
||||
step :math:`j` in E is obtained by concatenating the embeddings for timestep :math:`j \% l1` in E1 and :math:`j // l1`
|
||||
in E2.
|
||||
|
|
|
|||
|
|
@ -1,9 +1,9 @@
|
|||
Multi-lingual models
|
||||
=======================================================================================================================
|
||||
|
||||
Most of the models available in this library are mono-lingual models (English, Chinese and German). A few
|
||||
multi-lingual models are available and have a different mechanisms than mono-lingual models.
|
||||
This page details the usage of these models.
|
||||
Most of the models available in this library are mono-lingual models (English, Chinese and German). A few multi-lingual
|
||||
models are available and have a different mechanisms than mono-lingual models. This page details the usage of these
|
||||
models.
|
||||
|
||||
The two models that currently support multiple languages are BERT and XLM.
|
||||
|
||||
|
|
@ -28,8 +28,8 @@ This section concerns the following checkpoints:
|
|||
|
||||
These checkpoints require language embeddings that will specify the language used at inference time. These language
|
||||
embeddings are represented as a tensor that is of the same shape as the input ids passed to the model. The values in
|
||||
these tensors depend on the language used and are identifiable using the ``lang2id`` and ``id2lang`` attributes
|
||||
from the tokenizer.
|
||||
these tensors depend on the language used and are identifiable using the ``lang2id`` and ``id2lang`` attributes from
|
||||
the tokenizer.
|
||||
|
||||
Here is an example using the ``xlm-clm-enfr-1024`` checkpoint (Causal language modeling, English-French):
|
||||
|
||||
|
|
@ -78,8 +78,9 @@ You can then feed it all as input to your model:
|
|||
>>> outputs = model(input_ids, langs=langs)
|
||||
|
||||
|
||||
The example `run_generation.py <https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py>`__
|
||||
can generate text using the CLM checkpoints from XLM, using the language embeddings.
|
||||
The example `run_generation.py
|
||||
<https://github.com/huggingface/transformers/blob/master/examples/text-generation/run_generation.py>`__ can generate
|
||||
text using the CLM checkpoints from XLM, using the language embeddings.
|
||||
|
||||
XLM without Language Embeddings
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
|
@ -89,8 +90,8 @@ This section concerns the following checkpoints:
|
|||
- ``xlm-mlm-17-1280`` (Masked language modeling, 17 languages)
|
||||
- ``xlm-mlm-100-1280`` (Masked language modeling, 100 languages)
|
||||
|
||||
These checkpoints do not require language embeddings at inference time. These models are used to have generic
|
||||
sentence representations, differently from previously-mentioned XLM checkpoints.
|
||||
These checkpoints do not require language embeddings at inference time. These models are used to have generic sentence
|
||||
representations, differently from previously-mentioned XLM checkpoints.
|
||||
|
||||
|
||||
BERT
|
||||
|
|
@ -101,15 +102,15 @@ BERT has two checkpoints that can be used for multi-lingual tasks:
|
|||
- ``bert-base-multilingual-uncased`` (Masked language modeling + Next sentence prediction, 102 languages)
|
||||
- ``bert-base-multilingual-cased`` (Masked language modeling + Next sentence prediction, 104 languages)
|
||||
|
||||
These checkpoints do not require language embeddings at inference time. They should identify the language
|
||||
used in the context and infer accordingly.
|
||||
These checkpoints do not require language embeddings at inference time. They should identify the language used in the
|
||||
context and infer accordingly.
|
||||
|
||||
XLM-RoBERTa
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
XLM-RoBERTa was trained on 2.5TB of newly created clean CommonCrawl data in 100 languages. It provides strong
|
||||
gains over previously released multi-lingual models like mBERT or XLM on downstream taks like classification,
|
||||
sequence labeling and question answering.
|
||||
XLM-RoBERTa was trained on 2.5TB of newly created clean CommonCrawl data in 100 languages. It provides strong gains
|
||||
over previously released multi-lingual models like mBERT or XLM on downstream taks like classification, sequence
|
||||
labeling and question answering.
|
||||
|
||||
Two XLM-RoBERTa checkpoints can be used for multi-lingual tasks:
|
||||
|
||||
|
|
|
|||
|
|
@ -1,86 +1,69 @@
|
|||
Perplexity of fixed-length models
|
||||
=======================================================================================================================
|
||||
|
||||
Perplexity (PPL) is one of the most common metrics for evaluating language
|
||||
models. Before diving in, we should note that the metric applies specifically
|
||||
to classical language models (sometimes called autoregressive or causal
|
||||
language models) and is not well defined for masked language models like BERT
|
||||
(see :doc:`summary of the models <model_summary>`).
|
||||
Perplexity (PPL) is one of the most common metrics for evaluating language models. Before diving in, we should note
|
||||
that the metric applies specifically to classical language models (sometimes called autoregressive or causal language
|
||||
models) and is not well defined for masked language models like BERT (see :doc:`summary of the models
|
||||
<model_summary>`).
|
||||
|
||||
Perplexity is defined as the exponentiated average log-likelihood of a
|
||||
sequence. If we have a tokenized sequence :math:`X = (x_0, x_1, \dots, x_t)`,
|
||||
then the perplexity of :math:`X` is,
|
||||
Perplexity is defined as the exponentiated average log-likelihood of a sequence. If we have a tokenized sequence
|
||||
:math:`X = (x_0, x_1, \dots, x_t)`, then the perplexity of :math:`X` is,
|
||||
|
||||
.. math::
|
||||
|
||||
\text{PPL}(X)
|
||||
= \exp \left\{ {-\frac{1}{t}\sum_i^t \log p_\theta (x_i|x_{<i}) } \right\}
|
||||
|
||||
where :math:`\log p_\theta (x_i|x_{<i})` is the log-likelihood of the ith
|
||||
token conditioned on the preceding tokens :math:`x_{<i}` according to our
|
||||
model. Intuitively, it can be thought of as an evaluation of the model's
|
||||
ability to predict uniformly among the set of specified tokens in a corpus.
|
||||
Importantly, this means that the tokenization procedure has a direct impact
|
||||
on a model's perplexity which should always be taken into consideration when
|
||||
comparing different models.
|
||||
where :math:`\log p_\theta (x_i|x_{<i})` is the log-likelihood of the ith token conditioned on the preceding tokens
|
||||
:math:`x_{<i}` according to our model. Intuitively, it can be thought of as an evaluation of the model's ability to
|
||||
predict uniformly among the set of specified tokens in a corpus. Importantly, this means that the tokenization
|
||||
procedure has a direct impact on a model's perplexity which should always be taken into consideration when comparing
|
||||
different models.
|
||||
|
||||
This is also equivalent to the exponentiation of the cross-entropy between
|
||||
the data and model predictions. For more intuition about perplexity and its
|
||||
relationship to Bits Per Character (BPC) and data compression, check out this
|
||||
`fantastic blog post on The Gradient
|
||||
<https://thegradient.pub/understanding-evaluation-metrics-for-language-models/>`_.
|
||||
This is also equivalent to the exponentiation of the cross-entropy between the data and model predictions. For more
|
||||
intuition about perplexity and its relationship to Bits Per Character (BPC) and data compression, check out this
|
||||
`fantastic blog post on The Gradient <https://thegradient.pub/understanding-evaluation-metrics-for-language-models/>`_.
|
||||
|
||||
Calculating PPL with fixed-length models
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
If we weren't limited by a model's context size, we would evaluate the
|
||||
model's perplexity by autoregressively factorizing a sequence and
|
||||
conditioning on the entire preceding subsequence at each step, as shown
|
||||
below.
|
||||
If we weren't limited by a model's context size, we would evaluate the model's perplexity by autoregressively
|
||||
factorizing a sequence and conditioning on the entire preceding subsequence at each step, as shown below.
|
||||
|
||||
.. image:: imgs/ppl_full.gif
|
||||
:width: 600
|
||||
:alt: Full decomposition of a sequence with unlimited context length
|
||||
|
||||
When working with approximate models, however, we typically have a constraint
|
||||
on the number of tokens the model can process. The largest version
|
||||
of :doc:`GPT-2 <model_doc/gpt2>`, for example, has a fixed length of 1024
|
||||
tokens, so we cannot calculate :math:`p_\theta(x_t|x_{<t})` directly when
|
||||
:math:`t` is greater than 1024.
|
||||
When working with approximate models, however, we typically have a constraint on the number of tokens the model can
|
||||
process. The largest version of :doc:`GPT-2 <model_doc/gpt2>`, for example, has a fixed length of 1024 tokens, so we
|
||||
cannot calculate :math:`p_\theta(x_t|x_{<t})` directly when :math:`t` is greater than 1024.
|
||||
|
||||
Instead, the sequence is typically broken into subsequences equal to the
|
||||
model's maximum input size. If a model's max input size is :math:`k`, we
|
||||
then approximate the likelihood of a token :math:`x_t` by conditioning only
|
||||
on the :math:`k-1` tokens that precede it rather than the entire context.
|
||||
When evaluating the model's perplexity of a sequence, a tempting but
|
||||
suboptimal approach is to break the sequence into disjoint chunks and
|
||||
add up the decomposed log-likelihoods of each segment independently.
|
||||
Instead, the sequence is typically broken into subsequences equal to the model's maximum input size. If a model's max
|
||||
input size is :math:`k`, we then approximate the likelihood of a token :math:`x_t` by conditioning only on the
|
||||
:math:`k-1` tokens that precede it rather than the entire context. When evaluating the model's perplexity of a
|
||||
sequence, a tempting but suboptimal approach is to break the sequence into disjoint chunks and add up the decomposed
|
||||
log-likelihoods of each segment independently.
|
||||
|
||||
.. image:: imgs/ppl_chunked.gif
|
||||
:width: 600
|
||||
:alt: Suboptimal PPL not taking advantage of full available context
|
||||
|
||||
This is quick to compute since the perplexity of each segment can be computed
|
||||
in one forward pass, but serves as a poor approximation of the
|
||||
fully-factorized perplexity and will typically yield a higher (worse) PPL
|
||||
because the model will have less context at most of the prediction steps.
|
||||
This is quick to compute since the perplexity of each segment can be computed in one forward pass, but serves as a poor
|
||||
approximation of the fully-factorized perplexity and will typically yield a higher (worse) PPL because the model will
|
||||
have less context at most of the prediction steps.
|
||||
|
||||
Instead, the PPL of fixed-length models should be evaluated with a
|
||||
sliding-window strategy. This involves repeatedly sliding the
|
||||
context window so that the model has more context when making each
|
||||
prediction.
|
||||
Instead, the PPL of fixed-length models should be evaluated with a sliding-window strategy. This involves repeatedly
|
||||
sliding the context window so that the model has more context when making each prediction.
|
||||
|
||||
.. image:: imgs/ppl_sliding.gif
|
||||
:width: 600
|
||||
:alt: Sliding window PPL taking advantage of all available context
|
||||
|
||||
This is a closer approximation to the true decomposition of the
|
||||
sequence probability and will typically yield a more favorable score.
|
||||
The downside is that it requires a separate forward pass for each token in
|
||||
the corpus. A good practical compromise is to employ a strided sliding
|
||||
window, moving the context by larger strides rather than sliding by 1 token a
|
||||
time. This allows computation to procede much faster while still giving the
|
||||
model a large context to make predictions at each step.
|
||||
This is a closer approximation to the true decomposition of the sequence probability and will typically yield a more
|
||||
favorable score. The downside is that it requires a separate forward pass for each token in the corpus. A good
|
||||
practical compromise is to employ a strided sliding window, moving the context by larger strides rather than sliding by
|
||||
1 token a time. This allows computation to procede much faster while still giving the model a large context to make
|
||||
predictions at each step.
|
||||
|
||||
Example: Calculating perplexity with GPT-2 in 🤗 Transformers
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
|
@ -95,10 +78,9 @@ Let's demonstrate this process with GPT-2.
|
|||
model = GPT2LMHeadModel.from_pretrained(model_id).to(device)
|
||||
tokenizer = GPT2TokenizerFast.from_pretrained(model_id)
|
||||
|
||||
We'll load in the WikiText-2 dataset and evaluate the perplexity using a few
|
||||
different sliding-window strategies. Since this dataset is small and we're
|
||||
just doing one forward pass over the set, we can just load and encode the
|
||||
entire dataset in memory.
|
||||
We'll load in the WikiText-2 dataset and evaluate the perplexity using a few different sliding-window strategies. Since
|
||||
this dataset is small and we're just doing one forward pass over the set, we can just load and encode the entire
|
||||
dataset in memory.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -106,16 +88,13 @@ entire dataset in memory.
|
|||
test = load_dataset('wikitext', 'wikitext-2-raw-v1', split='test')
|
||||
encodings = tokenizer('\n\n'.join(test['text']), return_tensors='pt')
|
||||
|
||||
With 🤗 Transformers, we can simply pass the ``input_ids`` as the ``labels``
|
||||
to our model, and the average log-likelihood for each token is returned as
|
||||
the loss. With our sliding window approach, however, there is overlap in the
|
||||
tokens we pass to the model at each iteration. We don't want the
|
||||
log-likelihood for the tokens we're just treating as context to be included
|
||||
in our loss, so we can set these targets to ``-100`` so that they are
|
||||
ignored. The following is an example of how we could do this with a stride of
|
||||
``512``. This means that the model will have at least 512 tokens for context
|
||||
when calculating the conditional likelihood of any one token (provided there
|
||||
are 512 preceding tokens available to condition on).
|
||||
With 🤗 Transformers, we can simply pass the ``input_ids`` as the ``labels`` to our model, and the average
|
||||
log-likelihood for each token is returned as the loss. With our sliding window approach, however, there is overlap in
|
||||
the tokens we pass to the model at each iteration. We don't want the log-likelihood for the tokens we're just treating
|
||||
as context to be included in our loss, so we can set these targets to ``-100`` so that they are ignored. The following
|
||||
is an example of how we could do this with a stride of ``512``. This means that the model will have at least 512 tokens
|
||||
for context when calculating the conditional likelihood of any one token (provided there are 512 preceding tokens
|
||||
available to condition on).
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -139,14 +118,11 @@ are 512 preceding tokens available to condition on).
|
|||
|
||||
ppl = torch.exp(torch.stack(lls).sum() / end_loc)
|
||||
|
||||
Running this with the stride length equal to the max input length is
|
||||
equivalent to the suboptimal, non-sliding-window strategy we discussed above.
|
||||
The smaller the stride, the more context the model will have in making each
|
||||
prediction, and the better the reported perplexity will typically be.
|
||||
Running this with the stride length equal to the max input length is equivalent to the suboptimal, non-sliding-window
|
||||
strategy we discussed above. The smaller the stride, the more context the model will have in making each prediction,
|
||||
and the better the reported perplexity will typically be.
|
||||
|
||||
When we run the above with ``stride = 1024``, i.e. no overlap, the resulting
|
||||
PPL is ``19.64``, which is about the same as the ``19.93`` reported in the
|
||||
GPT-2 paper. By using ``stride = 512`` and thereby employing our striding
|
||||
window strategy, this jumps down to ``16.53``. This is not only a more
|
||||
favorable score, but is calculated in a way that is closer to the true
|
||||
autoregressive decomposition of a sequence likelihood.
|
||||
When we run the above with ``stride = 1024``, i.e. no overlap, the resulting PPL is ``19.64``, which is about the same
|
||||
as the ``19.93`` reported in the GPT-2 paper. By using ``stride = 512`` and thereby employing our striding window
|
||||
strategy, this jumps down to ``16.53``. This is not only a more favorable score, but is calculated in a way that is
|
||||
closer to the true autoregressive decomposition of a sequence likelihood.
|
||||
|
|
|
|||
|
|
@ -12,15 +12,15 @@ The library was designed with two strong goals in mind:
|
|||
- Be as easy and fast to use as possible:
|
||||
|
||||
- We strongly limited the number of user-facing abstractions to learn, in fact, there are almost no abstractions,
|
||||
just three standard classes required to use each model: :doc:`configuration <main_classes/configuration>`,
|
||||
just three standard classes required to use each model: :doc:`configuration <main_classes/configuration>`,
|
||||
:doc:`models <main_classes/model>` and :doc:`tokenizer <main_classes/tokenizer>`.
|
||||
- All of these classes can be initialized in a simple and unified way from pretrained instances by using a common
|
||||
:obj:`from_pretrained()` instantiation method which will take care of downloading (if needed), caching and
|
||||
loading the related class instance and associated data (configurations' hyper-parameters, tokenizers' vocabulary,
|
||||
and models' weights) from a pretrained checkpoint provided on
|
||||
`Hugging Face Hub <https://huggingface.co/models>`__ or your own saved checkpoint.
|
||||
loading the related class instance and associated data (configurations' hyper-parameters, tokenizers' vocabulary,
|
||||
and models' weights) from a pretrained checkpoint provided on `Hugging Face Hub
|
||||
<https://huggingface.co/models>`__ or your own saved checkpoint.
|
||||
- On top of those three base classes, the library provides two APIs: :func:`~transformers.pipeline` for quickly
|
||||
using a model (plus its associated tokenizer and configuration) on a given task and
|
||||
using a model (plus its associated tokenizer and configuration) on a given task and
|
||||
:func:`~transformers.Trainer`/:func:`~transformers.TFTrainer` to quickly train or fine-tune a given model.
|
||||
- As a consequence, this library is NOT a modular toolbox of building blocks for neural nets. If you want to
|
||||
extend/build-upon the library, just use regular Python/PyTorch/TensorFlow/Keras modules and inherit from the base
|
||||
|
|
@ -52,10 +52,10 @@ Main concepts
|
|||
|
||||
The library is built around three types of classes for each model:
|
||||
|
||||
- **Model classes** such as :class:`~transformers.BertModel`, which are 30+ PyTorch models
|
||||
(`torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`__) or Keras models
|
||||
(`tf.keras.Model <https://www.tensorflow.org/api_docs/python/tf/keras/Model>`__) that work with the pretrained
|
||||
weights provided in the library.
|
||||
- **Model classes** such as :class:`~transformers.BertModel`, which are 30+ PyTorch models (`torch.nn.Module
|
||||
<https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`__) or Keras models (`tf.keras.Model
|
||||
<https://www.tensorflow.org/api_docs/python/tf/keras/Model>`__) that work with the pretrained weights provided in the
|
||||
library.
|
||||
- **Configuration classes** such as :class:`~transformers.BertConfig`, which store all the parameters required to build
|
||||
a model. You don't always need to instantiate these yourself. In particular, if you are using a pretrained model
|
||||
without any modification, creating the model will automatically take care of instantiating the configuration (which
|
||||
|
|
@ -66,8 +66,8 @@ The library is built around three types of classes for each model:
|
|||
All these classes can be instantiated from pretrained instances and saved locally using two methods:
|
||||
|
||||
- :obj:`from_pretrained()` lets you instantiate a model/configuration/tokenizer from a pretrained version either
|
||||
provided by the library itself (the supported models are provided in the list :doc:`here <pretrained_models>`
|
||||
or stored locally (or on a server) by the user,
|
||||
provided by the library itself (the supported models are provided in the list :doc:`here <pretrained_models>` or
|
||||
stored locally (or on a server) by the user,
|
||||
- :obj:`save_pretrained()` lets you save a model/configuration/tokenizer locally so that it can be reloaded using
|
||||
:obj:`from_pretrained()`.
|
||||
|
||||
|
|
|
|||
|
|
@ -17,7 +17,7 @@ work properly.
|
|||
the text you give it in tokens the same way for the pretraining corpus, and it will use the same correspondence
|
||||
token to index (that we usually call a `vocab`) as during pretraining.
|
||||
|
||||
To automatically download the vocab used during pretraining or fine-tuning a given model, you can use the
|
||||
To automatically download the vocab used during pretraining or fine-tuning a given model, you can use the
|
||||
:func:`~transformers.AutoTokenizer.from_pretrained` method:
|
||||
|
||||
.. code-block::
|
||||
|
|
@ -39,10 +39,10 @@ is its ``__call__``: you just need to feed your sentence to your tokenizer objec
|
|||
'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
|
||||
'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
|
||||
|
||||
This returns a dictionary string to list of ints.
|
||||
The `input_ids <glossary.html#input-ids>`__ are the indices corresponding to each token in our sentence. We will see
|
||||
below what the `attention_mask <glossary.html#attention-mask>`__ is used for and in
|
||||
:ref:`the next section <sentence-pairs>` the goal of `token_type_ids <glossary.html#token-type-ids>`__.
|
||||
This returns a dictionary string to list of ints. The `input_ids <glossary.html#input-ids>`__ are the indices
|
||||
corresponding to each token in our sentence. We will see below what the `attention_mask
|
||||
<glossary.html#attention-mask>`__ is used for and in :ref:`the next section <sentence-pairs>` the goal of
|
||||
`token_type_ids <glossary.html#token-type-ids>`__.
|
||||
|
||||
The tokenizer can decode a list of token ids in a proper sentence:
|
||||
|
||||
|
|
@ -51,10 +51,10 @@ The tokenizer can decode a list of token ids in a proper sentence:
|
|||
>>> tokenizer.decode(encoded_input["input_ids"])
|
||||
"[CLS] Hello, I'm a single sentence! [SEP]"
|
||||
|
||||
As you can see, the tokenizer automatically added some special tokens that the model expects. Not all models need special
|
||||
tokens; for instance, if we had used` gtp2-medium` instead of `bert-base-cased` to create our tokenizer, we would have
|
||||
seen the same sentence as the original one here. You can disable this behavior (which is only advised if you have added
|
||||
those special tokens yourself) by passing ``add_special_tokens=False``.
|
||||
As you can see, the tokenizer automatically added some special tokens that the model expects. Not all models need
|
||||
special tokens; for instance, if we had used` gtp2-medium` instead of `bert-base-cased` to create our tokenizer, we
|
||||
would have seen the same sentence as the original one here. You can disable this behavior (which is only advised if you
|
||||
have added those special tokens yourself) by passing ``add_special_tokens=False``.
|
||||
|
||||
If you have several sentences you want to process, you can do this efficiently by sending them as a list to the
|
||||
tokenizer:
|
||||
|
|
@ -114,9 +114,9 @@ You can do all of this by using the following options when feeding your list of
|
|||
[1, 1, 1, 1, 1, 0, 0, 0, 0],
|
||||
[1, 1, 1, 1, 1, 1, 1, 1, 0]])}
|
||||
|
||||
It returns a dictionary with string keys and tensor values. We can now see what the `attention_mask <glossary.html#attention-mask>`__ is
|
||||
all about: it points out which tokens the model should pay attention to and which ones it should not (because they
|
||||
represent padding in this case).
|
||||
It returns a dictionary with string keys and tensor values. We can now see what the `attention_mask
|
||||
<glossary.html#attention-mask>`__ is all about: it points out which tokens the model should pay attention to and which
|
||||
ones it should not (because they represent padding in this case).
|
||||
|
||||
|
||||
Note that if your model does not have a maximum length associated to it, the command above will throw a warning. You
|
||||
|
|
@ -127,9 +127,9 @@ can safely ignore it. You can also pass ``verbose=False`` to stop the tokenizer
|
|||
Preprocessing pairs of sentences
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Sometimes you need to feed a pair of sentences to your model. For instance, if you want to classify if two sentences in a
|
||||
pair are similar, or for question-answering models, which take a context and a question. For BERT models, the input is
|
||||
then represented like this: :obj:`[CLS] Sequence A [SEP] Sequence B [SEP]`
|
||||
Sometimes you need to feed a pair of sentences to your model. For instance, if you want to classify if two sentences in
|
||||
a pair are similar, or for question-answering models, which take a context and a question. For BERT models, the input
|
||||
is then represented like this: :obj:`[CLS] Sequence A [SEP] Sequence B [SEP]`
|
||||
|
||||
You can encode a pair of sentences in the format expected by your model by supplying the two sentences as two arguments
|
||||
(not a list since a list of two sentences will be interpreted as a batch of two single sentences, as we saw before).
|
||||
|
|
@ -146,8 +146,8 @@ This will once again return a dict string to list of ints:
|
|||
This shows us what the `token_type_ids <glossary.html#token-type-ids>`__ are for: they indicate to the model which part
|
||||
of the inputs correspond to the first sentence and which part corresponds to the second sentence. Note that
|
||||
`token_type_ids` are not required or handled by all models. By default, a tokenizer will only return the inputs that
|
||||
its associated model expects. You can force the return (or the non-return) of any of those special arguments by
|
||||
using ``return_input_ids`` or ``return_token_type_ids``.
|
||||
its associated model expects. You can force the return (or the non-return) of any of those special arguments by using
|
||||
``return_input_ids`` or ``return_token_type_ids``.
|
||||
|
||||
If we decode the token ids we obtained, we will see that the special tokens have been properly added.
|
||||
|
||||
|
|
@ -215,7 +215,7 @@ three arguments you need to know for this are :obj:`padding`, :obj:`truncation`
|
|||
a single sequence).
|
||||
- :obj:`'max_length'` to pad to a length specified by the :obj:`max_length` argument or the maximum length accepted
|
||||
by the model if no :obj:`max_length` is provided (``max_length=None``). If you only provide a single sequence,
|
||||
padding will still be applied to it.
|
||||
padding will still be applied to it.
|
||||
- :obj:`False` or :obj:`'do_not_pad'` to not pad the sequences. As we have seen before, this is the default
|
||||
behavior.
|
||||
|
||||
|
|
@ -238,9 +238,9 @@ three arguments you need to know for this are :obj:`padding`, :obj:`truncation`
|
|||
truncation/padding to :obj:`max_length` is deactivated.
|
||||
|
||||
Here is a table summarizing the recommend way to setup padding and truncation. If you use pair of inputs sequence in
|
||||
any of the following examples, you can replace :obj:`truncation=True` by a :obj:`STRATEGY` selected in
|
||||
:obj:`['only_first', 'only_second', 'longest_first']`, i.e. :obj:`truncation='only_second'` or
|
||||
:obj:`truncation= 'longest_first'` to control how both sequence in the pair are truncated as detailed before.
|
||||
any of the following examples, you can replace :obj:`truncation=True` by a :obj:`STRATEGY` selected in
|
||||
:obj:`['only_first', 'only_second', 'longest_first']`, i.e. :obj:`truncation='only_second'` or :obj:`truncation=
|
||||
'longest_first'` to control how both sequence in the pair are truncated as detailed before.
|
||||
|
||||
+--------------------------------------+-----------------------------------+---------------------------------------------------------------------------------------------+
|
||||
| Truncation | Padding | Instruction |
|
||||
|
|
|
|||
|
|
@ -3,7 +3,8 @@ Pretrained models
|
|||
|
||||
Here is the full list of the currently provided pretrained models together with a short presentation of each model.
|
||||
|
||||
For a list that includes community-uploaded models, refer to `https://huggingface.co/models <https://huggingface.co/models>`__.
|
||||
For a list that includes community-uploaded models, refer to `https://huggingface.co/models
|
||||
<https://huggingface.co/models>`__.
|
||||
|
||||
+--------------------+------------------------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------+
|
||||
| Architecture | Shortcut name | Details of the model |
|
||||
|
|
|
|||
|
|
@ -1,8 +1,8 @@
|
|||
Quick tour
|
||||
=======================================================================================================================
|
||||
|
||||
Let's have a quick look at the 🤗 Transformers library features. The library downloads pretrained models for
|
||||
Natural Language Understanding (NLU) tasks, such as analyzing the sentiment of a text, and Natural Language Generation (NLG),
|
||||
Let's have a quick look at the 🤗 Transformers library features. The library downloads pretrained models for Natural
|
||||
Language Understanding (NLU) tasks, such as analyzing the sentiment of a text, and Natural Language Generation (NLG),
|
||||
such as completing a prompt with new text or translating in another language.
|
||||
|
||||
First we will see how to easily leverage the pipeline API to quickly use those pretrained models at inference. Then, we
|
||||
|
|
@ -29,8 +29,8 @@ provides the following tasks out of the box:
|
|||
- Translation: translate a text in another language.
|
||||
- Feature extraction: return a tensor representation of the text.
|
||||
|
||||
Let's see how this work for sentiment analysis (the other tasks are all covered in the
|
||||
:doc:`task summary </task_summary>`):
|
||||
Let's see how this work for sentiment analysis (the other tasks are all covered in the :doc:`task summary
|
||||
</task_summary>`):
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -160,9 +160,10 @@ To apply these steps on a given text, we can just feed it to our tokenizer:
|
|||
|
||||
>>> inputs = tokenizer("We are very happy to show you the 🤗 Transformers library.")
|
||||
|
||||
This returns a dictionary string to list of ints. It contains the `ids of the tokens <glossary.html#input-ids>`__,
|
||||
as mentioned before, but also additional arguments that will be useful to the model. Here for instance, we also have an
|
||||
`attention mask <glossary.html#attention-mask>`__ that the model will use to have a better understanding of the sequence:
|
||||
This returns a dictionary string to list of ints. It contains the `ids of the tokens <glossary.html#input-ids>`__, as
|
||||
mentioned before, but also additional arguments that will be useful to the model. Here for instance, we also have an
|
||||
`attention mask <glossary.html#attention-mask>`__ that the model will use to have a better understanding of the
|
||||
sequence:
|
||||
|
||||
|
||||
.. code-block::
|
||||
|
|
@ -191,8 +192,8 @@ and get tensors back. You can specify all of that to the tokenizer:
|
|||
... return_tensors="tf"
|
||||
... )
|
||||
|
||||
The padding is automatically applied on the side expected by the model (in this case, on the right), with the
|
||||
padding token the model was pretrained with. The attention mask is also adapted to take the padding into account:
|
||||
The padding is automatically applied on the side expected by the model (in this case, on the right), with the padding
|
||||
token the model was pretrained with. The attention mask is also adapted to take the padding into account:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -213,8 +214,8 @@ Using the model
|
|||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Once your input has been preprocessed by the tokenizer, you can send it directly to the model. As we mentioned, it will
|
||||
contain all the relevant information the model needs. If you're using a TensorFlow model, you can pass the
|
||||
dictionary keys directly to tensors, for a PyTorch model, you need to unpack the dictionary by adding :obj:`**`.
|
||||
contain all the relevant information the model needs. If you're using a TensorFlow model, you can pass the dictionary
|
||||
keys directly to tensors, for a PyTorch model, you need to unpack the dictionary by adding :obj:`**`.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -223,8 +224,8 @@ dictionary keys directly to tensors, for a PyTorch model, you need to unpack the
|
|||
>>> ## TENSORFLOW CODE
|
||||
>>> tf_outputs = tf_model(tf_batch)
|
||||
|
||||
In 🤗 Transformers, all outputs are tuples (with only one element potentially). Here, we get a tuple with just the
|
||||
final activations of the model.
|
||||
In 🤗 Transformers, all outputs are tuples (with only one element potentially). Here, we get a tuple with just the final
|
||||
activations of the model.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -239,11 +240,10 @@ final activations of the model.
|
|||
[ 0.08181786, -0.04179301]], dtype=float32)>,)
|
||||
|
||||
The model can return more than just the final activations, which is why the output is a tuple. Here we only asked for
|
||||
the final activations, so we get a tuple with one element.
|
||||
.. note::
|
||||
the final activations, so we get a tuple with one element. .. note::
|
||||
|
||||
All 🤗 Transformers models (PyTorch or TensorFlow) return the activations of the model *before* the final
|
||||
activation function (like SoftMax) since this final activation function is often fused with the loss.
|
||||
All 🤗 Transformers models (PyTorch or TensorFlow) return the activations of the model *before* the final activation
|
||||
function (like SoftMax) since this final activation function is often fused with the loss.
|
||||
|
||||
Let's apply the SoftMax activation to get predictions.
|
||||
|
||||
|
|
@ -281,11 +281,11 @@ If you have labels, you can provide them to the model, it will return a tuple wi
|
|||
>>> import tensorflow as tf
|
||||
>>> tf_outputs = tf_model(tf_batch, labels = tf.constant([1, 0]))
|
||||
|
||||
Models are standard `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`__ or
|
||||
`tf.keras.Model <https://www.tensorflow.org/api_docs/python/tf/keras/Model>`__ so you can use them in your usual
|
||||
training loop. 🤗 Transformers also provides a :class:`~transformers.Trainer` (or :class:`~transformers.TFTrainer` if
|
||||
you are using TensorFlow) class to help with your training (taking care of things such as distributed training, mixed
|
||||
precision, etc.). See the :doc:`training tutorial <training>` for more details.
|
||||
Models are standard `torch.nn.Module <https://pytorch.org/docs/stable/nn.html#torch.nn.Module>`__ or `tf.keras.Model
|
||||
<https://www.tensorflow.org/api_docs/python/tf/keras/Model>`__ so you can use them in your usual training loop. 🤗
|
||||
Transformers also provides a :class:`~transformers.Trainer` (or :class:`~transformers.TFTrainer` if you are using
|
||||
TensorFlow) class to help with your training (taking care of things such as distributed training, mixed precision,
|
||||
etc.). See the :doc:`training tutorial <training>` for more details.
|
||||
|
||||
.. note::
|
||||
|
||||
|
|
@ -336,13 +336,13 @@ The :obj:`AutoModel` and :obj:`AutoTokenizer` classes are just shortcuts that wi
|
|||
pretrained model. Behind the scenes, the library has one model class per combination of architecture plus class, so the
|
||||
code is easy to access and tweak if you need to.
|
||||
|
||||
In our previous example, the model was called "distilbert-base-uncased-finetuned-sst-2-english", which means it's
|
||||
using the :doc:`DistilBERT </model_doc/distilbert>` architecture. As
|
||||
:class:`~transformers.AutoModelForSequenceClassification` (or :class:`~transformers.TFAutoModelForSequenceClassification`
|
||||
if you are using TensorFlow) was used, the model automatically created is then a
|
||||
:class:`~transformers.DistilBertForSequenceClassification`. You can look at its documentation for all details relevant
|
||||
to that specific model, or browse the source code. This is how you would directly instantiate model and tokenizer
|
||||
without the auto magic:
|
||||
In our previous example, the model was called "distilbert-base-uncased-finetuned-sst-2-english", which means it's using
|
||||
the :doc:`DistilBERT </model_doc/distilbert>` architecture. As
|
||||
:class:`~transformers.AutoModelForSequenceClassification` (or
|
||||
:class:`~transformers.TFAutoModelForSequenceClassification` if you are using TensorFlow) was used, the model
|
||||
automatically created is then a :class:`~transformers.DistilBertForSequenceClassification`. You can look at its
|
||||
documentation for all details relevant to that specific model, or browse the source code. This is how you would
|
||||
directly instantiate model and tokenizer without the auto magic:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
|
|||
|
|
@ -5,16 +5,18 @@ Exporting transformers models
|
|||
ONNX / ONNXRuntime
|
||||
=======================================================================================================================
|
||||
|
||||
Projects `ONNX (Open Neural Network eXchange) <http://onnx.ai>`_ and `ONNXRuntime (ORT) <https://microsoft.github.io/onnxruntime/>`_ are part of an effort from leading industries in the AI field
|
||||
to provide a unified and community-driven format to store and, by extension, efficiently execute neural network leveraging a variety
|
||||
Projects `ONNX (Open Neural Network eXchange) <http://onnx.ai>`_ and `ONNXRuntime (ORT)
|
||||
<https://microsoft.github.io/onnxruntime/>`_ are part of an effort from leading industries in the AI field to provide a
|
||||
unified and community-driven format to store and, by extension, efficiently execute neural network leveraging a variety
|
||||
of hardware and dedicated optimizations.
|
||||
|
||||
Starting from transformers v2.10.0 we partnered with ONNX Runtime to provide an easy export of transformers models to
|
||||
the ONNX format. You can have a look at the effort by looking at our joint blog post `Accelerate your NLP pipelines using
|
||||
Hugging Face Transformers and ONNX Runtime <https://medium.com/microsoftazure/accelerate-your-nlp-pipelines-using-hugging-face-transformers-and-onnx-runtime-2443578f4333>`_.
|
||||
the ONNX format. You can have a look at the effort by looking at our joint blog post `Accelerate your NLP pipelines
|
||||
using Hugging Face Transformers and ONNX Runtime
|
||||
<https://medium.com/microsoftazure/accelerate-your-nlp-pipelines-using-hugging-face-transformers-and-onnx-runtime-2443578f4333>`_.
|
||||
|
||||
Exporting a model is done through the script `convert_graph_to_onnx.py` at the root of the transformers sources.
|
||||
The following command shows how easy it is to export a BERT model from the library, simply run:
|
||||
Exporting a model is done through the script `convert_graph_to_onnx.py` at the root of the transformers sources. The
|
||||
following command shows how easy it is to export a BERT model from the library, simply run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -27,62 +29,66 @@ The conversion tool works for both PyTorch and Tensorflow models and ensures:
|
|||
* The generated model can be correctly loaded through onnxruntime.
|
||||
|
||||
.. note::
|
||||
Currently, inputs and outputs are always exported with dynamic sequence axes preventing some optimizations
|
||||
on the ONNX Runtime. If you would like to see such support for fixed-length inputs/outputs, please
|
||||
open up an issue on transformers.
|
||||
Currently, inputs and outputs are always exported with dynamic sequence axes preventing some optimizations on the
|
||||
ONNX Runtime. If you would like to see such support for fixed-length inputs/outputs, please open up an issue on
|
||||
transformers.
|
||||
|
||||
|
||||
Also, the conversion tool supports different options which let you tune the behavior of the generated model:
|
||||
|
||||
* **Change the target opset version of the generated model.** (More recent opset generally supports more operators and enables faster inference)
|
||||
* **Change the target opset version of the generated model.** (More recent opset generally supports more operators and
|
||||
enables faster inference)
|
||||
|
||||
* **Export pipeline-specific prediction heads.** (Allow to export model along with its task-specific prediction head(s))
|
||||
* **Export pipeline-specific prediction heads.** (Allow to export model along with its task-specific prediction
|
||||
head(s))
|
||||
|
||||
* **Use the external data format (PyTorch only).** (Lets you export model which size is above 2Gb (`More info <https://github.com/pytorch/pytorch/pull/33062>`_))
|
||||
* **Use the external data format (PyTorch only).** (Lets you export model which size is above 2Gb (`More info
|
||||
<https://github.com/pytorch/pytorch/pull/33062>`_))
|
||||
|
||||
|
||||
Optimizations
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
ONNXRuntime includes some transformers-specific transformations to leverage optimized operations in the graph.
|
||||
Below are some of the operators which can be enabled to speed up inference through ONNXRuntime (*see note below*):
|
||||
ONNXRuntime includes some transformers-specific transformations to leverage optimized operations in the graph. Below
|
||||
are some of the operators which can be enabled to speed up inference through ONNXRuntime (*see note below*):
|
||||
|
||||
* Constant folding
|
||||
* Attention Layer fusing
|
||||
* Skip connection LayerNormalization fusing
|
||||
* FastGeLU approximation
|
||||
|
||||
Some of the optimizations performed by ONNX runtime can be hardware specific and thus lead to different performances
|
||||
if used on another machine with a different hardware configuration than the one used for exporting the model.
|
||||
For this reason, when using ``convert_graph_to_onnx.py`` optimizations are not enabled,
|
||||
ensuring the model can be easily exported to various hardware.
|
||||
Optimizations can then be enabled when loading the model through ONNX runtime for inference.
|
||||
Some of the optimizations performed by ONNX runtime can be hardware specific and thus lead to different performances if
|
||||
used on another machine with a different hardware configuration than the one used for exporting the model. For this
|
||||
reason, when using ``convert_graph_to_onnx.py`` optimizations are not enabled, ensuring the model can be easily
|
||||
exported to various hardware. Optimizations can then be enabled when loading the model through ONNX runtime for
|
||||
inference.
|
||||
|
||||
|
||||
.. note::
|
||||
When quantization is enabled (see below), ``convert_graph_to_onnx.py`` script will enable optimizations on the model
|
||||
because quantization would modify the underlying graph making it impossible for ONNX runtime to do the optimizations
|
||||
afterwards.
|
||||
When quantization is enabled (see below), ``convert_graph_to_onnx.py`` script will enable optimizations on the
|
||||
model because quantization would modify the underlying graph making it impossible for ONNX runtime to do the
|
||||
optimizations afterwards.
|
||||
|
||||
.. note::
|
||||
For more information about the optimizations enabled by ONNXRuntime, please have a look at the (`ONNXRuntime Github <https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers>`_)
|
||||
For more information about the optimizations enabled by ONNXRuntime, please have a look at the (`ONNXRuntime Github
|
||||
<https://github.com/microsoft/onnxruntime/tree/master/onnxruntime/python/tools/transformers>`_)
|
||||
|
||||
Quantization
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
ONNX exporter supports generating a quantized version of the model to allow efficient inference.
|
||||
|
||||
Quantization works by converting the memory representation of the parameters in the neural network
|
||||
to a compact integer format. By default, weights of a neural network are stored as single-precision float (`float32`)
|
||||
which can express a wide-range of floating-point numbers with decent precision.
|
||||
These properties are especially interesting at training where you want fine-grained representation.
|
||||
Quantization works by converting the memory representation of the parameters in the neural network to a compact integer
|
||||
format. By default, weights of a neural network are stored as single-precision float (`float32`) which can express a
|
||||
wide-range of floating-point numbers with decent precision. These properties are especially interesting at training
|
||||
where you want fine-grained representation.
|
||||
|
||||
On the other hand, after the training phase, it has been shown one can greatly reduce the range and the precision of `float32` numbers
|
||||
without changing the performances of the neural network.
|
||||
On the other hand, after the training phase, it has been shown one can greatly reduce the range and the precision of
|
||||
`float32` numbers without changing the performances of the neural network.
|
||||
|
||||
More technically, `float32` parameters are converted to a type requiring fewer bits to represent each number, thus reducing
|
||||
the overall size of the model. Here, we are enabling `float32` mapping to `int8` values (a non-floating, single byte, number representation)
|
||||
according to the following formula:
|
||||
More technically, `float32` parameters are converted to a type requiring fewer bits to represent each number, thus
|
||||
reducing the overall size of the model. Here, we are enabling `float32` mapping to `int8` values (a non-floating,
|
||||
single byte, number representation) according to the following formula:
|
||||
|
||||
.. math::
|
||||
y_{float32} = scale * x_{int8} - zero\_point
|
||||
|
|
@ -96,9 +102,9 @@ Leveraging tiny-integers has numerous advantages when it comes to inference:
|
|||
* Integer operations execute a magnitude faster on modern hardware
|
||||
* Integer operations require less power to do the computations
|
||||
|
||||
In order to convert a transformers model to ONNX IR with quantized weights you just need to specify ``--quantize``
|
||||
when using ``convert_graph_to_onnx.py``. Also, you can have a look at the ``quantize()`` utility-method in this
|
||||
same script file.
|
||||
In order to convert a transformers model to ONNX IR with quantized weights you just need to specify ``--quantize`` when
|
||||
using ``convert_graph_to_onnx.py``. Also, you can have a look at the ``quantize()`` utility-method in this same script
|
||||
file.
|
||||
|
||||
Example of quantized BERT model export:
|
||||
|
||||
|
|
@ -111,26 +117,27 @@ Example of quantized BERT model export:
|
|||
|
||||
.. note::
|
||||
When exporting quantized model you will end up with two different ONNX files. The one specified at the end of the
|
||||
above command will contain the original ONNX model storing `float32` weights.
|
||||
The second one, with ``-quantized`` suffix, will hold the quantized parameters.
|
||||
above command will contain the original ONNX model storing `float32` weights. The second one, with ``-quantized``
|
||||
suffix, will hold the quantized parameters.
|
||||
|
||||
|
||||
TorchScript
|
||||
=======================================================================================================================
|
||||
|
||||
.. note::
|
||||
This is the very beginning of our experiments with TorchScript and we are still exploring its capabilities
|
||||
with variable-input-size models. It is a focus of interest to us and we will deepen our analysis in upcoming
|
||||
releases, with more code examples, a more flexible implementation, and benchmarks comparing python-based codes
|
||||
with compiled TorchScript.
|
||||
This is the very beginning of our experiments with TorchScript and we are still exploring its capabilities with
|
||||
variable-input-size models. It is a focus of interest to us and we will deepen our analysis in upcoming releases,
|
||||
with more code examples, a more flexible implementation, and benchmarks comparing python-based codes with compiled
|
||||
TorchScript.
|
||||
|
||||
|
||||
According to Pytorch's documentation: "TorchScript is a way to create serializable and optimizable models from PyTorch code".
|
||||
Pytorch's two modules `JIT and TRACE <https://pytorch.org/docs/stable/jit.html>`_ allow the developer to export
|
||||
According to Pytorch's documentation: "TorchScript is a way to create serializable and optimizable models from PyTorch
|
||||
code". Pytorch's two modules `JIT and TRACE <https://pytorch.org/docs/stable/jit.html>`_ allow the developer to export
|
||||
their model to be re-used in other programs, such as efficiency-oriented C++ programs.
|
||||
|
||||
We have provided an interface that allows the export of 🤗 Transformers models to TorchScript so that they can
|
||||
be reused in a different environment than a Pytorch-based python program. Here we explain how to export and use our models using TorchScript.
|
||||
We have provided an interface that allows the export of 🤗 Transformers models to TorchScript so that they can be reused
|
||||
in a different environment than a Pytorch-based python program. Here we explain how to export and use our models using
|
||||
TorchScript.
|
||||
|
||||
Exporting a model requires two things:
|
||||
|
||||
|
|
@ -145,13 +152,14 @@ Implications
|
|||
|
||||
TorchScript flag and tied weights
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
This flag is necessary because most of the language models in this repository have tied weights between their
|
||||
``Embedding`` layer and their ``Decoding`` layer. TorchScript does not allow the export of models that have tied weights, therefore
|
||||
it is necessary to untie and clone the weights beforehand.
|
||||
|
||||
This implies that models instantiated with the ``torchscript`` flag have their ``Embedding`` layer and ``Decoding`` layer
|
||||
separate, which means that they should not be trained down the line. Training would de-synchronize the two layers,
|
||||
leading to unexpected results.
|
||||
This flag is necessary because most of the language models in this repository have tied weights between their
|
||||
``Embedding`` layer and their ``Decoding`` layer. TorchScript does not allow the export of models that have tied
|
||||
weights, therefore it is necessary to untie and clone the weights beforehand.
|
||||
|
||||
This implies that models instantiated with the ``torchscript`` flag have their ``Embedding`` layer and ``Decoding``
|
||||
layer separate, which means that they should not be trained down the line. Training would de-synchronize the two
|
||||
layers, leading to unexpected results.
|
||||
|
||||
This is not the case for models that do not have a Language Model head, as those do not have tied weights. These models
|
||||
can be safely exported without the ``torchscript`` flag.
|
||||
|
|
@ -160,8 +168,8 @@ Dummy inputs and standard lengths
|
|||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
The dummy inputs are used to do a model forward pass. While the inputs' values are propagating through the layers,
|
||||
Pytorch keeps track of the different operations executed on each tensor. These recorded operations are then used
|
||||
to create the "trace" of the model.
|
||||
Pytorch keeps track of the different operations executed on each tensor. These recorded operations are then used to
|
||||
create the "trace" of the model.
|
||||
|
||||
The trace is created relatively to the inputs' dimensions. It is therefore constrained by the dimensions of the dummy
|
||||
input, and will not work for any other sequence length or batch size. When trying with a different size, an error such
|
||||
|
|
@ -185,8 +193,8 @@ Below is an example, showing how to save, load models as well as how to use the
|
|||
Saving a model
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This snippet shows how to use TorchScript to export a ``BertModel``. Here the ``BertModel`` is instantiated
|
||||
according to a ``BertConfig`` class and then saved to disk under the filename ``traced_bert.pt``
|
||||
This snippet shows how to use TorchScript to export a ``BertModel``. Here the ``BertModel`` is instantiated according
|
||||
to a ``BertConfig`` class and then saved to disk under the filename ``traced_bert.pt``
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
|
|
@ -2,30 +2,30 @@ Summary of the tasks
|
|||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
This page shows the most frequent use-cases when using the library. The models available allow for many different
|
||||
configurations and a great versatility in use-cases. The most simple ones are presented here, showcasing usage
|
||||
for tasks such as question answering, sequence classification, named entity recognition and others.
|
||||
configurations and a great versatility in use-cases. The most simple ones are presented here, showcasing usage for
|
||||
tasks such as question answering, sequence classification, named entity recognition and others.
|
||||
|
||||
These examples leverage auto-models, which are classes that will instantiate a model according to a given checkpoint,
|
||||
automatically selecting the correct model architecture. Please check the :class:`~transformers.AutoModel` documentation
|
||||
for more information.
|
||||
Feel free to modify the code to be more specific and adapt it to your specific use-case.
|
||||
for more information. Feel free to modify the code to be more specific and adapt it to your specific use-case.
|
||||
|
||||
In order for a model to perform well on a task, it must be loaded from a checkpoint corresponding to that task. These
|
||||
checkpoints are usually pre-trained on a large corpus of data and fine-tuned on a specific task. This means the
|
||||
following:
|
||||
|
||||
- Not all models were fine-tuned on all tasks. If you want to fine-tune a model on a specific task, you can leverage
|
||||
one of the `run_$TASK.py` scripts in the
|
||||
`examples <https://github.com/huggingface/transformers/tree/master/examples>`__ directory.
|
||||
- Fine-tuned models were fine-tuned on a specific dataset. This dataset may or may not overlap with your use-case
|
||||
and domain. As mentioned previously, you may leverage the
|
||||
`examples <https://github.com/huggingface/transformers/tree/master/examples>`__ scripts to fine-tune your model, or you
|
||||
may create your own training script.
|
||||
one of the `run_$TASK.py` scripts in the `examples
|
||||
<https://github.com/huggingface/transformers/tree/master/examples>`__ directory.
|
||||
- Fine-tuned models were fine-tuned on a specific dataset. This dataset may or may not overlap with your use-case and
|
||||
domain. As mentioned previously, you may leverage the `examples
|
||||
<https://github.com/huggingface/transformers/tree/master/examples>`__ scripts to fine-tune your model, or you may
|
||||
create your own training script.
|
||||
|
||||
In order to do an inference on a task, several mechanisms are made available by the library:
|
||||
|
||||
- Pipelines: very easy-to-use abstractions, which require as little as two lines of code.
|
||||
- Direct model use: Less abstractions, but more flexibility and power via a direct access to a tokenizer (PyTorch/TensorFlow) and full inference capacity.
|
||||
- Direct model use: Less abstractions, but more flexibility and power via a direct access to a tokenizer
|
||||
(PyTorch/TensorFlow) and full inference capacity.
|
||||
|
||||
Both approaches are showcased here.
|
||||
|
||||
|
|
@ -40,15 +40,17 @@ Both approaches are showcased here.
|
|||
Sequence Classification
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
Sequence classification is the task of classifying sequences according to a given number of classes. An example
|
||||
of sequence classification is the GLUE dataset, which is entirely based on that task. If you would like to fine-tune
|
||||
a model on a GLUE sequence classification task, you may leverage the
|
||||
`run_glue.py <https://github.com/huggingface/transformers/tree/master/examples/text-classification/run_glue.py>`__ and
|
||||
`run_pl_glue.py <https://github.com/huggingface/transformers/tree/master/examples/text-classification/run_pl_glue.py>`__ or
|
||||
`run_tf_glue.py <https://github.com/huggingface/transformers/tree/master/examples/text-classification/run_tf_glue.py>`__ scripts.
|
||||
Sequence classification is the task of classifying sequences according to a given number of classes. An example of
|
||||
sequence classification is the GLUE dataset, which is entirely based on that task. If you would like to fine-tune a
|
||||
model on a GLUE sequence classification task, you may leverage the `run_glue.py
|
||||
<https://github.com/huggingface/transformers/tree/master/examples/text-classification/run_glue.py>`__ and
|
||||
`run_pl_glue.py
|
||||
<https://github.com/huggingface/transformers/tree/master/examples/text-classification/run_pl_glue.py>`__ or
|
||||
`run_tf_glue.py
|
||||
<https://github.com/huggingface/transformers/tree/master/examples/text-classification/run_tf_glue.py>`__ scripts.
|
||||
|
||||
Here is an example of using pipelines to do sentiment analysis: identifying if a sequence is positive or negative.
|
||||
It leverages a fine-tuned model on sst2, which is a GLUE task.
|
||||
Here is an example of using pipelines to do sentiment analysis: identifying if a sequence is positive or negative. It
|
||||
leverages a fine-tuned model on sst2, which is a GLUE task.
|
||||
|
||||
This returns a label ("POSITIVE" or "NEGATIVE") alongside a score, as follows:
|
||||
|
||||
|
|
@ -67,18 +69,16 @@ This returns a label ("POSITIVE" or "NEGATIVE") alongside a score, as follows:
|
|||
label: POSITIVE, with score: 0.9999
|
||||
|
||||
|
||||
Here is an example of doing a sequence classification using a model to determine if two sequences are paraphrases
|
||||
of each other. The process is the following:
|
||||
Here is an example of doing a sequence classification using a model to determine if two sequences are paraphrases of
|
||||
each other. The process is the following:
|
||||
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. The model is
|
||||
identified as a BERT model and loads it with the weights stored in the
|
||||
checkpoint.
|
||||
2. Build a sequence from the two sentences, with the correct model-specific
|
||||
separators token type ids and attention masks
|
||||
(:func:`~transformers.PreTrainedTokenizer.encode` and
|
||||
:func:`~transformers.PreTrainedTokenizer.__call__` take care of this).
|
||||
3. Pass this sequence through the model so that it is classified in one of the
|
||||
two available classes: 0 (not a paraphrase) and 1 (is a paraphrase).
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a BERT model and loads it
|
||||
with the weights stored in the checkpoint.
|
||||
2. Build a sequence from the two sentences, with the correct model-specific separators token type ids and attention
|
||||
masks (:func:`~transformers.PreTrainedTokenizer.encode` and :func:`~transformers.PreTrainedTokenizer.__call__` take
|
||||
care of this).
|
||||
3. Pass this sequence through the model so that it is classified in one of the two available classes: 0 (not a
|
||||
paraphrase) and 1 (is a paraphrase).
|
||||
4. Compute the softmax of the result to get probabilities over the classes.
|
||||
5. Print the results.
|
||||
|
||||
|
|
@ -155,14 +155,15 @@ Extractive Question Answering
|
|||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
Extractive Question Answering is the task of extracting an answer from a text given a question. An example of a
|
||||
question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune
|
||||
a model on a SQuAD task, you may leverage the
|
||||
`run_squad.py <https://github.com/huggingface/transformers/tree/master/examples/question-answering/run_squad.py>`__ and
|
||||
`run_tf_squad.py <https://github.com/huggingface/transformers/tree/master/examples/question-answering/run_tf_squad.py>`__ scripts.
|
||||
question answering dataset is the SQuAD dataset, which is entirely based on that task. If you would like to fine-tune a
|
||||
model on a SQuAD task, you may leverage the `run_squad.py
|
||||
<https://github.com/huggingface/transformers/tree/master/examples/question-answering/run_squad.py>`__ and
|
||||
`run_tf_squad.py
|
||||
<https://github.com/huggingface/transformers/tree/master/examples/question-answering/run_tf_squad.py>`__ scripts.
|
||||
|
||||
|
||||
Here is an example of using pipelines to do question answering: extracting an answer from a text given a question.
|
||||
It leverages a fine-tuned model on SQuAD.
|
||||
Here is an example of using pipelines to do question answering: extracting an answer from a text given a question. It
|
||||
leverages a fine-tuned model on SQuAD.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -176,8 +177,8 @@ It leverages a fine-tuned model on SQuAD.
|
|||
... a model on a SQuAD task, you may leverage the examples/question-answering/run_squad.py script.
|
||||
... """
|
||||
|
||||
This returns an answer extracted from the text, a confidence score, alongside "start" and "end" values, which
|
||||
are the positions of the extracted answer in the text.
|
||||
This returns an answer extracted from the text, a confidence score, alongside "start" and "end" values, which are the
|
||||
positions of the extracted answer in the text.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -192,16 +193,13 @@ are the positions of the extracted answer in the text.
|
|||
|
||||
Here is an example of question answering using a model and a tokenizer. The process is the following:
|
||||
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. The model is
|
||||
identified as a BERT model and loads it with the weights stored in the
|
||||
checkpoint.
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a BERT model and loads it
|
||||
with the weights stored in the checkpoint.
|
||||
2. Define a text and a few questions.
|
||||
3. Iterate over the questions and build a sequence from the text and the current
|
||||
question, with the correct model-specific separators token type ids and
|
||||
attention masks.
|
||||
4. Pass this sequence through the model. This outputs a range of scores across
|
||||
the entire sequence tokens (question and text), for both the start and end
|
||||
positions.
|
||||
3. Iterate over the questions and build a sequence from the text and the current question, with the correct
|
||||
model-specific separators token type ids and attention masks.
|
||||
4. Pass this sequence through the model. This outputs a range of scores across the entire sequence tokens (question and
|
||||
text), for both the start and end positions.
|
||||
5. Compute the softmax of the result to get probabilities over the tokens.
|
||||
6. Fetch the tokens from the identified start and stop values, convert those tokens to a string.
|
||||
7. Print the results.
|
||||
|
|
@ -299,22 +297,22 @@ Here is an example of question answering using a model and a tokenizer. The proc
|
|||
Language Modeling
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
Language modeling is the task of fitting a model to a corpus, which can be domain specific. All popular transformer-based
|
||||
models are trained using a variant of language modeling, e.g. BERT with masked language modeling, GPT-2 with
|
||||
causal language modeling.
|
||||
Language modeling is the task of fitting a model to a corpus, which can be domain specific. All popular
|
||||
transformer-based models are trained using a variant of language modeling, e.g. BERT with masked language modeling,
|
||||
GPT-2 with causal language modeling.
|
||||
|
||||
Language modeling can be useful outside of pre-training as well, for example to shift the model distribution to be
|
||||
domain-specific: using a language model trained over a very large corpus, and then fine-tuning it to a news dataset
|
||||
or on scientific papers e.g. `LysandreJik/arxiv-nlp <https://huggingface.co/lysandre/arxiv-nlp>`__.
|
||||
domain-specific: using a language model trained over a very large corpus, and then fine-tuning it to a news dataset or
|
||||
on scientific papers e.g. `LysandreJik/arxiv-nlp <https://huggingface.co/lysandre/arxiv-nlp>`__.
|
||||
|
||||
Masked Language Modeling
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Masked language modeling is the task of masking tokens in a sequence with a masking token, and prompting the model to
|
||||
fill that mask with an appropriate token. This allows the model to attend to both the right context (tokens on the
|
||||
right of the mask) and the left context (tokens on the left of the mask). Such a training creates a strong basis
|
||||
for downstream tasks, requiring bi-directional context such as SQuAD (question answering,
|
||||
see `Lewis, Lui, Goyal et al. <https://arxiv.org/abs/1910.13461>`__, part 4.2).
|
||||
right of the mask) and the left context (tokens on the left of the mask). Such a training creates a strong basis for
|
||||
downstream tasks, requiring bi-directional context such as SQuAD (question answering, see `Lewis, Lui, Goyal et al.
|
||||
<https://arxiv.org/abs/1910.13461>`__, part 4.2).
|
||||
|
||||
Here is an example of using pipelines to replace a mask from a sequence:
|
||||
|
||||
|
|
@ -324,8 +322,7 @@ Here is an example of using pipelines to replace a mask from a sequence:
|
|||
|
||||
>>> nlp = pipeline("fill-mask")
|
||||
|
||||
This outputs the sequences with the mask filled, the confidence score, and the token id in the tokenizer
|
||||
vocabulary:
|
||||
This outputs the sequences with the mask filled, the confidence score, and the token id in the tokenizer vocabulary:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -359,14 +356,12 @@ vocabulary:
|
|||
|
||||
Here is an example of doing masked language modeling using a model and a tokenizer. The process is the following:
|
||||
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. The model is
|
||||
identified as a DistilBERT model and loads it with the weights stored in the
|
||||
checkpoint.
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a DistilBERT model and
|
||||
loads it with the weights stored in the checkpoint.
|
||||
2. Define a sequence with a masked token, placing the :obj:`tokenizer.mask_token` instead of a word.
|
||||
3. Encode that sequence into a list of IDs and find the position of the masked token in that list.
|
||||
4. Retrieve the predictions at the index of the mask token: this tensor has the
|
||||
same size as the vocabulary, and the values are the scores attributed to each
|
||||
token. The model gives higher score to tokens it deems probable in that
|
||||
4. Retrieve the predictions at the index of the mask token: this tensor has the same size as the vocabulary, and the
|
||||
values are the scores attributed to each token. The model gives higher score to tokens it deems probable in that
|
||||
context.
|
||||
5. Retrieve the top 5 tokens using the PyTorch :obj:`topk` or TensorFlow :obj:`top_k` methods.
|
||||
6. Replace the mask token by the tokens and print the results
|
||||
|
|
@ -427,9 +422,12 @@ Causal language modeling is the task of predicting the token following a sequenc
|
|||
model only attends to the left context (tokens on the left of the mask). Such a training is particularly interesting
|
||||
for generation tasks.
|
||||
|
||||
Usually, the next token is predicted by sampling from the logits of the last hidden state the model produces from the input sequence.
|
||||
Usually, the next token is predicted by sampling from the logits of the last hidden state the model produces from the
|
||||
input sequence.
|
||||
|
||||
Here is an example of using the tokenizer and model and leveraging the :func:`~transformers.PreTrainedModel.top_k_top_p_filtering` method to sample the next token following an input sequence of tokens.
|
||||
Here is an example of using the tokenizer and model and leveraging the
|
||||
:func:`~transformers.PreTrainedModel.top_k_top_p_filtering` method to sample the next token following an input sequence
|
||||
of tokens.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -490,12 +488,16 @@ This outputs a (hopefully) coherent next token following the original sequence,
|
|||
>>> print(resulting_string)
|
||||
Hugging Face is based in DUMBO, New York City, and has
|
||||
|
||||
In the next section, we show how this functionality is leveraged in :func:`~transformers.PreTrainedModel.generate` to generate multiple tokens up to a user-defined length.
|
||||
In the next section, we show how this functionality is leveraged in :func:`~transformers.PreTrainedModel.generate` to
|
||||
generate multiple tokens up to a user-defined length.
|
||||
|
||||
Text Generation
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In text generation (*a.k.a* *open-ended text generation*) the goal is to create a coherent portion of text that is a continuation from the given context. The following example shows how *GPT-2* can be used in pipelines to generate text. As a default all models apply *Top-K* sampling when used in pipelines, as configured in their respective configurations (see `gpt-2 config <https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json>`__ for example).
|
||||
In text generation (*a.k.a* *open-ended text generation*) the goal is to create a coherent portion of text that is a
|
||||
continuation from the given context. The following example shows how *GPT-2* can be used in pipelines to generate text.
|
||||
As a default all models apply *Top-K* sampling when used in pipelines, as configured in their respective configurations
|
||||
(see `gpt-2 config <https://s3.amazonaws.com/models.huggingface.co/bert/gpt2-config.json>`__ for example).
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -507,8 +509,9 @@ In text generation (*a.k.a* *open-ended text generation*) the goal is to create
|
|||
|
||||
|
||||
|
||||
Here, the model generates a random text with a total maximal length of *50* tokens from context *"As far as I am concerned, I will"*.
|
||||
The default arguments of ``PreTrainedModel.generate()`` can be directly overriden in the pipeline, as is shown above for the argument ``max_length``.
|
||||
Here, the model generates a random text with a total maximal length of *50* tokens from context *"As far as I am
|
||||
concerned, I will"*. The default arguments of ``PreTrainedModel.generate()`` can be directly overriden in the pipeline,
|
||||
as is shown above for the argument ``max_length``.
|
||||
|
||||
Here is an example of text generation using ``XLNet`` and its tokenzier.
|
||||
|
||||
|
|
@ -569,25 +572,30 @@ Here is an example of text generation using ``XLNet`` and its tokenzier.
|
|||
>>> print(generated)
|
||||
Today the weather is really nice and I am planning on anning on taking a nice...... of a great time!<eop>...............
|
||||
|
||||
Text generation is currently possible with *GPT-2*, *OpenAi-GPT*, *CTRL*, *XLNet*, *Transfo-XL* and *Reformer* in PyTorch and for most models in Tensorflow as well. As can be seen in the example above *XLNet* and *Transfo-XL* often need to be padded to work well.
|
||||
GPT-2 is usually a good choice for *open-ended text generation* because it was trained on millions of webpages with a causal language modeling objective.
|
||||
Text generation is currently possible with *GPT-2*, *OpenAi-GPT*, *CTRL*, *XLNet*, *Transfo-XL* and *Reformer* in
|
||||
PyTorch and for most models in Tensorflow as well. As can be seen in the example above *XLNet* and *Transfo-XL* often
|
||||
need to be padded to work well. GPT-2 is usually a good choice for *open-ended text generation* because it was trained
|
||||
on millions of webpages with a causal language modeling objective.
|
||||
|
||||
For more information on how to apply different decoding strategies for text generation, please also refer to our text generation blog post `here <https://huggingface.co/blog/how-to-generate>`__.
|
||||
For more information on how to apply different decoding strategies for text generation, please also refer to our text
|
||||
generation blog post `here <https://huggingface.co/blog/how-to-generate>`__.
|
||||
|
||||
|
||||
Named Entity Recognition
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
Named Entity Recognition (NER) is the task of classifying tokens according to a class, for example, identifying a
|
||||
token as a person, an organisation or a location.
|
||||
An example of a named entity recognition dataset is the CoNLL-2003 dataset, which is entirely based on that task.
|
||||
If you would like to fine-tune a model on an NER task, you may leverage the
|
||||
`run_ner.py <https://github.com/huggingface/transformers/tree/master/examples/token-classification/run_ner.py>`__ (PyTorch),
|
||||
`run_pl_ner.py <https://github.com/huggingface/transformers/tree/master/examples/token-classification/run_pl_ner.py>`__ (leveraging pytorch-lightning) or the
|
||||
`run_tf_ner.py <https://github.com/huggingface/transformers/tree/master/examples/token-classification/run_tf_ner.py>`__ (TensorFlow) scripts.
|
||||
Named Entity Recognition (NER) is the task of classifying tokens according to a class, for example, identifying a token
|
||||
as a person, an organisation or a location. An example of a named entity recognition dataset is the CoNLL-2003 dataset,
|
||||
which is entirely based on that task. If you would like to fine-tune a model on an NER task, you may leverage the
|
||||
`run_ner.py <https://github.com/huggingface/transformers/tree/master/examples/token-classification/run_ner.py>`__
|
||||
(PyTorch), `run_pl_ner.py
|
||||
<https://github.com/huggingface/transformers/tree/master/examples/token-classification/run_pl_ner.py>`__ (leveraging
|
||||
pytorch-lightning) or the `run_tf_ner.py
|
||||
<https://github.com/huggingface/transformers/tree/master/examples/token-classification/run_tf_ner.py>`__ (TensorFlow)
|
||||
scripts.
|
||||
|
||||
Here is an example of using pipelines to do named entity recognition, specifically, trying to identify tokens as belonging to one
|
||||
of 9 classes:
|
||||
Here is an example of using pipelines to do named entity recognition, specifically, trying to identify tokens as
|
||||
belonging to one of 9 classes:
|
||||
|
||||
- O, Outside of a named entity
|
||||
- B-MIS, Beginning of a miscellaneous entity right after another miscellaneous entity
|
||||
|
|
@ -599,8 +607,8 @@ of 9 classes:
|
|||
- B-LOC, Beginning of a location right after another location
|
||||
- I-LOC, Location
|
||||
|
||||
It leverages a fine-tuned model on CoNLL-2003, fine-tuned by `@stefan-it <https://github.com/stefan-it>`__ from
|
||||
`dbmdz <https://github.com/dbmdz>`__.
|
||||
It leverages a fine-tuned model on CoNLL-2003, fine-tuned by `@stefan-it <https://github.com/stefan-it>`__ from `dbmdz
|
||||
<https://github.com/dbmdz>`__.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -612,8 +620,8 @@ It leverages a fine-tuned model on CoNLL-2003, fine-tuned by `@stefan-it <https:
|
|||
... "close to the Manhattan Bridge which is visible from the window."
|
||||
|
||||
|
||||
This outputs a list of all words that have been identified as one of the entities from the 9 classes defined above. Here are the
|
||||
expected results:
|
||||
This outputs a list of all words that have been identified as one of the entities from the 9 classes defined above.
|
||||
Here are the expected results:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -633,24 +641,21 @@ expected results:
|
|||
{'word': 'Bridge', 'score': 0.990249514579773, 'entity': 'I-LOC'}
|
||||
]
|
||||
|
||||
Note, how the tokens of the sequence "Hugging Face" have been identified as an organisation, and "New York City", "DUMBO" and
|
||||
"Manhattan Bridge" have been identified as locations.
|
||||
Note, how the tokens of the sequence "Hugging Face" have been identified as an organisation, and "New York City",
|
||||
"DUMBO" and "Manhattan Bridge" have been identified as locations.
|
||||
|
||||
Here is an example of doing named entity recognition, using a model and a tokenizer. The process is the following:
|
||||
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. The model is
|
||||
identified as a BERT model and loads it with the weights stored in the
|
||||
checkpoint.
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. The model is identified as a BERT model and loads it
|
||||
with the weights stored in the checkpoint.
|
||||
2. Define the label list with which the model was trained on.
|
||||
3. Define a sequence with known entities, such as "Hugging Face" as an organisation and "New York City" as a location.
|
||||
4. Split words into tokens so that they can be mapped to predictions. We use a
|
||||
small hack by, first, completely encoding and decoding the sequence, so that
|
||||
we're left with a string that contains the special tokens.
|
||||
4. Split words into tokens so that they can be mapped to predictions. We use a small hack by, first, completely
|
||||
encoding and decoding the sequence, so that we're left with a string that contains the special tokens.
|
||||
5. Encode that sequence into IDs (special tokens are added automatically).
|
||||
6. Retrieve the predictions by passing the input to the model and getting the
|
||||
first output. This results in a distribution over the 9 possible classes for
|
||||
each token. We take the argmax to retrieve the most likely class for each
|
||||
token.
|
||||
6. Retrieve the predictions by passing the input to the model and getting the first output. This results in a
|
||||
distribution over the 9 possible classes for each token. We take the argmax to retrieve the most likely class for
|
||||
each token.
|
||||
7. Zip together each token with its prediction and print it.
|
||||
|
||||
.. code-block::
|
||||
|
|
@ -713,9 +718,9 @@ Here is an example of doing named entity recognition, using a model and a tokeni
|
|||
>>> predictions = tf.argmax(outputs, axis=2)
|
||||
|
||||
|
||||
This outputs a list of each token mapped to its corresponding prediction. Differently from the pipeline, here every token has
|
||||
a prediction as we didn't remove the "0"th class, which means that no particular entity was found on that token. The
|
||||
following array should be the output:
|
||||
This outputs a list of each token mapped to its corresponding prediction. Differently from the pipeline, here every
|
||||
token has a prediction as we didn't remove the "0"th class, which means that no particular entity was found on that
|
||||
token. The following array should be the output:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -727,11 +732,13 @@ Summarization
|
|||
|
||||
Summarization is the task of summarizing a document or an article into a shorter text.
|
||||
|
||||
An example of a summarization dataset is the CNN / Daily Mail dataset, which consists of long news articles and was created for the task of summarization.
|
||||
If you would like to fine-tune a model on a summarization task, various approaches are described in this
|
||||
`document <https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md>`__.
|
||||
An example of a summarization dataset is the CNN / Daily Mail dataset, which consists of long news articles and was
|
||||
created for the task of summarization. If you would like to fine-tune a model on a summarization task, various
|
||||
approaches are described in this `document
|
||||
<https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md>`__.
|
||||
|
||||
Here is an example of using the pipelines to do summarization. It leverages a Bart model that was fine-tuned on the CNN / Daily Mail data set.
|
||||
Here is an example of using the pipelines to do summarization. It leverages a Bart model that was fine-tuned on the CNN
|
||||
/ Daily Mail data set.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -758,9 +765,9 @@ Here is an example of using the pipelines to do summarization. It leverages a Ba
|
|||
... If convicted, Barrientos faces up to four years in prison. Her next court appearance is scheduled for May 18.
|
||||
... """
|
||||
|
||||
Because the summarization pipeline depends on the ``PreTrainedModel.generate()`` method, we can override the default arguments
|
||||
of ``PreTrainedModel.generate()`` directly in the pipeline for ``max_length`` and ``min_length`` as shown below.
|
||||
This outputs the following summary:
|
||||
Because the summarization pipeline depends on the ``PreTrainedModel.generate()`` method, we can override the default
|
||||
arguments of ``PreTrainedModel.generate()`` directly in the pipeline for ``max_length`` and ``min_length`` as shown
|
||||
below. This outputs the following summary:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -769,12 +776,14 @@ This outputs the following summary:
|
|||
|
||||
Here is an example of doing summarization using a model and a tokenizer. The process is the following:
|
||||
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. Summarization is usually done using an encoder-decoder model, such as ``Bart`` or ``T5``.
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. Summarization is usually done using an encoder-decoder
|
||||
model, such as ``Bart`` or ``T5``.
|
||||
2. Define the article that should be summarized.
|
||||
3. Add the T5 specific prefix "summarize: ".
|
||||
4. Use the ``PreTrainedModel.generate()`` method to generate the summary.
|
||||
|
||||
In this example we use Google`s T5 model. Even though it was pre-trained only on a multi-task mixed dataset (including CNN / Daily Mail), it yields very good results.
|
||||
In this example we use Google`s T5 model. Even though it was pre-trained only on a multi-task mixed dataset (including
|
||||
CNN / Daily Mail), it yields very good results.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -802,14 +811,13 @@ Translation
|
|||
|
||||
Translation is the task of translating a text from one language to another.
|
||||
|
||||
An example of a translation dataset is the WMT English to German dataset, which has sentences in English as the input data
|
||||
and the corresponding sentences in German as the target data.
|
||||
If you would like to fine-tune a model on a translation task, various approaches are described in this
|
||||
`document <https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md>`__.
|
||||
An example of a translation dataset is the WMT English to German dataset, which has sentences in English as the input
|
||||
data and the corresponding sentences in German as the target data. If you would like to fine-tune a model on a
|
||||
translation task, various approaches are described in this `document
|
||||
<https://github.com/huggingface/transformers/blob/master/examples/seq2seq/README.md>`__.
|
||||
|
||||
Here is an example of using the pipelines to do translation.
|
||||
It leverages a T5 model that was only pre-trained on a multi-task mixture dataset (including WMT), yet, yielding impressive
|
||||
translation results.
|
||||
Here is an example of using the pipelines to do translation. It leverages a T5 model that was only pre-trained on a
|
||||
multi-task mixture dataset (including WMT), yet, yielding impressive translation results.
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -819,12 +827,13 @@ translation results.
|
|||
>>> print(translator("Hugging Face is a technology company based in New York and Paris", max_length=40))
|
||||
[{'translation_text': 'Hugging Face ist ein Technologieunternehmen mit Sitz in New York und Paris.'}]
|
||||
|
||||
Because the translation pipeline depends on the ``PreTrainedModel.generate()`` method, we can override the default arguments
|
||||
of ``PreTrainedModel.generate()`` directly in the pipeline as is shown for ``max_length`` above.
|
||||
Because the translation pipeline depends on the ``PreTrainedModel.generate()`` method, we can override the default
|
||||
arguments of ``PreTrainedModel.generate()`` directly in the pipeline as is shown for ``max_length`` above.
|
||||
|
||||
Here is an example of doing translation using a model and a tokenizer. The process is the following:
|
||||
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. Summarization is usually done using an encoder-decoder model, such as ``Bart`` or ``T5``.
|
||||
1. Instantiate a tokenizer and a model from the checkpoint name. Summarization is usually done using an encoder-decoder
|
||||
model, such as ``Bart`` or ``T5``.
|
||||
2. Define the article that should be summarizaed.
|
||||
3. Add the T5 specific prefix "translate English to German: "
|
||||
4. Use the ``PreTrainedModel.generate()`` method to perform the translation.
|
||||
|
|
|
|||
|
|
@ -12,17 +12,26 @@ There are 2 test suites in the repository:
|
|||
How transformers are tested
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
1. Once a PR is submitted it gets tested with 9 CircleCi jobs. Every new commit to that PR gets retested. These jobs are defined in this `config file <https://github.com/huggingface/transformers/blob/master/.circleci/config.yml>`__, so that if needed you can reproduce the same environment on your machine.
|
||||
|
||||
1. Once a PR is submitted it gets tested with 9 CircleCi jobs. Every new commit to that PR gets retested. These jobs
|
||||
are defined in this `config file <https://github.com/huggingface/transformers/blob/master/.circleci/config.yml>`__,
|
||||
so that if needed you can reproduce the same environment on your machine.
|
||||
|
||||
These CI jobs don't run ``@slow`` tests.
|
||||
|
||||
|
||||
2. There are 3 jobs run by `github actions <https://github.com/huggingface/transformers/actions>`__:
|
||||
|
||||
* `torch hub integration <https://github.com/huggingface/transformers/blob/master/.github/workflows/github-torch-hub.yml>`__: checks whether torch hub integration works.
|
||||
* `torch hub integration
|
||||
<https://github.com/huggingface/transformers/blob/master/.github/workflows/github-torch-hub.yml>`__: checks
|
||||
whether torch hub integration works.
|
||||
|
||||
* `self-hosted (push) <https://github.com/huggingface/transformers/blob/master/.github/workflows/self-push.yml>`__: runs fast tests on GPU only on commits on ``master``. It only runs if a commit on ``master`` has updated the code in one of the following folders: ``src``, ``tests``, ``.github`` (to prevent running on added model cards, notebooks, etc.)
|
||||
|
||||
* `self-hosted runner <https://github.com/huggingface/transformers/blob/master/.github/workflows/self-scheduled.yml>`__: runs normal and slow tests on GPU in ``tests`` and ``examples``:
|
||||
* `self-hosted (push) <https://github.com/huggingface/transformers/blob/master/.github/workflows/self-push.yml>`__:
|
||||
runs fast tests on GPU only on commits on ``master``. It only runs if a commit on ``master`` has updated the code
|
||||
in one of the following folders: ``src``, ``tests``, ``.github`` (to prevent running on added model cards,
|
||||
notebooks, etc.)
|
||||
|
||||
* `self-hosted runner
|
||||
<https://github.com/huggingface/transformers/blob/master/.github/workflows/self-scheduled.yml>`__: runs normal and
|
||||
slow tests on GPU in ``tests`` and ``examples``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -43,7 +52,8 @@ Running tests
|
|||
Choosing which tests to run
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This document goes into many details of how tests can be run. If after reading everything, you need even more details you will find them `here <https://docs.pytest.org/en/latest/usage.html>`__.
|
||||
This document goes into many details of how tests can be run. If after reading everything, you need even more details
|
||||
you will find them `here <https://docs.pytest.org/en/latest/usage.html>`__.
|
||||
|
||||
Here are some most useful ways of running tests.
|
||||
|
||||
|
|
@ -90,7 +100,7 @@ All tests of a given test file:
|
|||
pytest tests/test_optimization.py --collect-only -q
|
||||
|
||||
|
||||
|
||||
|
||||
Run a specific test module
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
|
|
@ -99,12 +109,13 @@ To run an individual test module:
|
|||
.. code-block:: bash
|
||||
|
||||
pytest tests/test_logging.py
|
||||
|
||||
|
||||
|
||||
Run specific tests
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Since unittest is used inside most of the tests, to run specific subtests you need to know the name of the unittest class containing those tests. For example, it could be:
|
||||
Since unittest is used inside most of the tests, to run specific subtests you need to know the name of the unittest
|
||||
class containing those tests. For example, it could be:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -131,7 +142,7 @@ As mentioned earlier you can see what tests are contained inside the ``Optimizat
|
|||
|
||||
pytest tests/test_optimization.py::OptimizationTest --collect-only -q
|
||||
|
||||
|
||||
|
||||
You can run tests by keyword expressions.
|
||||
|
||||
To run only tests whose name contains ``adam``:
|
||||
|
|
@ -158,7 +169,9 @@ And you can combine the two patterns in one:
|
|||
Run only modified tests
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
You can run the tests related to the unstaged files or the current branch (according to Git) by using `pytest-picked <https://github.com/anapaulagomes/pytest-picked>`__. This is a great way of quickly testing your changes didn't break anything, since it won't run the tests related to files you didn't touch.
|
||||
You can run the tests related to the unstaged files or the current branch (according to Git) by using `pytest-picked
|
||||
<https://github.com/anapaulagomes/pytest-picked>`__. This is a great way of quickly testing your changes didn't break
|
||||
anything, since it won't run the tests related to files you didn't touch.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -168,17 +181,14 @@ You can run the tests related to the unstaged files or the current branch (accor
|
|||
|
||||
pytest --picked
|
||||
|
||||
All tests will be run from files and folders which are modified, but not
|
||||
yet committed.
|
||||
All tests will be run from files and folders which are modified, but not yet committed.
|
||||
|
||||
Automatically rerun failed tests on source modification
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
`pytest-xdist <https://github.com/pytest-dev/pytest-xdist>`__ provides a
|
||||
very useful feature of detecting all failed tests, and then waiting for
|
||||
you to modify files and continuously re-rerun those failing tests until
|
||||
they pass while you fix them. So that you don't need to re start pytest
|
||||
after you made the fix. This is repeated until all tests pass after
|
||||
`pytest-xdist <https://github.com/pytest-dev/pytest-xdist>`__ provides a very useful feature of detecting all failed
|
||||
tests, and then waiting for you to modify files and continuously re-rerun those failing tests until they pass while you
|
||||
fix them. So that you don't need to re start pytest after you made the fix. This is repeated until all tests pass after
|
||||
which again a full run is performed.
|
||||
|
||||
.. code-block:: bash
|
||||
|
|
@ -187,10 +197,9 @@ which again a full run is performed.
|
|||
|
||||
To enter the mode: ``pytest -f`` or ``pytest --looponfail``
|
||||
|
||||
File changes are detected by looking at ``looponfailroots`` root
|
||||
directories and all of their contents (recursively). If the default for
|
||||
this value does not work for you, you can change it in your project by
|
||||
setting a configuration option in ``setup.cfg``:
|
||||
File changes are detected by looking at ``looponfailroots`` root directories and all of their contents (recursively).
|
||||
If the default for this value does not work for you, you can change it in your project by setting a configuration
|
||||
option in ``setup.cfg``:
|
||||
|
||||
.. code-block:: ini
|
||||
|
||||
|
|
@ -204,17 +213,17 @@ or ``pytest.ini``/``tox.ini`` files:
|
|||
[pytest]
|
||||
looponfailroots = transformers tests
|
||||
|
||||
This would lead to only looking for file changes in the respective
|
||||
directories, specified relatively to the ini-file’s directory.
|
||||
This would lead to only looking for file changes in the respective directories, specified relatively to the ini-file’s
|
||||
directory.
|
||||
|
||||
`pytest-watch <https://github.com/joeyespo/pytest-watch>`__ is an
|
||||
alternative implementation of this functionality.
|
||||
`pytest-watch <https://github.com/joeyespo/pytest-watch>`__ is an alternative implementation of this functionality.
|
||||
|
||||
|
||||
Skip a test module
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If you want to run all test modules, except a few you can exclude them by giving an explicit list of tests to run. For example, to run all except ``test_modeling_*.py`` tests:
|
||||
If you want to run all test modules, except a few you can exclude them by giving an explicit list of tests to run. For
|
||||
example, to run all except ``test_modeling_*.py`` tests:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -224,8 +233,7 @@ If you want to run all test modules, except a few you can exclude them by giving
|
|||
Clearing state
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
CI builds and when isolation is important (against speed), cache should
|
||||
be cleared:
|
||||
CI builds and when isolation is important (against speed), cache should be cleared:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -234,24 +242,23 @@ be cleared:
|
|||
Running tests in parallel
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
As mentioned earlier ``make test`` runs tests in parallel via ``pytest-xdist`` plugin (``-n X`` argument, e.g. ``-n 2`` to run 2 parallel jobs).
|
||||
As mentioned earlier ``make test`` runs tests in parallel via ``pytest-xdist`` plugin (``-n X`` argument, e.g. ``-n 2``
|
||||
to run 2 parallel jobs).
|
||||
|
||||
``pytest-xdist``'s ``--dist=`` option allows one to control how the tests are grouped. ``--dist=loadfile`` puts the tests located in one file onto the same process.
|
||||
``pytest-xdist``'s ``--dist=`` option allows one to control how the tests are grouped. ``--dist=loadfile`` puts the
|
||||
tests located in one file onto the same process.
|
||||
|
||||
Since the order of executed tests is different and unpredictable, if
|
||||
running the test suite with ``pytest-xdist`` produces failures (meaning
|
||||
we have some undetected coupled tests), use
|
||||
`pytest-replay <https://github.com/ESSS/pytest-replay>`__ to replay the
|
||||
tests in the same order, which should help with then somehow reducing
|
||||
that failing sequence to a minimum.
|
||||
Since the order of executed tests is different and unpredictable, if running the test suite with ``pytest-xdist``
|
||||
produces failures (meaning we have some undetected coupled tests), use `pytest-replay
|
||||
<https://github.com/ESSS/pytest-replay>`__ to replay the tests in the same order, which should help with then somehow
|
||||
reducing that failing sequence to a minimum.
|
||||
|
||||
Test order and repetition
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
It's good to repeat the tests several times, in sequence, randomly, or
|
||||
in sets, to detect any potential inter-dependency and state-related bugs
|
||||
(tear down). And the straightforward multiple repetition is just good to
|
||||
detect some problems that get uncovered by randomness of DL.
|
||||
It's good to repeat the tests several times, in sequence, randomly, or in sets, to detect any potential
|
||||
inter-dependency and state-related bugs (tear down). And the straightforward multiple repetition is just good to detect
|
||||
some problems that get uncovered by randomness of DL.
|
||||
|
||||
|
||||
Repeat tests
|
||||
|
|
@ -268,10 +275,10 @@ And then run every test multiple times (50 by default):
|
|||
.. code-block:: bash
|
||||
|
||||
pytest --flake-finder --flake-runs=5 tests/test_failing_test.py
|
||||
|
||||
|
||||
.. note::
|
||||
This plugin doesn't work with ``-n`` flag from ``pytest-xdist``.
|
||||
|
||||
|
||||
.. note::
|
||||
There is another plugin ``pytest-repeat``, but it doesn't work with ``unittest``.
|
||||
|
||||
|
|
@ -283,14 +290,11 @@ Run tests in a random order
|
|||
|
||||
pip install pytest-random-order
|
||||
|
||||
Important: the presence of ``pytest-random-order`` will automatically
|
||||
randomize tests, no configuration change or command line options is
|
||||
required.
|
||||
Important: the presence of ``pytest-random-order`` will automatically randomize tests, no configuration change or
|
||||
command line options is required.
|
||||
|
||||
As explained earlier this allows detection of coupled tests - where one
|
||||
test's state affects the state of another. When ``pytest-random-order``
|
||||
is installed it will print the random seed it used for that session,
|
||||
e.g:
|
||||
As explained earlier this allows detection of coupled tests - where one test's state affects the state of another. When
|
||||
``pytest-random-order`` is installed it will print the random seed it used for that session, e.g:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -299,8 +303,7 @@ e.g:
|
|||
Using --random-order-bucket=module
|
||||
Using --random-order-seed=573663
|
||||
|
||||
So that if the given particular sequence fails, you can reproduce it by
|
||||
adding that exact seed, e.g.:
|
||||
So that if the given particular sequence fails, you can reproduce it by adding that exact seed, e.g.:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -309,11 +312,9 @@ adding that exact seed, e.g.:
|
|||
Using --random-order-bucket=module
|
||||
Using --random-order-seed=573663
|
||||
|
||||
It will only reproduce the exact order if you use the exact same list of
|
||||
tests (or no list at all). Once you start to manually narrowing
|
||||
down the list you can no longer rely on the seed, but have to list them
|
||||
manually in the exact order they failed and tell pytest to not randomize
|
||||
them instead using ``--random-order-bucket=none``, e.g.:
|
||||
It will only reproduce the exact order if you use the exact same list of tests (or no list at all). Once you start to
|
||||
manually narrowing down the list you can no longer rely on the seed, but have to list them manually in the exact order
|
||||
they failed and tell pytest to not randomize them instead using ``--random-order-bucket=none``, e.g.:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -325,12 +326,13 @@ To disable the shuffling for all tests:
|
|||
|
||||
pytest --random-order-bucket=none
|
||||
|
||||
By default ``--random-order-bucket=module`` is implied, which will
|
||||
shuffle the files on the module levels. It can also shuffle on
|
||||
``class``, ``package``, ``global`` and ``none`` levels. For the complete
|
||||
details please see its `documentation <https://github.com/jbasko/pytest-random-order>`__.
|
||||
By default ``--random-order-bucket=module`` is implied, which will shuffle the files on the module levels. It can also
|
||||
shuffle on ``class``, ``package``, ``global`` and ``none`` levels. For the complete details please see its
|
||||
`documentation <https://github.com/jbasko/pytest-random-order>`__.
|
||||
|
||||
Another randomization alternative is: ``pytest-randomly`` <https://github.com/pytest-dev/pytest-randomly>`__. This module has a very similar functionality/interface, but it doesn't have the bucket modes available in ``pytest-random-order``. It has the same problem of imposing itself once installed.
|
||||
Another randomization alternative is: ``pytest-randomly`` <https://github.com/pytest-dev/pytest-randomly>`__. This
|
||||
module has a very similar functionality/interface, but it doesn't have the bucket modes available in
|
||||
``pytest-random-order``. It has the same problem of imposing itself once installed.
|
||||
|
||||
Look and feel variations
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
|
@ -338,13 +340,11 @@ Look and feel variations
|
|||
pytest-sugar
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
`pytest-sugar <https://github.com/Frozenball/pytest-sugar>`__ is a
|
||||
plugin that improves the look-n-feel, adds a progressbar, and show tests
|
||||
that fail and the assert instantly. It gets activated automatically upon
|
||||
installation.
|
||||
`pytest-sugar <https://github.com/Frozenball/pytest-sugar>`__ is a plugin that improves the look-n-feel, adds a
|
||||
progressbar, and show tests that fail and the assert instantly. It gets activated automatically upon installation.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
||||
pip install pytest-sugar
|
||||
|
||||
To run tests without it, run:
|
||||
|
|
@ -360,8 +360,7 @@ or uninstall it.
|
|||
Report each sub-test name and its progress
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
For a single or a group of tests via ``pytest`` (after
|
||||
``pip install pytest-pspec``):
|
||||
For a single or a group of tests via ``pytest`` (after ``pip install pytest-pspec``):
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -372,9 +371,8 @@ For a single or a group of tests via ``pytest`` (after
|
|||
Instantly shows failed tests
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
`pytest-instafail <https://github.com/pytest-dev/pytest-instafail>`__
|
||||
shows failures and errors instantly instead of waiting until the end of
|
||||
test session.
|
||||
`pytest-instafail <https://github.com/pytest-dev/pytest-instafail>`__ shows failures and errors instantly instead of
|
||||
waiting until the end of test session.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -390,18 +388,20 @@ To GPU or not to GPU
|
|||
On a GPU-enabled setup, to test in CPU-only mode add ``CUDA_VISIBLE_DEVICES=""``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
||||
CUDA_VISIBLE_DEVICES="" pytest tests/test_logging.py
|
||||
|
||||
or if you have multiple gpus, you can specify which one is to be used by ``pytest``. For example, to use only the second gpu if you have gpus ``0`` and ``1``, you can run:
|
||||
or if you have multiple gpus, you can specify which one is to be used by ``pytest``. For example, to use only the
|
||||
second gpu if you have gpus ``0`` and ``1``, you can run:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
||||
CUDA_VISIBLE_DEVICES="1" pytest tests/test_logging.py
|
||||
|
||||
This is handy when you want to run different tasks on different GPUs.
|
||||
|
||||
Some tests must be run on CPU-only, others on either CPU or GPU or TPU, yet others on multiple-GPUs. The following skip decorators are used to set the requirements of tests CPU/GPU/TPU-wise:
|
||||
Some tests must be run on CPU-only, others on either CPU or GPU or TPU, yet others on multiple-GPUs. The following skip
|
||||
decorators are used to set the requirements of tests CPU/GPU/TPU-wise:
|
||||
|
||||
* ``require_torch`` - this test will run only under torch
|
||||
* ``require_torch_gpu`` - as ``require_torch`` plus requires at least 1 GPU
|
||||
|
|
@ -423,7 +423,8 @@ If a test requires ``tensorflow`` use the ``require_tf`` decorator. For example:
|
|||
@require_tf
|
||||
def test_tf_thing_with_tensorflow():
|
||||
|
||||
These decorators can be stacked. For example, if a test is slow and requires at least one GPU under pytorch, here is how to set it up:
|
||||
These decorators can be stacked. For example, if a test is slow and requires at least one GPU under pytorch, here is
|
||||
how to set it up:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -431,7 +432,8 @@ These decorators can be stacked. For example, if a test is slow and requires at
|
|||
@slow
|
||||
def test_example_slow_on_gpu():
|
||||
|
||||
Some decorators like ``@parametrized`` rewrite test names, therefore ``@require_*`` skip decorators have to be listed last for them to work correctly. Here is an example of the correct usage:
|
||||
Some decorators like ``@parametrized`` rewrite test names, therefore ``@require_*`` skip decorators have to be listed
|
||||
last for them to work correctly. Here is an example of the correct usage:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -439,7 +441,8 @@ Some decorators like ``@parametrized`` rewrite test names, therefore ``@require_
|
|||
@require_torch_multigpu
|
||||
def test_integration_foo():
|
||||
|
||||
This order problem doesn't exist with ``@pytest.mark.parametrize``, you can put it first or last and it will still work. But it only works with non-unittests.
|
||||
This order problem doesn't exist with ``@pytest.mark.parametrize``, you can put it first or last and it will still
|
||||
work. But it only works with non-unittests.
|
||||
|
||||
Inside tests:
|
||||
|
||||
|
|
@ -450,16 +453,22 @@ Inside tests:
|
|||
torch.cuda.device_count()
|
||||
|
||||
|
||||
|
||||
|
||||
Distributed training
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
``pytest`` can't deal with distributed training directly. If this is attempted - the sub-processes don't do the right thing and end up thinking they are ``pytest`` and start running the test suite in loops. It works, however, if one spawns a normal process that then spawns off multiple workers and manages the IO pipes.
|
||||
``pytest`` can't deal with distributed training directly. If this is attempted - the sub-processes don't do the right
|
||||
thing and end up thinking they are ``pytest`` and start running the test suite in loops. It works, however, if one
|
||||
spawns a normal process that then spawns off multiple workers and manages the IO pipes.
|
||||
|
||||
This is still under development but you can study 2 different tests that perform this successfully:
|
||||
|
||||
* `test_seq2seq_examples_multi_gpu.py <https://github.com/huggingface/transformers/blob/master/examples/seq2seq/test_seq2seq_examples_multi_gpu.py>`__ - a ``pytorch-lightning``-running test (had to use PL's ``ddp`` spawning method which is the default)
|
||||
* `test_finetune_trainer.py <https://github.com/huggingface/transformers/blob/master/examples/seq2seq/test_finetune_trainer.py>`__ - a normal (non-PL) test
|
||||
* `test_seq2seq_examples_multi_gpu.py
|
||||
<https://github.com/huggingface/transformers/blob/master/examples/seq2seq/test_seq2seq_examples_multi_gpu.py>`__ - a
|
||||
``pytorch-lightning``-running test (had to use PL's ``ddp`` spawning method which is the default)
|
||||
* `test_finetune_trainer.py
|
||||
<https://github.com/huggingface/transformers/blob/master/examples/seq2seq/test_finetune_trainer.py>`__ - a normal
|
||||
(non-PL) test
|
||||
|
||||
To jump right into the execution point, search for the ``execute_async_std`` function in those tests.
|
||||
|
||||
|
|
@ -474,12 +483,10 @@ You will need at least 2 GPUs to see these tests in action:
|
|||
Output capture
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
During test execution any output sent to ``stdout`` and ``stderr`` is
|
||||
captured. If a test or a setup method fails, its according captured
|
||||
output will usually be shown along with the failure traceback.
|
||||
During test execution any output sent to ``stdout`` and ``stderr`` is captured. If a test or a setup method fails, its
|
||||
according captured output will usually be shown along with the failure traceback.
|
||||
|
||||
To disable output capturing and to get the ``stdout`` and ``stderr``
|
||||
normally, use ``-s`` or ``--capture=no``:
|
||||
To disable output capturing and to get the ``stdout`` and ``stderr`` normally, use ``-s`` or ``--capture=no``:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
|
@ -512,9 +519,8 @@ Creating a URL for each test failure:
|
|||
|
||||
pytest --pastebin=failed tests/test_logging.py
|
||||
|
||||
This will submit test run information to a remote Paste service and
|
||||
provide a URL for each failure. You may select tests as usual or add for
|
||||
example -x if you only want to send one particular failure.
|
||||
This will submit test run information to a remote Paste service and provide a URL for each failure. You may select
|
||||
tests as usual or add for example -x if you only want to send one particular failure.
|
||||
|
||||
Creating a URL for a whole test session log:
|
||||
|
||||
|
|
@ -527,18 +533,22 @@ Creating a URL for a whole test session log:
|
|||
Writing tests
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
🤗 transformers tests are based on ``unittest``, but run by ``pytest``, so most of the time features from both systems can be used.
|
||||
🤗 transformers tests are based on ``unittest``, but run by ``pytest``, so most of the time features from both systems
|
||||
can be used.
|
||||
|
||||
You can read `here <https://docs.pytest.org/en/stable/unittest.html>`__ which features are supported, but the important thing to remember is that most ``pytest`` fixtures don't work. Neither parametrization, but we use the module ``parameterized`` that works in a similar way.
|
||||
You can read `here <https://docs.pytest.org/en/stable/unittest.html>`__ which features are supported, but the important
|
||||
thing to remember is that most ``pytest`` fixtures don't work. Neither parametrization, but we use the module
|
||||
``parameterized`` that works in a similar way.
|
||||
|
||||
|
||||
Parametrization
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Often, there is a need to run the same test multiple times, but with different arguments. It could be done from within the test, but then there is no way of running that test for just one set of arguments.
|
||||
Often, there is a need to run the same test multiple times, but with different arguments. It could be done from within
|
||||
the test, but then there is no way of running that test for just one set of arguments.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
||||
# test_this1.py
|
||||
import unittest
|
||||
from parameterized import parameterized
|
||||
|
|
@ -551,7 +561,8 @@ Often, there is a need to run the same test multiple times, but with different a
|
|||
def test_floor(self, name, input, expected):
|
||||
assert_equal(math.floor(input), expected)
|
||||
|
||||
Now, by default this test will be run 3 times, each time with the last 3 arguments of ``test_floor`` being assigned the corresponding arguments in the parameter list.
|
||||
Now, by default this test will be run 3 times, each time with the last 3 arguments of ``test_floor`` being assigned the
|
||||
corresponding arguments in the parameter list.
|
||||
|
||||
and you could run just the ``negative`` and ``integer`` sets of params with:
|
||||
|
||||
|
|
@ -565,14 +576,15 @@ or all but ``negative`` sub-tests, with:
|
|||
|
||||
pytest -k "not negative" tests/test_mytest.py
|
||||
|
||||
Besides using the ``-k`` filter that was just mentioned, you can find out the exact name of each sub-test and run any or all of them using their exact names.
|
||||
|
||||
Besides using the ``-k`` filter that was just mentioned, you can find out the exact name of each sub-test and run any
|
||||
or all of them using their exact names.
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
||||
pytest test_this1.py --collect-only -q
|
||||
|
||||
and it will list:
|
||||
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
test_this1.py::TestMathUnitTest::test_floor_0_negative
|
||||
|
|
@ -584,10 +596,12 @@ So now you can run just 2 specific sub-tests:
|
|||
.. code-block:: bash
|
||||
|
||||
pytest test_this1.py::TestMathUnitTest::test_floor_0_negative test_this1.py::TestMathUnitTest::test_floor_1_integer
|
||||
|
||||
The module `parameterized <https://pypi.org/project/parameterized/>`__ which is already in the developer dependencies of ``transformers`` works for both: ``unittests`` and ``pytest`` tests.
|
||||
|
||||
If, however, the test is not a ``unittest``, you may use ``pytest.mark.parametrize`` (or you may see it being used in some existing tests, mostly under ``examples``).
|
||||
The module `parameterized <https://pypi.org/project/parameterized/>`__ which is already in the developer dependencies
|
||||
of ``transformers`` works for both: ``unittests`` and ``pytest`` tests.
|
||||
|
||||
If, however, the test is not a ``unittest``, you may use ``pytest.mark.parametrize`` (or you may see it being used in
|
||||
some existing tests, mostly under ``examples``).
|
||||
|
||||
Here is the same example, this time using ``pytest``'s ``parametrize`` marker:
|
||||
|
||||
|
|
@ -606,14 +620,16 @@ Here is the same example, this time using ``pytest``'s ``parametrize`` marker:
|
|||
def test_floor(name, input, expected):
|
||||
assert_equal(math.floor(input), expected)
|
||||
|
||||
Same as with ``parameterized``, with ``pytest.mark.parametrize`` you can have a fine control over which sub-tests are run, if the ``-k`` filter doesn't do the job. Except, this parametrization function creates a slightly different set of names for the sub-tests. Here is what they look like:
|
||||
|
||||
Same as with ``parameterized``, with ``pytest.mark.parametrize`` you can have a fine control over which sub-tests are
|
||||
run, if the ``-k`` filter doesn't do the job. Except, this parametrization function creates a slightly different set of
|
||||
names for the sub-tests. Here is what they look like:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
|
||||
pytest test_this2.py --collect-only -q
|
||||
|
||||
and it will list:
|
||||
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
test_this2.py::test_floor[integer-1-1.0]
|
||||
|
|
@ -628,16 +644,20 @@ So now you can run just the specific test:
|
|||
|
||||
as in the previous example.
|
||||
|
||||
|
||||
|
||||
|
||||
Temporary files and directories
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Using unique temporary files and directories are essential for parallel test running, so that the tests won't overwrite each other's data. Also we want to get the temp files and directories removed at the end of each test that created them. Therefore, using packages like ``tempfile``, which address these needs is essential.
|
||||
Using unique temporary files and directories are essential for parallel test running, so that the tests won't overwrite
|
||||
each other's data. Also we want to get the temp files and directories removed at the end of each test that created
|
||||
them. Therefore, using packages like ``tempfile``, which address these needs is essential.
|
||||
|
||||
However, when debugging tests, you need to be able to see what goes into the temp file or directory and you want to know it's exact path and not having it randomized on every test re-run.
|
||||
However, when debugging tests, you need to be able to see what goes into the temp file or directory and you want to
|
||||
know it's exact path and not having it randomized on every test re-run.
|
||||
|
||||
A helper class :obj:`transformers.test_utils.TestCasePlus` is best used for such purposes. It's a sub-class of :obj:`unittest.TestCase`, so we can easily inherit from it in the test modules.
|
||||
A helper class :obj:`transformers.test_utils.TestCasePlus` is best used for such purposes. It's a sub-class of
|
||||
:obj:`unittest.TestCase`, so we can easily inherit from it in the test modules.
|
||||
|
||||
Here is an example of its usage:
|
||||
|
||||
|
|
@ -650,23 +670,27 @@ Here is an example of its usage:
|
|||
|
||||
This code creates a unique temporary directory, and sets :obj:`tmp_dir` to its location.
|
||||
|
||||
In this and all the following scenarios the temporary directory will be auto-removed at the end of test, unless ``after=False`` is passed to the helper function.
|
||||
In this and all the following scenarios the temporary directory will be auto-removed at the end of test, unless
|
||||
``after=False`` is passed to the helper function.
|
||||
|
||||
* Create a temporary directory of my choice and delete it at the end - useful for debugging when you want to monitor a specific directory:
|
||||
* Create a temporary directory of my choice and delete it at the end - useful for debugging when you want to monitor a
|
||||
specific directory:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def test_whatever(self):
|
||||
tmp_dir = self.get_auto_remove_tmp_dir(tmp_dir="./tmp/run/test")
|
||||
|
||||
* Create a temporary directory of my choice and do not delete it at the end---useful for when you want to look at the temp results:
|
||||
* Create a temporary directory of my choice and do not delete it at the end---useful for when you want to look at the
|
||||
temp results:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
def test_whatever(self):
|
||||
tmp_dir = self.get_auto_remove_tmp_dir(tmp_dir="./tmp/run/test", after=False)
|
||||
|
||||
* Create a temporary directory of my choice and ensure to delete it right away---useful for when you disabled deletion in the previous test run and want to make sure the that temporary directory is empty before the new test is run:
|
||||
* Create a temporary directory of my choice and ensure to delete it right away---useful for when you disabled deletion
|
||||
in the previous test run and want to make sure the that temporary directory is empty before the new test is run:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -674,38 +698,33 @@ In this and all the following scenarios the temporary directory will be auto-rem
|
|||
tmp_dir = self.get_auto_remove_tmp_dir(tmp_dir="./tmp/run/test", before=True)
|
||||
|
||||
.. note::
|
||||
In order to run the equivalent of ``rm -r`` safely, only subdirs of the project repository checkout are allowed if an explicit obj:`tmp_dir` is used, so that by mistake no ``/tmp`` or similar important part of the filesystem will get nuked. i.e. please always pass paths that start with ``./``.
|
||||
In order to run the equivalent of ``rm -r`` safely, only subdirs of the project repository checkout are allowed if
|
||||
an explicit obj:`tmp_dir` is used, so that by mistake no ``/tmp`` or similar important part of the filesystem will
|
||||
get nuked. i.e. please always pass paths that start with ``./``.
|
||||
|
||||
.. note::
|
||||
Each test can register multiple temporary directories and they all will get auto-removed, unless requested otherwise.
|
||||
Each test can register multiple temporary directories and they all will get auto-removed, unless requested
|
||||
otherwise.
|
||||
|
||||
|
||||
Skipping tests
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
This is useful when a bug is found and a new test is written, yet the
|
||||
bug is not fixed yet. In order to be able to commit it to the main
|
||||
repository we need make sure it's skipped during ``make test``.
|
||||
This is useful when a bug is found and a new test is written, yet the bug is not fixed yet. In order to be able to
|
||||
commit it to the main repository we need make sure it's skipped during ``make test``.
|
||||
|
||||
Methods:
|
||||
|
||||
- A **skip** means that you expect your test to pass only if some
|
||||
conditions are met, otherwise pytest should skip running the test
|
||||
altogether. Common examples are skipping windows-only tests on
|
||||
non-windows platforms, or skipping tests that depend on an external
|
||||
resource which is not available at the moment (for example a
|
||||
database).
|
||||
- A **skip** means that you expect your test to pass only if some conditions are met, otherwise pytest should skip
|
||||
running the test altogether. Common examples are skipping windows-only tests on non-windows platforms, or skipping
|
||||
tests that depend on an external resource which is not available at the moment (for example a database).
|
||||
|
||||
- A **xfail** means that you expect a test to fail for some reason. A
|
||||
common example is a test for a feature not yet implemented, or a bug
|
||||
not yet fixed. When a test passes despite being expected to fail
|
||||
(marked with pytest.mark.xfail), it’s an xpass and will be reported
|
||||
in the test summary.
|
||||
- A **xfail** means that you expect a test to fail for some reason. A common example is a test for a feature not yet
|
||||
implemented, or a bug not yet fixed. When a test passes despite being expected to fail (marked with
|
||||
pytest.mark.xfail), it’s an xpass and will be reported in the test summary.
|
||||
|
||||
One of the important differences between the two is that ``skip``
|
||||
doesn't run the test, and ``xfail`` does. So if the code that's buggy
|
||||
causes some bad state that will affect other tests, do not use
|
||||
``xfail``.
|
||||
One of the important differences between the two is that ``skip`` doesn't run the test, and ``xfail`` does. So if the
|
||||
code that's buggy causes some bad state that will affect other tests, do not use ``xfail``.
|
||||
|
||||
Implementation
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
|
@ -772,7 +791,7 @@ or:
|
|||
|
||||
@unittest.skipIf(torch_device == "cpu", "Can't do half precision")
|
||||
def test_feature_x():
|
||||
|
||||
|
||||
or skip the whole module:
|
||||
|
||||
.. code-block:: python
|
||||
|
|
@ -786,7 +805,9 @@ More details, example and ways are `here <https://docs.pytest.org/en/latest/skip
|
|||
Slow tests
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
The library of tests is ever-growing, and some of the tests take minutes to run, therefore we can't afford waiting for an hour for the test suite to complete on CI. Therefore, with some exceptions for essential tests, slow tests should be marked as in the example below:
|
||||
The library of tests is ever-growing, and some of the tests take minutes to run, therefore we can't afford waiting for
|
||||
an hour for the test suite to complete on CI. Therefore, with some exceptions for essential tests, slow tests should be
|
||||
marked as in the example below:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -799,8 +820,9 @@ Once a test is marked as ``@slow``, to run such tests set ``RUN_SLOW=1`` env var
|
|||
.. code-block:: bash
|
||||
|
||||
RUN_SLOW=1 pytest tests
|
||||
|
||||
Some decorators like ``@parameterized`` rewrite test names, therefore ``@slow`` and the rest of the skip decorators ``@require_*`` have to be listed last for them to work correctly. Here is an example of the correct usage:
|
||||
|
||||
Some decorators like ``@parameterized`` rewrite test names, therefore ``@slow`` and the rest of the skip decorators
|
||||
``@require_*`` have to be listed last for them to work correctly. Here is an example of the correct usage:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -808,39 +830,55 @@ Some decorators like ``@parameterized`` rewrite test names, therefore ``@slow``
|
|||
@slow
|
||||
def test_integration_foo():
|
||||
|
||||
As explained at the beginning of this document, slow tests get to run on a scheduled basis, rather than in PRs CI checks. So it's possible that some problems will be missed during a PR submission and get merged. Such problems will get caught during the next scheduled CI job. But it also means that it's important to run the slow tests on your machine before submitting the PR.
|
||||
As explained at the beginning of this document, slow tests get to run on a scheduled basis, rather than in PRs CI
|
||||
checks. So it's possible that some problems will be missed during a PR submission and get merged. Such problems will
|
||||
get caught during the next scheduled CI job. But it also means that it's important to run the slow tests on your
|
||||
machine before submitting the PR.
|
||||
|
||||
Here is a rough decision making mechanism for choosing which tests should be marked as slow:
|
||||
|
||||
If the test is focused on one of the library's internal components (e.g., modeling files, tokenization files, pipelines), then we should run that test in the non-slow test suite. If it's focused on an other aspect of the library, such as the documentation or the examples, then we should run these tests in the slow test suite. And then, to refine this approach we should have exceptions:
|
||||
If the test is focused on one of the library's internal components (e.g., modeling files, tokenization files,
|
||||
pipelines), then we should run that test in the non-slow test suite. If it's focused on an other aspect of the library,
|
||||
such as the documentation or the examples, then we should run these tests in the slow test suite. And then, to refine
|
||||
this approach we should have exceptions:
|
||||
|
||||
* All tests that need to download a heavy set of weights (e.g., model or tokenizer integration tests, pipeline integration tests) should be set to slow. If you're adding a new model, you should create and upload to the hub a tiny version of it (with random weights) for integration tests. This is discussed in the following paragraphs.
|
||||
* All tests that need to download a heavy set of weights (e.g., model or tokenizer integration tests, pipeline
|
||||
integration tests) should be set to slow. If you're adding a new model, you should create and upload to the hub a
|
||||
tiny version of it (with random weights) for integration tests. This is discussed in the following paragraphs.
|
||||
* All tests that need to do a training not specifically optimized to be fast should be set to slow.
|
||||
* We can introduce exceptions if some of these should-be-non-slow tests are excruciatingly slow, and set them to ``@slow``. Auto-modeling tests, which save and load large files to disk, are a good example of tests that are marked as ``@slow``.
|
||||
* We can introduce exceptions if some of these should-be-non-slow tests are excruciatingly slow, and set them to
|
||||
``@slow``. Auto-modeling tests, which save and load large files to disk, are a good example of tests that are marked
|
||||
as ``@slow``.
|
||||
* If a test completes under 1 second on CI (including downloads if any) then it should be a normal test regardless.
|
||||
|
||||
Collectively, all the non-slow tests need to cover entirely the different internals, while remaining fast.
|
||||
For example, a significant coverage can be achieved by testing with specially created tiny models with random weights. Such models have the very minimal number of layers (e.g., 2), vocab size (e.g., 1000), etc.
|
||||
Then the ``@slow`` tests can use large slow models to do qualitative testing. To see the use of these simply look for *tiny* models with:
|
||||
Collectively, all the non-slow tests need to cover entirely the different internals, while remaining fast. For example,
|
||||
a significant coverage can be achieved by testing with specially created tiny models with random weights. Such models
|
||||
have the very minimal number of layers (e.g., 2), vocab size (e.g., 1000), etc. Then the ``@slow`` tests can use large
|
||||
slow models to do qualitative testing. To see the use of these simply look for *tiny* models with:
|
||||
|
||||
.. code-block:: bash
|
||||
|
||||
grep tiny tests examples
|
||||
|
||||
Here is a an example of a `script <https://github.com/huggingface/transformers/blob/master/scripts/fsmt/fsmt-make-tiny-model.py>`__ that created the tiny model `stas/tiny-wmt19-en-de <https://huggingface.co/stas/tiny-wmt19-en-de>`__. You can easily adjust it to your specific model's architecture.
|
||||
Here is a an example of a `script
|
||||
<https://github.com/huggingface/transformers/blob/master/scripts/fsmt/fsmt-make-tiny-model.py>`__ that created the tiny
|
||||
model `stas/tiny-wmt19-en-de <https://huggingface.co/stas/tiny-wmt19-en-de>`__. You can easily adjust it to your
|
||||
specific model's architecture.
|
||||
|
||||
It's easy to measure the run-time incorrectly if for example there is an overheard of downloading a huge model, but if you test it locally the downloaded files would be cached and thus the download time not measured. Hence check the execution speed report in CI logs instead (the output of ``pytest --durations=0 tests``).
|
||||
It's easy to measure the run-time incorrectly if for example there is an overheard of downloading a huge model, but if
|
||||
you test it locally the downloaded files would be cached and thus the download time not measured. Hence check the
|
||||
execution speed report in CI logs instead (the output of ``pytest --durations=0 tests``).
|
||||
|
||||
That report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast. If you notice that the test suite starts getting slow on CI, the top listing of this report will show the slowest tests.
|
||||
That report is also useful to find slow outliers that aren't marked as such, or which need to be re-written to be fast.
|
||||
If you notice that the test suite starts getting slow on CI, the top listing of this report will show the slowest
|
||||
tests.
|
||||
|
||||
|
||||
Testing the stdout/stderr output
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In order to test functions that write to ``stdout`` and/or ``stderr``,
|
||||
the test can access those streams using the ``pytest``'s `capsys
|
||||
system <https://docs.pytest.org/en/latest/capture.html>`__. Here is how
|
||||
this is accomplished:
|
||||
In order to test functions that write to ``stdout`` and/or ``stderr``, the test can access those streams using the
|
||||
``pytest``'s `capsys system <https://docs.pytest.org/en/latest/capture.html>`__. Here is how this is accomplished:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -859,8 +897,8 @@ this is accomplished:
|
|||
assert msg in out
|
||||
assert msg in err
|
||||
|
||||
And, of course, most of the time, ``stderr`` will come as a part of an
|
||||
exception, so try/except has to be used in such a case:
|
||||
And, of course, most of the time, ``stderr`` will come as a part of an exception, so try/except has to be used in such
|
||||
a case:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -892,16 +930,13 @@ Another approach to capturing stdout is via ``contextlib.redirect_stdout``:
|
|||
# test:
|
||||
assert msg in out
|
||||
|
||||
An important potential issue with capturing stdout is that it may
|
||||
contain ``\r`` characters that in normal ``print`` reset everything that
|
||||
has been printed so far. There is no problem with ``pytest``, but with
|
||||
``pytest -s`` these characters get included in the buffer, so to be able
|
||||
to have the test run with and without ``-s``, you have to make an extra
|
||||
cleanup to the captured output, using ``re.sub(r'~.*\r', '', buf, 0, re.M)``.
|
||||
An important potential issue with capturing stdout is that it may contain ``\r`` characters that in normal ``print``
|
||||
reset everything that has been printed so far. There is no problem with ``pytest``, but with ``pytest -s`` these
|
||||
characters get included in the buffer, so to be able to have the test run with and without ``-s``, you have to make an
|
||||
extra cleanup to the captured output, using ``re.sub(r'~.*\r', '', buf, 0, re.M)``.
|
||||
|
||||
But, then we have a helper context manager wrapper to automatically take
|
||||
care of it all, regardless of whether it has some ``\r``'s in it or
|
||||
not, so it's a simple:
|
||||
But, then we have a helper context manager wrapper to automatically take care of it all, regardless of whether it has
|
||||
some ``\r``'s in it or not, so it's a simple:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -921,8 +956,7 @@ Here is a full test example:
|
|||
print(msg + final)
|
||||
assert cs.out == final+"\n", f"captured: {cs.out}, expecting {final}"
|
||||
|
||||
If you'd like to capture ``stderr`` use the :obj:`CaptureStderr` class
|
||||
instead:
|
||||
If you'd like to capture ``stderr`` use the :obj:`CaptureStderr` class instead:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -931,8 +965,7 @@ instead:
|
|||
function_that_writes_to_stderr()
|
||||
print(cs.err)
|
||||
|
||||
If you need to capture both streams at once, use the parent
|
||||
:obj:`CaptureStd` class:
|
||||
If you need to capture both streams at once, use the parent :obj:`CaptureStd` class:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -964,7 +997,8 @@ If you need to validate the output of a logger, you can use :obj:`CaptureLogger`
|
|||
Testing with environment variables
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
If you want to test the impact of environment variables for a specific test you can use a helper decorator ``transformers.testing_utils.mockenv``
|
||||
If you want to test the impact of environment variables for a specific test you can use a helper decorator
|
||||
``transformers.testing_utils.mockenv``
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -978,8 +1012,8 @@ If you want to test the impact of environment variables for a specific test you
|
|||
Getting reproducible results
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
In some situations you may want to remove randomness for your tests. To
|
||||
get identical reproducable results set, you will need to fix the seed:
|
||||
In some situations you may want to remove randomness for your tests. To get identical reproducable results set, you
|
||||
will need to fix the seed:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
|
|||
|
|
@ -1,12 +1,12 @@
|
|||
Tokenizer summary
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
In this page, we will have a closer look at tokenization. As we saw in
|
||||
:doc:`the preprocessing tutorial <preprocessing>`, tokenizing a text is splitting it into words or subwords, which then
|
||||
are converted to ids. The second part is pretty straightforward, here we will focus on the first part. More
|
||||
specifically, we will look at the three main different kinds of tokenizers used in 🤗 Transformers:
|
||||
:ref:`Byte-Pair Encoding (BPE) <byte-pair-encoding>`, :ref:`WordPiece <wordpiece>` and
|
||||
:ref:`SentencePiece <sentencepiece>`, and provide examples of models using each of those.
|
||||
In this page, we will have a closer look at tokenization. As we saw in :doc:`the preprocessing tutorial
|
||||
<preprocessing>`, tokenizing a text is splitting it into words or subwords, which then are converted to ids. The second
|
||||
part is pretty straightforward, here we will focus on the first part. More specifically, we will look at the three main
|
||||
different kinds of tokenizers used in 🤗 Transformers: :ref:`Byte-Pair Encoding (BPE) <byte-pair-encoding>`,
|
||||
:ref:`WordPiece <wordpiece>` and :ref:`SentencePiece <sentencepiece>`, and provide examples of models using each of
|
||||
those.
|
||||
|
||||
Note that on each model page, you can look at the documentation of the associated tokenizer to know which of those
|
||||
algorithms the pretrained model used. For instance, if we look at :class:`~transformers.BertTokenizer`, we can see it's
|
||||
|
|
@ -16,8 +16,8 @@ Introduction to tokenization
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Splitting a text in smaller chunks is a task that's harder than it looks, and there are multiple ways of doing it. For
|
||||
instance, let's look at the sentence "Don't you love 🤗 Transformers? We sure do." A first simple way of tokenizing
|
||||
this text is just to split it by spaces, which would give:
|
||||
instance, let's look at the sentence "Don't you love 🤗 Transformers? We sure do." A first simple way of tokenizing this
|
||||
text is just to split it by spaces, which would give:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -46,9 +46,8 @@ rule-based tokenizers. On the text above, they'd output something like:
|
|||
|
||||
Space/punctuation-tokenization and rule-based tokenization are both examples of word tokenization, which is splitting a
|
||||
sentence into words. While it's the most intuitive way to separate texts in smaller chunks, it can have a problem when
|
||||
you have a huge corpus: it usually yields a very big vocabulary (the set of all unique tokens used).
|
||||
:doc:`Transformer XL <model_doc/transformerxl>` for instance uses space/punctuation-tokenization, and has a vocabulary
|
||||
size of 267,735!
|
||||
you have a huge corpus: it usually yields a very big vocabulary (the set of all unique tokens used). :doc:`Transformer
|
||||
XL <model_doc/transformerxl>` for instance uses space/punctuation-tokenization, and has a vocabulary size of 267,735!
|
||||
|
||||
A huge vocabulary size means a huge embedding matrix at the start of the model, which will cause memory problems.
|
||||
TransformerXL deals with it by using a special kind of embeddings called adaptive embeddings, but in general,
|
||||
|
|
@ -69,9 +68,8 @@ decomposed as "annoying" and "ly". This is especially useful in agglutinative la
|
|||
form (almost) arbitrarily long complex words by stringing together some subwords.
|
||||
|
||||
This allows the model to keep a reasonable vocabulary while still learning useful representations for common words or
|
||||
subwords. This also enables the model to process words it has never seen before, by decomposing them into
|
||||
subwords it knows. For instance, the base :class:`~transformers.BertTokenizer` will tokenize "I have a new GPU!" like
|
||||
this:
|
||||
subwords. This also enables the model to process words it has never seen before, by decomposing them into subwords it
|
||||
knows. For instance, the base :class:`~transformers.BertTokenizer` will tokenize "I have a new GPU!" like this:
|
||||
|
||||
.. code-block::
|
||||
|
||||
|
|
@ -81,8 +79,8 @@ this:
|
|||
['i', 'have', 'a', 'new', 'gp', '##u', '!']
|
||||
|
||||
Since we are considering the uncased model, the sentence was lowercased first. Then all the words were present in the
|
||||
vocabulary of the tokenizer, except for "gpu", so the tokenizer splits it in subwords it knows: "gp" and "##u". The "##"
|
||||
means that the rest of the token should be attached to the previous one, without space (for when we need to decode
|
||||
vocabulary of the tokenizer, except for "gpu", so the tokenizer splits it in subwords it knows: "gp" and "##u". The
|
||||
"##" means that the rest of the token should be attached to the previous one, without space (for when we need to decode
|
||||
predictions and reverse the tokenization).
|
||||
|
||||
Another example is when we use the base :class:`~transformers.XLNetTokenizer` to tokenize our previous text:
|
||||
|
|
@ -106,9 +104,9 @@ Byte-Pair Encoding
|
|||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
|
||||
Byte-Pair Encoding was introduced in `this paper <https://arxiv.org/abs/1508.07909>`__. It relies on a pretokenizer
|
||||
splitting the training data into words, which can be a simple space tokenization
|
||||
(:doc:`GPT-2 <model_doc/gpt2>` and :doc:`Roberta <model_doc/roberta>` uses this for instance) or a rule-based tokenizer
|
||||
(:doc:`XLM <model_doc/xlm>` use Moses for most languages, as does :doc:`FlauBERT <model_doc/flaubert>`),
|
||||
splitting the training data into words, which can be a simple space tokenization (:doc:`GPT-2 <model_doc/gpt2>` and
|
||||
:doc:`Roberta <model_doc/roberta>` uses this for instance) or a rule-based tokenizer (:doc:`XLM <model_doc/xlm>` use
|
||||
Moses for most languages, as does :doc:`FlauBERT <model_doc/flaubert>`),
|
||||
|
||||
:doc:`GPT <model_doc/gpt>` uses Spacy and ftfy, and counts the frequency of each word in the training corpus.
|
||||
|
||||
|
|
@ -148,10 +146,10 @@ represented as
|
|||
|
||||
('hug', 10), ('p' 'ug', 5), ('p' 'un', 12), ('b' 'un', 4), ('hug' 's', 5)
|
||||
|
||||
If we stop there, the tokenizer can apply the rules it learned to new words (as long as they don't contain characters that
|
||||
were not in the base vocabulary). For instance 'bug' would be tokenized as ``['b', 'ug']`` but mug would be tokenized as
|
||||
``['<unk>', 'ug']`` since the 'm' is not in the base vocabulary. This doesn't happen to letters in general (since the
|
||||
base corpus uses all of them), but to special characters like emojis.
|
||||
If we stop there, the tokenizer can apply the rules it learned to new words (as long as they don't contain characters
|
||||
that were not in the base vocabulary). For instance 'bug' would be tokenized as ``['b', 'ug']`` but mug would be
|
||||
tokenized as ``['<unk>', 'ug']`` since the 'm' is not in the base vocabulary. This doesn't happen to letters in general
|
||||
(since the base corpus uses all of them), but to special characters like emojis.
|
||||
|
||||
As we said before, the vocabulary size (which is the base vocabulary size + the number of merges) is a hyperparameter
|
||||
to choose. For instance :doc:`GPT <model_doc/gpt>` has a vocabulary size of 40,478 since they have 478 base characters
|
||||
|
|
@ -161,24 +159,24 @@ Byte-level BPE
|
|||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
To deal with the fact the base vocabulary needs to get all base characters, which can be quite big if one allows for
|
||||
all unicode characters, the
|
||||
`GPT-2 paper <https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf>`__
|
||||
introduces a clever trick, which is to use bytes as the base vocabulary (which gives a size of 256). With some
|
||||
additional rules to deal with punctuation, this manages to be able to tokenize every text without needing an unknown
|
||||
token. For instance, the :doc:`GPT-2 model <model_doc/gpt>` has a vocabulary size of 50,257, which corresponds to the
|
||||
256 bytes base tokens, a special end-of-text token and the symbols learned with 50,000 merges.
|
||||
all unicode characters, the `GPT-2 paper
|
||||
<https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf>`__ introduces a
|
||||
clever trick, which is to use bytes as the base vocabulary (which gives a size of 256). With some additional rules to
|
||||
deal with punctuation, this manages to be able to tokenize every text without needing an unknown token. For instance,
|
||||
the :doc:`GPT-2 model <model_doc/gpt>` has a vocabulary size of 50,257, which corresponds to the 256 bytes base tokens,
|
||||
a special end-of-text token and the symbols learned with 50,000 merges.
|
||||
|
||||
.. _wordpiece:
|
||||
|
||||
WordPiece
|
||||
=======================================================================================================================
|
||||
|
||||
WordPiece is the subword tokenization algorithm used for :doc:`BERT <model_doc/bert>` (as well as
|
||||
:doc:`DistilBERT <model_doc/distilbert>` and :doc:`Electra <model_doc/electra>`) and was outlined in
|
||||
`this paper <https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf>`__. It relies
|
||||
on the same base as BPE, which is to initialize the vocabulary to every character present in the corpus and
|
||||
progressively learn a given number of merge rules, the difference is that it doesn't choose the pair that is the most
|
||||
frequent but the one that will maximize the likelihood on the corpus once merged.
|
||||
WordPiece is the subword tokenization algorithm used for :doc:`BERT <model_doc/bert>` (as well as :doc:`DistilBERT
|
||||
<model_doc/distilbert>` and :doc:`Electra <model_doc/electra>`) and was outlined in `this paper
|
||||
<https://static.googleusercontent.com/media/research.google.com/ja//pubs/archive/37842.pdf>`__. It relies on the same
|
||||
base as BPE, which is to initialize the vocabulary to every character present in the corpus and progressively learn a
|
||||
given number of merge rules, the difference is that it doesn't choose the pair that is the most frequent but the one
|
||||
that will maximize the likelihood on the corpus once merged.
|
||||
|
||||
What does this mean? Well, in the previous example, it means we would only merge 'u' and 'g' if the probability of
|
||||
having 'ug' divided by the probability of having 'u' then 'g' is greater than for any other pair of symbols. It's
|
||||
|
|
@ -198,10 +196,10 @@ with :ref:`SentencePiece <sentencepiece>`.
|
|||
|
||||
More specifically, at a given step, unigram computes a loss from the corpus we have and the current vocabulary, then,
|
||||
for each subword, evaluate how much the loss would increase if the subword was removed from the vocabulary. It then
|
||||
sorts the subwords by this quantity (that represents how much worse the loss becomes if the token is removed) and removes
|
||||
all the worst p tokens (for instance p could be 10% or 20%). It then repeats the process until the vocabulary has
|
||||
reached the desired size, always keeping the base characters (to be able to tokenize any word written with them, like
|
||||
BPE or WordPiece).
|
||||
sorts the subwords by this quantity (that represents how much worse the loss becomes if the token is removed) and
|
||||
removes all the worst p tokens (for instance p could be 10% or 20%). It then repeats the process until the vocabulary
|
||||
has reached the desired size, always keeping the base characters (to be able to tokenize any word written with them,
|
||||
like BPE or WordPiece).
|
||||
|
||||
Contrary to BPE and WordPiece that work out rules in a certain order that you can then apply in the same order when
|
||||
tokenizing new text, Unigram will have several ways of tokenizing a new text. For instance, if it ends up with the
|
||||
|
|
@ -217,9 +215,9 @@ training corpus. You can then give a probability to each tokenization (which is
|
|||
tokens forming it) and pick the most likely one (or if you want to apply some data augmentation, you could sample one
|
||||
of the tokenization according to their probabilities).
|
||||
|
||||
Those probabilities define the loss that trains the tokenizer: if our corpus consists of the
|
||||
words :math:`x_{1}, \dots, x_{N}` and if for the word :math:`x_{i}` we note :math:`S(x_{i})` the set of all possible
|
||||
tokenizations of :math:`x_{i}` (with the current vocabulary), then the loss is defined as
|
||||
Those probabilities define the loss that trains the tokenizer: if our corpus consists of the words :math:`x_{1}, \dots,
|
||||
x_{N}` and if for the word :math:`x_{i}` we note :math:`S(x_{i})` the set of all possible tokenizations of
|
||||
:math:`x_{i}` (with the current vocabulary), then the loss is defined as
|
||||
|
||||
.. math::
|
||||
\mathcal{L} = -\sum_{i=1}^{N} \log \left ( \sum_{x \in S(x_{i})} p(x) \right )
|
||||
|
|
@ -236,8 +234,8 @@ SentencePiece (introduced in `this paper <https://arxiv.org/pdf/1808.06226.pdf>`
|
|||
includes the space in the set of characters to use, then uses BPE or unigram to construct the appropriate vocabulary.
|
||||
|
||||
That's why in the example we saw before using :class:`~transformers.XLNetTokenizer` (which uses SentencePiece), we had
|
||||
the '▁' character, that represents space. Decoding a tokenized text is then super easy: we just have to concatenate
|
||||
all of them together and replace '▁' with space.
|
||||
the '▁' character, that represents space. Decoding a tokenized text is then super easy: we just have to concatenate all
|
||||
of them together and replace '▁' with space.
|
||||
|
||||
All transformers models in the library that use SentencePiece use it with unigram. Examples of models using it are
|
||||
:doc:`ALBERT <model_doc/albert>`, :doc:`XLNet <model_doc/xlnet>` or the :doc:`Marian framework <model_doc/marian>`.
|
||||
|
|
|
|||
|
|
@ -1,18 +1,14 @@
|
|||
Training and fine-tuning
|
||||
=======================================================================================================================
|
||||
|
||||
Model classes in 🤗 Transformers are designed to be compatible with native
|
||||
PyTorch and TensorFlow 2 and can be used seemlessly with either. In this
|
||||
quickstart, we will show how to fine-tune (or train from scratch) a model
|
||||
using the standard training tools available in either framework. We will also
|
||||
show how to use our included :func:`~transformers.Trainer` class which
|
||||
handles much of the complexity of training for you.
|
||||
Model classes in 🤗 Transformers are designed to be compatible with native PyTorch and TensorFlow 2 and can be used
|
||||
seemlessly with either. In this quickstart, we will show how to fine-tune (or train from scratch) a model using the
|
||||
standard training tools available in either framework. We will also show how to use our included
|
||||
:func:`~transformers.Trainer` class which handles much of the complexity of training for you.
|
||||
|
||||
This guide assume that you are already familiar with loading and use our
|
||||
models for inference; otherwise, see the :doc:`task summary <task_summary>`. We also assume
|
||||
that you are familiar with training deep neural networks in either PyTorch or
|
||||
TF2, and focus specifically on the nuances and tools for training models in
|
||||
🤗 Transformers.
|
||||
This guide assume that you are already familiar with loading and use our models for inference; otherwise, see the
|
||||
:doc:`task summary <task_summary>`. We also assume that you are familiar with training deep neural networks in either
|
||||
PyTorch or TF2, and focus specifically on the nuances and tools for training models in 🤗 Transformers.
|
||||
|
||||
Sections:
|
||||
|
||||
|
|
@ -26,25 +22,19 @@ Sections:
|
|||
Fine-tuning in native PyTorch
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Model classes in 🤗 Transformers that don't begin with ``TF`` are
|
||||
`PyTorch Modules <https://pytorch.org/docs/master/generated/torch.nn.Module.html>`_,
|
||||
meaning that you can use them just as you would any model in PyTorch for
|
||||
both inference and optimization.
|
||||
Model classes in 🤗 Transformers that don't begin with ``TF`` are `PyTorch Modules
|
||||
<https://pytorch.org/docs/master/generated/torch.nn.Module.html>`_, meaning that you can use them just as you would any
|
||||
model in PyTorch for both inference and optimization.
|
||||
|
||||
Let's consider the common task of fine-tuning a masked language model like
|
||||
BERT on a sequence classification dataset. When we instantiate a model with
|
||||
:func:`~transformers.PreTrainedModel.from_pretrained`, the model
|
||||
configuration and pre-trained weights
|
||||
of the specified model are used to initialize the model. The
|
||||
library also includes a number of task-specific final layers or 'heads' whose
|
||||
weights are instantiated randomly when not present in the specified
|
||||
Let's consider the common task of fine-tuning a masked language model like BERT on a sequence classification dataset.
|
||||
When we instantiate a model with :func:`~transformers.PreTrainedModel.from_pretrained`, the model configuration and
|
||||
pre-trained weights of the specified model are used to initialize the model. The library also includes a number of
|
||||
task-specific final layers or 'heads' whose weights are instantiated randomly when not present in the specified
|
||||
pre-trained model. For example, instantiating a model with
|
||||
``BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)``
|
||||
will create a BERT model instance with encoder weights copied from the
|
||||
``bert-base-uncased`` model and a randomly initialized sequence
|
||||
classification head on top of the encoder with an output size of 2. Models
|
||||
are initialized in ``eval`` mode by default. We can call ``model.train()`` to
|
||||
put it in train mode.
|
||||
``BertForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)`` will create a BERT model instance
|
||||
with encoder weights copied from the ``bert-base-uncased`` model and a randomly initialized sequence classification
|
||||
head on top of the encoder with an output size of 2. Models are initialized in ``eval`` mode by default. We can call
|
||||
``model.train()`` to put it in train mode.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -52,20 +42,17 @@ put it in train mode.
|
|||
model = BertForSequenceClassification.from_pretrained('bert-base-uncased', return_dict=True)
|
||||
model.train()
|
||||
|
||||
This is useful because it allows us to make use of the pre-trained BERT
|
||||
encoder and easily train it on whatever sequence classification dataset we
|
||||
choose. We can use any PyTorch optimizer, but our library also provides the
|
||||
:func:`~transformers.AdamW` optimizer which implements gradient bias
|
||||
correction as well as weight decay.
|
||||
This is useful because it allows us to make use of the pre-trained BERT encoder and easily train it on whatever
|
||||
sequence classification dataset we choose. We can use any PyTorch optimizer, but our library also provides the
|
||||
:func:`~transformers.AdamW` optimizer which implements gradient bias correction as well as weight decay.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
from transformers import AdamW
|
||||
optimizer = AdamW(model.parameters(), lr=1e-5)
|
||||
|
||||
The optimizer allows us to apply different hyperpameters for specific
|
||||
parameter groups. For example, we can apply weight decay to all parameters
|
||||
other than bias and layer normalization terms:
|
||||
The optimizer allows us to apply different hyperpameters for specific parameter groups. For example, we can apply
|
||||
weight decay to all parameters other than bias and layer normalization terms:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -75,11 +62,9 @@ other than bias and layer normalization terms:
|
|||
{'params': [p for n, p in model.named_parameters() if any(nd in n for nd in no_decay)], 'weight_decay': 0.0}
|
||||
]
|
||||
optimizer = AdamW(optimizer_grouped_parameters, lr=1e-5)
|
||||
|
||||
Now we can set up a simple dummy training batch using
|
||||
:func:`~transformers.PreTrainedTokenizer.__call__`. This returns a
|
||||
:func:`~transformers.BatchEncoding` instance which
|
||||
prepares everything we might need to pass to the model.
|
||||
|
||||
Now we can set up a simple dummy training batch using :func:`~transformers.PreTrainedTokenizer.__call__`. This returns
|
||||
a :func:`~transformers.BatchEncoding` instance which prepares everything we might need to pass to the model.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -90,10 +75,9 @@ prepares everything we might need to pass to the model.
|
|||
input_ids = encoding['input_ids']
|
||||
attention_mask = encoding['attention_mask']
|
||||
|
||||
When we call a classification model with the ``labels`` argument, the first
|
||||
returned element is the Cross Entropy loss between the predictions and the
|
||||
passed labels. Having already set up our optimizer, we can then do a
|
||||
backwards pass and update the weights:
|
||||
When we call a classification model with the ``labels`` argument, the first returned element is the Cross Entropy loss
|
||||
between the predictions and the passed labels. Having already set up our optimizer, we can then do a backwards pass and
|
||||
update the weights:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -103,8 +87,8 @@ backwards pass and update the weights:
|
|||
loss.backward()
|
||||
optimizer.step()
|
||||
|
||||
Alternatively, you can just get the logits and calculate the loss yourself.
|
||||
The following is equivalent to the previous example:
|
||||
Alternatively, you can just get the logits and calculate the loss yourself. The following is equivalent to the previous
|
||||
example:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -115,12 +99,10 @@ The following is equivalent to the previous example:
|
|||
loss.backward()
|
||||
optimizer.step()
|
||||
|
||||
Of course, you can train on GPU by calling ``to('cuda')`` on the model and
|
||||
inputs as usual.
|
||||
Of course, you can train on GPU by calling ``to('cuda')`` on the model and inputs as usual.
|
||||
|
||||
We also provide a few learning rate scheduling tools. With the following, we
|
||||
can set up a scheduler which warms up for ``num_warmup_steps`` and then
|
||||
linearly decays to 0 by the end of training.
|
||||
We also provide a few learning rate scheduling tools. With the following, we can set up a scheduler which warms up for
|
||||
``num_warmup_steps`` and then linearly decays to 0 by the end of training.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -135,19 +117,16 @@ Then all we have to do is call ``scheduler.step()`` after ``optimizer.step()``.
|
|||
optimizer.step()
|
||||
scheduler.step()
|
||||
|
||||
We highly recommend using :func:`~transformers.Trainer`, discussed below,
|
||||
which conveniently handles the moving parts of training 🤗 Transformers models
|
||||
with features like mixed precision and easy tensorboard logging.
|
||||
We highly recommend using :func:`~transformers.Trainer`, discussed below, which conveniently handles the moving parts
|
||||
of training 🤗 Transformers models with features like mixed precision and easy tensorboard logging.
|
||||
|
||||
|
||||
Freezing the encoder
|
||||
-----------------------------------------------------------------------------------------------------------------------
|
||||
|
||||
In some cases, you might be interested in keeping the weights of the
|
||||
pre-trained encoder frozen and optimizing only the weights of the head
|
||||
layers. To do so, simply set the ``requires_grad`` attribute to ``False`` on
|
||||
the encoder parameters, which can be accessed with the ``base_model``
|
||||
submodule on any task-specific model in the library:
|
||||
In some cases, you might be interested in keeping the weights of the pre-trained encoder frozen and optimizing only the
|
||||
weights of the head layers. To do so, simply set the ``requires_grad`` attribute to ``False`` on the encoder
|
||||
parameters, which can be accessed with the ``base_model`` submodule on any task-specific model in the library:
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -160,10 +139,8 @@ submodule on any task-specific model in the library:
|
|||
Fine-tuning in native TensorFlow 2
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Models can also be trained natively in TensorFlow 2. Just as with PyTorch,
|
||||
TensorFlow models can be instantiated with
|
||||
:func:`~transformers.PreTrainedModel.from_pretrained` to load the weights of
|
||||
the encoder from a pretrained model.
|
||||
Models can also be trained natively in TensorFlow 2. Just as with PyTorch, TensorFlow models can be instantiated with
|
||||
:func:`~transformers.PreTrainedModel.from_pretrained` to load the weights of the encoder from a pretrained model.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -171,11 +148,9 @@ the encoder from a pretrained model.
|
|||
model = TFBertForSequenceClassification.from_pretrained('bert-base-uncased')
|
||||
|
||||
Let's use ``tensorflow_datasets`` to load in the `MRPC dataset
|
||||
<https://www.tensorflow.org/datasets/catalog/glue#gluemrpc>`_ from GLUE. We
|
||||
can then use our built-in
|
||||
:func:`~transformers.data.processors.glue.glue_convert_examples_to_features`
|
||||
to tokenize MRPC and convert it to a TensorFlow ``Dataset`` object. Note that
|
||||
tokenizers are framework-agnostic, so there is no need to prepend ``TF`` to
|
||||
<https://www.tensorflow.org/datasets/catalog/glue#gluemrpc>`_ from GLUE. We can then use our built-in
|
||||
:func:`~transformers.data.processors.glue.glue_convert_examples_to_features` to tokenize MRPC and convert it to a
|
||||
TensorFlow ``Dataset`` object. Note that tokenizers are framework-agnostic, so there is no need to prepend ``TF`` to
|
||||
the pretrained tokenizer name.
|
||||
|
||||
.. code-block:: python
|
||||
|
|
@ -197,8 +172,8 @@ The model can then be compiled and trained as any Keras model:
|
|||
model.compile(optimizer=optimizer, loss=loss)
|
||||
model.fit(train_dataset, epochs=2, steps_per_epoch=115)
|
||||
|
||||
With the tight interoperability between TensorFlow and PyTorch models, you
|
||||
can even save the model and then reload it as a PyTorch model (or vice-versa):
|
||||
With the tight interoperability between TensorFlow and PyTorch models, you can even save the model and then reload it
|
||||
as a PyTorch model (or vice-versa):
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -212,12 +187,9 @@ can even save the model and then reload it as a PyTorch model (or vice-versa):
|
|||
Trainer
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
We also provide a simple but feature-complete training and evaluation
|
||||
interface through :func:`~transformers.Trainer` and
|
||||
:func:`~transformers.TFTrainer`. You can train, fine-tune,
|
||||
and evaluate any 🤗 Transformers model with a wide range of training options and
|
||||
with built-in features like logging, gradient accumulation, and mixed
|
||||
precision.
|
||||
We also provide a simple but feature-complete training and evaluation interface through :func:`~transformers.Trainer`
|
||||
and :func:`~transformers.TFTrainer`. You can train, fine-tune, and evaluate any 🤗 Transformers model with a wide range
|
||||
of training options and with built-in features like logging, gradient accumulation, and mixed precision.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -264,21 +236,16 @@ precision.
|
|||
eval_dataset=tfds_test_dataset # tensorflow_datasets evaluation dataset
|
||||
)
|
||||
|
||||
Now simply call ``trainer.train()`` to train and ``trainer.evaluate()`` to
|
||||
evaluate. You can use your own module as well, but the first
|
||||
argument returned from ``forward`` must be the loss which you wish to
|
||||
optimize.
|
||||
Now simply call ``trainer.train()`` to train and ``trainer.evaluate()`` to evaluate. You can use your own module as
|
||||
well, but the first argument returned from ``forward`` must be the loss which you wish to optimize.
|
||||
|
||||
:func:`~transformers.Trainer` uses a built-in default function to collate
|
||||
batches and prepare them to be fed into the model. If needed, you can also
|
||||
use the ``data_collator`` argument to pass your own collator function which
|
||||
takes in the data in the format provided by your dataset and returns a
|
||||
batch ready to be fed into the model. Note that
|
||||
:func:`~transformers.TFTrainer` expects the passed datasets to be dataset
|
||||
objects from ``tensorflow_datasets``.
|
||||
:func:`~transformers.Trainer` uses a built-in default function to collate batches and prepare them to be fed into the
|
||||
model. If needed, you can also use the ``data_collator`` argument to pass your own collator function which takes in the
|
||||
data in the format provided by your dataset and returns a batch ready to be fed into the model. Note that
|
||||
:func:`~transformers.TFTrainer` expects the passed datasets to be dataset objects from ``tensorflow_datasets``.
|
||||
|
||||
To calculate additional metrics in addition to the loss, you can also define
|
||||
your own ``compute_metrics`` function and pass it to the trainer.
|
||||
To calculate additional metrics in addition to the loss, you can also define your own ``compute_metrics`` function and
|
||||
pass it to the trainer.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
|
|
@ -296,8 +263,8 @@ your own ``compute_metrics`` function and pass it to the trainer.
|
|||
'recall': recall
|
||||
}
|
||||
|
||||
Finally, you can view the results, including any calculated metrics, by
|
||||
launching tensorboard in your specified ``logging_dir`` directory.
|
||||
Finally, you can view the results, including any calculated metrics, by launching tensorboard in your specified
|
||||
``logging_dir`` directory.
|
||||
|
||||
|
||||
.. _additional-resources:
|
||||
|
|
@ -308,11 +275,12 @@ Additional resources
|
|||
- `A lightweight colab demo <https://colab.research.google.com/drive/1-JIJlao4dI-Ilww_NnTc0rxtp-ymgDgM?usp=sharing>`_
|
||||
which uses ``Trainer`` for IMDb sentiment classification.
|
||||
|
||||
- `🤗 Transformers Examples <https://github.com/huggingface/transformers/tree/master/examples>`_
|
||||
including scripts for training and fine-tuning on GLUE, SQuAD, and several other tasks.
|
||||
- `🤗 Transformers Examples <https://github.com/huggingface/transformers/tree/master/examples>`_ including scripts for
|
||||
training and fine-tuning on GLUE, SQuAD, and several other tasks.
|
||||
|
||||
- `How to train a language model <https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb>`_,
|
||||
a detailed colab notebook which uses ``Trainer`` to train a masked language model from scratch on Esperanto.
|
||||
- `How to train a language model
|
||||
<https://colab.research.google.com/github/huggingface/blog/blob/master/notebooks/01_how_to_train.ipynb>`_, a detailed
|
||||
colab notebook which uses ``Trainer`` to train a masked language model from scratch on Esperanto.
|
||||
|
||||
- `🤗 Transformers Notebooks <notebooks.html>`_ which contain dozens of example notebooks from the community for
|
||||
training and using 🤗 Transformers on a variety of tasks.
|
||||
|
|
|
|||
|
|
@ -14,18 +14,19 @@ def swish(x):
|
|||
|
||||
|
||||
def _gelu_python(x):
|
||||
"""Original Implementation of the gelu activation function in Google Bert repo when initially created.
|
||||
For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
|
||||
0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
|
||||
This is now written in C in torch.nn.functional
|
||||
Also see https://arxiv.org/abs/1606.08415
|
||||
"""
|
||||
Original Implementation of the gelu activation function in Google Bert repo when initially created. For
|
||||
information: OpenAI GPT's gelu is slightly different (and gives slightly different results): 0.5 * x * (1 +
|
||||
torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) This is now written in C in
|
||||
torch.nn.functional Also see https://arxiv.org/abs/1606.08415
|
||||
"""
|
||||
return x * 0.5 * (1.0 + torch.erf(x / math.sqrt(2.0)))
|
||||
|
||||
|
||||
def gelu_new(x):
|
||||
"""Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT).
|
||||
Also see https://arxiv.org/abs/1606.08415
|
||||
"""
|
||||
Implementation of the gelu activation function currently in Google Bert repo (identical to OpenAI GPT). Also see
|
||||
https://arxiv.org/abs/1606.08415
|
||||
"""
|
||||
return 0.5 * x * (1.0 + torch.tanh(math.sqrt(2.0 / math.pi) * (x + 0.044715 * torch.pow(x, 3.0))))
|
||||
|
||||
|
|
|
|||
|
|
@ -4,11 +4,11 @@ import tensorflow as tf
|
|||
|
||||
|
||||
def gelu(x):
|
||||
"""Gaussian Error Linear Unit.
|
||||
Original Implementation of the gelu activation function in Google Bert repo when initially created.
|
||||
For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
|
||||
0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3))))
|
||||
Also see https://arxiv.org/abs/1606.08415
|
||||
"""
|
||||
Gaussian Error Linear Unit. Original Implementation of the gelu activation function in Google Bert repo when
|
||||
initially created. For information: OpenAI GPT's gelu is slightly different (and gives slightly different results):
|
||||
0.5 * x * (1 + torch.tanh(math.sqrt(2 / math.pi) * (x + 0.044715 * torch.pow(x, 3)))) Also see
|
||||
https://arxiv.org/abs/1606.08415
|
||||
"""
|
||||
x = tf.convert_to_tensor(x)
|
||||
cdf = 0.5 * (1.0 + tf.math.erf(x / tf.math.sqrt(2.0)))
|
||||
|
|
@ -17,11 +17,12 @@ def gelu(x):
|
|||
|
||||
|
||||
def gelu_new(x):
|
||||
"""Gaussian Error Linear Unit.
|
||||
This is a smoother version of the GELU.
|
||||
Original paper: https://arxiv.org/abs/1606.08415
|
||||
"""
|
||||
Gaussian Error Linear Unit. This is a smoother version of the GELU. Original paper: https://arxiv.org/abs/1606.0841
|
||||
|
||||
Args:
|
||||
x: float Tensor to perform activation.
|
||||
x: float Tensor to perform activation
|
||||
|
||||
Returns:
|
||||
`x` with the GELU activation applied.
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -46,8 +46,9 @@ class PyTorchBenchmarkArguments(BenchmarkArguments):
|
|||
]
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
"""This __init__ is there for legacy code. When removing
|
||||
deprecated args completely, the class can simply be deleted
|
||||
"""
|
||||
This __init__ is there for legacy code. When removing deprecated args completely, the class can simply be
|
||||
deleted
|
||||
"""
|
||||
for deprecated_arg in self.deprecated_args:
|
||||
if deprecated_arg in kwargs:
|
||||
|
|
|
|||
|
|
@ -43,8 +43,9 @@ class TensorFlowBenchmarkArguments(BenchmarkArguments):
|
|||
]
|
||||
|
||||
def __init__(self, **kwargs):
|
||||
"""This __init__ is there for legacy code. When removing
|
||||
deprecated args completely, the class can simply be deleted
|
||||
"""
|
||||
This __init__ is there for legacy code. When removing deprecated args completely, the class can simply be
|
||||
deleted
|
||||
"""
|
||||
for deprecated_arg in self.deprecated_args:
|
||||
if deprecated_arg in kwargs:
|
||||
|
|
|
|||
|
|
@ -1,147 +1,145 @@
|
|||
# coding=utf-8
|
||||
# Copyright 2018 The HuggingFace Inc. team.
|
||||
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import dataclasses
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from time import time
|
||||
from typing import List
|
||||
|
||||
from ..utils import logging
|
||||
|
||||
|
||||
logger = logging.get_logger(__name__)
|
||||
|
||||
|
||||
def list_field(default=None, metadata=None):
|
||||
return field(default_factory=lambda: default, metadata=metadata)
|
||||
|
||||
|
||||
@dataclass
|
||||
class BenchmarkArguments:
|
||||
"""
|
||||
BenchMarkArguments are arguments we use in our benchmark scripts
|
||||
**which relate to the training loop itself**.
|
||||
|
||||
Using `HfArgumentParser` we can turn this class
|
||||
into argparse arguments to be able to specify them on
|
||||
the command line.
|
||||
"""
|
||||
|
||||
models: List[str] = list_field(
|
||||
default=[],
|
||||
metadata={
|
||||
"help": "Model checkpoints to be provided to the AutoModel classes. Leave blank to benchmark the base version of all available models"
|
||||
},
|
||||
)
|
||||
|
||||
batch_sizes: List[int] = list_field(
|
||||
default=[8], metadata={"help": "List of batch sizes for which memory and time performance will be evaluated"}
|
||||
)
|
||||
|
||||
sequence_lengths: List[int] = list_field(
|
||||
default=[8, 32, 128, 512],
|
||||
metadata={"help": "List of sequence lengths for which memory and time performance will be evaluated"},
|
||||
)
|
||||
|
||||
inference: bool = field(
|
||||
default=True,
|
||||
metadata={"help": "Whether to benchmark inference of model. Inference can be disabled via --no-inference."},
|
||||
)
|
||||
cuda: bool = field(
|
||||
default=True,
|
||||
metadata={"help": "Whether to run on available cuda devices. Cuda can be disabled via --no-cuda."},
|
||||
)
|
||||
tpu: bool = field(
|
||||
default=True, metadata={"help": "Whether to run on available tpu devices. TPU can be disabled via --no-tpu."}
|
||||
)
|
||||
fp16: bool = field(default=False, metadata={"help": "Use FP16 to accelerate inference."})
|
||||
training: bool = field(default=False, metadata={"help": "Benchmark training of model"})
|
||||
verbose: bool = field(default=False, metadata={"help": "Verbose memory tracing"})
|
||||
speed: bool = field(
|
||||
default=True,
|
||||
metadata={"help": "Whether to perform speed measurements. Speed measurements can be disabled via --no-speed."},
|
||||
)
|
||||
memory: bool = field(
|
||||
default=True,
|
||||
metadata={
|
||||
"help": "Whether to perform memory measurements. Memory measurements can be disabled via --no-memory"
|
||||
},
|
||||
)
|
||||
trace_memory_line_by_line: bool = field(default=False, metadata={"help": "Trace memory line by line"})
|
||||
save_to_csv: bool = field(default=False, metadata={"help": "Save result to a CSV file"})
|
||||
log_print: bool = field(default=False, metadata={"help": "Save all print statements in a log file"})
|
||||
env_print: bool = field(default=False, metadata={"help": "Whether to print environment information"})
|
||||
multi_process: bool = field(
|
||||
default=True,
|
||||
metadata={
|
||||
"help": "Whether to use multiprocessing for memory and speed measurement. It is highly recommended to use multiprocessing for accurate CPU and GPU memory measurements. This option should only be disabled for debugging / testing and on TPU."
|
||||
},
|
||||
)
|
||||
inference_time_csv_file: str = field(
|
||||
default=f"inference_time_{round(time())}.csv",
|
||||
metadata={"help": "CSV filename used if saving time results to csv."},
|
||||
)
|
||||
inference_memory_csv_file: str = field(
|
||||
default=f"inference_memory_{round(time())}.csv",
|
||||
metadata={"help": "CSV filename used if saving memory results to csv."},
|
||||
)
|
||||
train_time_csv_file: str = field(
|
||||
default=f"train_time_{round(time())}.csv",
|
||||
metadata={"help": "CSV filename used if saving time results to csv for training."},
|
||||
)
|
||||
train_memory_csv_file: str = field(
|
||||
default=f"train_memory_{round(time())}.csv",
|
||||
metadata={"help": "CSV filename used if saving memory results to csv for training."},
|
||||
)
|
||||
env_info_csv_file: str = field(
|
||||
default=f"env_info_{round(time())}.csv",
|
||||
metadata={"help": "CSV filename used if saving environment information."},
|
||||
)
|
||||
log_filename: str = field(
|
||||
default=f"log_{round(time())}.csv",
|
||||
metadata={"help": "Log filename used if print statements are saved in log."},
|
||||
)
|
||||
repeat: int = field(default=3, metadata={"help": "Times an experiment will be run."})
|
||||
only_pretrain_model: bool = field(
|
||||
default=False,
|
||||
metadata={
|
||||
"help": "Instead of loading the model as defined in `config.architectures` if exists, just load the pretrain model weights."
|
||||
},
|
||||
)
|
||||
|
||||
def to_json_string(self):
|
||||
"""
|
||||
Serializes this instance to a JSON string.
|
||||
"""
|
||||
return json.dumps(dataclasses.asdict(self), indent=2)
|
||||
|
||||
@property
|
||||
def model_names(self):
|
||||
assert (
|
||||
len(self.models) > 0
|
||||
), "Please make sure you provide at least one model name / model identifier, *e.g.* `--models bert-base-cased` or `args.models = ['bert-base-cased']."
|
||||
return self.models
|
||||
|
||||
@property
|
||||
def do_multi_processing(self):
|
||||
if not self.multi_process:
|
||||
return False
|
||||
elif self.is_tpu:
|
||||
logger.info("Multiprocessing is currently not possible on TPU.")
|
||||
return False
|
||||
else:
|
||||
return True
|
||||
# coding=utf-8
|
||||
# Copyright 2018 The HuggingFace Inc. team.
|
||||
# Copyright (c) 2018, NVIDIA CORPORATION. All rights reserved.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
import dataclasses
|
||||
import json
|
||||
from dataclasses import dataclass, field
|
||||
from time import time
|
||||
from typing import List
|
||||
|
||||
from ..utils import logging
|
||||
|
||||
|
||||
logger = logging.get_logger(__name__)
|
||||
|
||||
|
||||
def list_field(default=None, metadata=None):
|
||||
return field(default_factory=lambda: default, metadata=metadata)
|
||||
|
||||
|
||||
@dataclass
|
||||
class BenchmarkArguments:
|
||||
"""
|
||||
BenchMarkArguments are arguments we use in our benchmark scripts **which relate to the training loop itself**.
|
||||
|
||||
Using `HfArgumentParser` we can turn this class into argparse arguments to be able to specify them on the command
|
||||
line.
|
||||
"""
|
||||
|
||||
models: List[str] = list_field(
|
||||
default=[],
|
||||
metadata={
|
||||
"help": "Model checkpoints to be provided to the AutoModel classes. Leave blank to benchmark the base version of all available models"
|
||||
},
|
||||
)
|
||||
|
||||
batch_sizes: List[int] = list_field(
|
||||
default=[8], metadata={"help": "List of batch sizes for which memory and time performance will be evaluated"}
|
||||
)
|
||||
|
||||
sequence_lengths: List[int] = list_field(
|
||||
default=[8, 32, 128, 512],
|
||||
metadata={"help": "List of sequence lengths for which memory and time performance will be evaluated"},
|
||||
)
|
||||
|
||||
inference: bool = field(
|
||||
default=True,
|
||||
metadata={"help": "Whether to benchmark inference of model. Inference can be disabled via --no-inference."},
|
||||
)
|
||||
cuda: bool = field(
|
||||
default=True,
|
||||
metadata={"help": "Whether to run on available cuda devices. Cuda can be disabled via --no-cuda."},
|
||||
)
|
||||
tpu: bool = field(
|
||||
default=True, metadata={"help": "Whether to run on available tpu devices. TPU can be disabled via --no-tpu."}
|
||||
)
|
||||
fp16: bool = field(default=False, metadata={"help": "Use FP16 to accelerate inference."})
|
||||
training: bool = field(default=False, metadata={"help": "Benchmark training of model"})
|
||||
verbose: bool = field(default=False, metadata={"help": "Verbose memory tracing"})
|
||||
speed: bool = field(
|
||||
default=True,
|
||||
metadata={"help": "Whether to perform speed measurements. Speed measurements can be disabled via --no-speed."},
|
||||
)
|
||||
memory: bool = field(
|
||||
default=True,
|
||||
metadata={
|
||||
"help": "Whether to perform memory measurements. Memory measurements can be disabled via --no-memory"
|
||||
},
|
||||
)
|
||||
trace_memory_line_by_line: bool = field(default=False, metadata={"help": "Trace memory line by line"})
|
||||
save_to_csv: bool = field(default=False, metadata={"help": "Save result to a CSV file"})
|
||||
log_print: bool = field(default=False, metadata={"help": "Save all print statements in a log file"})
|
||||
env_print: bool = field(default=False, metadata={"help": "Whether to print environment information"})
|
||||
multi_process: bool = field(
|
||||
default=True,
|
||||
metadata={
|
||||
"help": "Whether to use multiprocessing for memory and speed measurement. It is highly recommended to use multiprocessing for accurate CPU and GPU memory measurements. This option should only be disabled for debugging / testing and on TPU."
|
||||
},
|
||||
)
|
||||
inference_time_csv_file: str = field(
|
||||
default=f"inference_time_{round(time())}.csv",
|
||||
metadata={"help": "CSV filename used if saving time results to csv."},
|
||||
)
|
||||
inference_memory_csv_file: str = field(
|
||||
default=f"inference_memory_{round(time())}.csv",
|
||||
metadata={"help": "CSV filename used if saving memory results to csv."},
|
||||
)
|
||||
train_time_csv_file: str = field(
|
||||
default=f"train_time_{round(time())}.csv",
|
||||
metadata={"help": "CSV filename used if saving time results to csv for training."},
|
||||
)
|
||||
train_memory_csv_file: str = field(
|
||||
default=f"train_memory_{round(time())}.csv",
|
||||
metadata={"help": "CSV filename used if saving memory results to csv for training."},
|
||||
)
|
||||
env_info_csv_file: str = field(
|
||||
default=f"env_info_{round(time())}.csv",
|
||||
metadata={"help": "CSV filename used if saving environment information."},
|
||||
)
|
||||
log_filename: str = field(
|
||||
default=f"log_{round(time())}.csv",
|
||||
metadata={"help": "Log filename used if print statements are saved in log."},
|
||||
)
|
||||
repeat: int = field(default=3, metadata={"help": "Times an experiment will be run."})
|
||||
only_pretrain_model: bool = field(
|
||||
default=False,
|
||||
metadata={
|
||||
"help": "Instead of loading the model as defined in `config.architectures` if exists, just load the pretrain model weights."
|
||||
},
|
||||
)
|
||||
|
||||
def to_json_string(self):
|
||||
"""
|
||||
Serializes this instance to a JSON string.
|
||||
"""
|
||||
return json.dumps(dataclasses.asdict(self), indent=2)
|
||||
|
||||
@property
|
||||
def model_names(self):
|
||||
assert (
|
||||
len(self.models) > 0
|
||||
), "Please make sure you provide at least one model name / model identifier, *e.g.* `--models bert-base-cased` or `args.models = ['bert-base-cased']."
|
||||
return self.models
|
||||
|
||||
@property
|
||||
def do_multi_processing(self):
|
||||
if not self.multi_process:
|
||||
return False
|
||||
elif self.is_tpu:
|
||||
logger.info("Multiprocessing is currently not possible on TPU.")
|
||||
return False
|
||||
else:
|
||||
return True
|
||||
|
|
|
|||
Разница между файлами не показана из-за своего большого размера
Загрузить разницу
|
|
@ -16,9 +16,9 @@ def convert_command_factory(args: Namespace):
|
|||
)
|
||||
|
||||
|
||||
IMPORT_ERROR_MESSAGE = """transformers can only be used from the commandline to convert TensorFlow models in PyTorch,
|
||||
In that case, it requires TensorFlow to be installed. Please see
|
||||
https://www.tensorflow.org/install/ for installation instructions.
|
||||
IMPORT_ERROR_MESSAGE = """
|
||||
transformers can only be used from the commandline to convert TensorFlow models in PyTorch, In that case, it requires
|
||||
TensorFlow to be installed. Please see https://www.tensorflow.org/install/ for installation instructions.
|
||||
"""
|
||||
|
||||
|
||||
|
|
|
|||
|
|
@ -164,9 +164,9 @@ class ServeCommand(BaseTransformersCLICommand):
|
|||
|
||||
def tokenize(self, text_input: str = Body(None, embed=True), return_ids: bool = Body(False, embed=True)):
|
||||
"""
|
||||
Tokenize the provided input and eventually returns corresponding tokens id:
|
||||
- **text_input**: String to tokenize
|
||||
- **return_ids**: Boolean flags indicating if the tokens have to be converted to their integer mapping.
|
||||
Tokenize the provided input and eventually returns corresponding tokens id: - **text_input**: String to
|
||||
tokenize - **return_ids**: Boolean flags indicating if the tokens have to be converted to their integer
|
||||
mapping.
|
||||
"""
|
||||
try:
|
||||
tokens_txt = self._pipeline.tokenizer.tokenize(text_input)
|
||||
|
|
@ -187,10 +187,9 @@ class ServeCommand(BaseTransformersCLICommand):
|
|||
cleanup_tokenization_spaces: bool = Body(True, embed=True),
|
||||
):
|
||||
"""
|
||||
Detokenize the provided tokens ids to readable text:
|
||||
- **tokens_ids**: List of tokens ids
|
||||
- **skip_special_tokens**: Flag indicating to not try to decode special tokens
|
||||
- **cleanup_tokenization_spaces**: Flag indicating to remove all leading/trailing spaces and intermediate ones.
|
||||
Detokenize the provided tokens ids to readable text: - **tokens_ids**: List of tokens ids -
|
||||
**skip_special_tokens**: Flag indicating to not try to decode special tokens - **cleanup_tokenization_spaces**:
|
||||
Flag indicating to remove all leading/trailing spaces and intermediate ones.
|
||||
"""
|
||||
try:
|
||||
decoded_str = self._pipeline.tokenizer.decode(tokens_ids, skip_special_tokens, cleanup_tokenization_spaces)
|
||||
|
|
|
|||
|
|
@ -37,9 +37,8 @@ class AlbertConfig(PretrainedConfig):
|
|||
arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar
|
||||
configuration to that of the ALBERT `xxlarge <https://huggingface.co/albert-xxlarge-v2>`__ architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 30000):
|
||||
|
|
@ -61,15 +60,15 @@ class AlbertConfig(PretrainedConfig):
|
|||
inner_group_num (:obj:`int`, `optional`, defaults to 1):
|
||||
The number of inner repetition of attention and ffn.
|
||||
hidden_act (:obj:`str` or :obj:`Callable`, `optional`, defaults to :obj:`"gelu_new"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout_prob (:obj:`float`, `optional`, defaults to 0):
|
||||
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_probs_dropout_prob (:obj:`float`, `optional`, defaults to 0):
|
||||
The dropout ratio for the attention probabilities.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something
|
||||
large (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
(e.g., 512 or 1024 or 2048).
|
||||
type_vocab_size (:obj:`int`, `optional`, defaults to 2):
|
||||
The vocabulary size of the :obj:`token_type_ids` passed when calling :class:`~transformers.AlbertModel` or
|
||||
:class:`~transformers.TFAlbertModel`.
|
||||
|
|
|
|||
|
|
@ -258,8 +258,8 @@ class AutoConfig:
|
|||
r"""
|
||||
Instantiate one of the configuration classes of the library from a pretrained model configuration.
|
||||
|
||||
The configuration class to instantiate is selected based on the :obj:`model_type` property of the config
|
||||
object that is loaded, or when it's missing, by falling back to using pattern matching on
|
||||
The configuration class to instantiate is selected based on the :obj:`model_type` property of the config object
|
||||
that is loaded, or when it's missing, by falling back to using pattern matching on
|
||||
:obj:`pretrained_model_name_or_path`:
|
||||
|
||||
List options
|
||||
|
|
@ -287,9 +287,8 @@ class AutoConfig:
|
|||
Whether or not to delete incompletely received files. Will attempt to resume the download if such a
|
||||
file exists.
|
||||
proxies (:obj:`Dict[str, str]`, `optional`):
|
||||
A dictionary of proxy servers to use by protocol or endpoint, e.g.,
|
||||
:obj:`{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on each
|
||||
request.
|
||||
A dictionary of proxy servers to use by protocol or endpoint, e.g., :obj:`{'http': 'foo.bar:3128',
|
||||
'http://hostname': 'foo.bar:4012'}`. The proxies are used on each request.
|
||||
return_unused_kwargs (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
If :obj:`False`, then this function returns just the final configuration object.
|
||||
|
||||
|
|
@ -298,8 +297,8 @@ class AutoConfig:
|
|||
the part of ``kwargs`` which has not been used to update ``config`` and is otherwise ignored.
|
||||
kwargs(additional keyword arguments, `optional`):
|
||||
The values in kwargs of any keys which are configuration attributes will be used to override the loaded
|
||||
values. Behavior concerning key/value pairs whose keys are *not* configuration attributes is
|
||||
controlled by the ``return_unused_kwargs`` keyword parameter.
|
||||
values. Behavior concerning key/value pairs whose keys are *not* configuration attributes is controlled
|
||||
by the ``return_unused_kwargs`` keyword parameter.
|
||||
|
||||
Examples::
|
||||
|
||||
|
|
|
|||
|
|
@ -36,9 +36,8 @@ class BartConfig(PretrainedConfig):
|
|||
This is the configuration class to store the configuration of a :class:`~transformers.BartModel`. It is used to
|
||||
instantiate a BART model according to the specified arguments, defining the model architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 50265):
|
||||
|
|
@ -59,8 +58,8 @@ class BartConfig(PretrainedConfig):
|
|||
encoder_ffn_dim (:obj:`int`, `optional`, defaults to 4096):
|
||||
Dimensionality of the "intermediate" (often named feed-forward) layer in decoder.
|
||||
activation_function (:obj:`str` or :obj:`function`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
|
|
@ -70,8 +69,8 @@ class BartConfig(PretrainedConfig):
|
|||
classifier_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The dropout ratio for classifier.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 1024):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
init_std (:obj:`float`, `optional`, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
add_bias_logits (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
|
|
@ -95,11 +94,11 @@ class BartConfig(PretrainedConfig):
|
|||
bos_token_id (:obj:`int`, `optional`, defaults to 0)
|
||||
Beginning of stream token id.
|
||||
encoder_layerdrop: (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The LayerDrop probability for the encoder. See the `LayerDrop paper
|
||||
<see https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
The LayerDrop probability for the encoder. See the `LayerDrop paper <see
|
||||
https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
decoder_layerdrop: (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The LayerDrop probability for the decoder. See the `LayerDrop paper
|
||||
<see https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
The LayerDrop probability for the decoder. See the `LayerDrop paper <see
|
||||
https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
extra_pos_embeddings: (:obj:`int`, `optional`, defaults to 2):
|
||||
How many extra learned positional embeddings to use. Should be set to :obj:`pad_token_id+1`.
|
||||
num_labels: (:obj:`int`, `optional`, defaults to 3):
|
||||
|
|
@ -107,8 +106,8 @@ class BartConfig(PretrainedConfig):
|
|||
is_encoder_decoder (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
Whether this is an encoder/decoder model.
|
||||
force_bos_token_to_be_generated (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether or not to force BOS token to be generated at step 1 (after ``decoder_start_token_id``),
|
||||
only :obj:`True` for `bart-large-cnn`.
|
||||
Whether or not to force BOS token to be generated at step 1 (after ``decoder_start_token_id``), only
|
||||
:obj:`True` for `bart-large-cnn`.
|
||||
"""
|
||||
model_type = "bart"
|
||||
|
||||
|
|
|
|||
|
|
@ -51,13 +51,12 @@ BERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
|||
class BertConfig(PretrainedConfig):
|
||||
r"""
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.BertModel` or a
|
||||
:class:`~transformers.TFBertModel`. It is used to instantiate a BERT model according to the specified
|
||||
arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar
|
||||
configuration to that of the BERT `bert-base-uncased <https://huggingface.co/bert-base-uncased>`__ architecture.
|
||||
:class:`~transformers.TFBertModel`. It is used to instantiate a BERT model according to the specified arguments,
|
||||
defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration
|
||||
to that of the BERT `bert-base-uncased <https://huggingface.co/bert-base-uncased>`__ architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
|
||||
Args:
|
||||
|
|
@ -74,15 +73,15 @@ class BertConfig(PretrainedConfig):
|
|||
intermediate_size (:obj:`int`, `optional`, defaults to 3072):
|
||||
Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
|
||||
hidden_act (:obj:`str` or :obj:`Callable`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_probs_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout ratio for the attention probabilities.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
type_vocab_size (:obj:`int`, `optional`, defaults to 2):
|
||||
The vocabulary size of the :obj:`token_type_ids` passed when calling :class:`~transformers.BertModel` or
|
||||
:class:`~transformers.TFBertModel`.
|
||||
|
|
|
|||
|
|
@ -23,9 +23,8 @@ class BertGenerationConfig(PretrainedConfig):
|
|||
:class:`~transformers.BertGenerationPreTrainedModel`. It is used to instantiate a BertGeneration model according to
|
||||
the specified arguments, defining the model architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 50358):
|
||||
|
|
@ -40,15 +39,15 @@ class BertGenerationConfig(PretrainedConfig):
|
|||
intermediate_size (:obj:`int`, `optional`, defaults to 3072):
|
||||
Dimensionality of the "intermediate" (often called feed-forward) layer in the Transformer encoder.
|
||||
hidden_act (:obj:`str` or :obj:`function`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_probs_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout ratio for the attention probabilities.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
initializer_range (:obj:`float`, `optional`, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
layer_norm_eps (:obj:`float`, `optional`, defaults to 1e-12):
|
||||
|
|
|
|||
|
|
@ -14,7 +14,10 @@
|
|||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
# LICENSE file in the root directory of this source tree.
|
||||
"""BlenderbotConfig has the same signature as BartConfig. We only rewrite the signature in order to document blenderbot-90M defaults."""
|
||||
"""
|
||||
BlenderbotConfig has the same signature as BartConfig. We only rewrite the signature in order to document
|
||||
blenderbot-90M defaults.
|
||||
"""
|
||||
from .configuration_bart import BartConfig
|
||||
|
||||
|
||||
|
|
@ -26,12 +29,12 @@ BLENDERBOT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
|||
|
||||
class BlenderbotConfig(BartConfig):
|
||||
r"""
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.BlenderbotForConditionalGeneration`.
|
||||
It inherits from :class:`~transformers.BartConfig` and has the same signature with different defaults.
|
||||
This is the configuration class to store the configuration of a
|
||||
:class:`~transformers.BlenderbotForConditionalGeneration`. It inherits from :class:`~transformers.BartConfig` and
|
||||
has the same signature with different defaults.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 54944):
|
||||
|
|
@ -52,8 +55,8 @@ class BlenderbotConfig(BartConfig):
|
|||
encoder_ffn_dim (:obj:`int`, `optional`, defaults to 2048):
|
||||
Dimensionality of the "intermediate" (often named feed-forward) layer in decoder.
|
||||
activation_function (:obj:`str` or :obj:`function`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
|
|
@ -63,8 +66,8 @@ class BlenderbotConfig(BartConfig):
|
|||
classifier_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The dropout ratio for classifier.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
init_std (:obj:`float`, `optional`, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
add_bias_logits (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
|
|
@ -88,11 +91,11 @@ class BlenderbotConfig(BartConfig):
|
|||
bos_token_id (:obj:`int`, `optional`, defaults to 0)
|
||||
Beginning of stream token id.
|
||||
encoder_layerdrop: (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The LayerDrop probability for the encoder. See the `LayerDrop paper
|
||||
<see https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
The LayerDrop probability for the encoder. See the `LayerDrop paper <see
|
||||
https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
decoder_layerdrop: (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The LayerDrop probability for the decoder. See the `LayerDrop paper
|
||||
<see https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
The LayerDrop probability for the decoder. See the `LayerDrop paper <see
|
||||
https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
extra_pos_embeddings: (:obj:`int`, `optional`, defaults to 2):
|
||||
How many extra learned positional embeddings to use. Should be set to :obj:`pad_token_id+1`.
|
||||
is_encoder_decoder (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
|
|
|
|||
|
|
@ -30,8 +30,8 @@ CAMEMBERT_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
|||
|
||||
class CamembertConfig(RobertaConfig):
|
||||
"""
|
||||
This class overrides :class:`~transformers.RobertaConfig`. Please check the
|
||||
superclass for the appropriate documentation alongside usage examples.
|
||||
This class overrides :class:`~transformers.RobertaConfig`. Please check the superclass for the appropriate
|
||||
documentation alongside usage examples.
|
||||
"""
|
||||
|
||||
model_type = "camembert"
|
||||
|
|
|
|||
|
|
@ -26,13 +26,12 @@ CTRL_PRETRAINED_CONFIG_ARCHIVE_MAP = {"ctrl": "https://s3.amazonaws.com/models.h
|
|||
class CTRLConfig(PretrainedConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.CTRLModel` or a
|
||||
:class:`~transformers.TFCTRLModel`. It is used to instantiate a CTRL model according to the specified
|
||||
arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar
|
||||
configuration to that of the `ctrl <https://huggingface.co/ctrl>`__ architecture from SalesForce.
|
||||
:class:`~transformers.TFCTRLModel`. It is used to instantiate a CTRL model according to the specified arguments,
|
||||
defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration
|
||||
to that of the `ctrl <https://huggingface.co/ctrl>`__ architecture from SalesForce.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 246534):
|
||||
|
|
@ -40,8 +39,8 @@ class CTRLConfig(PretrainedConfig):
|
|||
:obj:`inputs_ids` passed when calling :class:`~transformers.CTRLModel` or
|
||||
:class:`~transformers.TFCTRLModel`.
|
||||
n_positions (:obj:`int`, `optional`, defaults to 256):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
n_ctx (:obj:`int`, `optional`, defaults to 256):
|
||||
Dimensionality of the causal mask (usually same as n_positions).
|
||||
n_embd (:obj:`int`, `optional`, defaults to 1280):
|
||||
|
|
|
|||
|
|
@ -45,16 +45,16 @@ class DebertaConfig(PretrainedConfig):
|
|||
intermediate_size (:obj:`int`, `optional`, defaults to 3072):
|
||||
Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
|
||||
hidden_act (:obj:`str` or :obj:`Callable`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"`, :obj:`"gelu"`, :obj:`"tanh"`, :obj:`"gelu_fast"`,
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"`, :obj:`"gelu"`, :obj:`"tanh"`, :obj:`"gelu_fast"`,
|
||||
:obj:`"mish"`, :obj:`"linear"`, :obj:`"sigmoid"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_probs_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout ratio for the attention probabilities.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
type_vocab_size (:obj:`int`, `optional`, defaults to 2):
|
||||
The vocabulary size of the :obj:`token_type_ids` passed when calling :class:`~transformers.DebertaModel` or
|
||||
:class:`~transformers.TFDebertaModel`.
|
||||
|
|
@ -65,15 +65,15 @@ class DebertaConfig(PretrainedConfig):
|
|||
relative_attention (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether use relative position encoding.
|
||||
max_relative_positions (:obj:`int`, `optional`, defaults to 1):
|
||||
The range of relative positions :obj:`[-max_position_embeddings, max_position_embeddings]`.
|
||||
Use the same value as :obj:`max_position_embeddings`.
|
||||
The range of relative positions :obj:`[-max_position_embeddings, max_position_embeddings]`. Use the same
|
||||
value as :obj:`max_position_embeddings`.
|
||||
pad_token_id (:obj:`int`, `optional`, defaults to 0):
|
||||
The value used to pad input_ids.
|
||||
position_biased_input (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
Whether add absolute position embedding to content embedding.
|
||||
pos_att_type (:obj:`List[str]`, `optional`):
|
||||
The type of relative position attention, it can be a combination of :obj:`["p2c", "c2p", "p2p"]`,
|
||||
e.g. :obj:`["p2c"]`, :obj:`["p2c", "c2p"]`, :obj:`["p2c", "c2p", 'p2p"]`.
|
||||
The type of relative position attention, it can be a combination of :obj:`["p2c", "c2p", "p2p"]`, e.g.
|
||||
:obj:`["p2c"]`, :obj:`["p2c", "c2p"]`, :obj:`["p2c", "c2p", 'p2p"]`.
|
||||
layer_norm_eps (:obj:`float`, optional, defaults to 1e-12):
|
||||
The epsilon used by the layer normalization layers.
|
||||
"""
|
||||
|
|
|
|||
|
|
@ -36,21 +36,20 @@ class DistilBertConfig(PretrainedConfig):
|
|||
This is the configuration class to store the configuration of a :class:`~transformers.DistilBertModel` or a
|
||||
:class:`~transformers.TFDistilBertModel`. It is used to instantiate a DistilBERT model according to the specified
|
||||
arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar
|
||||
configuration to that of the DistilBERT
|
||||
`distilbert-base-uncased <https://huggingface.co/distilbert-base-uncased>`__ architecture.
|
||||
configuration to that of the DistilBERT `distilbert-base-uncased
|
||||
<https://huggingface.co/distilbert-base-uncased>`__ architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 30522):
|
||||
Vocabulary size of the DistilBERT model. Defines the number of different tokens that can be represented by the
|
||||
:obj:`inputs_ids` passed when calling :class:`~transformers.DistilBertModel` or
|
||||
Vocabulary size of the DistilBERT model. Defines the number of different tokens that can be represented by
|
||||
the :obj:`inputs_ids` passed when calling :class:`~transformers.DistilBertModel` or
|
||||
:class:`~transformers.TFDistilBertModel`.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
sinusoidal_pos_embds (:obj:`boolean`, `optional`, defaults to :obj:`False`):
|
||||
Whether to use sinusoidal positional embeddings.
|
||||
n_layers (:obj:`int`, `optional`, defaults to 6):
|
||||
|
|
@ -66,8 +65,8 @@ class DistilBertConfig(PretrainedConfig):
|
|||
attention_dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout ratio for the attention probabilities.
|
||||
activation (:obj:`str` or :obj:`Callable`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
initializer_range (:obj:`float`, `optional`, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
qa_dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
|
|
|
|||
|
|
@ -32,20 +32,19 @@ DPR_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
|||
|
||||
class DPRConfig(PretrainedConfig):
|
||||
r"""
|
||||
:class:`~transformers.DPRConfig` is the configuration class to store the configuration of a
|
||||
`DPRModel`.
|
||||
:class:`~transformers.DPRConfig` is the configuration class to store the configuration of a `DPRModel`.
|
||||
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.DPRContextEncoder`,
|
||||
:class:`~transformers.DPRQuestionEncoder`, or a :class:`~transformers.DPRReader`. It is used to instantiate the
|
||||
components of the DPR model.
|
||||
|
||||
This class is a subclass of :class:`~transformers.BertConfig`. Please check the
|
||||
superclass for the documentation of all kwargs.
|
||||
This class is a subclass of :class:`~transformers.BertConfig`. Please check the superclass for the documentation of
|
||||
all kwargs.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 30522):
|
||||
Vocabulary size of the DPR model. Defines the different tokens that
|
||||
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.BertModel`.
|
||||
Vocabulary size of the DPR model. Defines the different tokens that can be represented by the `inputs_ids`
|
||||
passed to the forward method of :class:`~transformers.BertModel`.
|
||||
hidden_size (:obj:`int`, `optional`, defaults to 768):
|
||||
Dimensionality of the encoder layers and the pooler layer.
|
||||
num_hidden_layers (:obj:`int`, `optional`, defaults to 12):
|
||||
|
|
@ -55,15 +54,15 @@ class DPRConfig(PretrainedConfig):
|
|||
intermediate_size (:obj:`int`, `optional`, defaults to 3072):
|
||||
Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
|
||||
hidden_act (:obj:`str` or :obj:`function`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_probs_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout ratio for the attention probabilities.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
type_vocab_size (:obj:`int`, `optional`, defaults to 2):
|
||||
The vocabulary size of the `token_type_ids` passed into :class:`~transformers.BertModel`.
|
||||
initializer_range (:obj:`float`, `optional`, defaults to 0.02):
|
||||
|
|
@ -73,8 +72,8 @@ class DPRConfig(PretrainedConfig):
|
|||
gradient_checkpointing (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
If True, use gradient checkpointing to save memory at the expense of slower backward pass.
|
||||
projection_dim (:obj:`int`, `optional`, defaults to 0):
|
||||
Dimension of the projection for the context and question encoders.
|
||||
If it is set to zero (default), then no projection is done.
|
||||
Dimension of the projection for the context and question encoders. If it is set to zero (default), then no
|
||||
projection is done.
|
||||
"""
|
||||
model_type = "dpr"
|
||||
|
||||
|
|
|
|||
|
|
@ -36,12 +36,11 @@ class ElectraConfig(PretrainedConfig):
|
|||
This is the configuration class to store the configuration of a :class:`~transformers.ElectraModel` or a
|
||||
:class:`~transformers.TFElectraModel`. It is used to instantiate a ELECTRA model according to the specified
|
||||
arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar
|
||||
configuration to that of the ELECTRA
|
||||
`google/electra-small-discriminator <https://huggingface.co/google/electra-small-discriminator>`__ architecture.
|
||||
configuration to that of the ELECTRA `google/electra-small-discriminator
|
||||
<https://huggingface.co/google/electra-small-discriminator>`__ architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
|
||||
Args:
|
||||
|
|
@ -60,15 +59,15 @@ class ElectraConfig(PretrainedConfig):
|
|||
intermediate_size (:obj:`int`, `optional`, defaults to 1024):
|
||||
Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
|
||||
hidden_act (:obj:`str` or :obj:`Callable`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_probs_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout ratio for the attention probabilities.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
type_vocab_size (:obj:`int`, `optional`, defaults to 2):
|
||||
The vocabulary size of the :obj:`token_type_ids` passed when calling :class:`~transformers.ElectraModel` or
|
||||
:class:`~transformers.TFElectraModel`.
|
||||
|
|
|
|||
|
|
@ -29,9 +29,8 @@ class EncoderDecoderConfig(PretrainedConfig):
|
|||
:class:`~transformers.EncoderDecoderModel`. It is used to instantiate an Encoder Decoder model according to the
|
||||
specified arguments, defining the encoder and decoder configs.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
kwargs (`optional`):
|
||||
|
|
@ -93,7 +92,8 @@ class EncoderDecoderConfig(PretrainedConfig):
|
|||
cls, encoder_config: PretrainedConfig, decoder_config: PretrainedConfig, **kwargs
|
||||
) -> PretrainedConfig:
|
||||
r"""
|
||||
Instantiate a :class:`~transformers.EncoderDecoderConfig` (or a derived class) from a pre-trained encoder model configuration and decoder model configuration.
|
||||
Instantiate a :class:`~transformers.EncoderDecoderConfig` (or a derived class) from a pre-trained encoder model
|
||||
configuration and decoder model configuration.
|
||||
|
||||
Returns:
|
||||
:class:`EncoderDecoderConfig`: An instance of a configuration object
|
||||
|
|
|
|||
|
|
@ -34,20 +34,19 @@ class FlaubertConfig(XLMConfig):
|
|||
:class:`~transformers.TFFlaubertModel`. It is used to instantiate a FlauBERT model according to the specified
|
||||
arguments, defining the model architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
pre_norm (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to apply the layer normalization before or after the feed forward layer following the
|
||||
attention in each layer (Vaswani et al., Tensor2Tensor for Neural Machine Translation. 2018)
|
||||
Whether to apply the layer normalization before or after the feed forward layer following the attention in
|
||||
each layer (Vaswani et al., Tensor2Tensor for Neural Machine Translation. 2018)
|
||||
layerdrop (:obj:`float`, `optional`, defaults to 0.0):
|
||||
Probability to drop layers during training (Fan et al., Reducing Transformer Depth on Demand
|
||||
with Structured Dropout. ICLR 2020)
|
||||
Probability to drop layers during training (Fan et al., Reducing Transformer Depth on Demand with
|
||||
Structured Dropout. ICLR 2020)
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 30145):
|
||||
Vocabulary size of the FlauBERT model. Defines the number of different tokens that can be represented by the
|
||||
:obj:`inputs_ids` passed when calling :class:`~transformers.FlaubertModel` or
|
||||
Vocabulary size of the FlauBERT model. Defines the number of different tokens that can be represented by
|
||||
the :obj:`inputs_ids` passed when calling :class:`~transformers.FlaubertModel` or
|
||||
:class:`~transformers.TFFlaubertModel`.
|
||||
emb_dim (:obj:`int`, `optional`, defaults to 2048):
|
||||
Dimensionality of the encoder layers and the pooler layer.
|
||||
|
|
@ -56,8 +55,7 @@ class FlaubertConfig(XLMConfig):
|
|||
n_head (:obj:`int`, `optional`, defaults to 16):
|
||||
Number of attention heads for each attention layer in the Transformer encoder.
|
||||
dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probability for all fully connected
|
||||
layers in the embeddings, encoder, and pooler.
|
||||
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probability for the attention mechanism
|
||||
gelu_activation (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
|
|
@ -65,28 +63,25 @@ class FlaubertConfig(XLMConfig):
|
|||
sinusoidal_embeddings (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether or not to use sinusoidal positional embeddings instead of absolute positional embeddings.
|
||||
causal (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether or not the model shoul behave in a causal manner.
|
||||
Causal models use a triangular attention mask in order to only attend to the left-side context instead
|
||||
if a bidirectional context.
|
||||
Whether or not the model shoul behave in a causal manner. Causal models use a triangular attention mask in
|
||||
order to only attend to the left-side context instead if a bidirectional context.
|
||||
asm (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether or not to use an adaptive log softmax projection layer instead of a linear layer for the prediction
|
||||
layer.
|
||||
n_langs (:obj:`int`, `optional`, defaults to 1):
|
||||
The number of languages the model handles. Set to 1 for monolingual models.
|
||||
use_lang_emb (:obj:`bool`, `optional`, defaults to :obj:`True`)
|
||||
Whether to use language embeddings. Some models use additional language embeddings, see
|
||||
`the multilingual models page <http://huggingface.co/transformers/multilingual.html#xlm-language-embeddings>`__
|
||||
for information on how to use them.
|
||||
Whether to use language embeddings. Some models use additional language embeddings, see `the multilingual
|
||||
models page <http://huggingface.co/transformers/multilingual.html#xlm-language-embeddings>`__ for
|
||||
information on how to use them.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might
|
||||
ever be used with. Typically set this to something large just in case
|
||||
(e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
embed_init_std (:obj:`float`, `optional`, defaults to 2048^-0.5):
|
||||
The standard deviation of the truncated_normal_initializer for
|
||||
initializing the embedding matrices.
|
||||
The standard deviation of the truncated_normal_initializer for initializing the embedding matrices.
|
||||
init_std (:obj:`int`, `optional`, defaults to 50257):
|
||||
The standard deviation of the truncated_normal_initializer for
|
||||
initializing all weight matrices except the embedding matrices.
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices except the
|
||||
embedding matrices.
|
||||
layer_norm_eps (:obj:`float`, `optional`, defaults to 1e-12):
|
||||
The epsilon used by the layer normalization layers.
|
||||
bos_index (:obj:`int`, `optional`, defaults to 0):
|
||||
|
|
@ -134,8 +129,7 @@ class FlaubertConfig(XLMConfig):
|
|||
mask_token_id (:obj:`int`, `optional`, defaults to 0):
|
||||
Model agnostic parameter to identify masked tokens when generating text in an MLM context.
|
||||
lang_id (:obj:`int`, `optional`, defaults to 1):
|
||||
The ID of the language used by the model. This parameter is used when generating
|
||||
text in a given language.
|
||||
The ID of the language used by the model. This parameter is used when generating text in a given language.
|
||||
"""
|
||||
|
||||
model_type = "flaubert"
|
||||
|
|
|
|||
|
|
@ -28,8 +28,7 @@ FSMT_PRETRAINED_CONFIG_ARCHIVE_MAP = {}
|
|||
|
||||
class DecoderConfig(PretrainedConfig):
|
||||
r"""
|
||||
Configuration class for FSMT's decoder specific things.
|
||||
note: this is a private helper class
|
||||
Configuration class for FSMT's decoder specific things. note: this is a private helper class
|
||||
"""
|
||||
model_type = "fsmt_decoder"
|
||||
|
||||
|
|
@ -44,9 +43,8 @@ class FSMTConfig(PretrainedConfig):
|
|||
This is the configuration class to store the configuration of a :class:`~transformers.FSMTModel`. It is used to
|
||||
instantiate a FSMT model according to the specified arguments, defining the model architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
langs (:obj:`List[str]`):
|
||||
|
|
@ -72,8 +70,8 @@ class FSMTConfig(PretrainedConfig):
|
|||
encoder_ffn_dim (:obj:`int`, `optional`, defaults to 4096):
|
||||
Dimensionality of the "intermediate" (often named feed-forward) layer in decoder.
|
||||
activation_function (:obj:`str` or :obj:`Callable`, `optional`, defaults to :obj:`"relu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
|
|
@ -81,8 +79,8 @@ class FSMTConfig(PretrainedConfig):
|
|||
activation_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The dropout ratio for activations inside the fully connected layer.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 1024):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
init_std (:obj:`float`, `optional`, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
scale_embedding (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
|
|
@ -104,14 +102,13 @@ class FSMTConfig(PretrainedConfig):
|
|||
tie_word_embeddings (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether to tie input and output embeddings.
|
||||
num_beams (:obj:`int`, `optional`, defaults to 5)
|
||||
Number of beams for beam search that will be used by default in the :obj:`generate` method
|
||||
of the model. 1 means no beam search.
|
||||
Number of beams for beam search that will be used by default in the :obj:`generate` method of the model. 1
|
||||
means no beam search.
|
||||
length_penalty (:obj:`float`, `optional`, defaults to 1)
|
||||
Exponential penalty to the length that will be used by default in the :obj:`generate` method
|
||||
of the model.
|
||||
Exponential penalty to the length that will be used by default in the :obj:`generate` method of the model.
|
||||
early_stopping (:obj:`bool`, `optional`, defaults to :obj:`False`)
|
||||
Flag that will be used by default in the :obj:`generate` method of the model. Whether to stop
|
||||
the beam search when at least ``num_beams`` sentences are finished per batch or not.
|
||||
Flag that will be used by default in the :obj:`generate` method of the model. Whether to stop the beam
|
||||
search when at least ``num_beams`` sentences are finished per batch or not.
|
||||
|
||||
Examples::
|
||||
|
||||
|
|
|
|||
|
|
@ -42,9 +42,8 @@ class FunnelConfig(PretrainedConfig):
|
|||
configuration to that of the Funnel Transformer `funnel-transformer/small
|
||||
<https://huggingface.co/funnel-transformer/small>`__ architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 30522):
|
||||
|
|
@ -66,8 +65,8 @@ class FunnelConfig(PretrainedConfig):
|
|||
d_inner (:obj:`int`, `optional`, defaults to 3072):
|
||||
Inner dimension in the feed-forward blocks.
|
||||
hidden_act (:obj:`str` or :obj:`callable`, `optional`, defaults to :obj:`"gelu_new"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
|
|
@ -75,8 +74,8 @@ class FunnelConfig(PretrainedConfig):
|
|||
activation_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The dropout probability used between the two layers of the feed-forward blocks.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
type_vocab_size (:obj:`int`, `optional`, defaults to 3):
|
||||
The vocabulary size of the :obj:`token_type_ids` passed when calling :class:`~transformers.FunnelModel` or
|
||||
:class:`~transformers.TFFunnelModel`.
|
||||
|
|
@ -90,19 +89,17 @@ class FunnelConfig(PretrainedConfig):
|
|||
layer_norm_eps (:obj:`float`, `optional`, defaults to 1e-9):
|
||||
The epsilon used by the layer normalization layers.
|
||||
pooling_type (:obj:`str`, `optional`, defaults to :obj:`"mean"`):
|
||||
Possible values are ``"mean"`` or ``"max"``. The way pooling is performed at the beginning of each
|
||||
block.
|
||||
Possible values are ``"mean"`` or ``"max"``. The way pooling is performed at the beginning of each block.
|
||||
attention_type (:obj:`str`, `optional`, defaults to :obj:`"relative_shift"`):
|
||||
Possible values are ``"relative_shift"`` or ``"factorized"``. The former is faster on CPU/GPU while
|
||||
the latter is faster on TPU.
|
||||
Possible values are ``"relative_shift"`` or ``"factorized"``. The former is faster on CPU/GPU while the
|
||||
latter is faster on TPU.
|
||||
separate_cls (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
Whether or not to separate the cls token when applying pooling.
|
||||
truncate_seq (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
When using ``separate_cls``, whether or not to truncate the last token when pooling, to avoid getting
|
||||
a sequence length that is not a multiple of 2.
|
||||
When using ``separate_cls``, whether or not to truncate the last token when pooling, to avoid getting a
|
||||
sequence length that is not a multiple of 2.
|
||||
pool_q_only (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether or not to apply the pooling only to the query or to query, key and values for the attention
|
||||
layers.
|
||||
Whether or not to apply the pooling only to the query or to query, key and values for the attention layers.
|
||||
"""
|
||||
model_type = "funnel"
|
||||
|
||||
|
|
|
|||
|
|
@ -33,13 +33,12 @@ GPT2_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
|||
class GPT2Config(PretrainedConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.GPT2Model` or a
|
||||
:class:`~transformers.TFGPT2Model`. It is used to instantiate a GPT-2 model according to the specified
|
||||
arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar
|
||||
configuration to that of the GPT-2 `small <https://huggingface.co/gpt2>`__ architecture.
|
||||
:class:`~transformers.TFGPT2Model`. It is used to instantiate a GPT-2 model according to the specified arguments,
|
||||
defining the model architecture. Instantiating a configuration with the defaults will yield a similar configuration
|
||||
to that of the GPT-2 `small <https://huggingface.co/gpt2>`__ architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
|
||||
Args:
|
||||
|
|
@ -48,8 +47,8 @@ class GPT2Config(PretrainedConfig):
|
|||
:obj:`inputs_ids` passed when calling :class:`~transformers.GPT2Model` or
|
||||
:class:`~transformers.TFGPT2Model`.
|
||||
n_positions (:obj:`int`, `optional`, defaults to 1024):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
n_ctx (:obj:`int`, `optional`, defaults to 1024):
|
||||
Dimensionality of the causal mask (usually same as n_positions).
|
||||
n_embd (:obj:`int`, `optional`, defaults to 768):
|
||||
|
|
@ -73,8 +72,8 @@ class GPT2Config(PretrainedConfig):
|
|||
initializer_range (:obj:`float`, `optional`, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
summary_type (:obj:`string`, `optional`, defaults to :obj:`"cls_index"`):
|
||||
Argument used when doing sequence summary, used in the models
|
||||
:class:`~transformers.GPT2DoubleHeadsModel` and :class:`~transformers.TFGPT2DoubleHeadsModel`.
|
||||
Argument used when doing sequence summary, used in the models :class:`~transformers.GPT2DoubleHeadsModel`
|
||||
and :class:`~transformers.TFGPT2DoubleHeadsModel`.
|
||||
|
||||
Has to be one of the following options:
|
||||
|
||||
|
|
@ -84,8 +83,8 @@ class GPT2Config(PretrainedConfig):
|
|||
- :obj:`"cls_index"`: Supply a Tensor of classification token position (like GPT/GPT-2).
|
||||
- :obj:`"attn"`: Not implemented now, use multi-head attention.
|
||||
summary_use_proj (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
Argument used when doing sequence summary, used in the models
|
||||
:class:`~transformers.GPT2DoubleHeadsModel` and :class:`~transformers.TFGPT2DoubleHeadsModel`.
|
||||
Argument used when doing sequence summary, used in the models :class:`~transformers.GPT2DoubleHeadsModel`
|
||||
and :class:`~transformers.TFGPT2DoubleHeadsModel`.
|
||||
|
||||
Whether or not to add a projection after the vector extraction.
|
||||
summary_activation (:obj:`str`, `optional`):
|
||||
|
|
@ -94,13 +93,13 @@ class GPT2Config(PretrainedConfig):
|
|||
|
||||
Pass :obj:`"tanh"` for a tanh activation to the output, any other value will result in no activation.
|
||||
summary_proj_to_labels (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
Argument used when doing sequence summary, used in the models
|
||||
:class:`~transformers.GPT2DoubleHeadsModel` and :class:`~transformers.TFGPT2DoubleHeadsModel`.
|
||||
Argument used when doing sequence summary, used in the models :class:`~transformers.GPT2DoubleHeadsModel`
|
||||
and :class:`~transformers.TFGPT2DoubleHeadsModel`.
|
||||
|
||||
Whether the projection outputs should have :obj:`config.num_labels` or :obj:`config.hidden_size` classes.
|
||||
summary_first_dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
Argument used when doing sequence summary, used in the models
|
||||
:class:`~transformers.GPT2DoubleHeadsModel` and :class:`~transformers.TFGPT2DoubleHeadsModel`.
|
||||
Argument used when doing sequence summary, used in the models :class:`~transformers.GPT2DoubleHeadsModel`
|
||||
and :class:`~transformers.TFGPT2DoubleHeadsModel`.
|
||||
|
||||
The dropout ratio to be used after the projection and activation.
|
||||
gradient_checkpointing (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
|
|
|
|||
|
|
@ -29,20 +29,19 @@ LAYOUTLM_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
|||
|
||||
class LayoutLMConfig(BertConfig):
|
||||
r"""
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.LayoutLMModel`.
|
||||
It is used to instantiate a LayoutLM model according to the specified arguments, defining the model
|
||||
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
|
||||
the LayoutLM `layoutlm-base-uncased <https://huggingface.co/microsoft/layoutlm-base-uncased>`__ architecture.
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.LayoutLMModel`. It is used to
|
||||
instantiate a LayoutLM model according to the specified arguments, defining the model architecture. Instantiating a
|
||||
configuration with the defaults will yield a similar configuration to that of the LayoutLM `layoutlm-base-uncased
|
||||
<https://huggingface.co/microsoft/layoutlm-base-uncased>`__ architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.BertConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.BertConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.BertConfig` and can be used to control the model outputs.
|
||||
Read the documentation from :class:`~transformers.BertConfig` for more information.
|
||||
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 30522):
|
||||
Vocabulary size of the LayoutLM model. Defines the different tokens that
|
||||
can be represented by the `inputs_ids` passed to the forward method of :class:`~transformers.LayoutLMModel`.
|
||||
Vocabulary size of the LayoutLM model. Defines the different tokens that can be represented by the
|
||||
`inputs_ids` passed to the forward method of :class:`~transformers.LayoutLMModel`.
|
||||
hidden_size (:obj:`int`, `optional`, defaults to 768):
|
||||
Dimensionality of the encoder layers and the pooler layer.
|
||||
num_hidden_layers (:obj:`int`, `optional`, defaults to 12):
|
||||
|
|
@ -52,15 +51,15 @@ class LayoutLMConfig(BertConfig):
|
|||
intermediate_size (:obj:`int`, `optional`, defaults to 3072):
|
||||
Dimensionality of the "intermediate" (i.e., feed-forward) layer in the Transformer encoder.
|
||||
hidden_act (:obj:`str` or :obj:`function`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_probs_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout ratio for the attention probabilities.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
type_vocab_size (:obj:`int`, `optional`, defaults to 2):
|
||||
The vocabulary size of the :obj:`token_type_ids` passed into :class:`~transformers.LayoutLMModel`.
|
||||
initializer_range (:obj:`float`, `optional`, defaults to 0.02):
|
||||
|
|
@ -70,8 +69,8 @@ class LayoutLMConfig(BertConfig):
|
|||
gradient_checkpointing (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
If True, use gradient checkpointing to save memory at the expense of slower backward pass.
|
||||
max_2d_position_embeddings (:obj:`int`, `optional`, defaults to 1024):
|
||||
The maximum value that the 2D position embedding might ever used.
|
||||
Typically set this to something large just in case (e.g., 1024).
|
||||
The maximum value that the 2D position embedding might ever used. Typically set this to something large
|
||||
just in case (e.g., 1024).
|
||||
|
||||
Examples::
|
||||
|
||||
|
|
|
|||
|
|
@ -37,19 +37,19 @@ class LongformerConfig(RobertaConfig):
|
|||
:class:`~transformers.TFLongformerModel`. It is used to instantiate a Longformer model according to the specified
|
||||
arguments, defining the model architecture.
|
||||
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.LongformerModel`.
|
||||
It is used to instantiate an Longformer model according to the specified arguments, defining the model
|
||||
architecture. Instantiating a configuration with the defaults will yield a similar configuration to that of
|
||||
the RoBERTa `roberta-base <https://huggingface.co/roberta-base>`__ architecture with a sequence length 4,096.
|
||||
This is the configuration class to store the configuration of a :class:`~transformers.LongformerModel`. It is used
|
||||
to instantiate an Longformer model according to the specified arguments, defining the model architecture.
|
||||
Instantiating a configuration with the defaults will yield a similar configuration to that of the RoBERTa
|
||||
`roberta-base <https://huggingface.co/roberta-base>`__ architecture with a sequence length 4,096.
|
||||
|
||||
The :class:`~transformers.LongformerConfig` class directly inherits :class:`~transformers.RobertaConfig`.
|
||||
It reuses the same defaults. Please check the parent class for more information.
|
||||
The :class:`~transformers.LongformerConfig` class directly inherits :class:`~transformers.RobertaConfig`. It reuses
|
||||
the same defaults. Please check the parent class for more information.
|
||||
|
||||
Args:
|
||||
attention_window (:obj:`int` or :obj:`List[int]`, `optional`, defaults to 512):
|
||||
Size of an attention window around each token. If an :obj:`int`, use the same size for all layers.
|
||||
To specify a different window size for each layer, use a :obj:`List[int]` where
|
||||
``len(attention_window) == num_hidden_layers``.
|
||||
Size of an attention window around each token. If an :obj:`int`, use the same size for all layers. To
|
||||
specify a different window size for each layer, use a :obj:`List[int]` where ``len(attention_window) ==
|
||||
num_hidden_layers``.
|
||||
|
||||
Example::
|
||||
|
||||
|
|
|
|||
|
|
@ -32,9 +32,8 @@ class LxmertConfig(PretrainedConfig):
|
|||
:class:`~transformers.TFLxmertModel`. It is used to instantiate a LXMERT model according to the specified
|
||||
arguments, defining the model architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
|
||||
Args:
|
||||
|
|
@ -55,15 +54,15 @@ class LxmertConfig(PretrainedConfig):
|
|||
intermediate_size (:obj:`int`, `optional`, defaults to 3072):
|
||||
Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
|
||||
hidden_act (:obj:`str` or :obj:`Callable`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_probs_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout ratio for the attention probabilities.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
type_vocab_size (:obj:`int`, `optional`, defaults to 2):
|
||||
The vocabulary size of the `token_type_ids` passed into :class:`~transformers.BertModel`.
|
||||
initializer_range (:obj:`float`, `optional`, defaults to 0.02):
|
||||
|
|
@ -71,15 +70,14 @@ class LxmertConfig(PretrainedConfig):
|
|||
layer_norm_eps (:obj:`float`, `optional`, defaults to 1e-12):
|
||||
The epsilon used by the layer normalization layers.
|
||||
visual_feat_dim (:obj:`int`, `optional`, defaults to 2048):
|
||||
This represents the last dimension of the pooled-object features used as input for the model,
|
||||
representing the size of each object feature itself.
|
||||
This represents the last dimension of the pooled-object features used as input for the model, representing
|
||||
the size of each object feature itself.
|
||||
visual_pos_dim (:obj:`int`, `optional`, defaults to 4):
|
||||
This represents the number of spacial features that are mixed into the visual features.
|
||||
The default is set to 4 because most commonly this will represent the location of a bounding box.
|
||||
i.e., (x, y, width, height)
|
||||
This represents the number of spacial features that are mixed into the visual features. The default is set
|
||||
to 4 because most commonly this will represent the location of a bounding box. i.e., (x, y, width, height)
|
||||
visual_loss_normalizer (:obj:`float`, `optional`, defaults to 1/15):
|
||||
This represents the scaling factor in which each visual loss is multiplied by if during pretraining,
|
||||
one decided to train with multiple vision-based loss objectives.
|
||||
This represents the scaling factor in which each visual loss is multiplied by if during pretraining, one
|
||||
decided to train with multiple vision-based loss objectives.
|
||||
num_qa_labels (:obj:`int`, `optional`, defaults to 9500):
|
||||
This represents the total number of different question answering (QA) labels there are. If using more than
|
||||
one dataset with QA, the user will need to account for the total number of labels that all of the datasets
|
||||
|
|
@ -91,8 +89,8 @@ class LxmertConfig(PretrainedConfig):
|
|||
This represents the total number of semantically unique attributes that lxmert will be able to classify a
|
||||
pooled-object feature as possessing.
|
||||
task_matched (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
This task is used for sentence-image matching. If the sentence correctly describes the image the label
|
||||
will be 1. If the sentence does not correctly describe the image, the label will be 0.
|
||||
This task is used for sentence-image matching. If the sentence correctly describes the image the label will
|
||||
be 1. If the sentence does not correctly describe the image, the label will be 0.
|
||||
task_mask_lm (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
Whether or not to add masked language modeling (as used in pretraining models such as BERT) to the loss
|
||||
objective.
|
||||
|
|
@ -108,8 +106,8 @@ class LxmertConfig(PretrainedConfig):
|
|||
visual_feat_loss (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
Whether or not to calculate the feature-regression loss objective
|
||||
output_attentions (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether or not the model should return the attentions from the vision, langauge, and cross-modality
|
||||
layers should be returned.
|
||||
Whether or not the model should return the attentions from the vision, langauge, and cross-modality layers
|
||||
should be returned.
|
||||
output_hidden_states (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
Whether or not the model should return the hidden states from the vision, langauge, and cross-modality
|
||||
layers should be returned.
|
||||
|
|
|
|||
|
|
@ -27,9 +27,8 @@ class MarianConfig(BartConfig):
|
|||
This is the configuration class to store the configuration of a :class:`~transformers.MarianMTModel`. It is used to
|
||||
instantiate a Marian model according to the specified arguments, defining the model architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 58101):
|
||||
|
|
@ -50,8 +49,8 @@ class MarianConfig(BartConfig):
|
|||
encoder_ffn_dim (:obj:`int`, `optional`, defaults to 2048):
|
||||
Dimensionality of the "intermediate" (i.e., feed-forward) layer in decoder.
|
||||
activation_function (:obj:`str` or :obj:`function`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
|
|
@ -61,8 +60,8 @@ class MarianConfig(BartConfig):
|
|||
classifier_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The dropout ratio for classifier.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
init_std (:obj:`float`, `optional`, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
add_bias_logits (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
|
|
@ -84,11 +83,11 @@ class MarianConfig(BartConfig):
|
|||
bos_token_id (:obj:`int`, `optional`, defaults to 0)
|
||||
Beginning of stream token id.
|
||||
encoder_layerdrop: (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The LayerDrop probability for the encoder. See the `LayerDrop paper
|
||||
<see https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
The LayerDrop probability for the encoder. See the `LayerDrop paper <see
|
||||
https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
decoder_layerdrop: (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The LayerDrop probability for the decoder. See the `LayerDrop paper
|
||||
<see https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
The LayerDrop probability for the decoder. See the `LayerDrop paper <see
|
||||
https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
extra_pos_embeddings: (:obj:`int`, `optional`, defaults to 2):
|
||||
How many extra learned positional embeddings to use.
|
||||
is_encoder_decoder (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
|
|
|
|||
|
|
@ -29,12 +29,11 @@ MBART_PRETRAINED_CONFIG_ARCHIVE_MAP = {
|
|||
class MBartConfig(BartConfig):
|
||||
"""
|
||||
This is the configuration class to store the configuration of a
|
||||
:class:`~transformers.MBartForConditionalGeneration`. It is used to
|
||||
instantiate a BART model according to the specified arguments, defining the model architecture.
|
||||
:class:`~transformers.MBartForConditionalGeneration`. It is used to instantiate a BART model according to the
|
||||
specified arguments, defining the model architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 250027):
|
||||
|
|
@ -55,8 +54,8 @@ class MBartConfig(BartConfig):
|
|||
encoder_ffn_dim (:obj:`int`, `optional`, defaults to 4096):
|
||||
Dimensionality of the "intermediate" (i.e., feed-forward) layer in decoder.
|
||||
activation_function (:obj:`str` or :obj:`function`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
dropout (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probabilitiy for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
|
|
@ -66,8 +65,8 @@ class MBartConfig(BartConfig):
|
|||
classifier_dropout (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The dropout ratio for classifier.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 1024):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
init_std (:obj:`float`, `optional`, defaults to 0.02):
|
||||
The standard deviation of the truncated_normal_initializer for initializing all weight matrices.
|
||||
add_bias_logits (:obj:`bool`, `optional`, defaults to :obj:`False`):
|
||||
|
|
@ -89,11 +88,11 @@ class MBartConfig(BartConfig):
|
|||
bos_token_id (:obj:`int`, `optional`, defaults to 0)
|
||||
Beginning of stream token id.
|
||||
encoder_layerdrop: (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The LayerDrop probability for the encoder. See the `LayerDrop paper
|
||||
<see https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
The LayerDrop probability for the encoder. See the `LayerDrop paper <see
|
||||
https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
decoder_layerdrop: (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The LayerDrop probability for the decoder. See the `LayerDrop paper
|
||||
<see https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
The LayerDrop probability for the decoder. See the `LayerDrop paper <see
|
||||
https://arxiv.org/abs/1909.11556>`__ for more details.
|
||||
extra_pos_embeddings: (:obj:`int`, `optional`, defaults to 2):
|
||||
How many extra learned positional embeddings to use. Should be equal to :obj:`pad_token_id+1`.
|
||||
is_encoder_decoder (:obj:`bool`, `optional`, defaults to :obj:`True`):
|
||||
|
|
|
|||
|
|
@ -29,9 +29,8 @@ class MobileBertConfig(PretrainedConfig):
|
|||
:class:`~transformers.TFMobileBertModel`. It is used to instantiate a MobileBERT model according to the specified
|
||||
arguments, defining the model architecture.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
|
||||
Args:
|
||||
|
|
@ -48,15 +47,15 @@ class MobileBertConfig(PretrainedConfig):
|
|||
intermediate_size (:obj:`int`, `optional`, defaults to 512):
|
||||
Dimensionality of the "intermediate" (often named feed-forward) layer in the Transformer encoder.
|
||||
hidden_act (:obj:`str` or :obj:`function`, `optional`, defaults to :obj:`"relu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
hidden_dropout_prob (:obj:`float`, `optional`, defaults to 0.0):
|
||||
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
attention_probs_dropout_prob (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout ratio for the attention probabilities.
|
||||
max_position_embeddings (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
type_vocab_size (:obj:`int`, `optional`, defaults to 2):
|
||||
The vocabulary size of the :obj:`token_type_ids` passed when calling :class:`~transformers.MobileBertModel`
|
||||
or :class:`~transformers.TFMobileBertModel`.
|
||||
|
|
@ -84,7 +83,7 @@ class MobileBertConfig(PretrainedConfig):
|
|||
normalization_type (:obj:`str`, `optional`, defaults to :obj:`"no_norm"`):
|
||||
The normalization type in MobileBERT.
|
||||
|
||||
Examples:
|
||||
Examples::
|
||||
|
||||
>>> from transformers import MobileBertModel, MobileBertConfig
|
||||
|
||||
|
|
@ -97,9 +96,8 @@ class MobileBertConfig(PretrainedConfig):
|
|||
>>> # Accessing the model configuration
|
||||
>>> configuration = model.config
|
||||
|
||||
Attributes:
|
||||
pretrained_config_archive_map (Dict[str, str]):
|
||||
A dictionary containing all the available pre-trained checkpoints.
|
||||
Attributes: pretrained_config_archive_map (Dict[str, str]): A dictionary containing all the available pre-trained
|
||||
checkpoints.
|
||||
"""
|
||||
pretrained_config_archive_map = MOBILEBERT_PRETRAINED_CONFIG_ARCHIVE_MAP
|
||||
model_type = "mobilebert"
|
||||
|
|
|
|||
|
|
@ -33,9 +33,8 @@ class OpenAIGPTConfig(PretrainedConfig):
|
|||
arguments, defining the model architecture. Instantiating a configuration with the defaults will yield a similar
|
||||
configuration to that of the `GPT <https://huggingface.co/openai-gpt>`__ architecture from OpenAI.
|
||||
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used
|
||||
to control the model outputs. Read the documentation from :class:`~transformers.PretrainedConfig`
|
||||
for more information.
|
||||
Configuration objects inherit from :class:`~transformers.PretrainedConfig` and can be used to control the model
|
||||
outputs. Read the documentation from :class:`~transformers.PretrainedConfig` for more information.
|
||||
|
||||
Args:
|
||||
vocab_size (:obj:`int`, `optional`, defaults to 40478):
|
||||
|
|
@ -43,8 +42,8 @@ class OpenAIGPTConfig(PretrainedConfig):
|
|||
:obj:`inputs_ids` passed when calling :class:`~transformers.OpenAIGPTModel` or
|
||||
:class:`~transformers.TFOpenAIGPTModel`.
|
||||
n_positions (:obj:`int`, `optional`, defaults to 512):
|
||||
The maximum sequence length that this model might ever be used with.
|
||||
Typically set this to something large just in case (e.g., 512 or 1024 or 2048).
|
||||
The maximum sequence length that this model might ever be used with. Typically set this to something large
|
||||
just in case (e.g., 512 or 1024 or 2048).
|
||||
n_ctx (:obj:`int`, `optional`, defaults to 512):
|
||||
Dimensionality of the causal mask (usually same as n_positions).
|
||||
n_embd (:obj:`int`, `optional`, defaults to 768):
|
||||
|
|
@ -54,8 +53,8 @@ class OpenAIGPTConfig(PretrainedConfig):
|
|||
n_head (:obj:`int`, `optional`, defaults to 12):
|
||||
Number of attention heads for each attention layer in the Transformer encoder.
|
||||
afn (:obj:`str` or :obj:`Callable`, `optional`, defaults to :obj:`"gelu"`):
|
||||
The non-linear activation function (function or string) in the encoder and pooler.
|
||||
If string, :obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
The non-linear activation function (function or string) in the encoder and pooler. If string,
|
||||
:obj:`"gelu"`, :obj:`"relu"`, :obj:`"swish"` and :obj:`"gelu_new"` are supported.
|
||||
resid_pdrop (:obj:`float`, `optional`, defaults to 0.1):
|
||||
The dropout probability for all fully connected layers in the embeddings, encoder, and pooler.
|
||||
embd_pdrop (:obj:`int`, `optional`, defaults to 0.1):
|
||||
|
|
|
|||
Некоторые файлы не были показаны из-за слишком большого количества измененных файлов Показать больше
Загрузка…
Ссылка в новой задаче