This commit is contained in:
Sylvain Gugger 2021-02-20 15:46:54 -05:00
Родитель a0dfc2d30f
Коммит a2e379743c
1 изменённых файлов: 3 добавлений и 2 удалений

Просмотреть файл

@ -3054,11 +3054,12 @@ class PreTrainedTokenizerBase(SpecialTokensMixin):
def convert_tokens_to_string(self, tokens: List[str]) -> str:
"""
Converts a sequence of tokens in a single string. The most simple way to do it is ``" ".join(tokens)`` but
we often want to remove sub-word tokenization artifacts at the same time.
Converts a sequence of tokens in a single string. The most simple way to do it is ``" ".join(tokens)`` but we
often want to remove sub-word tokenization artifacts at the same time.
Args:
tokens (:obj:`List[str]`): The token to join in a string.
Returns:
:obj:`str`: The joined tokens.
"""