Funtowicz Morgan 9e03364999 Ability to pickle/unpickle BatchEncoding pickle (reimport) (#5039) ... * Added is_fast property on BatchEncoding to indicate if the object comes from a Fast Tokenizer. * Added __get_state__() & __set_state__() to be pickable. * Correct tokens() return type from List[int] to List[str] * Added unittest for BatchEncoding pickle/unpickle * Added unittest for BatchEncoding is_fast * More careful checking on BatchEncoding unpickle tests. * Formatting. * is_fast should assertTrue on Rust tokenizers. * Ensure tensorflow has correct way of checking array_equal * More formatting.		2020-06-16 09:25:25 +02:00
..
fixtures	…
__init__.py	…
test_activations.py	…
test_benchmark.py	…
test_configuration_auto.py	…
test_configuration_common.py	…
test_doc_samples.py	…
test_hf_api.py	…
test_hf_argparser.py	…
test_model_card.py	…
test_modeling_albert.py	Add AlbertForMultipleChoice (#4959)	2020-06-12 14:20:19 -04:00
test_modeling_auto.py	[AutoModel] Split AutoModelWithLMHead into clm, mlm, encoder-decoder (#4933)	2020-06-12 10:01:49 +02:00
test_modeling_bart.py	[Bart] Question Answering Model is added to tests (#5024)	2020-06-15 22:50:09 +02:00
test_modeling_bert.py	[AutoModel] Split AutoModelWithLMHead into clm, mlm, encoder-decoder (#4933)	2020-06-12 10:01:49 +02:00
test_modeling_camembert.py	…
test_modeling_common.py	[Bart] Question Answering Model is added to tests (#5024)	2020-06-15 22:50:09 +02:00
test_modeling_ctrl.py	[ctrl] fix pruning of MultiHeadAttention (#4904)	2020-06-10 14:06:55 -04:00
test_modeling_distilbert.py	Add DistilBertForMultipleChoice (#5032)	2020-06-15 18:31:41 -04:00
test_modeling_electra.py	ElectraForQuestionAnswering (#4913)	2020-06-10 15:17:52 -04:00
test_modeling_encoder_decoder.py	Split LMBert model in two (#4874)	2020-06-10 18:26:42 -04:00
test_modeling_flaubert.py	…
test_modeling_gpt2.py	…
test_modeling_longformer.py	Add more models to common tests (#4910)	2020-06-10 13:19:53 -04:00
test_modeling_marian.py	…
test_modeling_openai.py	…
test_modeling_reformer.py	…
test_modeling_roberta.py	Add more models to common tests (#4910)	2020-06-10 13:19:53 -04:00
test_modeling_t5.py	…
test_modeling_tf_albert.py	…
test_modeling_tf_auto.py	…
test_modeling_tf_bert.py	Support multiple choice in tf common model tests (#4920)	2020-06-11 10:31:26 -04:00
test_modeling_tf_camembert.py	…
test_modeling_tf_common.py	Support multiple choice in tf common model tests (#4920)	2020-06-11 10:31:26 -04:00
test_modeling_tf_ctrl.py	…
test_modeling_tf_distilbert.py	…
test_modeling_tf_electra.py	…
test_modeling_tf_flaubert.py	…
test_modeling_tf_gpt2.py	…
test_modeling_tf_openai_gpt.py	…
test_modeling_tf_roberta.py	…
test_modeling_tf_t5.py	…
test_modeling_tf_transfo_xl.py	…
test_modeling_tf_xlm.py	…
test_modeling_tf_xlm_roberta.py	…
test_modeling_tf_xlnet.py	…
test_modeling_transfo_xl.py	Fix resize_token_embeddings for Transformer-XL (#4759)	2020-06-10 19:03:06 -04:00
test_modeling_xlm.py	…
test_modeling_xlm_roberta.py	…
test_modeling_xlnet.py	Add more models to common tests (#4910)	2020-06-10 13:19:53 -04:00
test_onnx.py	…
test_optimization.py	…
test_optimization_tf.py	…
test_pipelines.py	…
test_tokenization_albert.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_auto.py	…
test_tokenization_bert.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_bert_japanese.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_common.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_ctrl.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_distilbert.py	…
test_tokenization_fast.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_gpt2.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_marian.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_openai.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_roberta.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_t5.py	…
test_tokenization_transfo_xl.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_utils.py	Ability to pickle/unpickle BatchEncoding pickle (reimport) (#5039)	2020-06-16 09:25:25 +02:00
test_tokenization_xlm.py	[HUGE] Refactoring tokenizers backend - padding - truncation - pre-tokenized pipeline - fast tokenizers - tests (#4510)	2020-06-15 17:12:51 -04:00
test_tokenization_xlm_roberta.py	…
test_tokenization_xlnet.py	…
test_trainer.py	Make DataCollator a callable (#5015)	2020-06-15 11:58:33 -04:00
test_trainer_distributed.py	Make DataCollator a callable (#5015)	2020-06-15 11:58:33 -04:00
utils.py	…