Tarek Mahmoud Sayed bad82989a0 Adding needed Tokenizer's APIs (#7047) ... * Adding needed Tokenizer's APIs * Address the feedback * Small update to the newly exposed APIs * fix comments * Update the APIs signatures * More feedback addressing * Fix the comments		2024-03-06 19:04:15 -08:00
..
Data	Address the feedback on the tokenizer's library (#7024)	2024-02-26 10:57:18 -08:00
BpeTests.cs	Adding needed Tokenizer's APIs (#7047)	2024-03-06 19:04:15 -08:00
EnglishRobertaTests.cs	Adding needed Tokenizer's APIs (#7047)	2024-03-06 19:04:15 -08:00
Microsoft.ML.Tokenizers.Tests.csproj	Packaging cleanup (#6939)	2024-02-27 16:05:43 -08:00
NormalizerTests.cs	Address the feedback on the tokenizer's library (#7024)	2024-02-26 10:57:18 -08:00
PreTokenizerTests.cs	Address the feedback on the tokenizer's library (#7024)	2024-02-26 10:57:18 -08:00
TitokenTests.cs	Adding needed Tokenizer's APIs (#7047)	2024-03-06 19:04:15 -08:00
TokenizerTests.cs	Adding needed Tokenizer's APIs (#7047)	2024-03-06 19:04:15 -08:00
Utils.cs	Tokenizers Support (#6272)	2022-08-10 14:43:13 -07:00