Tarek Mahmoud Sayed 7cce7535b7 Add the components governance file `cgmanifest.json` for tokenizer's vocab files (#7283) ... * Add the governance file cgmanifest.json for tokenizer's vocab files * Address the feedback * apply more schema requirements on the doc		2024-11-01 15:20:59 -07:00
..
Data	Support Gpt-4o tokenizer model (#7157)	2024-05-21 08:32:08 -07:00
BertTokenizerTests.cs	Introducing WordPiece and Bert tokenizers (#7275)	2024-10-22 12:33:58 -07:00
BpeTests.cs	Introducing WordPiece and Bert tokenizers (#7275)	2024-10-22 12:33:58 -07:00
CodeGenTests.cs	Misc Changes (#7264)	2024-10-11 16:06:22 -07:00
EnglishRobertaTests.cs	Misc Changes (#7264)	2024-10-11 16:06:22 -07:00
LlamaTests.cs	Misc Changes (#7264)	2024-10-11 16:06:22 -07:00
Microsoft.ML.Tokenizers.Tests.csproj	Misc Changes (#7264)	2024-10-11 16:06:22 -07:00
NormalizerTests.cs	Tokenizer APIs Update (#7190)	2024-07-15 07:48:58 -07:00
PreTokenizerTests.cs	Introducing WordPiece and Bert tokenizers (#7275)	2024-10-22 12:33:58 -07:00
TiktokenTests.cs	Add the components governance file `cgmanifest.json` for tokenizer's vocab files (#7283)	2024-11-01 15:20:59 -07:00
TokenizerTests.cs	Misc Changes (#7264)	2024-10-11 16:06:22 -07:00
Utils.cs	Move the Tokenizer's data into separate packages. (#7248)	2024-10-04 14:47:37 -07:00
WordPieceTests.cs	Introducing WordPiece and Bert tokenizers (#7275)	2024-10-22 12:33:58 -07:00