Wenbing Li a8bce4328b Add the tokenizer C ABI (#693) ... * initial checkins * fix the selectedops build failures * add the tokenization implementation * update the windows DEF file for c abi in cmake file * fix the build on linux * fix some warnings and remove the unused code * initial import of unit tests from tfmtok * add streaming API support * fix the merges loading issues * complete export from tfmtok - needs input id fixing * fix the unit test failures. * fix all unit test failure * refactor streaming code * remove the unused code --------- Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>		2024-04-29 16:45:49 -07:00
..
tokenizer.json	Add the tokenizer C ABI (#693)	2024-04-29 16:45:49 -07:00
tokenizer_config.json	Add the tokenizer C ABI (#693)	2024-04-29 16:45:49 -07:00