Граф коммитов

7 Коммитов

Автор SHA1 Сообщение Дата
Wenbing Li 711a2cfa69
add a convert_token_string_to_an_id API for the prompt ids (#794)
* add a convert token string to an id API for the prompt ids

* fix the build issues on Linux
2024-08-19 16:44:07 -07:00
Wenbing Li c3145b8f52
add the decoder_prompt_id for whisper tokenizer (#775)
* add the decoder_prompt_id for whisper tokenizer

* temporarily disable android prebuilt

* disable the prebuilt for android

* disable the prebuilt for android 2

* Add a unit test

* correct test ids
2024-07-29 14:21:17 -07:00
Wenbing Li c3c5f1cbb1
Remove C++ filesystem library dependency for the compatibility of old system (#721)
* Remove C++ filesystem library dependency for the compatibility of old OS.

* Update file_sys.h
2024-05-18 07:23:45 -07:00
Wenbing Li 311dd35401
Add ImageProcessor for Multimodel model Pre-processing (#715)
* only keep the image decoder from opencv

* initial build

* refine the code

* Add clear functions

* Update CMakeLists.txt

* Update opencv.cmake

* change the output type to float

* get the result

* align image-process with original Python

* move the LoadRawImages into library

* fix the calculation error

* fix the pipeline build issue

* fix the build breaks in ci pipeline

* support json configuration file and refactor the code.
2024-05-15 14:35:14 -07:00
Wenbing Li a8bce4328b
Add the tokenizer C ABI (#693)
* initial checkins

* fix the selectedops build failures

* add the tokenization implementation

* update the windows DEF file for c abi in cmake file

* fix the build on linux

* fix some warnings and remove the unused code

* initial import of unit tests from tfmtok

* add streaming API support

* fix the merges loading issues

* complete export from tfmtok - needs input id fixing

* fix the unit test failures.

* fix all unit test failure

* refactor streaming code

* remove the unused code

---------

Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
2024-04-29 16:45:49 -07:00
Wenbing Li f9290e8bac
Add a status class for future tokenizer API implementation (#690)
* Add a status class for future API implementation

* Update bpe_kernels.cc

* fix the ios package pipeline

* update mistral test model name
2024-04-18 21:12:14 -07:00
Wenbing Li 646462790b
Refactor the header file directory and integrate the eager tensor implementation (#689)
* refactor the header file in include folder

* fix the basic-token eager unit test case

* a more flexible way to handle string tensor shape.

* fix the unit test path issue

* remove the multi-inherits to avoid issue during pointer casting

* add api cmake build support

* undo some temporary changes

* code refinement

* fix variadic arg

* only expose the context for ort version >= 17

* fix a shape bug

* fix the cuda build issue

* change ifdef condition of GetAllocator

* finalize the ort c abi wrapper file name

* fix the iOS build break

* align gtest version with triton

* Update ext_apple_framework.cmake for iOS header files

---------

Co-authored-by: Cheng Tang <chenta@a100.crj0ad2y1kku1j4yxl4sj10o4e.gx.internal.cloudapp.net>
2024-04-17 12:58:19 -07:00