* initial api for tokenizer
* More fixings and test data refinement
* add a simple wrapper for pre-processing APIs
* fix the test issues
* test if the tokenizer is spm based
* fix the failed test cases
* json pointer does not work
* support tokenizer build only in C API mode
* fix the python build.
* fix the selectedops build
---------
Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com>
* Feature extraction C API for whipser model
* Update the docs
* Update the docs2
* refine the code
* fix some issues
* fix the Linux build
* fix more data consistency issue
* More code refinements
* initial checkins
* fix the selectedops build failures
* add the tokenization implementation
* update the windows DEF file for c abi in cmake file
* fix the build on linux
* fix some warnings and remove the unused code
* initial import of unit tests from tfmtok
* add streaming API support
* fix the merges loading issues
* complete export from tfmtok - needs input id fixing
* fix the unit test failures.
* fix all unit test failure
* refactor streaming code
* remove the unused code
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* add UT for neg_pos_cuda in eager mode and fix build break in Windows
* fix Linux build break
* adjust argument and path
* remove old cudaContext
* add ort cuda test back
* fix cuda tests
* undo debug code
* undo useless change
---------
Co-authored-by: jslhcl <jslhcl@gmail.com>
Co-authored-by: Cheng Tang <chenta@a100.crj0ad2y1kku1j4yxl4sj10o4e.gx.internal.cloudapp.net>
Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com>
* refactor the header file in include folder
* fix the basic-token eager unit test case
* a more flexible way to handle string tensor shape.
* fix the unit test path issue
* remove the multi-inherits to avoid issue during pointer casting
* add api cmake build support
* undo some temporary changes
* code refinement
* fix variadic arg
* only expose the context for ort version >= 17
* fix a shape bug
* fix the cuda build issue
* change ifdef condition of GetAllocator
* finalize the ort c abi wrapper file name
* fix the iOS build break
* align gtest version with triton
* Update ext_apple_framework.cmake for iOS header files
---------
Co-authored-by: Cheng Tang <chenta@a100.crj0ad2y1kku1j4yxl4sj10o4e.gx.internal.cloudapp.net>
- ifdef out some test code that requires RE2 if RE2 is not enabled
- add ability to plugin custom output validator for C++ unit tests
- OpenAI responses can have different punctuation. used in the new unit tests that will be in the refactoring PR
* new CI configuration
* Set up CI with Azure Pipelines
[skip ci]
* install numpy in cibuildwheel
* add pyproject.toml
* upgrade vmImage
* update the build python versions
* remove the pytest
* move the wheel build files
* enable sdist setup.py as well.
* use git command line
* Update wheels.yml for Azure Pipelines
* disable the pypy package for macos;
* fix the external repo code tag
* fix the ctest problem
* fix the unicode 8217.
* fix the locale base test
* inital commit
* update
* bring selectedoplist back
* remove unnessary change
* update
* fix unittest
* remove test
* fix windows building
* udpate
* update
* undo the changes on test cases
* add the missing C++ flags
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
Co-authored-by: Wenbing Li <wenli@microsoft.com>
* Reorg the code folder structure
* update the math test case
* Add an matrix inverse op.
* turn off the ctest by default.
* disbable jpeg lib in dlib for Linux build issue.
* Linux build fixing
* typo
* enable dlib library on Win32 build
* rename ocos to operators
* add the missing operator folder
* add vector_to_string
* fix merge conflict
* fix building failure
* remove debug code
* fix test
* move back unicode
* fix typo
* move base64 back
* move the right place
* support only int64_t
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
* basic changes
build issues fixing
runing on Windows Platform
deploy the ort library in CMake
update gitignore
* Add C++ shared tests
* enable ctest
* fixing the python build issue
* remove cc test
* why does macos needs openmp package?
Wenbing Li
Tang, Cheng
cao lei
Scott McKay
Mojimi
Xavier Dupré