* - Ensure we log an error message before throwing on Android
- message in exception will be lost due to how the shared libraries are built (both onnxruntime and extensions use static libc++ so there are no shared exception types between them)
- support static or dynamic build of curl/openssl on android
- TBD which we want to use.
- add infra for anything deriving from BaseKernel to log messages using the ORT logger
- ensures messages from custom kernels end up in the same place as messages from ORT
* Fix GPT2 and Falcon tokenizer cvt for AutoTokenizer imp
* fix fast tokenizer issue
* small fix
* use slow tokenizer in test script
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* Fix LNK4098 warning from sentencepiece forcibly changing the build flags.
> LINK : warning LNK4098: defaultlib 'LIBCMT' conflicts with use of other libs; use /NODEFAULTLIB:library
* Use CMAKE_MSVC_RUNTIME_LIBRARY to determine whether /MT should be used.
* Add TrieTokenizer for RWKV-like LLM models
* add more tests
* fix the windows build
* downloading file instead of check in the vocab file
* a small bug fixing
* Refactor setup for Azure ops to try and make common things more re-usable, and for the actual ops to simply layer in the specific input/output constraints for that type of request.
Currently builds on Linux, Windows (x64 only) and Android
Android requires a manual pre-build of openssl and curl.
Linux requires a manual pre-install of openssl.
Windows currently only works for x64. Other targets need the triplet adjusted.
* Address PR comments
* Fix could of android build warnings.
* Update .gitignore to remove old path
* Fix build break from merge
* Add Falcon-7b and Falcon-40b tokenizer support
* fix alignment and add tokenizer file in test/data to speed up compute
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
- ifdef out some test code that requires RE2 if RE2 is not enabled
- add ability to plugin custom output validator for C++ unit tests
- OpenAI responses can have different punctuation. used in the new unit tests that will be in the refactoring PR
* set before-test
* test cmd
* clean in yml
* restore toml
* add ut for triton endpoints
* reset working path
* rename suffix
* install ort
* pip install
* make env
* add extra env
* make executable
* set dir for linux
* add switch
* set env default
* skip tests
* simplify env
* clean env for official
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
* Update ci.yml for Azure Pipelines
macOS ci pipeline fixing.
* Update ci.yml for Azure Pipelines
* Update ci.yml for Azure Pipelines
* drop python 3.8 support in macOS due ADO
* fix macos wheel pipeline
* revert the change to add 3.9 back.
* Nodes can be called concurrently and Compute needs to be stateless due to that.
Update the kernels to make Compute const.
* Fix test that uses ustring.h.
Would be better to not have duplicate declarations for GetTensorMutableDataString and FillTensorDataString in ustring.h and string_tensor.h.
* initial checkins
* test pass
* basic impl
* first unit test pass
* merge error
* refine a little bit
* add more unit test
* fix unit test
* Fix the unit test.
* add one more whisper audiodecoder test case
* update the docs
* More updates
* upgrade gradle wrapper to unblock android CI
* Revert "upgrade gradle wrapper to unblock android CI"
This reverts commit 4df4f253aa.
* set timeout of gradlew
* "re-update it"
* Update gradle-wrapper.properties
* using gradle 8.0.1
* reformat the files
* fix the build.gradle issue
* one more fixing
* delay gradle starts
* rerun the pipeline
* add perf changes for CLIP and Roberta
* add perf improvement for BERT
* remove global var
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* using the latest Ort header instead of minimum compatible headers
* Update ext_ortlib.cmake
* Update ortcustomops.def
* change the default ORT API version value
Pass `-gpu swiftshader_indirect` emulator args as a workaround to get Android emulator running on the MacOS 13 hosted agents.
See https://github.com/actions/runner-images/issues/7671
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
* release the dependency package version
* upgrade the dependency versions
* more fixing for 3.11
* Update ci.yml
* upgrade torch version in MacOS
* fix torch version in MacOS
* upgrade Python support versions
* pybind11 switch URL mode
* more URL modes
* use lite custom op api for math
* add vision ops
* add cx2 ops
* remove useless code
* support register custom kernel struct
* add string tensor support
* add more text kernels
* fix issue with std stringg as scalar
* migrate all text ops
* initial tokenizer change
* migrate all tokenizers
* Resolve conflict with main (#433)
* resolve conflict
* resolve conflict
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* Update custom-op-lite PR (#440)
* add the onnxruntime 1.14 release into the CI pipeline (#387)
* add the onnxruntime 1.14 release into the CI pipeline
* torch 2.0 crashed on Linux
* Fix size_t overflow issue for RobertaTokenizer (#388)
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* Pre and Post processing example for openAI Whisper model (#380)
* add a stft-norm custom op for log-mel spectrum.
* undo the debug change
* Support ONNX standard STFT op signature.
* Add a unit test onnx STFT compatible mode.
* add whisper pre-/post- processing example
* Update dlib.cmake
* undo test code changes
* Update setup.cfg
* update the end2end example with STFT op
* Added optional outputs for GPT2, CLIP and Roberta Tokenizers (#389)
* Initial optional i/o for robertap
* Small fix
* Added working optional output functionality to RobertaTokenizer with tests
* Added optional outputs to CLIPTokenizer
* Added optional outputs to GPT2Tokenizer
* Use ternary operators
---------
Authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* ignore the unknown token id on bpe deocder (#391)
* Use dependency name 'nlohmann_json' which is the same name that ORT uses. (#393)
* Add an audio decoder custom op for whisper end-to-end processing (#385)
* evaluate the audio decoder library
* MP3 Decoder
* rename it to test_audio_codec
* add the audio decoder to whisper model
* whisper end-to-end draft
* fix the mp3 decoder
* Running with ONNX models
* Add more audio format supports
* refine the end-to-end script
* Update operators/audio/audio_decoder.hpp
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
* Update operators/audio/audio_decoder.hpp
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
* Update operators/audio/audio_decoder.hpp
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
* some fixings of comments and more test cases.
* changes for review comments.
* Update audio_decoder.hpp
* Update audio_decoder.hpp
* code refinement
* Update operators/audio/audio_decoder.hpp
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
---------
Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com>
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
* make tensorflow be optional for unittest (#394)
* make tensorflow be optional for unitest.
* typo
* built-in bounding box op (#382)
* built-in bounding box op
* update boundary check
* assert policy
* more boundary test and check
* XYXY--> X horizon
---------
Co-authored-by: Scott McKay <skottmckay@gmail.com>
* a quick nuget package impl. (#396)
* Update wheels_linux.yml: change the linux machine pool name (#398)
* Add a merge step in whisper end-to-end script and fixed some issues (#399)
* add merged models in whisper model
* verify the final model
* support batch > 1 in BpeDecoder (#400)
* support batch > 1 in BpeDecoder
* update the shape in helper function
* [object detection ppp] YoLo as example (#397)
* object detection
* Unit test
add e2e fastestdet model test
---------
Co-authored-by: Changming Sun <chasun@microsoft.com>
Co-authored-by: Scott McKay <skottmckay@gmail.com>
* some fixing for python package (#401)
* more code fixing related whisper models (#403)
* Added windows nuget work temporarily for testing (#402)
* Added windows nuget work temporarily for testing
* Cleanup
* Add back onnxruntime.lib in props file for possible future ORT need
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
* Remove unnecessary nupkg file and update nuspec (#405)
* Add nuget pack to build.bat and small nuget changes for demo
* Temporarily adding nuget.exe to build package until we can add to CI machine
* Switch back from Release to RelWithDebInfo
* Remove unnecessary changes
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* Add initial NuGet pipeline for Windows x64 build (#406)
* initial nuget pipeline
* Update nuget.yml for Azure Pipelines
* update nuget.yml for extensions specific packaging
TODO: add certain template yml files
* added component governance template yaml
* change template yaml path
* remove RoslynAnalyzers
* Add packDestination to nuget pack task (change from default)
* fix nuspec path
* Update nuget.yml for Azure Pipelines
* Update nuget.yml for Azure Pipelines
* Update nuget.yml for Azure Pipelines
* Update 2 nuget.yml for Azure Pipelines
* Update NativeNuget.nuspec
* Update nuget.yml for Azure Pipelines
* update nuspec
* Update 3 nuget.yml for Azure Pipelines
* Update 4 nuget.yml for Azure Pipelines
* Update 7 nuget.yml for Azure Pipelines
* Remove unnecessary nupkg file and update nuspec (#405)
* Add nuget pack to build.bat and small nuget changes for demo
* Temporarily adding nuget.exe to build package until we can add to CI machine
* Switch back from Release to RelWithDebInfo
* Remove unnecessary changes
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* Update 8 nuget.yml for Azure Pipelines
* Update 9 nuget.yml for Azure Pipelines
* add DLL signing
* Update nuget.yml for Azure Pipelines
* fix indendation
* Update 11 nuget.yml for Azure Pipelines
* Update 12 nuget.yml for Azure Pipelines
* Update 12 nuget.yml for Azure Pipelines
* Revert some unneccesary changes on nuget.yml
* clean up nuget.yml and update nuspec release notes
* small changes
* update commit id and release notes
---------
Co-authored-by: Wenbing Li <wenbingl@outlook.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* Compatible with onnxruntime-gpu package (#410)
* be compatible without onnxruntime-gpu version
* some fixing
* Add nuget README and remove ort lib references from props (#409)
* Add nuget README and remove ort lib references from props
* replace commit id in nuspec dynamically
* remove $ sign for commit id token
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
* Add an C# demo project for NuGet package (#407)
* Add a nuget test app
* remove unused file
* Compatible with onnxruntime-gpu package (#410)
* be compatible without onnxruntime-gpu version
* some fixing
* turn it as a .net demo project
---------
Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com>
* Make Whisper E2E script more portable (#412)
This PR makes the Whisper E2E script more portable for other environments.
* Update macos wheel timeout to 180 min (#390)
* Update ci timeout to 120 min
* Only update WindowsPython job timeout
* Update ci timeout to 90 min
* update macos wheel timeout to 180 min
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* Fix OneBranch PR pipeline CodeQL issue (#413)
* test codeql 3000
* switch codeql from compiled to python
* switch back to compiled
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* Adding down-sampling and stereo mixing features for AudioDecoder (#420)
* initial draft
* second
* third
* polishing
* fix the M_PI name in LINUX platform
* fix bessel function issue
* add a unit test case
* fix the unit test name
* Fix Secure Supply Chain Analysis Warning in PR pipeline (#414)
* remove package sources
* remove NuGet.config
* add .sscignore for cfs0011
* change sscignore
* add CFS0013 to sscignore
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* fix onnx version to 1.13.1 (#422)
* [NuGet] All platform package pipeline (#408)
* nuget ci package
* disable macos arm64 build for err
* Get the iOS xcframework build working with the split build/pack approach. (#416)
* refine build_xcframework.py
Cleanup/clarify various things
- naming of parameters and files
- consistency
Make handling of additional build args more generic
Update the artifact download dir/extract dir to more intuitive names
Update scripts
- make usage from CI pipeline clearer (e.g. don't hide directory names inside script)
- keep comments in nuspec
- remove unused args
- make additional arg handling more
Co-authored-by: Scott McKay <skottmckay@gmail.com>
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
* Add new required pre/post processing ops to Android and iOS packages. (#415)
* Revert "Pin onnx version to 1.13.1" (#423)
* Revert "fix onnx version to 1.13.1 (#422)"
This reverts commit eb29d225a7.
* Update requirements.txt
* PyOp attribute supports int and float data type (#425)
* Fix Android AAR in nuget package. Requires libortextensions.so. (#429)
* build for mac M1 (#430)
* Fix the unit test failure with ONNX 1.14 package. (#428)
* Fix the unit test failure with ONNX 1.14 package.
* more tests
* Update whisper_e2e.py
* Add nuget.org publish version option (#426)
* Add nuget.org publish version option
* typo
* small fix
* typo
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* resolve conflict
* resolve conflict
* minor fix
* rename from TensorT to Tensor
* fix string tensor
* Add OrtLiteCustomOp
* switch to string view
* fix regex ops
---------
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com>
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
Co-authored-by: JiCheng <247153481@qq.com>
Co-authored-by: Scott McKay <skottmckay@gmail.com>
Co-authored-by: Changming Sun <chasun@microsoft.com>
Co-authored-by: Wenbing Li <wenbingl@outlook.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* Fix a build err (#442)
* resolve conflict
* resolve conflict
* minor fix
* rename from TensorT to Tensor
* fix string tensor
* Add OrtLiteCustomOp
* switch to string view
* fix regex ops
* fix build
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* Fix build err on ort 141 (#444)
* resolve conflict
* resolve conflict
* minor fix
* rename from TensorT to Tensor
* fix string tensor
* Add OrtLiteCustomOp
* switch to string view
* fix regex ops
* fix build
* fix a build err
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* Remove shape from span (#445)
* resolve conflict
* resolve conflict
* minor fix
* rename from TensorT to Tensor
* fix string tensor
* Add OrtLiteCustomOp
* switch to string view
* fix regex ops
* fix build
* fix a build err
* remove shape
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* Fix python tests (#446)
* resolve conflict
* resolve conflict
* minor fix
* rename from TensorT to Tensor
* fix string tensor
* Add OrtLiteCustomOp
* switch to string view
* fix regex ops
* fix build
* fix a build err
* remove shape
* fix python tests
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* Fix max build (#449)
* resolve conflict
* resolve conflict
* minor fix
* rename from TensorT to Tensor
* fix string tensor
* Add OrtLiteCustomOp
* switch to string view
* fix regex ops
* fix build
* fix a build err
* remove shape
* fix python tests
* fix packaging err
* fix mac build
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* Fix comments (#452)
* resolve conflict
* resolve conflict
* minor fix
* rename from TensorT to Tensor
* fix string tensor
* Add OrtLiteCustomOp
* switch to string view
* fix regex ops
* fix build
* fix a build err
* remove shape
* fix python tests
* fix packaging err
* fix mac build
* fixing the universal2 python package for macOS (#448)
* Remove onnx<1.14 from requirements.txt (#447)
* remove onnx<1.14 from requirements.txt
* downgrade protobuf
* move protobuf req to requirements-dev.txt
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
* fix comments
* comment version macro
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com>
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* Fix build err (#453)
* resolve conflict
* resolve conflict
* minor fix
* rename from TensorT to Tensor
* fix string tensor
* Add OrtLiteCustomOp
* switch to string view
* fix regex ops
* fix build
* fix a build err
* remove shape
* fix python tests
* fix packaging err
* fix mac build
* fix comments
* comment version macro
* define Compute for StftNormal
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* Merge latest main (#461)
* resolve conflict
* resolve conflict
* minor fix
* rename from TensorT to Tensor
* fix string tensor
* Add OrtLiteCustomOp
* switch to string view
* fix regex ops
* fix build
* fix a build err
* remove shape
* fix python tests
* fix packaging err
* fix mac build
* fix comments
* comment version macro
* define Compute for StftNormal
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* revert wanted changes in test
* revert unwanted changed
* add string_strip op
---------
Co-authored-by: Cheng Tang <chenta@microsoft.com@orttrainingdev9.d32nl1ml4oruzj4qz3bqlggovf.px.internal.cloudapp.net>
Co-authored-by: RandySheriffH <48490400+RandySheriffH@users.noreply.github.com>
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com>
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
Co-authored-by: JiCheng <247153481@qq.com>
Co-authored-by: Scott McKay <skottmckay@gmail.com>
Co-authored-by: Changming Sun <chasun@microsoft.com>
Co-authored-by: Wenbing Li <wenbingl@outlook.com>
Co-authored-by: kunal-vaishnavi <115581922+kunal-vaishnavi@users.noreply.github.com>
Wenbing Li
Scott McKay
Sayan Shaw
kunal-vaishnavi
RandySheriffH
Changming Sun
Rachel Guo
JiCheng
Edward Chen
Tang, Cheng
Aidan Ryan (MSFT)