* optimize the tokenizer for efficiency
* fix the unit test failures.
* fix the api test case failures
* removed the unused code.
* More test cases fixings
* One more fixing
* fix macOS build issues
* refine the test
* add more diagnosis info.
* fix unit test in CI Linux
* fix the pp_api test failure
* initial api for tokenizer
* More fixings and test data refinement
* add a simple wrapper for pre-processing APIs
* fix the test issues
* test if the tokenizer is spm based
* fix the failed test cases
* json pointer does not work
* support tokenizer build only in C API mode
* fix the python build.
* fix the selectedops build
---------
Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com>
* reimplement resize cpu kernel for image processing
* accuracy fixing and code refinement
* fix the build issues
* fix Linux build issue
* more fixings
* Fix the pipeline issue
* fix the ci script
* try to fix CUDA machine pool
* Feature extraction C API for whipser model
* Update the docs
* Update the docs2
* refine the code
* fix some issues
* fix the Linux build
* fix more data consistency issue
* More code refinements
* add an experimental CUDA python unit test pipeline
* typo
* in ci.yml?
* winpycuda
* move it in optional
* enable cuda pytest in linuxbuild
* build in docker
* add the cuda pytest for windows
* cuda flag fixing
* minor fixing
* typo
---------
Co-authored-by: Yi Zhang <zhanyi@microsoft.com>
* add test .nuspec for Windows and respective docs
* minor doc change
* keep only x64 file entry and use build.bat output dll
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
* Add details to Azure ops documentation (#552)
* doc ops
* typo
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* rename the input and output as example
---------
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* initial api-docs implementation
* change pip install extensions
* add examples README as it is needed to build Sphinx gallery + some minor changes
* call sphinx directly from workflow
* fixed commit id issue for workflow
* fix path issue in publish-python-apidocs.yml
* add tools API and cleanup
* update copyright from current year
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
* doc ops (#529)
* Try and make CIs pass with Azure ops enabled by default.
Misc. other cleanups
* Fix some CI issues.
Cleanups some bits and pieces.
* Fix a couple of issues.
* Fix arg to build.bat
* Increase warning in triton client build to make binskim happy (hopefully).
* Try patching the warning level in the triton grpc branch as well. Shouldn't matter but...
* Run triton patch command for windows as well.
* Add patch.exe directly so windows builds work.
* override auth gen for AOAI
* fix build
* switch to windows-static
* update model for azure chat
* document triton invoker
* doc chat endpoint
* document triton invoker
* format
* format
* format
---------
Co-authored-by: Scott McKay <Scott.McKay@microsoft.com>
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* address comments
* move doc sect
* typo
* typo
---------
Co-authored-by: Scott McKay <Scott.McKay@microsoft.com>
Co-authored-by: Randy Shuai <rashuai@microsoft.com>
* initial checkins
* test pass
* basic impl
* first unit test pass
* merge error
* refine a little bit
* add more unit test
* fix unit test
* Fix the unit test.
* add one more whisper audiodecoder test case
* update the docs
* More updates
* release the dependency package version
* upgrade the dependency versions
* more fixing for 3.11
* Update ci.yml
* upgrade torch version in MacOS
* fix torch version in MacOS
* upgrade Python support versions
* pybind11 switch URL mode
* more URL modes
* evaluate the audio decoder library
* MP3 Decoder
* rename it to test_audio_codec
* add the audio decoder to whisper model
* whisper end-to-end draft
* fix the mp3 decoder
* Running with ONNX models
* Add more audio format supports
* refine the end-to-end script
* Update operators/audio/audio_decoder.hpp
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
* Update operators/audio/audio_decoder.hpp
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
* Update operators/audio/audio_decoder.hpp
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
* some fixings of comments and more test cases.
* changes for review comments.
* Update audio_decoder.hpp
* Update audio_decoder.hpp
* code refinement
* Update operators/audio/audio_decoder.hpp
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
---------
Co-authored-by: Sayan Shaw <52221015+sayanshaw24@users.noreply.github.com>
Co-authored-by: Edward Chen <18449977+edgchen1@users.noreply.github.com>
- Generate AAR with associated artifacts (javadoc.jar, sources.jar, pom).
- Fix JAR/AAR package descriptions.
- Remove `gradle -b build-android.gradle` invocation from CMake. This creates the AAR and it is not necessary to run it for each Android ABI build.
- Support building on Windows.
* update the main doc and add a developer doc
* add it back
* fix some typo
* Update README.md
Co-authored-by: Nat Kershaw (MSFT) <nakersha@microsoft.com>
Co-authored-by: Nat Kershaw (MSFT) <nakersha@microsoft.com>
* more C++ code fixing and polish for release
* fixing for android build
* build flags for android release
* add missing exporting function
* imint
* first versoin
* more C++ code fixing and polish for release (#275)
* more C++ code fixing and polish for release
* fixing for android build
* build flags for android release
* add missing exporting function
* support build_id on Python package building (#281)
* support buildid in package building
* undo the change on build.sh
* build.sh issue on macos
* Add `$schema` to `cgmanifest.json` (#284)
Co-authored-by: Jamie Magee <jamie.magee@microsoft.com>
* test package with a simple java app
* demo app
* some fixing for windows platform
* refine the example app
* fix the missing symobls issue for Linux build
* fix the package package build issue
* typo
* a missing change
* fix PythonOp
* fix Android test issue
* one more Android change
* replace build flags in ci pipeline
* android AAR package build
* refine the code for android package
Co-authored-by: Jamie Magee <jamie.magee@gmail.com>
Co-authored-by: Jamie Magee <jamie.magee@microsoft.com>
* initial checkins
* restructure the implementation.
* refine the Python interface
* Finalize the interface.
* Add the custmop class for the customization.
* Test the eager_op with vector_to_string customop
* Refine the customop conversion interface.
* initial onnx builder
* Runable with incorrect result.
* reformat the onnx_ops calls
* a few of operators working on tracing
* handcraft all op conversion
* Add the unit testing for mytorch
* unit test passed.
* Add some documents...
* Move non-torch API into onnxruntime_customops.utils module.
* Fix the unit test issues.
* Fix some typos.
* add batch_mode and padding for GPT2Tokenizer
* fix text
* fix test and add doc
* fix test
* fix comments
* delete header
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
* add vector_to_string
* fix merge conflict
* fix building failure
* remove debug code
* fix test
* move back unicode
* fix typo
* move base64 back
* move the right place
* support only int64_t
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
* Add attribute global_replace to StringRegexReplace
Signed-off-by: xavier dupré <xavier.dupre@gmail.com>
* fix potential wrong pointer
Signed-off-by: xavier dupré <xavier.dupre@gmail.com>
* update sep
Signed-off-by: xavier dupré <xavier.dupre@gmail.com>
Co-authored-by: xavier dupré <xavier.dupre@gmail.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
> It seems to be working now.
I enabled some less secured option in pipeline. let's see how it goes.
* test attribute
* finish improvement
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
Wenbing Li
Zhipeng Han
cao lei
Sayan Shaw
RandySheriffH
Edward Chen
Nat Kershaw (MSFT)
Mojimi
Zuwei Zhao
Xavier Dupré