b045e66396
make onnx package to be optional. ( #653 )
...
* putting onnx package to be optional
* update the ci.yml
* add more message of missing ONNX package
2024-02-15 14:09:04 -08:00
394377b988
add a Mistral model unit test case ( #593 )
...
* add a Mistral model unit test case
* add the dependency
* Update test_autotokenizer.py
* Update test_autotokenizer.py
* Update test_autotokenizer.py
* Update test_autotokenizer.py
* Update test_autotokenizer.py
* Update test_autotokenizer.py
* Update test_autotokenizer.py
* Update test_autotokenizer.py 11
* Update ci.yml for Azure Pipelines
* Update ci.yml
* Update test_autotokenizer.py 12
* Update test_autotokenizer.py 13
2023-11-07 09:53:54 -08:00
92d17b0a52
remove ftfy imports and add warning if not installed ( #578 )
...
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
2023-10-19 14:09:55 -07:00
4d2930e35a
Fix newline and apostrophe handling for BPE ( #574 )
...
* Fix certain BPE issues
* minor changes
* change newline handling for unix/linux/windows builds
* small test case
* move apostrophe testing into test_cliptok.py
* fix CLIP inconsistency with ftfy install
* add ftfy to requirements-dev.txt
* remove HF CLIP bug testing
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
2023-10-19 00:17:34 -07:00
978ada6d60
Add TrieTokenizer for RWKV-like LLM models ( #509 )
...
* Add TrieTokenizer for RWKV-like LLM models
* add more tests
* fix the windows build
* downloading file instead of check in the vocab file
* a small bug fixing
2023-08-08 16:47:38 -07:00
1c04e95bda
upgrade all dependency versions ( #466 )
...
* release the dependency package version
* upgrade the dependency versions
* more fixing for 3.11
* Update ci.yml
* upgrade torch version in MacOS
* fix torch version in MacOS
* upgrade Python support versions
* pybind11 switch URL mode
* more URL modes
2023-06-03 20:09:41 -07:00
77cf3e6d2c
Remove onnx<1.14 from requirements.txt ( #447 )
...
* remove onnx<1.14 from requirements.txt
* downgrade protobuf
* move protobuf req to requirements-dev.txt
---------
Co-authored-by: Sayan Shaw <sayanshaw@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2023-05-21 23:34:09 -07:00
0f45fef2d9
Compatible with onnxruntime-gpu package ( #410 )
...
* be compatible without onnxruntime-gpu version
* some fixing
2023-04-26 17:17:23 -07:00
e8611d036d
Include requirements.txt from requirements-dev.txt. ( #328 )
...
Include requirements.txt from requirements-dev.txt so that pip has context to avoid installing incompatible dependencies.
2022-12-13 10:21:32 -08:00
8df25bf5a5
Fix ci pipelines failure caused by the latest transformer package ( #324 )
...
* test ci pipeline
* transformer 4.25 broken on pytorch 1.9/1.10
2022-12-06 15:21:06 -08:00
72790873e5
suppress the protobuf warning. ( #299 )
2022-10-05 00:29:46 -07:00
a7deb7b52c
Update CI build workflow matrix
...
Upgraded the onnxruntime headers from v1.6 to v1.9
Update workflow matrix so it's consistent across the platforms and using
newer versions of the dependencies. Current supported matrix -
+------------+------------------------+-----------------------+-----------------------+----------------------+
|Python | 3.7 | 3.8 | 3.9 | 3.10 |
+------------+------------------------+-----------------------+-----------------------+----------------------+
|Onnxruntime | 1.9.0 (Sept 22, 2021) | 1.10.0 (Dec 7, 2021) | 1.11.0 (Mar 26, 2022) | 1.12.1 (Aug 4, 2022) |
|Torch | 1.9.1 (Sept 22, 2021) | 1.10.0 (Oct 21, 2021) | 1.11.0 (Mar 10, 2022) | 1.12.1 (Aug 5, 2022) |
|TorchVision | 0.10.1 (Jun 15, 2021) | 0.11.1 (Oct 21, 2021) | 0.12.0 (Mar 10, 2022) | 0.13.1 (Aug 5, 2022) |
|TorchAudio | 0.9.0 (Jun 15, 2021) | 0.10.0 (Oct 21, 2021) | 0.11.0 (Mar 10, 2022) | 0.12.1 (Aug 5, 2022) |
+------------+------------------------+-----------------------+-----------------------+----------------------+
Release versions strictly follow the convention of onnxruntime being one
release ahead of all its dependencies.
2022-09-02 23:05:58 -07:00
dddd85397d
Add android pipeline ( #183 )
...
* build locally success
* update
* fix pipeline
* fix pipeline
* fix pipeline
* fix tool chain file
* validate quickly
* fix tool chain
* update
* finished
* fix bug
* fix bugs
* remove tree
* update android ndk
* fix bugs
* remove java install
* bring back build
* fix model
* resolve conflict
* remove uncessary file
* remove tensorflow_text version 2.6.0
Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-11-09 10:25:03 -08:00
a98c29f6d2
Implement custom operators for sentancepiece ( #41 )
...
* implements sentancepiecetokozenizer
* add ragged to sparse
* move one input to attribute
2021-01-27 23:55:50 +01:00
c7b2f864c6
Add Huggingface GPT2Tokenizer Support ( #35 )
...
* initialize a bbpe tokenizer
* add the json library.
* gpt2 tokenizer cpp implementation.
* Tom/add tutorial (#32 )
* Added getting started instructions for Windows
Signed-off-by: Tom Wildenhain <tomwi@microsoft.com>
* Created a tutorial for converting models with custom ops. WIP
* Removed long outputs
* Changed to keras syntax and added setup instructions
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
* rename gpt2 test case file
* polish the symbol names in the sources
* polish it again.
* fix the build issue on macos
* another fixing
* another fixing 3
Co-authored-by: TomWildenhain-Microsoft <67606533+TomWildenhain-Microsoft@users.noreply.github.com>
2020-12-21 17:12:32 -08:00
e36205ee83
Handles dummy python operators for double and strings ( #7 )
...
* refactor tests
* Update mshost.yaml
* Implements dummy operators with double and strings
* udpate CI
* Implements StringUpper C++ version
* Fix runtime issue preventing from registering multiple python ops
* add c++ operator StringJoin
* Support multi output for python and C++ operators
* remove torch in requirements-dev.txt
2020-10-30 11:20:18 +01:00
Wenbing Li
Sayan Shaw
Edward Chen
shaahji
Mojimi
Xavier Dupré