Граф коммитов

17 Коммитов

Автор SHA1 Сообщение Дата
Wenbing Li a7f1a728d0
the doc for android/java custom build. (#260)
* the doc for android/java custom build.

* Update custom_build.md

* Update custom_build.md

* Update custom_build.md
2022-07-08 10:26:00 -07:00
Nat Kershaw (MSFT) f12f42a007
Update README.md (#238) 2022-07-06 16:18:35 -07:00
Mojimi abdd5b1bd8
Add MaskedFill (#182)
* add StringRemove

* update

Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-11-04 09:29:17 +08:00
Mojimi 4290400ed3
Add doc for new operators (#161)
* add initial doc

* update doc

* finish all docs

Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-09-29 07:59:09 +08:00
Wenbing Li 06c902253f
re-package onnx process code (#93)
* re-package onnx process code

* update the docs and examples as well.
2021-05-14 09:23:50 -07:00
Wenbing Li 3806e29421
rename the package to onnxruntime_extensions (#92)
* rename the package to onnxruntime_extensions

* fix the cmake file
2021-05-12 12:02:57 -07:00
Zuwei Zhao 794890985e
Support custom operator: StringToVector. (#84)
Co-authored-by: Zuwei Zhao <zuzhao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-04-22 13:29:43 +08:00
Wenbing Li aa846c3116
Support the pre/post processing Python and pytorch code snippets to be converted and be merged into the ONNX model. (#76)
* initial checkins

* restructure the implementation.

* refine the Python interface

* Finalize the interface.

* Add the custmop class for the customization.

* Test the eager_op with vector_to_string customop

* Refine the customop conversion interface.

* initial onnx builder

* Runable with incorrect result.

* reformat the onnx_ops calls

* a few of operators working on tracing

* handcraft all op conversion

* Add the unit testing for mytorch

* unit test passed.

* Add some documents...

* Move non-torch API into onnxruntime_customops.utils module.

* Fix the unit test issues.

* Fix some typos.
2021-03-12 10:39:21 -08:00
Xavier Dupré 4bc5c962b0
Add WordpieceTokenizer (#72)
* add Wordpiece tokenizer
* add RaggedTensorToDense
* update documentation
2021-03-11 19:19:49 +01:00
Mojimi 9653f52341
Add batch query and attention mask support for GPT2Tokenizer (#75)
* add batch_mode and padding for GPT2Tokenizer

* fix text

* fix test and add doc

* fix test

* fix comments

* delete header

Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-03-10 13:57:07 -08:00
Mojimi 2378ca116b
add StringConcat (#70)
Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-02-25 10:44:30 +08:00
Mojimi da41b75467
Add op: VectorToString (#57)
* add vector_to_string

* fix merge conflict

* fix building failure

* remove debug code

* fix test

* move back unicode

* fix typo

* move base64 back

* move the right place

* support only int64_t

Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-02-09 09:41:00 -08:00
Xavier Dupré b3a300d7bf
Add attribute global_replace to StringRegexReplace (#55)
* Add attribute global_replace to StringRegexReplace

Signed-off-by: xavier dupré <xavier.dupre@gmail.com>

* fix potential wrong pointer

Signed-off-by: xavier dupré <xavier.dupre@gmail.com>

* update sep

Signed-off-by: xavier dupré <xavier.dupre@gmail.com>

Co-authored-by: xavier dupré <xavier.dupre@gmail.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>

> It seems to be working now.

I enabled some less secured option in pipeline. let's see how it goes.
2021-02-05 09:08:02 -08:00
Xavier Dupré 4c201e7800
Change Fix type of nbtest_size into int64 (#54)
Signed-off-by: xavier dupré <xavier.dupre@gmail.com>

Co-authored-by: xavier dupré <xavier.dupre@gmail.com>
2021-01-29 14:21:50 -05:00
Xavier Dupré a32f9bc28c
Documentation for SentencepieceTokenizer (#52) 2021-01-28 19:07:48 +01:00
Mojimi a9a498501c
Improve GPT2 (#48)
* test attribute

* finish improvement

Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-01-27 15:26:37 -08:00
Mojimi 4a0f892949
Operator Schemas of text processing ops (#42)
* add a schema doc for new Op
Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-01-21 15:07:32 +08:00