70aa18e14e
add a native unit test for regex_split op ( #166 )
...
* add a native unit test for regex_split op
* fix the case of shape [1, 0]
* Update mshost.yaml
* downgrade the test model version.
* upgrade torch version on Windows CI
* disable windows python 3.7 pipeline.
2021-10-06 15:58:46 -07:00
4d7004bf6e
Correct memory indexing issue. ( #165 )
...
* Correct memory reservation.
* Fix the vmImage version for MacOS CI pipeline.
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-10-04 16:26:34 -07:00
4ebb8667da
Integrate the changes from the 0.4 release branch. ( #162 )
...
* generate the release packages on release branch
* no pyd file on Windows
* undo setup.py simplify
2021-09-29 13:18:36 -07:00
4290400ed3
Add doc for new operators ( #161 )
...
* add initial doc
* update doc
* finish all docs
Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-09-29 07:59:09 +08:00
b6e65f8780
Change farmhash to use uint128_t from the local namespace. ( #160 )
2021-09-27 19:49:03 -07:00
9eef22cb81
embedded farmhash source files to have some fixing ( #158 )
...
* embedded farmhash source files to have some fixing.
* update CMakefile
2021-09-27 16:04:46 -07:00
cddb3de158
Add ORT_API_CALL to AddExternalCustomOp so that the calling convention of the declaration in ocos.h matches the calling convention of the definition in ortcustomops.cc for x86 builds. ( #159 )
2021-09-27 16:04:28 -07:00
6ad95f6df1
upgrade ort and ir version mapping ( #157 )
...
* some bugs fixing for the gpt2 tools
* upgrade ort and ir version mapping
2021-09-27 14:56:22 -07:00
2d6cf0b4ea
Reduce bert tokenize memory usage ( #156 )
...
* add BertTokenizerVocab
* improve format
Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-09-27 11:19:57 -07:00
d8cdb8e042
reduce memory usage ( #154 )
...
Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-09-27 13:45:47 +08:00
9f3abe20fd
Prepare for 0.4.0 release ( #151 )
...
* new CI configuration
* Set up CI with Azure Pipelines
[skip ci]
* install numpy in cibuildwheel
* add pyproject.toml
* upgrade vmImage
* update the build python versions
* remove the pytest
* move the wheel build files
* enable sdist setup.py as well.
* use git command line
* Update wheels.yml for Azure Pipelines
* disable the pypy package for macos;
* fix the external repo code tag
* fix the ctest problem
* fix the unicode 8217.
* fix the locale base test
2021-09-25 00:40:12 -07:00
98c32dfe4a
Add new operator StringMapping ( #148 )
...
* update
* update python wrapper
* fix comments
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-09-14 09:13:08 +08:00
519755fcc0
fix bug ( #149 )
2021-09-13 10:29:08 -07:00
3542434876
Fix op selection ( #147 )
...
* fix segement_extraction
* update
Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-09-11 01:23:35 +08:00
d4b2aff0c8
Improve regex ( #146 )
...
* add test
* bring back test case
* add ignore case for regex
Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-09-09 13:24:49 +08:00
6d7a865913
Disable c++ exceptions in onnxruntime-extensions. ( #143 )
...
* Disable c++ exceptions in onnxruntime-extensions.
* Remove cxx flags for extensions.
* Remove redundant lines.
Co-authored-by: Zuwei Zhao <zuzhao@microsoft.com>
2021-09-09 08:21:40 +08:00
8649d98839
update ( #145 )
...
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-09-08 14:26:26 +08:00
cce66310b2
Improve recent checkin operators ( #144 )
...
* update
* update
* update
* remove tokenizer space
* fix bugs
Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-09-07 13:34:47 +08:00
2842d2208e
support the non-exception compiling for the text domain. ( #142 )
...
* support the non-exception compiling for the text domain.
* fix an path error.
2021-09-02 11:19:18 -07:00
97ec950751
Add SegementExtraction and BertTokenizerDecoder ( #140 )
...
* status
* update
* update
* fix bug
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-08-27 13:54:02 -07:00
cb47ee4d44
Standardize the public header files. ( #139 )
...
* a couple of fixing
* add a library alias
2021-08-27 12:51:02 -07:00
8d47fb7925
Fix comma error. ( #138 )
...
Co-authored-by: Zuwei Zhao <zuzhao@microsoft.com>
2021-08-27 10:15:38 -07:00
aef5ef1ef1
Add BertTokenizer ( #135 )
...
* init
* update
* update
* update
* update
* update
* update
* Modify relative path of generated cmake file.
* update
* udapte
* fix the bug
* update
* fix bugs
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
Co-authored-by: Zuwei Zhao <zuzhao@microsoft.com>
2021-08-26 13:50:03 -07:00
6c3b496e3f
Support the domain name in the PyOp. ( #136 )
2021-08-26 11:04:45 -07:00
00448bc78c
Replace Re2 with std::re2 ( #129 )
...
* inital commit
* update
* bring selectedoplist back
* remove unnessary change
* update
* fix unittest
* remove test
* fix windows building
* udpate
* update
* undo the changes on test cases
* add the missing C++ flags
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
Co-authored-by: Wenbing Li <wenli@microsoft.com>
2021-08-25 16:09:35 -07:00
f74770feed
Customize string operators list for cmake build. ( #134 )
...
* Customize operators list for build.
* Modify cmake_helper.py to not depend on onnxruntime-extensions build/install.
* Add new line at last.
* Move cmake_helper.py to ci_build/tools/ folder.
* undo some unwanted changes.
* Update CMakeLists.txt
* Update gen_selectedops.py
Co-authored-by: Zuwei Zhao <zuzhao@microsoft.com>
Co-authored-by: Wenbing Li <wenli@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-08-24 12:31:05 -07:00
52922ecc54
Update wheels.yml
2021-08-20 16:42:37 -07:00
e5e766d6dc
Update wheels.yml
2021-08-20 16:38:07 -07:00
4777213e31
Create wheels.yml ( #133 )
2021-08-20 16:29:29 -07:00
ef4e07c5f4
Add a dockerfile and update for build script. ( #131 )
...
* Add a dockerfile and update for build script.
* update
2021-08-19 10:38:05 -07:00
1ae69c0f7a
Users/zetao/fix link order ( #125 )
...
* fix link order
* fix format
* update
* bring option back
* bring selectedoplist back
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-08-17 13:21:12 +08:00
4eaa5ac4ae
support non-exception build ( #126 )
...
* support non-exception build
* fix the ir_version issue in the test
2021-08-16 11:08:03 -07:00
3fa03e1855
Update README.md
...
Update the build status link
2021-08-13 15:53:39 -07:00
0c57e02e1f
fix the ir_version issue due to the latest onnx release ( #128 )
2021-08-13 15:03:14 -07:00
983de7c0fe
support the operator list for build flags ( #122 )
...
* support the operator list for build flags
* revert the flag
* update the file name
* little refinement
2021-07-30 12:43:47 -07:00
a428be447c
optimize the gpt2-bs model after the generation ( #119 )
2021-07-08 15:36:11 -07:00
7fd96c8e97
fix the input shape in the gptbs model. ( #118 )
2021-07-02 18:00:09 -07:00
b708aeaaf4
add a disable_tokenizer option for the beam search process ( #116 )
...
* add an enable_tokenizer options
* a formal name
2021-06-29 15:05:14 -07:00
3e82549bcb
Add TextToSentenceTokenizer ( #113 )
...
* add depence
* init
* add test
* implement
* finished
* find file path througth pathlib
* rename the op
* update reminder
Co-authored-by: Ze Tao <zetao@microsoft.com>
Co-authored-by: Wenbing Li <10278425+wenbingl@users.noreply.github.com>
2021-06-24 14:29:16 +08:00
800e360ef3
add convert gpt-2 function in the bs script as well ( #114 )
2021-06-17 14:51:47 -07:00
fd1216b027
fix some build and env issues. ( #111 )
2021-06-10 09:57:49 -07:00
f4e1be286a
add sentencepiece pre-trained model test ( #110 )
2021-06-09 15:50:12 -07:00
cb81344392
install emsdk out of source ( #109 )
...
Co-authored-by: Ze Tao <zetao@microsoft.com>
2021-06-09 14:45:26 +08:00
803d9a3db9
Update patch file to integrate CustomOps into ONNXRuntime. ( #107 )
...
Co-authored-by: Zuwei Zhao <zuzhao@microsoft.com>
2021-06-08 18:26:14 -07:00
ba200b4a0e
import the opset version from the embeded ONNX model ( #105 )
...
* import the opset version from the embeded ONNX model
* real fix
2021-06-04 18:43:18 -07:00
88a3c0e42d
fix the onnxprocess for the empty input and name ( #104 )
...
* fixing the onnxprocess for the empty input and name
* fix the crash on onnxruntime 1.8
2021-06-03 21:23:13 -07:00
0851eacfeb
Update mshost.yaml for Azure Pipelines ( #102 )
2021-06-01 22:58:22 -07:00
48be0efea2
more refinement on the release doc and script ( #99 )
...
* more refinement on the release doc and script
* fix the build script.
2021-06-01 14:02:55 -07:00
e7b7eb5960
prepare the 0.3.0 release ( #98 )
...
* prepare the 0.3.0 release
* fix the install wheel
* Update mshost.yaml for Azure Pipelines
* update the yaml file
* update the yaml file
* update the yaml file
* put the wheels in one zipped file
* Revert "put the wheels in one zipped file"
This reverts commit bf4a086c94
2021-05-30 23:46:33 -07:00
567527b3b3
Create linux_release_x86_64.yml ( #97 )
...
* Create linux_release_x86_64.yml
* update release scripts
* Create linux_release_x86_64.yml
* temp1
* Update linux_release_x86_64.yml
kick off
* Update linux_release_x86_64.yml
* Update linux_release_x86_64.yml
* Update linux_release_x86_64.yml
* fix the cmake file.
* Update linux_release_x86_64.yml
* Update linux_release_x86_64.yml
* Update linux_release_x86_64.yml
* Fix the python3.x.lib missing issue.
2021-05-29 20:17:15 -07:00
Wenbing Li
joburkho
Mojimi
Zuwei Zhao