Summary:
In multi-threaded applications where we have multiple inferences on the
same model in parallel (consider e.g. a TTS system handling multiple
requests), it can be useful to share the parameters of a model amongst
these multiple instances. This improves the cache utilization behaviour
of the system, as multiple cores can use the same set of weights instead
of evicting the identical copies of weights in a shared cache.
As the underlying `NDArray` instances in `data_entry_` implement a
ref-counted based sharing system, this is a simple modification of the
`GraphRuntime::LoadParams` logic to instead copy parameters from an
existing GraphRuntime instance. This is a little ugly in that we need
both the pre-existing GraphRuntime instance, as well as the 'serialized'
params (since we need to know the set of names we should copy), but
without imposing additional assumptions (i.e. storing the set of param
names in GraphRuntime, and enforcing that shared param names are
identical to the parameters set in the preceding `LoadParams` call),
this seems unavoidable.
Test Plan:
Unit test added.
save
save
save
upstream
lint
remove bad changes
fix build
save
save
please the ci god
Update src/relay/pass/partial_eval.cc
Co-Authored-By: Wei Chen <ipondering.weic@gmail.com>
save
fix test
ci is ANGRY
fix rebase problem
fix rebase
add test
save
save
comment
* Add ability to load Prelude from disk
* Port over id
* Define compose
* Linting errors and style changes
* Eliminate unnecessary parens
* Rename identType to typeIdent (makes more sense)
* Another unnecessary paren
* Bump the version number for the text format
* Ensure .rly (Relay text files) are permitted
* Correct release number and simplify grammar rule
* Correct load_prelude docstring
* Corrections to _parser
* Add Apache headers to prelude source file
* Remove test_prelude (redundant)
* Correct misleading error message
* Add check that parser is enabled in Prelude
* Commit pre-generated parser, ensure generated files are treated as binaries, and have parser tests always fire
* Permit parser files and git attributes files
* Exclude gitattributes and parser files from apache check
* Another attempt at appeasing Apache audit checker
* Corrections to rat-excludes
* Apache should be truly appeased now
* Ignore Relay parser files by name
* Mark parser files as generated so they don't show up on Github
* Add parsing helper function for tests
* Mark parser files as not detectable
The exist code of a posix compilant shell is 0..255. Attempting to
return -1 will error in some shells and implicitly cast to 255 in
others. Fix it by returning a legal return value.
The existing RAT ASF license auditing logic ignores any failure in the
shell pipeline rather than just the exit code of the final grep.
Adjust the logic such that failure of the various tools in the
pipeline are not elided away.
* Fix x86 depthwise conv2d alter_op_layout
* Small fix
* Add test case
* Fix test
* Assert kernel layout
* Minor fix
* Add get_shape function
* Minor change
Currently, the BindVar function does not take Var type into account. This causes
two same graph structures with different var shapes to have same hash.
Structural hash is used for keeping track of which operators we have
already compiled. Because of this, two operators with different shapes end up
pointing to same compiled code. The failure is encountered at runtime, where the
expected input shape asserts are not met.
* [ARITH] Improve div/mod in rewrite simplifier
* Fix lint error
* Fuller file name in src/arithmetic/modular_set.h
Co-Authored-By: Wei Chen <ipondering.weic@gmail.com>
* Generalize some rules
* Replace gcd factoring with specialized rules
* Mark rules that don't work for non-truncated division
* More tests
* Some bug fixes in tensorflow graph converter and added DepthToSpace operator.
* Made DepthToSpace better comply with other function syntax.
* Added better shape inference for unusual situations.
* Lint fixes.
* Added depthtospace test.
* Added test cases for value inference and depthtospace.
* Added fill testing.
* Made comment changes and added BroadcastTo op and tests.
* Fixed underlining and unneeded opt_level forcing.
* Added _infer_value assertion that all values to infer are available in passed parameters.
* Register and use custom datatypes in TVM
This patch adds the ability to register and use a custom datatype from Python,
using the `register_datatype` call. The datatype can then be passed as the
`dtype` parameter using the syntax `dtype="custom[<type_name>]bitsxlanes"`.
* Removes extra file
* Register custom datatypes with TVM; specify Cast and Add lowering
This commit adds functionality for registering custom datatypes with TVM, and
furthermore adding custom lowering functions to lower those custom datatypes.
This commit only adds lowering for the Cast and Add ops; more ops will be added
soon.
Check out some custom datatype samples in my repository of samples:
https://github.com/gussmith23/tvm-custom-datatype-samples
* Register and lower casts from Python
* Formatting
* Fix include; was including too much
* Add comment
* Add DatatypeRegistered
* Add storage size field to custom datatypes
This field indicates the bitwidth of the opaque block of data into which
instances of the datatype will be stored, when TVM compiles. For example, if I
create a datatype with a storage size of 16, then
- Constants of that datatype will be created as unsigned 16-bit ints
- Calls to external functions taking that datatype will pass the data as
unsigned 16-bit ints
- External functions returning that datatype will be assumed to return unsigned
16-bit ints.
* Change how lowering funcs (Cast and other ops) are named in registry
tvm.datatypes.lower.<target>.cast.<dst-type>.<src-type>
becomes
tvm.datatypes.lower.<target>.Cast.<dst-type>.<src-type>
And fixes some sloppy code around how the other ops were being formatted.
* Update Python register_datatype to accept storage size
* Oops, left out one cast->Cast change
* Look up storage size when parsing `custom[typename]`
When we encounter this type string in Python, it will be parsed into a Halide
type object in C++. Some of my original code supported this parsing, but we now
have to attach the storage type to the type (by setting the bits field).
* Change how external calls for casting/other ops are done
Firstly, we now use the storage size of the custom type when determining
input/output types; e.g. a cast to a custom type with storage size 16 is seen as
a call to an external function returning an opaque uint of size 16.
Secondly, write a macro to handle the other ops. Originally I thought I could
handle these at runtime, with a single `_register_op` global. I transitioned
instead to using individual `_register_Add` etc. calls generated with a macro,
but I don't remember why.
* When encountering a custom type immediate, generate UIntImm
* Translate custom types to LLVM type
* Generate correct return type in Casts
Originally I was assuming that the result type from casts was always a custom
datatype, and so I was making the Call return a UInt type.
* Use TVM-idiomatic recursion style in DatatypesLowerer
This was actually a bug, I'm pretty sure; we wouldn't have recursed deep on any
complex programs. As a result of making this change, I also uncovered another
potential bug, where the datatypes lowering pass would attempt to lower a Load
of a custom type. By commenting out the `Mutate_` for Load, I was able to stop
the error from cropping up, but frankly, I'm not satisfied with the solution;
how is it that we are able to run codegen when Loads of custom datatypes are
present in the IR? I have not written any code, to my knowledge, that will
support this. Perhaps Load does not care about the underlying datatype?
* Use CHECK
* Add comment about which Mutate_s are needed
* Add comments
* Add GetCustomDatatypeRegistered as an extern C function
* Formatting, comments, casting
* Change how datatype string is formatted
* Use bits() instead of GetStorageSize
Use bits() instead of GetStorageSize
* Change comment
* Add datatype.py
* Change registered function name (datatypes->datatype)
* Remove GetStorageSize
* Format custom datatypes like any other datatype
Specifically, we now print the bits and lanes after the `custom[...]` string.
* Correctly implement datatype lowering in Python
* Remove unneeded include
* Make function naming consistent
* Use CHECK instead of internal_assert
* Rename macro
* Formatting
* Rename functions
* Implement Cast lowering
`_datatype_register_op` is now able to lower both binary ops and Casts.
* Formatting
* Formatting
* Clang format, google style
* Fix std::string/extern "C" warnings
* Formatting
* Formatting
* Lower Allocates and Loads during datatype lowering
This should ensure that there are no custom datatypes remaining once datatype
lowering is done. This will allow us to remove the code in the LLVM codegen
which deals with custom datatypes.
* Revert additions to codegen_llvm.cc which are now unneeded
* Pass cpplint on lower_datatypes.cc
* Add clarifying comment
* Remove datatype lowering registration funcs from C++
* Add CHECKs
* Remove TODO
* Remove all references to storage size
* Move and rename function
* Rename function
* Remove done TODOs and other handled comments
* Remove irrelevant Load code and comments
* Comment out the IR node types I'm not sure about yet
* Add bfloat16 datatype unittest
* Fix MakeConstScalar
MakeConstScalar for a custom datatype will now call out to a function which can
be registered on a per-datatype basis. The function will take a double and
return the equivalent value in the custom datatype format.
Note that these code paths are not actually used or tested at the moment. I have
not yet written an example which uses const scalars of a custom datatype.
* Formatting
* Change pass name
* Allow users to register whatever lowering function they want
Tianqi pointed out that users should be able to register whatever lowering
function they want, and should not be constrained to registering lowering
functions which just call out to external libraries.
I still provide a function for making lowering functions which call out to
external libraries, for convenience.
* Add clarifying comment
* Remove unneeded comment
* Remove unneeded function
* Rename file
* Undo unnecessary change
* Undo unnecessary change
* Make naming consistent
Rename "datatypes" to "custom datatypes" in most contexts.
* Revert an artifact of old code
* Fix build warnings, add TODO
* Lint
* Remove unnecessary use of extern C by separating decl and impl
* Error checking
* Remove TODO
* Missed a name change
* Lint
* Python lint
* Correctly format datatype
* Move bfloat16 to 3rdparty
* "custom_datatypes" --> "datatype" in most places
I left the pass as "LowerCustomDatatypes" to indicate that we're not lowering
anything other than custom datatypes. Otherwise, everything else has been
changed.
* Upgrade datatype unittest
I used a float calculator to generate some real testcases for the unittest.
* Separate public includes and private implementation
Specifically, create cleaner decoupling between datatypes stuff in packed_func
and the datatype registry implementation.
* Formatting
* Limit custom datatype codes to >128
* Add TODOs
* Fix comment
* Formatting
* Clean up datatype unittest
* Remove un-exported functions in public headers; UIntImm->FloatImm
More places where I accidentally was using implementation-only functions in
public headers.
Additionally, store custom datatype immediates as FloatImms. A later change will
add new lowering logic to lower these FloatImms to UIntImms.
Plus formatting change.
* Lint
* Use FloatImm (not UIntImm) to hold immediates of custom datatypes
This change switches from using UIntImm to FloatImm for storing immediates of
custom datatypes. The value of the number is stored in a double, which should be
enough precision for now, for most custom types we will explore in the immediate
future.
In line with this change, we change the datatype lowering so that FloatImms are
lowered to UInts of the appropriate size. Originally, this was going to be done
by allowing the user to register a double->uint_<storage size>_t conversion
which would be called at compile time to convert the value from the FloatImm to
a UInt and store it in a UIntImm. After discussions with Tianqi, we decided to
take the simpler route, and lower FloatImms just as we lower all other ops: by
replacing them with Call nodes. In this case, presumably the user will Call out
to a conversion function in their datatype library.
The justification for this decision is due to the functionality added in #1486.
This pull request adds the ability to load LLVM bytecode in at compile time.
This applies in our case as follows:
1. The user writes their custom datatype programs and registers their lowering
functions in the same way we've been doing it so far. All operations over
custom datatypes are lowered to Calls to the datatype library.
2. The user compiles their datatype library to LLVM bytecode.
3. At TVM compile time, the user loads the LLVM bytecode. Depending on how the
datatype library is written, Clang should be able to perform constant
folding over the custom datatype immediates, even if their conversions are
done with calls to the library.
Additionally adds test to test the FloatImm codepath.
* Re-add a change I removed accidentally during rebase
* Cleanup
* Remove unnecessary TVM_DLLs
* Add custom datatype utilities source file to Go runtime pack
* Revert "Remove unnecessary TVM_DLLs"
This reverts commit 4b742b99557fd3bf0ce6617f033c8b444b74eda4.
* Mark bfloat code as TVM_DLL
* Moves custom datatype runtime utilities to c_runtime_api.cc
* Revert "Add custom datatype utilities source file to Go runtime pack"
This reverts commit aecbcde0b2cc09a2693955b77037fe20f93b5bfd.
* Move datatype parsing to its own function
* Change comments
* Remove unneeded function
* Formatting
* Formatting
* Documentation
* Add kCustomBegin, use it for checking for custom types
* Documentation
* Formatting
* Move static definition to implementation
* Remove comment
* Decide toBeLowered before lowering arguments of Expr
In the past, e.g. when lowering custom datatypes for an Add, we would lower a
and b first, and then decide whether the resulting new Add needed to be lowered
based on the (new) types of a and b. Now, instead, we need to check the types of
a and b first (to see if they're custom types), and then lower them (so they'll
become non-custom types), and then lower the new Add.
* Revert "Move datatype parsing to its own function"
This reverts commit d554a5881afcf69af1c070d882a7651022703a09.
This broke parsing. Will figure this out later. There isn't a really clean way
to separate this out given how the rest of the function is written.
* Replace comment
* Documentation
* Remove comment and TVM_DLL
* Better error messages
* Remove artifact of rebase
* Separate datatypes parsing to its own function
* Add \returns
* Comment changes; add TODO
* Refactor tests
* ssd gluoncv gpu op updated
* ssd gluoncv gpu op updated
* tutorials and testes modified
* tutorials and testes modified
* fix lint
* fix lint
* address comment
* multibox bug fixed
* space line added
* use less threads per block
* use less threads per block
* less threads per block for get valid count
* less threads per block for get valid count
* merge with master
* Revert "less threads per block for get valid count"
This reverts commit 08896cfccc34b0b2a1646d01d01ea4cad73941c4.
* Revert "less threads per block for get valid count"
This reverts commit 08896cfccc34b0b2a1646d01d01ea4cad73941c4.
* typo fixed
* elem length made to a variable
* fix lint error
* fix lint error
* lint fixed
* bug fixed
* bug fixed
* lint fixed
* error fixed
* error fixed
* test ci
* test ci
* seperate argsort to be an independent op
* seperate argsort to be an independent op
* fix lint
* fix lint
* remove unsupported models
* typo fixed
* argsort added to realy
* solve conflicts with master
* fix lint
* fix lint
* test push
* Revert "test push"
This reverts commit 6db00883fab6cc06bddf564c926bb27c874397d8.
* fix lint error
* fix more lint
* cpu test_sort udpated
* debug ci
* nms fixed
* expose argsort to relay frontend
* test ci
* fix lint
* sort register error fixed
* fix nnvm
* nms type fixed
* adaptive pooling added to relay
* Revert "adaptive pooling added to relay"
This reverts commit 1119f1f2c055753e0cc5611627597749134c5c8c.
* fix lint
* expose argsort op
* fix lint
* fix lint
* fix lint
* sort test updated
* sort bug fixed
* nnvm error fixed
* fix argsort default data type returned to be float insteaf of int
* fix lint
* fix lint
* test fixed
* fix valid count
* fix titanx bug
* tutorial add both targets
* titanx error fixed
* try to fix CI old gpu error
* try to solve CI GPU error
* get_valid_count added
* reverse get_valid_count
* get valid count optimized
* address comments
* fix ci error
* remove unessesary block sync
* add back one sync
* address comments
* address more comments
* more comments
* move sort to be indepent algorithm
* typo fixed
* more typos
* comments addressed
* doc updated
* fix pylint
* address final comments
* apache license added
* Fixed issue #3069 by adding in_channels
* Registerd group_conv2d_nchw as topi compute
* Improved by checking tag value
* Removed group_conv2d_nchw topi registration
* Added test for relay group_conv2d_nchw
* Added assertions to forbid small group size
* Removed hard-coded oc_block_factor
* Added explanatory comments to group_conv2d_nchw_cuda
* Updated group_conv2d_nchw_cuda schedule
Removed 'direct' CUDA tests
* Reverted an accidental change in a conv2d test
* Fixed indentation problems
* Fixed a mis-commented line
* Reverted change in group_conv2d_nchw tag
* Removed commented int8 group_conv2d test
* Fixed group size assertions in group_conv2d_nchw_cuda
Due to the previous changes the frontend resnet example failed to build. So this patch
1) fixes it
2) adds ~~a local `run_tests.sh` to remedy non-existence of MXNet CI (used in python build example)~~ the example build to CI with random weights and a flag for pretrained resnet weights
Please review: @tqchen @nhynes @kazimuth
lint
lint
save
save
add more case
save
error
lint
lint
commit
do
lint
save
fix lint
wrap it back as func
lint
save
remove dead comment
fix style
fix lint
Update src/relay/pass/partial_eval.cc
Co-Authored-By: MarisaKirisame <lolisa@marisa.moe>
Update src/relay/pass/partial_eval.cc
Co-Authored-By: MarisaKirisame <lolisa@marisa.moe>
Update src/relay/pass/partial_eval.cc
Co-Authored-By: MarisaKirisame <lolisa@marisa.moe>
Update src/relay/pass/partial_eval.cc
Co-Authored-By: MarisaKirisame <lolisa@marisa.moe>
Update src/relay/pass/partial_eval.cc
Co-Authored-By: MarisaKirisame <lolisa@marisa.moe>
Update src/relay/pass/partial_eval.cc
Co-Authored-By: MarisaKirisame <lolisa@marisa.moe>
address review feedback
pe now handle freevar. as a result preserving function is now trivial.
test
add basic test, implement pretty printing for generic function
test
lint
fix segfault
save
save
do
test
fix another error
address comment
commit
save
address review feedback
add test for invalidate, fix error in lookup
rename cont to boduy
fix error and add regression test
fix error, add test case
Update src/relay/pass/partial_eval.cc
Co-Authored-By: MarisaKirisame <lolisa@marisa.moe>
fix lint
remove extra line
save
save
* [Relay] InferCorrectLayout for strided_slice
* Add min_num_branches option to CombineParallelConv2D
* Return undef if original layout contains splitted axes
* Update take
* Add special case for canonical simplify and fix test cases
* Use lower case for wrap and clip
* remove unnecssary lower
* Fix mxnet converter for take
* fix
A number of test scripts use the '|| exit 1' idiom. This has two
issues, first process exit codes are defined to be in the range 0-255.
Second, more importantly, the idiom is fragile because it requires
that every possible failure point be explicitly coded. This patch
removes the idiom in favour of "set -e" as used in the docker scripts
as a more robust mechanism to ensure that script failures are always
caught and propagated by default.
Code like this can't be built with NV OpenCL, and it needs an explicit type
converison for ternary expression if return type is uchar.
uchar i = 0, j = 0;
uchar t = max((uchar)j, ((i > 0) ? (uchar)1 : (uchar)0));
* initial commit
* add python frontend and module tests
* add unit tests for function pass and optimize interface
* add ExprPass
* remove PassState and pass context for run
* add required_passes
* return module
* remove move
* fix minor reviews
* remove optimizer, optimizer->pass_manager, make pass a the base class of all
* remove deleted files
* move resolvedependency to sequential pass, use ir_pass namespace
* add todo
* add disabled passes in sequetialpass
* fix minor
* fix currying doc
* remove pass_kind from passnode
* remove pass kind from test
* fix doc
* fix per @tqchen's comments
* remove pass_manager.py create separate classes
* simplify pass_func
* inline using passfunc
* update doc
* disable test_quantize_pass for now
* create PassInfo class to contain the meta data
* flatten passinfo for interface
* retrigger ci
* remove required method
* make Pass python class lighter
* create pass -> decorator
* make the api consistent for all classes
雾雨魔理沙
Altan Haan
ttyang1018
Yao Wang
Alexander Pivovarov
Yong Wu
Andrew Tulloch
Sammy
Haichen Shen
ziheng
Wei Chen
Jessica Davies
zhengdi
Tianqi Chen
Zhi
Jared Roesch
Wuwei Lin
Sheng Zha
Hua
Steven S. Lyubomirsky
hlu1
Marcus Shawcroft
Alexey Romanov
Ramana Radhakrishnan
Sergei Grechanik
Animesh Jain
Balint Cristian
masahi
Nick Hynes
Siju
eqy
Zhao Wu
lixiaoquan
Josh Fromm
Siva
llyfacebook
Gus Smith
Joshua Z. Zhang
Oldpan
Lianmin Zheng
Bing Xu
songqun
Leyuan Wang
Yizhi Liu
Gemfield
Ruizhe Zhao (Vincent)
Salem Derisavi
Hiroyuki Makino
Josh Pollock
Yong Wu
Steven S. Lyubomirsky
Ehsan M. Kermani
MORINAGA
Josh Pollock
Alexey Romanov
Sunwoong Joo
MORITA Kazutaka
Mr You
Yutetsu TAKATSUKASA
Siyuan Feng
Hao Jin
Leonardo lontra
Ashutosh Parkhi