* Initial unit tests and KNN build test
* Fix linking error
* Fix TPT tests
* Change test files and fix tpt test issues
* Fix linking issues
* Fix buildssd tests and add new tests - new bug with SPTAG logger when running tests
* Add benchmark tests for PQ optimization
---------
Co-authored-by: MaggieQi <chenqi871025@gmail.com>
* outline convenience function changes
* implement GetPostingDebug
* fix make_shared of abstract class
* dont need to change VectorIndex.h interface
* resolve build
* Add support for and error checking for 384 dim and other PQ dimensions
* Fix error message for GPU code
---------
Co-authored-by: MaggieQi <chenqi871025@gmail.com>
* add protections against overflow and size mismatch, and invalid IDs
* avoid compilation issue
* add more safety to rebuild job
* add check to RNG
* try to resolve gcc compiler issue
* debug-guard expensive check
* fix the logging macro
* make it more branch-predictor friendly
* transform macro into function
* turn KDT macro to function
* special case for index==-1
* Update to LL_ERROR
* prevent extra allocations in BKT search by templated search function
* static dispatch in KDT index
* fix RNG prefetching
* skip checking index in graph traversal, since we will check in At
* dispatch by switch instead of if for fewer branches, use AlwaysTrue when filterFunc is null to allow compiler to optimize
* make the template function naming easier to understand, formatting improvements
* make checks in RebuildNeighbors IF_DEBUG only
* Allow setting a different Logger at runtime
Mutex is needed here because Logger::Logging is not const, so we can mangle things if shared_ptr is not updated atomically. And specialization for std::atomic<std::shared_ptr<T>> isn't availible to us in C++17
* Missed one file
* use atomics
* add filter support for BKT index
* put the nullptr check into the upper function instead of #define, and add filter checking before duplicated check
---------
Co-authored-by: Qianxi Zhang <qiazh@microsoft.com>
Co-authored-by: Qianxi Zhang <Qianxi.Zhang@microsoft.com>
Co-authored-by: MaggieQi <chenqi871025@gmail.com>
I have gone to "https://sourceforge.net/projects/boost/files/boost-binaries/1.67.0/" and downloaded "boost_1_67_0-msvc-14.1-64.exe", but encountered this failure during cmake
-- Could NOT find Boost (missing: system thread serialization wserialization regex filesystem) (found suitable version "1.67.0", minimum required is "1.66")
CMake Error at src/legacy/sptag/SPTAG/CMakeLists.txt:90 (message):
Could not find Boost >= 1.67!
I've notice that there are no precomiled libs, and had to run bootstrap + build exe to get the compiled libs.
Co-authored-by: Philip Adams <35666630+PhilipBAdams@users.noreply.github.com>
* modify for thread_local context
* fix initialization issue
* fix ExtraWorkSpace id issue
* fix workSpacePool
* set thread affinity
* add more affinity strategies
* fix cmake compiler
* fix linux libnuma compile
* fix compiling and core bind
* fix NumaStrategy and OrderStrategy enum type
* remove space
* Clear the workspace to ensure the heap size and pagebuffer size
* User-overrideable worskpace implementation draft (#362)
* make it possible to override workspace implementation
* bool -> ErrrorCode
* SPANN index should allow setting child index workspace
* finish replacing by workspace factory
* switch to unique_ptr
* unresolved external
* linux build error
* windows build error
---------
Co-authored-by: cheqi <cheqi@SRGSSD-07>
Co-authored-by: Philip Adams <35666630+PhilipBAdams@users.noreply.github.com>
* add winrt projects
* add gitignore for VS files
* retarget to vc142
* make neighborCount uint32
* api takes byte[] for metadata
* enable CFG and disable incremental linking to make BinSkim pass
* format
* remove edit and continue /ZI since it's incompatible for CFG
* remove arm/arm64 platforms
* Addresses static initialization fiasco problem with the logger and code calling into it from other translation units
* fix namespace
* make logger init multithreading safe
* actually we can use magic statics to simplify
* Fix#356
* gate GMH/GPA on Windows only
* update python to python3
* use dynamic linking
* enable ANNIndexTestTool code compiling
* add python version in wheel package
* update nuspec
* enable to set different maxcheck and hashexponent
* change to configure python version
* trigger azurepipeline
* trigger
* fix python version
* fix python version
* fix python version in windows
* fix cosine kmeans
* clean avx/see header files
* fix nuspec
Co-authored-by: cheqi <cheqi@SRGSSD-07>
* Remove Transposed Point to save shared Memory
* Add Generate GT int8 Dim to 768, add CUDA CHECK to debug, Add infty for uint32
* Fix sharedmem usage for K=100 gt
Co-authored-by: Diego Cai <diegocai@microsoft.com>
Co-authored-by: MaggieQi <chenqi871025@gmail.com>
* Fixed bugs with accuracy fix for TPT build with recon vectors
* Accuracy fix working for GPU index build with PQ/OPQ enabled
* Fix bug with accuracy fix
* Trigger CI
* Fix low accuracy issue with GPU index build for int8/cosine configuration (#339)
* Fix bug with int8/cosine configuration, and enabled hardware optimization for this case.
* Trigger CI
Co-authored-by: diegocai <diegocai@microsoft.com>
* Add nni_auto_tune example (#325)
* add nni_auto_tune
* support other data format and add more result
* add result picture
* refine readme
* refine readme
* refine code style
* update readme and datareader
* add aml training config
* update config
* fix licence
* refactor for data type
* refactor for data type
* refactor for data type
* fix overflow on bruteforce
* fix compute metric by index
* add limits and preprocessing
* change code dir
Co-authored-by: Guoxin <suiguoxin@gmail.com>
Co-authored-by: MaggieQi <chenqi871025@gmail.com>
* Fixed bugs with accuracy fix for TPT build with recon vectors
* Accuracy fix working for GPU index build with PQ/OPQ enabled
* Fix bug with accuracy fix
* Trigger CI
Co-authored-by: diegocai <diegocai@microsoft.com>
Co-authored-by: smallv0221 <33639025+smallv0221@users.noreply.github.com>
Co-authored-by: Guoxin <suiguoxin@gmail.com>
Co-authored-by: MaggieQi <chenqi871025@gmail.com>
* add nni_auto_tune
* support other data format and add more result
* add result picture
* refine readme
* refine readme
* refine code style
* update readme and datareader
* add aml training config
* update config
* fix licence
* refactor for data type
* refactor for data type
* refactor for data type
* fix overflow on bruteforce
* fix compute metric by index
* add limits and preprocessing
* change code dir
Co-authored-by: Guoxin <suiguoxin@gmail.com>
Co-authored-by: MaggieQi <chenqi871025@gmail.com>
* Fix bug with int8/cosine configuration, and enabled hardware optimization for this case.
* Trigger CI
Co-authored-by: diegocai <diegocai@microsoft.com>
* GPU KNN Intergrate
* Merged cuda hxx files
* Remove extra files, fix CoreLibrary build
* GPU code needs to be in cu file
* break circular including
* Rearranged the method
* Included the files like Neighborhood Graph. h did.
Only included need files in Kernel.cu
* Relocate the Quey_KNN to KNN.hxx (templates cannot be in kernel.cu
* Try to Instantizate before compile
* Fixed compile error. Need GPU to test rest
* Set Mem Failure
* Adjusted the max dimension for vectors up ot 184
* Change DistCalMethod
* Print Thread Start/End
* Shared memory
* Cannot launch Kernelr with 60% MaxSharedMemory
* Root Cause: Transpose_Mem
* L2 480s, Cosine 703s after Transpose
* Two functions for Shared and local ThreadHeap, two launch setting with shared and local DistPair
* Test version for Ben. 32 Threads defined in params.h
query_KNN has transpose and all shared memory.
query_KNN has transpose and heamMem in local.
* Found the debugger issue, next step: fix dist calc
* 1. Fixed int8 dist calc
2. Tested batch splitting on 35M
3. Moved the malloc before launching to avoid waste of memry
* 45cap, monitor mem usage, track where Convert failed
* Succeed on 400M 100D, Fixed int overflow, major change: int to size_t
* Multi-GPU detection
* Fixed the CPU mem over-usage, located the bug in updating results
* Fixed multi free results
* Fixed hard coded Cosine DistCalcMethod
* Relocate the Point & Transposed Point to GPUKNNDistance.hxx
* Merge SPTAG current changes, complaining about cuh cuda lib
* Move Generate Truth to TruthSet.cpp
* Fix optimization issue, add GPUCoreLibrary/GPUSSDServing to default build
* Fix Error for Linux cmake.
* Remove Unnecessary Changes.
* restore datasets
* missing s in folder name
* Restore build config to Lib, restore sln Config
* update tlog for lib config
* Remove Wrappers
* Removed not ignored log info
* Remove Static in TruthSet.cpp
Remove debug command, build GPU SSDServing to exe in Debug
* Recover CL compile for GPU SSDServing/main.cpp
Co-authored-by: diegocai <diegocai@microsoft.com>
Co-authored-by: Philip Adams <philipadams@microsoft.com>
Co-authored-by: Diego-Cai <103398280+Diego-Cai@users.noreply.github.com>
Ben Karsin
dependabot[bot]
Huisheng Liu
zqxjjj
dlmodric
MaggieQi
Alexander Sklar
Philip Adams
microsoft-github-policy-service[bot]
jinwei14
limenghao
yangliu-ms
awesomelewis2007
Renan S
Yuming-Xu
Jie (Diego) Cai
smallv0221
diegocai