* Update .vsts-ci.yml for Azure Pipelines
Upgrade Azure VM Scale Set to use Mariner (Azure Linux) systems.
* Update .vsts-ci.yml
* Update .vsts-ci.yml
* Update .vsts-ci.yml
* Update .vsts-ci.yml
* Update .vsts-ci.yml
* upgrade clang to clang-17
* add -y in apt install
* update name of mariner agent pool
* fix clang 17 compiler name
* apt update before install clang 17
* update libomp version for clang 17
* use clang-17 only for Mariner agents
* fix CXX and CC
* add quantized training (first stage)
* add histogram construction functions for integer gradients
* add stochastic rounding
* update docs
* fix compilation errors by adding template instantiations
* update files for compilation
* fix compilation of gpu version
* initialize gradient discretizer before share states
* add a test case for quantized training
* add quantized training for data distributed training
* Delete origin.pred
* Delete ifelse.pred
* Delete LightGBM_model.txt
* remove useless changes
* fix lint error
* remove debug loggings
* fix mismatch of vector and allocator types
* remove changes in main.cpp
* fix bugs with uninitialized gradient discretizer
* initialize ordered gradients in gradient discretizer
* disable quantized training with gpu and cuda
fix msvc compilation errors and warnings
* fix bug in data parallel tree learner
* make quantized training test deterministic
* make quantized training in test case more accurate
* refactor test_quantized_training
* fix leaf splits initialization with quantized training
* check distributed quantized training result
* add cuda gradient discretizer
* add quantized training for CUDA version in tree learner
* remove cuda computability 6.1 and 6.2
* fix parts of gpu quantized training errors and warnings
* fix build-python.sh to install locally built version
* fix memory access bugs
* fix lint errors
* mark cuda quantized training on cuda with categorical features as unsupported
* rename cuda_utils.h to cuda_utils.hu
* enable quantized training with cuda
* fix cuda quantized training with sparse row data
* allow using global memory buffer in histogram construction with cuda quantized training
* recover build-python.sh
enlarge allowed package size to 100M
* Add integrated OpenCL build on Linux
* Build integrated OpenCL Linux wheel in CI
* Fix test_dual.py on Linux arm64
* Enable integrated OpenCL Linux wheel arm64 testing in CI
* Update documentation
* Add comment about gpu_use_dp
* add missing fi dropped in merge conflict resolution
* install opencl-headers on bdist task
* use new CI image for x86_64
* update check_dynamic_dependencies script
* use main CI image
Co-authored-by: James Lamb <jaylamb20@gmail.com>
* Run OpenCL tests against POCL instead of the AMD App SDK
* Update .ci/setup.sh
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* Run Linux gpu source on default Python version
* [docs] Update GPU Targets Table
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* new cuda framework
* add histogram construction kernel
* before removing multi-gpu
* new cuda framework
* tree learner cuda kernels
* single tree framework ready
* single tree training framework
* remove comments
* boosting with cuda
* optimize for best split find
* data split
* move boosting into cuda
* parallel synchronize best split point
* merge split data kernels
* before code refactor
* use tasks instead of features as units for split finding
* refactor cuda best split finder
* fix configuration error with small leaves in data split
* skip histogram construction of too small leaf
* skip split finding of invalid leaves
stop when no leaf to split
* support row wise with CUDA
* copy data for split by column
* copy data from host to CPU by column for data partition
* add synchronize best splits for one leaf from multiple blocks
* partition dense row data
* fix sync best split from task blocks
* add support for sparse row wise for CUDA
* remove useless code
* add l2 regression objective
* sparse multi value bin enabled for CUDA
* fix cuda ranking objective
* support for number of items <= 2048 per query
* speedup histogram construction by interleaving global memory access
* split optimization
* add cuda tree predictor
* remove comma
* refactor objective and score updater
* before use struct
* use structure for split information
* use structure for leaf splits
* return CUDASplitInfo directly after finding best split
* split with CUDATree directly
* use cuda row data in cuda histogram constructor
* clean src/treelearner/cuda
* gather shared cuda device functions
* put shared CUDA functions into header file
* change smaller leaf from <= back to < for consistent result with CPU
* add tree predictor
* remove useless cuda_tree_predictor
* predict on CUDA with pipeline
* add global sort algorithms
* add global argsort for queries with many items in ranking tasks
* remove limitation of maximum number of items per query in ranking
* add cuda metrics
* fix CUDA AUC
* remove debug code
* add regression metrics
* remove useless file
* don't use mask in shuffle reduce
* add more regression objectives
* fix cuda mape loss
add cuda xentropy loss
* use template for different versions of BitonicArgSortDevice
* add multiclass metrics
* add ndcg metric
* fix cross entropy objectives and metrics
* fix cross entropy and ndcg metrics
* add support for customized objective in CUDA
* complete multiclass ova for CUDA
* separate cuda tree learner
* use shuffle based prefix sum
* clean up cuda_algorithms.hpp
* add copy subset on CUDA
* add bagging for CUDA
* clean up code
* copy gradients from host to device
* support bagging without using subset
* add support of bagging with subset for CUDAColumnData
* add support of bagging with subset for dense CUDARowData
* refactor copy sparse subrow
* use copy subset for column subset
* add reset train data and reset config for CUDA tree learner
add deconstructors for cuda tree learner
* add USE_CUDA ifdef to cuda tree learner files
* check that dataset doesn't contain CUDA tree learner
* remove printf debug information
* use full new cuda tree learner only when using single GPU
* disable all CUDA code when using CPU version
* recover main.cpp
* add cpp files for multi value bins
* update LightGBM.vcxproj
* update LightGBM.vcxproj
fix lint errors
* fix lint errors
* fix lint errors
* update Makevars
fix lint errors
* fix the case with 0 feature and 0 bin
fix split finding for invalid leaves
create cuda column data when loaded from bin file
* fix lint errors
hide GetRowWiseData when cuda is not used
* recover default device type to cpu
* fix na_as_missing case
fix cuda feature meta information
* fix UpdateDataIndexToLeafIndexKernel
* create CUDA trees when needed in CUDADataPartition::UpdateTrainScore
* add refit by tree for cuda tree learner
* fix test_refit in test_engine.py
* create set of large bin partitions in CUDARowData
* add histogram construction for columns with a large number of bins
* add find best split for categorical features on CUDA
* add bitvectors for categorical split
* cuda data partition split for categorical features
* fix split tree with categorical feature
* fix categorical feature splits
* refactor cuda_data_partition.cu with multi-level templates
* refactor CUDABestSplitFinder by grouping task information into struct
* pre-allocate space for vector split_find_tasks_ in CUDABestSplitFinder
* fix misuse of reference
* remove useless changes
* add support for path smoothing
* virtual destructor for LightGBM::Tree
* fix overlapped cat threshold in best split infos
* reset histogram pointers in data partition and spllit finder in ResetConfig
* comment useless parameter
* fix reverse case when na is missing and default bin is zero
* fix mfb_is_na and mfb_is_zero and is_single_feature_column
* remove debug log
* fix cat_l2 when one-hot
fix gradient copy when data subset is used
* switch shared histogram size according to CUDA version
* gpu_use_dp=true when cuda test
* revert modification in config.h
* fix setting of gpu_use_dp=true in .ci/test.sh
* fix linter errors
* fix linter error
remove useless change
* recover main.cpp
* separate cuda_exp and cuda
* fix ci bash scripts
add description for cuda_exp
* add USE_CUDA_EXP flag
* switch off USE_CUDA_EXP
* revert changes in python-packages
* more careful separation for USE_CUDA_EXP
* fix CUDARowData::DivideCUDAFeatureGroups
fix set fields for cuda metadata
* revert config.h
* fix test settings for cuda experimental version
* skip some tests due to unsupported features or differences in implementation details for CUDA Experimental version
* fix lint issue by adding a blank line
* fix lint errors by resorting imports
* fix lint errors by resorting imports
* fix lint errors by resorting imports
* merge cuda.yml and cuda_exp.yml
* update python version in cuda.yml
* remove cuda_exp.yml
* remove unrelated changes
* fix compilation warnings
fix cuda exp ci task name
* recover task
* use multi-level template in histogram construction
check split only in debug mode
* ignore NVCC related lines in parameter_generator.py
* update job name for CUDA tests
* apply review suggestions
* Update .github/workflows/cuda.yml
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* Update .github/workflows/cuda.yml
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* update header
* remove useless TODOs
* remove [TODO(shiyu1994): constrain the split with min_data_in_group] and record in #5062
* #include <LightGBM/utils/log.h> for USE_CUDA_EXP only
* fix include order
* fix include order
* remove extra space
* address review comments
* add warning when cuda_exp is used together with deterministic
* add comment about gpu_use_dp in .ci/test.sh
* revert changing order of included headers
Co-authored-by: Yu Shi <shiyu1994@qq.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* [ci] use conda-forge in CI jobs (fixes#4948)
* comment out more jobs
* try reverting graphviz patch, running more cuda jobs
* get graphviz from PyPI and try removing some patches for r-lintr
* start running appveyor again
* use conda-forge if using conda
* fix commands
* conda install graphviz
* try newer openmp
* pin below openmp 11.x
* focus on gpu task
* trying to narrow down error
* maybe gcc11 is the issue
* start adding other tests back
* pin openmp too
* maybe need to pin to gcc less than 10.x
* pin libgfortran and libstdcxx as well
* pin to gcc 9.3.0
* move constraints up to initial environment
* add all CI jobs back
* try installing python-graphviz separately
* try new lightgbm/vsts-agent image
* fix typo
* test if pinning gcc for linux gpu_source build is still necessary
* ok yes, pinning gcc is necessary
* test if Linux gpu_source works with Python 3.9.6
* no special exception for Linux gpu_source job
* pin to Python 3.9.6 in Linux gpu_source
* try explicitly asking for libstdcxx-ng for every linux build
* swap compilers
* switch compilers back
* revert accidental whitespace change
* comment out CI
* try Linux gpu_source with different Python versions
* Revert "try Linux gpu_source with different Python versions"
This reverts commit f6f63cbb9b.
* Revert "comment out CI"
This reverts commit ece191f01e.
* remove libxml2 install, change CONDA path
* avoid installing conda in rchk job
* empty commit 1
* empty commit 2
* empty commit 3
* empty commit 4
* add more verbose logging around installation of python-graphviz
* empty commit 1
* get mamba info
* get more conda info
* add another mamba info call
* allow for other macOS environments in GHA configuration
* Revert "allow for other macOS environments in GHA configuration"
This reverts commit a3c7a19926.
* get more logs from mamba
* get Build.ArtifactsStagingDirectory
* get more logs and try to force re-installing everything
* clean cache after every step
* remove --update-all and make logs less verbose
* remove more print statements and uncomment jobs
* test if conda-clean issue fixes segfaults for gpu_source
* pin python version for gpu_source
* empty commit 1
* use miniforge instead
* empty commit 1
* Apply suggestions from code review
* bring workarounds back
* remove duplicated graphviz system-wide installation (reverts #4095, #4097, #4238)
* empty commit 1
* empty commit 2
* empty commit 3
* empty commit 4
* empty commit 5
* empty commit 6
* empty commit 7
* empty commit 8
* empty commit 9
* empty commit 10
* empty commit 10
* empty commit 10
* empty commit 10
* empty commit 11
* one more try
* try to downgrade Python version for Linux GPU job
* swap compilers
* Revert "swap compilers"
This reverts commit f04dc27b17.
Co-authored-by: Nikita Titov <nekit94-12@hotmail.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* [ci] manually create symlinks to R entrypoints on macOS (fixes#4988)
* exclude non-R CI jobs
* upgrade to R 4.1.2
* get logs for R install
* pin R 4.1.x jobs to macOS-10.15
* update to R 4.1.2 on Windows
* allow for non-latest macOS builds in GHA configuration
* fix prefix check
* fix config check
* more direct check for mac version
* uncomment other CIs
* update R version in CI job names
shiyu1994
James Lamb
david-cortes
Oliver Borchert
Nikita Titov
Jonathan Giannuzzi
CuriousCorrelation
Saumya
Antoni Baum