* Update .vsts-ci.yml for Azure Pipelines
Upgrade Azure VM Scale Set to use Mariner (Azure Linux) systems.
* Update .vsts-ci.yml
* Update .vsts-ci.yml
* Update .vsts-ci.yml
* Update .vsts-ci.yml
* Update .vsts-ci.yml
* upgrade clang to clang-17
* add -y in apt install
* update name of mariner agent pool
* fix clang 17 compiler name
* apt update before install clang 17
* update libomp version for clang 17
* use clang-17 only for Mariner agents
* fix CXX and CC
* Add integrated OpenCL build on Linux
* Build integrated OpenCL Linux wheel in CI
* Fix test_dual.py on Linux arm64
* Enable integrated OpenCL Linux wheel arm64 testing in CI
* Update documentation
* Add comment about gpu_use_dp
* add missing fi dropped in merge conflict resolution
* install opencl-headers on bdist task
* use new CI image for x86_64
* update check_dynamic_dependencies script
* use main CI image
Co-authored-by: James Lamb <jaylamb20@gmail.com>
* Run OpenCL tests against POCL instead of the AMD App SDK
* Update .ci/setup.sh
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* Run Linux gpu source on default Python version
* [docs] Update GPU Targets Table
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* new cuda framework
* add histogram construction kernel
* before removing multi-gpu
* new cuda framework
* tree learner cuda kernels
* single tree framework ready
* single tree training framework
* remove comments
* boosting with cuda
* optimize for best split find
* data split
* move boosting into cuda
* parallel synchronize best split point
* merge split data kernels
* before code refactor
* use tasks instead of features as units for split finding
* refactor cuda best split finder
* fix configuration error with small leaves in data split
* skip histogram construction of too small leaf
* skip split finding of invalid leaves
stop when no leaf to split
* support row wise with CUDA
* copy data for split by column
* copy data from host to CPU by column for data partition
* add synchronize best splits for one leaf from multiple blocks
* partition dense row data
* fix sync best split from task blocks
* add support for sparse row wise for CUDA
* remove useless code
* add l2 regression objective
* sparse multi value bin enabled for CUDA
* fix cuda ranking objective
* support for number of items <= 2048 per query
* speedup histogram construction by interleaving global memory access
* split optimization
* add cuda tree predictor
* remove comma
* refactor objective and score updater
* before use struct
* use structure for split information
* use structure for leaf splits
* return CUDASplitInfo directly after finding best split
* split with CUDATree directly
* use cuda row data in cuda histogram constructor
* clean src/treelearner/cuda
* gather shared cuda device functions
* put shared CUDA functions into header file
* change smaller leaf from <= back to < for consistent result with CPU
* add tree predictor
* remove useless cuda_tree_predictor
* predict on CUDA with pipeline
* add global sort algorithms
* add global argsort for queries with many items in ranking tasks
* remove limitation of maximum number of items per query in ranking
* add cuda metrics
* fix CUDA AUC
* remove debug code
* add regression metrics
* remove useless file
* don't use mask in shuffle reduce
* add more regression objectives
* fix cuda mape loss
add cuda xentropy loss
* use template for different versions of BitonicArgSortDevice
* add multiclass metrics
* add ndcg metric
* fix cross entropy objectives and metrics
* fix cross entropy and ndcg metrics
* add support for customized objective in CUDA
* complete multiclass ova for CUDA
* separate cuda tree learner
* use shuffle based prefix sum
* clean up cuda_algorithms.hpp
* add copy subset on CUDA
* add bagging for CUDA
* clean up code
* copy gradients from host to device
* support bagging without using subset
* add support of bagging with subset for CUDAColumnData
* add support of bagging with subset for dense CUDARowData
* refactor copy sparse subrow
* use copy subset for column subset
* add reset train data and reset config for CUDA tree learner
add deconstructors for cuda tree learner
* add USE_CUDA ifdef to cuda tree learner files
* check that dataset doesn't contain CUDA tree learner
* remove printf debug information
* use full new cuda tree learner only when using single GPU
* disable all CUDA code when using CPU version
* recover main.cpp
* add cpp files for multi value bins
* update LightGBM.vcxproj
* update LightGBM.vcxproj
fix lint errors
* fix lint errors
* fix lint errors
* update Makevars
fix lint errors
* fix the case with 0 feature and 0 bin
fix split finding for invalid leaves
create cuda column data when loaded from bin file
* fix lint errors
hide GetRowWiseData when cuda is not used
* recover default device type to cpu
* fix na_as_missing case
fix cuda feature meta information
* fix UpdateDataIndexToLeafIndexKernel
* create CUDA trees when needed in CUDADataPartition::UpdateTrainScore
* add refit by tree for cuda tree learner
* fix test_refit in test_engine.py
* create set of large bin partitions in CUDARowData
* add histogram construction for columns with a large number of bins
* add find best split for categorical features on CUDA
* add bitvectors for categorical split
* cuda data partition split for categorical features
* fix split tree with categorical feature
* fix categorical feature splits
* refactor cuda_data_partition.cu with multi-level templates
* refactor CUDABestSplitFinder by grouping task information into struct
* pre-allocate space for vector split_find_tasks_ in CUDABestSplitFinder
* fix misuse of reference
* remove useless changes
* add support for path smoothing
* virtual destructor for LightGBM::Tree
* fix overlapped cat threshold in best split infos
* reset histogram pointers in data partition and spllit finder in ResetConfig
* comment useless parameter
* fix reverse case when na is missing and default bin is zero
* fix mfb_is_na and mfb_is_zero and is_single_feature_column
* remove debug log
* fix cat_l2 when one-hot
fix gradient copy when data subset is used
* switch shared histogram size according to CUDA version
* gpu_use_dp=true when cuda test
* revert modification in config.h
* fix setting of gpu_use_dp=true in .ci/test.sh
* fix linter errors
* fix linter error
remove useless change
* recover main.cpp
* separate cuda_exp and cuda
* fix ci bash scripts
add description for cuda_exp
* add USE_CUDA_EXP flag
* switch off USE_CUDA_EXP
* revert changes in python-packages
* more careful separation for USE_CUDA_EXP
* fix CUDARowData::DivideCUDAFeatureGroups
fix set fields for cuda metadata
* revert config.h
* fix test settings for cuda experimental version
* skip some tests due to unsupported features or differences in implementation details for CUDA Experimental version
* fix lint issue by adding a blank line
* fix lint errors by resorting imports
* fix lint errors by resorting imports
* fix lint errors by resorting imports
* merge cuda.yml and cuda_exp.yml
* update python version in cuda.yml
* remove cuda_exp.yml
* remove unrelated changes
* fix compilation warnings
fix cuda exp ci task name
* recover task
* use multi-level template in histogram construction
check split only in debug mode
* ignore NVCC related lines in parameter_generator.py
* update job name for CUDA tests
* apply review suggestions
* Update .github/workflows/cuda.yml
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* Update .github/workflows/cuda.yml
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* update header
* remove useless TODOs
* remove [TODO(shiyu1994): constrain the split with min_data_in_group] and record in #5062
* #include <LightGBM/utils/log.h> for USE_CUDA_EXP only
* fix include order
* fix include order
* remove extra space
* address review comments
* add warning when cuda_exp is used together with deterministic
* add comment about gpu_use_dp in .ci/test.sh
* revert changing order of included headers
Co-authored-by: Yu Shi <shiyu1994@qq.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* [ci] use conda-forge in CI jobs (fixes#4948)
* comment out more jobs
* try reverting graphviz patch, running more cuda jobs
* get graphviz from PyPI and try removing some patches for r-lintr
* start running appveyor again
* use conda-forge if using conda
* fix commands
* conda install graphviz
* try newer openmp
* pin below openmp 11.x
* focus on gpu task
* trying to narrow down error
* maybe gcc11 is the issue
* start adding other tests back
* pin openmp too
* maybe need to pin to gcc less than 10.x
* pin libgfortran and libstdcxx as well
* pin to gcc 9.3.0
* move constraints up to initial environment
* add all CI jobs back
* try installing python-graphviz separately
* try new lightgbm/vsts-agent image
* fix typo
* test if pinning gcc for linux gpu_source build is still necessary
* ok yes, pinning gcc is necessary
* test if Linux gpu_source works with Python 3.9.6
* no special exception for Linux gpu_source job
* pin to Python 3.9.6 in Linux gpu_source
* try explicitly asking for libstdcxx-ng for every linux build
* swap compilers
* switch compilers back
* revert accidental whitespace change
* comment out CI
* try Linux gpu_source with different Python versions
* Revert "try Linux gpu_source with different Python versions"
This reverts commit f6f63cbb9b.
* Revert "comment out CI"
This reverts commit ece191f01e.
* remove libxml2 install, change CONDA path
* avoid installing conda in rchk job
* empty commit 1
* empty commit 2
* empty commit 3
* empty commit 4
* add more verbose logging around installation of python-graphviz
* empty commit 1
* get mamba info
* get more conda info
* add another mamba info call
* allow for other macOS environments in GHA configuration
* Revert "allow for other macOS environments in GHA configuration"
This reverts commit a3c7a19926.
* get more logs from mamba
* get Build.ArtifactsStagingDirectory
* get more logs and try to force re-installing everything
* clean cache after every step
* remove --update-all and make logs less verbose
* remove more print statements and uncomment jobs
* test if conda-clean issue fixes segfaults for gpu_source
* pin python version for gpu_source
* empty commit 1
* use miniforge instead
* empty commit 1
* Apply suggestions from code review
* bring workarounds back
* remove duplicated graphviz system-wide installation (reverts #4095, #4097, #4238)
* empty commit 1
* empty commit 2
* empty commit 3
* empty commit 4
* empty commit 5
* empty commit 6
* empty commit 7
* empty commit 8
* empty commit 9
* empty commit 10
* empty commit 10
* empty commit 10
* empty commit 10
* empty commit 11
* one more try
* try to downgrade Python version for Linux GPU job
* swap compilers
* Revert "swap compilers"
This reverts commit f04dc27b17.
Co-authored-by: Nikita Titov <nekit94-12@hotmail.com>
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* [ci] move CI jobs from Travis to Azure DevOps (fixes#3519)
* comment out other CIs to avoid wasting cycles
* try without docker
* add container back
* stop using --user in pip install
* run check-docs and lint without container
* job names
* move more jobs to Azure-hosted Linux pool
* fix PATH for check-docs
* uncomment other CI
* uncomment windows
* remove uses of maxParallel
* try moving macos-latest jobs to GitHub Actions
* fix config
* fix missing conda env
* set Python version
* remove commented-out code
* add more to GitHub Actions
* try to fix GPU
* remove static_analysis to prevent conflicts with #3726
* change workflow name
* try using ubuntu:latest docker
* fix conda
* trying to find where permissions first break
* add workaround for sudo
* please azure please
* image syntax
* more sudo
* noninteractive
* LC_ALL
* more sudo
* more stuff
* CONDA dir
* paths
* get path
* missing CONDA
* fix path stuff
* more tests
* fix graphviz
* stuff
* more graphviz
* install xorg-libxau
* graphviz works, run more jobs
* stuff
* enable more tests
* uncomment GitHub Actions
* uncomment all other CIs
* Apply suggestions from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* add travis.yml to Rbuildignore
* add Rbuildignore rule for fmt
* add libomp for clang builds
* changes from code review
Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
* GitHub Actions
* ok
* fixing on list
* stuff
* directories
* directories
* things
* env variables
* working dir
* running a bunch of tasks
* more builds
* PATH
* actually use R task
* TASK
* be right, often
* doing stuff
* trying stuff
* more paths
* conda activate
* updating PATH
* trying bash
* where the hell is activate
* WHERE IS ACTIVATE
* set up conda
* more conda
* PLEASE WORK
* installing cpplint
* try r-package
* R version
* try windows job
* make windows work
* use powershell
* exe
* use conda
* conda init powershell
* different conda approach
* make it work
* cleaning up
* init powershell
* fixing windows
* more windows
* build directory
* no way right
* maybe it will work
* trying Visual Studio
* do this
* Windows is interesting
* put back check-output
* set compiler
* stuff
* more fixes
* fix the broken things
* updating jobs
* continuing
* poweshell is bad
* ok so maybe not powershell
* cmon now
* ok so
* fixing env variables
* maybe this
* MINGW job
* cleaning up
* conda init powershell
* moving more R stuff into GitHub Actions
* everything else
* use powershell
* cmon now powershell
* ttry to Continue
* override powershell
* peg MiKTeX URL
* what is happening
* try powershell -File
* trying stuff
* path
* more testing of output
* Matches uppercase
* more regex stuff
* this is getting ridiculous
* back to powershell I guess
* more commands
* this might work
* adding more reliable miktex download
* trying to download miktex
* installing httr
* fix error in MiKTeX script
* remove comments
* redirect output
* redirect output
* move linting back to Travis
* change redirection
* switch back to just mac and linux
* put linting exclude back
* renamed R_TRAVIS_LINUX
* revert changes to non-R tasks and update events
* simplify
Nikita Titov
James Lamb
Oliver Borchert
shiyu1994
Jonathan Giannuzzi