[R-package] Remove ability to install precompiled lib_lightgbm (fixes #3320) (#3360)

* [R-package] [docs] Reorganize installation instructions (fixes #3320)

* more changes

* remove ability to use precompiled lib_lightgbm

* remove language about installing from the CRAN section

* move installation stuff

* Apply suggestions from code review

Co-authored-by: Nikita Titov <nekit94-08@mail.ru>

* fix anchor

Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
This commit is contained in:
James Lamb 2020-10-11 00:50:41 +01:00 коммит произвёл GitHub
Родитель 3f71d75467
Коммит 785eb60c78
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
2 изменённых файлов: 201 добавлений и 202 удалений

Просмотреть файл

@ -5,22 +5,51 @@
### Contents
* [Installation](#installation)
- [Installing the CRAN Package](#installing-the-cran-package)
- [Installing from Source with CMake](#install)
- [Installing a GPU-enabled Build](#installing-a-gpu-enabled-build)
- [Installing Precompiled Binaries](#installing-precompiled-binaries)
- [Installing from a Pre-compiled lib_lightgbm](#lib_lightgbm)
* [Examples](#examples)
* [Testing](#testing)
* [Preparing a CRAN Package and Installing It](#preparing-a-cran-package-and-installing-it)
* [Preparing a CRAN Package](#preparing-a-cran-package)
* [External Repositories](#external-unofficial-repositories)
* [Known Issues](#known-issues)
Installation
------------
For the easiest installation, go to ["Installing the CRAN package"](#installing-the-cran-package).
If you experience any issues with that, try ["Installing from Source with CMake"](#install). This can produce a more efficient version of the library on Windows systems with Visual Studio.
To build a GPU-enabled version of the package, follow the steps in ["Installing a GPU-enabled Build"](#installing-a-gpu-enabled-build).
If any of the above options do not work for you or do not meet your needs, please let the maintainers know by [opening an issue](https://github.com/microsoft/LightGBM/issues).
When your package installation is done, you can check quickly if your LightGBM R-package is working by running the following:
```r
library(lightgbm)
data(agaricus.train, package='lightgbm')
train <- agaricus.train
dtrain <- lgb.Dataset(train$data, label = train$label)
model <- lgb.cv(
params = list(
objective = "regression"
, metric = "l2"
)
, data = dtrain
)
```
### Installing the CRAN package
As of this writing, `LightGBM`'s R package is not available on CRAN. However, start with `LightGBM` 3.0.0, you can install a released source distribution. This is the same type of package that you'd install from CRAN. It does not require `CMake`, Visual Studio, or anything else outside the CRAN toolchain.
As of this writing, `LightGBM`'s R package is not available on CRAN. However, start with `LightGBM` 3.0.0, you can install a released source distribution. This is the same type of package that you'd install from CRAN. It does not require `CMake`, `Visual Studio`, or anything else outside the CRAN toolchain.
To install this package on any operating system:
1. Choose a release from [the "Releases" page](https://github.com/microsoft/LightGBM/releases)
1. Choose a release from [the "Releases" page](https://github.com/microsoft/LightGBM/releases).
2. Look for the artifact with a name like `lightgbm-{VERSION}-r-cran.tar.gz`. Right-click it and choose "copy link address".
3. Copy that link into `PKG_URL` in the code below and run it.
@ -30,47 +59,31 @@ PKG_URL <- "https://github.com/microsoft/LightGBM/releases/download/v3.0.0/light
remotes::install_url(PKG_URL)
```
### Installing Precompiled Binaries
#### Custom Installation (Linux, Mac)
Starting with `LightGBM` 3.0.0, precompiled binaries for the R package are created for each release. These packages do not require compilation, so they will be faster and easier to install than packages that are built from source. These packages are created with R 4.0 and are not guaranteed to work with other R versions.
The steps above should work on most systems, but users with highly-customized environments might want to change how R builds packages from source.
Binaries are available for Windows, Mac, and Linux systems. They are not guaranteed to work with all variants and versions of these operating systems. Please [open an issue](https://github.com/microsoft/LightGBM/issues) if you encounter any problems.
To change the compiler used when installing the CRAN package, you can create a file `~/.R/Makevars` which overrides `CC` (`C` compiler) and `CXX` (`C++` compiler).
To install a binary for the R package:
For example, to use `gcc` instead of `clang` on Mac, you could use something like the following:
1. Choose a release from [the "Releases" page](https://github.com/microsoft/LightGBM/releases).
2. Choose a file based on your operating system. Right-click it and choose "copy link address".
* Linux: `lightgbm-{VERSION}-r40-linux.tgz`
* Mac: `lightgbm-{VERSION}-r40-macos.tgz`
* Windows: `lightgbm-{VERSION}-r40-windows.zip`
3. Copy that link into `PKG_URL` in the code below and run it.
This sample code installs version 3.0.0-1 of the R package on Mac.
```r
PKG_URL <- "https://github.com/microsoft/LightGBM/releases/download/v3.0.0rc1/lightgbm-3.0.0-1-r40-macos.tgz"
local_file <- paste0("lightgbm.", tools::file_ext(PKG_URL))
download.file(
url = PKG_URL
, destfile = local_file
)
install.packages(
pkgs = local_file
, type = "binary"
, repos = NULL
)
```make
# ~/.R/Makevars
CC=gcc-8
CXX=g++-8
CXX11=g++-8
```
### Preparation
### Installing from Source with CMake <a name="install"></a>
You need to install git and [CMake](https://cmake.org/) first.
Note: 32-bit (i386) R/Rtools is currently not supported.
Note: this method is only supported on 64-bit systems. If you need to run LightGBM on 32-bit Windows (i386), follow the instructions in ["Installing the CRAN Package"](#installing-the-cran-package).
#### Windows Preparation
NOTE: Windows users may need to run with administrator rights (either R or the command prompt, depending on the way you are installing this package).
Installing a 64-bit version of [Rtools](https://cran.r-project.org/bin/windows/Rtools/) is mandatory.
After installing `Rtools` and `CMake`, be sure the following paths are added to the environment variable `PATH`. These may have been automatically added when installing other software.
@ -127,9 +140,9 @@ You can perform installation either with **Apple Clang** or **gcc**. In case you
export CXX=/usr/local/bin/g++-8 CC=/usr/local/bin/gcc-8
```
### Install
#### Install with CMake
Build and install R-package with the following commands:
After following the "preparation" steps above for your operating system, build and install the R-package with the following commands:
```sh
git clone --recursive https://github.com/microsoft/LightGBM
@ -141,23 +154,53 @@ The `build_r.R` script builds the package in a temporary directory called `light
Note: for the build with Visual Studio/VS Build Tools in Windows, you should use the Windows CMD or Powershell.
Windows users may need to run with administrator rights (either R or the command prompt, depending on the way you are installing this package). Linux users might require the appropriate user write permissions for packages.
### Installing a GPU-enabled Build
Set `use_gpu` to `TRUE` in `R-package/src/install.libs.R` to enable the build with GPU support. You will need to install Boost and OpenCL first: details for installation can be found in [Installation-Guide](https://github.com/microsoft/LightGBM/blob/master/docs/Installation-Guide.rst#build-gpu-version).
If you are using a precompiled dll/lib locally, you can move the dll/lib into LightGBM root folder, modify `LightGBM/R-package/src/install.libs.R`'s 2nd line (change `use_precompile <- FALSE` to `use_precompile <- TRUE`), and install R-package as usual.
After installing these other libraries, follow the steps in ["Installing from Source with CMake"](#install).
When your package installation is done, you can check quickly if your LightGBM R-package is working by running the following:
### Installing Precompiled Binaries
**NOTE:** As of this writing, the precompiled binaries of the R package should be considered experimental. If you try them an experience any problems, please [open an issue](https://github.com/microsoft/LightGBM/issues).
Starting with `LightGBM` 3.0.0, precompiled binaries for the R package are created for each release. These packages do not require compilation, so they will be faster and easier to install than packages that are built from source. These packages are created with R 4.0 and are not guaranteed to work with other R versions.
Binaries are available for Windows, Mac, and Linux systems. They are not guaranteed to work with all variants and versions of these operating systems. Please [open an issue](https://github.com/microsoft/LightGBM/issues) if you encounter any problems.
To install a binary for the R package:
1. Choose a release from [the "Releases" page](https://github.com/microsoft/LightGBM/releases).
2. Choose a file based on your operating system. Right-click it and choose "copy link address".
* Linux: `lightgbm-{VERSION}-r40-linux.tgz`
* Mac: `lightgbm-{VERSION}-r40-macos.tgz`
* Windows: `lightgbm-{VERSION}-r40-windows.zip`
3. Copy that link into `PKG_URL` in the code below and run it.
This sample code installs version 3.0.0-1 of the R package on Mac.
```r
library(lightgbm)
data(agaricus.train, package='lightgbm')
train <- agaricus.train
dtrain <- lgb.Dataset(train$data, label=train$label)
params <- list(objective="regression", metric="l2")
model <- lgb.cv(params, dtrain, 10, nfold=5, min_data=1, learning_rate=1, early_stopping_rounds=10)
PKG_URL <- "https://github.com/microsoft/LightGBM/releases/download/v3.0.0rc1/lightgbm-3.0.0-1-r40-macos.tgz"
local_file <- paste0("lightgbm.", tools::file_ext(PKG_URL))
download.file(
url = PKG_URL
, destfile = local_file
)
install.packages(
pkgs = local_file
, type = "binary"
, repos = NULL
)
```
### Installing from a Pre-compiled lib_lightgbm <a name="lib_lightgbm"></a>
Previous versions of LightGBM offered the ability to first compile the C++ library (`lib_lightgbm.so` or `lib_lightgbm.dll`) and then build an R package that wraps it.
As of version 3.0.0, this is no longer supported. If building from source is difficult for you, please [open an issue](https://github.com/microsoft/LightGBM/issues).
Examples
--------
@ -194,8 +237,8 @@ Rscript -e " \
"
```
Preparing a CRAN Package and Installing It
------------------------------------------
Preparing a CRAN Package
------------------------
This section is primarily for maintainers, but may help users and contributors to understand the structure of the R package.
@ -221,17 +264,6 @@ After building the package, install it with a command like the following:
R CMD install lightgbm_*.tar.gz
```
#### Custom Installation (Linux, Mac)
To change the compiler used when installing the package, you can create a file `~/.R/Makevars` which overrides `CC` (`C` compiler) and `CXX` (`C++` compiler). For example, to use `gcc` instead of `clang` on Mac, you could use something like the following:
```make
# ~/.R/Makevars
CC=gcc-8
CXX=g++-8
CXX11=g++-8
```
### Changing the CRAN Package
A lot of details are handled automatically by `R CMD build` and `R CMD install`, so it can be difficult to understand how the files in the R package are related to each other. An extensive treatment of those details is available in ["Writing R Extensions"](https://cran.r-project.org/doc/manuals/r-release/R-exts.html).

Просмотреть файл

@ -1,5 +1,4 @@
# User options
use_precompile <- FALSE
use_gpu <- FALSE
# For Windows, the package will be built with Visual Studio
@ -107,176 +106,144 @@ if (!write_succeeded) {
source_dir <- file.path(R_PACKAGE_SOURCE, "src", fsep = "/")
build_dir <- file.path(source_dir, "build", fsep = "/")
# Check for precompilation
if (!use_precompile) {
# Prepare building package
dir.create(
build_dir
, recursive = TRUE
, showWarnings = FALSE
)
setwd(build_dir)
# Prepare building package
dir.create(
build_dir
, recursive = TRUE
, showWarnings = FALSE
use_visual_studio <- !(use_mingw || use_msys2)
# If using MSVC to build, pull in the script used
# to create R.def from R.dll
if (WINDOWS && use_visual_studio) {
write_succeeded <- file.copy(
"../../inst/make-r-def.R"
, file.path(build_dir, "make-r-def.R")
, overwrite = TRUE
)
setwd(build_dir)
use_visual_studio <- !(use_mingw || use_msys2)
# If using MSVC to build, pull in the script used
# to create R.def from R.dll
if (WINDOWS && use_visual_studio) {
write_succeeded <- file.copy(
"../../inst/make-r-def.R"
, file.path(build_dir, "make-r-def.R")
, overwrite = TRUE
)
if (!write_succeeded) {
stop("Copying make-r-def.R failed")
}
if (!write_succeeded) {
stop("Copying make-r-def.R failed")
}
}
# Prepare installation steps
cmake_args <- NULL
build_cmd <- "make"
build_args <- "_lightgbm"
lib_folder <- file.path(source_dir, fsep = "/")
# Prepare installation steps
cmake_args <- NULL
build_cmd <- "make"
build_args <- "_lightgbm"
lib_folder <- file.path(source_dir, fsep = "/")
WINDOWS_BUILD_TOOLS <- list(
"MinGW" = c(
build_tool = "mingw32-make.exe"
, makefile_generator = "MinGW Makefiles"
)
, "MSYS2" = c(
build_tool = "make.exe"
, makefile_generator = "MSYS Makefiles"
)
WINDOWS_BUILD_TOOLS <- list(
"MinGW" = c(
build_tool = "mingw32-make.exe"
, makefile_generator = "MinGW Makefiles"
)
, "MSYS2" = c(
build_tool = "make.exe"
, makefile_generator = "MSYS Makefiles"
)
)
if (use_mingw) {
windows_toolchain <- "MinGW"
} else if (use_msys2) {
if (use_mingw) {
windows_toolchain <- "MinGW"
} else if (use_msys2) {
windows_toolchain <- "MSYS2"
} else {
# Rtools 4.0 moved from MinGW to MSYS toolchain. If user tries
# Visual Studio install but that fails, fall back to the toolchain
# supported in Rtools
if (R_ver >= 4.0) {
windows_toolchain <- "MSYS2"
} else {
# Rtools 4.0 moved from MinGW to MSYS toolchain. If user tries
# Visual Studio install but that fails, fall back to the toolchain
# supported in Rtools
if (R_ver >= 4.0) {
windows_toolchain <- "MSYS2"
} else {
windows_toolchain <- "MinGW"
}
windows_toolchain <- "MinGW"
}
windows_build_tool <- WINDOWS_BUILD_TOOLS[[windows_toolchain]][["build_tool"]]
windows_makefile_generator <- WINDOWS_BUILD_TOOLS[[windows_toolchain]][["makefile_generator"]]
}
windows_build_tool <- WINDOWS_BUILD_TOOLS[[windows_toolchain]][["build_tool"]]
windows_makefile_generator <- WINDOWS_BUILD_TOOLS[[windows_toolchain]][["makefile_generator"]]
if (use_gpu) {
cmake_args <- c(cmake_args, "-DUSE_GPU=ON")
}
cmake_args <- c(cmake_args, "-D__BUILD_FOR_R=ON")
if (use_gpu) {
cmake_args <- c(cmake_args, "-DUSE_GPU=ON")
}
cmake_args <- c(cmake_args, "-D__BUILD_FOR_R=ON")
# Pass in R version, used to help find R executable for linking
R_version_string <- paste(
R.Version()[["major"]]
, R.Version()[["minor"]]
, sep = "."
)
r_version_arg <- sprintf("-DCMAKE_R_VERSION='%s'", R_version_string)
cmake_args <- c(cmake_args, r_version_arg)
# Pass in R version, used to help find R executable for linking
R_version_string <- paste(
R.Version()[["major"]]
, R.Version()[["minor"]]
, sep = "."
)
r_version_arg <- sprintf("-DCMAKE_R_VERSION='%s'", R_version_string)
cmake_args <- c(cmake_args, r_version_arg)
# the checks below might already run `cmake -G`. If they do, set this flag
# to TRUE to avoid re-running it later
makefiles_already_generated <- FALSE
# the checks below might already run `cmake -G`. If they do, set this flag
# to TRUE to avoid re-running it later
makefiles_already_generated <- FALSE
# Check if Windows installation (for gcc vs Visual Studio)
if (WINDOWS) {
if (!use_visual_studio) {
message(sprintf("Trying to build with %s", windows_toolchain))
# Check if Windows installation (for gcc vs Visual Studio)
if (WINDOWS) {
if (!use_visual_studio) {
message(sprintf("Trying to build with %s", windows_toolchain))
# Must build twice for Windows due sh.exe in Rtools
cmake_args <- c(cmake_args, "-G", shQuote(windows_makefile_generator))
.run_shell_command("cmake", c(cmake_args, ".."), strict = FALSE)
build_cmd <- windows_build_tool
build_args <- "_lightgbm"
} else {
visual_studio_succeeded <- .generate_vs_makefiles(cmake_args)
if (!isTRUE(visual_studio_succeeded)) {
warning(sprintf("Building with Visual Studio failed. Attempting with %s", windows_toolchain))
# Must build twice for Windows due sh.exe in Rtools
cmake_args <- c(cmake_args, "-G", shQuote(windows_makefile_generator))
.run_shell_command("cmake", c(cmake_args, ".."), strict = FALSE)
build_cmd <- windows_build_tool
build_args <- "_lightgbm"
} else {
visual_studio_succeeded <- .generate_vs_makefiles(cmake_args)
if (!isTRUE(visual_studio_succeeded)) {
warning(sprintf("Building with Visual Studio failed. Attempting with %s", windows_toolchain))
# Must build twice for Windows due sh.exe in Rtools
cmake_args <- c(cmake_args, "-G", shQuote(windows_makefile_generator))
.run_shell_command("cmake", c(cmake_args, ".."), strict = FALSE)
build_cmd <- windows_build_tool
build_args <- "_lightgbm"
} else {
build_cmd <- "cmake"
build_args <- c("--build", ".", "--target", "_lightgbm", "--config", "Release")
lib_folder <- file.path(source_dir, "Release", fsep = "/")
makefiles_already_generated <- TRUE
}
}
} else {
.run_shell_command("cmake", c(cmake_args, ".."))
build_cmd <- "cmake"
build_args <- c("--build", ".", "--target", "_lightgbm", "--config", "Release")
lib_folder <- file.path(source_dir, "Release", fsep = "/")
makefiles_already_generated <- TRUE
}
}
# generate build files
if (!makefiles_already_generated) {
.run_shell_command("cmake", c(cmake_args, ".."))
}
# R CMD check complains about the .NOTPARALLEL directive created in the cmake
# Makefile. We don't need it here anyway since targets are built serially, so trying
# to remove it with this hack
generated_makefile <- file.path(
build_dir
, "Makefile"
)
if (file.exists(generated_makefile)) {
makefile_txt <- readLines(
con = generated_makefile
)
makefile_txt <- gsub(
pattern = ".*NOTPARALLEL.*"
, replacement = ""
, x = makefile_txt
)
writeLines(
text = makefile_txt
, con = generated_makefile
, sep = "\n"
)
}
# build the library
message("Building lib_lightgbm")
.run_shell_command(build_cmd, build_args)
src <- file.path(lib_folder, paste0("lib_lightgbm", SHLIB_EXT), fsep = "/")
} else {
# Has precompiled package
lib_folder <- file.path(R_PACKAGE_SOURCE, "../", fsep = "/")
shared_object_file <- file.path(
lib_folder
, paste0("lib_lightgbm", SHLIB_EXT)
, fsep = "/"
)
release_file <- file.path(
lib_folder
, paste0("Release/lib_lightgbm", SHLIB_EXT)
, fsep = "/"
)
windows_shared_object_file <- file.path(
lib_folder
, paste0("/windows/x64/DLL/lib_lightgbm", SHLIB_EXT)
, fsep = "/"
)
if (file.exists(shared_object_file)) {
src <- shared_object_file
} else if (file.exists(release_file)) {
src <- release_file
} else {
# Expected result: installation will fail if it is not here or any other
src <- windows_shared_object_file
}
.run_shell_command("cmake", c(cmake_args, ".."))
makefiles_already_generated <- TRUE
}
# generate build files
if (!makefiles_already_generated) {
.run_shell_command("cmake", c(cmake_args, ".."))
}
# R CMD check complains about the .NOTPARALLEL directive created in the cmake
# Makefile. We don't need it here anyway since targets are built serially, so trying
# to remove it with this hack
generated_makefile <- file.path(
build_dir
, "Makefile"
)
if (file.exists(generated_makefile)) {
makefile_txt <- readLines(
con = generated_makefile
)
makefile_txt <- gsub(
pattern = ".*NOTPARALLEL.*"
, replacement = ""
, x = makefile_txt
)
writeLines(
text = makefile_txt
, con = generated_makefile
, sep = "\n"
)
}
# build the library
message("Building lib_lightgbm")
.run_shell_command(build_cmd, build_args)
src <- file.path(lib_folder, paste0("lib_lightgbm", SHLIB_EXT), fsep = "/")
# Packages with install.libs.R need to copy some artifacts into the
# expected places in the package structure.
# see https://cran.r-project.org/doc/manuals/r-devel/R-exts.html#Package-subdirectories,