LightGBM/.ci/test-windows.ps1

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

143 строки
6.4 KiB
PowerShell
Исходник Обычный вид История

function Check-Output {
param( [bool]$success )
if (!$success) {
$host.SetShouldExit(-1)
exit 1
}
}
$env:CONDA_ENV = "test-env"
$env:LGB_VER = (Get-Content $env:BUILD_SOURCESDIRECTORY\VERSION.txt).trim()
# Use custom temp directory to avoid
# > warning MSB8029: The Intermediate directory or Output directory cannot reside under the Temporary directory
# > as it could lead to issues with incremental build.
# And make sure this directory is always clean
$env:TMPDIR = "$env:USERPROFILE\tmp"
Remove-Item $env:TMPDIR -Force -Recurse -ErrorAction Ignore
[Void][System.IO.Directory]::CreateDirectory($env:TMPDIR)
[R-package] [ci] Add Windows CI for R package (fixes #2335) (#2936) * first test of appveyor * strings are strings * lil bit of build script * fixing paths * removed unnecessary file * updated CRAN URL * added a lot more printing * fixing paths * more stuff * fixed paths * more stuff * more path guessing * even more paths * more stuff * moar logz * and now for something totally different * please work * ok could be a thing * changing directories * we might be in business * fixed install syntax * tryinv mingw * more mingw * ignore Suggests check * trying Azure DevOps * just run bare minimum for Azure DevOps * fixed build dir thing * trying to set libPaths * more testing * trying R 3.6.3 * R 3.6.3 * this feels right * still messing around with libraries * more paths * removed duplication in Windows testing code * simpler * fixed conda stuff * more conda stuff * more fixes * fixed testing script * moved AppVeyor setup to the top * commenting * ch-ch-ch-ch-chaaaanges * paths * plz work * fixed conda stuff in Windows CI * uncommented stuff to test a full build * fixed quotes and removed some unnecessary stuff * added install.libs.R change * quotes are impoortant * added commented-out stuff back in * added Windows script, download retries, and MSVC linking * minor fixes * cleaned up debugging code in FindLibR * cleaned up debugging code and moved R first in CI * fixed vsts-ci indentation * cut documentation stuff out of MSVC build * fix R CMD check for Azure * misc whitespace changes * Added echoing of build logs from R CMD check * cut out more documentation tests * fixed NOTE about imports from Matrix * moved some changes out of this PR and into #2963 * fixed whitespace stuff * added check on number of NOTES * adding better checks * fixing check on NOTEs * removing unnecessary variable * Update .ci/test_r_package_windows.ps1 Co-Authored-By: Nikita Titov <nekit94-08@mail.ru> * some changes * fix quoting * trying MINGW on Azure DevOps * fixing paths * more paths * fixing paths * testing paths * fixing slashes * pinned CTAN mirror * get better logs * made sure Azure finds MinGW, fixed search for LIBR_CORE_LIBRARY, stopped building R docs on Azure * Apply suggestions from code review Co-Authored-By: Nikita Titov <nekit94-08@mail.ru> * added CXX, CC for Windows builds and changed back to building docs on all MINGW builds * stored LIBR_CORE_LIBRARY hints in one variable * Apply suggestions from code review Co-Authored-By: Nikita Titov <nekit94-08@mail.ru> * changes from code review * increased parallel builds for Azure CI * Apply suggestions from code review Co-Authored-By: Nikita Titov <nekit94-08@mail.ru> Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
2020-04-27 00:05:45 +03:00
if ($env:TASK -eq "r-package") {
& .\.ci\test-r-package-windows.ps1 ; Check-Output $?
[R-package] [ci] Add Windows CI for R package (fixes #2335) (#2936) * first test of appveyor * strings are strings * lil bit of build script * fixing paths * removed unnecessary file * updated CRAN URL * added a lot more printing * fixing paths * more stuff * fixed paths * more stuff * more path guessing * even more paths * more stuff * moar logz * and now for something totally different * please work * ok could be a thing * changing directories * we might be in business * fixed install syntax * tryinv mingw * more mingw * ignore Suggests check * trying Azure DevOps * just run bare minimum for Azure DevOps * fixed build dir thing * trying to set libPaths * more testing * trying R 3.6.3 * R 3.6.3 * this feels right * still messing around with libraries * more paths * removed duplication in Windows testing code * simpler * fixed conda stuff * more conda stuff * more fixes * fixed testing script * moved AppVeyor setup to the top * commenting * ch-ch-ch-ch-chaaaanges * paths * plz work * fixed conda stuff in Windows CI * uncommented stuff to test a full build * fixed quotes and removed some unnecessary stuff * added install.libs.R change * quotes are impoortant * added commented-out stuff back in * added Windows script, download retries, and MSVC linking * minor fixes * cleaned up debugging code in FindLibR * cleaned up debugging code and moved R first in CI * fixed vsts-ci indentation * cut documentation stuff out of MSVC build * fix R CMD check for Azure * misc whitespace changes * Added echoing of build logs from R CMD check * cut out more documentation tests * fixed NOTE about imports from Matrix * moved some changes out of this PR and into #2963 * fixed whitespace stuff * added check on number of NOTES * adding better checks * fixing check on NOTEs * removing unnecessary variable * Update .ci/test_r_package_windows.ps1 Co-Authored-By: Nikita Titov <nekit94-08@mail.ru> * some changes * fix quoting * trying MINGW on Azure DevOps * fixing paths * more paths * fixing paths * testing paths * fixing slashes * pinned CTAN mirror * get better logs * made sure Azure finds MinGW, fixed search for LIBR_CORE_LIBRARY, stopped building R docs on Azure * Apply suggestions from code review Co-Authored-By: Nikita Titov <nekit94-08@mail.ru> * added CXX, CC for Windows builds and changed back to building docs on all MINGW builds * stored LIBR_CORE_LIBRARY hints in one variable * Apply suggestions from code review Co-Authored-By: Nikita Titov <nekit94-08@mail.ru> * changes from code review * increased parallel builds for Azure CI * Apply suggestions from code review Co-Authored-By: Nikita Titov <nekit94-08@mail.ru> Co-authored-by: Nikita Titov <nekit94-08@mail.ru>
2020-04-27 00:05:45 +03:00
Exit 0
}
if ($env:TASK -eq "cpp-tests") {
cmake -B build -S . -DBUILD_CPP_TEST=ON -DUSE_OPENMP=OFF -DUSE_DEBUG=ON -A x64
cmake --build build --target testlightgbm --config Debug ; Check-Output $?
.\Debug\testlightgbm.exe ; Check-Output $?
Exit 0
}
if ($env:TASK -eq "swig") {
$env:JAVA_HOME = $env:JAVA_HOME_8_X64 # there is pre-installed Eclipse Temurin 8 somewhere
$ProgressPreference = "SilentlyContinue" # progress bar bug extremely slows down download speed
Invoke-WebRequest -Uri "https://sourceforge.net/projects/swig/files/latest/download" -OutFile $env:BUILD_SOURCESDIRECTORY/swig/swigwin.zip -UserAgent "curl"
Add-Type -AssemblyName System.IO.Compression.FileSystem
[System.IO.Compression.ZipFile]::ExtractToDirectory("$env:BUILD_SOURCESDIRECTORY/swig/swigwin.zip", "$env:BUILD_SOURCESDIRECTORY/swig") ; Check-Output $?
$SwigFolder = Get-ChildItem -Directory -Name -Path "$env:BUILD_SOURCESDIRECTORY/swig"
$env:PATH = "$env:BUILD_SOURCESDIRECTORY/swig/$SwigFolder;" + $env:PATH
$BuildLogFileName = "$env:BUILD_SOURCESDIRECTORY\cmake_build.log"
cmake -B build -S . -A x64 -DUSE_SWIG=ON *> "$BuildLogFileName" ; $build_succeeded = $?
Write-Output "CMake build logs:"
Get-Content -Path "$BuildLogFileName"
Check-Output $build_succeeded
$checks = Select-String -Path "${BuildLogFileName}" -Pattern "-- Found SWIG.*${SwigFolder}/swig.exe"
$checks_cnt = $checks.Matches.length
if ($checks_cnt -eq 0) {
Write-Output "Wrong SWIG version was found (expected '${SwigFolder}'). Check the build logs."
Check-Output $False
}
cmake --build build --target ALL_BUILD --config Release ; Check-Output $?
if ($env:AZURE -eq "true") {
cp ./build/lightgbmlib.jar $env:BUILD_ARTIFACTSTAGINGDIRECTORY/lightgbmlib_win.jar ; Check-Output $?
}
Exit 0
}
# setup for Python
conda init powershell
conda activate
conda config --set always_yes yes --set changeps1 no
conda update -q -y conda "python=$env:PYTHON_VERSION[build=*cpython]"
if ($env:PYTHON_VERSION -eq "3.7") {
$env:CONDA_REQUIREMENT_FILE = "$env:BUILD_SOURCESDIRECTORY/.ci/conda-envs/ci-core-py37.txt"
} elseif ($env:PYTHON_VERSION -eq "3.8") {
$env:CONDA_REQUIREMENT_FILE = "$env:BUILD_SOURCESDIRECTORY/.ci/conda-envs/ci-core-py38.txt"
} else {
$env:CONDA_REQUIREMENT_FILE = "$env:BUILD_SOURCESDIRECTORY/.ci/conda-envs/ci-core.txt"
}
conda create `
-y `
-n $env:CONDA_ENV `
--file $env:CONDA_REQUIREMENT_FILE `
"python=$env:PYTHON_VERSION[build=*cpython]" ; Check-Output $?
if ($env:TASK -ne "bdist") {
conda activate $env:CONDA_ENV
}
cd $env:BUILD_SOURCESDIRECTORY
if ($env:TASK -eq "regular") {
cmake -B build -S . -A x64 ; Check-Output $?
cmake --build build --target ALL_BUILD --config Release ; Check-Output $?
sh ./build-python.sh install --precompile ; Check-Output $?
cp ./Release/lib_lightgbm.dll $env:BUILD_ARTIFACTSTAGINGDIRECTORY
cp ./Release/lightgbm.exe $env:BUILD_ARTIFACTSTAGINGDIRECTORY
}
elseif ($env:TASK -eq "sdist") {
sh ./build-python.sh sdist ; Check-Output $?
sh ./.ci/check-python-dists.sh ./dist ; Check-Output $?
cd dist; pip install @(Get-ChildItem *.gz) -v ; Check-Output $?
}
elseif ($env:TASK -eq "bdist") {
# Import the Chocolatey profile module so that the RefreshEnv command
# invoked below properly updates the current PowerShell session environment.
$module = "$env:ChocolateyInstall\helpers\chocolateyProfile.psm1"
Import-Module "$module" ; Check-Output $?
[python] Windows wheels that support GPUs via OpenCL (#3403) * Add specialized OpenCL/Python package build path * Refer to upstream OpenCL repository * Reset build job count in setup.py * Run test for dual CPU/GPU code in CI * TEMPORARY: refer to OpenCL fork to ensure Linux CI builds succeed * Try switching to OpenCL 2.0 * Add --opencl-python-pacakge option to wheel * Fix some typos. * Remove intermediate cmake target * Restrict OpenCL headers to documented API version * Refresh the environment after installation * Try testing with two processes * Use command line definition to activate integrated build * Try newer OpenCL runtime * If we hit OpenCL code path that's good enough * Make tests run correctly * Correct directory * Flag reference to unofficial repo with FIXME * TEMPORARY: update private repo tag for dependency * Further post-merge fixes. * Fix attribute name. * Reformat with black. * Rewrite config.h before library compilation for Windows wheel * Run dual cpu/gpu test once only for Windows bdist CI * Avoid C API tests when building gpu library for wheel * TEMPORARY -- add clinfo tool to vm for debugging * Increase compilation logging * Rename OpenCL registry parameter after driver installation * Install legacy OpenCL runtime * ... * Remove config.h change, clinfo installation, latest opencl runtime reference * Add explicit script exit * revert to 18.1 intel driver * Run test_dual.py in two different modes * Revert "Run test_dual.py in two different modes" This reverts commit cbfec02659816f0cb321e1184994a9c2f37d2a77. * Remove unused expected failure test * Install OpenCL in separate powershell script * fix setup.py imports * activate conda env after session update * separate update functions; exit opencl installation on error * nothing; start pipeline * download from original source * specify tls version for installer download * eliminate variable from command * download opencl installer in parts * remove duplicate setup.py command class entry * download smaller parts * remove redundant bdist wheel class * Import chocolatey modules and use RefreshEnv * insert pipeline cache for opencl installer * rerun to ensure pipeline cache hit * add comments, test another pipeline cache hit * remove cache task; download full installer file from new location * try another installer download timing * reinsert pipeline cache * retain cache task; download full file on cache miss * test cache hit again * remove todo Co-authored-by: TP Boudreau <tpboudreau@gmail.com>
2020-12-19 18:05:04 +03:00
RefreshEnv
[python] Windows wheels that support GPUs via OpenCL (#3403) * Add specialized OpenCL/Python package build path * Refer to upstream OpenCL repository * Reset build job count in setup.py * Run test for dual CPU/GPU code in CI * TEMPORARY: refer to OpenCL fork to ensure Linux CI builds succeed * Try switching to OpenCL 2.0 * Add --opencl-python-pacakge option to wheel * Fix some typos. * Remove intermediate cmake target * Restrict OpenCL headers to documented API version * Refresh the environment after installation * Try testing with two processes * Use command line definition to activate integrated build * Try newer OpenCL runtime * If we hit OpenCL code path that's good enough * Make tests run correctly * Correct directory * Flag reference to unofficial repo with FIXME * TEMPORARY: update private repo tag for dependency * Further post-merge fixes. * Fix attribute name. * Reformat with black. * Rewrite config.h before library compilation for Windows wheel * Run dual cpu/gpu test once only for Windows bdist CI * Avoid C API tests when building gpu library for wheel * TEMPORARY -- add clinfo tool to vm for debugging * Increase compilation logging * Rename OpenCL registry parameter after driver installation * Install legacy OpenCL runtime * ... * Remove config.h change, clinfo installation, latest opencl runtime reference * Add explicit script exit * revert to 18.1 intel driver * Run test_dual.py in two different modes * Revert "Run test_dual.py in two different modes" This reverts commit cbfec02659816f0cb321e1184994a9c2f37d2a77. * Remove unused expected failure test * Install OpenCL in separate powershell script * fix setup.py imports * activate conda env after session update * separate update functions; exit opencl installation on error * nothing; start pipeline * download from original source * specify tls version for installer download * eliminate variable from command * download opencl installer in parts * remove duplicate setup.py command class entry * download smaller parts * remove redundant bdist wheel class * Import chocolatey modules and use RefreshEnv * insert pipeline cache for opencl installer * rerun to ensure pipeline cache hit * add comments, test another pipeline cache hit * remove cache task; download full installer file from new location * try another installer download timing * reinsert pipeline cache * retain cache task; download full file on cache miss * test cache hit again * remove todo Co-authored-by: TP Boudreau <tpboudreau@gmail.com>
2020-12-19 18:05:04 +03:00
Write-Output "Current OpenCL drivers:"
Get-ItemProperty -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Khronos\OpenCL\Vendors
conda activate $env:CONDA_ENV
sh "build-python.sh" bdist_wheel --integrated-opencl ; Check-Output $?
sh ./.ci/check-python-dists.sh ./dist ; Check-Output $?
cd dist; pip install @(Get-ChildItem *py3-none-win_amd64.whl) ; Check-Output $?
cp @(Get-ChildItem *py3-none-win_amd64.whl) $env:BUILD_ARTIFACTSTAGINGDIRECTORY
} elseif (($env:APPVEYOR -eq "true") -and ($env:TASK -eq "python")) {
if ($env:COMPILER -eq "MINGW") {
sh ./build-python.sh install --mingw ; Check-Output $?
} else {
sh ./build-python.sh install; Check-Output $?
}
}
if (($env:TASK -eq "sdist") -or (($env:APPVEYOR -eq "true") -and ($env:TASK -eq "python"))) {
[python] Windows wheels that support GPUs via OpenCL (#3403) * Add specialized OpenCL/Python package build path * Refer to upstream OpenCL repository * Reset build job count in setup.py * Run test for dual CPU/GPU code in CI * TEMPORARY: refer to OpenCL fork to ensure Linux CI builds succeed * Try switching to OpenCL 2.0 * Add --opencl-python-pacakge option to wheel * Fix some typos. * Remove intermediate cmake target * Restrict OpenCL headers to documented API version * Refresh the environment after installation * Try testing with two processes * Use command line definition to activate integrated build * Try newer OpenCL runtime * If we hit OpenCL code path that's good enough * Make tests run correctly * Correct directory * Flag reference to unofficial repo with FIXME * TEMPORARY: update private repo tag for dependency * Further post-merge fixes. * Fix attribute name. * Reformat with black. * Rewrite config.h before library compilation for Windows wheel * Run dual cpu/gpu test once only for Windows bdist CI * Avoid C API tests when building gpu library for wheel * TEMPORARY -- add clinfo tool to vm for debugging * Increase compilation logging * Rename OpenCL registry parameter after driver installation * Install legacy OpenCL runtime * ... * Remove config.h change, clinfo installation, latest opencl runtime reference * Add explicit script exit * revert to 18.1 intel driver * Run test_dual.py in two different modes * Revert "Run test_dual.py in two different modes" This reverts commit cbfec02659816f0cb321e1184994a9c2f37d2a77. * Remove unused expected failure test * Install OpenCL in separate powershell script * fix setup.py imports * activate conda env after session update * separate update functions; exit opencl installation on error * nothing; start pipeline * download from original source * specify tls version for installer download * eliminate variable from command * download opencl installer in parts * remove duplicate setup.py command class entry * download smaller parts * remove redundant bdist wheel class * Import chocolatey modules and use RefreshEnv * insert pipeline cache for opencl installer * rerun to ensure pipeline cache hit * add comments, test another pipeline cache hit * remove cache task; download full installer file from new location * try another installer download timing * reinsert pipeline cache * retain cache task; download full file on cache miss * test cache hit again * remove todo Co-authored-by: TP Boudreau <tpboudreau@gmail.com>
2020-12-19 18:05:04 +03:00
# cannot test C API with "sdist" task
$tests = $env:BUILD_SOURCESDIRECTORY + "/tests/python_package_test"
} else {
[python] Windows wheels that support GPUs via OpenCL (#3403) * Add specialized OpenCL/Python package build path * Refer to upstream OpenCL repository * Reset build job count in setup.py * Run test for dual CPU/GPU code in CI * TEMPORARY: refer to OpenCL fork to ensure Linux CI builds succeed * Try switching to OpenCL 2.0 * Add --opencl-python-pacakge option to wheel * Fix some typos. * Remove intermediate cmake target * Restrict OpenCL headers to documented API version * Refresh the environment after installation * Try testing with two processes * Use command line definition to activate integrated build * Try newer OpenCL runtime * If we hit OpenCL code path that's good enough * Make tests run correctly * Correct directory * Flag reference to unofficial repo with FIXME * TEMPORARY: update private repo tag for dependency * Further post-merge fixes. * Fix attribute name. * Reformat with black. * Rewrite config.h before library compilation for Windows wheel * Run dual cpu/gpu test once only for Windows bdist CI * Avoid C API tests when building gpu library for wheel * TEMPORARY -- add clinfo tool to vm for debugging * Increase compilation logging * Rename OpenCL registry parameter after driver installation * Install legacy OpenCL runtime * ... * Remove config.h change, clinfo installation, latest opencl runtime reference * Add explicit script exit * revert to 18.1 intel driver * Run test_dual.py in two different modes * Revert "Run test_dual.py in two different modes" This reverts commit cbfec02659816f0cb321e1184994a9c2f37d2a77. * Remove unused expected failure test * Install OpenCL in separate powershell script * fix setup.py imports * activate conda env after session update * separate update functions; exit opencl installation on error * nothing; start pipeline * download from original source * specify tls version for installer download * eliminate variable from command * download opencl installer in parts * remove duplicate setup.py command class entry * download smaller parts * remove redundant bdist wheel class * Import chocolatey modules and use RefreshEnv * insert pipeline cache for opencl installer * rerun to ensure pipeline cache hit * add comments, test another pipeline cache hit * remove cache task; download full installer file from new location * try another installer download timing * reinsert pipeline cache * retain cache task; download full file on cache miss * test cache hit again * remove todo Co-authored-by: TP Boudreau <tpboudreau@gmail.com>
2020-12-19 18:05:04 +03:00
$tests = $env:BUILD_SOURCESDIRECTORY + "/tests"
}
if ($env:TASK -eq "bdist") {
[python] Windows wheels that support GPUs via OpenCL (#3403) * Add specialized OpenCL/Python package build path * Refer to upstream OpenCL repository * Reset build job count in setup.py * Run test for dual CPU/GPU code in CI * TEMPORARY: refer to OpenCL fork to ensure Linux CI builds succeed * Try switching to OpenCL 2.0 * Add --opencl-python-pacakge option to wheel * Fix some typos. * Remove intermediate cmake target * Restrict OpenCL headers to documented API version * Refresh the environment after installation * Try testing with two processes * Use command line definition to activate integrated build * Try newer OpenCL runtime * If we hit OpenCL code path that's good enough * Make tests run correctly * Correct directory * Flag reference to unofficial repo with FIXME * TEMPORARY: update private repo tag for dependency * Further post-merge fixes. * Fix attribute name. * Reformat with black. * Rewrite config.h before library compilation for Windows wheel * Run dual cpu/gpu test once only for Windows bdist CI * Avoid C API tests when building gpu library for wheel * TEMPORARY -- add clinfo tool to vm for debugging * Increase compilation logging * Rename OpenCL registry parameter after driver installation * Install legacy OpenCL runtime * ... * Remove config.h change, clinfo installation, latest opencl runtime reference * Add explicit script exit * revert to 18.1 intel driver * Run test_dual.py in two different modes * Revert "Run test_dual.py in two different modes" This reverts commit cbfec02659816f0cb321e1184994a9c2f37d2a77. * Remove unused expected failure test * Install OpenCL in separate powershell script * fix setup.py imports * activate conda env after session update * separate update functions; exit opencl installation on error * nothing; start pipeline * download from original source * specify tls version for installer download * eliminate variable from command * download opencl installer in parts * remove duplicate setup.py command class entry * download smaller parts * remove redundant bdist wheel class * Import chocolatey modules and use RefreshEnv * insert pipeline cache for opencl installer * rerun to ensure pipeline cache hit * add comments, test another pipeline cache hit * remove cache task; download full installer file from new location * try another installer download timing * reinsert pipeline cache * retain cache task; download full file on cache miss * test cache hit again * remove todo Co-authored-by: TP Boudreau <tpboudreau@gmail.com>
2020-12-19 18:05:04 +03:00
# Make sure we can do both CPU and GPU; see tests/python_package_test/test_dual.py
$env:LIGHTGBM_TEST_DUAL_CPU_GPU = "1"
}
pytest $tests ; Check-Output $?
if (($env:TASK -eq "regular") -or (($env:APPVEYOR -eq "true") -and ($env:TASK -eq "python"))) {
cd $env:BUILD_SOURCESDIRECTORY/examples/python-guide
@("import matplotlib", "matplotlib.use('Agg')") + (Get-Content "plot_example.py") | Set-Content "plot_example.py"
(Get-Content "plot_example.py").replace('graph.render(view=True)', 'graph.render(view=False)') | Set-Content "plot_example.py" # prevent interactive window mode
conda install -y -n $env:CONDA_ENV "h5py>=3.10" "ipywidgets>=8.1.2" "notebook>=7.1.2"
foreach ($file in @(Get-ChildItem *.py)) {
@("import sys, warnings", "warnings.showwarning = lambda message, category, filename, lineno, file=None, line=None: sys.stdout.write(warnings.formatwarning(message, category, filename, lineno, line))") + (Get-Content $file) | Set-Content $file
python $file ; Check-Output $?
} # run all examples
cd $env:BUILD_SOURCESDIRECTORY/examples/python-guide/notebooks
(Get-Content "interactive_plot_example.ipynb").replace('INTERACTIVE = False', 'assert False, \"Interactive mode disabled\"') | Set-Content "interactive_plot_example.ipynb"
jupyter nbconvert --ExecutePreprocessor.timeout=180 --to notebook --execute --inplace *.ipynb ; Check-Output $? # run all notebooks
}