124 строки
4.7 KiB
Markdown
124 строки
4.7 KiB
Markdown
# Installation and Setup
|
|
|
|
Here we present the instructions for setting up the project on an [Ubuntu Azure VM](https://azure.microsoft.com/en-us/services/virtual-machines/). The VM we used for the experiment was a NV24 with 4 NVIDIA M60 GPUs. The OS was Ubuntu 16.04. We recommend to use the [Azure Data Science VM](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/microsoft-ads.standard-data-science-vm) which comes with many machine learning tools already installed.
|
|
|
|
## Setting up the environment
|
|
|
|
Clone this repo to your desired location
|
|
```bash
|
|
git clone https://github.com/Azure/fast_retraining.git
|
|
```
|
|
|
|
Create a conda environment if you haven't already done so. The command below creates a python 3 environment called fast.
|
|
```bash
|
|
conda create --name fast python=3.5 anaconda
|
|
```
|
|
|
|
Edit [activate_env_vars.sh](environment/activate_env_vars.sh ) and [deactivate_env_vars.sh](environment/deactivate_env_vars.sh )
|
|
so that they contain the correct information.
|
|
|
|
Install command line json parser
|
|
```bash
|
|
apt-get install jq
|
|
```
|
|
|
|
Activate the conda environment and install the requirements.
|
|
```bash
|
|
source activate fast
|
|
pip install -r requirements.txt
|
|
```
|
|
|
|
Get the currently activated environment and assign it to env_path.
|
|
Get info of current env and output to json | look for default_prefix element in JSON | remove all quotes
|
|
```bash
|
|
env_path=$(conda info --json | jq '.default_prefix' | tr -d '"')
|
|
```
|
|
|
|
Make sure you are in the environment folder of the project and run the following
|
|
```bash
|
|
activate_script_path=$(readlink -f activate_env_vars.sh)
|
|
deactivate_script_path=$(readlink -f deactivate_env_vars.sh)
|
|
```
|
|
|
|
Then we create the activation and deactivation scripts and make sure they point to our now modified activation
|
|
and deactivation scripts in our environment folder
|
|
```bash
|
|
mkdir -p $env_path/etc/conda/activate.d
|
|
mkdir -p $env_path/etc/conda/deactivate.d
|
|
echo 'source '$activate_script_path >> $env_path/etc/conda/activate.d/env_vars.sh
|
|
echo 'source '$deactivate_script_path >> $env_path/etc/conda/deactivate.d/env_vars.sh
|
|
```
|
|
|
|
Exit the environment
|
|
```bash
|
|
source deactivate
|
|
```
|
|
|
|
Enter the environment again
|
|
```bash
|
|
source activate fast
|
|
```
|
|
|
|
Finally, to register the environment in the jupyter notebook:
|
|
```bash
|
|
python -m ipykernel install --user --name fast --display-name "Python Fast"
|
|
```
|
|
|
|
## Installation of boosted tree libraries
|
|
|
|
We need to install [XGBoost](https://github.com/dmlc/xgboost) and [LightGBM](https://github.com/microsoft/LightGBM). Even though both libraries have pypi versions, for creating the experiments contained in this repo we compiled from source.
|
|
|
|
To install XGBoost you can follow the [installation guide](https://xgboost.readthedocs.io/en/latest/build.html). To build in CPU, using the specific commit we used:
|
|
|
|
git clone --recursive https://github.com/dmlc/xgboost
|
|
cd xgboost
|
|
git checkout 6776292951565c8cd72e69afd9d94de1474f00c0
|
|
git submodule update --recursive
|
|
make -j$(nproc)
|
|
|
|
In case you want to use the last version, just skip the commands `git checkout` and `git submodule`.
|
|
|
|
If you want to build in GPU, the instructions are [here](https://github.com/dmlc/xgboost/tree/master/plugin/updater_gpu). You first need to download and unzip [CUB 1.6.4](https://nvlabs.github.io/cub/).
|
|
|
|
git clone --recursive https://github.com/dmlc/xgboost
|
|
cd xgboost
|
|
git checkout 6776292951565c8cd72e69afd9d94de1474f00c0
|
|
git submodule update --recursive
|
|
mkdir build
|
|
cd build
|
|
cmake .. -DPLUGIN_UPDATER_GPU=ON -DCUB_DIRECTORY=/path/to/cub-1.6.4
|
|
make -j$(nproc)
|
|
|
|
To install LighGBM you can follow the [installation guide](https://github.com/Microsoft/LightGBM/wiki/Installation-Guide). To build on CPU:
|
|
|
|
git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
|
|
git checkout 73968a96829e212b333c88cd44725c8c39c03ad1
|
|
mkdir build ; cd build
|
|
cmake ..
|
|
make -j$(nproc)
|
|
|
|
To install the GPU version:
|
|
|
|
git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
|
|
git checkout 73968a96829e212b333c88cd44725c8c39c03ad1
|
|
mkdir build ; cd build
|
|
cmake .. -DUSE_GPU=1
|
|
make -j$(nproc)
|
|
|
|
To install the python biddings you have to compile in the python directory. Both libraries have the exact same name for the python package, so you just need to do the following step in both libraries:
|
|
|
|
cd python-package
|
|
python setup.py install
|
|
|
|
Finally, to check that the libraries are correctly installed, try to load them from python:
|
|
|
|
python -c "import xgboost; import lightgbm"
|
|
|
|
|
|
## Installation of bokeh functionality to export plots
|
|
|
|
To generate png exports with bokeh you have to follow the instructions explained in [this link](http://bokeh.pydata.org/en/0.12.6/docs/user_guide/export.html).
|
|
|
|
sudo apt-get install npm
|
|
sudo npm install -g phantomjs-prebuilt
|