fast_retraining/INSTALL.md

Installation and Setup

Here we present the instructions for setting up the project on an Ubuntu Azure VM. The VM we used for the experiment was a NV24 with 4 NVIDIA M60 GPUs. The OS was Ubuntu 16.04. We recommend to use the Azure Data Science VM which comes with many machine learning tools already installed.

Setting up the environment

Clone this repo to your desired location

git clone https://github.com/Azure/fast_retraining.git

Create a conda environment if you haven't already done so. The command below creates a python 3 environment called fast.

conda create --name fast python=3.5 anaconda

Edit activate_env_vars.sh and deactivate_env_vars.sh so that they contain the correct information.

Install command line json parser

apt-get install jq

Activate the conda environment and install the requirements.

source activate fast
pip install -r requirements.txt

Get the currently activated environment and assign it to env_path. Get info of current env and output to json | look for default_prefix element in JSON | remove all quotes

env_path=$(conda info --json | jq '.default_prefix' | tr -d '"')

Make sure you are in the environment folder of the project and run the following

activate_script_path=$(readlink -f activate_env_vars.sh)
deactivate_script_path=$(readlink -f deactivate_env_vars.sh)

Then we create the activation and deactivation scripts and make sure they point to our now modified activation and deactivation scripts in our environment folder

mkdir -p $env_path/etc/conda/activate.d
mkdir -p $env_path/etc/conda/deactivate.d
echo 'source '$activate_script_path >> $env_path/etc/conda/activate.d/env_vars.sh
echo 'source '$deactivate_script_path >> $env_path/etc/conda/deactivate.d/env_vars.sh

Exit the environment

source deactivate

Enter the environment again

source activate fast

Finally, to register the environment in the jupyter notebook:

python -m ipykernel install --user --name fast --display-name "Python Fast"

Installation of boosted tree libraries

We need to install XGBoost and LightGBM. Even though both libraries have pypi versions, for creating the experiments contained in this repo we compiled from source.

To install XGBoost you can follow the installation guide. To build in CPU, using the specific commit we used:

git clone --recursive https://github.com/dmlc/xgboost
cd xgboost
git checkout 6776292951565c8cd72e69afd9d94de1474f00c0
git submodule update --recursive
make -j$(nproc)

In case you want to use the last version, just skip the commands git checkout and git submodule.

If you want to build in GPU, the instructions are here. You first need to download and unzip CUB 1.6.4.

git clone --recursive https://github.com/dmlc/xgboost
cd xgboost
git checkout 6776292951565c8cd72e69afd9d94de1474f00c0
git submodule update --recursive
mkdir build
cd build
cmake .. -DPLUGIN_UPDATER_GPU=ON -DCUB_DIRECTORY=/path/to/cub-1.6.4
make -j$(nproc)

To install LighGBM you can follow the installation guide. To build on CPU:

git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
git checkout 73968a96829e212b333c88cd44725c8c39c03ad1
mkdir build ; cd build
cmake ..
make -j$(nproc)

To install the GPU version:

git clone --recursive https://github.com/Microsoft/LightGBM ; cd LightGBM
git checkout 73968a96829e212b333c88cd44725c8c39c03ad1
mkdir build ; cd build
cmake .. -DUSE_GPU=1
make -j$(nproc)

To install the python biddings you have to compile in the python directory. Both libraries have the exact same name for the python package, so you just need to do the following step in both libraries:

cd python-package
python setup.py install

Finally, to check that the libraries are correctly installed, try to load them from python:

python -c "import xgboost; import lightgbm"

Installation of bokeh functionality to export plots

To generate png exports with bokeh you have to follow the instructions explained in this link.

sudo apt-get install npm
sudo npm install -g phantomjs-prebuilt