addressing comments and removing dependencies that we don't use anymore

INSTALL.md

@@ -1,8 +1,16 @@
 # Installation guide
 
+In this guide we show how to install all the dependencies to run the notebooks of this repo. 
+
+We have three different environments for our algorithms:
+* Python CPU
+* Python GPU
+* PySpark
+
 ## Requirements
 
 - [Anaconda Python 3.6](https://conda.io/miniconda.html)
+- The library dependencies are generated depending on the environment we choose and can be found in this [script](scripts/generate_conda_file.sh).
 - Machine with GPU (optional)
 - Machine with Spark (optional)
 
@@ -12,7 +20,7 @@ As a pre-requisite, we may want to make sure that Conda is up-to-date:
 
     conda update conda
 
-We have three different environments: Python CPU, Python GPU and PySpark. We provided a script to [generate a conda file](scripts/generate_conda_file.sh), depending of the environment we want to use.
+We provided a script to [generate a conda file](scripts/generate_conda_file.sh), depending of the environment we want to use.
 
 To install each environment, first we need to generate a conda yml file and then install the environment. We can choose the environment name with the input `-n`. In the following examples, we provide a name example.
 
@@ -37,7 +45,23 @@ To install the PySpark environment, which by default installs the CPU environmen
     ./scripts/generate_conda_file.sh --pyspark
     conda env create -n reco_pyspark -f conda_gpu.yaml
 
-For this environment, we need to set the environment variables `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` to point to the conda python executable. In this [guide](https://conda.io/docs/user-guide/tasks/manage-environments.html#macos-and-linux), it is shown how these variables can be added every time the environment is activated.
+For this environment, we need to set the environment variables `PYSPARK_PYTHON` and `PYSPARK_DRIVER_PYTHON` to point to the conda python executable. 
+
+For setting these variables every time the environment is activated, we can follow the steps of this [guide](https://conda.io/docs/user-guide/tasks/manage-environments.html#macos-and-linux). Assuming that we have installed the environment in `/anaconda/envs/reco_pyspark`, we create the file `/anaconda/envs/reco_pyspark/activate.d/env_vars.sh` and add:
+
+```bash
+#!/bin/sh
+export PYSPARK_PYTHON=/anaconda/envs/reco_pyspark/bin/python
+export PYSPARK_DRIVER_PYTHON=/anaconda/envs/reco_pyspark/bin/python
+```
+
+This will export the variables every time we do `source activate reco_pyspark`. To unset these variables when we deactivate the environment, we create the file `/anaconda/envs/reco_pyspark/activate.d/env_vars.sh` and add:
+
+```bash
+#!/bin/sh
+unset PYSPARK_PYTHON
+unset PYSPARK_DRIVER_PYTHON
+```
 
 ### All environments
 
@@ -46,3 +70,13 @@ To install all three environments:
     ./scripts/generate_conda_file.sh  --gpu --pyspark
     conda env create -n reco_full -f conda_full.yaml
 
+### Register the conda environment in Jupyter notebook
+
+We can register our created conda environment to appear as a kernel in the Jupyter notebooks. 
+
+    python -m ipykernel install --user --name my_env_name --display-name "Python (my_env_name)"
+
+## Troubleshooting
+
+* We found that there could be problems if the Spark version of the machine is not the same as the one in the conda file. You will have to adapt the conda file to your machine. 
+

scripts/generate_conda_file.sh

@@ -75,7 +75,6 @@ channels:
 - conda-forge
 - defaults
 dependencies:
-- jupyter==1.0.0
 - python==3.6
 - numpy>=1.13.3
 - dask>=0.17.1
@@ -93,10 +92,6 @@ ${pyspark}- pyspark==2.3.1
   - azure-storage>=0.36.0
   - tffm==1.0.1
   - pytest==3.6.4
-  - pytest-cov
-  - pytest-datafiles>=1.0
-  - pylint>=2.0.1
-  - pytest-pylint==0.11.0
   - papermill>=0.15.0
   - black>=18.6b4
 EOM