* merge package management doc

* merge packagement docs

* address review feedback (#232)

* address review feedback

* add reference to github and bioconductor packages in worker

* update package management sample

* update package management doc (#233)

* address review feedback

* add reference to github and bioconductor packages in worker

* update package management sample

* update package management doc

* remove cluster.json

* remove package installation
This commit is contained in:
zfengms 2018-04-26 00:50:38 -07:00 коммит произвёл GitHub
Родитель fa75afbbd2
Коммит 0fbfd4cb07
Не найден ключ, соответствующий данной подписи
Идентификатор ключа GPG: 4AEE18F83AFDEB23
4 изменённых файлов: 39 добавлений и 77 удалений

Просмотреть файл

@ -4,7 +4,26 @@ The doAzureParallel package allows you to install packages to your pool in two w
- Installing on pool creation - Installing on pool creation
- Installing per-*foreach* loop - Installing per-*foreach* loop
Packages installed at the pool level benefit from only needing to be installed once per node. Each iteration of the foreach can load the library without needing to install them again. Packages installed in the foreach benefit from specifying any dependencies required only for that instance of the loop.
## Installing Packages on Pool Creation ## Installing Packages on Pool Creation
Pool level packages support CRAN, GitHub and BioConductor packages. The packages are installed in a shared directory on the node. It is important to note that it is required to add it to .packages parameter (or github or bioconductor for github or bioconductor packages), or explicitly load any packages installed at the pool level within the foreach loop. For example, if you installed xml2 on the cluster, you must explicitly load it or add it to .packages before using it.
```R
foreach (i = 1:4) %dopar% {
# Load the libraries you want to use.
library(xml2)
xml2::as_list(...)
}
```
or
```R
foreach (i = 1:4, .packages=c('xml2')) %dopar% {
xml2::as_list(...)
}
```
You can install packages by specifying the package(s) in your JSON pool configuration file. This will then install the specified packages at the time of pool creation. You can install packages by specifying the package(s) in your JSON pool configuration file. This will then install the specified packages at the time of pool creation.
```R ```R

Просмотреть файл

@ -92,6 +92,14 @@ for (package in azbatchenv$packages) {
library(package, character.only = TRUE) library(package, character.only = TRUE)
} }
for (package in azbatchenv$github) {
library(package, character.only = TRUE)
}
for (package in azbatchenv$bioconductor) {
library(package, character.only = TRUE)
}
ls(azbatchenv) ls(azbatchenv)
parent.env(azbatchenv$exportenv) <- getparentenv(azbatchenv$pkgName) parent.env(azbatchenv$exportenv) <- getparentenv(azbatchenv$pkgName)

Просмотреть файл

@ -1,69 +0,0 @@
# Using package management
doAzureParallel supports installing packages at either the cluster level or during the execution of the foreach loop. Packages installed at the cluster level benefit from only needing to be installed once per node. Each iteration of the foreach can load the library without needing to install them again. Packages installed in the foreach benefit from specifying any specific dependencies required only for that instance of the loop.
## Cluster level packages
Cluster level packages support CRAN, GitHub and BioConductor packages. The packages are installed in a shared directory on the node. It is important to note that it is required to explicitly load any packages installed at the cluster level within the foreach loop. For example, if you installed xml2 on the cluster, you must explicityly load it before using it.
```R
foreach (i = 1:4) %dopar% {
# Load the libraries you want to use.
library(xml2)
xml2::as_list(...)
}
```
### CRAN
CRAN packages can be insatlled on the cluster by adding them to the collection of _cran_ packages in the cluster specification.
```json
"rPackages": {
"cran": ["package1", "package2", "..."],
"github": [],
"bioconductor": []
}
```
### GitHub
GitHub packages can be insatlled on the cluster by adding them to the collection of _github_ packages in the cluster specification.
```json
"rPackages": {
"cran": [],
"github": ["repo1/name1", "repo1/name2", "repo2/name1", "..."],
"bioconductor": []
}
```
**NOTE** When using packages from a private GitHub repository, you must add your GitHub authentication token to your credentials.json file.
### BioConductor
Installing bioconductor packages is now supported via the cluster configuration. Simply add the list of packages you want to have installed in the cluster configuration file and they will get automatically applied
```json
"rPackages": {
"cran": [],
"github": [],
"bioconductor": ["IRanges", "GenomeInofDb"]
}
```
**IMPORTANT** doAzureParallel uses the rocker/tidyverse Docker images by default, which comes with BioConductor pre-installed. If you use a different container image, make sure that bioconductor is installed on it.
## Foreach level packages
Foreach level packages currently only support CRAN packages. Unlike cluster level pacakges, when specifying packages on the foreach loop, packages will be automatically installed _and loaded_ for use.
### CRAN
```R
foreach(i = 1:4, .packages = c("xml2")) %dopar% {
# xml2 is automatically loaded an can be used without calling library(xml2)
xml2::as_list(...)
}
```

Просмотреть файл

@ -1,21 +1,19 @@
# install packages #Please see documentation at docs/20-package-management.md for more details on packagement management.
library(devtools)
install_github("azure/doazureparallel")
# import the doAzureParallel library and its dependencies # import the doAzureParallel library and its dependencies
library(doAzureParallel) library(doAzureParallel)
# set your credentials # set your credentials
setCredentials("credentials.json") doAzureParallel::setCredentials("credentials.json")
# Create your cluster if not exist # Create your cluster if not exist
cluster <- makeCluster("bioconductor_cluster.json") cluster <- doAzureParallel::makeCluster("bioconductor_cluster.json")
# register your parallel backend # register your parallel backend
registerDoAzureParallel(cluster) doAzureParallel::registerDoAzureParallel(cluster)
# check that your workers are up # check that your workers are up
getDoParWorkers() doAzureParallel::getDoParWorkers()
summary <- foreach(i = 1:1) %dopar% { summary <- foreach(i = 1:1) %dopar% {
library(GenomeInfoDb) # Already installed as part of the cluster configuration library(GenomeInfoDb) # Already installed as part of the cluster configuration
@ -23,7 +21,13 @@ summary <- foreach(i = 1:1) %dopar% {
sessionInfo() sessionInfo()
# Your algorithm # Your algorithm
}
summary
summary <- foreach(i = 1:1, bioconductor=c('GenomeInfoDb', 'IRanges')) %dopar% {
sessionInfo()
# Your algorithm
} }
summary summary