Граф коммитов

18 Коммитов

Автор SHA1 Сообщение Дата
Pablo Selem a6e51c964e
Feature/container (#153)
* force add PATH to current user

* checkin docker setup script

* Update cluster_setup.sh

* install docker and start container on cluster setup

* WIP: Run task in container

* fix merge conflict

* run tasks and merge task from within container

* refactor code to proper docker commands and make a single R container per job

* refactor command line utils into its own file

* refactor job utilities into its own file

* move cluster setup script to inst folder

* remove unnecessary curl installs

* remove starting container from setup script

* check in WIP

* add apt_install file

* make required directories

* update cluster setup files as needed

* include libxml2 packages in apt installs

* working cluster create with cran and github dependencies

* update job prep to install apt-get and not each task

* use rocker containers instead of r-base

* remove unused & commented code

* remove unused install function

* address several lintr issues

* initial test dockerfile

* add spacing between commands

* temporarily point wget to feature branch

* update bioconductor install for non-jobPrep installs

* Delete Dockerfile

* minor changes to install bioc

* resolve merge conflicts

* update cluster to correctly install BioC packages using install_bioconductor

* fix issue where some packages were not getting installed

* add missing BioConductorCommand initializer

* remove print lines

* initial dockerfile implementations

* update docker files

* Only install packages if they are required

* Remove requirement on bioconductor installer script on start task

* remove duplicate environment variable entry

* update docs for container support

* update version to 0.6.0

* refactor changes updates

* remove poorly formatted whitespaces

* add full path to pacakges directory

* fix docker command line

* update file share sample

* update azure files cluster name

* update mandelbrot sample

* update package management sample

* update plyr samples

* make montecarlo sample more consistent

* update montecarlo sample

* remove plyr example

* fix bad environment pointer

* fix linter issues

* more linter fixes

* more linter issues

* use latest rAzureBatch version

* update resource files example

* remove reference to deleted sample

* pr feedback

* PR docs feedback

* Print errors from worker (#154)

* Fixed pool package command line lintr test

* Package installation tests fixed - too long lines

* Fixed json in customize cluster docs

* Fix: Typos in customize cluster docs

* Cleaning up files

* Feature/githubbiopackage (#150)

* install github package worked for foreach loop

* fix lintr error

* tests for github and bioc packages installation

* lintr fix

* add back lost code due to merge and update docs

* The Travis CI build failed for feature/githubbiopackage

* remove incorrect parameter for install_github

* Updated job prep task to have default command

* Use the latest version of rAzureBatch

* Updated description + Generate cluster config

* Fix: Bioconductor and Github packages installation (#155)

* Added multiple package install test and fix obj reading args

* Fixed naming for packages install

* Replaced validation exclusion for linter

* Fixed test validate test

* Fixing all interactive tests with skip

* Fixed renaming validation

* Removed default test - cannot be tested

* Removed  in validation

* Added cluster package install tests (#156)
2017-11-03 10:06:40 -07:00
zfengms 907812228c address review feedback 2017-10-02 16:58:56 -07:00
zfengms 7ec1306aeb fix bug in metadata handling for packages and enableCloudCombine (#133)
* fix bug in metadata handling for packages and enableCloudCombine

* call long running job api in test

* update test

* add test for long running job feature

* code style fix

* update job state description in readme

* use list for job state filter
2017-09-30 21:45:25 -07:00
zfengms 2e8aff58c3 Merge from feature/getjobresult for long running job support (#130)
* Added set chunk size

* Added cluster configuration validation function (#30)

* Added pool config test validation

* Added a fix for validation

* Added if checks for null tests and more validation tests

* Install R packages at job run time (#29)

* Added cran/github installation scripts

* Added package installation tests

* Upgraded package version to 0.3.2

* Output file support (#40)

* Output files support

* Added createOutputFile method

* output files readme documentation

* added tests and find container sas

* Added more detailed variable names

* Enable/disable merge task (#39)

* Merge task pass params

* Fixed enableMerge cases

* Merge task documentation on README.md

* Fixed typo on merge task description

* Update doAzureParallel.R

* Changed enableMerge to enableCloudCombine

* convert getJobResult output from binary to text

* Only write vector to temp file

* save cloud merge enabled, chunk size and packages as job metadata

* update cloudMergeEnabled to cloudCombineEnabled

* Fix/backwards compatible (#68)

* Added backwards compatible in make cluster

* Added deprecated config validator

* Added mismatch label

* Added validation for quota limits and bad getPool requests in waitForNodesToComplete (#52)

* Added validation for quota limits and bad getPool requests

* Fixed based on PR

* Fixed progress bar layout to use switch statements instead of if statements

* Changed clusterId to poolId

* Added comments and fixed messages

* Added running state to the node status

* Reformatted lines for function

* Added end statement for node completion

* Feature/custom script and reduce (#70)

* Added custom scripts and removed dependencies parameter

* Updated roxygen tool version

* Added parallelThreads support

* Added test coverage

* Removed verbose message on command line

* Added Reduce function for group of tasks

* Fix build because of doc semantics mismatch with function

* Removed unused function

* Added command line arg

* Added docs for custom script

* Moved customize cluster to separate doc for future usage

* Fixed typo

* Bug - Waiting for tasks to completion function ends too early (#69)

* Moved wait for tasks to complete to doAzureParallel utility

* Removed unneeded variables and progress

* Fixed camel case for skiptoken

* Travis/lintr (#72)

* Added lintr config file

* Added travis github package installation

* Removed snake case rule

* Fixed documents on doAzureParallel

* Based on lintr default_settins docs, correctly added default rules

* Updated lintr package to use object_name_style

* Added package :: operator

* Reformatted after merge

* Fixed command line tests

* Upgraded roxygen to 6.0.1

* Cluster config docs

* Removed additional delete job

* add getJob api (#84)

* add getJob api

* reformat code

* update styling in utility file

* fix code styling

* update chunksize to chunkSize and remove unused code

* handle job metadata in getJob api

* fix styling issue

* update getJobList parameter from list of job ids to filter object, and output jobs status in data frame (#128)

long running job support, getJob, getJobList and getJobResult implementation

* reformat code

* update styling in utility file

* fix code styling

* update chunksize to chunkSize and remove unused code

* handle job metadata in getJob api

* fix styling issue

* use counting service api in getJobList

* fix coding style

* return data frame from getJobList

* update getJobList parameter from job id list to filter by state

* reformat code

* update description for getJobList

* remove dup code

* address review feedback

* jobId parameter check for getJobResult

* update documentation for long run job

* update version to 0.5.0

* update version

* address review feedback

* update chunkSizeValue to chunkSizeKeyValuePair
2017-09-29 16:49:57 -07:00
zfengms 91db8a959e readme.md update 2017-09-29 14:17:09 -07:00
Brian 3dadf0803e Fixed worker and merger scripts (#116)
* Fixed worker and merger scripts

* Fixed verbose logs based on PR comments

* Added documentation on error handling

* Fixed header on table markdown

* Fixed based on PR comments
2017-09-28 15:18:06 -07:00
Dan Tenenbaum 368eeb85c8 Change True/False to TRUE/FALSE in README example (#124)
* add .gitiattrributes file to track line endings

* True and False are not valid in R; changed to TRUE and FALSE
2017-09-27 14:29:32 -07:00
Pablo Selem 0f03d171f2 update docs to include github auth feature (#75)
* update docs to include github auth feature

* pr feedback for more verbose context

* move github auth instructions to pacakge management document

* remove github specific link from readme
2017-08-25 13:12:02 -07:00
Brian 3ef734500a Milestone/0.4.0 (#74)
* Added set chunk size

* Added cluster configuration validation function (#30)

* Added pool config test validation

* Added a fix for validation

* Added if checks for null tests and more validation tests

* Install R packages at job run time (#29)

* Added cran/github installation scripts

* Added package installation tests

* Upgraded package version to 0.3.2

* Output file support (#40)

* Output files support

* Added createOutputFile method

* output files readme documentation

* added tests and find container sas

* Added more detailed variable names

* Enable/disable merge task (#39)

* Merge task pass params

* Fixed enableMerge cases

* Merge task documentation on README.md

* Fixed typo on merge task description

* Update doAzureParallel.R

* Changed enableMerge to enableCloudCombine

* Fix/backwards compatible (#68)

* Added backwards compatible in make cluster

* Added deprecated config validator

* Added mismatch label

* Added validation for quota limits and bad getPool requests in waitForNodesToComplete (#52)

* Added validation for quota limits and bad getPool requests

* Fixed based on PR

* Fixed progress bar layout to use switch statements instead of if statements

* Changed clusterId to poolId

* Added comments and fixed messages

* Added running state to the node status

* Reformatted lines for function

* Added end statement for node completion

* Feature/custom script and reduce (#70)

* Added custom scripts and removed dependencies parameter

* Updated roxygen tool version

* Added parallelThreads support

* Added test coverage

* Removed verbose message on command line

* Added Reduce function for group of tasks

* Fix build because of doc semantics mismatch with function

* Removed unused function

* Added command line arg

* Added docs for custom script

* Moved customize cluster to separate doc for future usage

* Fixed typo

* Bug - Waiting for tasks to completion function ends too early (#69)

* Moved wait for tasks to complete to doAzureParallel utility

* Removed unneeded variables and progress

* Fixed camel case for skiptoken

* Travis/lintr (#72)

* Added lintr config file

* Added travis github package installation

* Removed snake case rule

* Fixed documents on doAzureParallel

* Based on lintr default_settins docs, correctly added default rules

* Updated lintr package to use object_name_style

* Added package :: operator

* Reformatted after merge

* Fixed command line tests

* Upgraded roxygen to 6.0.1

* Cluster config docs

* Removed additional delete job

* Fixed warning descriptions in makeCluster

* Ramped up versions for DESCRIPTION

* Updated CHANGELOG to 0.4.0

* Fixed environmentSettings style from merge conflict

* Fixed cluster warning naming style
2017-08-22 16:49:28 -07:00
Brian c3a6384924 Added travis ci banner 2017-06-13 22:12:10 -07:00
JS 32f2145fe3 Update README.md 2017-05-23 16:12:32 -07:00
Brian 02c5eac6bf v0.3.0 Release (#20)
* Added set chunk size

* Adding resource files on pool creation

* renaming generate file functions

* Moved worker/merger scripts to doAzureParallel and created common job env

* Added stdout and stderr logs in uploads

* added to docs / README

* Switched params for cluster and added examples

* setCreds, resizeCluster, job management

* cred generator update

* Added samples, moved autoscale, and low-pri/output files

* Added documentation on methods for ??R feature

* Added export for makeCluster

* Namespace missing export

* clusterSetting param name

* cluster id param name

* NumOfNodes param for wait nodes completion fix

* Added proper naming for registerDoAzureParallel

* readme update'

* typo readme

* low pri in readme

* monte carlo simulation

* Added new sample for sas resource files

* caret + annotation on montecarlo sim

* samples readme.md

* samples readme

* Fixed the resource files to use proper storage account for example

* Update README.md

* Update 11-autoscale.md

* Fixed autoscale formula for task queue to take maxTaskPerNode

* Added named args to createSasToken

* Update resource-files-example.R

* Update 21-distributing-data.md

* Renamed samples files to underscore format

* Update 21-distributing-data.md

* Update README.md

* Update README.md

* Update README.md

* Edited changelog file

* Update plyr_example.R

* Update README.md
2017-05-22 17:22:40 -07:00
JS 56cc34ea37 README / Documentation Updates for v0.2.0 (#7)
* README updates - json file pointers

* documentation - autoscale formula update

* README - reconfigured getting started

* README - persistent storage update

* README - update with autoscaling

* README - pool limitations

* README - limitations added to doc README

* README - update to maxcpu formula

* README - chunkSize documentation

* README json update

* package management update
2017-03-15 13:34:15 -07:00
jiata d532b5037f README update 2017-03-13 21:54:41 +00:00
JS 6563ac21de Updates to documentation
* typo in README.md and job quota limitation update

* added intro code to README.md

* doc typos and separated out contributing content

* README typo

* removed contributing section from README.md
2017-02-17 07:46:25 -08:00
JS 726e10b9d1 Created documentation
* put images into vignette folder

* Created 00-azure-introduction.md

* Created 10-vm-sizes.md

* Update VM sizes link in docs/README.md

* Create 20-package-management.md

* Updated README.md with Azure Batch limitations

* Create 21-distributing-data.md

* Create 22-parallelizing-cores.md

* Create 23-persistent-storage.md

* standardized foreach keyword and pool keyword
2017-02-15 21:27:55 -08:00
JS 46fc9acefb Updating README.md
* Introduction to README.md

* Updated README

* instructional images to get keys and account info

* Added images to README
2017-02-15 17:54:04 -08:00
Microsoft Open Source 24b8d08cca Initial commit 2017-02-14 16:17:57 -08:00