Граф коммитов

923 Коммитов

Автор SHA1 Сообщение Дата
Fred Park d6e939d58e
Use singularity env var over param for tasks
- Doc fixups
2019-06-19 20:11:09 +00:00
Fred Park 824f6de415
Merge branch 'develop' into singularity3
- Move username/password run options to settings for singularity
2019-06-18 20:49:38 +00:00
Fred Park bc4be6dbc3
Proxy non-native task execution via script
- Resolves #235
2019-06-18 20:11:12 +00:00
Vincent Labonté 6a0f90d509 Singularity list images and run ORAS images (#284)
* Remove unused directories

* Augment pool images list to support Singularity images

* Fix specific image update with private ORAS registries

* Add support to run ORAS image from a private registry

* Only log in used registries

* Fix checks

* Update documentation

* Resolve PR comments

* Resolve PR comments
2019-06-17 08:29:25 -07:00
Vincent Labonté 8293a20be3 Support multiple Singularity registries (#283)
* Add support for multiple singularity registries (://docker and ://oras)

* Resolve PR comments

* Resolve PR comments
2019-06-13 10:22:18 -07:00
Vincent Labonté 5307f1779d Fix image update command (#281)
* Create one log file per container mode

* Make singularity 3 work

* Minor fixes

* Fix cascade with docker image and singularity image

* Add capability to pull from library://

* Add singularity signed images to config file

* Add singularity signed images to the global resource table

* Pull and verify signed singularity images

* Put the singularity sypgp directory in the mount directory

* Add ability to provide key file to verify a singularity image

* Resolve PR comments

* Fix Singularity registry credemtials

* Extract cascade logic from nodeprep

* Re-run cascade if the image update command has no specified image

* Fix prefix errors when using shipyard docker image

* Make sure that the cascade log files are not overridden

* Fix wrong parameter name

* Clarify error message when trying to update images on Windows

* Update documentation

* Fix checks

* Resolve PR comments
2019-06-10 10:35:28 -07:00
Vincent Labonté 305d376cdc Support Singularity signed image verification (#280)
* Create one log file per container mode

* Make singularity 3 work

* Minor fixes

* Fix cascade with docker image and singularity image

* Add capability to pull from library://

* Add singularity signed images to config file

* Add singularity signed images to the global resource table

* Pull and verify signed singularity images

* Put the singularity sypgp directory in the mount directory

* Add ability to provide key file to verify a singularity image

* Resolve PR comments

* Fix Singularity registry credemtials
2019-06-05 11:14:37 -07:00
Vincent Labonté f9d0ad9a7f Initial support for Singularity 3 and SIF (#279)
* Create one log file per container mode

* Make singularity 3 work

* Minor fixes

* Fix cascade with docker image and singularity image

* Add capability to pull from library://
2019-05-29 14:15:22 -07:00
Fred Park 509834b3fb
Cascade Docker/Singularity image split 2019-05-23 21:29:24 +00:00
Vincent Labonté a68579c095 Prepare for Singularity3 work (#276)
* Remove torrent functionality

* Remove torrent storage

* Fix singularity permissions

* Add container mode in cascade.py

* Fix errors

* Fix PR comments

* Fix flake8 errors
2019-05-22 14:46:44 -07:00
Fred Park a16e125056
Doc updates 2019-05-10 14:26:14 +00:00
Fred Park d9a9f3865b
Add CG detection task 2019-04-16 18:19:23 +00:00
Fred Park cbb3689bd5
Fix missed breaking change in BatchServiceClient
- Resolves #274
2019-04-16 15:50:14 +00:00
Fred Park 6767ce2326
Update recipe README 2019-04-04 08:10:45 -07:00
Fred Park c39a919bf7
Add more Slurm partition settings
- Partition preemption settings
- Partition other options
2019-04-02 15:59:36 -07:00
Fred Park ec7af5b7c1
Various updates
- Update docs
- Update azure-batch dependency
- Set Slurm scheduling option defer mode
2019-03-22 14:50:26 -07:00
Fred Park 97dac7d5aa
Update blobxfer to 1.7.1
- Update some docs
2019-03-05 08:05:02 -08:00
Fred Park 2301e20cfc
Tag for 3.7.0 release 2019-02-28 14:03:35 -08:00
Fred Park c42652d504
Migrate Docker builds to Azure DevOps 2019-02-28 13:48:21 -08:00
Fred Park 163f1d0cb6
Suspend and restart support for Slurm clusters 2019-02-28 13:48:20 -08:00
Fred Park ed5a21d416
Doc updates
- Add support for CentOS 7.6 native
2019-02-28 13:48:16 -08:00
Fred Park 81a260f0bd
Update to Alpine 3.9
- Fix some slurm deps
2019-02-28 12:11:19 -08:00
Fred Park 6e8d2a119f
Component updates
- Update blobxfer to 1.7.0
- Update Batch Insights to 1.2.0
- Update LIS
- Update NV driver to 410.92
- Update NC/ND driver to 410.104
2019-02-28 12:11:19 -08:00
Fred Park f5cde186bf
Update Docker CE to 18.09.2
- runc CVE-2019-5736
2019-02-28 12:11:18 -08:00
Fred Park 4fa60af37a
Fix accelerated networking provisioning
- Add pool exists command
- Add recreate option to pool add
2019-02-28 12:11:18 -08:00
Fred Park 314037f76f
Slurm on Batch feature
- Package and use Slurm 18.08 instead of default from distro repo
- Slurm "master" contains separate controller and login nodes
- Integrate RemoteFS shared file system into Slurm cluster
- Auto feature tagging on Slurm nodes
- Support CentOS 7, Ubuntu 16.04, Ubuntu 18.04 Batch pools as Slurm
  node targets
- Unify login and Batch pools on cluster user based on login user
- Auto provision passwordless SSH user on compute nodes with login user
  context
- Add slurm cluster commands, including orchestrate command
- Add separate SSH for controller, login, nodes
- Add Slurm configuration doc
- Add Slurm guide
- Add Slurm recipe
- Update usage doc
- Remove deprecated MSI VM extension from monitoring and federation
- Fix pool nodes count on non-existent pool
- Refactor SSH info to allow offsets
- Add fs cluster orchestrate command
2019-02-28 12:11:10 -08:00
Fred Park d33f06457d
Pin flake8 to 3.6.0 2019-02-26 13:05:49 -08:00
Fred Park 6f59ce07e1
Add missing support files for binaries
- Resolves #258
2019-01-16 13:04:40 -08:00
Fred Park 15aa25bb1f
Update to Python 3.7.2
- Windows Docker images
- Pre-built Binaries
2019-01-16 13:04:34 -08:00
Fred Park a30cb674ca
Migrate to Azure Batch Python SDK 6.0.0
- Fix breaking changes
- Update dependencies
- Gate some debug messages behind the verbose flag
2019-01-16 13:03:30 -08:00
Fred Park 1253ee0062
Component updates
- Update blobxfer to 1.6.0
- Update Singularity to 2.6.1
- Update Docker CE to 18.09.1
- Move monitor setup to after GPU driver installation
2019-01-16 13:03:30 -08:00
Fred Park 070a9892b9
Force ASCII encoding on version.txt
- Resolves #260
2019-01-16 13:03:29 -08:00
Fred Park 458c69e6a9
Improve task factory generation/submission speed
- Amortize copy cost over single deep copy and pop
- Add more feedback during generation/submission for large task
  factories and task sets
2019-01-16 13:03:29 -08:00
Fred Park 775f602e5a
Add Batch Insights integration
- Resolves #259
2019-01-16 13:03:29 -08:00
Fred Park e210e6032f
Re-organize environment variables on node prep
- Resolve #252
2019-01-16 13:03:28 -08:00
jackpimbert 30ec7ad03b
Add custom environment variable options to pool configuration (#253)
* Add env vars option to pool configuration

- Add the option to include custom environment variables in the pool
  configuration
- This allows users to setup Batch environment variables for the start
  task
- This in turn could allow for the ability to use batch insights

* Add env var spec to pool schema for validation

- CHANGELOG updated with pool env var feature

* Add a keyvault option for pool env vars

- Added a keyvault option to the pool configuration
- Updated docs, schemas and templates accordingly
- Changed schema type from `str` to `text` for env vars

* Add keyvault client check

- Add a keyvault client check to ensure a valid client is used when
  using keyvault to add env vars to the start task.
2019-01-16 13:03:28 -08:00
Fred Park f53eee7bbd
Block job submission on non-active pools
- Resolves #251
2019-01-10 13:47:08 -08:00
Fred Park 1a3a6ebd88
Fix BeeGFS broken links 2018-12-17 08:38:22 -08:00
Fred Park 788b786797
Update max task active limitation 2018-12-11 08:29:06 -08:00
Fred Park 296c39b39b
Add Windows recipes
- Resolves #246
2018-12-10 10:58:47 -08:00
Fred Park 17e26f091b
Fix minor nodeid None issues 2018-12-10 10:54:54 -08:00
Fred Park 4bc4ab8557
Fix typos 2018-12-10 10:54:54 -08:00
Fred Park 70f2c80de0
Update Dockerfiles 2018-12-03 10:59:38 -08:00
Fred Park 99f15879cd
Tag for 3.6.1 release 2018-12-03 09:04:08 -08:00
Fred Park eea4286724
Update dependencies
- NC/ND driver to 410.79
- NV Grid driver to 410.71 with CUDA10 support
- LIS
2018-12-03 09:03:49 -08:00
Fred Park ad4e832363
Fix Windows node prep script 2018-12-03 09:03:48 -08:00
Fred Park 2adf9456ad
Fix NV provisioning 2018-11-29 09:39:34 -08:00
Fred Park 70532fa4ae
Add Genomics recipes
- BLAST and RNASeq pipelines
- Fix adding tasks to an existing job with existing merge tasks
- Add support for force_enable_task_dependencies at the job level
- Fix doc typos
2018-11-29 08:58:01 -08:00
Fred Park 0d97a8cee6
Update Docker base images for Windows 2018-11-29 08:58:01 -08:00
Fred Park 387fd14d54
Fix --tail console output
- Fix occurrences where the stream would occassionally repeat characters
- Allow incremental unicode decoding of the stream
2018-11-29 08:58:01 -08:00