Граф коммитов

80 Коммитов

Автор SHA1 Сообщение Дата
Fred Park e9130f83f4
MCR migration
- Migrate images to Microsoft Container Registry
- Fix Shellcheck issues
- Related to #278
2019-08-14 03:23:03 +00:00
Fred Park 290209381e
Update Dependencies
- Update NVIDIA compute driver to 418.67
- Update NVIDIA grid driver to 430.30
- Update Batch Insights to 1.3.0
- Update blobxfer to 1.9.0
- Update Python dependencies
- Drop Python 3.4 support
2019-08-12 20:42:32 +00:00
Fred Park 00f1c95b1d
Update Dockerfiles to Alpine 3.10 2019-08-08 20:11:33 +00:00
Vincent Labonté 9f58ad0042 Fixes for Singularity 3 support (#285)
* Fix credentials when running task with Singularity docker:// images

* Fix Singularity cache directory's ownership

* Fix images update command

* Fix running cascade with use_shipyard_docker_image

* Remove envfile dump in task runner
2019-06-21 10:36:28 -07:00
Vincent Labonté 8293a20be3 Support multiple Singularity registries (#283)
* Add support for multiple singularity registries (://docker and ://oras)

* Resolve PR comments

* Resolve PR comments
2019-06-13 10:22:18 -07:00
Vincent Labonté 5307f1779d Fix image update command (#281)
* Create one log file per container mode

* Make singularity 3 work

* Minor fixes

* Fix cascade with docker image and singularity image

* Add capability to pull from library://

* Add singularity signed images to config file

* Add singularity signed images to the global resource table

* Pull and verify signed singularity images

* Put the singularity sypgp directory in the mount directory

* Add ability to provide key file to verify a singularity image

* Resolve PR comments

* Fix Singularity registry credemtials

* Extract cascade logic from nodeprep

* Re-run cascade if the image update command has no specified image

* Fix prefix errors when using shipyard docker image

* Make sure that the cascade log files are not overridden

* Fix wrong parameter name

* Clarify error message when trying to update images on Windows

* Update documentation

* Fix checks

* Resolve PR comments
2019-06-10 10:35:28 -07:00
Vincent Labonté 305d376cdc Support Singularity signed image verification (#280)
* Create one log file per container mode

* Make singularity 3 work

* Minor fixes

* Fix cascade with docker image and singularity image

* Add capability to pull from library://

* Add singularity signed images to config file

* Add singularity signed images to the global resource table

* Pull and verify signed singularity images

* Put the singularity sypgp directory in the mount directory

* Add ability to provide key file to verify a singularity image

* Resolve PR comments

* Fix Singularity registry credemtials
2019-06-05 11:14:37 -07:00
Vincent Labonté f9d0ad9a7f Initial support for Singularity 3 and SIF (#279)
* Create one log file per container mode

* Make singularity 3 work

* Minor fixes

* Fix cascade with docker image and singularity image

* Add capability to pull from library://
2019-05-29 14:15:22 -07:00
Fred Park 509834b3fb
Cascade Docker/Singularity image split 2019-05-23 21:29:24 +00:00
Vincent Labonté a68579c095 Prepare for Singularity3 work (#276)
* Remove torrent functionality

* Remove torrent storage

* Fix singularity permissions

* Add container mode in cascade.py

* Fix errors

* Fix PR comments

* Fix flake8 errors
2019-05-22 14:46:44 -07:00
Fred Park 81a260f0bd
Update to Alpine 3.9
- Fix some slurm deps
2019-02-28 12:11:19 -08:00
Fred Park 6e8d2a119f
Component updates
- Update blobxfer to 1.7.0
- Update Batch Insights to 1.2.0
- Update LIS
- Update NV driver to 410.92
- Update NC/ND driver to 410.104
2019-02-28 12:11:19 -08:00
Fred Park 1253ee0062
Component updates
- Update blobxfer to 1.6.0
- Update Singularity to 2.6.1
- Update Docker CE to 18.09.1
- Move monitor setup to after GPU driver installation
2019-01-16 13:03:30 -08:00
Fred Park 70f2c80de0
Update Dockerfiles 2018-12-03 10:59:38 -08:00
Fred Park 2519f3cedd
Update dependencies 2018-11-19 11:20:08 -08:00
Fred Park a53bb2a044
Fix Singularity issues in latest update 2018-11-05 11:24:17 -08:00
Fred Park ab9cc70828
Update build to Python 3.7.1
- Update Windows Docker images to Python 3.7.1
- Fix flake8 errors
- Fix shellcheck errors
- Various build updates and fixes
2018-10-30 14:24:31 -07:00
Fred Park 584dada9f8
Update Singularity and Alpine
- Update to 3.8, rebuild 3.7 due to CVE
- Update Singularity to 2.6.0
2018-09-20 08:48:15 -07:00
Fred Park f73f1e25a7
Fix shellcheck errors 2018-09-18 13:56:26 -07:00
Fred Park 1d666ae6aa
Update dependencies
- Update blobxfer to 1.5.0
2018-09-18 13:56:26 -07:00
Fred Park 7a92ed39b4
Improve Docker login robustness
- Omit registry info for native container pools when
  delay_docker_image_preload is enabled
- Allow cascade image management to scale to large node counts
2018-07-31 10:18:49 -07:00
Fred Park e069e72564
Support Docker image preload delay
- This option is only available for Linux non-native pools. All other
  pool types ignore this option.
2018-07-28 18:44:57 -07:00
Fred Park 7060366213
Update dependencies 2018-07-17 11:18:20 -07:00
Fred Park 66e77ac397
Update dependencies
- Fixes in scripts for cascade and monitor cert renewal
2018-06-27 15:32:22 -07:00
Fred Park 3f30ba8d07
Support a fallback registry for system images
- Resolves #217
- Add misc mirror-images command
- Pass Singularity version to bootstrap
- Fix GlusterFS on compute provisioning, resolves #220
2018-06-26 12:22:09 -07:00
Fred Park b77a147766
Continue Prometheus integration support
- Add nginx reverse proxy and letsencrypt cert support
- Add let's encrypt options
- Add picket into compose
- crontab cert renewal
- Add inbound rule management for temporary ACME challenge on port 80
- Update to node exporter 0.16
- Fixup various issues
2018-06-04 09:04:30 -07:00
Fred Park 0bf9399061
Fix some post-release issues
- Shellcheck-induced regressions in containers
- Split Docker image build into own build env on appveyor
- Update to Python 3.6.5 for cargo Docker image build on Windows
2018-05-02 10:24:38 -07:00
Fred Park 7c5ca646dc
Update Singularity to 2.5.0 2018-05-01 10:57:15 -07:00
Fred Park 383bcfd7c0
Update Singularity to 2.4.5 2018-04-04 13:34:35 -07:00
Fred Park c1a92e4138
Fix scripts to be Shellcheck clean (#178) 2018-04-04 13:34:09 -07:00
Fred Park 2b3ecac70b
Update Singularity to 2.4.4 2018-03-19 14:07:12 -07:00
Fred Park a195d7e242
Update dependencies 2018-02-28 15:08:33 -08:00
Fred Park b4e6e4320d
Various updates
- Fix image update to work in multi-instance mode with registry logins
- Allow CentOS 7.3 provisioning to continue to work
- Allow CentOS-HPC 7.1 provisioning
- Add CentOS 7.4 support
- Add Debian 9 support
- Update dependencies
2018-02-16 09:18:34 -08:00
Fred Park 5853b4787f
Update Docker images
- Update to alpine 3.7
- Update Windows images to Python 3.6.4
- Update libtorrent image
- Update to Singularity 2.4.2
2018-01-22 14:14:39 -08:00
Fred Park f9765c98a2 Fix file naming for Docker image under singularity 2017-11-15 10:01:13 -08:00
Fred Park 17497ff06d Fix non-Ubuntu/CentOS cascade failures 2017-11-13 09:25:25 -08:00
Fred Park 90283298e6 Update dependencies 2017-11-10 09:23:15 -08:00
Fred Park 5afa97de45 Fix default Singularity image names with tags 2017-11-09 13:07:10 -08:00
Fred Park 5b2af24f00 Retry image configuration errors
- Add TensorFlow-GPU Singularity recipe
2017-10-29 20:29:34 -07:00
Fred Park 6da607c9b9 Multi-instance/IB support for Singularity tasks
- Make cascade work in Docker container
2017-10-22 13:59:35 -07:00
Fred Park 48172e115e Add initial Singularity task support
- Auto-GPU
- Fix ownership issues with Singularity image pre-load
2017-10-20 23:10:12 -07:00
Fred Park 4e5d5abf6b Add Singularity support into cascade
- Remove singularity suport in native container support pools as it's
  impossible to execute a singularity container in this mode
2017-10-17 18:51:27 -07:00
Fred Park 607bfd252e Migrate to storage split library
- Remove queue deletion code
- Resolves #133
2017-10-05 21:40:50 -07:00
Fred Park 01995e97b6 Tag for 3.0.0a1 release 2017-10-04 09:26:01 -07:00
Fred Park e783744e00 Container registry logic overhaul
- Remove private registry back to Azure storage blob support (#44)
- Require fully qualified Docker image names (#106)
- Support multiple public/private registries on a single pool (#127)
2017-10-03 18:24:42 -07:00
Fred Park 9602608871 Tag for 2.9.4 release 2017-09-12 08:56:03 -07:00
Fred Park 832a32e375 Use multi-stage build for cascade 2017-08-15 19:14:37 -07:00
Fred Park e32fc4d93e Add Autopool support
- Resolves #33
- Add --poolid to storage clear and storage del
- jobs del and jobs term now cleanup storage data if autopool is
  detected
2017-07-21 11:10:03 -07:00
Fred Park 5291ff1130 Move to blob leasing for download ticketing
- Greatly increase resource file SAS expiry timedelta
- Make concurrent_source_downloads generic, remove non-p2p option
- Update Dockerfiles
- Update to latest azure-storage
2017-07-21 11:10:03 -07:00
Fred Park de45b18a67 Add backoff to cascade docker image pull retries 2017-07-01 01:25:30 -07:00