Azure
/
batch-shipyard
зеркало из
1
0
Форкнуть 0
Код Задачи Пакеты Проекты Релизы Вики Активность
84 коммитов 6 Ветки 71 Тег 8,9 MiB
Граф коммитов

50 Коммитов

Автор SHA1 Сообщение Дата
jasper-schneider b28df08730 Reorder parsing of --configdir switch (#2) 
* Reorder parsing of --configdir switch

* Set args.pool when only --configdir is supplied
 2016-09-22 13:27:21 -07:00
Fred Park a4e271d93f Increase max stream file time to 30min 2016-09-21 13:05:46 -07:00
Fred Park 6aea6782ce Update Azure File DVD to 0.5.1 
- Update quickstart to accommodate choice
- Change STANDARD_F1 to STANDARD_D1_V2 for some recipes
 2016-09-21 09:23:00 -07:00
Fred Park 1188a1e885 Fix shipyard container detach/cleanup 
- Add @FIRSTRUNNING task id for streamfile/gettaskfile
 2016-09-18 17:01:42 -07:00
Fred Park cd9a4e5bb5 Deterministic remote login settings output 2016-09-16 22:20:07 -07:00
Fred Park d4d32fb699 Fix inter-node comm omitted keyerror 2016-09-16 17:54:49 -07:00
Fred Park a2f99720f7 Add --configdir argument for convenience 2016-09-16 14:39:44 -07:00
Fred Park ea702d3a43 Add quickstart doc 
- Disable ssh tunnel creation without public key on Windows
 2016-09-16 13:45:12 -07:00
Fred Park 9d12dab8be Fix cascade start issue without private registry 
- Add GlusterFS support for ubuntu and opensuse/sles
- Add --filespec and --verbose parameters
 2016-09-16 11:46:59 -07:00
Fred Park 08204092be Add CentOS GlusterFS support 
- Update recipes
 2016-09-15 12:47:43 -07:00
Fred Park 646cff6631 Add TensorFlow-Distributed recipe 
- Fix SSH user expiry within 1 day
- Fix some README/dockerfile typos
 2016-09-13 11:43:28 -07:00
Fred Park af98bdfb57 Add sample configs for all existing recipes 
- Fix temp file creation for cross-platform
 2016-09-09 13:49:59 -07:00
Fred Park e8d5e7a8a3 Automatically detect nvidia driver version 
- Fix azure-storage dependencies for non-shipyard docker image setup
- Add no-install-recommends to apt-gets in node prep
 2016-09-08 21:06:21 -07:00
Fred Park 0338fc612f Add gettaskfile/getnodefile actions 
- Add .gitattributes to designate text files eol as LF
- Update configuration doc with required/optional tags
 2016-09-08 20:15:11 -07:00
Fred Park 028768a61a Add CNTK recipes 
- Add TCP optimization
- Fix job autocompletion
- Update azure-storage requirement to 0.33.0
 2016-09-07 21:40:57 -07:00
Fred Park eea254aeb9 Add job auto-completion for multi-instance tasks 2016-09-06 10:32:42 -07:00
Fred Park ada1feb00a Add GPU support 2016-09-02 01:42:54 -07:00
Fred Park 6a414ca829 Add multi-instance task doc 
- Ensure --net=host for multi-instance task coordination command
 2016-09-01 14:00:14 -07:00
Fred Park 53c6e5a673 Add del mi cleanup job and stream file actions 
- Add base recipes readme
 2016-09-01 09:07:05 -07:00
Fred Park a666d7d9f4 Add explicit inter node comm property for pool 
- Allow inter node comm to work independently of p2p transfers
 2016-08-31 22:20:18 -07:00
Fred Park 75a27746a4 Complete guide/docs 
- Add pool_current_dedicated support for multi-instance tasks
 2016-08-31 20:44:44 -07:00
Fred Park 7f49641074 First part of the guide/docs 
- Modify placement of some configuration settings
 2016-08-31 15:35:33 -07:00
Fred Park d53436b6ca Prep for ghpages 
- Update for pathlib2
- Add bare start point for docs
 2016-08-31 11:16:43 -07:00
Fred Park e813bbeae7 Enforce VM size check for infiniband tasks 2016-08-31 09:07:33 -07:00
Fred Park c5368207cd Add more features to pool/tasks 
- Add infiniband support
- Add max tasks per node
- Properly handle multi-instance tasks with docker run <-> exec
- Add docker multi-instance cleanup helper
 2016-08-31 02:21:50 -07:00
Fred Park 351800344d Add multi-instance support for tasks 2016-08-30 15:31:23 -07:00
Fred Park 0d48919afa Azure file dvd support for all supported hosts 
- Add default init scripts to avoid surprise config changes
- Add block flag to pool for image ready
 2016-08-29 12:12:50 -07:00
Fred Park 35fb3f588b Add support for more host OSes 
- Ubuntu 14.04, Debian 8, CentOS 7.x, RHEL 7.x, OpenSUSE 13.2/42.1,
  SLES 12/12-sp1
- Improve graphing
- Prevent metadata clear on existing pool
 2016-08-28 19:43:53 -07:00
Fred Park 4cb1f79c07 Update docstrings and typing information 
- Add MIT license text to all py files
 2016-08-27 17:42:36 -07:00
Fred Park 703e9e7fd1 Reorganize project 2016-08-27 11:35:32 -07:00
Fred Park 0da28d7e0a Add stderr redirect to logger 
- Refactor some common functions
- Rename services table to images
 2016-08-27 11:30:55 -07:00
Fred Park e9ce32e69f Fix shipyard container issues 
- Downgrade libtorrent to 1.0.9 due to DHT issues
- Add task dependency support
- Add CONTRIBUTING.md, requirements.txt and .travis.yml
 2016-08-26 23:09:10 -07:00
Fred Park aa4add34d5 Support shipyard as docker container 
- Add Dockerfile
- Add command file for container
 2016-08-26 15:34:30 -07:00
Fred Park bc30023330 Scale fixes 
- Add timing recording toggle
 2016-08-25 13:07:57 -07:00
Fred Park 1fbad08ee5 Add ssh docker user/tunnel generation 
- Compact some vars
 2016-08-24 20:32:53 -07:00
Fred Park 359dfdad85 Use proper logging in shipyard and cascade 
- Fix shipyard.py for Python2.7 compatibility
- Allow option to reboot nodes that go into start task failed state
- Correctly pin gr-done events
- Reduce chattiness of torrent info dumps
 2016-08-24 15:36:23 -07:00
Fred Park 39dd9bf5c6 Fix storage env var issues 
- Add support for additional resource files on docker jobs
 2016-08-24 09:09:37 -07:00
Fred Park a99bee7100 Pass through more storage endpoint info 
- Compact env vars
- Azure file endpoint
- Docker private registry realm
 2016-08-23 15:32:49 -07:00
Fred Park ee242c21d6 Split configuration files further 
- Allow docker public hub passthrough in private context
- Pass config through in more places in shipyard
 2016-08-23 14:50:17 -07:00
Fred Park 6fb2d7e221 Add azurefile docker volume driver support 
- Add auto packaging of registry:2 docker image if not present
 2016-08-19 15:13:33 -07:00
Fred Park 8182a73847 Add docker job/task support 
- Fix cascade compression bug
- Add full job and task spec support
 2016-08-19 11:28:29 -07:00
Fred Park f1a666db41 Add redirect for torrents to blob 
- Always install private registry to every node
- Fix uncompressed torrenting
 2016-08-18 15:17:04 -07:00
Fred Park e4f58c6ff4 Allow non-p2p and no dpr modes 
- Add jp block script
- Fix some data graph bugs
- Begin fixing non-reproducible .tar.gz docker save images
 2016-08-17 08:30:53 -07:00
Fred Park d4c44f811c Add more configurable options for P2P mode 
- Add WIP of graph.py
 2016-08-12 15:31:05 -07:00
Fred Park d06d3649a0 Configuration changes 
- Move more private registry settings from hardcode to config
- Split config file into two
 2016-08-12 09:21:33 -07:00
Fred Park 0271e54b1a Upload perf timings to table 2016-08-09 15:23:15 -07:00
Fred Park 2e2b2c0f96 Migrate to DHT table, load/register functionality 2016-08-09 12:47:58 -07:00
Fred Park 83a422c64c Updates for single session/DHT fixes 2016-08-08 09:38:12 -07:00
Fred Park 553380df8a Break out private registry setup from cascade 
- Fix torrent start, queue msg get and torrentinfo table merge
 2016-07-20 13:12:20 -07:00
Fred Park 41a6f7c258 Populate with initial code 2016-07-20 13:12:07 -07:00