MLOS/.devcontainer/scripts
Brian Kroth d50f1f4e83
MacOS test fixups and CI pipelines (#876)
# Pull Request

## Title

Fixes some test issues for Mac OS environments and enables CI pipelines.

---

## Description

- [x] mypy fixups
- [x] Enable MacOS build pipeline
- [x] ~Enable MacOS devcontainer tests~
  - [x] ~Needs `docker` enabled (which isn't currently possible)~
- [x] ~Related: add *basic* devcontainer build test and run for Windows
as well (also isn't currently possible)~
- [x] Fixes MacOS tests
  - [x] Address parallel runner issues with output dir cleanup
  - [x] Fix ssh tests (also a parallelization issue)
- [x] ~Publish multi arch docker images (amd64 and arm64) (can't see
above)~

Closes #875 
  
---

## Type of Change

- 🛠️ Bug fix
-  New feature
- 🧪 Tests

---

## Testing

Locally tested on MacOS, Windows, and Linux.

---

## Additional Notes (optional)

Leaves some currently disabled stub code for creating devcontainers in
the future on MacOS and Windows hosts. Unfortunately this isn't
currently possible with the Github Action runners.


Also did some cosmetic renaming of the CI pipelines for better
descriptions.
This affects the required tests to pass in the repo settings.
Will adjust that after this is approved.

---
2024-11-05 11:24:50 -06:00
..
common Introduce pyproject.toml build changes (#764) 2024-07-02 17:31:31 -05:00
README.md Update container purge docs (#871) 2024-10-14 21:42:42 +00:00
prep-container-build MacOS test fixups and CI pipelines (#876) 2024-11-05 11:24:50 -06:00
prep-container-build.cmd Merged PR 862: Add devcontainer support and other dev environment improvements 2023-02-07 21:43:01 +00:00
prep-container-build.ps1 devcontainer build fixups for macOS clients (#874) 2024-10-21 10:17:37 -05:00
run-cspell.sh Merged PR 1054: Prepwork for OSSing 2023-03-17 21:20:39 +00:00
run-devcontainer.ps1 MacOS test fixups and CI pipelines (#876) 2024-11-05 11:24:50 -06:00
run-devcontainer.sh devcontainer build fixups for macOS clients (#874) 2024-10-21 10:17:37 -05:00
run-markdown-link-check.sh Introduce basic experiment visualization module mlos_viz via dabl (#624) 2024-01-16 18:48:20 +00:00

README.md

Devcontainer setup scripts

This directory contains some script variants to preparing the devcontainer environment.

.env file hack

Allow touching (creating but not altering) a .env file prior to devcontainer start so those variables can be populated into the devcontainer environment.

See Also: https://github.com/microsoft/vscode-remote-release/issues/4568

mlos_deps.yml file hack

When building the devcontainer image, we don't want to include the MLOS source code initially, just its dependencies, so we filter out the MLOS source code from the mlos.yml file when building the image and keep the context to that smaller set of files.

When the devcontainer starts, we map the source code (and hence the mlos.yml file) into the devcontainer and then run conda env update to install the rest of the MLOS dependencies.

This makes the devcontainer base more cacheable.

Publishing the devcontainer image for cache reuse

The build pipeline publishes the devcontainer image to the Azure Container Registry (ACR) so that it can be reused by other builds.

The secrets for this are stored in the pipeline, but also available in a Key Vault in the MlosCore RG.

One can also use az acr login to push an image manually if need be.

Image cleanup

To save space in the ACR, we purge images older than 7 days.

#DRY_RUN_ARGS='--dry-run'

PURGE_CMD="acr purge --filter 'devcontainer-cli:.*' --filter 'mlos-devcontainer:.*' --untagged --ago 30d --keep 3 $DRY_RUN_ARGS"

# Setup a daily task:
az acr task create --name dailyPurgeTask --cmd "$PURGE_CMD" --registry mloscore --schedule "0 1 * * *" --context /dev/null

# Or, run it manually.
az acr run --cmd "$PURGE_CMD" --registry mloscore --context /dev/null