# Pull Request
## Title
Fixes some test issues for Mac OS environments and enables CI pipelines.
---
## Description
- [x] mypy fixups
- [x] Enable MacOS build pipeline
- [x] ~Enable MacOS devcontainer tests~
- [x] ~Needs `docker` enabled (which isn't currently possible)~
- [x] ~Related: add *basic* devcontainer build test and run for Windows
as well (also isn't currently possible)~
- [x] Fixes MacOS tests
- [x] Address parallel runner issues with output dir cleanup
- [x] Fix ssh tests (also a parallelization issue)
- [x] ~Publish multi arch docker images (amd64 and arm64) (can't see
above)~
Closes#875
---
## Type of Change
- 🛠️ Bug fix
- ✨ New feature
- 🧪 Tests
---
## Testing
Locally tested on MacOS, Windows, and Linux.
---
## Additional Notes (optional)
Leaves some currently disabled stub code for creating devcontainers in
the future on MacOS and Windows hosts. Unfortunately this isn't
currently possible with the Github Action runners.
Also did some cosmetic renaming of the CI pipelines for better
descriptions.
This affects the required tests to pass in the repo settings.
Will adjust that after this is approved.
---
# Pull Request
## Title
A test configuration for the environment based on local shell scripts
---
## Description
This is a sample MLOS configuration (CLI config, tunables, environment,
and the scripts) that launches a local environment and invokes local
shell scripts to set it up and run trials.
---
## Type of Change
This is purely a config-based update. We might add unit tests around
that setup later.
- ✨ New feature
- 📝 Documentation update
- 🧪 Tests
---
## Testing
Unit tests that validate the environment and run test experiments are
included in this PR.
To test the environment manually, run:
```bash
mlos_bench \
--config mlos_bench/mlos_bench/tests/config/cli/test-cli-local-env-bench.jsonc \
--globals experiment_test_local.jsonc \
--tunable_values tunable-values/tunable-values-local.jsonc
```
---
## Additional Notes (optional)
This setup is supposed to serve as an example of other local
environments with shell scripts.
---
# Pull Request
## Title
Add support for Python 3.13 and mark is as the expected default.
---
## Description
Upstream changed the default version of python, so this addresses
changes in checks in the pipelines.
So far it seems to "just work (tm)"
---
## Type of Change
- 🛠️ Bug fix
- ✨ New feature
- 🧪 Tests
---
## Testing
Usual CI tests.
---
# Pull Request
## Title
Fixups to the devcontainer build for macOS clients.
---
## Description
- Use a more portalable random number generator
- Address some differences in docker.sock privileges.
- Address differences in `stat` arguments.
- Switch to wget mode for conda installation to workaround lack of arm64
apt repo.
- When pulling base images to prime cache, use the appropriate
architecture (for Windows we only support amd64 for now).
- Remove `:latest` from `cache-from` args for `podman` compliance.
- Address some differences in `sed` syntax.
- Fixes#873
---
## Type of Change
- 🛠️ Bug fix
---
## Testing
- local MacBook testing
- CI testing for Linux
---
## Additional Notes
Doesn't currently do builds for arm64 platform in the pipeline.
Can work towards addressing that in the future.
# Pull Request
## Title
Combines container build steps to reduce devcontainer size
substantially.
---
## Description
Reduces container size from ~5.7GB to ~2.7GB by combining the following
steps:
- install conda (previously in a base image layer)
- create the base mlos environment
This reduces the space substantially because conda's usual attempt to do
hardlinks across environments is actually able to work.
The downside is that changes to base package level requirements require
reinstalling all of conda again (i.e., the single large combined
container layer is less cacheable).
Should help with issues like:
https://github.com/Microsoft-CISL/sqlite-autotuning/issues/7
---
## Type of Change
- 🔄 Refactor
- Dev environment
---
## Testing
- `time make devcontainer`
- "Rebuild devcontainer" inside VSCode
- `docker image ls` to check the sizes
---
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
# Pull Request
## Title
Quick fixups to the documentation build.
---
## Description
- Expands one of the ignored warnings.
- Reformats the sphinx conf for black and pylint.
- Removes a warning from that file around redefinition of
`html_theme_path`.
- Adds a dependency requirement tweak for `pyparsing` when installing
under python 3.8 (unrelated other CI bug).
- Adds a workaround to a mypy false positive with in checking `np.e` as
a "deleted variable" (e.g., `e` is used in a try/except block)
---
## Type of Change
- 🛠️ Bug fix
- 📝 Documentation update
---
## Testing
- Ran `make doc` and `make check-doc`.
- Ran `make notebook-exec-test`.
---
## Additional Notes (optional)
This is a quick fix to get the CI pipeline working again.
The more complete fix to remove warning ignores from doc generation,
improve cross reference linking, etc. will be handled later in #869
---
# Pull Request
## Title
Allows an empty dictionary (object) to represent the default tunable
values for tunable params.
---
## Description
This should make it easier to maintain configs that loop over a chosen
set of tunable values, where some of the tunable values are the default
values without needing to copy/paste those values from the tunable
params definitions.
- See Also: [discussion in
#855](https://github.com/microsoft/MLOS/pull/855#discussion_r1786896189)
---
## Type of Change
- ✨ New feature
---
## Testing
Extends existing testing infrastructure to check that this works.
---
# Pull Request
## Title
Include `strtobool` from a now deprecated version of python.
---
## Description
- Closes#865
---
## Type of Change
- 🛠️ Bug fix
---
---------
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
A comparison of the existing action run command API vs new managed
version:
https://learn.microsoft.com/en-us/azure/virtual-machines/run-command-overview
The main benefits of moving to the new managed version are:
- Much longer remote execution timeouts can be set beyond the current 90
mins. (Supposedly supports up to days)
- Returns the exit code of script ran.
- Can run multiple managed run commands in parallel.
- Supports storing large script output into blob storage (not used in
this PR)
The first point is really needed for steps like the benchbase loading
phase, where loading TPCC scale factor 200+ is a struggle due to hitting
the previous 90 min timeout limit.
This should be possible by setting `"pollTimeout": 5400` to much higher
values as needed in the relevant VM service configs.
---------
Co-authored-by: Eu Jing Chua <eujingchua@microsoft.com>
Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Avoids the following error that results when the [latest
libpq](https://anaconda.org/conda-forge/libpq) (17.0) is installed from
conda-forge:
```
Error: pg_config --includedir is empty
```
Refactors tests in preparation for testing for #720.
Adds `status()` output for MockEnv as well, though with a pending FIXME
on improving the Status type in the status output that we'll address in
a future PR.
Minor adjustments to both display in vscode and quiet some pylint
warnings.
Note: in the future I'd like to get some changes from #330 incorporated
to allow more self-contained config examples, with relative paths
support, added, which means changing a pile of APIs, in which case we
can probably make all of these require named position values and remove
those exceptions.
The `mlos_bench` CLI wrapper exits non-zero currently even on success.
This PR adds some basic sanity checks and makes sure we exit 0 when the
process looks roughly OK.
Further work to be expanded in #523.
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
Merge after (or instead of) #835
diff from #835 :: https://github.com/motus/MLOS/pull/15/filesCloses#803
---------
Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>
Co-authored-by: Brian Kroth <bpkroth@microsoft.com>
Closes#803
> Future PR will rename the config schema in order to reduce confusion
on the change in semantics, but also keep this PR smaller.
---------
Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>
Co-authored-by: Brian Kroth <bpkroth@microsoft.com>
- [x] enable notebook execution in CI pipelines to prevent these
mistakes in the future
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
This PR enables Python 3.12 in Windows environment and does not restrict
Python version in the default environments on Windows and Linux. Note
that although `mlos-3.12.yml` exists, its usage is still commented out
in `linux.yml`, because Python 3.12 is still not available on GitHub CI.
Issue #832 opened to track it
P.S. Also, we now use a lighter `matplotlib-base` instead of
`matplotlib` proper
~To be merged after #802~
---------
Co-authored-by: Brian Kroth <bpkroth@microsoft.com>
To use token-based authentication for `ShareClient`, I think we should
be passing in the credential object derived from `TokenCredential` (in
our case, some instance of `DefaultAzureCredential`.
Previously, we were passing specific string tokens to the `credential`
argument, which is being intepreted as a SAS token.
This leads to errors like:
`azure.core.exceptions.ClientAuthenticationError: Server failed to
authenticate the request. Make sure the value of Authorization header is
formed correctly including the signature.`
[ShareClient](https://learn.microsoft.com/en-us/python/api/azure-storage-file-share/azure.storage.fileshare.shareclient?view=azure-python)
documentation on the `credential` argument.
By passing in the whole `TokenCredential` object, I believe
`ShareClient` will manage the token lifecycle and we won't need to do so
as mentioned in #818.
---------
Co-authored-by: Eu Jing Chua <eujingchua@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Temporary fix to conda env to make it match the `pip` `setup.py`
restrictions for now.
Eventually, we'll wan to relax these, along with increased `numpy`
versions, etc.
See Also #826
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
Summary of changes:
* [x] Update dependencies to ConfigSpace >= 1.0.
* [x] Update dependencies to SMAC3 >= 2.2.
* [x] Migrate to new ConfigSpace API.
* [x] Better definition of ConfigSpace fixture in unit tests.
* [x] Fix mypy and pylint issues triggered by the migration.
Notes:
* Side effect: **Works with Python 3.12!**
* ~Supersedes #799~
* ~Depends on upcoming SMAC3 release with [SMAC3
#1124](https://github.com/automl/SMAC3/pull/1124) merged in and a new
release cut with it.~
* Quantization will be added back later:
* #803
* ~Tests fail due to LlamaTune issue:~
* #805Closes#727
---------
Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>
Co-authored-by: Brian Kroth <bpkroth@microsoft.com>
Minor tweaks to provide common attributes like
```python
mlos_core.__version__
# or
mlos_core.VERSION
```
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Revamp of #806 with the fix as well.
Adds code to handle appropriate scaling and type conversion and checking
of the inverted config.
Splits that code out to a separate method for individual configs to make
pylint happy and improve code readability.
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Eu Jing Chua <chuaeujing@gmail.com>
Co-authored-by: Eu Jing Chua <eujingchua@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
Test for #782
Makes `new_trial` a wrapper with a sanity check and renames the previous
`new_trial` abstractmethod to `_new_trial`.
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Make `AzureFileShareService` class use token-based credential instead of
the access key. This PR is part of #777
**NOTE:**
* We have to use late initialization of the `_share_client` to avoid
issues with JSON schema validation tests.
* More PRs will follow:
* Use to `azcopy` on the remote VMs instead of mounting the file share
using the access key
* Remove references to `"storageAccountKey"` from configurations and
tests and document the new mechanism
---------
Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>
Closes#785
Also, add more unit tests to make sure the optimizer handles string
inputs correctly.
---------
Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>
These changes are mainly to add flexibility to the auth service in
supporting azure authenticating via:
- A personal identity via az CLI for the local dev environment. This
will still run into expiring issues, but is probably fine for local dev.
- A personal or managed identity in an azure hosted environment. The
flexibility to use the managed identity allows us to avoid the expiring
issue without needing a separate SP, key-vault and cert setup, aligning
with new security recommendations.
---------
Co-authored-by: Eu Jing Chua <eujingchua@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Quiet a handful of messages from a new version of pylint and fix some
existing exceptions that got reformatted incorrectly.
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Follow on work to #766.
This enabled both formatters and applies their changes to the repo.
Additionally, since `black` does not make changes to comments nor
docstrings, we also enable `docformatter` to reformat docstrings which
better aligns with `pydocstyle` rules as well.
Without this additional change (and some manual fixups), `pycodestyle`
and `pylint` would still complain about line lengths, for instance.
Finally, we make a minor adjustment to the max line length setting it to
99 (which is also accepted and mentioned in pep8) instead of 88 to avoid
some comment (especially linter overrides) wrapping.
ConfigSpace 1.0.0 was just released and includes some important changes
that we'll eventually want to support, but in the interest of making
progress on other things for the time being, this PR pins us at the
previous stable version we were using.
https://github.com/automl/ConfigSpace/compare/v0.7.2...v1.0.0
Brian Kroth
Sergiy Matusevych
Eu Jing Chua