# Pull Request
## Title
Fixups to the devcontainer build for macOS clients.
---
## Description
- Use a more portalable random number generator
- Address some differences in docker.sock privileges.
- Address differences in `stat` arguments.
- Switch to wget mode for conda installation to workaround lack of arm64
apt repo.
- When pulling base images to prime cache, use the appropriate
architecture (for Windows we only support amd64 for now).
- Remove `:latest` from `cache-from` args for `podman` compliance.
- Address some differences in `sed` syntax.
- Fixes#873
---
## Type of Change
- 🛠️ Bug fix
---
## Testing
- local MacBook testing
- CI testing for Linux
---
## Additional Notes
Doesn't currently do builds for arm64 platform in the pipeline.
Can work towards addressing that in the future.
Minor adjustments to both display in vscode and quiet some pylint
warnings.
Note: in the future I'd like to get some changes from #330 incorporated
to allow more self-contained config examples, with relative paths
support, added, which means changing a pile of APIs, in which case we
can probably make all of these require named position values and remove
those exceptions.
Follow on work to #766.
This enabled both formatters and applies their changes to the repo.
Additionally, since `black` does not make changes to comments nor
docstrings, we also enable `docformatter` to reformat docstrings which
better aligns with `pydocstyle` rules as well.
Without this additional change (and some manual fixups), `pycodestyle`
and `pylint` would still complain about line lengths, for instance.
Finally, we make a minor adjustment to the max line length setting it to
99 (which is also accepted and mentioned in pep8) instead of 88 to avoid
some comment (especially linter overrides) wrapping.
Builds off of #762, #763, and #764.
Prepares rules and configs to enable isort and black formatters and
checks but doesn't enable them yet.
After these are enabled (next PR) we will reformat all files and ignore
that revision in git blame configs.
Then, we can convert configs stored in `setup.cfg` and `.pylintrc` to
the top level `pyproject.toml` and remove the older configs.
This PR builds off of #762 and #763
These are in part
1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled
in at build time as a build dependency only when conda an pip have
mismatched version issues), and
2. Modernization improvements to allow us to make better use of other
tools (e.g., `black` that only accept `pyproject.toml` files as their
configuration files).
To do so, we move some configs from `setup.py` to `pyproject.toml` for
each module.
However, to retain the ability to rewrite URLs in published README.md
files on PyPi as well as consistent version dependencies across modules
without the need to manually specify version numbers (e.g., using
`setuptools-scm`) we mark a few dependencies as dynamic and leave our
existing logic inside the `setup.py` file.
Finally, we reorganize the `version.py` file to be inside the module and
fix a few previous omissions for `mlos_viz`.
* [x] Make Scheduler class loadable from JSON configs
* [x] Update the Launcher and `run.py` to instantiate Scheduler from
JSON
* [x] Create JSON schema for the Scheduler config
* [x] Add unit tests for the new Scheduler JSON configs
Closes#700
---------
Co-authored-by: Brian Kroth <bpkroth@microsoft.com>
Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>
My VSCode adds `"."` to pytest arguments in config every time I rebuild
the container - and deletes the `--log-level=DEBUG` argument at teh same
time. With this small fix, vscode can accept the project config as we
have it in git without modifying it
Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>
Note: current includes changes from #557 as well.
Closes#563
Removes reliance on `--dist=loadgroup` and
`@pytest.mark.xdist_group(...)` to run SSH tests in a single worker by
implementing a portable file-lock based serialization method for test
service setup/teardown and separating out services that might interact
between tests.
In my local testing, this seems to solve both the "successful but
non-reported tests" issue, limited discoverability of tests in vscode
upon failure, and lack of parallel execution in all dev environments
(Windows, WSL Devcontainer, Linux).
On occassion in Windows I have seen the following error, but it is after
tests have passed, so seems to be part of pytest cleanup:
```
...
[gw4] [100%] PASSED mlos_bench/mlos_bench/tests/services/remote/ssh/test_ssh_host_service.py::test_ssh_service_reboot
=================================================================================================================================== warnings summary ====================================================================================================================================
mlos_bench/mlos_bench/tests/services/remote/ssh/test_ssh_fileshare.py::test_ssh_fileshare_upload_file_dne
C:\Users\bpkroth\.conda\envs\mlos\lib\site-packages\_pytest\threadexception.py:73: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-9 (_run_event_loop)
Traceback (most recent call last):
File "C:\Users\bpkroth\.conda\envs\mlos\lib\asyncio\windows_events.py", line 494, in finish_recv
return ov.getresult()
OSError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\bpkroth\.conda\envs\mlos\lib\asyncio\windows_events.py", line 826, in _poll
value = callback(transferred, key, ov)
File "C:\Users\bpkroth\.conda\envs\mlos\lib\asyncio\windows_events.py", line 498, in finish_recv
raise ConnectionResetError(*exc.args)
ConnectionResetError: [WinError 995] The I/O operation has been aborted because of either a thread exit or an application request
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "C:\Users\bpkroth\.conda\envs\mlos\lib\threading.py", line 1016, in _bootstrap_inner
self.run()
File "C:\Users\bpkroth\.conda\envs\mlos\lib\threading.py", line 953, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\bpkroth\src\MLOS\mlos_bench\mlos_bench\event_loop_context.py", line 48, in _run_event_loop
self._event_loop.run_forever()
File "C:\Users\bpkroth\.conda\envs\mlos\lib\asyncio\windows_events.py", line 321, in run_forever
super().run_forever()
File "C:\Users\bpkroth\.conda\envs\mlos\lib\asyncio\base_events.py", line 603, in run_forever
self._run_once()
File "C:\Users\bpkroth\.conda\envs\mlos\lib\asyncio\base_events.py", line 1871, in _run_once
event_list = self._selector.select(timeout)
File "C:\Users\bpkroth\.conda\envs\mlos\lib\asyncio\windows_events.py", line 444, in select
self._poll(timeout)
File "C:\Users\bpkroth\.conda\envs\mlos\lib\asyncio\windows_events.py", line 828, in _poll
f.set_exception(e)
File "C:\Users\bpkroth\.conda\envs\mlos\lib\asyncio\windows_events.py", line 88, in set_exception
super().set_exception(exception)
asyncio.exceptions.InvalidStateError: invalid state
warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg))
```
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
Adds tests for RemoteEnv `setup`, `run`, `teardown` using SshServices
Fixes some associated bugs:
- convert certain `SFTPError` exceptions to `FileNotFound` in
`SshFileShareService`
This is important for `LocalFileShareEnv` integration especially since
it always calls `download` during calls to `status`, but only handles
`FileNotFound` exceptions when `ignore_missing=True`
- Fixups to #517 for `download`, `upload` to connect via `self._params`
(loaded from the `required_args`) instead of `self.config`
Also included:
- basic configs for SshServices and LocalExec for easy inclusion
Closes#521
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
This change makes it so that running tests via VSCode's pytest plugin by
default runs them in parallel (respecting the setup.cfg args for
pytest), but when using the "Debug Test" option, parallel workers are
disabled so we can more quickly trace through code in debug mode without
additional processes to attach to.
Unfortunately, there is a [bug with the pytest-xdist loadgroup output
processing in the
vscode](https://github.com/microsoft/vscode-python/issues/19374) that
prevents a few tests that use that (namely the recently added ssh test)
from displaying as "completed successfully", even though they did.
- Adds SSH support for remote exec and file copy operations via
`asyncssy` library. Closes#379
- This requires an event loop thread to operate the async calls in the
background without blocking the rest of our main thread operations or
changing our APIs dramatically. Instead we interact with all operations
as futures after that.
- Add json schema config support for SSH
- Adds test infrastructure for running SSH servers inside containers via
`pytest-docker` which uses `docker compose` to start/stop them as
fixtures. This *should* work in all dev environments (Linux host, WSL
host, Windows host), but is quite sensitive to networking setups.
- [x] test the background thread cleanup logic
- [x] test error handling of a broken ssh endpoint
- [x] test basic exec commands
- [x] test host operations (e.g., reboot)
- [x] test basic copy commands
- [x] test for single connection reuse across services
- [x] test multi-server support
- [x] use random ports to avoid conflicts
- provide some config examples for using this
- add remote_env exec tests with this (increase code coverage)
- Will do these in a future PR instead - #521
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
Fixes a bug with mock responses not providing a `json()` method which is
only called during debug logging.
This was somewhat randomly showing up based I think upon the order of
the tests running.
One of them, seemed to be enabling `_LOG.setLevel(logging.DEBUG)` so
that future tests (for that worker) would also have the level set.
Once done, additional paths would be followed inside the `AzureServices`
code that caused it to try and do something like the following
`_LOG.debug(response.json())` which would fail because the mocked
response object didn't provide a `json()` method response.
This fixes that error, but doesn't address the `debug` log ordering.
I'm not certain we should switch all of our unit tests over to running
in DEBUG mode by default since it might cause us to depend on that
behavior.
As a compromise, for now I've enabled it only inside
`.vscode/settings.json` so that when we run `pytest` interactively
(e.g., during dev cycle or while debugging a single unit test), then
those code paths will be followed.
We can also run the following on demand and incorporate it to our
`Makefile` or nightly runs to do it at least some of the time:
```sh
pytest --log-level=DEBUG
```
or
```sh
export PYTEST_EXTRA_OPTIONS=--log-level=DEBUG
make test
```
- Support multiple instances of `--globals` and other arguments to build
a list.
- Provide some common mistype aliases for CLI args.
- [x] Adds some tests (additional ones to be added later)
See Also testing comments in #490
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
- Workaround a bug in SMAC that builds additional_configs across
instantiations.
- [ ] Look for more of these cases (maybe that's where the
non-determinism is coming from?)
- [x] Report it upstream: https://github.com/automl/SMAC3/pull/1067
- Adds more assertions to help check for related issues
- Addresses a TODO sanity check
- Adds some test code for FLAML's surrogate model by increasing
iterations
- Updates tests for easier tracing
Note: This still doesn't address all of the non-determinism across
platforms/python versions we've noticed recently (#482), and there's
some question yet about how much runhistory and some other features are
being handled correctly, but this still handles a few small improvements
of other issues I've noticed.
For the rest I've filed issues:
- #487
- #488
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
The recommended autopep8 extension doesn't seem to be working for us:
https://github.com/microsoft/vscode-autopep8/issues/16
So, for now I'm reverting to the old way of doing things.
Also, the previous extension relied on a newer version of autopep8 than
conda was installing anyways, so I switched to installing it from pip.
In the subsequent PRs, we will port the tunables' metadata and Linux
kernel configuration scripts that use it from our private repo.
A PR with unit tests will also follow (turns out, we don't have *any*
tests for `LocalEnv` atm).
- Fixes#381 to add nested inline CompositeEnv config support
- Enforces uniqueness on some more arrays.
- Cleanup some old comments.
- Add support for dynamic `$param_name` substitution in `const_args`
- [x] Add additional tests
Extends #340, #346, #349, #352, #359 to Service configs.
See Also: #331
- [x] main schema as a set of subschemas
- [x] connect schema validation in the mlos_bench code
- [x] test-cases
To do this we removed support for configs that were flat lists of service configs: `[ {"class": "service.class"} ]` and turned those into objects that have a single key `"services"` with the list instead. `{ "services": [ ... ] }`
This makes sure that all configs are dicts/objects that we can optionally put `"$schema"` attributes on but also simplifies some of the config loading logic.
Extends #340, #346, #349, #352 to Environment configs.
See Also: #331
- [x] main schema as a set of subschemas
- [x] connect schema validation in the mlos_bench code
- [x] test-cases
Extends #340, #346, #349 to cli configs.
See Also: #331
Also updates `--log_level` to support named logging levels (e.g. `INFO`)
- [x] add tests
- [x] also backfilling tests of loading cli config examples (similar to what #313 did)
- [x] global config handling
* Disable line buffering
* fail pylint checks on unused-import errors
* make sure pycodestyle checks for indentation
* increase code quality requirements
* add some new extensions
* bring pycodestyle checks inline with flake8
* add a prettier rc that instructs it to use our editorconfig
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
This PR introduces initial json schemas for mlos_bench optimizer configs and tunable_values configs, their validation at load time, and tests for both.
Future PRs will handle other config types.
See Also: #331
Adds static type checking via mypy for mlos_bench as well.
Builds on #305, #307
- [x] Add `Protocol`s for different `Service` types so `Environment`s can check that the appropriate `Service` mix-ins have been setup.
- [x] Config Loader
- [x] Local Exec
- [x] Remote Exec
- [x] Remote Fileshare
- [x] VM Ops
- [ ] Future PR: Split VM Ops to VM/Host and OS Ops (for local SSH support)
Attempts to workaround some upstream release version mismatch issues for now so we can build the devcontainer in the pipelines.
See Also: <https://github.com/conda/conda-libmamba-solver/issues/169#issuecomment-1490509810>
* fixups for missing extension id (github made it official)
* workaround an upstream conda release bug
Separating the Windows and Linux jobs into two different pipelines so we can make the Windows one optional since it takes a lot more time than the others and we don't care about it as much now that we have devcontainer support.
Note: once this is in, I will need to make a couple of changes on the Azure DevOps side to make use of it, so these checks won't run for this PR.
This PR was originally aimed at just fixing the intermittent documentation build errors.
That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency.
Along the way I found a handful of other improvements for build, pipeline, and documentation generation things.
- Make sure we do the nightly build regardless of whether code has changed
- Don't build the docs in parallel by default (trying to avoid an intermittent build failure)
- Add basic support for testing devcontainer manually outside for vscode
- Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer
- Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc
- Adjust the devcontainer build so that more dependencies are built upfront during the container build.
This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch.
- Import part of a fix from elsewhere
- Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs
- Update some docs and doc building
- [X] pydocstyle fixups
- [X] Add some log filtering of the `make doc` output for easier debugging
(e.g. don't care about missing examples for now)
- [X] doc link lint checks
- DONE
- [X] Split this out into multiple PRs
- [X] SpaceAdapterFactory: !959
- [X] docstyle fixups: !960
- [ ] doc build fixups
- [ ] devcontainer improvements
(leaving the last two as this PR)
This PR is about general dev environment improvements.
Adds devcontainer support to help automatically prepare a dev environment inside either vscode of github codespaces.
The container is build, tested, and published to an ACR in the MlosCore RG during the pipeline for easy reuse and quick dev env boot strapping and reproducible environments both locally and remotely.
It also
- Enables parallel Makefile builds
- Adds additional helpful extensions to the vscode (e.g. copilot)
- Moves some pip package installs to conda for better reproducibility
- Fixes some issues regarding PYTHONPATH adjustment requirements for pytest and pylint.
- And, since I noticed a config for it in vscode, I added pydocstyle checks, though they aren't configured for blocking atm since there's lots of complaints. Can decide whether to fix/force those later.
- include all python files in pylint checks
- improve version handling for pylint and update scripts
- make autopep8 available for vscode to use as a python formatter
- add more cspell words
- address some pylint warnings
- more pylint fixups
We will do our OS autotuning related work on mlos-bench there.
Main prior to this has been tagged as v0.0.3 for servicing Spark autotune needs if need be.
Plan is to reorganize the source tree after this so we can generate separate `mlos-core` and `mlos-bench` packages.
Adds basic `register` and `suggest` APIs for a couple of optimizer backends: `Skopt`, `Emukit`, `Random`.
Includes example notebook for starters.
Limited unit testing. Basic function documentation.
Azure DevOps CI pipelines for `pylint` and `pytest` via the `mlos_core` `conda` `environment.yml`.
Related work items: #274, #279
Brian Kroth
Sergiy Matusevych
Brian Kroth
Andreas Mueller