Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of
`ExperimentData` results relative to the experiment's objectives
(building off of #628 and https://github.com/dabl/dabl/pull/335).
Note: this PR currently omits unit tests for the new module due to the
complexity of testing visualizations. We intend to add this in future
PRs. There is however, a working example of its use here right now:
https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41
---------
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
- Adds SSH support for remote exec and file copy operations via
`asyncssy` library. Closes#379
- This requires an event loop thread to operate the async calls in the
background without blocking the rest of our main thread operations or
changing our APIs dramatically. Instead we interact with all operations
as futures after that.
- Add json schema config support for SSH
- Adds test infrastructure for running SSH servers inside containers via
`pytest-docker` which uses `docker compose` to start/stop them as
fixtures. This *should* work in all dev environments (Linux host, WSL
host, Windows host), but is quite sensitive to networking setups.
- [x] test the background thread cleanup logic
- [x] test error handling of a broken ssh endpoint
- [x] test basic exec commands
- [x] test host operations (e.g., reboot)
- [x] test basic copy commands
- [x] test for single connection reuse across services
- [x] test multi-server support
- [x] use random ports to avoid conflicts
- provide some config examples for using this
- add remote_env exec tests with this (increase code coverage)
- Will do these in a future PR instead - #521
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
The recommended autopep8 extension doesn't seem to be working for us:
https://github.com/microsoft/vscode-autopep8/issues/16
So, for now I'm reverting to the old way of doing things.
Also, the previous extension relied on a newer version of autopep8 than
conda was installing anyways, so I switched to installing it from pip.
Unpins some dependencies at older versions.
Also fixes up a few more DeprecationWarnings.
Closes#401Closes#429Closes#295
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Removes Emukit to avoid GPy issues. Closes#426
Switches most things to SMAC instead.
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
- Split out and fixup the build and dist-test script for Windows.
- Make sure conda isn't installing mismatch conda vs. pip versions.
- Add additional logs to help debug issues in the future.
* Disable line buffering
* fail pylint checks on unused-import errors
* make sure pycodestyle checks for indentation
* increase code quality requirements
* add some new extensions
* bring pycodestyle checks inline with flake8
* add a prettier rc that instructs it to use our editorconfig
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
This PR introduces initial json schemas for mlos_bench optimizer configs and tunable_values configs, their validation at load time, and tests for both.
Future PRs will handle other config types.
See Also: #331
* force function definitions to have type decorations
* type annotations
* ignore errors on listening to port 81 from 127.0.0.1 so we can run two devcontainer instances at a time
* more type check fixups
* ignore missing matplotlib stubs
* more typing
* spelling
* annotate test functions
* spelling
* types
* spelling
* more typing
* more typing fixups
* more strict
* more typing fixups
* fixups
* more
* spaces
* merge conflict fixups
* more merge conflicts
* fixup
* back that change out
* minor tweaks for unimplemented function
* fixups for py 3.89
* fixups for newer version of mypy
* minor tweaks to dmypy script
* update
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
* adding configs to be able to use bumpversion for managing versions
* Bump version: 0.0.4 → 0.1.0
* cleanup old file
* don't need to reinstall licenseheaders everytime if we include it in the conda env
* github actions checks out in shallow mode which setuptools-scm doesn't like, but doesn't cause issues, so we ignore the warning
- make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally
- ensures that the full set still runs in the pipeline though
- add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster
- optionally copy the code coverage data into our static html site for publishing (plus nice badges)
- also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier
- also includes which job the test came from in the results
- some other general `Makefile` cleanup
Future work:
- [X] Split the single pipeline into two pipelines and make the Windows one optional
(since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support)
!1040
This PR was originally aimed at just fixing the intermittent documentation build errors.
That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency.
Along the way I found a handful of other improvements for build, pipeline, and documentation generation things.
- Make sure we do the nightly build regardless of whether code has changed
- Don't build the docs in parallel by default (trying to avoid an intermittent build failure)
- Add basic support for testing devcontainer manually outside for vscode
- Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer
- Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc
- Adjust the devcontainer build so that more dependencies are built upfront during the container build.
This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch.
- Import part of a fix from elsewhere
- Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs
- Update some docs and doc building
- [X] pydocstyle fixups
- [X] Add some log filtering of the `make doc` output for easier debugging
(e.g. don't care about missing examples for now)
- [X] doc link lint checks
- DONE
- [X] Split this out into multiple PRs
- [X] SpaceAdapterFactory: !959
- [X] docstyle fixups: !960
- [ ] doc build fixups
- [ ] devcontainer improvements
(leaving the last two as this PR)
This PR is about general dev environment improvements.
Adds devcontainer support to help automatically prepare a dev environment inside either vscode of github codespaces.
The container is build, tested, and published to an ACR in the MlosCore RG during the pipeline for easy reuse and quick dev env boot strapping and reproducible environments both locally and remotely.
It also
- Enables parallel Makefile builds
- Adds additional helpful extensions to the vscode (e.g. copilot)
- Moves some pip package installs to conda for better reproducibility
- Fixes some issues regarding PYTHONPATH adjustment requirements for pytest and pylint.
- And, since I noticed a config for it in vscode, I added pydocstyle checks, though they aren't configured for blocking atm since there's lots of complaints. Can decide whether to fix/force those later.
- add nightly builds so we can catch issues with upstream changes sooner
- allow reading conda envs out of the environment
- handle only setting one of CONDA_EXPERIMENTAL_SOLVER and CONDA_SOLVER envs
- adjust conda envs to avoid python 3.11 for now
- adjust Makefile to use the same version of python from the current CONDA_ENV for our test environment
(this is quick way to respect that version constraint and only specify it in one place)
- adjust the expected version of configspace
- remove configspace from conda env file
(upstream doesn't have the version we need outside of conda-forge)
- move all other version restrictions to setup.py files
- add a 'full' mode for installing all extra_requires for each package
- use that in the conda envs instead
Related to additional build/test issues discovered in !850
This is a final version of the benchmark stack that integrates both local (scheduler-side) and remote environments. Summary of changes:
* Implement `LocalEnv` to run custom scripts locally and exchange the data between the script and the framework.
* Implement `LocalFileShareEnv` that runs the scripts locally and uploads/downloads data to and from the shared storage.
* Refactor `AppEnv` into `RemoteEnv` to run the scripts on the remote VM; move the post-processing functionality (that runs locally) into `LocalEnv`.
* Allow stacking of multiple environments in one config file. This way we can have a single file for several environments that always go together (e.g., a sequence of: generate data locally -> upload to shared storage -> apply the configuration on a remote VM -> run the benchmark -> download the results -> process the data locally).
* Implement `require` keyword for `const_args` that are defined externally (e.g., the security credentials).
* Update the benchmarking scripts to work with the new setup.
* Minor improvements:
* Use `JSON5` and `.jsonc` extension for our config files (so that we can have comments in JSON!)
* Actually add comments to our JSON configs :)
* Introduce the `--log-level` option and set the default logging level to `INFO`
* Small fixes to logging, docstrings, and such
Related work items: #600
Wrap up interactions with the Azure file share as a Service to use within mlos_bench.
Includes methods for uploading and downloading folders to / from the file share.
Related work items: #599
- include all python files in pylint checks
- improve version handling for pylint and update scripts
- make autopep8 available for vscode to use as a python formatter
- add more cspell words
- address some pylint warnings
- more pylint fixups
This is a follow up to !557 (Make conda work on Windows) and addresses some of the lingering comments from that PR.
- document the mamba slow solver workaround
- use the mamba solver if available in Makefile
- add windows jobs to the CI tasks as well
- Add conda env caching for build pipeline speed ups.
- Cache will use items from the last hour (within the same day), but only if the `environment.yml` and `setup.yml` file haven't changed.
- Adds testing for multiple Python versions.
- Mostly as a proof of concept. Right now just 3.8 and 3.9. 3.6 (the current default in Synapse) is broken and disabled.
- We can disable this and just use the `DefaultPython` version again for now as well.
- Also enabled parallel builds as a part of this change.
Related work items: #322
Brian Kroth
Konstantinos Kanellis
Brian Kroth
Sergiy Matusevych
Eu Jing Chua