# Pull Request
## Title
Add support for Python 3.13 and mark is as the expected default.
---
## Description
Upstream changed the default version of python, so this addresses
changes in checks in the pipelines.
So far it seems to "just work (tm)"
---
## Type of Change
- 🛠️ Bug fix
- ✨ New feature
- 🧪 Tests
---
## Testing
Usual CI tests.
---
Avoids the following error that results when the [latest
libpq](https://anaconda.org/conda-forge/libpq) (17.0) is installed from
conda-forge:
```
Error: pg_config --includedir is empty
```
This PR enables Python 3.12 in Windows environment and does not restrict
Python version in the default environments on Windows and Linux. Note
that although `mlos-3.12.yml` exists, its usage is still commented out
in `linux.yml`, because Python 3.12 is still not available on GitHub CI.
Issue #832 opened to track it
P.S. Also, we now use a lighter `matplotlib-base` instead of
`matplotlib` proper
~To be merged after #802~
---------
Co-authored-by: Brian Kroth <bpkroth@microsoft.com>
Temporary fix to conda env to make it match the `pip` `setup.py`
restrictions for now.
Eventually, we'll wan to relax these, along with increased `numpy`
versions, etc.
See Also #826
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
Summary of changes:
* [x] Update dependencies to ConfigSpace >= 1.0.
* [x] Update dependencies to SMAC3 >= 2.2.
* [x] Migrate to new ConfigSpace API.
* [x] Better definition of ConfigSpace fixture in unit tests.
* [x] Fix mypy and pylint issues triggered by the migration.
Notes:
* Side effect: **Works with Python 3.12!**
* ~Supersedes #799~
* ~Depends on upcoming SMAC3 release with [SMAC3
#1124](https://github.com/automl/SMAC3/pull/1124) merged in and a new
release cut with it.~
* Quantization will be added back later:
* #803
* ~Tests fail due to LlamaTune issue:~
* #805Closes#727
---------
Co-authored-by: Brian Kroth <bpkroth@users.noreply.github.com>
Co-authored-by: Brian Kroth <bpkroth@microsoft.com>
Follow on work to #766.
This enabled both formatters and applies their changes to the repo.
Additionally, since `black` does not make changes to comments nor
docstrings, we also enable `docformatter` to reformat docstrings which
better aligns with `pydocstyle` rules as well.
Without this additional change (and some manual fixups), `pycodestyle`
and `pylint` would still complain about line lengths, for instance.
Finally, we make a minor adjustment to the max line length setting it to
99 (which is also accepted and mentioned in pep8) instead of 88 to avoid
some comment (especially linter overrides) wrapping.
Builds off of #762, #763, and #764.
Prepares rules and configs to enable isort and black formatters and
checks but doesn't enable them yet.
After these are enabled (next PR) we will reformat all files and ignore
that revision in git blame configs.
Then, we can convert configs stored in `setup.cfg` and `.pylintrc` to
the top level `pyproject.toml` and remove the older configs.
This PR builds off of #762 and #763
These are in part
1. followup fixups for #746 (e.g., to allow setuptools-scm to be pulled
in at build time as a build dependency only when conda an pip have
mismatched version issues), and
2. Modernization improvements to allow us to make better use of other
tools (e.g., `black` that only accept `pyproject.toml` files as their
configuration files).
To do so, we move some configs from `setup.py` to `pyproject.toml` for
each module.
However, to retain the ability to rewrite URLs in published README.md
files on PyPi as well as consistent version dependencies across modules
without the need to manually specify version numbers (e.g., using
`setuptools-scm`) we mark a few dependencies as dynamic and leave our
existing logic inside the `setup.py` file.
Finally, we reorganize the `version.py` file to be inside the module and
fix a few previous omissions for `mlos_viz`.
A new git version outputs a different date format that older versions of
python's datetime module don't recognize.
https://github.com/pypa/setuptools_scm/issues/1038
setuptools-scm cut a new version to address that, but it's not available
in conda main channel yet, which is required, since without that the
conda pip phase can't execute in a single transaction
For now, we install things via conda-forge and adjust the channel
priority order so that the full set of dependencies could be resolved.
That of course brought in additional changes (e.g., `python=3.12` by
default, new `pylint`, `pycodestyle`, etc.), so this now also includes
some additional linting changes.
However, longer term, we need to switch to a pyproject.toml file to fix
this properly.
There we can specify prereqs for even loading the setup.py as well as
fix some other config complexities, though that is a broader change.
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
- Update the base image and make sure related dependencies are
consistently from either pip or conda, but not both (follow on fixes to
#699)
- Minor tweak to include the PATH in the local script exec environment
variables
- Another fix to the CI issue in #703
Simply makes sure that pyarrow is available in the conda environment to
make some warnings go away.
Assumes that future versions of pandas will declare their own dependency
on pyarrow and this can be removed.
Also includes changes from #680
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Fixups to #656 by switching to a new plugin and alerting to remove the
old one.
Includes changes from #680
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Adds a basic `mlos_viz.plot(exp)` style API for simple visualizations of
`ExperimentData` results relative to the experiment's objectives
(building off of #628 and https://github.com/dabl/dabl/pull/335).
Note: this PR currently omits unit tests for the new module due to the
complexity of testing visualizations. We intend to add this in future
PRs. There is however, a working example of its use here right now:
https://github.com/Microsoft-CISL/sqlite-autotuning/pull/41
---------
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
- Adds SSH support for remote exec and file copy operations via
`asyncssy` library. Closes#379
- This requires an event loop thread to operate the async calls in the
background without blocking the rest of our main thread operations or
changing our APIs dramatically. Instead we interact with all operations
as futures after that.
- Add json schema config support for SSH
- Adds test infrastructure for running SSH servers inside containers via
`pytest-docker` which uses `docker compose` to start/stop them as
fixtures. This *should* work in all dev environments (Linux host, WSL
host, Windows host), but is quite sensitive to networking setups.
- [x] test the background thread cleanup logic
- [x] test error handling of a broken ssh endpoint
- [x] test basic exec commands
- [x] test host operations (e.g., reboot)
- [x] test basic copy commands
- [x] test for single connection reuse across services
- [x] test multi-server support
- [x] use random ports to avoid conflicts
- provide some config examples for using this
- add remote_env exec tests with this (increase code coverage)
- Will do these in a future PR instead - #521
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Co-authored-by: Sergiy Matusevych <sergiy.matusevych@gmail.com>
The recommended autopep8 extension doesn't seem to be working for us:
https://github.com/microsoft/vscode-autopep8/issues/16
So, for now I'm reverting to the old way of doing things.
Also, the previous extension relied on a newer version of autopep8 than
conda was installing anyways, so I switched to installing it from pip.
Unpins some dependencies at older versions.
Also fixes up a few more DeprecationWarnings.
Closes#401Closes#429Closes#295
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
Removes Emukit to avoid GPy issues. Closes#426
Switches most things to SMAC instead.
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
- Split out and fixup the build and dist-test script for Windows.
- Make sure conda isn't installing mismatch conda vs. pip versions.
- Add additional logs to help debug issues in the future.
* Disable line buffering
* fail pylint checks on unused-import errors
* make sure pycodestyle checks for indentation
* increase code quality requirements
* add some new extensions
* bring pycodestyle checks inline with flake8
* add a prettier rc that instructs it to use our editorconfig
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
This PR introduces initial json schemas for mlos_bench optimizer configs and tunable_values configs, their validation at load time, and tests for both.
Future PRs will handle other config types.
See Also: #331
* force function definitions to have type decorations
* type annotations
* ignore errors on listening to port 81 from 127.0.0.1 so we can run two devcontainer instances at a time
* more type check fixups
* ignore missing matplotlib stubs
* more typing
* spelling
* annotate test functions
* spelling
* types
* spelling
* more typing
* more typing fixups
* more strict
* more typing fixups
* fixups
* more
* spaces
* merge conflict fixups
* more merge conflicts
* fixup
* back that change out
* minor tweaks for unimplemented function
* fixups for py 3.89
* fixups for newer version of mypy
* minor tweaks to dmypy script
* update
---------
Co-authored-by: Sergiy Matusevych <sergiym@microsoft.com>
* adding configs to be able to use bumpversion for managing versions
* Bump version: 0.0.4 → 0.1.0
* cleanup old file
* don't need to reinstall licenseheaders everytime if we include it in the conda env
* github actions checks out in shallow mode which setuptools-scm doesn't like, but doesn't cause issues, so we ignore the warning
- make test and check targets for mlos_core and mlos_bench independent for faster execution and more clear output for `make test` and `make check` iteration when only hacking on one module locally
- ensures that the full set still runs in the pipeline though
- add support for skipping code coverage locally by setting `SKIP_COVERAGE=true` to run tests faster
- optionally copy the code coverage data into our static html site for publishing (plus nice badges)
- also, rather than racing and publishing the test results and code coverage for all of the different sub jobs, we only run and upload coverage via `pytest-azurepipelines` for the main one we're interested in to make reporting easier
- also includes which job the test came from in the results
- some other general `Makefile` cleanup
Future work:
- [X] Split the single pipeline into two pipelines and make the Windows one optional
(since it's usually the one that takes the longest and we care a little bit less about it now that we have decontainer support)
!1040
This PR was originally aimed at just fixing the intermittent documentation build errors.
That issue ultimately stems from conflicts between conda and pip installs of two different versions of `mistune` dependency.
Along the way I found a handful of other improvements for build, pipeline, and documentation generation things.
- Make sure we do the nightly build regardless of whether code has changed
- Don't build the docs in parallel by default (trying to avoid an intermittent build failure)
- Add basic support for testing devcontainer manually outside for vscode
- Stop doing parallel builds so it's easier to debug, and move doc build/publishing to the devcontainer
- Split windows conda env off, stop using gpy from conda-forge, and just build it locally with clang, gcc, or msvc
- Adjust the devcontainer build so that more dependencies are built upfront during the container build.
This makes the process more cacheable so that we can just pull the upstream version and not have to build as much locally which could delay first launch.
- Import part of a fix from elsewhere
- Switch to docker-compose style devcontainer that automatically also starts an nginx container for viewing the generated docs
- Update some docs and doc building
- [X] pydocstyle fixups
- [X] Add some log filtering of the `make doc` output for easier debugging
(e.g. don't care about missing examples for now)
- [X] doc link lint checks
- DONE
- [X] Split this out into multiple PRs
- [X] SpaceAdapterFactory: !959
- [X] docstyle fixups: !960
- [ ] doc build fixups
- [ ] devcontainer improvements
(leaving the last two as this PR)
This PR is about general dev environment improvements.
Adds devcontainer support to help automatically prepare a dev environment inside either vscode of github codespaces.
The container is build, tested, and published to an ACR in the MlosCore RG during the pipeline for easy reuse and quick dev env boot strapping and reproducible environments both locally and remotely.
It also
- Enables parallel Makefile builds
- Adds additional helpful extensions to the vscode (e.g. copilot)
- Moves some pip package installs to conda for better reproducibility
- Fixes some issues regarding PYTHONPATH adjustment requirements for pytest and pylint.
- And, since I noticed a config for it in vscode, I added pydocstyle checks, though they aren't configured for blocking atm since there's lots of complaints. Can decide whether to fix/force those later.
- add nightly builds so we can catch issues with upstream changes sooner
- allow reading conda envs out of the environment
- handle only setting one of CONDA_EXPERIMENTAL_SOLVER and CONDA_SOLVER envs
- adjust conda envs to avoid python 3.11 for now
- adjust Makefile to use the same version of python from the current CONDA_ENV for our test environment
(this is quick way to respect that version constraint and only specify it in one place)
- adjust the expected version of configspace
- remove configspace from conda env file
(upstream doesn't have the version we need outside of conda-forge)
- move all other version restrictions to setup.py files
- add a 'full' mode for installing all extra_requires for each package
- use that in the conda envs instead
Related to additional build/test issues discovered in !850
This is a final version of the benchmark stack that integrates both local (scheduler-side) and remote environments. Summary of changes:
* Implement `LocalEnv` to run custom scripts locally and exchange the data between the script and the framework.
* Implement `LocalFileShareEnv` that runs the scripts locally and uploads/downloads data to and from the shared storage.
* Refactor `AppEnv` into `RemoteEnv` to run the scripts on the remote VM; move the post-processing functionality (that runs locally) into `LocalEnv`.
* Allow stacking of multiple environments in one config file. This way we can have a single file for several environments that always go together (e.g., a sequence of: generate data locally -> upload to shared storage -> apply the configuration on a remote VM -> run the benchmark -> download the results -> process the data locally).
* Implement `require` keyword for `const_args` that are defined externally (e.g., the security credentials).
* Update the benchmarking scripts to work with the new setup.
* Minor improvements:
* Use `JSON5` and `.jsonc` extension for our config files (so that we can have comments in JSON!)
* Actually add comments to our JSON configs :)
* Introduce the `--log-level` option and set the default logging level to `INFO`
* Small fixes to logging, docstrings, and such
Related work items: #600
Wrap up interactions with the Azure file share as a Service to use within mlos_bench.
Includes methods for uploading and downloading folders to / from the file share.
Related work items: #599
- include all python files in pylint checks
- improve version handling for pylint and update scripts
- make autopep8 available for vscode to use as a python formatter
- add more cspell words
- address some pylint warnings
- more pylint fixups
This is a follow up to !557 (Make conda work on Windows) and addresses some of the lingering comments from that PR.
- document the mamba slow solver workaround
- use the mamba solver if available in Makefile
- add windows jobs to the CI tasks as well
- Add conda env caching for build pipeline speed ups.
- Cache will use items from the last hour (within the same day), but only if the `environment.yml` and `setup.yml` file haven't changed.
- Adds testing for multiple Python versions.
- Mostly as a proof of concept. Right now just 3.8 and 3.9. 3.6 (the current default in Synapse) is broken and disabled.
- We can disable this and just use the `DefaultPython` version again for now as well.
- Also enabled parallel builds as a part of this change.
Related work items: #322
Brian Kroth
Sergiy Matusevych
Konstantinos Kanellis
Brian Kroth
Eu Jing Chua